"a tensor can be on the right device for one consumer and wrong for another" is one of those insights that sounds obvious once said but takes real profiling discipline to actually catch. a reporting tensor bouncing cpu→gpu→cpu while the data was already sitting on gpu is the kind of silent overhead that never shows up in a functional test - the output is correct, its just quietly costing you throughput on every step. splitting autoregressive ids (gpu) from reporting ids (cpu) is the clean fix. -7.1% latency and +14.8% throughput from removing one unnecessary device bounce. this is the unglamorous work that actually makes serving fast @ratishp22
Ratish P (@ratishp22)
1/2 a tensor can be on the right device for one consumer and wrong for another.
while profiling higgs tts async decode in sglang omni, i found a reporting tensor bouncing cpu→gpu→cpu even though the next-step token ids were already available on gpu.
— https://nitter.net/ratishp22/status/2084730613043695893#m