Speculative Listening speeds speech models using idle compute
Original title: Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models
It achieves 2.6-2.8x speedup on speak chunks and reduces delay by 73%.
Chunk-interleaved full-duplex speech language models process audio in chunks with asymmetric workload. Using idle compute during listen phases, Speculative Listening drafts next chunks under silence and verifies with audio. The framework is training-free and lossless, reducing decoding delay by 73% on TickTock-Bench.
- Next audio chunk unchanged in 95.3% of chunks
- 88% of output changes start at first token
- Maximum lag reduced by 94.4%
- Delays approach 200 ms human turn gap
- Organizations
- Hugging Face
- AI models
- speech language models
From the original article
1 min read at huggingface.co
Chunk-interleaved full-duplex speech language models listen and speak in fixed time chunks, and each chunk must be processed before the next one arrives. Their workload is asymmetric across chunks: listen chunks leave compute idle while awaiting audio, and speak and tool-call chunks require serial generation, which delays responses and tool calls.
By Yutong Bian, Yanglin Zhang, Yuxiang Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.