What Did I Just Say? Self-Listening for Full-Duplex Speech Models
Researchers propose Self-Listening, a full-duplex speech approach feeding realized model speech back as input to improve interruption recovery.
Full-duplex spoken language models can listen and speak simultaneously, but asynchronous text generation, speech synthesis, and playback cause mismatches between what a model believes it said and what the user heard. The paper defines the resulting recovery problem as anchor interruption and proposes Self-Listening, which interleaves user speech, model text, and played speech as input streams. The authors also release AnchorSpeech, a benchmark with homogeneous training and test splits tracking which ordered response items were actually spoken. Experiments show self-listening models achieve better anchoring performance than full-duplex baselines.
- Anchor interruption defined as recovery while tracking realized speech output
- Self-Listening feeds played speech back into the model as an input stream
- AnchorSpeech benchmark introduced for structured ordered spoken responses
- Self-listening models outperform full-duplex baselines on anchoring tasks
- Products
- AnchorSpeech
From the original article
1 min read at huggingface.co
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user.
By Xuanning Zhou, Junyi Ao, Xiaotong Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.