What Did I Just Say? Self-Listening for Full-Duplex Speech Models
Researchers propose Self-Listening, a full-duplex speech approach feeding realized model speech back as input to improve interruption recovery.
Full-duplex spoken language models can listen and speak simultaneously, but asynchronous text generation, speech synthesis, and playback cause mismatches between what a model believes it said and what the user heard. The paper defines the resulting recovery problem as anchor interruption and proposes Self-Listening, which interleaves user speech, model text, and played speech as input streams. The authors also release AnchorSpeech, a benchmark with homogeneous training and test splits tracking which ordered response items were actually spoken. Experiments show self-listening models achieve better anchoring performance than full-duplex baselines.
- Anchor interruption defined as recovery while tracking realized speech output
- Self-Listening feeds played speech back into the model as an input stream
- AnchorSpeech benchmark introduced for structured ordered spoken responses
- Self-listening models outperform full-duplex baselines on anchoring tasks
Full article174 words · extracted from huggingface.co · click to collapse
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.05592