SteerDuplex: Steerable Duplex Speech Dialogue Models
SteerDuplex fine-tunes the Moshi full-duplex speech model with two-stage RL and hybrid rewards to steer tone, persona, and pacing, alongside the new SteerBench benchmark.
The paper introduces SteerDuplex, a Moshi-based full-duplex speech dialogue model fine-tuned on natural and synthetic conversations, using two-stage reinforcement learning with hybrid rewards (verifiable interaction checks plus judge-based semantic feedback). It also introduces SteerBench, a benchmark of 390 spoken prompts and 1,067 human-authored binary audio and text rubrics covering tone, persona, style/accent, and speed/length. Supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest open baseline, and Audio MultiChallenge task average improves by 7 points. RL raises clean interruption response from 72.5% to 82.5% and cuts synthetic pause barge-in from 26.5% to 9%, though reward probes reveal reward hacking via incomplete responses.
- Moshi-based full-duplex speech model with instruction-driven control of tone, persona, rate, voice style.
- SteerBench: 390 spoken prompts, 1,067 human-authored binary audio/text rubrics.
- Two-stage RL with hybrid rewards improves timing and response continuity.
- Audio-steering pass rate up 44.5 points over strongest open baseline.
- Reward analysis exposes reward hacking through incomplete responses.
Full article235 words · extracted from huggingface.co · click to collapse
Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate full-duplex spoken steerability, we introduce SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style/accent, and speed/length. On SteerBench, supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest evaluated open baseline. On Audio MultiChallenge, task average pass rate improves by 7 points over its strongest evaluated open baseline. RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%. Steering and aggregate task scores remain comparable or higher, while reward probes reveal reward hacking through incomplete responses. Our model and benchmark support systematic research on spoken steerability, with reward analysis showing why timing gains must be evaluated alongside response completeness.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.12623