SteerDuplex: Steerable Duplex Speech Dialogue Models
SteerDuplex fine-tunes the Moshi full-duplex speech model with two-stage RL and hybrid rewards to steer tone, persona, and pacing, alongside the new SteerBench benchmark.
The paper introduces SteerDuplex, a Moshi-based full-duplex speech dialogue model fine-tuned on natural and synthetic conversations, using two-stage reinforcement learning with hybrid rewards (verifiable interaction checks plus judge-based semantic feedback). It also introduces SteerBench, a benchmark of 390 spoken prompts and 1,067 human-authored binary audio and text rubrics covering tone, persona, style/accent, and speed/length. Supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest open baseline, and Audio MultiChallenge task average improves by 7 points. RL raises clean interruption response from 72.5% to 82.5% and cuts synthetic pause barge-in from 26.5% to 9%, though reward probes reveal reward hacking via incomplete responses.
- Moshi-based full-duplex speech model with instruction-driven control of tone, persona, rate, voice style.
- SteerBench: 390 spoken prompts, 1,067 human-authored binary audio/text rubrics.
- Two-stage RL with hybrid rewards improves timing and response continuity.
- Audio-steering pass rate up 44.5 points over strongest open baseline.
- Reward analysis exposes reward hacking through incomplete responses.
From the original article
1 min read at huggingface.co
Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions.
By Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.