Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
VGBench shows audio LLMs rarely withhold tool use when a spoken command shifts to a bystander.
The paper introduces VGBench, a 1,018-item benchmark of whether voice agents withhold action across side-talk, self-talk, and speaker-switch scenarios. Each item shares an action space of silence, a tool call, or a natural-language answer, with speaker-switch pairs holding the words fixed while changing source and distance. Six raw audio LLMs and three training-free adaptations often selected the right tool but rarely stayed silent; the best raw switch mute rate was 14%. Supervised VoxGate training muted 91.3% of switched commands while choosing the correct tool for all nearby wearer commands, and an exploratory GRPO stage modestly improved side-talk accuracy and self-talk muting.
- VGBench has 1,018 items covering side-talk, self-talk, and speaker switches.
- Best raw model muted only 14% of wearer-to-bystander command switches.
- VoxGate supervised training muted 91.3% of switched commands.
- GRPO raised side-talk accuracy from 68.4% to 70.9% and self-talk muting from 52% to 60%.
- Results measure multi-cue acoustic gating, not speaker identity alone.
From the original article
1 min read at huggingface.co
Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer.
By Yanjie Zhang, Nanchen Hu, Yushi Sun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.