MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents
MTVA-Bench evaluates language models inside cascaded voice agents across 490 scenarios and 7 languages, exposing 24.4-point score gaps among seven tested models.
MTVA-Bench is a multi-turn benchmark that evaluates the language model inside cascaded voice-agent systems (ASR, LLM, TTS) under realistic phone-call conditions such as transcription errors, split caller audio, and language/script constraints. It covers 49 agents across 490 reviewed scenarios in 7 languages, combining deterministic tool-call checks with two LLM judges that must cite transcript messages. In a seven-model study, six models chose correct tools within 6.4 points of each other, but overall scores spanned 24.4 points, driven by argument values, action ordering, and rule compliance.
- Evaluates LLMs under realistic cascaded voice-agent conditions
- 49 agents, 490 reviewed scenarios, 7 languages, mock backend tools
- Dual LLM judges must cite specific transcript messages
- Tool selection is similar across models yet overall scores span 24.4 points
- Products
- MTVA-Bench
- AI models
- MTVA-Bench
From the original article
1 min read at arxiv.orgGenerally, most voice agents are cascaded systems, i. e. , an ASR model transcribes the caller's audio, a language model reads the transcript and decides what to say and which backend tools to call, and a TTS model speaks the reply.
By Pritish Mishra · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.