MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
MP-Bench is the first benchmark for voice agents in multiparty conversations, finding real-time agents near chance on turn-taking.
MP-Bench is the first benchmark designed to objectively evaluate conversational speech systems as active participants in multi-party conversations. It assesses agents on turn-taking awareness and response appropriateness, with comprehension-based question-answering as a complementary evaluation. Benchmarking 12 voice agents shows real-time agents score at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
- First benchmark evaluating voice agents as multiparty conversation participants
- Assesses turn-taking awareness and response appropriateness across 12 agents
- Real-time agents score at or below 22% on multiparty comprehension
From the original article
1 min read at arxiv.orgConversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations.
By Yi-Jen Shih · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.