Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations
VoxPolyMem's interaction-aware memory and agentic retrieval score 85.0 on multi-party spoken-conversation benchmark VoxPolyBench.
VoxPolyMem is an interaction-aware multimodal memory framework for multi-party spoken conversations, combining incremental speaker identification with interaction memory, fact memory, and participant profiles. Retrieval is treated as sequential decision-making, and Evidence-Gain GRPO assigns round-wise credit for acquiring complementary evidence. It scores 85.0 on the new VoxPolyBench, 23.6 points above the strongest baseline, and 89.6 and 74.4 on Mem-Gallery and H2HMem-Multi, exceeding public memory baselines by more than 8 points each.
- Memory hierarchy stores interactions, facts, and participant profiles across sessions.
- EG-GRPO rewards agents for gathering complementary evidence during retrieval.
- VoxPolyBench score is 85.0, 23.6 points above the best baseline.
- Also leads Mem-Gallery at 89.6 and H2HMem-Multi at 74.4.
- Products
- VoxPolyMemVoxPolyBench
From the original article
1 min read at huggingface.co
Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom.
By Wenxu Jia, Xize Cheng, Zihan Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.