OmniEcho: Spatial Audio Understanding for Embodied Agents
OmniEcho benchmarks spatial audio for embodied agents and reports strong perception and navigation results.
OmniEcho introduces a benchmark and model for spatial audio understanding in embodied agents. OmniEchoBench has six tasks over 197 real scenes, 2,972 question-answer pairs, and 900 navigation samples using first-order ambisonics from 30 environments. A controllable renderer supplies geometrically consistent training audio, and OmniEcho pairs an FOA spatial encoder with a pretrained semantic audio pathway. The model reports state-of-the-art spatial perception and sound-guided navigation close to traditional vision-language navigation, while fine localization and distance estimation remain open.
- OmniEchoBench covers six tasks, 197 scenes, 2,972 QA pairs, and 900 navigation samples.
- Audio is first-order ambisonics from 30 real environments, plus a controllable renderer.
- OmniEcho adds an FOA spatial encoder beside a pretrained semantic audio pathway.
- It leads spatial audio-visual perception and nears vision-language navigation performance.
- Products
- OmniEchoBench
- AI models
- OmniEcho
From the original article
1 min read at huggingface.co
Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings.
By Ruixun Liu, Yuxuan Wang, Jiacheng Xie · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.