VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control
VA-Bench evaluates whether MLLMs can complete embodied observe-reason-act-revise loops; the best model achieves only 53.93% macro-average task success.
VA-Bench measures embodied spatial intelligence through the complete observe-reason-act-revise loop, requiring models to actively select camera viewpoints and issue metric Cartesian commands without privileged poses, oracle trajectories, or learned action heads. The benchmark contains 14 base task families, seven held-out geometry and layout variants, and a long-horizon five-object composition track, with 12 primary model conditions evaluated over physically verified seeds. The best model's three-run macro-average task success is only 53.93%, active camera control raised success from 27.86% to 57.50% in one matched comparison, and no model completed a strict long-horizon episode.
- Tests full observe-reason-act-revise loop without privileged information
- 14 base task families plus seven held-out geometry and layout variants
- Best model achieves 53.93% macro-average task success across three runs
- Active camera control nearly doubles success in a matched comparison
From the original article
1 min read at huggingface.co
Spatial intelligence requires more than describing object locations. Under incomplete observation, models must identify and acquire missing evidence, interpret it in a common spatial frame, and act on it. We introduce VA-Bench to evaluate the complete observe-reason-act-revise loop.
By Zhongbo Zhang, Jiayi Jin, Yifan Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.