VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control
VA-Bench evaluates whether MLLMs can complete embodied observe-reason-act-revise loops; the best model achieves only 53.93% macro-average task success.
VA-Bench measures embodied spatial intelligence through the complete observe-reason-act-revise loop, requiring models to actively select camera viewpoints and issue metric Cartesian commands without privileged poses, oracle trajectories, or learned action heads. The benchmark contains 14 base task families, seven held-out geometry and layout variants, and a long-horizon five-object composition track, with 12 primary model conditions evaluated over physically verified seeds. The best model's three-run macro-average task success is only 53.93%, active camera control raised success from 27.86% to 57.50% in one matched comparison, and no model completed a strict long-horizon episode.
- Tests full observe-reason-act-revise loop without privileged information
- 14 base task families plus seven held-out geometry and layout variants
- Best model achieves 53.93% macro-average task success across three runs
- Active camera control nearly doubles success in a matched comparison
Full article221 words · extracted from huggingface.co · click to collapse
Spatial intelligence requires more than describing object locations. Under incomplete observation, models must identify and acquire missing evidence, interpret it in a common spatial frame, and act on it. We introduce VA-Bench to evaluate the complete observe-reason-act-revise loop. General-purpose MLLMs learn procedural context from RGB-only demonstrations, actively select camera viewpoints, issue metric Cartesian commands, and revise them from execution feedback. Models receive no privileged object poses, oracle trajectories, or learned action heads. A fixed model-agnostic controller executes only model-specified targets. VA-Bench contains 14 base task families (11 single-arm and three dual-arm), seven held-out geometry/layout variants, and a long-horizon five-object composition track. We evaluate 12 primary model conditions in three independent runs over the same 20 physically verified seeds per base task, reporting terminal success, nine trajectory-level behavioral diagnostics, and subtask progress. First, the best-performing model scores 100.0% on target localization and 78.9% on spatial relations in the annotated run. Its three-run macro-average task success is only 53.93+/-3.17%. Second, active camera control significantly improves task success over passive multi-view observation. In one matched comparison, success rises from 27.86% to 57.50%. Third, held-out geometric transfer can reduce task success by over 30 percentage points. No model completes a strict long-horizon episode, despite substantial partial progress. VA-Bench thus tests whether general-purpose MLLMs can turn visual demonstrations and actively acquired evidence into successful embodied action.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.19554