SATE-Bench tests MLLMs on active exploration of occluded scenes
Original title: Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?
The benchmark has 316 scenes and 2,525 multi-turn pairs; GRPO increases view-seeking but lowers accuracy.
Researchers introduce SATE-Bench to test whether multimodal large language models actively seek views when occlusion hides geometry, rather than guessing from a static image. Models start from a severely occluded view and query adjacent viewpoints, scored with a step-penalized efficiency reward. Evaluated models stop early while reporting high confidence and struggle with fine-grained 3D geometry. Fine-tuning with Group Relative Policy Optimization increases exploration but reduces accuracy because models over-explore and fail to fuse multi-view context.
- SATE-Bench includes 316 real-world scenes and 2,525 multi-turn QA pairs.
- Passive multi-view controls are compared with an active exploration setting.
- Matched-view controls attribute the gap mainly to insufficient evidence.
- GRPO increases view-seeking but lowers overall answer accuracy.
- Code and data are at gtc-gh.github.io/SATE-Bench.
From the original article
1 min read at huggingface.co
While Multimodal Large Language Models (MLLMs) excel at zero-shot reasoning, their spatial capabilities are primarily evaluated through static, passive observation. In complex, human-centric scenes, occlusion often hides crucial geometric evidence, and passive benchmarks force models to guess rather than recognize their own visual uncertainty.
By Tianchen Guo, Tengyue Zhang, Chen Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.