RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment
RoTA-Bench shows vision-language models struggle to recover target 3D projections through successive rotations.
RoTA-Bench evaluates whether vision-language models can turn 2D observations into precise 3D control by rotating an object until its projection matches a target silhouette. An automated pipeline builds 674 solvable tasks, including a stratified 100-task test set, from 2D icons. GPT-5.6-Sol succeeds on only 17%, while 20 undergraduates collectively complete the test set four times with 100% success. At matched states, the model worsens pose alignment while improving silhouette overlap more often than humans, 19.4% versus 10.9%.
- RoTA-Bench contains 674 tasks and a stratified 100-task test set.
- GPT-5.6-Sol succeeds on 17%, while undergraduates collectively reach 100%.
- Models more often improve silhouette overlap while worsening pose alignment.
- Without interaction history, 96.8% of model actions stay at 30-degree steps.
Full article198 words · extracted from huggingface.co · click to collapse
Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.17665