Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World
Researchers introduce Spatial-Interactor, a three-level curriculum that trains vision-language models to track physical state transitions.
Spatial-Interactor trains vision-language models to track physical-world state transitions using interaction trajectories rather than static spatial questions. The LSI-108K dataset, built from simulated and real trajectories, supports a three-level curriculum covering passive world-state transitions, active self-state transitions, and long-horizon trajectories. Supervised fine-tuning models local transitions, then privileged on-policy distillation teaches the student to integrate consecutive transitions. Experiments across multiple VLMs report consistent gains in local transition modeling and long-horizon spatial integration.
- LSI-108K pairs simulated and real interaction trajectories with level-aligned tasks.
- Curriculum covers passive transitions, active self-motion, and long-horizon trajectories.
- SFT trains local transitions; on-policy distillation integrates long trajectories.
From the original article
1 min read at huggingface.co
Spatial reasoning is essential for vision-language models (VLMs) to understand and act in the physical world. Reasoning in dynamic environments requires VLMs to perceive local state transitions caused by object motion and viewpoint changes and integrate them over long trajectories to maintain an updated spatial state, yet existing VLMs remain limited in both capabilities.
By Kaixiang Yao, Xu Wang, Miao Pan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.