Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models
Latent Interface Training improves robot foundation model generalization by constraining visual conditioning, boosting LIBERO-Plus success up to 10.7 points.
The paper identifies vision-action shortcuts where robot policies exploit task-irrelevant visual cues that fail under distribution shift. Latent Interface Training (LIT) first trains an action expert conditioned on language, robot state, and terminal SE(3) end-effector poses without images, then constrains visual input through a pose-supervised latent interface. Across four VLA and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves LIBERO-Plus success by 3.87-10.70 percentage points. Real-world tests show 13.30-16.70 percentage-point gains under unseen cameras, lighting, and distractors.
- Vision-action shortcuts degrade robot policy generalization under distribution shift
- Two-stage training separates goal-directed action learning from visual conditioning
- Tested on Pi0.5, MolmoAct2, FAST-WAM, ImageWAM architectures
- Real-world gains of 13-17 points under camera, lighting, and distractor shifts
From the original article
1 min read at huggingface.co
Robot foundation models achieve strong in-distribution performance but often degrade under visual distribution shifts. When learning to generate actions from pretrained visual representations, models may exploit task-irrelevant visual cues that correlate with demonstrated actions within the training distribution.
By Jianman Lin, Shailesh Shailesh, Zhongyi Luo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.