Activation study finds VLA motor programs ignore shifted scenes
Original title: Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Injected task-A activations made pi0.5 replay that trajectory in 99.6% of 1,968 episodes.
A mechanistic study reports that a fine-tuned vision-language-action policy binds motor programs to absolute workspace coordinates rather than the visible scene. On a physical UR10e, directed activation replay redirected the commanded task, and moving the source object 10–15 cm still produced grasps at recorded coordinates. Interventions across six VLAs and more than 420,000 rollouts separate goal selection from motor generation. The authors released Action Atlas.
- Directed UR10e replay succeeded in 26 of 26 rollouts one way and 18 of 26 the other.
- After a 10–15 cm move, recorded coordinates were used in 50 of 50 trials.
- Four of five architectures weighted the instruction below vision when the scene identified the goal.
- Action Atlas covers more than 520 trained SAEs and 79 identified concepts.
- Products
- UR10eAction Atlas
From the original article
1 min read at huggingface.co
A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing.
By Bryce Grant, Xijia Zhao, Peng Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.