PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training
PerturBot trains vision-language-action policies so task evidence, not modality shortcuts, drives actions.
PerturBot is a training method for vision-language-action policies that break modality shortcuts, where visual, lexical, or motor cues predict expert actions without the evidence needed for the decision. It applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. GroundingFscore is an offline score that diagnoses shortcut reliance, while task success only shows whether performance improved. Inference stays the same.
- VLA policies can succeed while ignoring the evidence that should decide the action.
- PerturBot uses wrist-view perturbations, decision captions, and relabeled failed segments.
- GroundingFscore measures shortcut reliance separately from task success rate.
- Inference is unchanged; the method changes which demonstrations are scaled.
From the original article
1 min read at huggingface.co
A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway.
By Mingyu Liu, Chonghao Sima, Tianjian Feng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.