RoboCasa-Shift measures stage-aware robustness of household VLA policies
Original title: RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation
GR00T N1.5 ran 1,040 rollouts; equal success drops hid different first-stage failure rates.
RoboCasa-Shift, built on RoboCasa365, pairs clean and perturbed rollouts that differ only by one controlled shift. GR00T N1.5 was evaluated on four long-horizon household tasks under four perturbation factors and three severity levels. Success did not always fall monotonically as severity rose, and identical end-to-end drops mapped to different failure depths. Code is public.
- One camera condition left success unchanged while 8 of 80 episodes flipped.
- Camera L_2 and Gaussian blur L_1 both changed success by 0.188.
- Their first-stage failure rates were 19.0% and 36.4%, respectively.
- Some perturbation settings drove the evaluated policy to complete task failure.
- Products
- RoboCasa-ShiftRoboCasa365
- AI models
- GR00T N1.5
From the original article
1 min read at huggingface.co
Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks.
By Haocheng Wang, Youquan Liu, Weiyu Guo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.