What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Researchers diagnose conditional visual grounding failures in visuomotor imitation policies and show targeted interventions substantially improve distractor robustness.
The paper studies why ACT-based visuomotor imitation policies fail when visually similar distractor objects or receptacles are introduced, finding sensitivity depends on both distractor type and manipulation stage. Interventions including distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting improve target selection while preserving spatial control information, with gains in simulation and on a physical UR3e. The same failure pattern is confirmed in a pretrained vision-language-action policy on a state-conditioned medical instrument-handling task.
- Distractor sensitivity varies with visual similarity type and manipulation stage
- Three complementary interventions improve target selection under distractors
- Failure pattern reproduced in a pretrained vision-language-action policy
- Robustness gains validated in simulation and on a physical UR3e robot
From the original article
1 min read at arxiv.orgVisuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state.
By Vivek Chavan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.