Visual perturbations alter VLA actions without changing text rationale
Original title: Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs
TNAS-guided attacks induce collisions in 52.6% of driving scenes while keeping rationales word-for-word identical.
Reasoning vision-language-action models produce readable rationales for physical actions. Visual perturbations can shift actions while leaving the generated text unchanged. The TNAS diagnostic isolates text-neutral directions in hidden states that affect actions far more than text gradients. In Alpamayo, such attacks cause new collisions in over half of obstacle scenes under full-image or localized perturbations.
- Text-neutral perturbations shift actions 4.64x-47.8x more than text gradients
- Rationale preservation rises from 83.0% to 89.5% with TNAS guidance
- Collisions occur in 52.6% of 154 scenes under full-image perturbations
- Monitors must verify action channels directly because rationales cannot certify safety
From the original article
1 min read at arxiv.orgReasoning vision-language-action (VLA) models pair physical control with natural-language rationales that supervisors can read to monitor embodied safety. Prior attacks on VLA reasoning corrupt the rationale itself and leave visible evidence in the text. We show that visual perturbations can instead alter the physical action while keeping the generated rationale word-for-word identical.
By Zihao Feng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.