RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
RoboFollow shows embodied agents' high success rates mask weak genuine instruction following.
RoboFollow argues that low scene entropy lets embodied policies score well without using language. Its scenes support multiple task branches, and a four-level protocol perturbs layout and semantics while separating intent from execution. Nine VLA and WAM policies do not reliably carry strong L0 results to L1-L3, and tested mitigations fail to close the gap. Code and the dataset are publicly released.
- Low scene entropy can make language redundant for high scores.
- An L0-L3 protocol separates intent from motor execution.
- Nine VLA and WAM policies fail to transfer beyond L0.
- Stronger backbones, QA co-training, and guidance do not close the gap.
Full article206 words · extracted from huggingface.co · click to collapse
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.25636