RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
RoboFollow shows embodied agents' high success rates mask weak genuine instruction following.
RoboFollow argues that low scene entropy lets embodied policies score well without using language. Its scenes support multiple task branches, and a four-level protocol perturbs layout and semantics while separating intent from execution. Nine VLA and WAM policies do not reliably carry strong L0 results to L1-L3, and tested mitigations fail to close the gap. Code and the dataset are publicly released.
- Low scene entropy can make language redundant for high scores.
- An L0-L3 protocol separates intent from motor execution.
- Nine VLA and WAM policies fail to transfer beyond L0.
- Stronger backbones, QA co-training, and guidance do not close the gap.
From the original article
1 min read at huggingface.co
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it.
By Chang Guo, Yukun Xie, Bohan Tan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.