Sim-trained ZAP robot policy hits 96.7% on real tasks
Original title: Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent
A 27B vision-language model fine-tuned only in simulation matches GPT-6 Astra and beats MolmoAct2's 33.3% in the same harness.
Zero-real Agent Policy (ZAP) uses a pretrained 27B vision-language model for scene understanding, then supervised fine-tuning and GRPO on simple simulated tasks to teach robot control through a tool harness. The authors report no real-world robot data during training, yet 96.7% success on real-world tasks, matching GPT-6 Astra while running 3x faster and 17x cheaper. They also report generalization to new objects, goals and language, plus transfer to an unseen single-arm Franka.
Unconfirmed: Success rates are the authors' measurements in their harness, not an independent evaluation.
- ZAP scores 55.9% on RoboLab-120, 4.9 points ahead of the leading VLA/WAM baseline.
- Reported new behaviors include mid-air handoff, drawing an X and playing chess.
- Those goals and manipulation types do not appear in the training data, the authors say.
- Products
- Franka
- AI models
- ZAPGPT-6 AstraMolmoAct2
From the original article
1 min read at arxiv.orgA robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf vision-language model to control a robot via a simple tool harness.
By Benjamin Pikus · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.