CaPtain turns small VLMs into robot manipulation agents
Original title: CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents
CaPtain trains small open VLMs into code-as-policy agents achieving 73.8% success on LIBERO benchmarks and running on real hardware without frontier models.
CaPtain is a post-training pipeline that converts small vision-language models into capable code-as-policy robot agents. It uses supervised fine-tuning on synthetic trajectories followed by multi-turn reinforcement learning with a physical critic. The framework operates on simulation and real robots after simulation-only adaptation.
- Reaches 73.8% success on LIBERO after RL from 0% zero-shot
- 78.5% on LIBERO-PRO surpassing frontier agents
- 83.3% on SimplerEnv and runs on real WidowX robot
From the original article
1 min read at huggingface.co
Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply.
By Yijia Fan, Xilin He, Yilei Jiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.