Grounded Action Model: 3D Grounding as a Foundation for Robotics
Grounded Action Models use 3D object grounding and beat prior robot policies on RoboTwin and LIBERO-PRO.
Grounded Action Models build robot policies around explicit 3D object grounding rather than learning metric location only from demonstrations. Language, point, or box prompts become a shared object-centric representation that a multi-stream transformer mixes with robot state to predict action chunks. On RoboTwin 2.0, GAM averages 55.3% success across 50 tasks and 47.6% under scene randomization. It reaches 61% on LIBERO-PRO and, on real robots, 17 of 20 successes under visual shift on a bimanual YAM, with a Molmo2 planner scoring 64.7% in-distribution step completion on a Franka.
- GAMs turn language, point, or box prompts into object-centric 3D representations.
- RoboTwin 2.0 average success is 55.3% across 50 tasks, versus 52.0% for Spatial Forcing.
- LIBERO-PRO average is 61% across 16 perturbation settings, versus 53% for pi0.5.
- On a bimanual YAM it keeps 17/20 successes under visual shift versus 4/20 for pi0.5.
- Products
- RoboTwin 2.0LIBERO-PRO
From the original article
1 min read at huggingface.co
Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations.
By Gehao Zhang, Weikai Huang, Shailesh Shailesh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.