Grounded Action Model: 3D Grounding as a Foundation for Robotics
Grounded Action Models use 3D object grounding and beat prior robot policies on RoboTwin and LIBERO-PRO.
Grounded Action Models build robot policies around explicit 3D object grounding rather than learning metric location only from demonstrations. Language, point, or box prompts become a shared object-centric representation that a multi-stream transformer mixes with robot state to predict action chunks. On RoboTwin 2.0, GAM averages 55.3% success across 50 tasks and 47.6% under scene randomization. It reaches 61% on LIBERO-PRO and, on real robots, 17 of 20 successes under visual shift on a bimanual YAM, with a Molmo2 planner scoring 64.7% in-distribution step completion on a Franka.
- GAMs turn language, point, or box prompts into object-centric 3D representations.
- RoboTwin 2.0 average success is 55.3% across 50 tasks, versus 52.0% for Spatial Forcing.
- LIBERO-PRO average is 61% across 16 perturbation settings, versus 53% for pi0.5.
- On a bimanual YAM it keeps 17/20 successes under visual shift versus 4/20 for pi0.5.
Full article255 words · extracted from huggingface.co · click to collapse
Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for π_{0.5}) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for π_{0.5}, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.23863