Embodied First Exam tests coding agents on 1,156 robot tasks
Original title: Embodied First Exam
GPT-6 Astra in Codex, strongest of eight agents, succeeds on 34% of 456 core tasks.
The Embodied First Exam lets coding agents choose direct control, code-to-control, or training a policy across 1,156 tasks in 35 suites on 47 robots, including nine suites beyond pick-and-place. The Standard Setting gives only sensor observations and, by default, no resets; a Realtime variant keeps the world running. A Privileged Setting exposes full simulator state and unlimited resets as an approximate upper bound. Failures are labeled by the stage at which attempts go wrong.
- Nine new suites cover puzzles, dexterity, multi-robot, and whole-body control.
- Each task can run in up to three evaluation settings.
- Privileged success rates are described as an approximate upper bound.
- Authors compare failures across agents, settings, and stage categories.
- Products
- Codex
- AI models
- GPT-6 Astra
From the original article
1 min read at huggingface.co
Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints.
By Kangrui Wang, Qineng Wang, Changhe Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.