RAPID: Robot Agentic Programming from Demonstrations
RAPID turns one visual human demonstration into verified, reusable robot programs via an agentic coding loop.
RAPID generates, verifies, and refines robot programs from a single visual human demonstration by inferring a task specification, action primitives, and an interactive test environment. Programs are object-centric and relational: primitives are trajectory-optimization problems for object-level effects, composed with run-time geometric constraints. It is evaluated in simulation on eight contact-rich nonprehensile tasks and on LIBERO-Pro prehensile tasks, and deployed on a real Franka arm for all eight nonprehensile tasks. Results report generalization across object pose, shape, material, and environment.
- Builds specification, primitives, and a verifier from one visual demonstration.
- Object-centric relational programs generalize beyond the demonstrated motion.
- Evaluated on eight contact-rich nonprehensile tasks in simulation and on Franka.
- Also tested on prehensile tasks in the LIBERO-Pro benchmark.
Full article199 words · extracted from arxiv.org · click to collapse
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.30249