Agent as Policy for Robotic Manipulation
Agent as Policy lets a general-purpose agent drive a physical robot via runtime reasoning and program generation, reaching 100% success on manipulation tasks.
The paper introduces Agent as Policy (AGP), which puts task planning and execution for a physical robot under a general-purpose agent's control with no task-specific or environment-specific training. The agent interprets visual evidence, writes executable programs, issues motion commands, and revises actions based on physical outcomes. AGP was evaluated on real-world manipulation tasks including assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. It achieved success rates of 100%, 100%, and 80% on three block construction configurations.
- General-purpose agent directly controls a robot without task-specific training
- Agent writes programs, issues motion commands, and revises actions from visual feedback
- Achieves 100%, 100%, and 80% success on three block construction configurations
- Tasks span assembly, die reorientation, targeted throwing, and bimanual towel folding
Full article153 words · extracted from huggingface.co · click to collapse
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.12541