In-Context Robot Learning with VLM Agents
GPT-Policy uses commercial VLM agents like GPT-6 Astra for in-context robot learning, translating demonstrations and feedback into verified robot actions without gradient updates.
The paper (arXiv 2609.19138) presents GPT-Policy, a framework combining a context compiler that preserves task-relevant visual transitions, a VLM such as GPT-6 Astra that proposes robot-tool actions, and a constrained controller that verifies and executes each action. Real-robot trials show human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. Evaluation covers success and efficiency metrics, matched model comparisons, and controlled context ablations.
- Framework combines a context compiler, VLM action proposer, and constrained controller that verifies and executes each action.
- Human video demonstrations improved real-robot task completion even without robot action labels.
- Aligned action references yielded further gains on contact-sensitive manipulation tasks.
- Products
- GPT-Policy
- AI models
- GPT-6 Astra
From the original article
1 min read at huggingface.co
Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies.
By Dongzhou Cheng, Taoran Yi, Ye Fang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.