Transferring the Intelligence of VLMs to Robotic Control
RoboDawn lets a VLM control robots via discrete commands, beating trained policies zero-shot and one-shot.
RoboDawn lets an agentic vision-language model control a robot through discrete translation, rotation, and gripper commands in a closed visual loop. A few in-context demonstrations ground both the interface and the task strategy, without task-specific robot training. On RoboTwin 2.0 C2R, success rises from 53.2% zero-shot to 73.6% one-shot, above the pi0.5 baseline at 46.0%. On RoboDojo it rises from 35.67% to 47.17%, and the same setup transfers to a Franka robot for block tasks.
- Closed-loop VLM control via translation, rotation, and gripper commands
- Zero-shot beats several policies trained on robot data
- RoboTwin: 53.2% zero-shot and 73.6% one-shot, versus pi0.5 at 46.0%
- Also improves RoboDojo and transfers to a real Franka robot
- AI models
- pi0.5
From the original article
1 min read at huggingface.co
Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control?
By Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.