Transferring the Intelligence of VLMs to Robotic Control
RoboDawn lets a VLM control robots via discrete commands, beating trained policies zero-shot and one-shot.
RoboDawn lets an agentic vision-language model control a robot through discrete translation, rotation, and gripper commands in a closed visual loop. A few in-context demonstrations ground both the interface and the task strategy, without task-specific robot training. On RoboTwin 2.0 C2R, success rises from 53.2% zero-shot to 73.6% one-shot, above the pi0.5 baseline at 46.0%. On RoboDojo it rises from 35.67% to 47.17%, and the same setup transfers to a Franka robot for block tasks.
- Closed-loop VLM control via translation, rotation, and gripper commands
- Zero-shot beats several policies trained on robot data
- RoboTwin: 53.2% zero-shot and 73.6% one-shot, versus pi0.5 at 46.0%
- Also improves RoboDojo and transfers to a real Franka robot
Full article238 words · extracted from huggingface.co · click to collapse
Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.22966