World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
World Action Agent lets VLMs pilot robots via visual rehearsal, reaching 75.6% success on LIBERO-Pro.
World Action Agent is a multi-agent harness that lets vision-language models pilot robots inside a visual action workspace with automatic contact views, action rehearsal, and in-view correction. It evolves multimodal skills from expert videos and human teaching and can consult them through a Skill Agent. On LIBERO-Pro, skills evolved only from LIBERO-90 reach 75.6% average success, ahead of end-to-end vision-language-action models, code-as-policy agents, and a same-backbone visual-harness baseline, and remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises out-of-domain success from 1.7% to 43.3%.
- WAA gives VLMs contact views, action rehearsal, and in-view correction before execution.
- Skills evolved only from LIBERO-90 reach 75.6% average success on LIBERO-Pro.
- Those skills remain effective on robosuite without further learning.
- Fine-tuning Qwen3.5-9B on harness traces lifts out-of-domain success from 1.7% to 43.3%.
Full article232 words · extracted from huggingface.co · click to collapse
General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.29964