World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
World Action Agent lets VLMs pilot robots via visual rehearsal, reaching 75.6% success on LIBERO-Pro.
World Action Agent is a multi-agent harness that lets vision-language models pilot robots inside a visual action workspace with automatic contact views, action rehearsal, and in-view correction. It evolves multimodal skills from expert videos and human teaching and can consult them through a Skill Agent. On LIBERO-Pro, skills evolved only from LIBERO-90 reach 75.6% average success, ahead of end-to-end vision-language-action models, code-as-policy agents, and a same-backbone visual-harness baseline, and remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises out-of-domain success from 1.7% to 43.3%.
- WAA gives VLMs contact views, action rehearsal, and in-view correction before execution.
- Skills evolved only from LIBERO-90 reach 75.6% average success on LIBERO-Pro.
- Those skills remain effective on robosuite without further learning.
- Fine-tuning Qwen3.5-9B on harness traces lifts out-of-domain success from 1.7% to 43.3%.
From the original article
1 min read at huggingface.co
General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act.
By Yehang Zhang, Haojian Huang, Yifan Chang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.