World Action Modeling with Progressive Visual Planning
ProWAM predicts actions and sparse visual sub-goals, posting new robotics scores and 70% real-world success.
ProWAM is a progressive world action model that jointly predicts robot actions and an ordered sequence of sparse visual sub-goals, which can be learned from action-free video. One video-backbone pass caches sub-goal features, and replanning uses lightweight action denoising instead of full video rollouts. It reports 85.8% on LIBERO-Plus and 75.7% on randomized RoboTwin, with relative gains up to 35.9%, plus 48.1% on RoboCasa365 and 18.2% on Composite-Unseen. Zero-shot real-world success was 70.0%, versus 55.0% for the strongest baseline.
- Predicts actions plus an ordered sequence of sparse visual sub-goals.
- Sub-goal prediction can be learned from large-scale action-free videos.
- LIBERO-Plus 85.8% and randomized RoboTwin 75.7%, relative gains up to 35.9%.
- Zero-shot real-world success rose from 55.0% to 70.0%.
From the original article
1 min read at huggingface.co
World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient.
By Fei Zhang, Zhaochong An, Duncan Frost · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.