DreamFree Repurposes Video Diffusion for World Action Models
Original title: Future Representation Distillation for Imagination-free World Action Models
DreamFree enables real-time robot control at 11Hz without generating future videos.
DreamFree is a World Action Model that repurposes pretrained video diffusion models as predictive visual encoders through representation distillation. The visual encoder learns to predict frozen video diffusion model representations from current observations. An action module extracts action-relevant information enabling closed-loop control without iterative denoising.
- 11Hz real-time closed-loop control with 14B video diffusion backbone
- Higher success rates than state-of-the-art VLAs and WAMs
- Translates stronger video priors into better robot control
- Enables learning new tasks from video demonstrations without action annotation
From the original article
1 min read at huggingface.co
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction.
By Jaehyeong Jo, Sangwon Jang, Sihyun Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.