SWAM studies how action learning affects language-video alignment
Original title: SWAM: Scaling World Action Models with Cross-Modal Alignment
Authors report 88.10% easy and 61.56% hard RoboTwin success, and 40.00% versus 23.33% on three real tasks.
SWAM studies world action models that jointly predict video and actions from language, focusing on whether action learning weakens language-video alignment. The authors curate about 47,000 hours of general video, egocentric video, UMI demonstrations, and robot demonstrations, then compare language-video supervision, language-video-action supervision, and both together. Under a fixed budget, mixing egocentric video and UMI demonstrations improved control generalization. Post-trained only on RoboTwin Clean demonstrations, they report 88.10% Easy and 61.56% Hard success, and 40.00% success on three real-world tasks versus 23.33% for the strongest baseline. They release the curated data, training code, and weights.
Unconfirmed: Success rates are the authors' reported results, not an independent evaluation.
- The data pool is about 47,000 hours across four video and demonstration sources.
- Mixing egocentric videos and UMI demonstrations improved control generalization.
- Easy and Hard RoboTwin rates are reported as 88.10% and 61.56%.
- Real-world success is reported as 40.00% versus 23.33% for the strongest baseline.
- Curated data, training code, and model weights are released.
From the original article
1 min read at huggingface.co
World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment.
By Yu Zhou, Xinyuan Xia, Xuankun Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.