Modality-Autoregressive World-Action Models
ModAR autoregressively denoises multiple future modalities (point tracks, DINO features, depth) before predicting actions, beating prior world-action models at all data scales.
ModAR is the first world-action model (WAM) to autoregressively denoise multiple future modalities before predicting actions, letting each prediction condition on previously generated modalities. Training from scratch shows WAMs benefit from predicting point tracks, DINO features, and depth maps, while future RGB adds no consistent benefit. ModAR's sequential generation outperforms existing WAM formulations with the highest average success rate at all evaluated data scales. It slightly beats video-model-initialized Flex-π (75% vs 72% success) using roughly 20x fewer training FLOPs and no pretraining, and wins on three real-world bimanual tasks.
- First WAM to autoregressively denoise multiple modalities before action prediction
- Point tracks, DINO features, depth help; future RGB adds no consistent benefit
- Beats Flex-π 75% vs 72% success with ~20x fewer training FLOPs
- Outperforms baselines on three real-world bimanual manipulation tasks
From the original article
1 min read at huggingface.co
World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question.
By Adam Hung, Bardienus P. Duisterhof, Deva Ramanan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.