Modality-Autoregressive World-Action Models
ModAR autoregressively denoises multiple future modalities (point tracks, DINO features, depth) before predicting actions, beating prior world-action models at all data scales.
ModAR is the first world-action model (WAM) to autoregressively denoise multiple future modalities before predicting actions, letting each prediction condition on previously generated modalities. Training from scratch shows WAMs benefit from predicting point tracks, DINO features, and depth maps, while future RGB adds no consistent benefit. ModAR's sequential generation outperforms existing WAM formulations with the highest average success rate at all evaluated data scales. It slightly beats video-model-initialized Flex-π (75% vs 72% success) using roughly 20x fewer training FLOPs and no pretraining, and wins on three real-world bimanual tasks.
- First WAM to autoregressively denoise multiple modalities before action prediction
- Point tracks, DINO features, depth help; future RGB adds no consistent benefit
- Beats Flex-π 75% vs 72% success with ~20x fewer training FLOPs
- Outperforms baselines on three real-world bimanual manipulation tasks
Full article185 words · extracted from huggingface.co · click to collapse
World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-π on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately 20times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.17524