FRED improves future video representations for world action models
Original title: Self-Distilling Future Video Representations for World Action Models
On tasks shown only in human videos, FRED reaches 50.0% success versus 19.4% for the best rival.
FRED trains world action models to predict future video representations from noise in one denoising step. Self-distillation uses representations from ground-truth future observations as targets, and historical observations add temporal context. Across real-world manipulation scenarios it outperforms competing WAMs, with larger gains on long-horizon and out-of-domain zero-shot tasks and better human-to-robot transfer.
- Action prediction depends on intermediate future video representations.
- Targets are representations extracted from ground-truth future observations.
- The 50.0% result is on tasks absent from robot training data.
- Gains are particularly large under zero-shot environment generalization.
From the original article
1 min read at huggingface.co
World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models.
By Huiwon Jang, Suhyeok Jang, Hyeonmo Kang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.