DreamX Video Diffusion Model as Universal Latent Action Learner
Original title: DreamX: Video Diffusion Model is a Universal Latent Action Learner
DreamX extracts compact latent actions from video diffusion models for transfer across embodiments and environments.
DreamX repurposes a pretrained video diffusion model to learn universal latent actions. It compresses each latent video frame by 300 times into 1,200 tokens while capturing action-relevant dynamics disentangled from visual appearance. The shared latent space supports any-to-any video transfer and zero-shot action transfer. On LARY-Bench it achieves state-of-the-art performance among latent action models, enabling compact policy learning with reduced computational cost.
- Compresses each latent video frame by 300 times.
- 1,200 arrows 4 tokens.
- Supports any-to-any video transfer.
- State-of-the-art on LARY-Bench.
- AI models
- DreamX
From the original article
1 min read at huggingface.co
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments.
By Jisoo Kim, Chaehyun Kim, Junsu Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.