DreamX repurposes video diffusion models as universal latent action learners
Original title: DreamX: Video Diffusion Model is a Universal Latent Action Learner
DreamX extracts 300x compressed latent actions from pretrained video diffusion models, enabling any-to-any video transfer and zero-shot action transfer across embodiments.
DreamX extracts compact latent actions from video diffusion models' spatiotemporal features, capturing action-relevant dynamics while disentangling them from visual appearance. The resulting shared latent space supports transfer between human and robot embodiments as well as across environments and action parameterizations. On LARY-Bench it reaches state-of-the-art performance among latent action models and reduces computational cost for video-diffusion-based policy learning.
- 300x compression: 1200 tokens to 4 tokens per frame
- Any-to-any video transfer supported
- Zero-shot action transfer across embodiments
- State-of-the-art on LARY-Bench
- Enables compact policy learning
From the original article
1 min read at arxiv.orgLearning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments.
By Jisoo Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.