SkeLAM splits robot latent actions into skeleton and residual
Original title: Skeleton-Grounded Latent Action Model for Robot World Models
On DROID, end-effector ADE falls 19.5% and LPIPS 11.1% versus DreamDojo, with zero-shot human-to-robot transfer shown qualitatively.
SkeLAM is a skeleton-guided latent action model for robot world models that separate kinematic motion from other visual change. Robot skeleton supervision grounds a kinematic latent, while a residual keeps complementary dynamics; cross-modal swap reconstruction makes RGB- and skeleton-derived kinematics interchangeable. A lightweight adapter maps robot actions into that kinematic space, and the world model is then fine-tuned. At inference it predicts future observations from the current view and robot actions. On DROID, the authors report 19.5% lower end-effector ADE and 11.1% lower LPIPS than DreamDojo, plus qualitative zero-shot human-to-robot motion transfer.
- Latent actions split into skeleton-grounded kinematics and a residual.
- Branch dropout limits reliance on future-derived residual information.
- Policy eval success rates are said to agree with real robot runs.
- Motion probing is cited as evidence of kinematic specialization.
- Qualitative rollouts show zero-shot human-to-robot motion transfer.
From the original article
1 min read at huggingface.co
Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls.
By Qi Lv, Jiayi Lin, Yuxin Jiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.