LaLa-World scales latent world models with language actions
Original title: LaLa-World: Scaling Latent World Models with Natural Language Actions
The paper reports comparable planning to an RAEv2 video baseline at 6 times less training cost.
LaLa-World is a language-conditioned latent world model that learns from simulation, robot trajectories, and broader image and video data. Numerical actions and visual changes are described in language so one predictor can train across heterogeneous datasets, and the predictor is built on a language model. The authors report higher average model-predictive-control planning performance than existing latent world models trained per domain, plus better robustness to action phrasing when image and video data are added. A matched comparison claims planning comparable to an RAEv2-based video model with 6 times less training cost and 13.9 times faster planning. Pretrained weights initialize LaLa-Policy, reported competitive with Cosmos Policy on LIBERO and RoboCasa.
Unconfirmed: Speed and planning comparisons are the authors' matched experiments.
- Planning is described as 13.9 times faster than the RAEv2 video baseline.
- LaLa-Policy jointly predicts actions and future visual representations.
- Inference is up to 5.5 times faster than Cosmos Policy, the paper says.
- Evaluations include LIBERO and RoboCasa policy comparisons.
- AI models
- LaLa-WorldLaLa-PolicyCosmos Policy
From the original article
1 min read at huggingface.co
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning.
By Junha Song, Geonmo Gu, Jeongeun Park · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.