Dwell lets vision-language agents learn policies in latent space
Original title: Vision-Language Agents Can Learn in Their Own Latent Worlds
The framework treats a pretrained VLM as a recurrent latent transition model and reports top trainable results on five visual environments with less imagination compute.
Researchers introduce Dwell, a model-based reinforcement learning framework that treats a pretrained vision-language model as a recurrent latent transition operator used as both world model and policy. The agent learns from real interactions and latent imagination without reconstructing future observations. Across five interactive visual environments, the authors report the strongest overall performance among trainable methods and substantially lower compute for imagination. Code is public.
- Dwell reframes a VLM as a recurrent latent transition operator.
- Evaluation covers five interactive visual environments.
- Authors say effective world models need only action-useful futures.
- Code is at github.com/Tom111-lhx/Dwell.
From the original article
1 min read at arxiv.orgLearning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction.
By Haoxi Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.