Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
Latent-Foresight jointly learns a tokenizer and flow model so vision latents stay predictable.
Latent-Foresight jointly learns a latent tokenizer and a flow-based generative dynamics model so Vision Foundation Model features are shaped for temporal prediction. Earlier pipelines compress features with fixed PCA or separate autoencoders, then train a predictor on a frozen latent space. The authors add optimization choices that prevent latent collapse and align reconstruction with generation. They report more temporally coherent latents and better results than two-stage baselines across future scene-understanding tasks and horizons, including high-resolution adaptation, and release code and weights.
- Jointly trains a latent tokenizer and a flow-based dynamics model.
- Removes frozen PCA or separate autoencoder stages used previously.
- Design choices prevent collapse and align reconstruction with generation.
- Beats two-stage baselines across tasks, horizons, and high resolution.
- Authors release implementation code and model weights.
- Tags
- world-modelslatent-representationsvision-foundation-modelsflow-modelsvideo-predictionlatent-foresight
- AI models
- Latent-Foresight
From the original article
1 min read at huggingface.co
Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.
By Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.