ALPS unifies language, reconstruction and state in one visual encoder
Original title: ALPS: Visual State Encoders for World Models
Frozen ALPS tokens reach 85.5% mean policy success on LIBERO and match a 3.5x larger V-JEPA 2-L on state probes.
ALPS is a pretrained visual state encoder meant to support prediction, instruction and inspection in world models. A ViT-B/16 is trained from scratch on still images and text with language, reconstruction and spatial-correspondence objectives, then refined by cross-view self-distillation. Across fifteen public encoders, the paper reports the lowest aggregate LIBERO state-probe error among ViT-B models, matching the 3.5x larger V-JEPA 2-L, plus strong language alignment and pixel reconstruction. Frozen tokens reach 85.5% mean policy success on LIBERO with the authors' text tower and match or exceed leading encoders on planning for PushT, Wall and Granular.
- Backbone is a ViT-B/16 pretrained from scratch on images and text.
- LIBERO mean policy success with frozen tokens is 85.5%.
- State-probe error matches V-JEPA 2-L, described as 3.5x larger.
- Planning tests cover PushT, Wall and Granular.
- Future tokens can be decoded to pixels for inspection.
From the original article
1 min read at huggingface.co
World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection.
By Yanqing Liu, Letian Zhang, Chengzhi Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.