| title | ALPS: Visual State Encoders for World Models |
|---|---|
| authors | Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie |
| date | 2026-10-21 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.30120 |
| affiliation | UCSC-VLAA |
| upvotes | 30 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| primer | World models |
| abstract | World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models. |
cite
@misc{liu2026alps,
title={ALPS: Visual State Encoders for World Models},
author={Yanqing Liu and Letian Zhang and Chengzhi Liu and Zijun Wang and Jieru Mei and Hongliang Fei and Yuyin Zhou and Cihang Xie},
year={2026},
eprint={2610.30120},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.30120}
}