titleFuture Representation Distillation for Imagination-free World Action Models
authorsJaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang
date2026-11-04
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.04527
affiliationKAIST AI
upvotes9 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf
cite
relatedOmniShowcase: Omnidirectional Subject-Consistent Video Generation from Multi-View Images
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
Language Models Can Control Their Own Attention
OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sources
GRACE: Generation-aware latent compression for efficient video generation
abstractWorld Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

bibtex

@misc{jo2026future,
  title={Future Representation Distillation for Imagination-free World Action Models},
  author={Jaehyeong Jo and Sangwon Jang and Sihyun Yu and Seonghyeon Ye and Yekyung Nah and Hojung Jung and Kimin Lee and Sung Ju Hwang},
  year={2026},
  eprint={2611.04527},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.04527}
}