| title | MemLearner: Learning to Query Context memory for Video World Models |
|---|---|
| authors | Jiwen Yu, Jianxiong Gao, Jianhong Bai, Yiran Qin, Kaiyi Huang, Quande Liu, Xintao Wang, Pengfei Wan, Kun Gai, Xihui Liu |
| date | 2026-06-30 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.31734 |
| upvotes | 27 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| primer | World models |
| abstract | Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios. |
cite
@misc{yu2026memlearner,
title={MemLearner: Learning to Query Context memory for Video World Models},
author={Jiwen Yu and Jianxiong Gao and Jianhong Bai and Yiran Qin and Kaiyi Huang and Quande Liu and Xintao Wang and Pengfei Wan and Kun Gai and Xihui Liu},
year={2026},
eprint={2606.31734},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.31734}
}