| title | Memory as Plans: World-Action Modeling with Memory-Grounded Planning |
|---|---|
| authors | Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang |
| date | 2026-09-10 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.11561 |
| upvotes | 21 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows. |
cite
@misc{zhao2026memory,
title={Memory as Plans: World-Action Modeling with Memory-Grounded Planning},
author={Sizhe Zhao and Haozhe Xie and Weiyu Zhao and Chenchu Zhang and Huan Wang and Chenyang Wang and Qinglin Liu and Shengping Zhang},
year={2026},
eprint={2609.11561},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.11561}
}