titleHeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation
authorsGyeonghun Kim, Donghoon Kim, Minsoo Choi, Suyun Lee
date2026-11-02
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.02660
affiliationSeoul National University
upvotes7 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf code
cite
primerVision-language-action models
related4D Human-Scene Reconstruction from Low-Overlap Captures
ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Articulated Object Reconstruction from Rest-State Observation
ResearchMath-14K: Scaling Research-Level Mathematics via Agents
In-Distribution Forcing for Long Video Generation at Test Time
abstractRecent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons.

bibtex

@misc{kim2026helm,
  title={HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation},
  author={Gyeonghun Kim and Donghoon Kim and Minsoo Choi and Suyun Lee},
  year={2026},
  eprint={2611.02660},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.02660}
}