| title | HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation |
|---|---|
| authors | Gyeonghun Kim, Donghoon Kim, Minsoo Choi, Suyun Lee |
| date | 2026-11-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.02660 |
| affiliation | Seoul National University |
| upvotes | 7 (Hugging Face Daily Papers, 2026-11-08) |
| links | abs pdf hf code |
| cite | |
| primer | Vision-language-action models |
| related | 4D Human-Scene Reconstruction from Low-Overlap Captures ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time? Articulated Object Reconstruction from Rest-State Observation ResearchMath-14K: Scaling Research-Level Mathematics via Agents In-Distribution Forcing for Long Video Generation at Test Time |
| abstract | Recent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons. |
bibtex
@misc{kim2026helm,
title={HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation},
author={Gyeonghun Kim and Donghoon Kim and Minsoo Choi and Suyun Lee},
year={2026},
eprint={2611.02660},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.02660}
}