| title | Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents |
|---|---|
| authors | Wei-Chieh Huang, Weizhi Zhang, Yuchen Wu, Yankai Chen, Eric Hanchen Jiang, Wooseong Yang, Yiwei Yang, Henry Peng Zou, Hanrong Zhang, Ying Nian Wu, Haolun Wu, Kai-Wei Chang, Philip S. Yu, Xue Liu, Aylin Caliskan |
| date | 2026-08-15 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.15008 |
| upvotes | 15 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| primer | Memory for AI agents |
| abstract | Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate, namely the underlying medium in which memory is represented and stored, should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability introduces a further routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents. Code will be made available upon acceptance. |
cite
@misc{huang2026harness,
title={Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents},
author={Wei-Chieh Huang and Weizhi Zhang and Yuchen Wu and Yankai Chen and Eric Hanchen Jiang and Wooseong Yang and Yiwei Yang and Henry Peng Zou and Hanrong Zhang and Ying Nian Wu and Haolun Wu and Kai-Wei Chang and Philip S. Yu and Xue Liu and Aylin Caliskan},
year={2026},
eprint={2608.15008},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.15008}
}