| title | Maglev: Sliding Recurrent Memory |
|---|---|
| authors | Bo Liu, Qiang Liu |
| date | 2026-08-05 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.02870 |
| affiliation | University of Texas at Austin |
| upvotes | 7 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | We introduce , a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. consists of two coupled models: a prefiller Q, which leverages full attentionIn practice, we use interleaved full and sliding-window attention for Q, as this yields stronger performance. The essential requirement is that Q be more expressive than P, with access to the full history. to produce memory targets m'_t, and a decoder P, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories m_t for next-token prediction. We train with a memory consistency loss that aligns m_t with m'_t, allowing inference to use P alone. Empirically, improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between P and Q reduces parameter memory while preserving most of the gains. |
cite
@misc{liu2026maglev,
title={Maglev: Sliding Recurrent Memory},
author={Bo Liu and Qiang Liu},
year={2026},
eprint={2608.02870},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.02870}
}