| title | Fractional State Space Transition for Long Sequence Modeling |
|---|---|
| authors | Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi, Lifeng Shang, Yufei Cui |
| date | 2026-09-28 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.36314 |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs. |
cite
@misc{kobyzev2026fractional,
title={Fractional State Space Transition for Long Sequence Modeling},
author={Ivan Kobyzev and Abbas Ghaddar and Ali Nasiri-Sarvi and Lifeng Shang and Yufei Cui},
year={2026},
eprint={2609.36314},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.36314}
}