| title | The State-Prediction Separation Hypothesis |
|---|---|
| authors | Giovanni Monea, Nathan Godey, Kianté Brantley, Yoav Artzi |
| date | 2026-07-01 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.01218 |
| affiliation | Cornell LIL Lab |
| upvotes | 9 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails. |
cite
@misc{monea2026state,
title={The State-Prediction Separation Hypothesis},
author={Giovanni Monea and Nathan Godey and Kianté Brantley and Yoav Artzi},
year={2026},
eprint={2607.01218},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.01218}
}