| title | Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs |
|---|---|
| authors | Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets, Elena Tutubalina |
| date | 2026-09-24 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.29845 |
| upvotes | 100 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass. |
cite
@misc{tikhonov2026your,
title={Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs},
author={Pavel Tikhonov and Anton Korznikov and Matvey Mikhalchuk and Nikita Dragunov and Temurbek Rahmatullaev and Polina Druzhinina and Anton Razzhigaev and Ivan Oseledets and Elena Tutubalina},
year={2026},
eprint={2609.29845},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.29845}
}