| title | When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings |
|---|---|
| authors | Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer |
| date | 2026-08-04 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.03994 |
| affiliation | CORAL NLP Research |
| upvotes | 8 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi. |
cite
@misc{schrder2026when,
title={When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings},
author={Christopher Schröder and Lukas Gienapp and Ferdinand Schlatt and Martin Potthast and Gerhard Heyer},
year={2026},
eprint={2608.03994},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.03994}
}