| title | Rethinking the Role of Efficient Attention in Hybrid Architectures |
|---|---|
| authors | Ziqing Qiao, Yinuo Xu, Chaojun Xiao, Zhou Su, Zihan Zhou, Yingfa Chen, Xiaoyue Xu, Xu Han, Zhiyuan Liu |
| date | 2026-06-13 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.15378 |
| affiliation | OpenBMB |
| upvotes | 20 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent sequence mixers. However, how these efficient modules shape model capabilities remains poorly understood. To address this gap, we conduct a systematic analysis across hybrid architectures from three perspectives: scaling behavior, mechanism analysis, and architecture design. First, from a scaling perspective, we find that efficient-attention design primarily affects how fast long-context capability emerges, while different hybrids eventually converge to comparable long-context performance under sufficient training. Second, mechanistically, we show that long-range retrieval is mainly carried by full attention, whereas efficient attention shapes its optimization trajectory. This explains a counter-intuitive phenomenon we call Large-Window Laziness: larger SWA windows can delay the formation of retrieval heads in full-attention layers. Third, guided by this mechanism, we show that applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance with negligible impact on short-context performance. |
cite
@misc{qiao2026rethinking,
title={Rethinking the Role of Efficient Attention in Hybrid Architectures},
author={Ziqing Qiao and Yinuo Xu and Chaojun Xiao and Zhou Su and Zihan Zhou and Yingfa Chen and Xiaoyue Xu and Xu Han and Zhiyuan Liu},
year={2026},
eprint={2606.15378},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.15378}
}