| title | SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference |
|---|---|
| authors | Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa |
| date | 2026-06-03 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.04511 |
| affiliation | NVIDIA |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Sparse attention reduces compute and memory bandwidth for long-context LLM inference. However, two key challenges remain: (1) KV cache capacity still grows with sequence length, and offloading to CPU memory introduces a PCIe transfer bottleneck; (2) the sparse selection step itself retains O(T^2) complexity and can dominate attention cost at long contexts. We propose SparDA, a decoupled sparse attention architecture that introduces a fourth per-layer projection, the Forecast, alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution. Because Forecast is decoupled from the attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi-head selector. SparDA adds <0.5% parameters and trains only the Forecast projections by matching the original selector's attention distribution. On two sparse-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1.25times prefill speedup and 1.7times decode speedup over the sparse-attention offload baseline. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5.3times higher decode throughput than the non-offload sparse baseline. Our source code is available at https://github.com/NVlabs/SparDA. |
cite
@misc{fu2026sparda,
title={SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference},
author={Yaosheng Fu and Guangxuan Xiao and Xin Dong and Song Han and Oreste Villa},
year={2026},
eprint={2606.04511},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.04511}
}