titleAccelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining
authorsYingqi Shi, Junru Lu, Yun Liu, Di Yin, Xing Sun
date2026-11-19
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.26268
affiliationTencent
upvotes12 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf code
cite
relatedEvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary World
A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
abstractMixture of Block Attention (MoBA) reduces long-context computation by selecting key-value (KV) blocks for each query. FlashMoBA implements MoBA efficiently but retains a fixed Top-k budget and query-centric schedule. Fixed Top-k applies one budget across different score distributions and couples routing to task granularity; query-range ownership fragments K/V reuse when several ranges select the same column. We present LightningMoBA, which co-designs selection and execution through dynamic routing, KV-centric ownership, and runtime pipelining. Dynamic routing thresholds MoBA's per-query block scores and directly emits variable-cardinality sparse metadata. KV-centric ownership groups selected rows by (query head, KV block) column, splits high-fanout columns into bounded tasks, and reuses each task's loaded KV tile. On Hopper, pipelining overlaps regular K/V transfers and irregular query gathers with Tensor Core computation. Together, these mechanisms decouple retention from GPU scheduling granularity. At matched Top-16 retention across five checkpoints, LightningMoBA accelerates end-to-end generation by 1.20 to 1.60x while raising mean RULER score from 74.42 to 75.28.

bibtex

@misc{shi2026accelerating,
  title={Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining},
  author={Yingqi Shi and Junru Lu and Yun Liu and Di Yin and Xing Sun},
  year={2026},
  eprint={2611.26268},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.26268}
}