| title | Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data |
|---|---|
| authors | Geunhyoung Kim, Sung Ju Hwang |
| date | 2026-10-27 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.37386 |
| affiliation | KAIST AI |
| upvotes | 17 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model's weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one and up to 2.69x at larger batches on the same three models. |
cite
@misc{kim2026channel,
title={Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data},
author={Geunhyoung Kim and Sung Ju Hwang},
year={2026},
eprint={2610.37386},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.37386}
}