| title | Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading |
|---|---|
| authors | Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia, Xinkai WAng, Mingxuan Zhang, Peng Tang, Jiacheng Liu, Chao Li |
| date | 2026-10-20 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2511.14102 |
| upvotes | 26 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos. |
cite
@misc{wang2026specter,
title={Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading},
author={Wenfeng Wang and Xiaofeng Hou and Xinfeng Xia and Xinkai WAng and Mingxuan Zhang and Peng Tang and Jiacheng Liu and Chao Li},
year={2026},
eprint={2511.14102},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2511.14102}
}