| title | SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving |
|---|---|
| authors | Gunho Park, Kyoungho Jeun, Juntaek Oh, Byeongjun Shin, Baeseong Park, Minsoo Rhu |
| date | 2026-09-28 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.34117 |
| upvotes | 18 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Mixture-of-experts (MoE) models activate few experts per token, yet batched decoding can access nearly the entire expert pool, making expert-weight traffic a major bottleneck. Expert pruning reduces this traffic, but conventional approaches also prune compute-bound prefill, sacrificing model quality for little throughput benefit. We present SlimWise, a serving framework that tailors the expert pool to each inference phase. SlimWise performs prefill with the full model and decode with a pruned model that directly reuses the prefill-generated KV cache without conversion. Across two MoE backbones and three pruning criteria, this training-free KV cache handoff substantially narrows accuracy gaps relative to the full model in many settings. We also show that benchmark accuracy can conceal substantial pruning-induced changes in generation length. To address these distortions and residual accuracy loss, SlimWise introduces a low-cost distillation stage that trains the decoder to continue from full-model KV caches while updating only a small subset of parameters. Implemented in vLLM, SlimWise supports both prefill-decode (PD) disaggregation and PD-colocated serving. On Qwen3.6-35B-A3B, SlimWise improves decode throughput by up to 1.81x at 50% expert pruning with minimal accuracy loss. |
cite
@misc{park2026slimwise,
title={SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving},
author={Gunho Park and Kyoungho Jeun and Juntaek Oh and Byeongjun Shin and Baeseong Park and Minsoo Rhu},
year={2026},
eprint={2609.34117},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.34117}
}