titleTideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices
authorsYifei Wang, Tsun-yu Yang, Yechen Xu, Hancheng Ye, Qinsi Wang, Baijun Zhou, Tingjun Chen, Hai "Helen" Li, Danyang Zhuo, Yiran Chen
date2026-11-10
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.14848
affiliationDuke University
upvotes65 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf code
cite
relatedGAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning
CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video
Who to Trust in Multi-Robot Systems: Joint Robot and Object Hypothesis Trust Inference via Neural-Symbolic Cross-Validation
dictyBase 2026: a reimplementation of the Dictyostelium model organism database on minimal infrastructure
AI-generated language remains systematically distinguishable from human communication
abstractLarge language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as independent caches and thrashing in inference. We present TIDEWAY, a system for efficient inference of large MoE models on memory-constrained consumer devices. Our core insight is to manage host RAM and GPU VRAM as a unified memory pool. Within this pool, TIDEWAY couples phase-aware memory layouts, which adapt to the distinct access patterns of prefill and decode, with an execution engine that optimizes CUDA Graph execution under expert paging. Our evaluations on representative consumer hardware with frontier MoE models demonstrate that TIDEWAY significantly outperforms state-of-the-art systems. On average, TIDEWAY achieves speedups of 1.97x in end-to-end latency and 2.40x in Time-to-First-Token (TTFT), along with 1.61x higher decode throughput against the fastest baseline. Our source code is publicly available at https://github.com/Afterglow231/Tideway .

bibtex

@misc{wang2026tideway,
  title={Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices},
  author={Yifei Wang and Tsun-yu Yang and Yechen Xu and Hancheng Ye and Qinsi Wang and Baijun Zhou and Tingjun Chen and Hai "Helen" Li and Danyang Zhuo and Yiran Chen},
  year={2026},
  eprint={2611.14848},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.14848}
}