| title | On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability |
|---|---|
| authors | Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu |
| date | 2026-08-31 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.30320 |
| affiliation | Qwen |
| upvotes | 61 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable. |
cite
@misc{qiu2026design,
title={On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability},
author={Zihan Qiu and Zekun Wang and Xiao Li and Yanpeng Li and Yang Xu and Yixuan Wang and Huaqing Zhang and Rui Men and Bochao Mao and Chengruidong Zhang and Fan Zhou and Hao Luo and Haofeng Huang and Haoran Lian and Haoyan Huang and Hongqing Chen and Jianwei Zhang and Jing Xu and Junjie Wang and Langshi Chen and Liangyu Wang and Linlang Jiang and Man Yuan and Minmin Sun and Peng Jin and Siqi Zhang and Siyu Wang and Xingzhang Ren and Yakai Wang and Yi Zhang and Yiming Dong and Yizhong Cao and Yubo Ma and Yunfei Mao and Bo Zheng and Dayiheng Liu},
year={2026},
eprint={2608.30320},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.30320}
}