| title | SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning |
|---|---|
| authors | Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, Yu Cheng |
| date | 2026-08-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.14277 |
| affiliation | Shanghai AI Laboratory |
| upvotes | 36 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain. |
cite
@misc{he2026simpleopd,
title={SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning},
author={Haonan He and Haodi Lei and Yun Luo and Haoran Zhang and Shunkai Zhang and Yizhuo Li and Shengji Tang and Zhilin Wang and Runzhe Zhan and Lei Bai and Ganqu Cui and Fangchen Yu and Yafu Li and Peng Ye and Ning Ding and Yu Cheng},
year={2026},
eprint={2608.14277},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.14277}
}