| title | Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD |
|---|---|
| authors | Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li |
| date | 2026-09-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.29142 |
| affiliation | East China Normal University |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Direct On-Policy Distillation (Direct-OPD) transfers reinforcement-learning-induced policy improvements from a small model to a larger student by using the token-level log-ratio between post-RL and pre-RL checkpoints as dense supervision on the student's own rollouts. This transfer rewards the policy shift at every state, yet the log-ratio measures only relative change: it can stay fixed even as the probability mass that both checkpoints assign to the student's candidate tokens vanishes. Through an exact construction, we show that the Direct-OPD reward and its update can remain unchanged while the Jensen-Shannon divergence (JSD) and both KL directions between the checkpoints vanish with this mass, and we note that a small JSD bounds how much the teacher's behavior changed. Motivated by this analysis, we propose Selective Supervision for Direct-OPD (S^2D-OPD), which ranks student-sampled states by their teacher-reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% of states per response. Across two teacher pairs and four student models ranging from 1.7B to 8B parameters, S^2D-OPD improves held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings and matches it in the eighth, without extra forward passes. |
cite
@misc{zhao2026not,
title={Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD},
author={Yibo Zhao and Zixuan Yang and Yunshi Lan and Xiang Li},
year={2026},
eprint={2609.29142},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.29142}
}