titleAnti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
authorsGuobin Shen, Xiang Cheng, Chenxiao Zhao, Lei Huang, Jindong Li, Dongcheng Zhao, Xing Yu
date2026-05-12
venuearXiv (preprint, not peer reviewed)
idarXiv:2605.11609
affiliationdots studio
upvotes30 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractOn-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.

cite

@misc{shen2026anti,
  title={Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information},
  author={Guobin Shen and Xiang Cheng and Chenxiao Zhao and Lei Huang and Jindong Li and Dongcheng Zhao and Xing Yu},
  year={2026},
  eprint={2605.11609},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.11609}
}