titleDistilled Reinforcement Learning for LLM Post-training
authorsChen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang
date2026-07-19
venuearXiv (preprint, not peer reviewed)
idarXiv:2607.17247
affiliationZhongguancun Academy
upvotes12 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractLarge language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

cite

@misc{wang2026distilled,
  title={Distilled Reinforcement Learning for LLM Post-training},
  author={Chen Wang and Zhaochun Li and Jionghao Bai and Yining Zhang and Hexuan Deng and Ge Lan and Yue Wang},
  year={2026},
  eprint={2607.17247},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2607.17247}
}