| title | Self-Distilled Policy Gradient |
|---|---|
| authors | Yifeng Liu, Shiyuan Zhang, Yifan Zhang, Quanquan Gu |
| date | 2026-06-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.04036 |
| affiliation | University of California, Los Angeles |
| upvotes | 27 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG. |
cite
@misc{liu2026self,
title={Self-Distilled Policy Gradient},
author={Yifeng Liu and Shiyuan Zhang and Yifan Zhang and Quanquan Gu},
year={2026},
eprint={2606.04036},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.04036}
}