| title | Trust Region Policy Distillation |
|---|---|
| authors | Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, Mao Yang |
| date | 2026-07-06 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.04751 |
| affiliation | HKUSTGZ |
| upvotes | 34 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm. |
cite
@misc{xie2026trust,
title={Trust Region Policy Distillation},
author={Zhengpeng Xie and Li Lyna Zhang and Zeke Xie and Mao Yang},
year={2026},
eprint={2607.04751},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.04751}
}