| title | On-Policy Distillation Teaches New Skills but Not New Knowledge |
|---|---|
| authors | Yixuan Tang, Yi Yang |
| date | 2026-10-07 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.09639 |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses. |
cite
@misc{tang2026policy,
title={On-Policy Distillation Teaches New Skills but Not New Knowledge},
author={Yixuan Tang and Yi Yang},
year={2026},
eprint={2610.09639},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.09639}
}