titleFrom Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
authorsSiqi Zhu, Suozhi Huang, Kaixuan Zhang, Yuheng Yang, Zhanyang Jin, Yihang Sun, Jiaxuan You
date2026-10-01
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.02179
affiliationUniversity of Illinois at Urbana-Champaign
upvotes11 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractMulti-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97\% of FP32 master weights differ from initialization, but only 7--11\% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.

cite

@misc{zhu2026from,
  title={From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation},
  author={Siqi Zhu and Suozhi Huang and Kaixuan Zhang and Yuheng Yang and Zhanyang Jin and Yihang Sun and Jiaxuan You},
  year={2026},
  eprint={2610.02179},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.02179}
}