titleThe Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs
authorsXin Li, Hao Jiang, Annan Wang, Yichi Zhang, Chau Yuen
date2026-05-09
venuearXiv (preprint, not peer reviewed)
idarXiv:2605.08737
affiliationNanyang Technological University
upvotes2 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractOn-policy distillation (OPD) is widely used for LLM post-training. When pushed with a reward-extrapolation coefficient lambda > 1, the student can lift past the teacher in domain, but past a threshold lambda* the same step violates the output contract on structured-output tasks. In a single-position Bernoulli reduction, we derive a closed-form base-relative clip-safety threshold lambda*(p,b,c) determined by three measurable quantities: the teacher modal probability, the warm-start mass, and the importance-sampling clip strength. Above lambda*, the extrapolated fixed point exits the clip-safe region, changing training from format-preserving to format-collapsing. We extend the rule to calibrated K-ary listwise JSON tasks where a single binding equivalence class dominates the output contract and SFT retains parse headroom. On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval, a budget-extension test, and a small-clip cross-prediction--fall within their locked prediction windows, with the small-clip value matching the closed-form prediction below grid resolution. Operating just below lambda*, ListOPD brings a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters. The gain is driven primarily by format adherence: NDCG@1 on parsed outputs remains flat across lambda, while parse validity sharply changes at the predicted boundary. The cliff diagnostic is rubric-independent, whereas the parity claim uses a Gemini-graded rubric and inherits that evaluator's exposure.

cite

@misc{li2026extrapolation,
  title={The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs},
  author={Xin Li and Hao Jiang and Annan Wang and Yichi Zhang and Chau Yuen},
  year={2026},
  eprint={2605.08737},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.08737}
}