| title | TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training |
|---|---|
| authors | Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, Jingjing Chen |
| date | 2026-07-07 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.05804 |
| affiliation | Tencent Hunyuan |
| upvotes | 20 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD. |
cite
@misc{zhou2026turnopd,
title={TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training},
author={Yuhang Zhou and Kai Zheng and Haoling Li and Dengyun Peng and Can Xu and Jingjing Chen},
year={2026},
eprint={2607.05804},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.05804}
}