| title | Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns |
|---|---|
| authors | Deyang Kong, Zhaoyi Li, Sijian Ren, Yuan Wei, Ying Wei, Defu Lian, Shikun Zhang, Wei Ye |
| date | 2026-11-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.19244 |
| upvotes | 7 (Hugging Face Daily Papers, 2026-11-19) |
| links | abs pdf hf |
| cite | |
| abstract | On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and τ^2-bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces τ^2-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises τ^2 accuracy from 11.5% to 29.9% with a source-learning trade-off. |
bibtex
@misc{kong2026agentic,
title={Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns},
author={Deyang Kong and Zhaoyi Li and Sijian Ren and Yuan Wei and Ying Wei and Defu Lian and Shikun Zhang and Wei Ye},
year={2026},
eprint={2611.19244},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.19244}
}