Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Heming Zou; Qi Wang; Yixiu Mao; Yun Qu; Yuyi Guo; Zeyuan Liu; Chenglin Li; Weijie Liu; Saiyong Yang; Xiangyang Ji

title

authors

Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji

date

2026-10-16

venue

arXiv (preprint, not peer reviewed)

arXiv:2610.23181

affiliation

Tencent Hunyuan

upvotes

53 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf

abstract

Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.

@misc{zou2026tree,
  title={Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning},
  author={Heming Zou and Qi Wang and Yixiu Mao and Yun Qu and Yuyi Guo and Zeyuan Liu and Chenglin Li and Weijie Liu and Saiyong Yang and Xiangyang Ji},
  year={2026},
  eprint={2610.23181},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.23181}
}