| title | Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents |
|---|---|
| authors | Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick, Sharon Li |
| date | 2026-06-24 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.26080 |
| affiliation | University of Wisconsin - Madison |
| upvotes | 13 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems. |
cite
@misc{oh2026neglected,
title={Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents},
author={Changdae Oh and Wendi Li and Seongheon Park and Samuel Yeh and Tanwi Mallick and Sharon Li},
year={2026},
eprint={2606.26080},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.26080}
}