| title | Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL |
|---|---|
| authors | Skylar Zhai, Haonan Wang, Zhining Liu, Linxin Song, Xuwei Ding, Jingcheng Liang, Jian Kang |
| date | 2026-10-15 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.21439 |
| affiliation | Mohamed Bin Zayed University of Artificial Intelligence |
| upvotes | 36 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods. |
cite
@misc{zhai2026each,
title={Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL},
author={Skylar Zhai and Haonan Wang and Zhining Liu and Linxin Song and Xuwei Ding and Jingcheng Liang and Jian Kang},
year={2026},
eprint={2610.21439},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.21439}
}