| title | Are Dense Rewards Worth Their Tokens in GRPO? |
|---|---|
| authors | Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary |
| date | 2026-11-10 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.13560 |
| affiliation | Case Western Reserve University |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-13) |
| links | abs pdf hf |
| cite | |
| abstract | RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it. Comparing such a reward with GRPO per training update, or vetting it with a gradient test on a single batch, leaves out what it costs: in our setting, the continuations alone take 91% of all generated tokens. We introduce a matched-compute protocol that gives every training intervention the same budget, measured in generated tokens and training GPU-hours, and tests it in preregistered studies with controls for side effects. Across 1,668 runs on six models, matching compute reverses the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward leads standard GRPO per update, yet ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time, and none of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test orders these rewards as training does but overestimates each by one to two points, as much as the gains it is meant to detect. In contrast, training on the prompts whose predicted success is nearest 1/2 improves accuracy at matched compute, by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. These gains replicate on fresh seeds and unseen problems, and a simple rule, that the candidate prompts are predicted to fail more often than not, predicted where they appear in five of six preregistered tests. Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them. We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others. |
bibtex
@misc{singh2026are,
title={Are Dense Rewards Worth Their Tokens in GRPO?},
author={Vikash Singh and Weicong Chen and Debargha Ganguly and Yanyan Zhang and Mohsen Hariri and Sreehari Sankar and Wang Yang and Chaoda Song and Alexander Nemecek and Osama Zafar and Yuting Shao and Kai Ye and Biyao Zhang and Xinpeng Li and Shouren Wang and Erman Ayday and Jing Ma and Sanmukh Rao Kuppannagari and Vipin Chaudhary},
year={2026},
eprint={2611.13560},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.13560}
}