| title | Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following |
|---|---|
| authors | Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim |
| date | 2026-10-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.19525 |
| upvotes | 9 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers. |
cite
@misc{yoon2026diagnosing,
title={Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following},
author={Sangyeon Yoon and Wooseok Seo and Hamish Ivison and Hyesoo Hong and Donghoon Kim and Wonjun Kang and Albert No and Jaehyung Kim},
year={2026},
eprint={2610.19525},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.19525}
}