| title | Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization |
|---|---|
| authors | Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li, Deng Cai, Min Chen, Hao Zhang |
| date | 2026-07-16 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.14614 |
| upvotes | 11 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance. |
cite
@misc{xu2026beyond,
title={Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization},
author={Weiwen Xu and Jia Liu and Hou Pong Chan and Long Li and Deng Cai and Min Chen and Hao Zhang},
year={2026},
eprint={2607.14614},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.14614}
}