Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

Yunho Choi; Jongwon Lim; Woojin Ahn; Minjae Oh; Jeonghoon Shim; Yohan Jo

title	Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
authors	Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo
date	2026-05-08
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2605.07579
affiliation	Seoul National University
upvotes	8 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf code
abstract	Reinforcement learning with verifiable rewards (RLVR) for Large Reasoning Models hinges on baseline estimation for variance reduction, but existing approaches pay a heavy price: PPO requires a policy-model scale critic, while GRPO needs multiple rollouts per prompt to keep its empirical group mean stable. We introduce Policy Optimization with Internal State Value Estimation), which obtains a baseline at negligible cost by using the policy model's internal signals already computed during the policy forward pass. A lightweight probe predicts the expected verifiable reward from the hidden states of the prompt and generated trajectory, as well as token-entropy statistics, and is trained online alongside the policy. To preserve gradient unbiasedness despite using trajectory-conditioned features, we introduce a cross-rollout construction that predicts each rollout's value from an independent rollout's internal states. Because POISE estimates prompt value using only a single rollout, it enables higher prompt diversity for a fixed compute budget during training. This reduces gradient variance for more stable learning and also eliminates the compute overhead of sampling costs for detecting zero-advantage prompts. On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B across math reasoning benchmarks, POISE matches DAPO while requiring less compute. Moreover, its value estimator shows similar performance to a separate LLM-scale value model and generalizes to various verifiable tasks. By leveraging the model's own internal representations, POISE enables more stable and efficient policy optimization.

cite

@misc{choi2026your,
  title={Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States},
  author={Yunho Choi and Jongwon Lim and Woojin Ahn and Minjae Oh and Jeonghoon Shim and Yohan Jo},
  year={2026},
  eprint={2605.07579},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.07579}
}