| title | RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards |
|---|---|
| authors | Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister |
| date | 2026-05-11 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.10899 |
| affiliation | |
| upvotes | 56 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| primer | Memory for AI agents |
| abstract | Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory. Based on this view, we introduce RubricEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution. RubricEM first makes research trajectories stage-aware by conditioning planning, evidence gathering, review, and synthesis on self-generated rubrics. It then assigns credit with Stage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback for long-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM. |
cite
@misc{li2026rubricem,
title={RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards},
author={Gaotang Li and Bhavana Dalvi Mishra and Zifeng Wang and Jun Yan and Yanfei Chen and Chun-Liang Li and Long T. Le and Rujun Han and George Lee and Hanghang Tong and Chen-Yu Lee and Tomas Pfister},
year={2026},
eprint={2605.10899},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.10899}
}