titlePaperCritic: Training AI Reviewers to Assess the Soundness of Papers
authorsHyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose
date2026-11-13
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.17834
affiliationCarnegie Mellon University
upvotes32 (Hugging Face Daily Papers, 2026-11-19)
linksabs pdf hf
cite
relatedBenchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Training Object Permanence in World Models
Into the Unknown: Benchmarking AI agents on open scientific research problems
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
abstractAs AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).

bibtex

@misc{ryu2026papercritic,
  title={PaperCritic: Training AI Reviewers to Assess the Soundness of Papers},
  author={Hyun Ryu and Ji Yong Cho and Eunho Yang and Moontae Lee and June Yong Yang and Sean Welleck and Carolyn Rose},
  year={2026},
  eprint={2611.17834},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.17834}
}