| title | PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews |
|---|---|
| authors | Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun |
| date | 2026-10-15 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.21046 |
| upvotes | 34 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency. |
cite
@misc{song2026premisecheck,
title={PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews},
author={Dingjie Song and Zhiling Yan and Ziqian Zhang and Yuxuan Zhang and Lichao Sun},
year={2026},
eprint={2610.21046},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.21046}
}