PremiseCheck fact-checks peer reviews at about half frontier cost
Original title: PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
An LLM rewrites comments as premises, then a decision model checks each premise against the paper.
The paper introduces ReviewFact, a benchmark of factual errors in human and AI peer-review comments across NLP, machine learning and the natural sciences, and PremiseCheck, a two-stage checker. An LLM rewrites each comment as explicit premises without reading the paper, and a cheap decision model checks each premise against the paper. When comments are checked one at a time, PremiseCheck matches GPT-5.5 accuracy across all three fields at about half the cost, and on NLP reviews at a quarter of the cost and less than half the latency.
- ReviewFact labels factual errors under one definition across three fields.
- Frontier checks require reading the whole paper for each comment.
- Cheap decision models fall short on comments that mix several implicit assertions.
- NLP reviews cost about a quarter of GPT-5.5 and take less than half the latency.
From the original article
1 min read at huggingface.co
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy.
By Dingjie Song, Zhiling Yan, Ziqian Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.