Benchmark plants 100 errors in psychology papers to test AI review
Original title: How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
The best system found 71 errors and the pooled systems found 93; seven missed errors were all omissions.
Researchers planted 100 known errors into 10 open-access psychology papers and ran the modified papers through frontier models and two commercial AI review tools. The best single system caught 71 errors and the worst caught 30; pooling outputs caught 93. Models only partly overlapped in what they found, so ensembling helped. Seven errors escaped every system, and all seven were omissions rather than inserted mistakes. Refine.ink contributed more unique catches than any other system, though at a higher cost per error.
Unconfirmed: False positives were not measured, and how the planted errors compare with real papers is unknown.
- Ten open-access psychology papers received the planted errors.
- Pooling every system caught 93 of 100 errors.
- All seven uncaught errors were deleted information, not inserted mistakes.
- Papers, errors, model outputs, and the experiment log are public.
- False positives were not measured.
From the original article
1 min read at arxiv.orgOne hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93.
By Paul Litvak · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.