Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers
Researchers introduce SciSlopBench, detecting AI-generated scientific papers at 85.9% accuracy via reasoning-breakdown measures, plus SciSlopHarness to mitigate slop.
The paper builds SciSlopBench, a benchmark of 390 AI-generated papers spanning computer, life, social, and natural sciences, each paired with a human-written paper matched by problem and contribution. Six measures covering Structure, Argument, and Artifacts identify the AI paper with 85.9% accuracy versus 68.7% for Binoculars. Higher slop accompanies lower ICLR ratings and distinguishes rejected from accepted papers above chance every year from 2017 to 2025. The proposed SciSlopHarness framework reduces the remaining AI-human gap by 63% over the strongest revision baseline while avoiding the reward hacking triggered by direct slop-aware prompting.
- SciSlopBench pairs 390 AI-generated papers with matched human-written papers
- Six measures detect AI papers at 85.9% accuracy, beating Binoculars' 68.7%
- Higher slop correlates with lower ICLR ratings and paper rejection
- SciSlopHarness cuts the AI-human gap by 63% without reward hacking
- Organizations
- ICLRBinocularsSciSlopBench
From the original article
1 min read at huggingface.co
AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however, has more complex patterns that cannot be easily detected by existing token-based AI detectors. Each part of such a paper looks plausible while the scientific reasoning that connects the parts breaks down, which can mislead how readers assess the work.
By Yerim Oh, Young-Jun Lee, Jaewoo Ahn · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.