AI reviewers fail to consistently detect paper revision improvements
Original title: Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories
Large-scale study of 6,959 AI paper trajectories shows major revisions rarely raise AI review scores and user opinions can override them.
AI reviewers assess scientific manuscripts but it is unclear if they reflect author revision gains. PaperTrajectory-2026 dataset provides 6,959 trajectories across five conferences with annotations on 14 revision dimensions. Independent assessments by eight AI reviewers find every reviewer leaves scores unchanged on most major improvements. Agreement on score change direction is weak. Even higher neutral scores can be erased by brief negative user opinions.
- Major improvements emphasize empirical evidence and framing
- Later revisions tend toward incremental improvement
- Reviewer-specific associations with score increases exist
- User opinions outweigh observed revision gains in some cases
From the original article
1 min read at huggingface.co
AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision.
By Yaochen Wang, Chenguang Wang, Ming Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.