SciΠvot: frontier models lag humans on research judgment
Original title: SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
Best project-condition accuracy is 63.5%, versus 73.4% for fresh researchers on the same cases.
SciΠvot is a 248-case benchmark of scientific judgment built from human corrections during six months of LLM-agent research. Senior researchers ran agents on 26 projects across six fields and intervened when judgments were wrong. Given only the pre-correction project record, an agent must choose the original LLM plan or the human-corrected alternative. At medium reasoning effort with the record, models prefer the original plan, and four of five frontier models score below chance on the selected cases.
- The benchmark has 248 cases from 26 projects across six scientific fields.
- Highest project-condition accuracy across models and effort settings is 63.5%.
- Fresh researchers score 73.4% on the same selected cases.
- At medium effort, every model is less accurate with the project record than without it.
- Reconsideration does not consistently improve overall accuracy.
From the original article
1 min read at huggingface.co
Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct SciΠvot, a 248-case benchmark of scientific judgment based on human corrections.
By Youran Sun, Xingyu Ren, Chugang Yi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.