SciResearch-Eval finds scientific agents overconfident when simulators omit physics
Original title: SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
On 93 tasks, most models' 95% intervals miss the truth, while GPT-6 Astra and Claude Fable 5.2 stay near calibrated.
SciResearch-Eval is an interactive benchmark where an agent designs heating experiments, uses a simulator that may be under-resolved or physically wrong, and reports material properties with a 95% interval. On a held-out split of 93 tasks, 9 models and 2,511 episodes, models handle procedural traps, but when physics is missing most stay highly confident while wrong. Coverage is 20-62% for most models, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Stating a strategy improves interval scores; maximum-effort reasoning lifts Claude 5.5 Opus more than tenfold. The environment, graders and harness are released.
Unconfirmed: The claim that this is the first benchmark combining those three properties is the authors' own.
- At default effort, Claude 5.5 Opus names the missing physics in 35% of episodes, versus 66% for Astra.
- Asking for wider intervals does not help; stating the strategy raises scores 1.6-2.5-fold.
- Maximum-effort Opus needs 13-31 times more reasoning and is then not detectably behind default Astra.
- Fable 5.2 scores 1.9 times Astra at default effort but writes 6 times as many tokens.
- Older and open-weight models gain nothing from the effort ablation.
From the original article
1 min read at arxiv.orgWe introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval.
By Brian T. Weston · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.