Skip to content
NewA calmer, faster ZeroHour

SciResearch-Eval finds scientific agents overconfident when simulators omit physics

Original title: SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

arXiv cs.AI / cs.LG / cs.CLby Brian T. Weston ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On 93 tasks, most models' 95% intervals miss the truth, while GPT-6 Astra and Claude Fable 5.2 stay near calibrated.

SciResearch-Eval is an interactive benchmark where an agent designs heating experiments, uses a simulator that may be under-resolved or physically wrong, and reports material properties with a 95% interval. On a held-out split of 93 tasks, 9 models and 2,511 episodes, models handle procedural traps, but when physics is missing most stay highly confident while wrong. Coverage is 20-62% for most models, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Stating a strategy improves interval scores; maximum-effort reasoning lifts Claude 5.5 Opus more than tenfold. The environment, graders and harness are released.

Unconfirmed: The claim that this is the first benchmark combining those three properties is the authors' own.

Key facts
  • At default effort, Claude 5.5 Opus names the missing physics in 35% of episodes, versus 66% for Astra.
  • Asking for wider intervals does not help; stating the strategy raises scores 1.6-2.5-fold.
  • Maximum-effort Opus needs 13-31 times more reasoning and is then not detectably behind default Astra.
  • Fable 5.2 scores 1.9 times Astra at default effort but writes 6 times as many tokens.
  • Older and open-weight models gain nothing from the effort ablation.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval.

arXiv cs.AI / cs.LG / cs.CL

By Brian T. Weston · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.