SciGame tests agents on real and counterfactual scientific laws
Original title: SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
On five frontier models, recovering the exact symbolic form stays the main bottleneck.
SciGame converts scientific literature into structured cards, each pairing an Original environment with a dimensionally consistent Counterfactual one under the same task interface. A Historical collection covers canonical laws and a Temporal collection uses papers published after model knowledge cutoffs. Agents are scored in Identification and Discovery under Prior-only, Fixed and Adaptive protocols. Experiments on five frontier models find familiar priors hurt once the governing relation changes, adaptive experimentation helps only the strongest models under limited budgets, and good explanations can hide quantitative failures.
Unconfirmed: The paper does not name the five frontier models.
- Existing benchmarks mix law inference with programming and score data fitting.
- Each card keeps the same task interface across original and counterfactual worlds.
- Adaptive experimentation harms weaker models when the experiment budget is limited.
- Inducing exact laws in unfamiliar settings remains an open challenge.
From the original article
1 min read at huggingface.co
Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards.
By Yusheng Zhao, Bowei He, Xue Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.