SciExam for ENSO: Can AI Agents Build Climate Models?
SciExam for ENSO tests whether AI agents can build competitive climate models of El Nino from observations alone.
SciExam for ENSO is a benchmark where language-model agents build low-order stochastic models of the El Nino-Southern Oscillation from real observations within a six-hour budget. Agents write diagnostics that are then frozen and may use only those diagnostics as feedback; hidden graders score statistics, recovery of unobserved variables, and forecasts of held-out years against a published model. Across twelve agent systems, six outscore the published baseline, mainly via better reconstruction and forecasting. Stronger models' simplified forms align with competing explanations of ENSO warm-cold asymmetry, and controls suggest scores are not from memorizing the observational record.
- Agents build ENSO models from observations inside a six-hour budget.
- Six of twelve systems score above a published reference model.
- Hidden graders test statistics, hidden variables, and held-out forecasts.
- Controls suggest success is not memorization of the climate record.
From the original article
1 min read at arxiv.orgLanguage-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid.
By Yinling Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.