WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
WhatWorkedBench tests whether AI agents can predict experiment outcomes across 36 tasks.
WhatWorkedBench measures whether AI research agents can predict how component changes affect outcomes after a limited experiment budget. Agents inspect code, choose measurements, and submit a response surface covering every configuration. Reference effects come from exhaustive CPU runs across 36 tasks, 30 data sources, eight workflow types, and 1,248 configuration records. Fitting a Gaussian process to the same observations raises effect recovery, and encoding code equivalences further improves it.
- Agents predict scores for every configuration after budgeted experimentation.
- Coverage includes 36 tasks, 30 sources, eight workflows, and 1,248 configurations.
- Gaussian process fitting raises Flash-cohort recovery from 0.632 to 0.698.
- Encoding code equivalences lifts recovery from 0.248 to 0.462.
From the original article
1 min read at huggingface.co
AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding, the accuracy of predictions about component changes after budgeted experimentation. Agents inspect code, select measurements, and submit a response surface, a table predicting scores for every configuration of component settings.
By Jingjie Ning, Xueqi Li, Yibo Kong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.