AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents
AutoSciBench automatically generates scientific-agent benchmarks that lower solver accuracy versus human-curated tests.
AutoSciBench represents scientific tasks as high-level concepts and low-level recipes, then revises them using solver trajectories and judge feedback. It was evaluated from existing benchmarks in computational biology, materials science, and clinical imaging. Generated benchmarks reduced average solver accuracy by 22.4 and 25.5 percentage points in computational biology and materials science, while scoring higher average quality across all three domains.
- Tasks pair a scientific concept with a verifiable construction recipe.
- Solver traces and judge feedback close shortcuts and raise difficulty.
- Accuracy fell 22.4 and 25.5 points in biology and materials science.
- Generated tasks received higher quality ratings across three domains.
From the original article
1 min read at huggingface.co
As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities.
By Dongki Kim, Namkyeong Lee, Surag Nair · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.