TwinLab-Bench tests AI scientists on self-designed laboratory experiments
Original title: TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs
Across 52 physical, life-science and engineering tasks, the paper's strongest model, Claude Fable 5.2, averages 45.0.
TwinLab-Bench evaluates AI agents that design their own experiments rather than only running a given workflow or choosing from predefined measurements. It covers 52 tasks from physical sciences, life sciences and engineering, with more than 100 experimental action types. Each task uses a TwinLab agent that executes designs against real laboratory records, and reports are scored against a hidden answer. The authors say frontier models still struggle and study where Claude Fable 5.2 fails.
- The benchmark has 52 tasks and more than 100 experimental action types.
- Scores compare agent reports with hidden answers from real laboratory records.
- The TwinLab agent nearly eliminates readings that deviate from those records.
- Closed-book probes control for answers recalled from memory.
- Claude Fable 5.2 is reported as the strongest model at an average of 45.0.
- Products
- TwinLab-Bench
- AI models
- Claude Fable 5.2
From the original article
1 min read at huggingface.coAI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates.
By Yang Wan, Linchao Zhu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.