ProteinGeoBench: frontier agents stronger at lookup than geometry
Original title: ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry
On 131 protein-structure tasks, three frontier models scored 0.785 to 0.816 while a fourth scored 0.707.
ProteinGeoBench is a 131-task suite for protein-structure agents. Across 4,249 rollouts, three of four frontier models scored 0.785 to 0.816 with no resolved gap among them, while the fourth scored 0.707. Three of the four did better on metadata lookup than numeric calculation; numeric items are 28 of 131 tasks but half of the 12 tasks all four models fail. Rankings also shift with run budget, harness, and access conditions, which the authors say should be reported by task family.
Unconfirmed: Network-access and domain-blocking contrasts used different passes and cannot be cleanly separated, and task subsets differ across experiments.
- A further 4,486 trials covered 26 models beyond the four frontier systems.
- With one run per task, 23.9% of bootstrap replicates changed the leader.
- Dropping network access lowered scores more than blocking eight database domains.
- Own command-line agents changed scores by -0.071 to +0.086.
- Across 26 models, numeric scores rose with non-numeric scores.
From the original article
1 min read at arxiv.orgFrontier agents retrieve structural metadata well and compute geometry from coordinates less reliably. We ask how much of a benchmark score reflects that gap and how much reflects the evaluation configuration around the model. We study ProteinGeoBench, a suite of 131 protein-structure tasks, using 4,249 rollouts from four frontier models and 4,486 further trials across 26 models.
By Matt Turk · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.