EvalResearchBench tests agents that design their own evaluators
Original title: EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
The best evaluators ordered about 75% of candidate pairs like the targets, below a 91% disagreement ceiling.
EvalResearchBench asks whether agents can select or synthesize tasks and graders under fixed time and API budgets, then freeze an executable evaluator for coding, co-work, and reasoning. The study covers 9 researchers and 13 candidate models, compared with 14 target benchmarks on score concordance and pairwise agreement. No researcher led every metric, and the best evaluator on development targets was not best on sealed targets.
- A human-designed sample of public tasks remained a strong baseline.
- The lowest recorded execution cost coincided with the highest pairwise agreement.
- Agent evaluators still truncated answers, exhausted budgets, or let a few questions dominate.
- Products
- EvalResearchBench
From the original article
1 min read at huggingface.co
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research.
By Yaolun Zhang, Tianyi Xu, Yujie Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.