VerifierBench tests whether agents can write executable solution verifiers
Original title: VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Across 19 models, normalized balanced accuracy ranged from 10.6 to 62.1 on 222 tasks.
VerifierBench measures whether an agent can write an executable verifier that accepts every valid solution and rejects invalid ones. It covers 222 tasks from six agent benchmarks and 4,993 hidden test cases, screened by Verifier4Verifier. Generated verifiers rejected invalid solutions more reliably than they accepted valid ones. Many false rejections came from verifiers that accepted no tested solution.
- Tasks span tool-call, state-based, and code verification.
- Most analyzed false rejections came from verifiers accepting no tested solution.
- Self-tests often reused mistaken assumptions about formats, tools, or code changes.
- The project page and dataset are publicly available.
- Products
- VerifierBenchVerifier4Verifier
From the original article
1 min read at huggingface.co
Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones.
By Jianing Yin, Ziyi Chen, Hao Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.