Veritas harness auto-generates agent-task verifiers that beat seven alternatives
Original title: Veritas: A Verifier Generation Harness for Agent Tasks
With the model fixed, generated verifiers matched official verdicts better than seven harnesses on all four execution-scored benchmarks.
Researchers propose Veritas, a harness that writes verifiers for agent tasks instead of relying on hand-authored ones. It adds an evidence header, a library of unstated domain rules, and authoring principles partly refined by a self-improvement loop. On six benchmarks, four execution-scored and two pairwise, Veritas agreed with official verdicts better than seven other harnesses on every execution benchmark and was at or near the best on the pairwise ones. Its mechanisms also improved 9 of 12 harness-benchmark pairs scored by execution across three harnesses.
Unconfirmed: Agreement is with official verdicts under a fixed model, not an independent measure of verifier correctness.
- Official verifiers on the four execution benchmarks were certified first.
- Generated verifiers also beat the vendor harness for two of three other model families.
- Evaluation covers four execution-scored benchmarks and two pairwise response benchmarks.
From the original article
1 min read at arxiv.orgThe effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically.
By Daoan Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.