ExecJudges writes executable programs to replace per-document LLM judges
Original title: Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling
On TREC DL 2019-2021, program labels match NIST about as well as Claude Sonnet 5.5, at $0.17 to $0.49 per topic.
Researchers propose ExecJudges, which uses a frontier model to turn assessment guidelines into a readable program for each topic instead of judging every document. Programs rely on lexical matching, standard NLP libraries and pool statistics. On TREC DL 2019-2021, the labels agree with NIST assessors at the level of Claude Sonnet 5.5 judges. With some human labels, ExecJudges-L meets or comes within about 0.05 Krippendorff's alpha of LLM judges given the same labels.
Unconfirmed: Agreement and cost figures are the authors' reported results, not independently replicated.
- Programs use lexical matching, parsers and pool statistics rather than per-document calls.
- ExecJudges-L can use a small number of human labels per topic.
- The study compares labels on seven TREC collections with NIST assessors.
- Labels rank official TREC DL runs with Kendall's tau of 0.87 to 0.94.
- After the one-time program write, grading has no additional model cost.
- Organizations
- NIST
- AI models
- Claude Sonnet 5.5
From the original article
1 min read at arxiv.orgLarge language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments.
By Eugene Agichtein · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.