Skip to content
NewA calmer, faster ZeroHour

ExecJudges writes executable programs to replace per-document LLM judges

Original title: Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

arXiv cs.AI / cs.LG / cs.CLby Eugene Agichtein ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On TREC DL 2019-2021, program labels match NIST about as well as Claude Sonnet 5.5, at $0.17 to $0.49 per topic.

Researchers propose ExecJudges, which uses a frontier model to turn assessment guidelines into a readable program for each topic instead of judging every document. Programs rely on lexical matching, standard NLP libraries and pool statistics. On TREC DL 2019-2021, the labels agree with NIST assessors at the level of Claude Sonnet 5.5 judges. With some human labels, ExecJudges-L meets or comes within about 0.05 Krippendorff's alpha of LLM judges given the same labels.

Unconfirmed: Agreement and cost figures are the authors' reported results, not independently replicated.

Key facts
  • Programs use lexical matching, parsers and pool statistics rather than per-document calls.
  • ExecJudges-L can use a small number of human labels per topic.
  • The study compares labels on seven TREC collections with NIST assessors.
  • Labels rank official TREC DL runs with Kendall's tau of 0.87 to 0.94.
  • After the one-time program write, grading has no additional model cost.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments.

arXiv cs.AI / cs.LG / cs.CL

By Eugene Agichtein · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.