ExecJudges writes executable programs to replace per-document LLM judges
On TREC DL 2019-2021, program labels match NIST about as well as Claude Sonnet 5.5, at $0.17 to $0.49 per topic.
8 stories, newest first · one card per story
On TREC DL 2019-2021, program labels match NIST about as well as Claude Sonnet 5.5, at $0.17 to $0.49 per topic.
In 43 shadow-mode days, TriAgent triaged 499 tickets and judges called 81.1% of verifiable assertions correct.

Combining Jev with Claude Opus provides frontier accuracy at over 4 times lower cost than frontier models alone for grading identity resolution.
Built from 3,079 Chinese financial reports across 99 types, judge pipelines agreed at kappa 0.71 and ranking tau 0.87.
DeCert raises position-consistent accuracy by up to 14.8 points without reference explanations.
A tool-augmented agent compiled 87.2% of 227 statements, but only 57.7% passed a two-judge semantic check.
Holdout Best-of-N unbiasedly estimates selection reward only when selector depth is below the score count.
Rubric Response Theory turns rubric verdicts into scalar RL rewards and cuts judge calls while beating GRPO.
Today 1K readers · 1.1K views · 101 reads · all time 211K views