JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
A decision-only JEV judge nearly matches a top LLM judge at a fraction of the cost.
JEV-as-a-judge is compared with sixteen generative and reward-model judges under blinded human adjudication. It stays within three percentage points of the strongest LLM judge on ordinary preference and evidence-grounded factuality at 0.36% of that judge's fee. Larger gaps appear when checking derivations or resisting elaborate wrong answers. A frozen cascade that escalates low-confidence cases retains 99% of comparator accuracy at lower cost.
- Within three points of the strongest comparator on preference and factuality.
- Its fee is 0.36% of the comparator's cost.
- Larger gaps appear on derivations and elaborate wrong answers.
- A confidence cascade retains 99% of comparator accuracy.
From the original article
1 min read at huggingface.co
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed.
By Yubo Li, Yidi Miao, Ramayya Krishnan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.