When LLM judges agree, should we believe them?
Amazon ICML paper uses Ising models to correct correlated LLM-judge votes, beating accuracy-weighted panels by 9-14%.
Amazon Science describes an ICML paper, "Dependence-aware label aggregation for LLM-as-a-judge via Ising models," addressing how correlated judge outputs inflate majority-vote confidence. The unsupervised method models pairwise dependence between judges, learning both reliability and similarity without human reference labels. Tested on relevance, toxicity, and summarization tasks with 10 judge models at temperature zero, it outperformed accuracy-weighted voting by 9% to 14%.
- Correlated LLM judges make majority votes look stronger than they are
- Ising model learns judge reliability plus pairwise dependence, unsupervised
- Outperforms accuracy-weighted baseline by 9-14% across three tasks
- Learned judge network helps identify redundant judges and shared blind spots
From the original article
5 min read at amazon.science
Imagine evaluating a retrieval-augmented-generation system. A user asks a question, the system retrieves a text passage, and an LLM judge decides whether it’s relevant. To reduce noise, you ask several judge models to evaluate the same passage.
Hacker News · AIBy Betelbuddy · amazon.science
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.