Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
A position-masking self-distillation method trains LLM judges from language feedback and beats outcome-only RL on subjective tasks.
The paper trains LLM judges from natural-language feedback rather than a single scalar outcome reward such as GRPO. Self-distillation uses the same model, conditioned on that feedback, as a teacher, and position masking keeps the lower tail of the teacher-student entropy-shift distribution so the student preserves semantic alternatives instead of memorizing one criterion phrase. Masking higher-entropy-shift positions improves out-of-distribution generalization compared with naive self-distillation. The resulting judges outperform outcome-supervised RL by 2 to 9 percentage points on subjective subcategories while remaining competitive on objective ones.
- Uses language feedback as dense position-level self-distillation supervision.
- Entropy-shift masking drops high-shift positions that encourage memorization.
- Improves out-of-distribution generalization over naive self-distillation.
- Gains 2 to 9 points on subjective tasks versus outcome-supervised RL.
From the original article
1 min read at huggingface.co
We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e. g.
By Ilgee Hong, Changlong Yu, Zhenghao Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.