Rubricon generates LLM rubrics without human preference labels
Original title: Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training
Rubric scores align 9% more with expert rubrics than the best compared synthetic method, and lift Qwen3.5-9B on HealthBench.
Rubricon is a pipeline that samples rubrics with any LLM, keeps the most discriminative candidate, adds overlooked criteria, and prunes misaligned or shallow ones, without human preference labels, rubric annotation, or generator-model training. Refinement uses LLM self-critique, test-time scaling, and observed criterion pass rates on example responses. Across five professional, consumer, clinical, and general-use benchmarks, its scores of frontier LLM responses show 9% greater alignment with expert-authored rubrics than the best alternative among RRD, QSRG, OpenRubrics, and agentic generation. On HealthBench, post-training Qwen3.5-9B with Rubricon rubrics yields 4.5% better performance than rubrics from RRD.
- Rubricon does not require human preference labels or generator-model training.
- Compared methods include RRD, QSRG, OpenRubrics, and Codex-style agentic generation.
- The 9% alignment gain is versus the best of those synthetic generators.
- The HealthBench gain is 4.5% over RRD when post-training Qwen3.5-9B.
From the original article
1 min read at arxiv.orgRubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training.
By Ulyana Tkachenko · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.