Rubricon generates LLM rubrics without human preference labels
Rubric scores align 9% more with expert rubrics than the best compared synthetic method, and lift Qwen3.5-9B on HealthBench.
Stories per day · 30 days
1 RSS
Relatedalignment1deep-research-agents1evaluation1financial-research1healthbench1health-coaching1llm-benchmark1llm-evaluation1llm-judges1precision1qwen1reinforcement-learning1reward-hacking1reward-modeling1safety-evaluation1sports1
9 stories, newest first · one card per story
Stories tagged #rubrics
RubricAtlas mines type-level rubrics to score deep research agents
Built from 3,079 Chinese financial reports across 99 types, judge pipelines agreed at kappa 0.71 and ranking tau 0.87.
MoveBench evaluates 15 LLMs on sports and health coaching
Models were scored on 809 questions and 5,012 rubric criteria covering safety, validity, and personalization.
FORK reward adds precision to curb rubric-based RL reward hacking
Against the base policy, win rate rises from 10.6% to 68.0% on RubricHub-Medical and to 74.1% on HealthBench.
SERA trains recursive agents to score their own subtasks
On two text benchmarks, SERA beat recursive-agent baselines by 5.38 and 13.14 points on average.
OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
Rubric-privileged on-policy distillation before RL beat SFT-then-RL on health and science benchmarks.
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
FinAutoRubric generates expert-guided, cutoff-aware rubrics for evaluating financial research agents.
Rubric Rewards from Item Response Theory
Rubric Response Theory turns rubric verdicts into scalar RL rewards and cuts judge calls while beating GRPO.
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
ImpossibleRubrics benchmark shows LLM-generated rubric reward signals are exploited 8-26% of the time by adversarial answers on impossible tasks.