OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
Rubric-privileged on-policy distillation before RL beat SFT-then-RL on health and science benchmarks.
The paper proposes a two-stage method for tasks that lack exact outcome verification: rubric-privileged on-policy distillation (RP-OPD), then rubric-based reinforcement learning. A student without the rubric matches a rubric-aware teacher's next-token distributions on student-generated prefixes; RL then improves beyond the distillation plateau. On open-weight models, RP-OPD plus RL scored highest on HealthBench, ResearchQA, and RubricHub Science, with fewer signs of reward hacking than an SFT-plus-RL baseline.
- RP-OPD distills a rubric-aware teacher into a student that never sees the rubric.
- A second RL stage then optimizes the rubric reward past the distillation plateau.
- Best scores on HealthBench, ResearchQA, and RubricHub Science among methods tested.
- SFT plus RL showed more reward hacking than RP-OPD plus RL on RubricHub Science.
From the original article
1 min read at huggingface.co
Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by scoring open-ended responses against explicit criteria. However, because the reward is assigned after the complete response, the training signal does not directly identify which individual decisions contributed to the final score.
By Xinpeng Wang, Wei Shi, Yu-Chia Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.