FORK reward adds precision to curb rubric-based RL reward hacking
Original title: Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Against the base policy, win rate rises from 10.6% to 68.0% on RubricHub-Medical and to 74.1% on HealthBench.
The paper says rubric-based rewards usually maximize coverage, so reinforcement-learning models become more complete but also more verbose and more likely to add unnecessary or incorrect content. FORK jointly optimizes recall and a precision objective that requires each response unit to be necessary and factually correct, using rubric-driven necessity credits and a factual-correctness grader. Compared with the recall-only reward, it improves precision and shortens responses while keeping similar rubric coverage. The authors report Pareto-optimal coverage-versus-quality results and higher win rates than the base policy.
Unconfirmed: Win-rate figures are the authors’ experimental results, not an independent evaluation.
- Win rate versus the base policy moves from 22.1% to 74.1% on HealthBench.
- On RubricHub-Medical the same comparison moves from 10.6% to 68.0%.
- Precision uses necessity credits plus a factual-correctness grader.
- The recall-only setup is described as producing less preferred, more incorrect responses.
From the original article
1 min read at huggingface.co
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective.
By Hyun Ryu, Jong-Heon Kim, Yiqing Xie · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.