HarnessRL co-evolves open-ended RL rewards with the policy
Original title: HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
On Rubric-Mix-10k it beats the strongest baselines by 2.8 to 3.1 points and nearly halves steps to the best checkpoint.
HarnessRL is a training-time harness for rubric-based reinforcement learning that updates task sampling, rollout guidance, and rubric criteria as the policy changes. A memory-backed evolver attributes non-discriminative or spurious rewards to an interface, then a proposer and just-in-time critic accept updates only after testing them on the frozen policy. The authors also release Rubric-Mix-10k, covering medical, science, writing, role-play, and instruction-following tasks. Code and data are published in the Qwen-Character repository.
- Gains of 2.8 to 3.1 points are reported at both 4B and 9B scales.
- Training steps to the best checkpoint are nearly halved versus baselines.
- Rubric-Mix-10k spans five domains for training and evaluation.
- The critic rejects updates that do not help the current frozen policy.
From the original article
1 min read at huggingface.co
Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability.
By Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.