Decoupling Exploration from Optimization in RLVR
Exploration-Distillation separates novelty search from RLVR optimization and beats DAPO on math benchmarks.
The paper introduces Exploration-Distillation (ExpDis) for reinforcement learning with verifiable rewards. Explorer policies are trained with a novelty bonus, their trajectories are filtered for correctness and quality, and the results are distilled into a separate student trained without that bonus. Repeating exploration and optimization lets exploration scale without degrading the student. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget and shows improved pass@k scaling.
- ExpDis separates novelty search from student optimization.
- Explorer trajectories are filtered for correctness, then distilled.
- ExpDis beats DAPO at the same wall-clock budget.
- Pass@k scaling improved across seven math benchmarks.
From the original article
1 min read at arxiv.orgModern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data.
By Saif Punjwani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.