Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Researchers propose DATPO, a difficulty-adaptive tree-structured RLVR training method that expands reasoning coverage (pass@k) and improves test-time scaling on math benchmarks.
The paper identifies three rollout design principles for RLVR: difficulty-adaptive rollout expands pass@k, tree-based rollout beats parallel sampling, and sentence-entropy-guided forking overcomes token-level branching localization. DATPO combines difficulty-adaptive tree search with a sibling-diversity advantage term to promote semantic diversity during training. On mathematical reasoning benchmarks, DATPO outperforms baselines in pass@k, directly translating to superior test-time scaling performance.
- Three rollout principles: difficulty-adaptive, tree-structured, sentence-entropy-guided forking.
- Sibling-diversity advantage term promotes semantic diversity during RLVR training.
- Targets pass@k reasoning coverage rather than single-sample accuracy.
- Improves test-time scaling on mathematical reasoning benchmarks.
From the original article
1 min read at huggingface.co
Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k.
By Youngjun Yu, Sanghwan Jang, Hwanjo Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.