Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Researchers propose DATPO, a difficulty-adaptive tree-structured RLVR training method that expands reasoning coverage (pass@k) and improves test-time scaling on math benchmarks.
The paper identifies three rollout design principles for RLVR: difficulty-adaptive rollout expands pass@k, tree-based rollout beats parallel sampling, and sentence-entropy-guided forking overcomes token-level branching localization. DATPO combines difficulty-adaptive tree search with a sibling-diversity advantage term to promote semantic diversity during training. On mathematical reasoning benchmarks, DATPO outperforms baselines in pass@k, directly translating to superior test-time scaling performance.
- Three rollout principles: difficulty-adaptive, tree-structured, sentence-entropy-guided forking.
- Sibling-diversity advantage term promotes semantic diversity during RLVR training.
- Targets pass@k reasoning coverage rather than single-sample accuracy.
- Improves test-time scaling on mathematical reasoning benchmarks.
Full article157 words · extracted from huggingface.co · click to collapse
Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.08650