Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Researchers propose DATPO, a difficulty-adaptive tree-structured RLVR training method that expands reasoning coverage (pass@k) and improves test-time scaling on math benchmarks.
The paper identifies three rollout design principles for RLVR: difficulty-adaptive rollout expands pass@k, tree-based rollout beats parallel sampling, and sentence-entropy-guided forking overcomes token-level branching localization. DATPO combines difficulty-adaptive tree search with a sibling-diversity advantage term to promote semantic diversity during training. On mathematical reasoning benchmarks, DATPO outperforms baselines in pass@k, directly translating to superior test-time scaling performance.
30