ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
ThinkPrior builds zero-rollout difficulty priors via an offline verifier-anchored pass, halving silent groups in RLVR and cutting wasted rollouts on Qwen2.5-Math-7B.
In GRPO-based RLVR, groups where all rollouts are correct or all are wrong yield zero advantages and consume about 39% of a run's rollouts under uniform sampling. ThinkPrior initializes a Beta posterior from an external anchor pass's verifier-scored pass rate, selecting prompts by expected learnability before any target-policy rollout, without changing the loss or optimizer. On Qwen2.5-Math-7B across sixteen seeds it more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, with no detected final-accuracy difference. The ThinkPrior+DAPO composition reduces generated rollouts by 10.6% at an equal 3,840-rollout update budget.
- Zero-advantage silent groups waste 39% of rollouts under uniform sampling
- Beta posterior initialized offline; no loss or optimizer change
- Early silent groups more than halved across sixteen seeds
- ThinkPrior+DAPO cuts generated rollouts 10.6% at equal update budget
- Products
- ThinkPrior
- AI models
- Qwen2.5-Math-7BDAPO
From the original article
1 min read at arxiv.orgIn reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation.
By Tommy Sha · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.