RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
RollVerify verifies and repairs partial RL rollouts so language models keep on-policy accuracy while cutting training cost.
Reinforcement learning for language-model reasoning depends on long, uneven rollouts that leave GPUs idle in fully synchronized on-policy training. Partial rollouts improve throughput but introduce stale off-policy trajectories. RollVerify measures that deviation with an off-policy shift metric, then truncates invalid suffixes at sequence and token level before training. On mathematical and tool-assisted mathematical reasoning it matches on-policy accuracy while reducing cost; code-generation results are preliminary.
- Partial rollouts raise throughput but inject stale off-policy samples
- OPS metric guides sequence- and token-level suffix verification
- Accuracy matches fully on-policy training at lower cost
- Evaluated on math, tool-assisted math, and preliminary code generation
From the original article
1 min read at arxiv.orgReinforcement learning is crucial for improving large language models' reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability.
By Yongqiang Yao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.