Score Centering Stabilizes Off-policy Reinforcement Learning
Paper attributes RL instability under training-inference mismatch to accumulating drift and introduces score centering, an additive correction that stabilizes training.
The authors show that instability of LLM reinforcement learning under the training-inference mismatch (TIM) is primarily caused by drift, a persistent bias between training and inference engines that accumulates each step. They derive an additive score centering correction that cancels drift, and on models from 0.6B to 30B parameters it matches or outperforms importance-sampling methods under quantization as mismatch grows. Because it is additive, score centering composes with importance sampling, outperforming pure importance-sampling baselines in staleness experiments.
- Identifies drift as the main cause of RL instability under training-inference mismatch
- Score centering is an additive correction that cancels accumulated drift
- Validated on models from 0.6B to 30B parameters against importance-sampling baselines
- Composes with importance sampling, beating pure baselines in staleness experiments
From the original article
1 min read at arxiv.orgReinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency.
By Martin Marek · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.