Score Centering Stabilizes Off-policy Reinforcement Learning
Paper attributes RL instability under training-inference mismatch to accumulating drift and introduces score centering, an additive correction that stabilizes training.
The authors show that instability of LLM reinforcement learning under the training-inference mismatch (TIM) is primarily caused by drift, a persistent bias between training and inference engines that accumulates each step. They derive an additive score centering correction that cancels drift, and on models from 0.6B to 30B parameters it matches or outperforms importance-sampling methods under quantization as mismatch grows. Because it is additive, score centering composes with importance sampling, outperforming pure importance-sampling baselines in staleness experiments.
- Identifies drift as the main cause of RL instability under training-inference mismatch
- Score centering is an additive correction that cancels accumulated drift
- Validated on models from 0.6B to 30B parameters against importance-sampling baselines
- Composes with importance sampling, beating pure baselines in staleness experiments
Full article144 words · extracted from arxiv.org · click to collapse
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20807