Score Centering Stabilizes Off-policy Reinforcement Learning
Paper attributes RL instability under training-inference mismatch to accumulating drift and introduces score centering, an additive correction that stabilizes training.
The authors show that instability of LLM reinforcement learning under the training-inference mismatch (TIM) is primarily caused by drift, a persistent bias between training and inference engines that accumulates each step. They derive an additive score centering correction that cancels drift, and on models from 0.6B to 30B parameters it matches or outperforms importance-sampling methods under quantization as mismatch grows. Because it is additive, score centering composes with importance sampling, outperforming pure importance-sampling baselines in staleness experiments.