DLC correction stabilizes LLM RL training-inference mismatch
Original title: On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Discrepancy-lag correction centers numerical errors and clips lag ratios to prevent feedback loops in on-policy LLM reinforcement learning.
Training-inference mismatch arises from engine numerics and policy lag in LLM RL. The discrepancy-lag correction splits errors and resynchronizes the sampler when effective sample size drops below half. It maintains KL divergence within twice that of independent on-policy training on Qwen3 models.
- Numerical discrepancy biases updates by a fixed amount
- Policy lag creates a feedback loop closed by importance weighting
- DLC prevents collapses that per-token corrections leave
From the original article
1 min read at huggingface.co
On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the training-inference mismatch between the two makes the training data off-policy.
By Yu Li, Yufa Zhou, Tian Lan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.