Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It
Calibrated importance sampling corrects token-probability mismatch between LLM training and inference engines.
The paper studies training-inference mismatch in reinforcement learning with verifiable rewards, where rollout sampling and gradient computation assign different probabilities to the same tokens. Calibrated importance sampling models the gap as an additive log-odds displacement and truncates large positive displacements with a confidence-aware cap. Theory replaces the unbounded second moment of exact importance sampling with a constant-bounded term, introducing bias only from the truncated excess. Across three mixture-of-experts models and five math reasoning benchmarks, CIS posts the highest five-benchmark average on every model among the baselines.
- CIS treats engine mismatch as additive log-odds displacement
- Confidence-aware truncation tightens the importance-ratio cap
- Replaces an unbounded second moment with a constant bound
- Highest five-benchmark average on all three MoE models
- Upward clipping of small importance weights hurts held-out accuracy
From the original article
1 min read at huggingface.co
We study training-inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gradients are computed by a training engine, and the two engines assign different probabilities to the same tokens. To account for this discrepancy in policy updates, we introduce calibrated importance sampling (CIS).
By Tianrun Yu, Kaixiang Zhao, Shangzhe Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.