TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
TRIAGE stabilizes native NVFP4 RL on Qwen3 models, matching full precision with up to 2.3x rollout throughput.
TRIAGE addresses instability when reinforcement learning for large language models runs in native NVFP4, where learner-sampler mismatch can skew policy-gradient updates. The authors find an early imbalance favoring negative-advantage, negative-gap updates, with tail tokens concentrating in a few response segments before mismatch spreads. TRIAGE uses segment-level diagnosis to rebalance updates and applies bounded repair while keeping weight-and-activation 4-bit forward passes on both sampler and learner. On Qwen3-4B and Qwen3-30B-A3B it stays stable, matches full-precision results on five math reasoning benchmarks, and delivers up to 2.3x higher rollout throughput than BF16.
- Learner-sampler mismatch in native NVFP4 can bias the policy-gradient direction.
- Early imbalance favors negative-advantage, negative-gap updates before mismatch spreads.
- TRIAGE rebalances updates and repairs severe mismatch while keeping W4A4 forward passes.
- On Qwen3-4B and Qwen3-30B-A3B it matches BF16 with up to 2.3x rollout throughput.
Full article170 words · extracted from huggingface.co · click to collapse
Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3x higher rollout throughput than BF16.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.07043