KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
KV4-RL uses selective INT4 KV quantization to speed Qwen3 GRPO rollouts nearly 2x while keeping benchmark accuracy.
KV4-RL applies calibration-free INT4 KV-cache quantization during asynchronous RLVR rollouts, protecting attention sinks and recent context in INT8. Across 400 GRPO iterations on Qwen3-8B-Base, it reports 1.80x cumulative rollout speedup and 1.27x full-iteration speedup versus BF16, rising to 1.97x rollout speedup over the last 50 iterations. Under a shared BF16 evaluator, final mean accuracy on six benchmarks stays within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.
- INT4 KV cache speeds asynchronous RLVR rollouts more than weight quantization.
- Selective INT8 protection of sinks and recent tokens limits sampler mismatch.
- Qwen3-8B-Base reaches 1.80x rollout and 1.27x full-iteration speedup.
- Six-benchmark accuracy stays within 0.5 points of BF16 training.
From the original article
1 min read at huggingface.co
Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step.
By Sihwan Park, Sung-Yub Kim, Doohyuk Jang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.