CodeQuant2 enhances low-precision quantization for large reasoning models
Original title: CodeQuant2: Enhanced Outlier Mitigation and Adaptive Clustering for Low-Precision LRMs
CodeQuant2 adapts codebook sizes and improves calibration for reasoning-time activation statistics.
Large reasoning models generate longer traces increasing inference costs. CodeQuant2 mitigates activation outliers with rotation and scaling, uses clustering for non-uniform weights and adapts codebook sizes per layer. Customized LUT CUDA kernel enables substantial accuracy improvements while achieving up to 2.85× speedup over BF16.
- Improves calibration for early decoding steps
- Adapts codebook sizes to quantization difficulty
- Customized LUT CUDA kernel
- Delivers accuracy improvements on reasoning benchmarks
From the original article
1 min read at arxiv.orgLarge reasoning models (LRMs) improve task performance by generating longer intermediate reasoning traces, but this substantially increases inference costs and makes low-precision quantization increasingly important. However, aggressively quantized models often perform poorly on reasoning tasks. We observe larger activation fluctuations during the first decoding steps of reasoning.
By Xingyu Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.