TempQuant learns sampling settings that recover quantized model accuracy
Original title: TempQuant: Learning Adaptive Temperatures for Quantized Language Models
On 3-bit Qwen3-4B, a global setting raises MATH accuracy from 34% to 51% without labeled answers.
The paper finds that quantization warms next-token distributions and that sampling temperature matters more for quantized models than for full-precision ones. TempQuant learns temperature and top-p, globally or per token, to move a quantized model’s probabilities closer to its full-precision counterpart. Global settings match temperatures found by tuning on labeled answers, and per-token settings warm flipped rankings while cooling preserved ones. When quantization hurts accuracy substantially, the method recovers much of it.
- Quantization raises generation entropy on average.
- Model cards usually reuse full-precision sampling settings.
- Per-token control warms flipped rankings and cools preserved ones.
- Lighter degradation still moves behavior toward the full-precision model.
From the original article
1 min read at arxiv.orgLarge language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities.
By Mukund Athmaram · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.