Skip to content
NewA calmer, faster ZeroHour

TempQuant learns sampling settings that recover quantized model accuracy

Original title: TempQuant: Learning Adaptive Temperatures for Quantized Language Models

arXiv cs.AI / cs.LG / cs.CLby Mukund Athmaram ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On 3-bit Qwen3-4B, a global setting raises MATH accuracy from 34% to 51% without labeled answers.

The paper finds that quantization warms next-token distributions and that sampling temperature matters more for quantized models than for full-precision ones. TempQuant learns temperature and top-p, globally or per token, to move a quantized model’s probabilities closer to its full-precision counterpart. Global settings match temperatures found by tuning on labeled answers, and per-token settings warm flipped rankings while cooling preserved ones. When quantization hurts accuracy substantially, the method recovers much of it.

Key facts
  • Quantization raises generation entropy on average.
  • Model cards usually reuse full-precision sampling settings.
  • Per-token control warms flipped rankings and cools preserved ones.
  • Lighter degradation still moves behavior toward the full-precision model.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities.

arXiv cs.AI / cs.LG / cs.CL

By Mukund Athmaram · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.