Paper escalates Qwen3-0.6B from 4-bit to 8-bit on uncertain steps
Original title: Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
An entropy gate at a 16% escalation rate scores 62.7% on GSM8K versus 64.0% for always using 8-bit weights.
The paper stores simulated 8-bit weights for Qwen3-0.6B, decodes with the top 4 bits, and redoes a step at 8 bits when entropy or the top-two margin looks uncertain. On 36,724 replayed steps, low-pass entropy ranks argmax flips with AUC 0.93; at a 15% budget it catches 78% of flips and the margin gate 94%. In 150 live GSM8K problems, all-high scores 64.0% and all-low 55.3%. An entropy gate with a 16% escalation rate scores 62.7% while reading 0.87 of the all-high weight bytes, and the live gaps are not significant.
Unconfirmed: Live results do not separate the entropy gate from all-high or all-low decoding.
- Cost is weight bytes read per token; quantization is simulated.
- The entropy gate’s paired gap versus all-high is -1.3 points (p=0.080).
- Versus all-low the gap is +7.3 points, with a confidence interval from 0.0 to +14.7.
- A live random control, a second model, and live KL were not run.
From the original article
1 min read at huggingface.coLocal decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.