Qwen3-0.6B mixes 4-bit decode with selective 8-bit steps
Original title: Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
An entropy gate at a 16% escalation rate scores 62.7% on 150 GSM8K problems, using 0.87 of all-high bytes.
The paper stores 8-bit weights for Qwen3-0.6B, decodes with the top 4 bits, and repeats a step at 8 bits when entropy or the top-two margin looks uncertain. Quantization is simulated, and cost is weight bytes read per token on GSM8K. Across 36,724 replayed steps, low-pass entropy ranks argmax flips with AUC 0.93; a 15% budget catches 78% of flips and the margin gate 94%. On 150 live problems, all-high scores 64.0% and all-low 55.3%, while a 16% entropy gate scores 62.7% without a statistically separated gap.
Unconfirmed: Live results do not statistically separate the entropy gate from all-high or all-low decoding.
- The evaluated model is Qwen3-0.6B and the benchmark is GSM8K.
- All-high scores 64.0% and all-low scores 55.3% on 150 problems.
- The entropy gate's paired gap versus all-high is -1.3 points, p=0.080.
- McNemar p-value for all-high versus all-low is 0.053.
- A live random control, second model, and live KL were not run.
From the original article
1 min read at arxiv.orgLocal decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.
By Mohammed Nayeem Khan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.