Low-Discrepancy Dither for Quantized Recurrent State Caches
Golden-ratio Weyl dither keeps quantized Mamba recurrent states closer to full precision than stochastic rounding.
Mamba-style and hybrid language models compress history into a fixed recurrent state rewritten at every token, so low-precision rounding errors can accumulate over long generations. The authors find a deterministic golden-ratio Weyl dither, which needs no random numbers, keeps quantized models closer to full precision than stochastic rounding across pure and hybrid models, storage formats, and long decoding horizons. Round-to-nearest discards small updates, so it can look best in short evaluations yet falls behind as generation length grows. A discrepancy analysis explains the ordering and notes implementation pitfalls that remove the benefit.
- Low-precision recurrent states feed every rounding error into the next token update.
- Deterministic golden-ratio Weyl dither beats stochastic rounding at no extra cost.
- Round-to-nearest can win short evals but its error grows over long generations.
- Results hold across pure and hybrid models, formats, and decoding horizons.
From the original article
1 min read at arxiv.orgMamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over long generations. Production systems round the state stochastically; we ask which rounding rule such caches should use.
By Snigdha Chandan Khilar · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.