PACE quantizes recurrent language-model states to four bits
Original title: PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
On hybrid models from 4B to 35B parameters, mixed four- and eight-bit heads approach eight-bit DAMP at about two-thirds of its memory.
The paper introduces PACE, a post-hoc four-bit quantizer for the fixed recurrent state of linear-attention language models. It rounds the state only once every few generated tokens, keeps intervening tokens in a small buffer, shares each scale within a column, and leaves heads that retain rounding error longest at eight bits. The authors report that mixed four- and eight-bit heads are competitive with eight-bit DAMP on the three larger of four hybrid models at about two-thirds of its memory, and that all-four-bit heads score higher than Q-Mamba at equal memory.
Unconfirmed: Accuracy and memory comparisons are the authors’ reported results, with no independent replication stated.
- Existing quantizers that round after every token are said to lose accuracy at four bits without wider bits or fine-tuning.
- PACE shares a quantization scale within one column of the gated-delta state.
- Rounding every few tokens is reported to preserve the decoding speedup of the token buffer.
- Evaluated models range from 4B to 35B parameters.
From the original article
1 min read at arxiv.orgLinear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator.
By Patrick Haller · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.