PACE quantizes linear-attention recurrent states to four bits post hoc
Original title: PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
Mixed four- and eight-bit PACE matches eight-bit DAMP on the three larger hybrid models at about two thirds of its memory.
PACE is a post-hoc four-bit quantizer for the fixed recurrent state of linear-attention language models. It rounds only every few generated tokens, buffers the tokens in between, and shares each quantization scale within a column of the gated-delta state. Heads that retain rounding error longest stay at eight bits. The authors say this is competitive with eight-bit DAMP at lower memory and beats Q-Mamba at equal memory.
Unconfirmed: Accuracy and memory figures are the authors' reported results, not independently verified.
- Existing four-bit quantizers lose accuracy unless width rises or models are fine-tuned.
- PACE rounds the recurrent state once every few generated tokens.
- It shares each quantization scale within one column of the gated-delta state.
- Mixed four- and eight-bit heads use about two thirds of eight-bit DAMP memory.
- Four-bit heads throughout score higher than Q-Mamba at equal memory.
From the original article
1 min read at huggingface.co
Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator.
By Patrick Haller, Filipe Laitenberger, Alan Akbik · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.