PACE quantizes linear-attention recurrent states to four bits post hoc
Mixed four- and eight-bit PACE matches eight-bit DAMP on the three larger hybrid models at about two thirds of its memory.
Stories per day · 30 days
4 stories, newest first · one card per story
Stories tagged #recurrent-state
Prediction Is Not a Sufficient Memory State
Theory shows a fixed recurrent memory matching current predictions can still be insufficient for future learning.
Low-Discrepancy Dither for Quantized Recurrent State Caches
Golden-ratio Weyl dither keeps quantized Mamba recurrent states closer to full precision than stochastic rounding.
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
STEPQuant quantizes delta-rule recurrent states by error lifetime and location, matching FP32 accuracy at 6 bits.