Kalman Delta Networks: Uncertainty-aware Associative Memory
Researchers propose Kalman Delta Networks, adding Kalman-filter uncertainty tracking to delta-rule linear attention, improving perplexity and accuracy at 750M and 1.3B parameters.
The paper introduces Kalman Delta Networks (KDNs), which reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model where the Kalman gain weights each write by accumulated evidence and observation reliability. Two scan-compatible approximations, Diagonal KDN via online mean-field variational inference and Isotropic KDN with a single uncertainty scalar per head, enable associative scans with logarithmic parallel depth. Delta-rule updates are shown to be a special case of this formulation. KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention baselines in controlled pretraining at 750M and 1.3B parameters.
- Reformulates recurrent associative memory as a linear-Gaussian state-space model with optimal Kalman-filter updates
- Delta-rule updates emerge as a special case using isotropic surrogate covariance without tracking
- Diagonal and Isotropic KDN variants support parallel associative scans via Mobius-map uncertainty recurrences
- Improves perplexity and downstream accuracy at 750M and 1.3B parameters over linear-attention baselines
- Tags
- associative-memorykalman-filterlanguage-modelslinear-attentionstate-space-modelstraining-methodsuncertainty
- Products
- Kalman Delta Networks
From the original article
1 min read at huggingface.co
Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require.
By Ngoc Bui, Tinglin Huang, Rex Ying · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.