Kalman Delta Networks: Uncertainty-aware Associative Memory
Kalman Delta Networks add uncertainty tracking to linear-attention associative memory, improving perplexity and downstream accuracy at 750M and 1.3B scales.
Kalman Delta Networks reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model, allowing the Kalman gain to weight each residual write by accumulated evidence and observation reliability; Delta-rule updates emerge as a special case lacking covariance tracking. Two scan-compatible approximations, Diagonal KDN (online mean-field variational inference) and Isotropic KDN (one uncertainty scalar per head), produce Mobius-map uncertainty recurrences enabling associative scans with logarithmic parallel depth. Controlled pretraining at 750M and 1.3B parameters consistently improves perplexity and mean downstream accuracy over state-of-the-art linear-attention models.
- Frames delta-rule associative memory as a linear-Gaussian state-space with Kalman-gain-weighted writes
- Diagonal and Isotropic KDN approximations are GPU-scan-compatible via Mobius map recurrences
- Consistent perplexity and downstream accuracy gains at 750M and 1.3B parameters
- Targets fixed-size recurrent memory write decisions for long-context linear attention
- Products
- Kalman Delta Networks
From the original article
1 min read at arxiv.orgLinear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require.
By Ngoc Bui · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.