Kalman Delta Networks: Uncertainty-aware Associative Memory

Kalman Delta Networks (KDNs) add Kalman-filter uncertainty tracking to delta-rule linear attention by reformulating recurrent associative memory as a linear-Gaussian state-space model; Diagonal and Isotropic KDN variants consistently improve perplexity and…

Two listings of the same paper — Hugging Face daily papers (2026-09-06T20:00:00Z) and arXiv cs.AI/cs.LG/cs.CL (2026-09-07T17:53:29Z) — agree on all technical claims and describe Kalman Delta Networks, which reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model so that the Kalman gain weights each residual write by accumulated evidence and observation reliability. Delta-rule updates emerge as a special case of this formulation that lacks covariance tracking. Two scan-compatible approximations are proposed: Diagonal KDN, via online mean-field variational inference, and Isotropic KDN, with a single uncertainty scalar per head. Both produce Mobius-map uncertainty recurrences that enable associative scans with logarithmic parallel depth (described as GPU-scan-compatible in the arXiv listing). In controlled pretraining at 750M and 1.3B parameters, KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention baselines. The arXiv listing additionally frames the work as targeting fixed-size recurrent memory write decisions for long-context linear attention.

• Kalman Delta Networks reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model, with the Kalman gain weighting each residual write by accumulated evidence and observation reliability.
• Delta-rule updates emerge as a special case of this formulation without covariance tracking; one listing describes this special case as using an isotropic surrogate covariance without tracking.
• Two scan-compatible approximations are proposed: Diagonal KDN via online mean-field variational inference and Isotropic KDN with a single uncertainty scalar per head.
• Both variants produce Mobius-map uncertainty recurrences enabling associative scans with logarithmic parallel depth; the arXiv listing describes them as GPU-scan-compatible.
• In controlled pretraining at 750M and 1.3B parameters, KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention baselines.
• Per the arXiv listing (2026-09-07), the method targets fixed-size recurrent memory write decisions for long-context linear attention.
• Both sources (Hugging Face daily papers, 2026-09-06; arXiv cs.AI/cs.LG/cs.CL, 2026-09-07) are consistent; the only substantive addition in the arXiv listing is the long-context write-decision framing.