Search: “Kalman Delta Networks”

2 stories

Researchers propose Kalman Delta Networks, adding Kalman-filter uncertainty tracking to delta-rule linear attention, improving perplexity and accuracy at 750M and 1.3B parameters.

The paper introduces Kalman Delta Networks (KDNs), which reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model where the Kalman gain weights each write by accumulated evidence and observation reliability. Two scan-compatible approximations, Diagonal KDN via online mean-field variational inference and Isotropic KDN with a single uncertainty scalar per head, enable associative scans with logarithmic parallel depth. Delta-rule updates are shown to be a special case of this formulation. KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention baselines in controlled pretraining at 750M and 1.3B parameters.

Hugging Face daily papersSep 6, 20:00 UTC · 9d agoAI research

Kalman Delta Networks: Uncertainty-aware Associative Memory

Kalman Delta Networks add uncertainty tracking to linear-attention associative memory, improving perplexity and downstream accuracy at 750M and 1.3B scales.

Kalman Delta Networks reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model, allowing the Kalman gain to weight each residual write by accumulated evidence and observation reliability; Delta-rule updates emerge as a special case lacking covariance tracking. Two scan-compatible approximations, Diagonal KDN (online mean-field variational inference) and Isotropic KDN (one uncertainty scalar per head), produce Mobius-map uncertainty recurrences enabling associative scans with logarithmic parallel depth. Controlled pretraining at 750M and 1.3B parameters consistently improves perplexity and mean downstream accuracy over state-of-the-art linear-attention models.

arXiv cs.AI / cs.LG / cs.CLSep 7, 17:53 UTC · 8d agoAI research1