Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating softmax attention supplies abstention and noise filtering, with filtering gains growing from 10M to 350M models.
The paper argues that value-pathway gates improve language-model pretraining because softmax attention lacks abstention and noise filtering. In matched models from 10M to 350M parameters, abstention is a learned per-head sink logit and filtering is a gate on each value. Abstention accounts for nearly all of the gating gain at 10M, while filtering accounts for most of it at 350M, and combining both is best at every scale. Controlled interference tests show the gate removes injected noise, though each gate form has a blind spot, and the added parameters remain compatible with the key-value cache.
- A learned per-head sink logit lets an attention head abstain instead of summing weights to one.
- A value gate filters interference from superposed features in the residual stream.
- Abstention explains nearly all gating gain at 10M; filtering dominates at 350M.
- Models with both primitives are best at every tested scale, with negligible extra parameters.
- Each studied gate form has a characteristic blind spot under controlled interference.
From the original article
1 min read at arxiv.orgGating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one.
By Richard Zhe Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.