Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating softmax attention supplies abstention and noise filtering, with filtering gains growing from 10M to 350M models.
The paper argues that value-pathway gates improve language-model pretraining because softmax attention lacks abstention and noise filtering. In matched models from 10M to 350M parameters, abstention is a learned per-head sink logit and filtering is a gate on each value. Abstention accounts for nearly all of the gating gain at 10M, while filtering accounts for most of it at 350M, and combining both is best at every scale. Controlled interference tests show the gate removes injected noise, though each gate form has a blind spot, and the added parameters remain compatible with the key-value cache.
- A learned per-head sink logit lets an attention head abstain instead of summing weights to one.
- A value gate filters interference from superposed features in the residual stream.
- Abstention explains nearly all gating gain at 10M; filtering dominates at 350M.
- Models with both primitives are best at every tested scale, with negligible extra parameters.
- Each studied gate form has a characteristic blind spot under controlled interference.
Full article234 words · extracted from arxiv.org · click to collapse
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one. The second is noise filtering, which allows the value pathway of an attention head to suppress interference from superposed features in the residual stream. In our experiments in matched models from 10M to 350M parameters, we supply abstention through a learned per-head sink logit in the softmax and noise filtering through a gate on each value. We report three empirical findings. First, the benefit of abstention, measured as the reduction in validation loss relative to a matched baseline, declines as models grow, whereas the benefit of noise filtering increases with scale. In particular, abstention accounts for nearly all of the gain from gating at 10M and filtering for most of it at 350M. Second, the best model at every scale is the one with both primitives built in. Third, injecting controlled interference into the values a head reads confirms that the gate removes such interference, and reveals that each of the two gate forms we study has a characteristic blind spot. Supplying both primitives adds negligible parameters and remains compatible with the key-value cache.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22005