Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts
A fixed softmax-sigmoid attention mix barely changes GPT-2 in-distribution loss but shifts out-of-distribution perplexity sharply.
The paper proposes Mixture of Function Attention (MoFA), a parameter-free change that fixes a ratio of softmax and sigmoid heads in multi-head attention. Across five ratios, five seeds, and a 124M-parameter GPT-2, in-distribution differences are statistically negligible for moderate mixtures. Under zero-shot shift across 15 domains, perplexity gaps widen by more than an order of magnitude, and the best ratio tracks short informal versus long technical text, explaining 78.3% of domain-response variance. Sigmoid heads show a sharper drop in attention entropy as their share increases.
- MoFA fixes a ratio of softmax and sigmoid attention heads before training.
- On 124M GPT-2, in-distribution differences across five ratios and seeds are small.
- Zero-shot perplexity gaps widen by more than 10x on several of 15 domains.
- Domain structure explains 78.3% of which head ratio performs best.
Full article238 words · extracted from arxiv.org · click to collapse
Developmental psychology holds that certain priors are given to infants prior to experience rather than induced from data, and that the influence of such priors is suppressed under strong, well-constrained conditions but reasserts itself under weak ones. We ask whether an analogous principle holds for the Transformer: can the activation function given to attention heads serve as an intrinsic inductive bias? We propose Mixture of Function Attention (MoFA), a parameter-free modification to multi-head attention that fixes a ratio of softmax and sigmoid heads before training. Across five ratios, a 124M-parameter GPT-2 model, and five seeds, we find that this given ratio has little effect in-distribution -- differences between ratios are statistically negligible for moderate mixtures and remain small even at the extremes -- but its influence re-emerges sharply under zero-shot distribution shift across 15 out-of-distribution domains. Perplexity gaps between ratios widen by more than an order of magnitude on several domains, and the best-performing ratio tracks a single axis of domain structure, separating short, informal text (softmax-favoring) from technical, long-form text (sigmoid-favoring), that explains 78.3% of the variance in domain response. This reorganization is visible at the head level: sigmoid heads show an accelerating drop in attention entropy as their ratio increases, while softmax heads respond more modestly, yielding a consistent division of labor between the two head types. Our results suggest that activation choice functions as a given prior whose influence is masked in-distribution and re-emerges out-of-distribution.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.39188