Paper describes numerical fixes for softmax and cross-entropy
The authors say the fixes remove the Slingshot effect and stop student logits drifting from the teacher in distillation.
Stories per day · 30 days
Relatedattention1backward-compatibility1continuum-attention1cross-entropy1distillation1gradient-descent1grokking1hopfield-networks1language-modeling1multiple-instance-learning1numerical-stability1phase-transitions1representation-learning1retrieval1transformers1
9 stories, newest first · one card per story
Stories tagged #softmax
Paper proposes simplex features for backward-compatible retrieval
The method needs no extra compatibility loss, learned map, or architecture change, including when classes are added.
Paper maps phase transitions in max-pooled multiple instance learning
A minimal model shows freezing and retrieval transitions as softmax sharpness rises, and simulations match the predicted flow.
Continuum Attention
Continuum Attention beats softmax and focal attention by integrating a continuous memory curve.
Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersion
Corrected two-level softmax samplers remove cluster-size and dispersion bias with little overhead.
Pretraining Transformers with Quantized Softmax in Attention
K-interval quantized softmax nearly matches full-softmax pretraining loss when calibration and the gradient surrogate are chosen carefully.
Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration
Approximating softmax in ten frozen LLMs yields a FlashAttention-4 kernel up to 25.8% faster on NVIDIA B200.
Softmax Reparameterization for Output-Head Quantization
A post-training softmax reparameterization cuts output-head quantization error, dropping Phi-4-mini AW-MSE KL from 0.936 to 0.256.
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating softmax attention supplies abstention and noise filtering, with filtering gains growing from 10M to 350M models.