Paper describes numerical fixes for softmax and cross-entropy
Original title: A Fix for Softmax and Cross-Entropy
The authors say the fixes remove the Slingshot effect and stop student logits drifting from the teacher in distillation.
The paper says standard softmax, log-softmax, and cross-entropy implementations have two numerical bugs: Softmax Collapse, which zeros a dominant target logit gradient so gradients no longer sum to zero, and Softmax Cancellation, which yields nonzero gradients when student and teacher logits are equal or close. The authors present implementations they say fix both bugs. They report that the fix eliminates the Slingshot phenomenon, speeds grokking with weight decay, prevents accuracy collapse without weight decay, and stops student networks drifting from the teacher during distillation.
- Softmax Collapse is said to zero the gradient of a much larger target logit.
- Softmax Cancellation is said to produce nonzero gradients for equal student and teacher logits.
- The fix is reported to speed grokking when weight decay is used.
- After memorization, logit growth is tied to weight decay and Adam epsilon.
From the original article
1 min read at arxiv.orgStandard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero.
By Jonathan T. Barron · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.