Pretraining Transformers with Quantized Softmax in Attention
K-interval quantized softmax nearly matches full-softmax pretraining loss when calibration and the gradient surrogate are chosen carefully.
1 stories, newest first · one card per story
K-interval quantized softmax nearly matches full-softmax pretraining loss when calibration and the gradient surrogate are chosen carefully.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views