Aligning BF16 rounding locations stabilizes LLM reinforcement learning
Keeping RoPE, SwiGLU, and residual RMSNorm intermediates in FP32 prevented collapse on four Qwen-family models.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #numerical-stability
Paper describes numerical fixes for softmax and cross-entropy
The authors say the fixes remove the Slingshot effect and stop student logits drifting from the teacher in distillation.