FP4 pretraining needs more exposures to learn rare facts
NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall while loss differed by 0.5% and 2%.
Stories per day · 30 days
1 RSS
Relatedbenchmark1fp161fp41gpu1llm-training1numerical-stability1nvfp41nvidia1pretraining1quantization1qwen1reinforcement-learning1
6 stories, newest first · one card per story
Stories tagged #bf16
GPU-unlock measures FP/BF16 rates on Nvidia 3000 cards
The item is only a Hacker News link with 44 points and 25 comments, with no technical write-up.
Aligning BF16 rounding locations stabilizes LLM reinforcement learning
Keeping RoPE, SwiGLU, and residual RMSNorm intermediates in FP32 prevented collapse on four Qwen-family models.
From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
A study of Qwen3-1.7B shows multi-teacher on-policy distillation is shaped by averaging, Adam, and BF16 precision.
From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
A distillation study shows averaging, Adam, and BF16 reshape how multi-teacher signals update student models.
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Reproduction study finds Orthrus speculative-decoding trajectories match the reference model in only ~45% of cases under BF16, but 100% under FP32.