FP4 pretraining needs more exposures to learn rare facts
Original title: What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall while loss differed by 0.5% and 2%.
The paper trains transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, alone and mixed into web text. NVFP4 required 1.32× and 1.52× as many exposures as BF16 to match recall, while validation loss differed by only 0.5% and 2%. With both models evaluated using BF16 multiplications, the ratios were 1.26× and 1.30×, and about 1.2× at matched loss under the primary readout. Noise analysis attributes much of the cost to updates rather than Adam's second moment, and the authors recommend measuring rare-fact acquisition alongside loss.
Unconfirmed: Results are from the authors' emulated 4-bit training on synthetic biographies, not production pretraining runs.
- Training used synthetic biographies with controlled exposure counts.
- At matched loss, the exposure cost was about 1.2× under the primary readout.
- Raising the learning rate did not recover the NVFP4 exposure cost.
- BF16 gradient statistics ranked format costs with Spearman 0.96.
From the original article
1 min read at arxiv.orgLow-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text.
By Chang Lu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.