Near-zero embeddings stall gated model training
Original title: Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models
Untied gated checkpoints contain near-zero embeddings that amplify gradients 30+ orders of magnitude and trigger FP32 clipping, freezing rows in 20% of Telugu pretraining steps.
Among 291 surveyed open-weight repos, 52.6% of untied gated models have reachable embeddings below 10^{-6}. Near-zero tokens from rare or template inputs cause gradient amplification up to 35 orders of magnitude, leading to RMSNorm floor issues and batch gradient zeroing. Remedies are GroupClip for released checkpoints and GateFloor for pretraining.
- 52.6% of untied gated checkpoints affected
- Gradient amplification exceeds 30 orders of magnitude
- GateFloor reduces near-zero gradients by 28 orders
- Perplexity remains comparable to baseline
- GroupClip heals triggered rows in one step
From the original article
1 min read at huggingface.co
Among 291 surveyed open-weight language model repositories (280 distinct embedding tables), 52. 6% of untied gated and hybrid checkpoints (20/38) contain reachable input embeddings with norms below 10^{-6}, compared with 0% of weight-tied models. Such near-zero rows can arise during pretraining when unseen rare or template tokens receive zero input gradients while AdamW weight decay shrinks untied embeddings.
By Xin Liu, Boyuan Sun, Yuxin Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.