The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Weight decay causally regulates massive-activation scale in transformers, with peak magnitude scaling near lambda to the -1/2.
The paper traces how massive activations, residual-stream coordinates linked to attention sinks, emerge, grow, and consolidate during transformer training. Sink-carrying channels differ across random seeds but stabilize early, then concentrate onto a few redundant carriers as surrounding channels erode. Controlled continuations show that removing weight decay near the peak lets global activation scale keep rising, while retaining decay produces decline even at a constant learning rate. Peak timing shifts approximately log-linearly with the decay coefficient lambda, and peak magnitudes scale approximately as lambda to the power of -1/2.
- Sink-carrying channels vary by seed but stabilize early in each run.
- Removing weight decay near the peak lets activation scale keep rising.
- Peak magnitude scales approximately as lambda to the power of -1/2.
- AdamW preconditioning sustains large channels against weight decay.
From the original article
1 min read at huggingface.co
Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation.
By S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios Saravanos · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.