The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Weight decay causally regulates massive-activation scale in transformers, with peak magnitude scaling near lambda to the -1/2.
Stories per day · 30 days
1 RSS
3 stories, newest first · one card per story
Stories tagged #massive-activations
Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
Visual activation spikes in vision-language models are brittle and can be created or removed with tiny image perturbations.
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Study shows attention sinks and massive activations stem from causal-mask self-concentration and value-non-mixing rather than RoPE, informing quantization work.