It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Study shows attention sinks and massive activations stem from causal-mask self-concentration and value-non-mixing rather than RoPE, informing quantization work.
The paper analyzes why attention sinks and massive activations emerge at initial sequence positions regardless of which token occupies them. Experiments attribute both phenomena to self-concentration of attention induced by the causal mask and the subsequent value-non-mixing in attention outputs. The findings provide empirical evidence on LLM internal dynamics and may inform low-bit quantization strategies, which massive activations currently complicate.
- Attention sinks and massive activations co-occur at the initial sequence position
- Massive activations pose challenges for low-bit quantization
- Causal-mask self-concentration and value-non-mixing are identified as the drivers
- Results clarify attention-layer mechanics relevant to future quantization
From the original article
1 min read at arxiv.orgLarge Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it.
By Raito Kiya · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.