It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Study shows attention sinks and massive activations stem from causal-mask self-concentration and value-non-mixing rather than RoPE, informing quantization work.
The paper analyzes why attention sinks and massive activations emerge at initial sequence positions regardless of which token occupies them. Experiments attribute both phenomena to self-concentration of attention induced by the causal mask and the subsequent value-non-mixing in attention outputs. The findings provide empirical evidence on LLM internal dynamics and may inform low-bit quantization strategies, which massive activations currently complicate.
- Attention sinks and massive activations co-occur at the initial sequence position
- Massive activations pose challenges for low-bit quantization
- Causal-mask self-concentration and value-non-mixing are identified as the drivers
- Results clarify attention-layer mechanics relevant to future quantization
Full article112 words · extracted from arxiv.org · click to collapse
Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.09085