How Local Mixing Encodes Relative Position in Global NoPE Attention
Hybrid local-mixing layers give global NoPE attention a recency bias that encodes relative position.
The paper explains how hybrid transformers that mix sliding-window attention or gated linear attention with global no-position-encoding layers can still represent relative position. Theory and experiments argue that local mixing induces a recency bias in the residual stream that global attention logits then select. Unlike position information arising only from the causal mask in pure global NoPE models, this bias can be maintained over long sequences and may aid length extrapolation.
- Sliding-window and gated linear attention create residual recency bias.
- Global NoPE attention logits select that bias as position.
- The bias can persist across long sequences.
From the original article
1 min read at arxiv.orgThe attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE).
By Cutter Dawes · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.