How Local Mixing Encodes Relative Position in Global NoPE Attention
Hybrid local-mixing layers give global NoPE attention a recency bias that encodes relative position.
1 stories, newest first · one card per story
Hybrid local-mixing layers give global NoPE attention a recency bias that encodes relative position.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views