Retrieval Capacity of Self-Attention Under Competition
Researchers measure how few high-attention tokens language models need to match full-context prediction loss.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #self-attention
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Researchers show repulsive self-attention in a minimal recurrent transformer produces chaos, condensation, and emergent locality.