Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Researchers show repulsive self-attention in a minimal recurrent transformer produces chaos, condensation, and emergent locality.
Researchers analyze a minimal recurrent transformer with normalized tokens, identity query and key maps, and a negative value map. In two dimensions, a regular-polygon fixed point loses stability through a flip bifurcation into period-two motion, chaos, and cluster-exchange states. Attention condensation appears when softmax sharpness scales as N squared, while high-dimensional simulations show a transition at beta of order one, including diffuse, consensus-flip, chaotic routing, and fragmented cluster-flip phases.
- A negative value map pushes tokens away from similarity-selected fields.
- The polygon fixed point loses stability through a flip bifurcation as feedback grows.
- Attention stays diffuse at fixed sharpness but condenses when beta scales as N squared.
- High-dimensional simulations show condensation near beta of order one.
From the original article
1 min read at arxiv.orgWe study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network.
By Qucheng Gao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.