Paper derives attention architectures from geometric knowledge spaces
Original title: Geometric knowledge spaces, Attention pairing and Architectures
The framework recovers standard, causal, linear, and tensor attention plus recurrent networks, then proposes new geometric spaces.
An arXiv paper proposes a geometric way to derive attention mechanisms and explore possible transformer architectures. It formalizes knowledge worlds and attention pairings, then shows how that framework leads to standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. It also introduces several new geometric spaces that the authors say may inspire novel neural network architectures.
- The paper starts from knowledge worlds and attention pairings.
- It connects the framework to recurrent neural networks as well as transformers.
- No implementation results or benchmarks are reported in the abstract.
From the original article
1 min read at arxiv.orgIn this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks.
By Shang Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.