Continuum Attention
Continuum Attention beats softmax and focal attention by integrating a continuous memory curve.
Continuum Attention replaces discrete key-value lookup in a causal Transformer with integration over a continuous, piecewise-linear memory curve. Each query induces an exponential relevance density, and closed-form exponential moments produce exact normalized attention, including half-token boundary terms. Gradients come from moments of order at most two, so the operator stays GPU-parallel while keeping quadratic sequence complexity and a standard decoder. Versus softmax and focal attention, mean validation negative log likelihood drops by 0.0200 and 0.0480 nats per byte; a disjoint slice shows a 0.0457 nats-per-byte gain over the strongest selected softmax baseline.
- Continuum Attention integrates a continuous piecewise-linear memory curve.
- Closed-form exponential moments yield exact normalized attention.
- Validation NLL falls 0.0200 nats/byte versus softmax attention.
- A disjoint slice shows 0.0457 nats/byte over the best softmax baseline.
From the original article
1 min read at arxiv.orgWe introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve.
By Afshin Khadangi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.