First Passage Attention selectively erases intermediate context
Original title: First Passage Attention
A 110M-parameter model matched FoX perplexity and perfectly retrieved passkeys at eight times training length with a compressed cache.
First Passage Attention penalizes a key by the deepest dip of a learned scalar field between that key and the query, rather than summing monotonic decay. The authors show it includes the Forgetting Transformer as a special case and that a constant-dimension head can realize any subset of keys left alive by later dips. In synthetic tests, one-layer FPA heads reached the target sets while softmax and FoX baselines failed on every seed. Scaled to a 110M-parameter language model, FPA matched FoX perplexity and achieved perfect passkey retrieval at 8 times the training length even with the key-value cache compressed to about 25%.
Unconfirmed: Perplexity parity and perfect long-context retrieval are results reported by the authors.
- FPA uses path extrema instead of summed decay along the key-query path.
- Softmax or token forget gates need head dimension linear in the key count.
- One-layer FPA converged on every synthetic seed; baselines did not.
- Passkey retrieval stayed perfect with the cache at about 25%.
From the original article
1 min read at huggingface.co
Augmenting softmax attention with forget gates typically relies on summing decay contributions along the key--query path. Because such accumulation is monotonic, these mechanisms cannot selectively erase intermediate context while preserving older, structurally relevant tokens.
By Yuxuan Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.