Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
Algebraic feed-forward layers lift small Transformer throughput 6–8% but cut downstream scores.
The paper replaces dense matrix multiplication in Transformer feed-forward layers with a sparser associative-algebra product that has quadratic arithmetic at a fixed block size. The construction meets the Alder-Strassen bilinear-rank bound and supports causal masking and KV-cached decoding. Two approximately 110M-parameter decoder-only models, trained on the same 12.3B-token budget, show a 6.2–7.8% generation throughput gain for the algebraic variant but lower scores on all three reported downstream metrics.
- Associative-algebra product replaces dense matmul in feed-forward layers.
- Construction is optimal for bilinear rank by the Alder-Strassen bound.
- Two ~110M models were trained on the same 12.3B-token budget.
- Throughput rose 6.2–7.8% while all three downstream scores fell.
From the original article
1 min read at arxiv.orgFast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether.
By Ilya Koziev · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.