Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
Algebraic Transformer layers raise small-model generation throughput 6.2-7.8% but lower downstream scores.
The paper replaces a Transformer's dense feed-forward matrix product with an associative-algebra product that has quadratic arithmetic when the physical block size is fixed, and is provably optimal for its bilinear rank by the Alder-Strassen bound. Two approximately 110M-parameter decoder-only language models were trained from the same recipe on a 12.3B-token budget, differing only in the feed-forward product. Across four prompt domains, the algebraic model increased end-to-end generation throughput by 6.2-7.8% but scored lower on all three reported downstream metrics. The authors frame this as a small-scale feasibility check.
- Associative-algebra layers replace dense matmul with a sparser fixed-block product.
- The construction is bilinear-rank optimal by the Alder-Strassen bound and supports causal masking.
- Two ~110M decoder-only models were trained on the same 12.3B-token budget.
- Throughput rose 6.2-7.8%, but all three downstream metrics were lower.
From the original article
1 min read at huggingface.co
Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether.
By Ilya Koziev, Ivan Oseledets · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.