Hybrid wave-attention beats tiny transformers on perplexity, not memory
Original title: Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
A 10.94M hybrid scores 5.95 perplexity, but its peak VRAM matches the pure transformer.
The paper defines a Dynamic Wave recurrent layer using input-dependent exponential damping and cosine phase modulation, then compares Pure Transformer, Pure Wave, and Hybrid Wave-Attention models of 10.94 million parameters on TinyStories up to 4096 tokens. The hybrid records the best held-out perplexity, 5.95 versus 6.93 and 7.26. Memory gains for Pure Wave disappear against a fused scaled-dot-product attention kernel, and in training Pure Wave is 8.3% more expensive than the transformer. The hybrid's peak VRAM matches the pure transformer within 0.1%.
Unconfirmed: Efficiency comparisons are the authors' measurements on TinyStories only.
- Against unfused attention, Pure Wave uses 78.7% less peak VRAM at 4096 tokens.
- Fused attention uses 1403.6 MB versus 1404.3 MB for Pure Wave.
- Hoisting elementwise terms speeds the wave layer 2.53-fold without a custom kernel.
- Authors say efficiency claims must name the attention backend and implementation.
From the original article
1 min read at arxiv.orgLarge Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives. This paper formalizes a Dynamic Wave recurrent layer, based on input-dependent exponential damping and cosine phase modulation, and evaluates three 10.
By Dmytro Babarytskyi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.