WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
WhiteMatter mixes key-value sources across Transformer layers, matching deeper models with a smaller KV cache.
WhiteMatter lets every Transformer layer reuse past-token representations from any depth through a learned mixer that combines selected depths into shared KV-cache channels. With the same training tokens and a full cache, it matches a standard Transformer with 50% more layers. At half the KV cache it outperforms matched Transformers at scales up to 1.3B parameters. Cyclic iteration for cross-layer dependencies converges 12.5x faster than standard Jacobi iteration.
- A learned mixer selects depths and shares KV channels across layers.
- Full-cache WhiteMatter matches a Transformer with 50% more layers.
- Half-size KV cache still beats matched models up to 1.3B parameters.
- Cyclic iteration converges 12.5 times faster than Jacobi iteration.
From the original article
1 min read at huggingface.co
When generating text, a Transformer produces representations of past tokens at every layer, but each layer can normally use only representations from the same depth. This restriction prevents the model from fully reusing information it has already computed. We introduce WhiteMatter, which allows every layer to draw on past-token representations from any depth.
By Wenbo Zhang, Xiang Ren · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.