BeNice! trains looped models to share one KV-cache across recurrences
Original title: BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
In simulated low-memory long-context inference, cache hits rose up to 31.4 percent and throughput up to 23.6 percent.
Looped models reuse the same layers, but each recurrence normally builds its own KV-cache, so memory grows with depth. The paper studies when the final recurrence's cache can be shared earlier without quality loss. Stochastic depth during pretraining is followed by BeNice!, which broadcasts the final KV-cache into the first recurrence. The authors say the result matches standard finetuning's unshared validation perplexity and composes with other KV compression and low-precision caches.
- BeNice! is a post-training method after stochastic-depth pretraining.
- Shared validation perplexity matches standard finetuning without KV sharing.
- It composes with multi-latent attention, Kimi delta attention, and fp8, mxfp8, and nvfp4 caches.
- p90 time-to-first-token fell by up to 83.6 percent in the simulated setting.
From the original article
1 min read at arxiv.orgLooped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.
By Hayden Prairie · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.