Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
SQD disaggregates LLM inference by quadratic versus subquadratic attention layers, improving energy efficiency up to 56% on heterogeneous systems versus GPU-only baselines.
SQD (SubQuadratic Disaggregation) splits decode not by operator type but by quadratic versus subquadratic attention, matching their distinct arithmetic intensity and memory footprints. For sparse attention LLMs it separates top-k selection (requiring full KV indexing) from top-k attention plus FFN; for linear and sliding-window models it separates dense attention layers from subquadratic layers plus FFN. On an adjusted 8xB200 heterogeneous proxy, tokens-per-joule improves 53% on GLM 5.2, 31% on Nemotron 3 Ultra, and 56% on Gemma 4 31B. A Rubin plus LPX analytical model shows 1.2x-1.5x tighter achievable latencies and up to 3.6x higher throughput versus attention-FFN disaggregation.
- Disaggregates decode by attention type rather than operator type
- 53-56% tokens-per-joule gains on GLM 5.2 and Gemma 4 31B proxies
- Up to 3.6x throughput over attention-FFN disaggregation in Rubin+LPX model
- Applies to sparse, linear, and sliding-window attention variants
From the original article
1 min read at arxiv.orgFrontier language models are more aggressively using subquadratic attention to reduce the memory footprint and compute requirements during inference while still delivering frontier accuracy.
By Arya Tschand · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.