Multi-layer Indexer optimizes DeepSeek sparse attention
Multi-layer Indexer speeds DeepSeek-V3.2 decode up to 14.5% without retraining.
Stories per day · 30 days
1 RSS
Relatedkv-cache3inference2attention-efficiency1context-transfer1cross-llm1hierarchical-index1hunyuanvideo1latent-space1llm-optimization1long-context1ministral1model-efficiency1multimodal1sequence-compression1subquadratic1test-time-regression1ultra-long-context1video-diffusion1
22 stories, newest first · one card per story
Stories tagged #sparse-attention
LSA maintains accuracy at ultra-long context via latent sparse attention
LSA clusters keys in learned latent space for subquadratic attention, speeding up layers by 22.7x at 10M tokens while matching full-attention accuracy.
ACCT transfers KV cache between LLMs for context reuse
ACCT reduces KV cache size by 75% while matching target model performance and speeding up prefill by 21x.
ZHA achieves subquadratic sparse attention for long-context LLMs
ZHA uses hierarchical tree indexers for subquadratic attention scaling beyond 1M tokens with maintained performance.
TTR-Compressor uses test-time regression for sequence compression
On visual token merging it matches an MLP merger while using 55.6% fewer visual tokens.
ALLOT allocates block-sparse attention for video diffusion transformers
Training-free ALLOT gains 11.8 dB PSNR on HunyuanVideo with XAttention at equal or shorter runtime.
Stratified Sparse Attention narrows the sparse-to-dense gap
On Ministral-3-8B at 50x sparsity, SSA lifts RULER-HARD from 63.8% to 69.3% at 128K.
Joint Subspace Sparse Attention for Efficient Long-Context Decoding
JSSA sparse attention stays near full-KV accuracy and raises 128K decoding throughput up to 6.43×.
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
FreeSolo adds linear-time sparse attention that extrapolates context length without long-context fine-tuning.
MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
Training-free MC-Sparse speeds diffusion-transformer denoising up to 2.32× with negligible quality loss.
MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse narrows the dense-sparse attention gap, speeding diffusion denoising up to 2.32× with little quality loss.
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Prism uses dynamic sparse attention to train joint 2K video-audio models 2.5× faster than full attention.
SparseEngine: Sparse-First Inference Engine
SparseEngine, a sparse-first LLM inference engine, reports over 10x throughput with KV eviction versus existing serving stacks.
The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends
A survey frames LLM attention as contextual memory across 14 model lineages and 59 releases.
NaiveAI/Naive-N0.5-Flash — new model trending #30 on Hugging Face
NaiveAI released Naive-N0.5-Flash, a 309B MoE open-weight model with 15.5B active parameters and a native 1M-token context.
4
CoWindow Attention: Full Causal Coverage Is a Collective Property
CoWindow Attention covers causal history collectively, cutting 128K training latency up to 8.6x versus FullAttn.
DeepSeek-v4.1 Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1 Flash is a 552B-parameter multimodal MoE model with 1M-token context achieving 4x KV cache compression for long-horizon agent workloads.
11
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom introduces per-query bit-depth scanning of offloaded 4-bit KV caches, making long-context decoding 1.67x faster on Qwen3-8B at one million tokens.
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS trains a gated sparse attention selector end-to-end with the language modeling loss, beating distillation-based sparsifiers on reasoning, long-context, and agentic tasks.
DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
DeepSeek released open-weight V4.1-Flash, a 552B MoE model with 1M context and FP4 KV cache, beating Opus-5 and GPT-5.6 Sol on agent benchmarks.
10
deepseek-ai/DeepSeek-V4.1-Flash — new model trending #28 on Hugging Face
DeepSeek releases DeepSeek-V4.1-Flash, a 552B-parameter multimodal MoE model with 1M-token context and KV cache cut to 890 bytes per token.
10
unsloth/Qwen3.8-Flash-Next-GGUF — new model trending #21 on Hugging Face
Qwen released Qwen3.8-Flash-Next, an experimental 125B-parameter open-weight MoE previewing the Qwen4 architecture, with Unsloth shipping optimized GGUF quants.
6