Infrathon tests language agents on long-horizon AI infrastructure tasks
Across 150 runs and over 10,000 GPU-hours, 135 still improved in the second half of a 24-hour budget.
Stories per day · 30 days
5 RSS
Relatedinference13compression5long-context5quantization5llm-inference4attention3fp83llm-agents3sglang3sparse-attention3agentic-ai2llama2llm-serving2on-policy-distillation2prefill2qwen2qwen32ruler2
84 stories, newest first · one card per story
Stories tagged #kv-cache
SemanticKV compresses KV cache for agentic video generation
Authors report a 3.35x KV-memory cut and 1.22x speedup on Causal Forcing at compression ratio 0.7.
Paper measures mismatch on-policy distillation can tolerate
Model-specific FP8 recipes kept quality and fit at least 1.93x as many KV-cache tokens as BF16.
Mirage bounds LLM KV-cache timing leaks without blocking sharing
On a 500-sample corpus and three models, token-emission delays pushed leakage near random guessing with lower TTFT than baselines.
BitDecoding-FP4 enables accurate FP4 KV cache decoding
Algorithm-system co-design achieves near-BF16 accuracy with 2.4-3x higher throughput for LLM serving.
RidgeFill speeds prefill with a training-free late-layer cache
A closed-form ridge map fills upper-layer cache; reported TTFT gains are 1.25–1.60x on tested Qwen setups.
NVIDIA claims codesign lifts agentic throughput per dollar up to 30x
From Blackwell Ultra to Vera Rubin, HBM per GPU rose about 3% while claimed agentic throughput per dollar rose up to 30x.
LowRAM compresses KV cache with low-rank keys and amortised values
Test-time value predictors plus cross-layer key compression give about 4× compression and up to 5× decode throughput.
KV-cache repair budget should focus on target chunk
Recomputing the target chunk against reused cache outperforms other repair strategies in most settings.
KVGuard detects KV cache safety bugs in LLM serving
KVGuard covers all 29 historical bugs from vLLM and SGLang plus 8 new ones with 3% overhead.
LoopVQ sub-1-bit KV cache quantization for looped transformers
LoopVQ jointly quantizes channels across multiple loop steps in looped transformers to reduce memory while preserving accuracy.
SIMPLE: a four-parameter law predicts LLM inference energy cost from architecture
Across 13 dense LLMs on H100 GPUs the law reaches R2 of 0.969 and 11.97% MAPE, and shows agentic requests cost 13x more energy than non-agentic ones.
DepthGraft shares KV hierarchically to regularize deep LLMs
On a 16B MoE model the authors report +7.0 points on MMLU-Pro while cutting KV cache by about one third.
ACCT transfers KV cache between LLMs for context reuse
ACCT reduces KV cache size by 75% while matching target model performance and speeding up prefill by 21x.
ZHA achieves subquadratic sparse attention for long-context LLMs
ZHA uses hierarchical tree indexers for subquadratic attention scaling beyond 1M tokens with maintained performance.
GAAC losslessly compresses BF16 KV caches about 1.41×
On NVIDIA DGX Spark, 64K-token decode attention sped up 1.19–1.27× and resident KV memory fell about 26%.
DCKV learns end-to-end compression of hybrid-model prompt KV caches
On Qwen3.5-2B, tokens use over 55 times fewer bytes than bf16 with at most 1.8 points lost on three long-context benchmarks.
Cloudflare GLM-5.3 leaked other customers' data through a KV cache bug
155 responses from August 27 to September 9 mixed in other users' data; Cloudflare found no deliberate exploitation.
1
Tsinghua open-sources AKV cache server for LLM agents on SGLang
The available page text is a cookie notice, so release details, benchmarks and license terms are not stated.
BeNice! trains looped models to share one KV-cache across recurrences
In simulated low-memory long-context inference, cache hits rose up to 31.4 percent and throughput up to 23.6 percent.
Paper proposes cache engineering so agents edit KV cache in place
A rolling-drop policy raised benchmark scores 1.7 points and cut cost 27.7% versus summarization.
Token-Energy Law ties LLM decode joules to active bytes and time
Held-out energy estimates on Hopper and Blackwell stayed within 3.73% of measured means.
AKV lets agents drop and reposition KV cache without re-prefill
A rolling-drop policy raises benchmark scores 1.7 points and cuts average cost 27.7% versus summarization.
Stratified Sparse Attention narrows the sparse-to-dense gap
On Ministral-3-8B at 50x sparsity, SSA lifts RULER-HARD from 63.8% to 69.3% at 128K.