Sable proves bit-exact quantized LLM inference on one GPU
Qwen2.5-32B takes 92.5 seconds for 4,096 tokens; proofs are not zero-knowledge.
Stories per day · 30 days
2 RSS
Relatedkv-cache4attention3speculative-decoding3quantization2adversarial-prompts1agentic-systems1benchmarking1bit-flip1cpu-inference1decision-models1delta-net1denial-of-service1diffusion-drafting1distillation1energy-efficiency1fhe1flashmap1flow-maps1
26 stories, newest first · one card per story
Stories tagged #llm-inference
LLM Inference notes repo on GitHub
GitHub repo shares detailed notes on LLM inference from 6 hours of YouTube and 85 pages of handtyped content.
Chatter Attack drives LLMs to responses up to 31.5 times longer
Prompts target short doubt-inducing phrases, and Bayesian Prompt Optimization extends the attack to black-box models.
Mirage bounds LLM KV-cache timing leaks without blocking sharing
On a 500-sample corpus and three models, token-emission delays pushed leakage near random guessing with lower TTFT than baselines.
CPU-ARC caches LLM reasoning traces for cheaper CPU reuse
The paper reports on-par results with frontier models on hard math and science tasks at 16–291x lower cost.
KV-cache repair budget should focus on target chunk
Recomputing the target chunk against reused cache outperforms other repair strategies in most settings.
D3D enables asynchronous speculative decoding
Standalone diffusion drafter removes sequential dependency for 5x LLM inference speedup.
LoopVQ enables sub 1-bit KV cache quantization for looped transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Perseus enables fast homomorphic inference for LLMs
Perseus is 3.96x faster than other FHE implementations for Llama-3-8B
SpecOPD branch-parallel distillation accelerates speculative decoding
SpecOPD parallelizes on-policy distillation across branches for up to 41.45% higher mean acceptance length and 3.56times generation speedup.
SIMPLE: a four-parameter law predicts LLM inference energy cost from architecture
Across 13 dense LLMs on H100 GPUs the law reaches R2 of 0.969 and 11.97% MAPE, and shows agentic requests cost 13x more energy than non-agentic ones.
Flide accelerates decision inference with exact state forking
Flide decouples logical tree structure from execution in a flattened token stream using ForkAttention, achieving up to 9.5x speedup over vLLM while preserving exact probabilities.
Polymathic AI and Princeton release FlashMap for faster LLM decoding
The headline describes FlashMap as a flow-map drafter claimed to speed decoding by 6.14 times.
1
GAAC losslessly compresses BF16 KV caches about 1.41×
On NVIDIA DGX Spark, 64K-token decode attention sped up 1.19–1.27× and resident KV memory fell about 26%.
Stanford and UC Berkeley introduce TokenGate for faster long-context prefill
The title claims token-gated attention speeds long-context prefill by up to 11.3 times; the article body was not available.
1
Rowhammer bit-flip in Triton slows or corrupts LLM inference
LLVMhammer raised single-GPU latency 4-16% on four models and could cut BERTScore to 25%.
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding prunes LLMs using decoding activations, speeding generation up to 1.48x.
DScout speeds diffusion-drafted speculative decoding on Qwen3
Mean speedups versus autoregressive decoding are 3.84 times on Qwen3-4B and 4.14 times on Qwen3-8B.
Learning the Cost of Reliable Inference
A reverse-auction procurement platform learns LLM quality and price, finding 10–71% margins across Llama and Qwen tasks.
Disaggregated Quantization: Specializing LLM Prefill and Decode
Disaggregated quantization tailors LLM prefill and decode formats, raising accuracy and speeding time-to-first-token.
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
SwitchSD uses internal LLM signals to adaptively switch between neural drafting and context-based copying in speculative decoding, boosting throughput up to 15%.
SpliTEE: Improving LLM Inference on Trusted Hardware with Differentially Private GPU Outsourcing
SpliTEE splits LLM inference between Intel TDX trusted execution and untrusted GPUs, using differential privacy instead of encryption to protect intermediate representations.
VU#369611: ExLlamaV3 contains Denial of Service vulnerability via insufficient bounds checking on kernel dispatch index
CERT/CC details CVE-2026-84286, an out-of-bounds memory access in ExLlamaV3's CUDA extension causing denial-of-service crashes; fixed via merged pull request.
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Uno pairs autoregressive LLMs with lightweight diffusion weights to draw multiple tokens in parallel, delivering up to 3x lossless speedup without a draft model.