BitDecoding-FP4 enables accurate FP4 KV cache decoding
Algorithm-system co-design achieves near-BF16 accuracy with 2.4-3x higher throughput for LLM serving.
Stories per day · 30 days
2 RSS
Relatedinference2kv-cache2agent-gateway1agentic-systems1benchmark1deception1fp41glm-5.31gpu-scheduling1hierarchical-workloads1honeypot1inference-api1lmcache1maservebench1mla1multi-agent1prefill1quantization1
11 stories, newest first · one card per story
Stories tagged #llm-serving
InferD honeypot answers AI-inference probes without executing models
Static response pools cover transport, protocol, vulnerability-surface and agentic-session deception planes.
MAServeBench Benchmarks Multi-Agent LLM Serving
MAServeBench captures hierarchical collaboration workloads for multi-agent LLM systems, revealing concurrency and memory bottlenecks in serving hardware.
TAO jointly schedules LLM agent sessions and tool returns
Across 12 reported settings, TAO cuts latency by up to 26.8% and mean TTFT by up to 36.3% versus the best baseline for each metric.
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Remote KV-cache hits slowed GLM-5.3 prefill until sharding and async loads cut p99 from 83s to 0.67s.
TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter raises token-level LLM routing decode throughput by 2.01–64.15× over existing serving systems.
Characterizing High Bandwidth Flash for LLM Serving
High-bandwidth flash cuts LLM serving completion time by 36-87% when scheduling preserves flash write lifetime.
SparseEngine: Sparse-First Inference Engine
SparseEngine, a sparse-first LLM inference engine, reports over 10x throughput with KV eviction versus existing serving stacks.
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
JustFit MLX runtime serves 200K-token contexts for Qwen3.8-27B on a 24 GiB MacBook via just-in-time state management.
Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
SQD disaggregates LLM inference by quadratic versus subquadratic attention layers, improving energy efficiency up to 56% on heterogeneous systems versus GPU-only baselines.
HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving
HoneyRoute detects malicious LLM serving requests and diverts them to a honeypot model, reaching F1 0.911 with 38 ms median added latency.