Open guide maps platform engineering skills to inference serving
Figures are dated October 2026 and cover vLLM, SGLang, KV cache, and a four-layer serving stack.
Stories per day · 30 days
3 RSS
Relatedinference7sglang4deepseek3open-source3qwen2channel-selection1coding-agents1constrained-decoding1decision-model1decision-models1deepinfer1forkattention1fp41gemma1gpu1grpo1huggingface1kimi1
46 stories, newest first · one card per story
Stories tagged #vllm
MACaQ quantizes weights and activations to MXFP4 layer by layer
On Qwen3-32B in vLLM it adds 2.3% overhead, and a 122B model was quantized in 64 minutes using 32.5GB GPU RAM.
Buoyant releases Kinference open-source inference control plane for Kubernetes and vLLM
Buoyant has released Kinference, an open source inference control plane for Kubernetes and vLLM.
1
KVGuard detects KV cache safety bugs in LLM serving
KVGuard covers all 29 historical bugs from vLLM and SGLang plus 8 new ones with 3% overhead.
Flide answers decision queries up to 9.5 times faster than vLLM
ForkAttention keeps each query branch identical to an isolated forward pass while reusing GPU-resident graphs and checkpoints.
vLLM Semantic Router releases d3-ultra, a 27B multimodal model
The release is described as an open multimodal decision model; the article body was not available.
1
Unofficial Qwen3.8-Flash-Next E384 FP4 model trends on Hugging Face
The NAS build keeps 384 of 512 experts, cuts weights about 16%, and was merged after GRPO-style LoRA.
MoE channel selection works without calibration data, paper finds
A weight-only reader keeps 98.1% of full accuracy at 17.8% of routed-expert reads and speeds MoE blocks versus vLLM.
vLLM constrained decoding lifts Kimi tool-call schema accuracy to 100%
Serving Kimi K2.6 on vLLM 0.28 matched the expected schema only 92% of the time, the authors report.
2
Small open-weight coding workers trail direct coding on checks
In one author's studies, a Qwen3-Coder worker passed 0/45 tasks while direct coding passed 45/45 on the same suite.
DeepSeek open-sources DeepInfer, its production inference engine
DeepSeek claims 155 trillion tokens a day and 96% cache hits; KV cache and load balancing are not fully released.
DeepSeek open-sources DeepInfer inference engine under Apache 2.0
DeepInfer serves V4.1-Flash on top of vLLM, and deepseek-recipe mirrors official API token processing for self-hosting.
What is Decision 3.0? vLLM Semantic Router’s New Open Decision Models
vLLM Semantic Router released Decision 3.0, five Apache-2.0 multimodal models from 0.8B to 27B.
DuoDiscover searches optimizations with cheap proxy evaluators
Authors say it doubles expert speedup on SWE-fficiency and reports 1.92 times vLLM and 1.38 times SQLite.
autotrust/JEV-35B-NVFP4 — new model trending #21 on Hugging Face
AutoTrust released JEV-35B-NVFP4, a 4-bit quantization of its 35B decision model that cuts GPU memory by 65%.
vllm-sr/d3 — new model trending #24 on Hugging Face
vLLM Semantic Router released d3, a 27B multimodal model that returns calibrated decisions instead of generated text.
14
Metasploit Wrap Up: A Collection of What Can Only Be Called Eclectic Modules
Rapid7's Metasploit wrap-up adds 12 modules, including unauthenticated RCE for N-central, Langflow, LiteLLM, and dizqueTV.
12
nvidia/Qwen3.8-Flash-Next-FP4-Local — new model trending #30 on Hugging Face
NVIDIA published an FP4-quantized Qwen3.8-Flash-Next checkpoint for local Blackwell inference, occupying about 96 GiB.
Critical LMCache RCE Vulnerability Remains Unpatched, Public PoC Exploit Available
Unpatched LMCache flaw CVE-2026-105192 allows unauthenticated remote code execution, and JFrog published a PoC.
9
Unpatched Critical LMCache Flaw Lets Unauthenticated Attackers Run Code Remotely
Unpatched LMCache flaw CVE-2026-105192 lets unauthenticated attackers run code via pickle over ZeroMQ.
Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute
Researchers store encrypted KV cache on NVMe so Gemma 4 recalls facts across 50 million tokens without recompute.
autotrust/GEV-26B-Decide-NVFP4 — new model trending #30 on Hugging Face
autotrust published GEV-26B-Decide-NVFP4, an NVFP4 expert-quantized 26B model that fits in about 17 GB.
Prime Intellect Launches Prime Inference: Serverless and Reserved Serving for Frontier Open Models
Prime Intellect launched Prime Inference, serverless and reserved serving for open models such as GLM-5.3.
7
Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
Galahad caches byte-exact KV state so repeated LLM document reading becomes a one-time cost.