British AI chip startup Fractile raises a $720 million Series C
The UK government puts the valuation above £5 billion, and the CEO says first chips and systems ship next year.
Stories per day · 30 days
12 RSS
Relatedkv-cache13gpu10speculative-decoding10nvidia9open-source7vllm7deepseek6quantization6qwen6agentic-ai5long-context5diffusion-language-models4sglang4tpu4ai-chips3attention3deepinfer3diffusion-models3
252 stories, newest first · one card per story
Stories tagged #inference
Fireworks AI discloses new breach after cloud provider hack
On October 23 attackers reached development machines and exposed billing data, metadata, and incomplete customer model weights.
1
OpenAI deploys Control Room to halt research model jobs
Inference engines must recheck permission, and output stops within 35 seconds of an intervention decision.
SGLang RCE exploited at NEAR AI; user and agent data taken
About 100 Private Chat accounts and 18 agents were exposed; two Cloud API keys were revoked and no CVE was named.
Paper argues mechanistic interpretability needs a logic of inference
The authors propose five principles as a recursive procedure for turning local findings into testable mechanistic claims.
C2-Routing cuts VLM visual tokens by routing crop or compress
With a 1,024-token cap already applied, the method reports 36.53% fewer FLOPs and a 0.77% mean relative score drop.
Qwen3-0.6B mixes 4-bit decode with selective 8-bit steps
An entropy gate at a 16% escalation rate scores 62.7% on 150 GSM8K problems, using 0.87 of all-high bytes.
Open guide maps platform engineering skills to inference serving
Figures are dated October 2026 and cover vLLM, SGLang, KV cache, and a four-layer serving stack.
NVIDIA donates OpenShell to CNCF and adds Kubernetes AI tools
A Crusoe test of Snapshot cut GLM-5.3 readiness from about eight minutes to 31 seconds.
Google unveils TPU 8tx and TPU 8i Max for AI workloads
TPU 8tx, designed with AMD, targets confidential training and inference; TPU 8i Max doubles memory in one package.
Second-order screen predicts quantization mismatch on five language models
On unseen Mistral-7B-v0.3, all six predictions fell in band and vLLM throughput was predicted within 0.4%.
NVIDIA Dynamo Recipes Bring Korea’s Open Models to Production
NVIDIA releases Dynamo-optimized recipes for four Korean open models delivering up to 2x performance on multi-GPU setups.
Buoyant releases Kinference open-source inference control plane for Kubernetes and vLLM
Buoyant has released Kinference, an open source inference control plane for Kubernetes and vLLM.
1
RidgeFill speeds prefill with a training-free late-layer cache
A closed-form ridge map fills upper-layer cache; reported TTFT gains are 1.25–1.60x on tested Qwen setups.
NVIDIA claims codesign lifts agentic throughput per dollar up to 30x
From Blackwell Ultra to Vera Rubin, HBM per GPU rose about 3% while claimed agentic throughput per dollar rose up to 30x.
Qualcomm launches Cloud AI 100 Ultra AI accelerator card
Qualcomm Cloud AI 100 Ultra offers 150W TDP, 128GB DRAM and 548GB/s bandwidth for cloud and on-premises AI inference.
Open model inference growth tracked via startup token volumes
Interconnects research finds open model inference is growing exponentially but remains small compared to closed models.
1
LowRAM compresses KV cache with low-rank keys and amortised values
Test-time value predictors plus cross-layer key compression give about 4× compression and up to 5× decode throughput.
NVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%
NVIDIA demonstrates three methods to accelerate Mixture-of-Experts inference, increasing Ling-3.0-flash throughput by 42.5% on DGX Spark.
Hacker News users discuss hardware for running local LLMs
Community shares Mac Studios, RTX 3090 systems, and GPU clusters for models like DeepSeek-V4-Flash
Hush-Up attack halts distributed LLM inference
A single malicious worker forces premature LLM termination with small output perturbations that evade local activation-based verification.
Flide answers decision queries up to 9.5 times faster than vLLM
ForkAttention keeps each query branch identical to an isolated forward pass while reusing GPU-resident graphs and checkpoints.
AB-Attention approximates dense video attention at high sparsity
On Wan2.1 and HunyuanVideo the authors report skipping 89–92% of attention with up to 2.23× speedup.
Metacognitive controller trims diffusion language-agent inference cost
Across five benchmarks, the plug-in cuts inference FLOPs 20.6% on average versus the strongest efficient baseline while keeping competitive accuracy.