DeepSeek open-sources DeepInfer, its production inference engine
Original title: [AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
DeepSeek claims 155 trillion tokens a day and 96% cache hits; KV cache and load balancing are not fully released.
DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, with production serving details. It supports distributed inference, speculative decoding, and structured output on NVIDIA and Huawei Ascend, and separates vision encoding, prefill, and decode. DeepSeek says KV-cache management and load balancing remain tied to internal infrastructure. vLLM reported decode throughput within 6% of DeepSeek’s published GB200 results.
Unconfirmed: Token volume, cache-hit rate, revenue, and margin figures are first-party or calculated claims, not independently audited.
- Reported figures include 400 ms average time to first token and 230 tokens per second.
- Coding generation is said to reach 300 tokens per second.
- DeepSeek cites $38 revenue per accelerator-hour at off-peak prices.
- vLLM says DeepInfer was built primarily on vLLM under Apache 2.0.
- Recommended shapes are eight-accelerator prefill nodes and 64–128-accelerator decode supernodes.
- Organizations
- DeepSeek
- AI models
- V4.1 Flash
From the original article
12 min read at latent.space
Proudly brought to you by SerpAPI: Today’s release covers a LOT of ground, but you can scrape all this into markdown for your agents ! 50% token cost of JSON. We last covered the Inference Inflection in April , and historically most of the advances have been closed source decacorns from Fireworks to Baseten to Modal .
latent.space
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.