DeepSeek open-sources DeepInfer inference engine with production-scale throughput and cost claims

DeepSeek open-sourced DeepInfer, the inference engine behind its DeepSeek-V4.1-Flash model, positioning it as a production-grade framework designed for high throughput and low latency. The company described DeepInfer as a customization of vLLM with additional contributions from SGLang and TRT-LLM, while noting that certain components—including its ultra-optimized KV cache and load-balancing—remain closed.

The release includes production-serving figures claiming a cluster throughput of 155 trillion tokens per day, 96% input-cache hit rate, 400 ms average time to first token, and generation rates up to 300 tokens per second for coding workloads. DeepSeek also reported $38 revenue per accelerator-hour at off-peak prices and an inference margin exceeding 80%, with a projected 81-day payback period on inference hardware.

DeepInfer’s architecture separates vision encoding, prefill, and decode into distinct stages, allowing each to scale independently against its primary bottlenecks. Deployment guidance recommends eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode, with wide expert parallelism used in the decode stage to distribute MoE experts while maintaining large effective batch sizes.

The scheduling approach emphasizes compute-aware allocation over raw token counts, using estimated compute cost to schedule attention layers and varying chunk sizes to fit time budgets rather than enforcing fixed chunk sizes across heterogeneous requests. Prefill routing leverages shared storage for cached prefixes, enabling requests to be routed to available workers regardless of prior cache affinity, which the engineering analysis highlights as a production advantage.

Decode routing balances both running-request count and accumulated KV-cache length to capture both scheduling pressure and memory/state pressure. Expert-parallel load balancing replicates popular experts periodically to reduce inter-device waiting caused by routing skew, trading additional HBM consumption for higher throughput with diminishing returns as replication increases.

The vLLM team published comparative results showing decode throughput within 6% of DeepInfer’s published figures on GB200 under a matched configuration, alongside reproduction recipes and AgentX results, which partially substantiates DeepSeek’s efficiency claims against existing open-source engines.

DeepSeek stated that future model inference implementations and optimizations will continue to be added to the DeepInfer repository, framing the release as an ongoing contribution to the open-source ecosystem rather than a static artifact.

Independent analysis noted that DeepInfer is not a complete reproduction of DeepSeek’s production service; while the engine is open source, components such as KV-cache management and certain orchestration layers remain coupled to internal infrastructure and were not released.