On 10/20, DeepSeek announced the full open-sourcing of its internal inference framework DeepInfer, another act of openness following its earlier open-sourcing of a series of large models, the TileLang compilation tool, and the inference operator library. The company pointed out that while the capability boundary of large models is drawn by the models themselves, how to provide ultra-large-scale model services at a reasonable cost also depends on the inference framework as an intermediate layer—it coordinates and schedules massive computing clusters, and its efficiency directly determines the computing cost and power consumption of AI services.
DeepInfer integrates high-performance operators and a deeply optimized inference engine, supporting capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output, while being compatible with both NVIDIA and Huawei Ascend platforms. Based on this framework, DeepSeek has already implemented high-throughput, low-latency large-scale inference services for DeepSeek V4.1Flash, and the inference code for future new models as well as subsequent optimizations will continue to be updated here.
The companion deepseek-recipe, open-sourced at the same time, is a production-grade Token encoding/decoding component fully aligned with the processing logic of DeepSeek's official API, ensuring the community gets an API experience consistent with the official deployment. Both parts of the code have been released on GitHub, and DeepSeek has also published a complete technical report on Zhihu.
According to the company, the purposes of open-sourcing the inference framework are, first, to help developers and enterprises optimize throughput and latency of large-scale inference and efficiently build private and public cloud deployments, thereby reducing computing costs and energy consumption, letting intelligence reach a broader population at lower prices, and it will continue to support the community in large-scale deployment optimization. The layer-by-layer opening of models, operator libraries, and inference frameworks continues the industry path of domestic large models lowering inference barriers and promoting universal access to intelligence through an open-source ecosystem.
Open-source repositories
DeepInfer is available at: https://github.com/deepseek-ai/DeepInfer
deepseek-recipe: https://github.com/
deepseek-ai/deepseek-recipe