DeepSeek announced officially on October 20 that, after gradually making a series of large models, the TileLang compilation tool, and the inference operator library open source, it has decided to take another step—making its internal inference framework DeepInfer fully open source.
The company has revealed a long-neglected intermediate layer: the upper limit of the capabilities of large models is determined by the models themselves, but how to provide ultra-large-scale model services at a reasonable cost truly depends on the inference framework. Training enables the models to “learn effectively,” while inference allows them to “be used by people.” In the face of today’s massive user calls, the inference framework is responsible for coordinating and managing large computing power clusters, and its efficiency directly determines the computing power costs and electricity consumption of AI services.
DeepSeek wants to share its internal deployment experiences with developers and enterprise organizations, helping them optimize the throughput and latency of large-scale inference, so they can build private or public cloud deployments more efficiently. This will reduce computing costs and energy consumption, enabling intelligent technologies to reach a wider audience at a lower price, thus realizing what it calls “large model accessibility for all”.
Based on DeepInfer, DeepSeek has successfully implemented the high-throughput, low-latency large-scale inference service of DeepSeek V4.1Flash; in the future, the inference code for new models and subsequent optimizations will also be continuously updated within this framework.
Specifically, the DeepInfer inference framework integrates high-performance operators and a deeply optimized inference engine. It supports distributed inference, speculation decoding, SWA Bounded Replay, and structured output capabilities, and is compatible with both NVIDIA and Huawei Ascend platforms. Alongside it, the deepseek-recipe encoding and decoding component—a production-grade Token encoding and decoding module—is also open-sourced. This module fully aligns with the processing logic of the DeepSeek official API, ensuring that the API experience for the community is consistent with official deployment. For those who want to build their own reasoning service, this means they don’t have to start from scratch; they can directly build on DeepSeek’s deployment experience.