IT Home On October 20, DeepSeek officially announced that, based on previously releasing a series of open-source large models, the TileLang compilation tool, and the inference operator library, it has decided to fully open-source its internal inference framework DeepInfer.
DeepSeek officially states that the capabilities of large models are defined by the models themselves, but how to provide ultra-large-scale model services at a reasonable cost depends on an intermediate layer that has not been publicly understood for a long time: reasoning frameworks.
Training enables the model to achieve results, while inference makes the model useful to people. In the face of today’s massive user calls, the inference framework is responsible for coordinating and managing large computing power clusters, and its efficiency directly determines the computing power cost and power consumption of AI services.
DeepSeek hopes to help developers and enterprise organizations optimize the throughput and latency of large-scale inference by sharing its internal deployment experiences, thereby efficiently building private and public cloud deployments, reducing computing costs, minimizing energy consumption, and enabling smart technologies to reach a wider audience at a lower price. Enabling true inclusive access to large models.
Based on DeepInfer, DeepSeek has officially launched a large-scale inference service with DeepSeek V4.1 Flash, featuring high throughput and low latency. In the future, the inference code for new DeepSeek models and subsequent optimizations will also be continuously updated here.
DeepInfer inference framework: Integrates high-performance operators and a deeply optimized inference engine, supporting distributed inference, speculation decoding, SWA Bounded Replay, structured output, etc., and is compatible with NVIDIA and Huawei Ascend platforms.
deepseek-recipe encoding-decoding component: A production-level Token encoding-decoding component that is open-source and synchronized with the official API processing logic of DeepSeek, ensuring the community gets an API experience consistent with official deployment.
The IT Home open-source address is as follows:
DeepInfer: https://github.com/deepseek-ai/DeepInfer
deepseek-recipe: https://github.com/deepseek-ai/deepseek-recipe
Ad notice: The text contains external link references (including but not limited to hyperlinks, QR codes, passwords, etc.) designed to provide more information, save selection time, and the results are for reference only. Articles on IT Home that include external links comply with this notice.