DeepSeek开源DeepInfer推理框架与工具链

Full text of DeepSeek's announcement in Chinese, translated (I'll omit fangirl noises):

DeepSeek Open-Sources Its Inference Framework: Toward a Future of Truly Universal Access

The capability boundaries of large models are defined by the models themselves, but serving ultra-large-scale models at a reasonable cost also depends on a middle layer that has long gone unnoticed by the public: the inference framework.

Training is how a model learns; inference is how it becomes useful to people. Faced with today's enormous volume of user requests, the inference framework orchestrates and schedules vast compute clusters, and its efficiency directly determines the compute cost and power consumption of AI services.

DeepSeek has long been committed to pushing inference performance to its limits. Building on our earlier open-sourcing of a series of large models, the TileLang compiler toolchain, and our inference operator library, we have decided to go further and fully open-source our internal inference framework, DeepInfer.

"Do good together with the many; share skill so that the many may be skilled." By sharing our internal deployment experience, we hope to help developers and enterprises optimize throughput and latency for large-scale inference, so they can efficiently build private and public cloud deployments, lower compute costs, and reduce energy consumption, bringing intelligence to a broader population at a lower price and making large models truly accessible to all. We will also continue to support the community in optimizing large-scale deployments, and we welcome you to join us in open source.

Using DeepInfer, we have achieved high-throughput, low-latency large-scale inference serving for DeepSeek V4.1 Flash. Going forward, inference code and subsequent optimizations for new DeepSeek models will continue to be released here.

DeepInfer inference framework: An inference engine that integrates high-performance operators with deep optimizations. It supports distributed inference, speculative decoding, SWA Bounded Replay, structured output, and more, and is compatible with both NVIDIA and Huawei Ascend platforms. deepseek-recipe encoding/decoding component: A production-grade token encoding/decoding component, open-sourced alongside DeepInfer. It is fully aligned with the processing logic of DeepSeek's official API, so the community gets an API experience consistent with our official deployment.

Open-source repositories

Copy the link into your browser or tap "Read the original" to view DeepSeek's technical report on Zhihu: https://zhuanlan.zhihu.com/p/2095560295930508953

Looking Ahead

With fully open-sourced models, operator libraries, and inference frameworks, we look forward to a near future in which the reach and quality of AI services worldwide expand and improve as never before, becoming fairer, more transparent, and more trustworthy. We hope this will accelerate technological progress across society and cut energy waste, letting limited resources flow more efficiently between infrastructure and end users, and delivering more powerful intelligence at a cost everyone can afford, along with the goodwill and boundless possibilities that come with it.

DeepSeek will work hand in hand with all of you, striving tirelessly to make this vision a reality.