DeepSeek开源DeepInfer高性能推理框架
deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models
DeepSeek官方开源的生产级推理引擎，包含大量独家优化细节（如DSpark、SWA重放），做部署的同学建议直接参考其配置参数与架构设计。
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
DeepInfer：面向 DeepSeek 模型的高性能大规模推理框架
GitHub repository created 2026-10-16 · 2316 stars in 4.1 days (about 568 a day) · Python
GitHub 仓库创建于 2026-10-16 · 4.1 天内获得 2316 颗星（日均约 568 颗）· Python
DeepInfer
DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.
DeepInfer 是支撑 DeepSeek 大规模生产服务的高性能推理框架。本次开源发布提供了用于规模化服务所需的核心功能的离线推理能力。
Supported models:
支持的模型：
- DeepSeek-V4.1-Flash
Supported features:
支持的功能：
- Optimized kernels, model and sampler runtime - Parallelism: Attention DP / balanced CP, MoE EP + EPLB - DSpark and confidence-scheduled verification - SWA bounded replay - Asynchronous scheduling (zero-overhead batch scheduler) - Batch-invariant inference - Deterministic sampling - Structured outputs
- 优化内核、模型与采样器运行时 - 并行策略：Attention DP / 平衡 CP，MoE EP + EPLB - DSpark 和基于置信度的调度验证 - SWA 有界重放 - 异步调度（零开销批调度器） - 批次不变推理 - 确定性采样 - 结构化输出
Feature details and configuration
功能详情与配置
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. - DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set `DEEPINFER_ENABLE_MTP_VARLEN=1`; `DEEPINFER_USE_NEXTN_DECODE` controls the maximum draft length (default: 5). - Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length. - Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default. - Batch-invariant inference: keep model results identical across different batch compositions. Enable with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only). - Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only). - Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams`; see examples.
- 优化内核：FlashMLA、DeepGEMM、DeepEP、TileKernels、DeepSelect 以及自定义 TileLang 内核。 - DSpark 和基于置信度的调度验证：通过草稿 token 加速解码，并利用置信度估计来调度验证。设置 `DEEPINFER_ENABLE_MTP_VARLEN=1`；`DEEPINFER_USE_NEXTN_DECODE` 控制最大草稿长度（默认值：5）。 - 解码器 SWA 有界重放：仅重放有界后缀以减少解码器预填充工作。对于默认的 V4.1 设置，请设置 `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` 和 `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140`；保留的后缀必须覆盖 SWA 窗口加上最大回溯长度。 - 异步调度：使 CPU 批调度与设备执行重叠。默认启用。 - 批次不变推理：确保在不同批次组成下模型结果保持一致。通过 `DEEPINFER_BATCH_INVARIANT=1` 启用（仅限 NVIDIA）。 - 确定性采样：为相同的输入 token 和采样参数生成相同的输出。设置 `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` 并向 `SamplingParams` 传递固定的 `seq_seed`（仅限 NVIDIA）。 - 结构化输出：将生成过程约束为正则表达式、JSON 模式或 EBNF 语法。向 `SamplingParams` 传递 `so_args={"schema_type": ..., "schema": ...}`；参见示例。
Supported hardware:
支持的硬件：
- NVIDIA SM100 GPUs - Huawei Ascend 950 NPUs
- NVIDIA SM100 GPU - 华为 Ascend 950 NPU
Required for a complete inference service (not included in this release):
完整的推理服务所需组件（未包含在此次发布中）：
- A KV cache system - VE/Prefill/Decode disaggregation (most required engine functionality is already implemented) - Load balancing across data-parallel replicas for both prefill and decode
- KV 缓存系统 - VE/预填充/解码分离（大部分所需的引擎功能已实现） - 预填充和解码在数据并行副本间的负载均衡
Getting started
快速开始
Prerequisites
前置条件
- Linux - GCC >= 13.0 - Python 3.12 - CUDA 13 (NVIDIA only) - CANN 9.2 (Ascend only)
- Linux - GCC >= 13.0 - Python 3.12 - CUDA 13（仅限 NVIDIA） - CANN 9.2（仅限 Ascend）
Build from source
从源代码构建
Before building DeepInfer, please create and activate an empty Python 3.12 environment.
在构建 DeepInfer 之前，请创建并激活一个空的 Python 3.12 环境。
Offline inference
离线推理
The default checkpoint, `deepseek-ai/DeepSeek-V4.1-Flash-1017`, is downloaded into the Hugging Face cache on the first run; set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead.
默认检查点 `deepseek-ai/DeepSeek-V4.1-Flash-1017` 将在首次运行时下载到 Hugging Face 缓存中；设置 `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` 以从本地副本运行。
See examples for details.
详见示例。
Performance benchmarks
性能基准测试
| Prefill | Decode | ||
|---|---|---|---|
Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.
在 4 块 SM100 GPU 上进行预填充测试，在 64 块 GPU 上进行解码测试——每组均在同一个 scale-up 域内。如需复现基准测试结果，请参阅 run_sweep.sh。
Evaluation benchmarks
评估基准测试
See eval/README.md.
参见 eval/README.md。
License
许可证
DeepInfer is released under the Apache License 2.0. See LICENSE for details.
DeepInfer 根据 Apache License 2.0 发布。详情请参阅 LICENSE。
Acknowledgments
致谢
DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.
DeepIner 最初主要基于 vLLM 构建，并受到 SGLang 和 TensorRT-LLM 的额外启发。我们感谢它们的开发者。
Citation
引用
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力