LLM推理六大并行策略详解与选型指南
6 parallelism strategies for LLM inference in production:
工程落地必读,清晰拆解了LLM推理并行的底层逻辑与选型权衡,直接指导集群部署调优。
6 parallelism strategies for LLM inference in production:
LLM 生产环境推理的 6 种并行策略:
(popular LLM interview question)
(热门 LLM 面试题)
Running a model across four GPUs does not merge their memory into one shared pool.
在四块 GPU 上运行模型并不会将它们的内存合并为一个共享池。
Each GPU has its own memory, so data stored on another GPU must cross the interconnect before a layer can use it.
每块 GPU 都有独立的内存,因此存储在另一块 GPU 上的数据必须通过互联通道传输后,某一层才能使用它。
This is the central trade-off behind parallelism. It removes local memory or compute limits but adds communication overhead.
这是并行背后的核心权衡。它消除了本地内存或计算限制,但增加了通信开销。
The visual below explains the six common strategies for parallelizing LLM inference:
下图解释了 LLM 推理并行的六种常见策略:
- Data parallelism runs complete model replicas on separate GPU groups. It increases throughput, but cannot make an oversized model fit.
- Tensor parallelism splits matrix operations inside each transformer layer. It reduces weight memory per GPU, but the GPUs must combine partial results throughout the layer stack.
- Pipeline parallelism assigns layer groups to different GPUs. Activations move between stages, while insufficient micro-batching leaves some stages idle.
- Context parallelism splits sequence positions in two ways. Prefill context parallelism divides a long input across GPUs to target lower TTFT. Decode context parallelism divides the stored KV history to increase cache capacity, but every decoder layer must merge partial attention results.
- Expert parallelism distributes MoE experts across GPUs. Token routing and uneven expert load add network cost and idle time.
- Hybrid parallelism combines multiple strategies in one deployment. For example, tensor parallelism can operate within a node, pipeline parallelism across nodes, and data parallelism across model replicas. This matches each communication pattern to the available links, but adds more coordination paths.
- 数据并行在独立的 GPU 组上运行完整的模型副本。它提高了吞吐量,但无法让超出容量的模型适配。
- 张量并行拆分每个 Transformer 层内的矩阵运算。它降低了每块 GPU 上的权重内存占用,但 GPU 必须在整个层堆栈中合并部分结果。
- 流水线并行将层组分配给不同的 GPU。激活值在各阶段间移动,而微批次不足会导致某些阶段空闲。
- 上下文并行以两种方式拆分序列位置。预填充上下文并行将长输入跨 GPU 划分,以降低首 token 延迟 (TTFT)。解码上下文并行拆分存储的 KV 历史以增加缓存容量,但每个解码器层都必须合并部分注意力结果。
- 专家并行将 MoE 专家分布到各 GPU 上。Token 路由和不均衡的专家负载增加了网络成本和空闲时间。
- 混合并行在一次部署中结合多种策略。例如,张量并行可在节点内运行,流水线并行跨节点运行,数据并行跨模型副本运行。这将每种通信模式匹配到可用的链路,但增加了更多的协调路径。
Production deployments often combine methods, but each one should fix a measured bottleneck.
生产部署通常组合使用这些方法,但每种方法都应解决一个可测量的瓶颈。
- If the weights do not fit, use tensor or pipeline parallelism. - If long-prompt prefill or KV-cache capacity is the limit, use the corresponding form of context parallelism. - MoE models may need expert parallelism to place expert weights. - If one replica meets its latency target but requests still queue, add data-parallel replicas.
- 如果权重无法容纳,请使用张量并行或流水线并行。 - 如果长提示预填充或 KV 缓存容量是限制因素,请使用相应的上下文并行形式。 - MoE 模型可能需要专家并行来放置专家权重。 - 如果一个副本满足其延迟目标但请求仍在排队,请增加数据并行副本。
Keep tensor parallelism on the fastest links. After each change, measure memory, TTFT, TPOT, and other metrics again.
将张量并行保持在最快的链路上。每次更改后,重新测量内存、TTFT、TPOT 和其他指标。
Add another method only when measurements reveal another bottleneck.
仅在测量揭示另一个瓶颈时添加其他方法。
I wrote a detailed article explaining how each strategy works, where communication occurs, and how to choose a layout for a real deployment.
我写了一篇详细文章,解释每种策略的工作原理、通信发生的位置以及如何为实际部署选择布局。
Read it below.
点击下方阅读。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力