跳到主内容
@wquguru
精选88Avi Chawla技巧与观点

大模型推理连续批处理机制解析与性能对比

Continuous batching in LLMs, explained visually:

原文
发到 X
推荐理由

做LLM部署的同学必看,这篇把连续批处理的原理和收益讲透了,还有实测数据支撑,直接优化推理吞吐量的硬核干货。

Continuous batching in LLMs, explained visually:

LLM 中的连续批处理,图解说明:

(a popular LLM interview question; bookmark this)

(一个流行的 LLM 面试问题;请收藏此内容)

In traditional ML inference, a batch is a matrix.

在传统 ML 推理中,批次是一个矩阵。

Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.

每个输入都被填充到相同长度,执行一次前向传播,每一行在同一时刻完成。

LLM decoding does not work that way.

LLM 的解码过程并非如此运作。

One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.

一次前向传播为每个序列生成一个 token,因此一个请求需要与其输出 token 数量相同的前向传播次数,而在模型发出停止 token 之前,没人知道这个数量。

Under static batching, membership is fixed when the batch starts.

在静态批处理中,批次成员在开始时即固定不变。

A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400.

一个在 30 个 token 处完成的请求会占用其槽位,直到同一批次中最慢的请求在 400 个 token 处完成。

The GPU keeps paying the full weight read for a batch that is mostly empty.

GPU 持续为大部分为空的批次支付完整的权重读取开销。

Loading model weights out of HBM costs the same whether four slots are producing tokens or one.

无论四个槽位在生成 token 还是一个槽位在生成 token,从 HBM 加载模型权重的成本都是一样的。

Continuous batching moves the decision boundary.

连续批处理改变了决策边界。

Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.

调度器不再按批次进行一次调度,而是运行单次前向传播,获取控制权后再次做出决策。

A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done.

一个已完成的请求会在下一个迭代边界离开,而队列中的请求随即填补其空出的槽位。不会再为已完成的工作保留槽位。

Anyscale benchmarked both OPT-13B on a single A100.

Anyscale 在单张 A100 上对 OPT-13B 进行了基准测试。

With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving.

在生成长度均匀的情况下,两种策略的表现大致相当(正如预期);随着输出长度的方差增大,静态批处理的吞吐量降至约每秒 81 个 token,而 vLLM 达到了朴素 Hugging Face 服务 23 倍的吞吐量。

Variance drives the entire gap.

方差导致了这一全部差距。

Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.

生产流量混合了 30 个 token 的回复和 400 个 token 的回复,这正是静态批处理处理最差的场景。

None of this alters the model.

这一切都不改变模型本身。

vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA implements the same mechanism under the name in-flight batching.

vLLM、SGLang、TGI 和 TensorRT-LLM 默认都运行它,NVIDIA 也在其内部以 in-flight batching 的名义实现了相同的机制。

The animation below depicts the continuous batching process. I took inspiration from Sreedath Panat's post on a recent post to create it.

下面的动画展示了连续批处理的过程。我受到 Sreedath Panat 近期文章中的帖子的启发制作了该动画。

To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it.

为了更深入地了解连续批处理,我撰写了一篇关于其底层调度器的完整解析。

It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation.

它涵盖了两次前向传播之间发生的情况、如何在固定预算下分配 token、为什么调度器不需要为预填充和解码设置单独的路径,以及当 KV 缓存在中途填满时抢占操作的成本。

Read it below.

请在下方阅读。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件