LLM 连续批处理机制详解
Continuous batching in LLMs, clearly explained:
做 LLM 推理优化的同学必看，这篇把连续批处理的原理和收益讲得很清楚，还附了调度器深度拆解，值得收藏。
Continuous batching in LLMs, clearly explained:
LLM中的连续批处理，清晰解释：
(a popular LLM interview question; bookmark this)
（一个流行的LLM面试问题；请收藏此内容）
In traditional ML inference, a batch is a matrix.
在传统机器学习推理中，一个批次是一个矩阵。
Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.
每个输入都被填充到相同长度，运行一次前向传播，每一行在同一时刻完成。
LLM decoding does not work that way.
LLM解码并非如此。
One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.
一次前向传播为每个序列生成一个token，因此一个请求需要与其输出token数量相同的传播次数，而直到模型发出停止token之前，没人知道这个数量。
Under static batching, membership is fixed when the batch starts.
在静态批处理下，批次开始时成员关系是固定的。
A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400.
一个在30个token内完成的请求会占用其槽位，直到同一批次中最慢的请求在400个token时完成。
The GPU keeps paying the full weight read for a batch that is mostly empty.
GPU持续为大部分为空的批次支付完整的权重读取成本。
Loading model weights out of HBM costs the same whether four slots are producing tokens or one.
从HBM加载模型权重的成本相同，无论四个槽位在生成token还是只有一个。
Continuous batching moves the decision boundary.
连续批处理移动了决策边界。
Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.
调度器不是每批调度一次，而是运行一次前向传播，获得控制权，然后再次决定。
A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done.
完成的请求在下一个迭代边界离开，排队的请求立即占据其槽位。没有槽位为已完成的工作保留。
Anyscale benchmarked both OPT-13B on a single A100.
Anyscale在单个A100上对OPT-13B进行了基准测试。
With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving.
在均匀生成长度下，两种策略大致持平（如预期），随着输出长度方差增加，静态批处理降至约每秒81个token，而vLLM达到了朴素Hugging Face服务吞吐量的23倍。
Variance drives the entire gap.
方差驱动了整个差距。
Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.
生产流量混合了30个token的回复和400个token的回复，这正是静态批处理处理最差的条件。
None of this alters the model.
这些都不会改变模型。
vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching.
vLLM、SGLang、TGI和TensorRT-LLM默认运行它，NVIDIA以in-flight batching的名称提供相同的机制。
The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep.
下面的动画在相同的16个请求和相同的4个槽位上运行两种策略，步调一致。
The only difference is when a new request is allowed in.
唯一的区别是新请求何时被允许进入。
To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it.
要更深入地了解连续批处理，我写了其底层调度器的完整分解。
It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation.
它涵盖了两次前向传播之间发生什么、如何根据固定预算分配token、为什么调度器不需要为预填充和解码分开路径，以及当KV缓存生成中途填满时抢占的成本。
Read it below.
在下面阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力