LLM推理优化：利用CUDA Graph消除内核间等待开销
A counterintuitive insight about LLM inference:
深入解析LLM推理底层性能瓶颈与CUDA Graph实战原理，给出可落地的工程优化思路，适合做推理引擎的同学参考。
A counterintuitive insight about LLM inference:
关于大语言模型推理的一个反直觉洞察：
LLM inference can get faster even when every GPU operation takes exactly the same amount of time.
即使每个 GPU 操作耗时完全相同，LLM 推理速度仍可能加快。
The improvement can come entirely from reducing the time between operations.
这种提升可以完全来自于减少操作之间的时间间隔。
An LLM forward pass appears to be one computation.
LLM 的前向传播看似是一次计算。
But on the GPU, it becomes a sequence of smaller operations for attention, matrix multiplication, normalization, sampling, and data movement.
但在 GPU 上，它变成了注意力机制、矩阵乘法、归一化、采样和数据移动等一系列较小的操作序列。
The CPU usually prepares and submits these operations individually.
CPU 通常单独准备并提交这些操作。
That overhead is small for one launch. But token-by-token decoding repeats a similar sequence for every generated token.
对于单次启动来说，这种开销很小。但逐 token 解码会为每个生成的 token 重复类似的序列。
Some kernels complete in only a few microseconds. The GPU can finish one operation before the CPU has submitted the next.
某些内核仅需几微秒即可完成。GPU 可以在 CPU 提交下一个操作之前完成一个操作。
The resulting gaps contain no useful model computation, and the profiler shows the GPU waiting for its next work.
由此产生的间隙中不包含任何有用的模型计算，分析器显示 GPU 正在等待其下一项工作。
This means optimizing inference is not always about accelerating the kernels themselves. Instead, sometimes the larger opportunity is removing the empty waiting time between them.
这意味着优化推理并不总是关于加速内核本身。相反，有时更大的机会在于消除它们之间空白的等待时间。
CUDA Graphs are designed for this situation.
CUDA Graphs 正是为这种情况而设计的。
A CUDA Graph is a saved execution plan containing a sequence of GPU operations and their dependencies.
CUDA Graph 是一个保存的执行计划，包含一系列 GPU 操作及其依赖关系。
The application first runs a warm-up pass. This handles one-time initialization and establishes the shapes, buffers, kernels, and memory addresses required by the workload.
应用程序首先运行预热传递。这处理一次性初始化，并建立工作负载所需的形状、缓冲区、内核和内存地址。
The runtime then captures the operation sequence as a graph.
然后运行时将操作序列捕获为图。
Later iterations can replay the entire graph through one launch instead of submitting every operation separately.
后续迭代可以通过一次启动重放整个图，而不是单独提交每个操作。
Replay does not reuse an earlier output.
重放不会复用之前的输出。
New input is placed into the expected buffers. The GPU runs the captured operations again and produces a new result.
新输入被放入预期的缓冲区。GPU 再次运行捕获的操作并产生新的结果。
The computation remains almost the same. What changes is how efficiently that computation reaches the GPU.
计算过程几乎保持不变。变化的是该计算到达 GPU 的效率。
This works particularly well during LLM decoding because every step processes a new token while following a largely repeated execution path.
这在 LLM 解码期间特别有效，因为每一步都处理一个新 token，同时遵循大致重复的执行路径。
The tradeoff is reduced flexibility.
代价是灵活性的降低。
A captured graph expects compatible operations, tensor shapes, control flow, and memory addresses across replays.
捕获的图期望在重放过程中具有兼容的操作、张量形状、控制流和内存地址。
Real inference traffic rarely has one fixed shape. Batch sizes change, requests enter and leave, and servers switch between prefill and decoding workloads.
实际的推理流量很少具有固定的单一形状。批量大小会变化，请求会进入和离开，服务器会在预填充和解码工作负载之间切换。
Serving engines often capture several graphs for common execution shapes. Other requests can be padded to a captured size or handled through eager execution.
服务引擎通常为常见的执行形状捕获多个图。其他请求可以填充到捕获的大小，或通过即时执行处理。
More captured shapes improve coverage, but consume additional GPU memory and increase startup work.
更多的捕获形状提高了覆盖率，但会消耗额外的 GPU 内存并增加启动工作量。
CUDA Graph replay is useful because it optimizes something kernel benchmarks can easily hide: the cost of repeatedly asking the GPU to begin its next piece of work.
CUDA Graph 回放之所以有用，是因为它优化了内核基准测试容易隐藏的一项内容：反复要求 GPU 开始执行下一项工作的成本。
The visual below shows how warm-up, capture, and replay remove that repeated launch path.
下图展示了预热、捕获和回放如何消除这种重复的启动路径。
This is one reason LLM inference needs a specialized serving engine rather than a standard prediction API.
这也是 LLM 推理需要专用推理引擎而非标准预测 API 的原因之一。
I also wrote an article that explains five architectural differences between traditional ML inference and LLM inference.
我还写了一篇文章，解释了传统 ML 推理与 LLM 推理之间的五个架构差异。
Read it below.
请阅读下文。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力