推测解码技术详解：让LLM推理提速2-3倍的四种方案
This technique makes LLM inference 2-3x faster:
工程落地价值极高，清晰对比了四种推测解码方案的取舍，做推理优化的同学可直接参考选型。
This technique makes LLM inference 2-3x faster:
这项技术使 LLM 推理速度提升 2-3 倍：
(Google Search uses this in production)
（Google Search 在生产环境中使用了此技术）
Under autoregressive decoding, the target model produces one token per forward pass.
在自回归解码下，目标模型每次前向传播生成一个 token。
Each token is conditioned on the sequence generated so far, so generation cannot advance until another pass runs through every transformer layer.
每个 token 都依赖于目前已生成的序列，因此必须等待另一次前向传播遍历所有 transformer 层后，生成过程才能继续推进。
At low batch sizes, this process is usually memory-bandwidth bound, leaving much of the GPU's compute idle.
在低批量大小（batch size）下，该过程通常受限于内存带宽，导致 GPU 的大量计算能力处于空闲状态。
Instead, a cheaper draft path can propose several future tokens. The large target model can then check the complete block in one forward pass (this is similar to prefill).
相反，成本较低的草稿路径可以提议多个未来 token。然后，大型目标模型可以在一次前向传播中检查整个块（这类似于预填充 prefill）。
This technique is called speculative decoding. Google uses it in production for AI Overviews in Search.
这项技术称为推测解码（speculative decoding）。Google 在生产环境中将其用于搜索中的 AI Overviews。
Suppose the drafter proposes five tokens. If the target accepts all five, it can also add one bonus token from the same verification pass.
假设草稿器提议了五个 token。如果目标模型接受了全部五个，它还可以从同一次验证过程中额外添加一个奖励 token。
If the fourth token is wrong, the first three are accepted. The target corrects the fourth, and the remaining draft is discarded.
如果第四个 token 是错误的，则前三个被接受。目标模型修正第四个 token，并丢弃剩余的草稿部分。
With the proper acceptance rule, the output distribution remains identical to the target model's distribution.
通过适当的接受规则，输出分布与目标模型的分布保持一致。
There are four common ways to generate drafter tokens:
生成草稿 token 有四种常见方法：
1. Two-model speculative decoding
1. 双模型推测解码
A smaller model generates a few tokens sequentially, and the larger model verifies them together.
较小的模型顺序生成少量 token，较大的模型一起验证它们。
This is the most basic setup since it leaves the target unchanged. The tradeoff is that you must maintain another set of weights, another KV cache, and more scheduling work.
这是最基本的设置，因为它不改变目标模型。权衡之处在于，你必须维护另一组权重、另一个 KV cache 以及更多的调度工作。
2. EAGLE
2. EAGLE
EAGLE replaces the separate draft LLM with a lightweight module trained on the target model's hidden states.
EAGLE 用轻量级模块替换了独立的草稿 LLM，该模块是在目标模型的隐藏状态上训练而成的。
It drafts from the target's internal representation, but the module must be trained for that checkpoint.
它基于目标模型的内部表示进行草稿生成，但该模块必须针对该检查点（checkpoint）进行训练。
3. Medusa
3. Medusa
Medusa attaches several prediction heads to the target model. Each head predicts a different future position, so several candidate tokens are produced in parallel.
Medusa 在目标模型上附加了几个预测头（prediction heads）。每个头预测不同的未来位置，从而并行产生多个候选 token。
The heads do not condition on each other, so their predictions may not form one consistent sequence. Medusa builds a tree and verifies several candidate paths together.
这些预测头之间不相互依赖，因此它们的预测可能无法形成一致的一个序列。Medusa 构建一棵树，并一起验证多个候选路径。
4. LayerSkip
4. LayerSkip
LayerSkip uses the target model's early layers as the drafter. The remaining layers verify and correct the proposed tokens.
LayerSkip 使用目标模型的早期层作为草稿器。剩余的层负责验证和修正提议的 token。
This removes the second model and additional prediction heads. However, it requires a checkpoint trained for reliable early exits using layer dropout and an early-exit loss.
这种方法去除了第二个模型和额外的预测头。然而，它需要一个经过专门训练的 checkpoint，以便利用层丢弃（layer dropout）和早退损失（early-exit loss）实现可靠的早退机制。
In production, the useful metric is accepted tokens per target pass after accounting for drafting time, verification overhead, and extra memory.
在生产环境中，有用的指标是扣除草稿时间、验证开销和额外内存消耗后，每次目标模型前向传播所接受的 token 数量。
A more accurate drafter can still make the complete system slower if producing its guesses costs too much.
如果生成猜测的成本过高，更精确的草稿模型仍可能使整个系统变慢。
To dive deeper, I wrote a deeper breakdown of speculative decoding, including the acceptance process and why the output distribution remains unchanged.
为了更深入地了解，我撰写了一篇关于推测解码（speculative decoding）的详细解析，包括接受过程以及为何输出分布保持不变。
Read it below.
请在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力