LLM推理四大缓存机制详解：KV、前缀、提示与语义缓存
Four caches in LLM serving, clearly explained:
AI工程师必读，清晰拆解了LLM推理中易混淆的四层缓存架构及适用场景，附带可落地的代码思路，对优化Agent循环性能极有帮助。
Four caches in LLM serving, clearly explained:
LLM 服务中的四种缓存，清晰解释：
Every LLM request reads the whole prompt and computes attention state for every token in it.
每个 LLM 请求都会读取整个提示词，并计算其中每个 token 的注意力状态。
This step is called prefill, and it impacts both the input bill and the time before the first token appears.
这一步称为预填充（prefill），它既影响输入账单，也影响首个 token 出现前的时间。
In an agent loop, most of the prompt comprises text that the model already processed in the previous turn.
在智能体循环中，大部分提示词由模型在前一轮已经处理过的文本组成。
There are four cache layers that prevent paying for the processed tokens at each turn.
有四种缓存层可以防止在每一轮都为已处理的 token 付费。
↳ The KV cache holds the key and value tensors for every token at every layer, for one active request.
↳ KV 缓存为单个活跃请求在每个层保存每个 token 的键和值张量。
↳ Prefix caching keeps those tensors on the server instead. vLLM stores them in 16-token blocks and identifies each block by a hash that chains in the previous block's hash, so a block only matches if everything before it matched too. The scheduler stops at the first miss and prefills the suffix from there.
↳ 前缀缓存将这些张量保留在服务器上。vLLM 将它们存储在 16 个 token 的块中，并通过哈希标识每个块；该哈希链接到前一个块的哈希，因此只有当之前的所有内容都匹配时，块才会匹配。调度器在遇到第一个未命中时停止，并从那里开始预填充后缀。
↳ Prompt caching is the same reuse that runs on a provider's hardware, with a price sheet attached. Anthropic charges 1.25x the base input rate to write an entry and 0.1x to read it.
↳ 提示词缓存是在提供商硬件上运行的相同复用机制，并附带价格表。Anthropic 写入一条记录的收费是基础输入费率的 1.25 倍，读取则是 0.1 倍。
↳ Semantic caching works differently. It embeds the incoming prompt, runs a similarity search over stored prompts, and returns a stored answer outright when the score is above a threshold.
↳ 语义缓存的工作方式不同。它对传入的提示词进行嵌入，在存储的提示词上运行相似度搜索，并在得分超过阈值时直接返回存储的答案。
That's why it saves output tokens as well as input tokens. It's also why every request pays for an embedding round trip, including every miss.
这就是为什么它既能节省输出 token，也能节省输入 token。这也是为什么每个请求都要支付一次嵌入往返的费用，包括每次未命中的情况。
The first three match on exact tokens and cannot change what the model produces.
前三者基于精确 token 匹配，无法改变模型产生的内容。
This technique matches on similarity, which means it is quite susceptible to generating a wrong response since embeddings may match to a wrong prompt.
这种技术基于相似度匹配，这意味着它很容易生成错误响应，因为嵌入可能会匹配到错误的提示词。
The diagram below depicts all these techniques.
下图展示了所有这些技术。
To use these techniques, you don't need to build a custom serving stack.
要使用这些技术，你不需要构建自定义的服务堆栈。
The transformers library already implements the cache as an object of KV vectors that you can preserve, so you can prefill a corpus once, retain the returned tensors, and reuse them across queries in about ten lines.
transformers 库已经将缓存实现为一个可保留的 KV 向量对象，因此你可以一次性预填充语料库，保留返回的张量，并在大约十行代码中跨查询重用它们。
And this KV cache is only one of four separate caching layers in an LLM stack.
而这个 KV 缓存只是 LLM 堆栈中四个独立缓存层之一。
The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely.
其他三种分别是服务器上的前缀缓存、按提供商计费的提示词缓存，以及完全跳过模型的语义缓存。
I wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each.
我写了一篇关于 LLM 服务中所有四种缓存的详细分解文章，这是作为 AI 工程师你应该了解的，并附有每种技术的代码。
Read it below.
请在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力