AI工程师必知的12种KV Cache优化技术
I have been using LLMs in production for over 2.5 years now!
做LLM推理部署的同学必看，这篇把KV Cache优化的12种主流技术及其适用场景讲透了，直接指导工程选型。
I have been using LLMs in production for over 2.5 years now!
我使用大语言模型（LLM）进行生产部署已经超过 2.5 年了！
Here are 12 KV cache reduction techniques every AI engineer should understand:
以下是每位 AI 工程师都应了解的 12 种 KV 缓存缩减技术：
KV cache memory grows with the number of layers, KV heads, retained tokens, dimensions, bytes per value, and concurrent requests.
KV 缓存内存的大小随层数、KV 头数、保留的 token 数量、维度、每个值的字节数以及并发请求数的增加而增长。
The techniques below reduce different parts of that cost. Some require a model built around a different attention architecture. Others work inside the serving engine.
以下技术针对降低该成本的不同部分。其中一些需要基于不同注意力架构构建的模型，另一些则可在推理引擎内部实现。
1. GQA and MQA
1. GQA 和 MQA
Multiple query heads share fewer key and value heads. The model stores fewer KV vectors for every token and layer.
多个查询头共享较少的键头和值头。模型为每个 token 和层存储更少的 KV 向量。
2. Cross-layer sharing
2. 跨层共享
Several transformer layers reuse the same KV cache instead of storing independent copies. Native support usually requires a model trained for this layout.
多个 Transformer 层复用相同的 KV 缓存，而不是存储独立的副本。原生支持通常需要为此布局专门训练的模型。
3. Sliding-window attention
3. 滑动窗口注意力
Local layers retain only the latest window of tokens. Their cache stops growing beyond that window, although global layers may still retain the full sequence.
局部层仅保留最新的 token 窗口。其缓存在此窗口之外不再增长，尽管全局层可能仍保留完整序列。
4. Multi-head latent attention
4. 多头潜在注意力 (MLA)
MLA stores a compressed latent representation instead of full keys and values. The attention layer reconstructs what it needs during decoding.
MLA 存储压缩后的潜在表示，而非完整的键和值。注意力层在解码过程中重建所需内容。
5. Hybrid Mamba models
5. 混合 Mamba 模型
Mamba layers keep fixed-size recurrent states instead of growing KV caches. Hybrid models need KV memory only for their remaining attention layers.
Mamba 层保持固定大小的循环状态，而非增长的 KV 缓存。混合模型仅需为其剩余的注意力层分配 KV 内存。
6. Compressed attention
6. 压缩注意力
The model stores compact summaries of older context and combines them with selected tokens and a recent window. This can reduce memory and attention work.
模型存储旧上下文的紧凑摘要，并将其与选定的 token 及近期窗口相结合。这可以减少内存使用和注意力计算量。
7. Query-aware sparse reads
7. 查询感知稀疏读取
The engine scores KV pages for the current query and reads only the most relevant ones. The complete cache remains stored, so this mainly reduces memory traffic and computation.
引擎根据当前查询对 KV 页进行评分，并仅读取最相关的页面。完整的缓存仍保留在存储中，因此这主要减少了内存流量和计算量。
8. KV quantization
8. KV 量化
Every cached value remains present, but each uses fewer bits. An 8-bit cache has the same shape as a 16-bit cache with a smaller byte footprint.
所有缓存的值均保留，但每个值使用的位数更少。8 位缓存的形状与 16 位缓存相同，但占用的字节数更小。
9. Token eviction
9. Token 驱逐
The engine permanently removes tokens judged less useful. Memory falls, but an evicted token cannot help a later query.
引擎永久移除被认为用处较小的 token。内存占用减少，但被驱逐的 token 无法帮助后续的查询。
10. Paged allocation
10. 分页分配
KV memory is divided into fixed-size blocks allocated on demand. This reduces fragmentation and allocation waste without deleting cached values.
KV 内存被划分为固定大小的块，按需分配。这在不删除缓存值的情况下减少了碎片和分配浪费。
11. Prefix reuse
11. 前缀复用
Requests with an identical prefix share already-computed KV blocks. This removes duplicate storage and skips repeated prefill computation.
具有相同前缀的请求共享已计算的 KV 块。这消除了重复存储并跳过了重复的预填充计算。
12. Offloading
12. 卸载
Inactive KV blocks move from GPU memory to CPU memory or disk. GPU capacity increases, but cache misses now pay a transfer cost.
非活跃的 KV 块从 GPU 内存移动到 CPU 内存或磁盘。GPU 容量增加，但缓存未命中现在会产生传输成本。
The important distinction is that these methods do not all compress the cache. Some shrink it, and some reduce reads, duplicated computation, fragmentation, or GPU residency.
重要的区别在于，这些方法并非都会压缩缓存。有些会缩小它，有些则减少读取、重复计算、碎片化或 GPU 驻留。
If you want to learn more, I wrote a full article about it, covering how these techniques work, what each technique actually saves, and the trade-offs to decide which one would fit your workload.
如果你想了解更多，我写了一篇关于它的完整文章，涵盖了这些技术的工作原理、每种技术实际节省的内容，以及决定哪种适合你的工作负载的权衡取舍。
Read it below.
在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力