RAG vs CAG：把预填充移出查询路径，用 KV 缓存省时
RAG vs. CAG, clearly explained!
做 LLM 推理优化的同学必看，CAG 把预填充移出查询路径，配合 KV 缓存能显著降延迟，赶紧拿你的静态知识库试一下。
RAG vs. CAG, clearly explained!
RAG 与 CAG，清晰解析！
In a standard RAG setup, every query hits the vector DB, including queries about a product manual or policy documents that haven't changed in months.
在标准的 RAG 设置中，每次查询都会命中向量数据库，包括关于产品手册或数月未变的政策文档的查询。
The retrieval adds latency, and then the model prefills those same retrieved chunks again on every subsequent query.
检索增加了延迟，然后模型在每次后续查询中再次预填充那些相同的检索块。
CAG is a technique that drops the vector search and moves the prefill off the query path.
CAG 是一种技术，它放弃了向量搜索，并将预填充移出查询路径。
The preprocessing step runs those documents through the model once, before any query arrives, and keeps the key and value tensors it produces for every token at every layer.
预处理步骤在任何查询到达之前，将这些文档通过模型运行一次，并保留它为每个 token 在每一层产生的键和值张量。
At query time, the model loads that state and starts decoding, with no vector search or prefill on the knowledge.
在查询时，模型加载该状态并开始解码，无需向量搜索或对知识进行预填充。
The amount of context you can store as cache isn't limited by the context length of the model but rather the GPU memory.
你可以作为缓存存储的上下文量不受模型上下文长度的限制，而是受 GPU 内存的限制。
For instance, in a 70B model at BF16, the cache takes around 300 KB/token, so even a small corpus can produce tens of GBs of cache to manage.
例如，在 BF16 的 70B 模型中，缓存大约占用 300 KB/token，因此即使是一个小的语料库也可能产生数十 GB 的缓存需要管理。
That's why production setups run both RAG and CAG together.
这就是为什么生产环境同时运行 RAG 和 CAG。
↳ Static, high-value knowledge that nearly every query reads gets cached once, like policies, product docs, and standing instructions.
↳ 静态、高价值的知识，如政策、产品文档和常规指令，几乎每次查询都会读取，这些会被缓存一次。
↳ Everything else stays in the vector DB, since a document that surfaces in one query out of a thousand doesn't justify holding its tensors on the GPU all day.
↳ 其他所有内容保留在向量数据库中，因为一个在千次查询中只出现一次的文档，不值得整天在 GPU 上保留其张量。
The diagram below depicts this.
下图描绘了这一点。
To use this in practice, you don't need to build a custom serving stack.
要在实践中使用这一点，你不需要构建自定义的服务栈。
The transformers library already implements the cache as an object of KV vectors that you can preserve, so you can prefill a corpus once, retain the returned tensors, and reuse them across queries in about ten lines.
transformers 库已经将缓存实现为 KV 向量的对象，你可以保留它，因此你可以预填充一次语料库，保留返回的张量，并在大约十行代码中跨查询重用它们。
And this KV cache is only one of four separate caching layers in an LLM stack.
而这个 KV 缓存只是 LLM 栈中四个独立缓存层之一。
The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely.
其他三个是服务器上的前缀缓存、提供商计费的提示缓存，以及完全跳过模型的语义缓存。
I wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each.
我写了一篇关于 LLM 服务中所有四种缓存的完整解析，作为 AI 工程师你应该了解，每种都附有代码。
Read it below.
在下面阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力