实测语义缓存：多数LLM系统不应运行，因错误率不可接受
I built a semantic cache for LLM calls, measured it, and concluded most systems should not run one
Agent开发必看，这篇用真实数据拆解了语义缓存的致命缺陷，直接给出工程取舍建议，避免踩坑。
Semantic caching for LLMs is everywhere: embed the prompt, find the nearest stored one, serve its answer if it's close enough.
LLM 的语义缓存无处不在：将提示词嵌入，找到最接近的已存储项，如果足够接近就返回其答案。
I wanted to know what "close enough" is actually worth, so I built 40 triples — an original prompt, a paraphrase that means the same thing, and a near-miss changed as little as possible so the correct answer is different ("a trip to Hawaii" -> "a trip to Japan").
我想知道“足够接近”到底值多少，于是我构建了 40 组三元组——一个原始提示词、一个意思相同但表述不同的改写版本，以及一个尽可能少改动但仍导致正确答案不同的近似错误版本（例如“去夏威夷旅行”->“去日本旅行”）。
Paraphrases scored a median cosine similarity of 0.836. Near-misses scored 0.911. The variants that are unsafe to serve are, on average, NEARER than the safe ones — 33 of 40 near-misses scored above the median paraphrase. It's not overlap, it's inversion, and it's a property of what embeddings are for: they encode what a sentence is about, and the one token that decides the answer is a rounding error.
改写版本的余弦相似度中位数为 0.836。近似错误版本的得分为 0.911。平均而言，不适合直接服务的变体比安全的变体更接近——40 个近似错误中有 33 个的得分高于改写版本的中位数。这不是重叠问题，而是反转问题，这是嵌入向量特性的体现：它们编码句子是关于什么的，而决定答案的那个 token 只是一个舍入误差。
Prior art agrees. GPTCache's own paper says hit rates don't exceed 90% with current embeddings and that the rerankers they tried didn't separate good hits from bad. AWS's ElastiCache benchmark on 63,796 real queries shows accuracy stuck between 91% and 93% whether you set the threshold to 0.75 or 0.99.
现有研究也支持这一观点。GPTCache 自己的论文指出，使用当前的嵌入向量时命中率不会超过 90%，且他们尝试的重排序器未能区分好的命中和坏的命中。AWS 对 63,796 个真实查询进行的 ElastiCache 基准测试显示，无论将阈值设置为 0.75 还是 0.99，准确率都卡在 91% 到 93% 之间。
There is a fix: similarity recalls 3 candidates, a small cheap model decides whether the stored question and the new one actually have the same answer. Built on DynamoDB's vector index inside an AgentCore Gateway interceptor, measured live:
有一个解决方案：相似度召回 3 个候选项，一个小而廉价模型判断存储的问题和新问题是否真的具有相同的答案。该方案构建在 AgentCore Gateway 拦截器内的 DynamoDB 向量索引之上，实测结果如下：
- verify on, recall >= 0.65: 90% of paraphrases served, 5% of near-misses wrongly served
- verify on, recall >= 0.80: 50% / 5%
- verify off, recall >= 0.80: 50% / 90%
- 验证开启，召回率 >= 0.65：90% 的改写版本被服务，5% 的近似错误被错误地服务
- 验证开启，召回率 >= 0.80：50% / 5%
- 验证关闭，召回率 >= 0.80：50% / 90%
Turning the verifier off doesn't degrade the cache, it inverts it — and it's faster, which is the trap. Both metrics a cache normally reports improve.
关闭验证器并不会降低缓存性能，而是将其反转——而且速度更快，这就是陷阱所在。缓存通常报告的两个指标都得到了改善。
Then the part that decided it for me. That 5% is a floor, not a setting, and it applies to every hit - so at a 20% hit rate you have bought a 16% cost reduction by giving 1% of ALL requests a confident answer to a question nobody asked. And you cannot detect it: a wrong cached answer is indistinguishable from a wrong model.
然后是让我下定论的部分。那 5% 是一个底线，而不是一个可设置的参数，并且它适用于每一次命中——所以在 20% 的命中率下，你通过给所有请求中的 1% 提供对一个无人提问的问题的自信答案，换取了 16% 的成本降低。而且你无法检测到它：错误的缓存答案与错误的模型输出是无法区分的。
The economics are fine, which is the trap. Against an expensive model the break-even is a 0.15% hit rate; against a cheap one it never pays (verification costs 32x the call it avoids). Cost is not the reason to skip this. The error rate is.
经济账是划算的，这本身就是陷阱。对于昂贵的模型，盈亏平衡点的命中率为 0.15%；对于便宜的模型，则永远不划算（验证成本是其所避免调用的 32 倍）。跳过此步骤的原因不是成本，而是错误率。
An exact cache is free, instant, and wrong zero percent of the time. It fires less often, and less often beats sometimes wrongly.
精确缓存是免费的、即时的，并且零百分比时间出错。它触发频率较低，较少出现有时错误获胜的情况。
Code, CloudFormation and the raw results are in the post.
代码、CloudFormation 模板和原始结果都在文章中。
https://brianfeeny.com/posts/semantic-caching-for-llms-on-aws/?utm\_source=reddit&utm\_medium=social&utm\_campaign=semantic-cache-2026-09
https://brianfeeny.com/posts/semantic-caching-for-llms-on-aws/?utm_source=reddit&utm_medium=social&utm_campaign=semantic-cache-2026-09
Disclosure: I work at AWS; this is personal work and the views are my own.
披露：我在 AWS 工作；这是个人作品，观点仅代表我自己。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力