实测LLM语义缓存：多数系统不应运行，因错误率不可控
I built a semantic cache for LLM calls, measured it, and concluded most systems should not run one
Agent开发者的避坑指南，用数据证明了语义缓存的致命缺陷，直接指导架构取舍。
Semantic caching for LLMs is everywhere: embed the prompt, find the nearest stored one, serve its answer if it's close enough.
LLM 的语义缓存无处不在：将提示词嵌入，找到最接近的已存储项，如果足够接近就返回其答案。
I wanted to know what "close enough" is actually worth, so I built 40 triples — an original prompt, a paraphrase that means the same thing, and a near-miss changed as little as possible so the correct answer is different ("a trip to Hawaii" -> "a trip to Japan").
我想知道“足够接近”到底值多少，于是构建了 40 组三元组——原始提示词、含义相同的改写版本，以及尽可能少改动但导致正确答案不同的近似错误版本（例如“去夏威夷旅行”变为“去日本旅行”）。
Paraphrases scored a median cosine similarity of 0.836. Near-misses scored 0.911. The variants that are unsafe to serve are, on average, NEARER than the safe ones — 33 of 40 near-misses scored above the median paraphrase. It's not overlap, it's inversion, and it's a property of what embeddings are for: they encode what a sentence is about, and the one token that decides the answer is a rounding error.
改写版本的余弦相似度中位数为 0.836。近似错误版本的得分为 0.911。平均而言，不宜直接服务的变体比安全变体更接近——40 个近似错误中有 33 个得分高于改写版本的中位数。这不是重叠问题，而是反转问题，这是嵌入表示的本质属性：它们编码句子是关于什么的，而决定答案的那个 token 只是一个舍入误差。
Prior art agrees. GPTCache's own paper says hit rates don't exceed 90% with current embeddings and that the rerankers they tried didn't separate good hits from bad. AWS's ElastiCache benchmark on 63,796 real queries shows accuracy stuck between 91% and 93% whether you set the threshold to 0.75 or 0.99.
现有研究也支持这一点。GPTCache 自己的论文指出，使用当前嵌入技术命中率无法超过 90%，且他们尝试的重排序器未能区分好命中和坏命中。AWS 对 63,796 个真实查询的 ElastiCache 基准测试显示，无论将阈值设为 0.75 还是 0.99，准确率都卡在 91% 到 93% 之间。
There is a fix: similarity recalls 3 candidates, a small cheap model decides whether the stored question and the new one actually have the same answer. Built on DynamoDB's vector index inside an AgentCore Gateway interceptor, measured live:
有一种解决方案：相似度召回 3 个候选项，一个小型廉价模型判断存储的问题和新问题是否真的具有相同的答案。该方案基于 DynamoDB 向量索引构建，位于 AgentCore Gateway 拦截器内，实测结果如下：
- verify on, recall >= 0.65: 90% of paraphrases served, 5% of near-misses wrongly served
- verify on, recall >= 0.80: 50% / 5%
- verify off, recall >= 0.80: 50% / 90%
- 验证开启，召回率 >= 0.65：90% 的改写版本被服务，5% 的近似错误被错误地服务
- 验证开启，召回率 >= 0.80：50% / 5%
- 验证关闭，召回率 >= 0.80：50% / 90%
Turning the verifier off doesn't degrade the cache, it inverts it — and it's faster, which is the trap. Both metrics a cache normally reports improve.
关闭验证器并不会降低缓存性能，而是使其反转——而且速度更快，这正是陷阱所在。缓存通常报告的两个指标都得到了改善。
Then the part that decided it for me. That 5% is a floor, not a setting, and it applies to every hit - so at a 20% hit rate you have bought a 16% cost reduction by giving 1% of ALL requests a confident answer to a question nobody asked. And you cannot detect it: a wrong cached answer is indistinguishable from a wrong model.
然后是我最终决定的部分。那 5% 是一个底线，而非可调节的设置，并且适用于每次命中——因此在 20% 的命中率下，你通过给所有请求中的 1% 提供对无人提问问题的自信答案，换取了 16% 的成本降低。而且你无法检测到它：错误的缓存答案与错误的模型输出无法区分。
The economics are fine, which is the trap. Against an expensive model the break-even is a 0.15% hit rate; against a cheap one it never pays (verification costs 32x the call it avoids). Cost is not the reason to skip this. The error rate is.
经济账是划算的，这又是陷阱。面对昂贵模型时，盈亏平衡点的命中率为 0.15%；面对廉价模型时则永远不划算（验证成本是所避免调用的 32 倍）。跳过此步骤的原因不是成本，而是错误率。
An exact cache is free, instant, and wrong zero percent of the time. It fires less often, and less often beats sometimes wrongly.
精确缓存免费、即时，且零次出错。它触发频率较低，较少出现有时错误匹配的情况。
Code, CloudFormation and the raw results are in the post.
代码、CloudFormation 模板和原始结果都在文章中。
https://brianfeeny.com/posts/semantic-caching-for-llms-on-aws/?utm\_source=reddit&utm\_medium=social&utm\_campaign=semantic-cache-2026-09
https://brianfeeny.com/posts/semantic-caching-for-llms-on-aws/?utm_source=reddit&utm_medium=social&utm_campaign=semantic-cache-2026-09
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力