LLM-as-judge评测成本降低70%的工程实践
I cut my LLM-as-judge eval costs ~70% with prompt-cache engineering and a cheap decision-model cascade
做评测的同学必看,这套Prompt缓存+决策模型级联的方案直接把LLM-as-judge成本砍了70%,且一致性极高,赶紧拿去优化你的评测链路。
I have put quite a lot of thought on running evals, both in terms of their correctness, but also on how to make them cheaper. And this post is on the latter, what I have learned running thousands of LLM-as-judge checks, and what actually moved the bill.
我在运行评估(evals)方面投入了大量思考,既关注其正确性,也关注如何降低成本。本文聚焦于后者:我通过运行数千次 LLM-as-judge 检查所获得的经验,以及真正影响账单的因素。
If running only one test, the options are limited, but when running a lot of similar tests, we get a lot of good engineering practices to prompt structures, cache point selections, and even the System One models to get the price for individual checks as low as possible.
如果只运行一项测试,选择有限;但当运行大量相似测试时,我们可以采用许多优秀的工程实践,例如提示词结构、缓存点选择,甚至选用 System One 模型,从而将单次检查的成本降至最低。
(I'm also building an eval tool called Apo, but all of this works with any provider.)
(我正在构建一个名为 Apo 的评估工具,但所有这些方法都适用于任何提供商。)
The prompt
提示词(The prompt)
With the current pricing, the reading from cache is in most cases 10x cheaper than reading without it. And prompt caching is a byte-prefix match, so you want everything that does not change at the start of the prompt, and everything that changes at the end. I'll use the parts of the judge prompts as: briefing (same on all calls), deliverable (the artifact we evaluate), and instruction (the actual question, e.g. "PASS if the name is Jane Doe"). Put the cache point (cache_control: { type: "ephemeral" }) at the end of the deliverables, and only the instruction is read uncached.
根据当前定价策略,大多数情况下,从缓存中读取的成本比不缓存便宜 10 倍。而提示词缓存基于字节前缀匹配,因此我们希望提示词开头部分保持不变,变化部分放在末尾。我将法官提示词(judge prompts)分为三部分:简报(briefing,所有调用相同)、交付物(deliverable,即我们要评估的产物)和指令(instruction,即具体问题,例如“如果名字是 Jane Doe 则 PASS”)。将缓存点(cache_control: { type: "ephemeral" })设置在交付物的末尾,这样只有指令部分是以非缓存方式读取的。
Order of the runs
运行顺序
You can't just fire all the checks at the same time, the cache is not yet active, and every request pays the full price. With three checks on the same deliverable, that is 69,879 billed tokens when they all race the cold cache:
你不能同时发起所有检查请求,因为此时缓存尚未激活,每个请求都需要支付全额费用。对于针对同一交付物的三项检查,当它们竞争冷缓存时,计费 token 数为 69,879:
So the runs need to be run in a way that only one request pays the full price and warms the cache, the rest are run after the cache is active. It adds some time, but makes it MUCH cheaper overall, 23,563 billed tokens, roughly two thirds off the input bill. Different deliverables lists get their own cache key (sha256(model + briefing + deliverable)) and their own warmer, so the lanes never wait on each other.
因此,运行方式应确保只有一个请求支付全额费用并预热缓存,其余请求在缓存激活后再运行。这会增加一些时间,但整体成本大幅降低,计费 token 数仅为 23,563,输入账单减少了约三分之二。不同的交付物列表拥有各自的缓存键(sha256(model + briefing + deliverable))和各自的预热机制,因此各通道之间不会相互等待。
Routing to the warm cache
路由到热缓存
As LLMs are not just a single service, if the routing goes to the service that does not contain our cache, we pay the full read price. So send a prompt_cache_key with the requests, then the servers know to route the requests to the same warm cache service. In my testing on my own proxy, forgetting to put that key really made the requests get 0 cached tokens.
由于 LLM 并非单一服务,如果请求被路由到不包含我们缓存的服务,我们将支付全额读取费用。因此,应在请求中附带 prompt_cache_key,这样服务器就知道将请求路由到同一个热缓存服务。在我对自己代理服务器的测试中,忘记设置该键会导致请求获得 0 个缓存 token。
Cascading checks
级联检查
And we don't even always need the LLM-as-judge. Instead we can use System One models (the fast and cheap kind) like Jev or OpenAI's decision models to do the initial evaluation, and if the confidence is high, we don't need to run LLM-as-judge. When confidence is low then run the judge:
我们甚至并不总是需要 LLM-as-judge。相反,我们可以使用 System One 模型(快速且便宜的类型),例如 Jev 或 OpenAI 的决策模型来进行初步评估;如果置信度高,我们就不需要运行 LLM-as-judge。当置信度低时,再运行裁判模型:
And this is not a guess: on my production checks, 72% of the tests land at or above 0.95 confidence, the band where the decision model and the LLM judge agree 99.5% of the time. With this alone, I managed to make the LLM-as-judges 70% cheaper on my test set, while only 0.4% of the checks changed verdicts. So, the cheaper way to run LLM-as-judges is not to run them, instead use cheaper models.
这并非猜测:在我的生产环境检查中,72% 的测试达到或超过 0.95 的置信度,这是决策模型与 LLM 裁判一致率达 99.5% 的区间。仅凭这一点,我就将测试集上 LLM-as-judge 的成本降低了 70%,而仅有 0.4% 的检查改变了判决结果。因此,降低 LLM-as-judge 成本的最便宜方法是不运行它们,而是使用更便宜的模型。
Tiered system
分层系统
Another point that saves is running tests that do not even need LLMs, deterministic code checks are much cheaper, and many times this is possible. The ladder per check on my runs:
另一个节省成本的点是运行那些根本不需要 LLM 的测试,确定性代码检查要便宜得多,而且很多时候是可行的。我运行中每次检查的阶梯式成本如下:
| Decider | Cost per check | Latency |
|---|---|---|
| deterministic code check | $0 | milliseconds |
| decision model (Jev) | ~$0.0001 | ~0.3 s |
| LLM-as-judge | ~$0.003 | ~25 s |
| agent-as-judge | >$0.003 | ~25 s, multi-step |
| 决策者 | 每次检查成本 | 延迟 |
|---|---|---|
| 确定性代码检查 | $0 | 毫秒级 |
| 决策模型 (Jev) | ~$0.0001 | ~0.3 秒 |
| LLM-as-judge | ~$0.003 | ~25 秒 |
| agent-as-judge | >$0.003 | ~25 秒,多步骤 |
With all of these together, running the whole suite on every commit is finally cheap enough to actually do it. I have the full write-up, with the rest of the details and the animated versions of these diagrams, in my blog, happy to answer questions in the comments.
结合所有这些方法,现在在每次提交时运行整个测试套件终于足够便宜,可以真正去执行了。我在博客上发布了完整的文章,包含其余细节和这些图表的动态版本,欢迎在评论区提问。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力