PromptCheckup：解决LLM提示词评估中的模型漂移问题
Your new prompt scored higher. That doesn't mean it's better. I built a free tool that tells you if it actually is
做Prompt工程的同学必看，这个配对比较思路直接解决了模型漂移带来的评估噪音，CI集成方案也很实用，建议直接试用。
This has been bugging me for a while.
这个问题困扰我有一段时间了。
Every time I tweak a prompt, I do the same thing. I check last week's score, ship the change, check this week's score, and if the number went up I call it a win.
每次我调整提示词（prompt）时，我都会做同样的事情：查看上周的得分，部署更改，查看本周的得分，如果数字上升了，我就称之为胜利。
But the model under you isn't standing still. Researchers at Stanford and Berkeley ran the exact same questions through GPT-4 in March and June 2023, and its accuracy at spotting prime numbers dropped from 84% to 51%. Same prompts, and nobody touched anything (paper). So when my number goes up, how much of that is my prompt and how much is the model having a good week?
但你模型的表现并非静止不变。斯坦福大学和伯克利大学的研究人员在 2023 年 3 月和 6 月对 GPT-4 进行了完全相同的提问测试，其识别质数的准确率从 84% 下降到 51%。使用相同的提示词，且期间没有任何人为干预（论文）。那么当我的得分上升时，有多少归功于我的提示词，又有多少是因为模型本身表现较好？
I couldn't answer that, so I built a thing called promptcheckup. It runs the old prompt and the new prompt on the same cases, back to back, and just tells you better, worse, or not sure yet, with a 95% interval. Both prompts see the exact same inputs, so whatever drifted in the background cancels out.
我无法回答这个问题，所以我构建了一个名为 promptcheckup 的工具。它在相同的案例上依次运行旧提示词和新提示词，并直接告诉你结果是更好、更差还是尚未确定，同时提供 95% 的置信区间。两个提示词看到完全相同的输入，因此背景中发生的任何漂移都会相互抵消。
I simulated it with weekly drift. The usual week vs week comparison picked the worse prompt 41% of the time. The paired version never got it wrong in 2,000 runs.
我用每周的性能漂移对其进行了模拟。通常的周与周对比有 41% 的概率选出了更差的提示词。而配对版本在 2000 次运行中从未出错。
You can drop it in CI too. It exits 0 if the new prompt is better, 1 if it's worse and 2 if it can't tell, so a bad prompt fails the build. It works with your own score or with an LLM judge, and the judge gets checked against itself first. If the judge is flaky, you get not sure yet instead of a fake answer. There are zero dependencies, just the Python standard library.
你也可以将其集成到 CI 中。如果新提示词更好，它退出码为 0；如果更差，退出码为 1；如果无法判断，退出码为 2，这样糟糕的提示词会导致构建失败。它支持使用你自己的评分标准或 LLM 裁判，并且裁判会先进行自我校验。如果裁判不稳定，你会得到“尚未确定”而不是虚假的答案。零依赖，仅使用 Python 标准库。
There's a demo that needs no API key. Run PYTHONPATH=src python3 examples/planted/planted.py in the repo.
有一个无需 API key 的演示版本。在仓库中运行 PYTHONPATH=src python3 examples/planted/planted.py。
Here's the repo. Small improvements need a lot of test cases, and it can't fix biased cases. If my stats are off, please tell me. I'd rather hear it here than find out in prod.
这是仓库链接。微小的改进需要大量的测试用例，而且它无法修复存在偏差的案例。如果我的统计有误，请告诉我。我宁愿在这里听到反馈，也不愿在生产环境中才发现。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力