LLM Router 拒答误判:接受检查如何影响成本与排查
Our model router logged five rejected answers as a provider error
Router 开发者必看,揭示了接受检查(Acceptance Check)对成本和排查方向的致命影响,提供了保留拒答原因以区分故障类型的工程实践思路。
In one OmniNode run, our router tried five times. Each attempt returned an answer, and our acceptance check rejected every one. The final record called it a provider error. The providers had answered promptly, so that label would have sent us investigating the wrong thing.
在一次 OmniNode 运行中,我们的路由器尝试了五次。每次尝试都返回了一个答案,但我们的接受检查全部拒绝了它们。最终记录将其标记为提供商错误。由于提供商响应迅速,该标签会引导我们调查错误的方向。
The router starts with the cheapest eligible model and tries a more capable one when the answer doesn't pass. That makes the acceptance check part of the spending decision. If the check is wrong, another attempt can cost more without getting us any closer to an accepted answer.
路由器从最便宜的合格模型开始,当答案未通过检查时,再尝试能力更强的模型。这使得接受检查成为支出决策的一部分。如果检查出错,另一次尝试可能会花费更多,却让我们离被接受的答案更近不了多少。
We had a separate case on September 28 where the check rejected a correct one-word answer, apparently because it didn't show enough effort. I don't have evidence that the five earlier answers were correct. These were different cases, but both made me want to inspect the check before blaming the model.
9 月 28 日我们遇到了一个单独案例,检查拒绝了一个正确的一词答案,显然因为它没有显示出足够的努力。我没有证据表明之前的五个答案是正确的。这些是不同的案例,但两者都让我想在责怪模型之前先检查这个判断逻辑。
I'm trying to keep the returned answer and the reason it was rejected available together. A transport failure, an answer that misses the requirement, and a check that rejects a valid answer need different investigations.
我试图将返回的答案及其被拒绝的原因放在一起保留。传输故障、未能满足要求的答案以及拒绝有效答案的检查,需要不同的调查方向。
How do you test that boundary in a router? Do you have cases where the model gives a valid short answer and the judge is expected to accept it?
你如何测试路由器中的这一边界情况?你有这样的案例吗:模型给出了一个有效的简短答案,而裁判预期会接受它?
The incident and its limits are in this write-up: https://jonahatomninode.substack.com/p/delegation-is-the-first-product
此次事件及其局限性详见本文:https://jonahatomninode.substack.com/p/delegation-is-the-first-product
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力