AI产品“不可解释失败”的常态化与工程反思
The Normalization of Inexplicable Failures
深度剖析AI落地中的工程陷阱，对正在构建Agent或集成LLM产品的团队极具警示意义，建议收藏并对照自查评估流程。
Sunday, September 27, 2026
2026年9月27日，星期日
the normalization of inexplicable failures
无法解释的故障常态化
In a recent episode of President Curtis, the President struggles with opening a door on two separate occasions.
在《柯蒂斯总统》（President Curtis）最近的一集中，总统在两个不同的场合努力尝试打开一扇门。
These doors don't work because there are obstructions in the way: a body initially, then roughly a billion dollars worth of gold.
这些门打不开是因为有障碍物挡路：起初是一具尸体，然后是一堆价值约十亿美元的金子。
In both instances, in response to the frustration, the character mutters "stupid thing sucks." This is not a reasonable model of doors! Doors should not "suck" inexplicably! I found these moments outrageously hilarious¹ but maybe my stupid brain just sucks.
在这两种情况下，面对挫折，角色都嘟囔着“这破玩意儿真烂”。这根本不是对门的合理描述！门不应该莫名其妙地“烂”！我觉得这些时刻荒谬得令人发笑¹，但也许是我那愚蠢的大脑真的不行。
Jev: Making more doors that suck
Jev：制造更多烂门
The Internet has been abuzz about Jev, an AI model developed by TypeSafe AI, which returns typed values with probability estimates. The important things about Jev are, as far as I can tell:
互联网上关于 Jev 的讨论热火朝天。Jev 是由 TypeSafe AI 开发的一个 AI 模型，它返回带有概率估计的类型化值。据我所知，Jev 的重要特点包括：
- it is fast and cheap,
- you can build on it quickly,
- it is fast, and
- it is cheap.
- 它速度快且成本低，
- 你可以快速在其基础上构建，
- 它速度快，并且
- 它成本低。
I'm not particularly good at understanding what technology will get adopted.
我不太擅长理解哪些技术会被采用。
I still don't understand² Slack.
我仍然不理解² Slack。
But this product is even more confusing to me. This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part. To know if Jev is working, you have to build evals and a ground-truth pipeline. If you have evals and a ground-truth pipeline, you're already most of the way to fine-tuning your own solution.
但这个产品更让我困惑。这可不是“获取一个 FTP 账户，用 curlftpfs 在本地挂载它，然后在挂载的文件系统上使用 SVN 或 CVS”——你仍然需要完成最困难的部分。要判断 Jev 是否有效，你必须构建评估体系（evals）和真实标签管道（ground-truth pipeline）。如果你已经有了评估体系和真实标签管道，那你已经完成了微调自己解决方案的大部分工作。
Wait.
等等。
Do you still have to do the hard part?
难道你还需要做那些困难的部分吗？
Maybe my problem is expecting products to work.
也许我的问题在于期望产品能正常工作。
Nobody buying this is running evals. They're just handing opaque questions to Jev and getting opaque responses. Charitably, this allows them to check the "AI-powered" box and ship before Friday, and when this breaks downstream logic, they can always shrug and say "well, AI makes mistakes."
购买这款产品的人并没有运行评估。他们只是把不透明的输入丢给 Jev，然后得到不透明的输出。善意地解读，这让他们可以勾选“AI 驱动”的选项并在周五之前发货；当下游逻辑因此出错时，他们耸耸肩说“好吧，AI 会犯错”就行了。
Error budgets? Failure modes? Test sets? All of those can be handled later. The user can discover the failure rate! You've already shipped!
错误预算？故障模式？测试集？所有这些都可以稍后处理。用户自己去发现失败率吧！你已经发货了！
False Confidence
虚假的信心
"Oh," the strawman responding to my post responds, "you haven't considered the fact that Jev gives you confidence scores!"
"哦，"那个针对我帖子进行反驳的稻草人回应道，"你没考虑到 Jev 会给你置信度分数这一事实！"
What are you going to do with those?
你要拿这些分数做什么？
For you to do something reasonable with confidence scores you need to have both an understanding of the calibration of those confidence scores and also a model for the costs of the uncertainty.
要想对置信度分数做出合理的处理，你需要同时了解这些置信度分数的校准情况，以及不确定性带来的成本模型。
On the calibration side: Jev's topline ad copy is mostly about how well they score on various benchmarks, but not about how calibrated their confidence scores are. There's a cookbook about using confidence scores to go up a tree of classification but that's fundamentally not about how good the confidence scores are.
在校准方面：Jev 的主要广告文案大多关注他们在各种基准测试中的得分，而不是其置信度分数的校准程度。有一本关于使用置信度分数进行分类树攀升的指南，但这从根本上与评估置信度分数的好坏无关。
On the modeling side: nobody wants to think about this. It always ends up being "Ehh, let's just accept any response over a threshold. 0.9 sounds about right? What's for lunch?" Their documentation itself makes up a 0.5 threshold for "do nothing" and a 0.9 threshold for "do high-risk actions" [archive] with an aside info box stating "the correct threshold values depend on your domain and the performance of the model for your use case."
在建模方面：没人愿意思考这个问题。最终总是变成“唉，我们就接受超过某个阈值的任何响应吧。0.9 听起来差不多？午饭吃什么？”他们的文档本身设定了 0.5 作为“什么都不做”的阈值，以及 0.9 作为“执行高风险操作”的阈值 [archive]，并在一个侧边信息框中注明“正确的阈值取决于你的领域以及模型在你用例中的表现。”
At best, people use confidence scores in a cargo cult manner. At worst, people use them as an excuse for why the API call failed. The model was only 73% confident! That means my error budget is 27%!
最好的情况是，人们以盲目崇拜的方式使用置信度分数。最坏的情况是，人们将其作为 API 调用失败的借口。“模型只有 73% 的置信度！这意味着我的错误预算有 27%！”
Accountability
问责制
When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. My DNS is broken. Somebody shipped some slop that has JavaScript syntax errors along only a certain path. A handler threw that wasn't expected to throw. I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.
当网站上的按钮出问题时，我有一个关于应该发生什么的模型。某处的契约被破坏了。我的 DNS 坏了。有人发布了一些带有 JavaScript 语法错误的代码，且仅在某些路径下触发。某个本不该抛出异常的处理器抛出了异常。我可能无法访问调试 HTTP 状态码 500 的信息，但我期望有人负责理解为什么端点会返回 500。所有权定义明确，尽管不透明³。
For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration. It seems like not much of a loss to remove the possibility of following a failure to a concrete cause. Sometimes things just suck.
然而，对于许多用户来说，实际体验大致只是“愚蠢的东西真烂”。软件已经显得反复无常；更多的失败只会增加挫败感的频率。移除追踪故障到具体原因的可能性似乎损失不大。有时事情就是很烂。
This leads to a normalization of inexplicability.
这导致了对不可解释性的常态化。
My fear is not that more things will fail when things are accelerated by LLM-driven development. They will. They have. Such is part of the price of building things in a novel manner.
我的担忧并不是说在 LLM 驱动的加速开发下会有更多东西会失败。它们确实会。它们已经发生了。这是以新颖方式构建事物的一部分代价。
My fear is that "sometimes it just sucks" is going to be more and more the accepted endpoint of investigations. This is sad because LLM-accelerated development can indeed help us solve some of these issues. There are plenty of automated QA workflows that aren't written because of lack of engineering time. The very eval that would get you most of the way to replacing (or even justifying the use of) Jev can be a few prompts away.
我的担忧是，“有时它就是很烂”将越来越成为调查的最终结论。这令人悲伤，因为 LLM 加速的开发确实可以帮助我们解决其中一些问题。由于缺乏工程时间，有许多自动化 QA 工作流没有被编写。那些能让你大部分完成替换（或至少证明使用）Jev 的评估，可能只需几个提示词就能实现。
The tragedy of software engineering today is that we are actively engineering systems where neither the user nor the builder seems to have any interest in checking whether or not there's a body behind the door.
当今软件工程领域的悲剧在于，我们正积极地构建这样的系统：用户和建造者似乎都没有兴趣去检查门后是否有人。
We just shrug and conclude: stupid thing sucks.
我们只是耸耸肩，得出结论：这破玩意儿真烂。
¹This reminds me of a saying that I find similarly hilarious: "sometimes you get the elevator, sometimes you get the shaft." This is also not a reasonable model of elevators!!
¹这让我想起一句我觉得同样好笑的话：“有时候你坐上了电梯，有时候你只拿到了井道。”这也不是对电梯的合理描述！！
²The lock-in network effect makes sense to me but I'm still bewildered as to how people standardized on a product that does not even reliably deliver messages. I have seen messages dropped on free, paid, and enterprise instances that only show up weeks later.
²锁定网络效应对我来说是合理的，但我仍然困惑于人们是如何标准化一款连消息都无法可靠投递的产品。我见过在免费、付费和企业版实例中丢失的消息，几周后才出现。
³Well, maybe "well-defined" is optimistic. After Bill Gates famously failed to download Movie Maker, everybody agreed that it was presumably somebody's problem, just not necessarily theirs. Ideally we can get even this level of accountability without the customer being Bill Gates.
³嗯，也许“定义明确”还算是乐观的说法。在比尔·盖茨 famously 未能下载 Movie Maker 之后，每个人都认为这大概是某人的问题，只不过不一定是他们自己的问题。理想情况下，我们可以在客户不是比尔·盖茨的情况下，也能达到这种问责水平。
Posted by patrickxia at 8:24 AM
由 patrickxia 发布于上午 8:24
Email ThisBlogThis!Share to XShare to FacebookShare to Pinterest
电子邮件此博客分享至 X分享至 Facebook分享至 Pinterest
No comments:
没有评论：
Post a Comment
发表评论
Older Post Home
较早的文章 主页
Subscribe to: Post Comments (Atom)
订阅：发布评论 (Atom)
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力