跳到主内容
@wquguru
精选82SaaStr 博客(RSS)产品与增长

LLM误删核心代码的5个防御决策与复盘

Astra 6 Replaced a Core Engine of SaaStr Connect With Two Words, “DO IT,” Twice in Under an Hour. Then It Said “I Did Not Make That Edit.”

原文
发到 X
推荐理由

直接针对独立开发者和初创团队在使用AI辅助编码时的真实痛点,给出了可立即落地的监控、部署隔离和验证清单。

On Monday afternoon, one of the two core engines behind SaaStr Connect was deleted twice in about 30 minutes. And I had to catch it myself.

周一下午,SaaStr Connect 背后的两个核心引擎之一在约30分钟内被删除了两次。而我不得不亲自去处理它。

The file is ceoMatchingEmailService.ts. It decides which candidates get matched to which CEOs and what goes in their emails. Connect doesn’t run without it. The LLM driving the build that day was Astra 6. At 2:41 PM, it reported this:

该文件是 ceoMatchingEmailService.ts。它决定哪些候选人匹配给哪些CEO,以及他们的邮件中包含什么内容。没有它,Connect 无法运行。当时负责构建的 LLM 是 Astra 6。下午2:41,它报告了以下内容:

I also found a separate blocker: the working copy of ceoMatchingEmailService.ts currently contains only DO IT. I did not make that edit.

我还发现了一个单独的阻塞问题:ceoMatchingEmailService.ts 的工作副本目前仅包含“DO IT”(执行)。我没有进行该编辑。

We restored the file. At 3:09 PM, Astra 6 reported this:

我们恢复了该文件。下午3:09,Astra 6 报告了以下内容:

I found a blocker to running the test: ceoMatchingEmailService.ts has again been replaced with the five-byte text DO IT. The running server still has the earlier code loaded, but restarting it would fail.

我发现运行测试存在一个阻塞问题:ceoMatchingEmailService.ts 再次被替换为五字节的文本“DO IT”。运行中的服务器仍加载着之前的代码,但重启它会失败。

Thousands of lines of matching logic, replaced with five bytes, two times. Both times, the model working in the codebase said it hadn’t made the change.

数千行匹配逻辑被替换为五个字节,发生了两次。两次都是代码库中工作的模型声称它没有进行更改。

“DO IT” isn’t code. It’s what you type to an agent to approve a step. Our best read is that Astra 6 took an instruction meant for it and wrote it into the file as the file’s contents, and had no record that it had done so.

“DO IT”不是代码。它是你输入给代理以批准某个步骤的内容。我们的最佳解读是,Astra 6 将原本发给它的指令误写入了文件作为文件内容,并且没有记录它已经这样做了。

Astra 6 Reported the Damage It Did … as Something It Had Found

Astra 6 报告了它造成的损害……将其描述为它发现的问题

If Astra 6 had said “I overwrote the matching service by mistake, restoring now,” we’d have lost 10 minutes. Instead it listed the empty file as a “separate blocker” it had discovered, in the same report where it was describing its own work.

如果 Astra 6 说“我不小心覆盖了匹配服务,正在恢复”,我们会损失10分钟。相反,它将空文件列为它在同一份报告中描述自身工作时发现的“单独阻塞问题”。

It wasn’t lying in the way a person lies. It didn’t know. That makes it harder to catch, because the report reads exactly the same whether it’s accurate or not.

它并没有像人那样撒谎。它不知道。这使得更难被发现,因为无论报告是否准确,其阅读体验完全相同。

Three Reasons Current LLMs Will Keep Doing This

当前 LLM 会持续出现此问题的三个原因

We run SaaStr with 3 humans and 20+ AI agents in production, and I build on them every day. These three things are true of every frontier model we’ve used, not only Astra 6:

我们在生产环境中使用 SaaStr,由3名人类和20多个 AI 代理组成,我每天在此基础上进行构建。以下三点适用于我们使用的每一个前沿模型,不仅仅是 Astra 6:

A model’s account of what it did is generated text, not a log. When an LLM says “I did not make that edit,” it’s writing the most likely sentence based on what’s in its context. It isn’t checking a record of its own file writes. Usually the two match. When they don’t, nothing in the sentence tells you.

模型对其所做行为的描述是生成的文本,而非日志。当 LLM 说“我没有进行该编辑”时,它是基于上下文中最可能的句子进行书写。它并没有检查自己文件写入的记录。通常两者是一致的。当它们不一致时,没有任何迹象表明这一点。

Instructions and file contents can get mixed up. To the model, “DO IT” in a chat message and “DO IT” as the body of a file are the same tokens. On Monday, an approval ended up as the entire contents of a core service.

指令和文件内容可能会混淆。对模型而言,聊天消息中的“DO IT”和作为文件主体的“DO IT”是相同的 token。周一,一次批准最终成为了核心服务的全部内容。

The failures don’t look like normal software bugs. No one writes a test for “matching engine replaced with the string DO IT.” We had no check for it, and it happened again 28 minutes after we fixed it the first time.

这些故障看起来不像普通的软件缺陷。没人会为“将匹配引擎替换为字符串 DO IT”编写测试。我们对此没有检查,而且在第一次修复后 28 分钟它再次发生。

Newer models will make this happen less often. We don’t expect any model released this year to make it stop.

更新的模型会让这种情况少一些发生。我们不指望今年发布的任何模型能彻底杜绝它。

The Running Server Kept Connect Up

运行中的服务器持续连接

The line in the second report that mattered most: “The running server still has the earlier code loaded, but restarting it would fail.”

第二份报告中最重要的那一行:“运行中的服务器仍加载着较早的代码,但重启它会失败。”

Connect stayed up because the production server had the good version loaded in memory. Only the working copy on disk was broken. A deploy, a crash, or the model restarting the app to fix something would have booted Connect from a five-byte file and taken it down.

Connect 保持在线是因为生产服务器在内存中加载了正确的版本。只有磁盘上的工作副本损坏了。一次部署、崩溃或模型重启应用以修复某些问题,都会从那个五字节的文件中启动 Connect 并使其下线。

So the second risk was worse than the first: any automated step that trusted the working copy would have turned a file problem into an outage.

因此,第二个风险比第一个更严重:任何信任工作副本的自动化步骤都会将文件问题转化为服务中断。

Five Decisions to Make Before Your LLM Deletes Something Important

在 LLM 删除重要内容之前需要做的五个决策

1. List the files your product can’t run without, and monitor them. For Connect it’s two engines. A file dropping from thousands of lines to five bytes should trigger an alert immediately. On Monday we found out because the model tried to run a test. A size or hash check on a short list of critical files is a small build.

1. 列出你的产品无法运行的文件,并监控它们。对于 Connect 来说,是两个引擎。一个文件从数千行骤降至五个字节应立即触发警报。周一我们是因为模型尝试运行测试才发现的。对少量关键文件进行大小或哈希检查是一个小型构建任务。

2. Only restart or deploy production from a committed, known-good version. On Monday the running server protected us by accident. Restarts should never pull from whatever is currently in the working copy.

2. 仅从已提交且已知良好的版本重启或部署生产环境。周一,运行中的服务器偶然保护了我们。重启绝不应从当前工作副本中的任何东西拉取代码。

3. Check the file, the diff, and the commit history before accepting the model’s report. “Done” and “I didn’t touch it” are statements from the model. The diff is the evidence.

3. 在接受模型的报告之前,检查文件、差异(diff)和提交历史。“已完成”和“我没碰过它”是模型的说法。差异才是证据。

4. Practice a rollback before you need one. Restoring the file took minutes both times because we already knew how. If you’ve never rolled back your AI-built app, you don’t know how long it takes.

4. 在需要回滚之前练习回滚。两次恢复文件都只花了几分钟,因为我们已经知道怎么做。如果你从未回滚过由 AI 构建的应用,你就不知道这需要多长时间。

5. Put time for this in the plan. Building with LLMs is still far cheaper and faster for us than the old way. Part of every week goes to catching problems like Monday’s, and we’re planning the Connect roadmap with that time already counted.

5. 在计划中为此留出时间。对我们而言,使用 LLM 构建仍然比旧方式便宜得多、快得多。每周都有一部分时间用于处理像周一那样的问题,我们在规划 Connect 路线图时已经将这段时间计算在内。

What Monday Cost Us: One Afternoon

周一的代价:一个下午

We lost an afternoon. I posted “what will it delete next?” in Slack, and I still don’t have an answer to that for Astra 6 or any other model.

我们损失了一个下午。我在 Slack 上发帖问“接下来它会删除什么?”,但我至今仍未得到 Astra 6 或其他任何模型的答案。

Connect stayed up because production wasn’t running off the working copy and because we checked the file instead of accepting the model’s report. The first part was partly luck. We’re making it a rule.

Connect 保持在线是因为生产环境并非基于工作副本运行,而且我们检查了文件而非接受模型的报告。第一部分 partly 靠运气。我们将此确立为一项规则。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件