SaaStr AI Agent误删数据复盘:5条生产环境安全规范
Two AI Agents Wiped Critical Data at SaaStr. Both Times, a Copy Nobody Planned Is What Saved Us. It Will Happen To You, Too. Plan For It.
直接给出AI Agent在生产环境落地的5条具体操作规范,含权限分级、验证流程与监控机制,创业者可直接照做。
We run SaaStr with three humans and 20+ AI agents in production. In the last few weeks, two different agents destroyed something we can’t run the business without. Both times we got it back within an hour, and both times the copy we restored from was one we had never set up as a backup.
我们在生产环境中运行 SaaStr,由三名人类员工和 20 多个 AI 智能体共同协作。在最近几周里,两个不同的智能体分别破坏了业务无法运行的关键部分。两次我们都在一小时内恢复了数据,而且恢复所用的副本都是我们从未设置为备份的。
Every team running agents in production is going to have deletions like these. When yours happens, the question is where the clean copy lives. In both of our cases, we got lucky.
每个在生产环境中运行智能体的团队都会遇到此类删除事故。当你的团队遭遇这种情况时,关键问题在于干净副本存储在哪里。在我们的两个案例中,我们都侥幸脱险。
A third party agent deleted all our Salesforce records literally minutes after we deployed it
一个第三方智能体在我们部署它后几分钟内,彻底删除了我们所有的 Salesforce 记录。
Luckily, Replit had stored backups
幸运的是,Replit 保存了备份。
Agents are still … unpredictable https://t.co/ixWuAf2nVz
智能体仍然……不可预测 https://t.co/ixWuAf2nVz
— Jason ✨👾SaaStr.Ai✨ Lemkin (@jasonlk) October 17, 2026
— Jason ✨👾SaaStr.Ai✨ Lemkin (@jasonlk) 2026年10月17日
Agent #1 Deleted Almost Every Opportunity in Our Salesforce
智能体 #1 几乎删除了我们 Salesforce 中的所有商机
A few weeks ago we connected a well-regarded agent from a well-known vendor to our Salesforce, mostly as a favor. Shortly after, it deleted nearly all of our opportunities. A few records survived. Most of the pipeline didn’t.
几周前,我们将一家知名厂商广受好评的智能体接入我们的 Salesforce,主要是出于人情帮忙。不久之后,它几乎删除了我们所有的商机。只有少数记录幸存下来,大部分销售管线都丢失了。
Every Friday morning Amelia, our Chief AI Officer, has a one-on-one with David, and they clean up the pipeline together. They had just finished when the agent ran. Amelia was in the office, gasped out loud, and when I asked what was wrong she said an agent had just wiped our data.
每周五上午,我们的首席 AI 官 Amelia 会与 David 进行一对一会议,他们一起清理销售管线。就在他们刚完成清理工作时,智能体开始运行。Amelia 当时在办公室,大声惊呼起来。当我询问发生了什么事时,她说有一个智能体刚刚擦除了我们的数据。
The vendor’s agent had no rollback. Restoring by hand would have meant a lot of manual work, and the deals Amelia and David had just updated would have been the hardest to rebuild.
该厂商的智能体没有回滚功能。手动恢复将意味着大量手工劳动,而 Amelia 和 David 刚刚更新的交易将是重建中最困难的部分。
We’re keeping the vendor nameless. The product is widely used, and any agent with write access to a CRM can do this.
我们隐去该厂商的名称。该产品被广泛使用,任何拥有 CRM 写入权限的智能体都能做到这一点。
10K Had Been Keeping Its Own Copy of Our Pipeline
10K 一直保留着我们销售管线的独立副本
10K is our AI VP of Revenue. It runs our Salesforce headless through the API, and we’ve built it on Replit over the last year. When Amelia told 10K what happened, it said it had a snapshot of the full Salesforce pipeline from about 11am that same Friday. It warned her that anything changed after 11am wouldn’t be in it, and offered to restore once she confirmed.
10K 是我们的 AI 收入副总裁。它通过 API 无头方式运行我们的 Salesforce,过去一年来我们一直在 Replit 上构建它。当 Amelia 告诉 10K 发生了什么时,它表示自己在当天周五上午 11 点左右有一个完整的 Salesforce 销售管线快照。它提醒 Amelia,11 点之后更改的任何内容都不会包含在内,并在她确认后提供恢复服务。
She confirmed, and 10K restored it.
她确认了,10K 随即完成了恢复。
Neither of us knew 10K had that snapshot until we needed it. Kody, the engineer at Replit who has worked with us since early on, later told us that every agent on Replit keeps a daily snapshot outside its normal checkpoints. We hadn’t designed any of this. The copy that saved our pipeline sat in a system that had nothing to do with the incident.
直到需要时,我们都不知道 10K 拥有那个快照。从早期就与我们合作的 Replit 工程师 Kody 后来告诉我们,Replit 上的每个智能体都会在正常检查点之外保留每日快照。这一切都不是我们设计的。拯救我们销售管线的副本存在于一个与此次事件毫无关联的系统中。
Astra and Another Agent Replaced a Core Engine of SaaStr Connect With “DO IT”
Astra 和另一个智能体用“DO IT”替换了 SaaStr Connect 的核心引擎
I wrote up the second incident in full here. Astra 6, the model driving a Connect build, replaced ceoMatchingEmailService.ts, one of the two engines Connect runs on, with the five-byte string “DO IT.” It did it twice, 28 minutes apart. Both times it reported the empty file as a blocker it had found, and added “I did not make that edit.”
我在此完整记录了第二次事件。Astra 6(驱动 Connect 构建的模型)用五字节字符串“DO IT”替换了 ceoMatchingEmailService.ts,这是 Connect 运行的两个引擎之一。它这样做了两次,间隔 28 分钟。每次它都将空文件报告为它发现的阻塞项,并补充说“我没有做这个编辑。”
Connect stayed up because the production server still had the good code loaded in memory. Only the working copy on disk was broken. A restart or deploy at the wrong moment would have taken Connect down. We restored the file from version history in minutes. Customers never noticed because a running process happened to hold the last good version.
Connect 保持运行是因为生产服务器上内存中仍加载着正确的代码。只有磁盘上的工作副本被破坏了。如果在错误时刻重启或部署,会导致 Connect 宕机。我们在几分钟内从版本历史中恢复了该文件。客户从未察觉,因为正在运行的进程恰好持有最后一个正确版本。
Astra 6 Replaced a Core Engine of SaaStr Connect With Two Words, “DO IT,” Twice in Under an Hour. Then It Said “I Did Not Make That Edit.”
Astra 6 在一小时内两次用两个词“DO IT”替换了 SaaStr Connect 的核心引擎。然后它声称“我没有做那个编辑。”
In Both Cases, the Clean Copy Sat Where the Agent Couldn’t Write
在这两种情况下,干净的副本都位于代理无法写入的位置
The agent doing the damage had write access to the thing it destroyed and no access to the copy that saved us. 10K’s snapshot was outside the vendor agent’s reach. The server’s memory was outside the working copy Astra 6 was editing.
造成破坏的代理拥有对其所破坏对象的写入权限,却无法访问保护我们的副本。10K 的快照超出了供应商代理的访问范围。服务器内存则超出了 Astra 6 正在编辑的工作副本的范围。
A Third Agent Caught a Failure 10K Didn’t Report
第三个代理捕获了 10K 未报告的故障
In the same few weeks, Amelia connected Muse, Meta’s personal agent, to 10K through a shared endpoint so 10K could send it a morning GTM brief. A few days in, Muse flagged that one of 10K’s 20+ sync connections was failing and that a critical workflow depended on it. 10K hadn’t told Amelia directly.
在同一两周内,Amelia 将 Meta 的个人代理 Muse 通过共享端点连接到 10K,以便 10K 向其发送每日 GTM 简报。几天后,Muse 标记出 10K 的 20 多个同步连接中的一个出现故障,且一个关键工作流依赖于该连接。10K 并未直接向 Amelia 报告此事。
Astra 6 reported its own damage as something it had found. 10K didn’t mention its own broken sync. Both times, the accurate signal came from something outside the system that failed.
Astra 6 将其自身造成的破坏报告为其发现的问题。10K 未提及自身的同步故障。两次准确的信号都来自系统外部的失败组件。
What We Changed After This
此后我们做出的更改
- For every system an agent can write to, we now name where the independent copy lives. For Salesforce, that’s 10K’s snapshots plus Replit’s daily snapshot. For Connect, it’s committed version history, and production only restarts from a known-good commit. If we can’t name the independent copy, the agent doesn’t get write access yet.
- A new vendor’s agent starts read-only on our systems of record. The agent that wiped our pipeline had write access on day one because we connected it as a favor. Write access now comes after the agent has run read-only long enough for us to see what it would have done.
- We run a test restore before turning on any new agent that can touch a critical system. 10K’s restore worked the first time because Amelia could check the snapshot’s timestamp and knew what would be missing.
- We check the diff, the provider log, or the record count instead of the agent’s report. “I did not make that edit” and a quiet failed sync are both things an agent told us, or didn’t tell us. The system itself is what we check.
- Every critical agent gets a second system checking its outputs. Muse catching 10K’s failed sync was an accident. We’re now setting that up on purpose for the agents we can’t run without.
- 对于每个代理可以写入的系统,我们现在明确独立副本的存储位置。对于 Salesforce,那是 10K 的快照加上 Replit 的每日快照。对于 Connect,它是已提交的版本历史,且生产环境仅从已知良好的提交进行重启。如果我们无法指明独立副本的位置,代理就不会获得写入权限。
- 新供应商的代理在我们的记录系统中以只读模式启动。抹除我们管道的代理之所以在第一天就拥有写入权限,是因为我们出于帮忙而将其接入。现在,只有在代理以只读模式运行足够长时间、让我们看到其潜在行为后,才会授予写入权限。
- 在启用任何可能触及关键系统的新代理之前,我们会执行测试恢复。10K 的恢复首次就成功了,因为 Amelia 能够检查快照的时间戳,并清楚知道哪些数据会缺失。
- 我们检查差异、提供程序日志或记录计数,而不是依赖代理的报告。“我没有进行该编辑”以及静默失败的同步都是代理告诉我们的,或者没告诉我们的。我们检查的是系统本身。
- 每个关键代理都有一个独立的第二系统来检查其输出。Muse 捕获到 10K 的失败同步纯属意外。我们现在正有意为那些不可或缺的代理设置此机制。
What the Two Incidents Cost Us: Under an Hour and One Afternoon
两起事故的成本:不到一小时与一个下午
The Salesforce wipe cost us under an hour, plus whatever Amelia and David changed after 11am. Astra 6 cost us an afternoon. If 10K hadn’t kept that snapshot, or if Connect’s server had restarted between the two “DO IT” writes, we’d have spent days rebuilding the pipeline or had Connect down for customers.
Salesforce 数据清除事件导致我们损失不到一小时,加上 Amelia 和 David 在上午 11 点之后所做的任何更改。Astra 6 事件导致我们损失了一个下午。如果 10K 没有保留该快照,或者如果在两次“DO IT”写入之间 Connect 服务器重启了,我们将不得不花费数天时间重建管道,或者导致 Connect 对客户不可用。
We still build with agents eight to ten hours a day, and it’s still far faster and cheaper than the old way. The difference now is that we know where every backup lives before an agent gets write access.
我们每天仍使用代理工作八到十小时,且速度仍然比旧方式快得多,成本也更低。现在的区别在于,我们在授予代理写入权限之前,就已经知道每个备份的位置。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力