Agent 化工作流四教训:收件箱优于任务列表,路由与自愈是关键
Four Lessons From Three Months Inside An Agentic Harness
给正在用 AI agent 自动化工作流的创业者:四条可落地的工程经验,含具体参数(错误率、耗时、回滚次数)和取舍,直接能抄。
My agent’s error rate went from zero to thirty-four percent. That was the moment three months of running an agentic harness turned into four lessons.
我的代理的错误率从零上升到了百分之三十四。那一刻,三个月的代理框架运行变成了四条经验教训。
First, an inbox beats a task list. A list beats a stream for one reason : you can see everything at once. So I moved my work queue from Asana into Gmail & built an agent to work through it. While it works, threads sit archived under a processing label. When it finishes, or needs me, the thread returns to the inbox. The agent never permanently archives mail. Only I do. The inbox became the “waiting on you” surface. Twenty-four threads used to rot silently under an error label. Now they surface with a one-line reason.
首先,收件箱优于任务列表。列表优于流的原因只有一个:你可以一眼看到所有内容。所以我把工作队列从 Asana 移到了 Gmail,并构建了一个代理来处理它。当它工作时,线程会归档在处理标签下。当它完成或需要我时,线程会返回收件箱。代理永远不会永久归档邮件。只有我会这样做。收件箱变成了“等待你”的界面。二十四条线程曾经在错误标签下默默腐烂。现在它们会带着一行原因浮出水面。
Second, routing is essential. One model became a router : a fast local worker, a local reasoner, & cloud fallback tiers. Local runs four to six minutes per task. Cloud runs about thirty-nine seconds. Local is cheap. Cloud is reliable. The router’s job is to know which the task can afford, & to record the real reason a task went to cloud instead of faking a local failure to justify the promotion.
其次,路由至关重要。一个模型变成了路由器:一个快速的本地工作器、一个本地推理器,以及云回退层级。本地运行每个任务需要四到六分钟。云运行大约需要三十九秒。本地便宜。云可靠。路由器的工作是知道任务能承受哪种,并记录任务转到云端的真实原因,而不是伪造本地失败来证明升级的合理性。
Third, self-healing works, but it is brittle & it surfaces more errors before it surfaces fewer. For six weeks my agent’s error rate was zero. Then it rose to 15 percent, then 34 percent. It did not rise because the system got worse. It rose because the system stopped hiding failures. Before the fix, twenty-one threads sat stranded in error, the oldest for eighteen days, & a hundred & twelve recoveries happened by hand-written SQL. After the fix, failures surface immediately, redrive four times with jittered backoff, then dead-letter. Today zero threads sit in error. But surfacing an error is not the same as fixing it. The system now reverts its own bad deploys, 65 so far, 42 of them because unit tests failed, & since late July it catches broken code before deploy instead of after. Broken main never ships. Still, auth failures need a human to refresh a token. Memory limits need a human. A model that narrates instead of working needs a human to notice.
第三,自我修复有效,但它很脆弱,而且在减少错误之前会先暴露更多错误。六周内,我的代理的错误率是零。然后上升到百分之十五,再上升到百分之三十四。它上升不是因为系统变差了。它上升是因为系统不再隐藏失败。在修复之前,二十一条线程被困在错误中,最长的有十八天,一百一十二次恢复是通过手写 SQL 完成的。修复后,失败立即浮出水面,用抖动退避重试四次,然后进入死信队列。今天,零条线程处于错误状态。但暴露错误不等于修复错误。系统现在会回滚自己的不良部署,到目前为止有六十五次,其中四十二次是因为单元测试失败,而且自七月底以来,它在部署前而不是部署后捕获损坏的代码。损坏的主分支永远不会发布。尽管如此,认证失败需要人工刷新令牌。内存限制需要人工。一个叙述而不是工作的模型需要人工注意。
Fourth, you need a quarterback even in a no-huddle offense. The work stopped being one monolithic prompt & became a graph of contracted nodes : the model produces a narrow JSON intent, an action, a domain, a reason, a confidence score. Deterministic code performs the write. An independent node re-reads to confirm. Maker & verifier are different nodes ; never ask the agent to grade its own homework. That separation lets plays run without a huddle. It does not remove the quarterback. Someone still has to call the plays that matter : which task goes to cloud, which failure gets a human, which deploy gets reverted. Three months in, that quarterback is still me, sitting one layer above the code that used to be the whole team.
第四,即便采用无聚商进攻,你仍需要一名四分卫。工作不再是一个单一的提示词,而是变成了一个由契约节点组成的图:模型产生一个窄化的JSON意图、一个动作、一个领域、一个原因、一个置信度分数。确定性代码执行写入操作。一个独立的节点重新读取以确认。制造者与验证者是不同的节点;绝不要让智能体给自己的作业打分。这种分离使得比赛无需聚商也能进行。但这并未移除四分卫。仍须有人来叫关键战术:哪个任务交给云端,哪个失败需要人工介入,哪个部署需要回滚。三个月过去了,那个四分卫依然是我,坐在一层之上,而那层代码曾经是整个团队。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力