Attest 1.1:零模型调用回放Agent运行并断言行为契约
Attest 1.1: capture an agent run, replay it in CI with zero model calls, and assert on what the agent did (open source, author here)
Agent 测试是工程落地痛点,这个方案用录制回放替代昂贵的模型调用做 CI 断言,思路清晰且给出了具体契约写法,值得做 Agent 的同学参考其工程实践。
I maintain Attest, an Apache-2.0 testing framework for LLM agents in Python and TypeScript. No paid tier, no hosted service, no telemetry. 1.1 went out on November 10.
我维护 Attest,这是一个用于 LLM 智能体的 Apache-2.0 测试框架,使用 Python 和 TypeScript 编写。无付费层级,无托管服务,无遥测数据。1.1 版本于 11 月 10 日发布。
The failure it's built around. Agent tests usually assert on the reply. The demo repo has a synthetic incident where that isn't enough: the payments API commits a refund, then times out; the agent retries without an idempotency key; the customer gets refunded twice. The reply is "Your refund of $84.00 has been processed." Every output assertion passes.
其围绕构建的核心失败案例。智能体测试通常对回复进行断言。演示仓库中有一个合成事件,证明仅靠回复断言是不够的:支付 API 提交了退款,然后超时;智能体在没有幂等键的情况下重试;客户被退款两次。回复是“您的 84.00 美元退款已处理。”每个输出断言都通过。
The workflow.
工作流程。
- attest capture --name inc-1042 -- <command> runs the agent behind a loopback proxy for OpenAI (Chat Completions, Responses) and Anthropic Messages, plus an OTLP receiver. It records the trace and every model exchange, redacts secrets, templates volatile values (ids, timestamps, emails), drafts contracts from what it saw and checks them against the run.
- A behavior contract states the rule that broke:
- expect(trace).tool_cardinality("issue_refund", per="order_id", at_most=1)
- An effect contract checks the end state of the simulated payments API: exactly one refund for that order.
- In PR CI the agent's current code runs while the proxy answers from the recording: 0 live model calls · $0.00. A changed prompt is a replay miss with a re-record command, not a silent mismatch.
- attest policy evaluate turns the report into a merge gate (0 pass, 1 block, 2 warn), and each failure is annotated on the failing test's line in the PR.
- attest mutate drops, duplicates and reorders tool calls in the recording to measure the suite. Output-only tests: 0 of 29 mutants caught. With contracts: 10 of 29.
- attest capture --name inc-1042 -- <command> 在 OpenAI(Chat Completions、Responses)和 Anthropic Messages 的环回代理以及 OTLP 接收器后面运行智能体。它记录跟踪记录和每次模型交互,脱敏机密信息,模板化易变值(ID、时间戳、电子邮件),根据所见内容起草契约并将其与运行结果进行检查。
- 行为契约声明了被打破的规则:
- expect(trace).tool_cardinality("issue_refund", per="order_id", at_most=1)
- 效果契约检查模拟支付 API 的最终状态:该订单恰好有一笔退款。
- 在 PR CI 中,智能体的当前代码运行,而代理从录制文件中响应:0 次实时模型调用 · $0.00。提示词变更会导致重放未命中并触发重新录制命令,而不是静默的不匹配。
- attest policy evaluate 将报告转换为合并门控(0 通过,1 阻止,2 警告),并且每个失败都会在 PR 中失败测试的行上进行注释。
- attest mutate 在录制中删除、重复和重新排序工具调用以测量套件。纯输出测试:29 个突变体中捕获 0 个。加上契约后:29 个突变体中捕获 10 个。
LLM judges (OpenAI, Anthropic, Gemini, any OpenAI-compatible endpoint) run after the deterministic layers. A judge outage or malformed reply is "not evaluated" with a reason, never a pass.
LLM 裁判(OpenAI、Anthropic、Gemini、任何 OpenAI 兼容端点)在确定性层之后运行。裁判宕机或格式错误的回复会被标记为“未评估”并附带原因,绝不会显示为通过。
What 1.1 fixed. An audit of 1.0 logged 73 findings, each with a reproduction or a code trace. Four made a failing assertion come back as passing: an engine notification on the stdio pipe shifted later verdicts by one, a timed-out response was read by the next test, concurrent Vitest tests swapped responses, and a pytest-xdist run wrote an empty report that the gate accepted. 1.1 fixes those, fails any expect chain that was built but never evaluated, and blocks a PR that deletes baseline assertions.
1.1 修复的内容。对 1.0 的审计记录了 73 个发现,每个都有复现步骤或代码跟踪。其中四个导致失败的断言变为通过:stdio 管道上的引擎通知使后续裁决偏移了一位,超时的响应被下一个测试读取,并发的 Vitest 测试交换了响应,以及 pytest-xdist 运行生成了空报告,而门控接受了该报告。1.1 修复了这些问题,使任何已构建但从未评估的 expect 链失败,并阻止删除基线断言的 PR。
How it was built. Most of the code was written by coding agents (Claude Code and Codex) executing milestone plans I wrote and reviewed. The audit above was part of that loop.
构建方式。大部分代码由编码智能体(Claude Code 和 Codex)编写,它们执行了我编写和审查的里程碑计划。上述审计是该循环的一部分。
What I'm looking for. It has no external users yet. If you run an agent that changes real systems (payments, tickets, infra) and have an incident you'd want a test to block, the design-partner form is below. Criticism of the approach is just as useful.
这正是我所寻找的。目前它还没有外部用户。如果你运行一个会改变真实系统(支付、工单、基础设施)的代理,并且发生了你希望测试能拦截的事件,请使用下方的设计合作伙伴表单。对该方法的批评同样具有价值。
- Repo: https://github.com/attest-framework/attest
- Web Page: https://attest-framework.github.io/
- Demo (no API key, about two minutes): https://github.com/attest-framework/refund-desk
- Design partners: https://github.com/attest-framework/attest/issues/new?template=design-partner.yml
- 代码库:https://github.com/attest-framework/attest
- 网页:https://attest-framework.github.io/
- 演示(无需 API 密钥,约两分钟):https://github.com/attest-framework/refund-desk
- 设计合作伙伴:https://github.com/attest-framework/attest/issues/new?template=design-partner.yml
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力