自建 Coding Agent 评测：35B 模型胜过 120B
I built a custom benchmark for my coding agent: the 35B model beat the 120B one, and pay-per-token beat renting GPUs
Agent 开发者必读的实战复盘，给出了具体的 Benchmark 设计思路、工具链选择及成本对比数据，直接指导如何科学选型。
I'm currently writing a coding agent from scratch, so I obviously need an open model I can use. I'm not choosing based on leaderboards or specs, though. Instead, I made a small custom benchmark.
我目前正在从头编写一个编码代理（coding agent），因此显然需要一个可用的开源模型。不过，我并不是根据排行榜或规格来选择，而是制作了一个小型自定义基准测试。
Here is how I designed my benchmark:
以下是我设计基准测试的方法：
It contains 19 tasks of varying difficulty: 7 "easy", 6 "medium", and 6 "hard". I modeled their structure after the Terminal-Bench. Each has an instruction, seed repository pinned to a particular commit, and a hidden verifier that checks the final state (0: pass; 1: fail).
它包含 19 个不同难度的任务：7 个“简单”，6 个“中等”，以及 6 个“困难”。我参考 Terminal-Bench 的结构对它们进行了建模。每个任务都有一个指令、一个固定到特定提交的种子仓库，以及一个检查最终状态的隐藏验证器（0：通过；1：失败）。
The tricky part is making sure the sandbox is isolated, so that each trial runs the agent as a separate process and writes its work as a git branch. The hidden tests are only copied in after the agent runs, so there's no opportunity to leak any solutions.
棘手的部分是确保沙箱隔离，以便每次试验都将代理作为独立进程运行，并将其工作结果写入 git 分支。隐藏测试仅在代理运行后才会被复制进来，因此没有机会泄露任何解决方案。
I use code verifiers, not LLM/human judges. For example, one task fails if the agent changes any file other than the designated target, or changes more than 8 lines.
我使用代码验证器，而不是 LLM/人工评判员。例如，如果一个任务中代理修改了除指定目标之外的任何文件，或者修改超过 8 行，则该任务失败。
The benchmark is small, but I've tried to cover a wide range of scenarios and edge cases. The trials were expensive to run, so I kept the number of tasks to a minimum. However, I've tried to ensure that each one addressed a different scenario or edge case.
这个基准测试规模虽小，但我试图覆盖广泛的场景和边缘情况。由于试验运行成本高昂，我将任务数量保持在最低限度。然而，我尽力确保每个任务都针对不同的场景或边缘情况。
I used Opik to track my agent's traces, version my dataset, and save each run as an experiment. That means if I want to compare two models, I only have to compare two experiments.
我使用 Opik 来跟踪我的代理的轨迹、对我的数据集进行版本控制，并将每次运行保存为实验。这意味着如果我想比较两个模型，我只需要比较两个实验。
Result 1: a 3-4x bigger model loses (both MoE models)
结果 1：大 3-4 倍的模型输了（两个 MoE 模型）
- Qwen3.6-35B: 95% pass@1
- GPT-OSS-120B: 53% pass@1
- Qwen3.6-35B：pass@1 通过率 95%
- GPT-OSS-120B：pass@1 通过率 53%
Both were scored on the same 19 tasks, using the same harness, and with one trial each.
两者都在相同的 19 个任务上评分，使用相同的测试框架，且各进行一次试验。
But the test isn't fair, and that's the point. I optimized the harness to work with Qwen during development. GPT-OSS-120B is untuned in this test and used as a baseline. The score difference illustrates how much harness tuning can affect performance. Maybe even more than the number of parameters. It's also why I'm hesitant to switch from Claude Code to Pi as my default tool for everyday use. On a new model, untuned for the harness, performance can suffer.
但这项测试并不公平，而这正是重点。我在开发过程中优化了测试框架以适配 Qwen。GPT-OSS-120B 在此测试中未经调整，仅作为基线使用。分数的差异说明了测试框架调优对性能的影响有多大，甚至可能比参数量影响更大。这也是我为何犹豫是否将 Pi 作为日常使用的默认工具来取代 Claude Code。对于未经过测试框架调优的新模型，性能可能会受到影响。
Result 2: cost
结果 2：成本
Same model, same workload: Qwen3.6-35B
同一模型，同一工作量：Qwen3.6-35B
Two billing models: pay-per-token vs. pay-per-GPU hour
两种计费模式：按 token 付费 vs. 按 GPU 小时付费
- pay-per-token(via OpenRouter): cost ~$0.13 for 1.2 million tokens
- pay-per-GPU hour (via Modal): cost ~$0.46 for running the same thing
- 按 token 付费（通过 OpenRouter）：120 万个 token 的成本约为 $0.13
- 按 GPU 小时付费（通过 Modal）：运行相同内容的成本约为 $0.46
The GPU is mostly idle, running one trial at a time, while you still pay for the whole GPU, which means rensting/serverless modes are cost-effective only if you know how to properly batch all your workload to keep your GPU at 100% usage.
GPU 大部分时间处于空闲状态，一次只运行一个试验，但你仍需为整个 GPU 付费，这意味着只有在知道如何正确批量处理所有工作负载以保持 GPU 100% 利用率的情况下，租用/无服务器模式才具有成本效益。
The takeaway is that leaderboards, parameter counts, and generic pricing charts don't tell you how a model will perform or what it will cost in your harness. Build a small benchmark with your own tasks.
关键要点是：排行榜、参数量以及通用的定价表无法告诉你模型在你的环境中表现如何，也无法告诉你它的花费。请用你自己的任务构建一个小型基准测试。
So how do you choose an open (or closed) model when building your harness/AI app? Vibes don't count
那么，在构建你的环境/AI 应用时，该如何选择开源（或闭源）模型？靠感觉是不行的
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力