JetBrains论文：编程Agent基准测试应模拟真实用户任务流
New JetBrains Research paper argues that coding-agent benchmarks should copy how…
做Agent评测的同学必看，这篇论文用真实数据戳破了现有基准的泡沫，SWE-TaskFlow的多轮拆解思路值得直接参考用于优化你的评估体系。
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages.
新的 JetBrains Research 论文认为，编码智能体基准测试应模仿其真实用户的实际工作方式，因为真实会话会在多条消息中混合并切换任务类型。
Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks.
使用编码智能体的真实开发者会提问、规划、审查并在会话中途切换任务，但像 SWE-Bench Pro 这样的基准测试只给智能体 1 个详细问题并评估 1 个最终补丁。在该基准测试中，新功能和错误修复占任务的 93.8%。
JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits.
JetBrains 研究人员分析了 4,782 个真实的智能体会话，其中功能和新功能的请求占比不到一半。在较长的会话中，用户不断在提问、规划、审查和编辑之间切换。
They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate.
他们构建了 SWE-TaskFlow，将每个基准测试问题拆分为多个回合，并添加可检查的代码相关问题。在 700 个 SWE-Bench Pro 任务上，拆分使智能体成本大致翻倍，成功率没有稳定变化。
If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it.
如果你要评估编码智能体，首先测量你自己的用户询问的内容，然后调整基准测试的对话以与之匹配。
– arxiv. org/abs/2610.09633
– arxiv.org/abs/2610.09633
Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"
标题："Coding-Agent Benchmarks Should Match Their Users' Task Flows"
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力