跳到主内容
精选85Rohan Paul论文研究

给编码会话每个提示附加技能文件通常是净损失

Attaching a skill file to every prompt in a coding session is usually a net loss…

原文
推荐理由

做 Agent 工程的同学必看,这项研究用严格对照实验证明技能文件注入在多数场景是负收益,还给出了按后端路由、早期任务暂缓注入的实操建议,值得拿你的链路验证一遍。

Attaching a skill file to every prompt in a coding session is usually a net loss.

在编码会话中,将技能文件附加到每个提示通常是一种净损失。

A skill file hurts most on the easy early tasks your agent already handles correctly.

技能文件在您的代理已经能正确处理的简单早期任务上伤害最大。

Whatever gain exists belongs to a particular skill, project, and model combination, not to the skill itself.

任何存在的收益都属于特定的技能、项目和模型组合,而非技能本身。

So treat injection as a routing decision you measure per backend, and hold the file back on early tasks until error rates rise.

因此,将注入视为按后端测量的路由决策,并在错误率上升前,在早期任务中保留文件。

Skills can help, but this paper finds the helpful cases are a minority that a single published ranking cannot pick out in advance, since per-pair effects barely transfer across models.

技能可能有帮助,但本文发现,有帮助的情况是少数,单一发布的排名无法提前识别,因为每对效果在模型间几乎不转移。

WebDev-Skills-Bench holds the harness, project, and decoding fixed and varies only the injected file: no skill, the stack-matched skill, an equally long irrelevant one, and slice-removed variants.

WebDev-Skills-Bench 固定了测试框架、项目和解码,仅变化注入的文件:无技能、堆栈匹配技能、同等长度的无关技能,以及移除片段变体。

Across all 4 models, injection lowers mean Pass@2 on stack-matched pairs by 1.3 to 4.2 points while raising token cost by at least 72%.

在所有4个模型中,注入将堆栈匹配对的平均 Pass@2 降低了1.3至4.2个百分点,同时将令牌成本提高了至少72%。

The losses concentrate on easy early tasks, running from 4.0 to 10.7 points there, while harder tasks show no consistent effect.

损失集中在简单的早期任务上,在那里下降了4.0至10.7个百分点,而较难的任务没有一致的效果。

That looks like retry lock-in: the skill fixes structural choices the model would otherwise vary on a second attempt, so a recoverable first mistake becomes a chain-terminating failure.

这看起来像重试锁定:技能固定了模型在第二次尝试中可能变化的结构选择,因此可恢复的首次错误变成了终止链的失败。

– arxiv. org/abs/2608.23067

– arxiv.org/abs/2608.23067

Title: "Signal or Noise? A Benchmark Study of Agent Skills in Web Development"

标题:“信号还是噪音?Web开发中代理技能的基准研究”

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

另一事件,读法相近