Perplexity用提示引导自蒸馏训练计算机Agent
Perplexity Trains Its Computer Agent on Real Mistakes With Hint-Guided Self-Distillation
Agent训练常忽略失败样本，这篇详细拆解了如何用提示引导自蒸馏从真实错误中学习，流程完整且数据扎实，做Agent落地的同学值得参考。
Perplexity Research published a new post-training study. It trains a model inside Perplexity Computer on real user sessions, including failed ones. The method pairs rejection sampling fine-tuning with hint-guided self-distillation. In a live A/B test, tool-call failures fell from 2.24% to 1.77% between 2 trained checkpoints. Perplexity team reports this as a statistically significant 21.2% relative reduction.
Perplexity Research 发布了一项新的后训练研究。它在 Perplexity Computer 中基于真实用户会话（包括失败的会话）对模型进行训练。该方法将拒绝采样微调与提示引导的自蒸馏相结合。在一次实时的 A/B 测试中，工具调用失败率从两个训练检查点之间的 2.24% 降至 1.77%。Perplexity 团队报告称，这代表了具有统计显著性的 21.2% 相对降低。
Is it deployable? Not directly. Perplexity has not released the post-trained weights or training code. The model runs only as a model option inside Perplexity Computer. The base model, GLM 5.2, is openly available on Hugging Face.
它可以直接部署吗？不能。Perplexity 尚未发布经过后训练的权重或训练代码。该模型仅在 Perplexity Computer 内部作为模型选项运行。基础模型 GLM 5.2 已在 Hugging Face 上公开可用。
Why Outcome-Only Filtering Falls Short
为何仅基于结果的过滤存在不足
Standard rejection sampling fine-tuning (RFT) judges each session and imitates only the successful ones. A successful outcome does not mean every step was correct. An agent can recover from a bad tool call and still deliver the right answer. Imitating that full trajectory can reinforce the error. Discarding failed sessions also throws away clear evidence of avoidable mistakes.
标准的拒绝采样微调（RFT）会对每个会话进行评估，并仅模仿成功的会话。一个成功的结果并不意味着每一步都是正确的。智能体可以从错误的工具调用中恢复，并最终给出正确答案。模仿整个轨迹可能会强化错误。丢弃失败的会话也会丢失关于可避免错误的明确证据。
Imitate, Correct, or Keep as Context
模仿、纠正或保留为上下文
Perplexity team separates 2 decisions: which sessions hold behavior worth imitating, and which turns hold mistakes worth correcting.
Perplexity 团队区分了两个决策：哪些会话包含值得模仿的行为，以及哪些回合包含值得纠正的错误。
Each assistant turn gets 1 of 3 treatments:
每个助手回合都会接受以下三种处理之一：
- Imitate: non-error turns in successful sessions receive cross-entropy (CE) loss.
- Correct: error turns with a validated hint receive Kullback-Leibler (KL) divergence loss, in any session.
- Keep as context: remaining turns stay in the input but receive no loss.
- 模仿：成功会话中的非错误回合会计算交叉熵（CE）损失。
- 纠正：在任何会话中，带有已验证提示的错误回合会计算 KL 散度（KL divergence）损失。
- 保留为上下文：剩余的回合保留在输入中，但不计算任何损失。
Successful sessions can supply both imitation and correction targets. Unsuccessful sessions supply only correction targets.
成功会话可以同时提供模仿和纠正的目标。不成功会话仅提供纠正目标。
How a Hint Becomes a Training Signal
提示如何成为训练信号
A hint is a short corrective instruction grounded in information the model already had. In one example, a search call set recency_filter to ‘year.’ The schema allowed only ‘day,’ ‘week,’ or ‘month.’ The hint names the failed call, includes the validation error, and suggests an allowed value or omitting the optional field.
提示是一种简短的纠正指令，基于模型已有的信息。在一个例子中，搜索调用将 recency_filter 设置为 'year'。但模式定义只允许 'day'、'week' 或 'month'。该提示指出了失败的调用，包含了验证错误，并建议采用允许的值或省略可选字段。
The corrective part uses On-Policy Self-Distillation (OPSD). The trainer runs the same GLM 5.2 checkpoint twice on the recorded turn. The teacher pass sees the hint; the student pass does not. Both use teacher forcing, so no replacement answer is generated. The teacher’s next-token probabilities are detached and act as a soft target through forward KL.
纠正部分使用在线策略自蒸馏（OPSD）。训练器在记录的回合上两次运行相同的 GLM 5.2 检查点。教师端看到提示；学生端则看不到。两者都使用教师强制（teacher forcing），因此不会生成替代答案。教师端的下一个 token 概率被断开连接，并通过前向 KL 散度作为软目标。
The combined loss is (CE + λ × KL), divided by the number of imitated tokens. Setting λ to 0 recovers standard SFT. The CE term matters. Correction-only training can let teacher and student agree by ignoring context.
组合损失为 (CE + λ × KL)，除以模仿的 token 数量。将 λ 设为 0 可恢复标准 SFT。交叉熵（CE）项至关重要。仅进行纠错训练可能使教师模型和学生模型通过忽略上下文达成一致。
Tracing Complaints to the Real Mistake
追溯投诉至真正的错误
The pipeline draws from training-eligible Computer sessions served by GLM 5.2. Sessions with personally identifiable information and users who opted out are excluded. An LLM judge keeps tasks rated 4 or 5 on a 5-point difficulty scale. Two LLM judges must both approve the final delivery for a session to count as successful.
该流程从由 GLM 5.2 提供的符合训练条件的 Computer 会话中抽取数据。排除包含个人身份信息以及选择退出的用户的会话。LLM 裁判保留难度评分为 4 或 5（满分 5 分）的任务。两个 LLM 裁判必须都批准最终交付结果，该会话才算作成功。
For user feedback, threes LLM judges locate the responsible turn, and at least 2 must agree. This is important because the last assistant turn before a complaint is the root cause only about half the time. Each hint is also checked against information available before the mistake. That check reduces hindsight bias.
对于用户反馈，三个 LLM 裁判定位负有责任的对话轮次，且至少 2 个需达成一致。这一点很重要，因为在投诉之前的最后一个助手回复轮次仅在约一半的情况下是根本原因。每个提示也会与错误发生前可获得的信息进行核对。该核对步骤减少了后见之明偏差。
One example: a user asked for their ‘w3’ on Paychex. The model assumed a W-2 typo and searched for the wrong form. The hint targets that earlier interpretation, not just the final answer.
一个示例：用户询问其在 Paychex 上的“w3”。模型假设这是 W-2 的拼写错误，并搜索了错误的表格。该提示针对的是更早前的解读，而不仅仅是最终答案。
Interactive Explainer
交互式解释器
What the Evaluations Show
评估结果显示
- Hints work before training: On 985 held-out tool-error turns, the unchanged base model avoided the original failure in 93.7% of cases with hints, up from 75.1%. The share taking the corrected action rose from 60.6% to 82.3%. On user-feedback turns, fixed or on-track rates rose from 40.0% to 75.0% for explicit evidence. For inferred intent, they rose from 32.5% to 80.0%.
- Offline tool errors fell: Recorded tool-error rates were 2.79% for stock GLM 5.2 and 1.35% for RFT only. The RFT plus OPSD checkpoint reached 0.87%. Perplexity notes these checkpoints used different training data, so this is not a matched ablation. Task-level benchmark results on suites like BrowseComp and SpreadsheetBench were mixed.
- Live results are narrower: Each A/B test used about 100,000 users per condition. An early checkpoint versus stock GLM 5.2 showed 2.82% versus 2.94% failures, which was not significant. The later checkpoint comparison produced the significant 21.2% drop, without hints at inference. Strong dissatisfaction moved from 2.58% to 2.54%, which was also not significant. Perplexity did not compare the later checkpoint directly against stock GLM 5.2 online.
- 提示在训练前即生效：在 985 个预留的工具错误轮次中，未更改的基础模型在使用提示的情况下避免了原始失败的比例为 93.7%，高于之前的 75.1%。采取纠正行动的比例从 60.6% 上升至 82.3%。在用户反馈轮次中，带有明确证据时修复或回归正轨的比例从 40.0% 上升至 75.0%；对于推断意图，该比例从 32.5% 上升至 80.0%。
- 离线工具错误减少：记录的工具错误率方面，原始 GLM 5.2 为 2.79%，仅 RFT 为 1.35%。RFT 加上 OPSD 检查点降至 0.87%。Perplexity 指出这些检查点使用了不同的训练数据，因此这不是匹配的控制变量实验。在 BrowseComp 和 SpreadsheetBench 等套件上的任务级基准测试结果喜忧参半。
- 在线结果较为有限：每次 A/B 测试每条件使用约 100,000 名用户。早期检查点与原始 GLM 5.2 的对比显示失败率分别为 2.82% 和 2.94%，差异不显著。后续检查点对比产生了显著的 21.2% 降幅，且在推理阶段不使用提示。强烈不满率从 2.58% 降至 2.54%，同样不显著。Perplexity 并未将后续检查点直接与原始 GLM 5.2 进行在线对比。
Key Takeaways
关键要点
- Perplexity learns from failed sessions, not just successful ones.
- Validated hints turn avoidable mistakes into KL correction targets.
- 1 model acts as teacher (with hint) and student (without).
- Live tool-call failures fell from 2.24% to 1.77%.
- User dissatisfaction showed no significant change.
- Perplexity 从失败的会话中学习，而不仅仅是成功的会话。
- 经过验证的提示将可避免的错误转化为 KL 校正目标。
- 1 个模型同时充当教师（带提示）和学生（不带提示）。
- 实时工具调用失败率从 2.24% 降至 1.77%。
- 用户满意度未出现显著变化。
Check out the Technical Details. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
查看技术细节。本项目的所有功劳归于该研究员。此外，欢迎在 Twitter 上关注我们，别忘了加入拥有超过 15 万成员的 ML SubReddit，并订阅我们的新闻通讯。等等！你在 Telegram 上吗？现在你也可以加入我们的 Telegram 群组。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力