跳到主内容
@wquguru
精选88r/MachineLearning(Reddit)技巧与观点

无需训练的推理模型早退法:将线性探针折叠进输出层权重

Training-free early exit for reasoning models: a linear probe folded into the </think> row of the output layer [p]

原文
发到 X
推荐理由

做推理模型部署的同学必看,这种把早退探针直接融进权重的做法零开销且可复现,能大幅降低推理成本,建议直接拿你的 GGUF 模型试压。

Reasoning models often reach the right answer early and then keep double-checking it for thousands of tokens. I got the model to stop at that point by changing a single row of weights. There's no training and no extra code at inference time; the edited model is an ordinary weights file that runs anywhere.

推理模型往往在很早就能得出正确答案,然后继续用数千个 token 反复核对。我通过修改一行权重,让模型在那个阶段就停止输出。无需训练,推理时也不需要额外代码;编辑后的模型就是一个普通的权重文件,可以在任何地方运行。

Processing img 5yk991rjlt1i1...

处理 img 5yk991rjlt1i1...

Background,

背景,

  • Reasoning models (DeepSeek, Qwen, and so on) "think" in a scratchpad before answering. The scratchpad ends when the model writes a special token, </think>.
  • To pick each next token, the model first computes a summary of everything so far: a long list of numbers called the hidden state. The last layer, the output layer, has one row of weights per word in the vocabulary. Each word's score is its row multiplied by the hidden state, and the highest-scoring words are the likeliest next words.
  • So the score for "stop thinking now" (</think>) comes from exactly one row of that layer.
  • 推理模型(如 DeepSeek、Qwen 等)在回答前会先在草稿区“思考”。当模型写出特殊标记 <think> 时,草稿区结束。
  • 为了选择下一个 token,模型首先计算到目前为止所有内容的摘要:一个称为隐藏状态的长数字列表。最后一层即输出层,词汇表中每个词对应一行权重。每个词的得分是其所在行与隐藏状态相乘的结果,得分最高的词最有可能成为下一个词。
  • 因此,“停止思考”(<think>)的得分恰好来自该层的一行权重。

What I did

我所做的操作

  • Ran the model on 24 maths problems and recorded its hidden state at every token of its thinking.
  • Marked the points where the right answer had already appeared and the model was only re-checking (specifically, the ends of paragraphs after that point), and treated everything else as "still working".
  • Found the direction in the hidden state that best separates those two groups. This is a probe: a simple linear classifier (linear discriminant analysis), computed directly with linear algebra, with no gradient training.
  • Added that direction, times a strength of 14, to the </think> row. The "stop" score now rises only when the model's state looks like "done, just re-checking". While it's still solving, nothing measurable changes.
  • 在 24 道数学题上运行模型,并记录其思考过程中每个 token 处的隐藏状态。
  • 标记出正确答案已经出现且模型仅在进行复核的点(具体而言是这些点之后的段落结尾),并将其他所有内容视为“仍在解题中”。
  • 找到隐藏状态中最能区分这两组的方向。这是一个探针:一个简单的线性分类器(线性判别分析),直接使用线性代数计算,无需梯度训练。
  • 将该方向乘以强度因子 14 后加到 <think> 对应的行上。现在,只有当模型的状态看起来像“已完成,仅在复核”时,“停止”得分才会上升。而在模型仍在解题时,没有任何可测量的变化。

A useful side effect: because only one token's score changes, I can compute exactly what the edited model would do from recordings of the original, without rerunning it. That's how I picked the strength and searched for cases where it would stop too early.

一个有用的副作用:由于只有一个 token 的得分发生变化,我可以仅凭对原始模型的记录,精确计算出编辑后模型的行为,而无需重新运行它。这就是我选择强度因子并搜索可能出现过早停止情况的方法。

Processing img 0r3nr3xrlt1i1...

处理 img 0r3nr3xrlt1i1...

On a problem it never saw: the answer appears at token 567, the original keeps thinking until token 7,415, and the edited model stops by token ~2,000 with 98% probability.

在一个从未见过的题目上:答案在第 567 个 token 处出现,原始模型一直思考到第 7,415 个 token,而编辑后的模型以 98% 的概率在第 ~2,000 个 token 处停止。

Results (small samples, one run per setting)

结果(小样本,每种设置运行一次)

Processing img psaikkt6lt1i1...

处理 img psaikkt6lt1i1...

  • Qwen3.5-4B (runs in stock llama.cpp): same accuracy on 8 unseen MATH-500 problems with 54% fewer tokens. On 3 AIME 2025 problems with a 32k-token limit, the original ran out of tokens on all three while re-checking answers it already had; the edited model solved 2.
  • DeepSeek V4.1 Flash (763B parameters, 2-bit, on my laptop with an 8 GB GPU, streaming weights from the SSD): 7/7 maths problems vs 6/7, with 40% fewer tokens.
  • As a coding agent building a small website: 84% less thinking. It still thinks after an error or anything unexpected, and skips thinking after routine steps. Page quality held up; an LLM rating put it highest, and the screenshots are below so you can judge for yourself.
  • Qwen3.5-4B(可在标准 llama.cpp 中运行):在 8 道未见过的 MATH-500 问题上保持相同准确率,但 token 使用量减少了 54%。在 3 道 AIME 2025 题目上(限制为 32k 个 token),原始模型在已得出答案的情况下仍不断复核,导致三题均因耗尽 token 而失败;而编辑后的模型成功解出了其中两题。
  • DeepSeek V4.1 Flash(763B 参数,2-bit,在我的配备 8 GB GPU 的笔记本电脑上运行,权重从 SSD 流式传输):数学题答对 7/7 vs 6/7,且 token 数减少 40%。
  • 作为构建小型网站的编码代理:思考时间减少了 84%。它在遇到错误或任何意外情况时仍会思考,而在常规步骤后则跳过思考。页面质量保持良好;LLM 评分将其评为最高,截图如下,供您自行评判。

Processing img 3q1n1yp8lt1i1...

处理图片 img 3q1n1yp8lt1i1...

What isn't new: using a probe on the hidden state to stop reasoning early is published (e.g. Zhang et al. 2025, arXiv 2504.05419). What's new is putting the probe inside that one row of weights, so nothing extra runs, plus these measurements.

并非新内容:使用探针作用于隐藏状态以提前停止推理的做法已有发表(例如 Zhang 等人 2025,arXiv 2504.05419)。新之处在于将探针置于那一行权重内部,因此无需额外运行任何东西,再加上这些测量结果。

Limitations: small sample sizes; the probe was fitted on maths only (the agent results are transfer); the AIME comparison uses a 32k budget, below the 81,920 Qwen recommends for competition maths.

局限性:样本量较小;探针仅针对数学进行拟合(代理结果是迁移的);AIME 比较使用的是 32k 预算,低于 Qwen 为竞赛数学推荐的 81,920。

Write-up: https://think-graft.vercel.app Code and all logs: https://github.com/nextgen-user/think-graft Model (GGUF): https://huggingface.co/vaibhavrpi/Qwen3.5-4B-think-graft-GGUF

文章链接:https://think-graft.vercel.app 代码和所有日志:https://github.com/nextgen-user/think-graft 模型(GGUF):https://huggingface.co/vaibhavrpi/Qwen3.5-4B-think-graft-GGUF

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件