本地大模型为何感觉更笨：推理后端与精度对输出的影响
本地大模型为何感觉比实际更笨：提示词优化指南
本地部署同学必看，详细拆解了推理后端选择如何悄悄改变模型输出，附具体实验参数与评估方法，直接指导调优。
Return to Level1Techs.com
返回 Level1Techs.com
Why your local LLM feels dumber than it is
为什么你本地的 LLM 感觉比实际更笨
High-Performance Computing Machine Learning, LLMs, & AI
高性能计算：机器学习、LLM 与 AI
ai
人工智能
thr3e August 16, 2026, 1:51am 1
thr3e 2026年8月16日，凌晨1:51 1
Quick Introduction
快速介绍
We have all been on forums, chats, reddit, discord, youtube, or somewhere and heard “Oh! Model XYZ is AMAZEBALLZ!zomgwtfbbq” then downloaded it (or more likely, some quantized form of it) and said “eww… This sucks!”
我们都曾在论坛、聊天室、Reddit、Discord、YouTube 或其他地方听到过“哦！模型 XYZ 简直太棒了！zomgwtfbbq”，然后下载了它（或者更可能是它的某种量化版本），接着说“呃……这太烂了！”
This post is going to be a rather technical series of experiments to demonstrate the impact of implementation-specific hazards with inference. I will be using the term “reference implementation” to describe the lab that published and offers first-party hosting of their models and posts original benchmark claims. Their hardware will be different than yours. Their software will be very different than yours. And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.
这篇文章将是一系列相当技术性的实验，旨在展示推理过程中特定实现带来的隐患影响。我将使用“参考实现”一词来描述发布其模型并提供官方托管服务且发布原始基准测试声明的实验室。他们的硬件会与你的不同，他们的软件也会与你的大相径庭。而且，本文中的比较并不是指在 Ollama 中运行某个 2.58-bit GGUF 格式模型并输入几个测试提示语那么简单。
I am intentionally glossing over entire emerging fields of study, mountains of research papers and lit review to make this more approachable for you the reader. Don’t nit pick my oversimplifications or I will make you read the really long unpleasant version with math.
我有意略过了整个新兴研究领域、堆积如山的论文和文献综述，以便让作为读者的你更容易理解。不要对我过度简化的内容吹毛求疵，否则我会让你去读那个包含数学公式的冗长且不愉快的版本。
Your local implementation sucks. But that’s ok, because everyone else’s does too.
你本地的实现确实很烂。但没关系，因为别人的也很烂。
Every single instance of hardware and software running an LLM today is a little bit different. or a lot different when it comes to some cases. The average home lab user might be mixing multiple different generations of GPU. The chips on those have different instruction sets. Those instruction sets will implement and execute math to calculate your next token differently from any other person, even when running the same exact weights.
如今，每一个运行 LLM 的软硬件实例都略有不同，在某些情况下甚至差异巨大。普通家庭实验室用户可能会混合使用多代不同的 GPU。这些芯片拥有不同的指令集。即使运行完全相同的权重，这些指令集也会以不同于任何其他人的方式实现和执行数学计算，从而生成你的下一个 token。
So that begs the first question: How much does your particular setup suck? Turns out there are a number of different ways to go about measuring that.
这就引出了第一个问题：你的特定配置到底有多烂？事实证明，有多种方法可以衡量这一点。
The practical approach is straight forward. Run standard benchmarks. A variety of them. terminal bench, hle, SWEthis, HELLAthat, MMLU-whatever… take your pick. Just make sure its representative of your actual workload/use case. Do not crank temperature to zero and paste in 3 test prompts then call it good/bad. Zero-shot tests are not a good analog of most agentic tasks. You need long-context tool-calling and domain specific knowledge evaluations to figure out where your setup is weak when running the same weights as somebody else replicating those same benchmarks.
实用方法很简单。运行标准基准测试。多种多样的基准测试。terminal bench、hle、SWEthis、HELLAthat、MMLU-whatever……随你挑选。只要确保它能代表你的实际工作负载/用例即可。不要将温度设为零，粘贴三个测试提示语就宣称好坏。零样本测试并不能很好地模拟大多数智能体任务。你需要通过长上下文工具调用和领域特定知识评估，才能找出在与他人复现相同基准测试并使用相同权重时，你的配置在哪些方面存在弱点。
But the purely mathematical answer is where my focus is going to begin because as @wendell said:
但纯粹的数学答案是我关注的起点，因为正如 @wendell 所说：
Math is Math!
数学就是数学！
“Logits” are the models scores for each possible next token. They are normalized into probabilities, passed through the configured sampler, and converted back into text by the detokenizer to generate THE→NE→XT→TOK→EN during decode.
"Logits"（对数几率）是模型为每个可能的下一个 token 给出的分数。它们被归一化为概率，通过配置的采样器处理，并由 detokenizer（去词元化器）转换回文本，从而在解码过程中生成 THE→NE→XT→TOK→EN。
A side note about sampler settings: the model card on HF usually specifies exactly what sampler settings (and chat template) you should be using. temp 1.0, top-p 0.95, etc. it varies by model so make sure you are using the right ones. btw, setting temp too low is why your qwen is sitting there looping unable to escape its THINK output. You’re welcome, glad I could fix that for you.
关于采样器设置的旁注：HF 上的模型卡片通常会明确规定你应该使用的具体采样器设置（以及聊天模板），例如 temp 1.0、top-p 0.95 等。这因模型而异，所以请确保你使用的是正确的设置。顺便说一句，将 temp 设置得过低正是你的 qwen 陷入循环、无法跳出 THINK 输出的原因。不客气，很高兴能帮你解决这个问题。
When the next token probability changes enough, THE→NE→XT becomes THE→NE→W→DAY… And while those small changes might be fine, odds are that’s the beginning of the niggling sensation in the back of your mind that something feels off.
当下一个 token 的概率发生足够大的变化时，THE→NE→XT 就会变成 THE→NE→W→DAY……虽然这些微小的变化可能看起来没问题，但这很可能是你脑海中那种隐隐约约觉得哪里不对劲的感觉的开端。
Some of you may have heard the term KLD before, or KL Divergence. Don’t worry, I won’t make you do any math or flood your brain with tables of very small decimal numbers. But just in case you wanted the simple version: convert the output logits into a probability distribution, and measure how far that distribution has moved from a chosen baseline. Lower KLD means closer to that baseline, not automatically ‘smarter’. KLD is also directional, so the order of the two distributions matters.
你们中的一些人可能听说过 KLD 这个词，或者叫 KL 散度。别担心，我不会让你做数学题，也不会用一堆极小的十进制数字表格淹没你的大脑。但以防万一你想要一个简单的版本：将输出的 logits 转换为概率分布，并测量该分布与选定基线之间的距离。较低的 KLD 意味着更接近该基线，但并不自动意味着更‘聪明’。KLD 也是有方向的，因此两个分布的顺序很重要。
A word of caution: Don’t get suckered in by impossibly low KLD claims on a quant HF model card. It is impossible to interpret a number unless the author discloses the reference checkpoints and full runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, any vocabulary truncation, and how the measurements were aggregated. The methodology matters as much as the number and plenty of people get it wrong.
警告：不要被量化 HF 模型卡片上那些不可能极低的 KLD 声明所迷惑。除非作者披露了参考检查点、完整的运行时环境、评估文本、校准数据、上下文长度、采样位置、KL 方向、任何词汇表截断方式以及测量结果的聚合方法，否则该数字是无法解释的。方法论与数字本身同样重要，而且很多人对此理解错误。
What the hell is vllm doing?
vllm 到底在做什么？
Now, we need to take a brief field trip down what the giant stack of software is doing on your inference engine to understand where some of those sources of divergence come from.
现在，我们需要简要地实地探访一下推理引擎中那堆庞大的软件究竟在做什么，以了解其中一些差异来源来自何处。
wtf is vllm doing659×610 16.8 KB
vllm 到底在搞什么鬼659×610 16.8 KB
At every step of this oversimplified diagram are components that can be configured or changed based on your specific hardware/software footprint, model, quant, tensor shape, etc.
在这个过度简化的图表的每一步中，都有可以根据你特定的硬件/软件配置、模型、量化方式、张量形状等因素进行配置或更改的组件。
The nightly VLLM container image I snagged had 734 (252 uv/pip Python) packages in it. That’s 734 codebases each with their own bugs and undocumented idiosyncrasies. The path your specific implementation takes through that mountain of code will be distinct.
我搞到的那个每晚更新的 VLLM 容器镜像里包含了 734 个（252 个 uv/pip Python）包。这意味着有 734 个代码库，每个都有自己的 bug 和未记录的怪癖。你的特定实现穿过这片代码大山的路径将是独一无二的。
Test 1: Precision Benchmarking Attention Backends
测试 1：精度基准测试注意力后端
Lets start with one piece of that inference flowchart. During prefill (prompt processing) there are a several attention backends your inference engine will select from. This impacts both speed and precision of prefill, while requiring different cuda kernels for every GPU family / SM compute capability 1.3. The CUDA platform — CUDA Programming Guide . Lets test them and compare.
让我们从推理流程图的一部分开始。在预填充（提示词处理）期间，你的推理引擎将从几个可用的注意力后端中进行选择。这会影响预填充的速度和精度，并且需要为每个 GPU 系列 / SM 计算能力使用不同的 CUDA 内核 1.3。CUDA 平台 — CUDA 编程指南。让我们来测试并比较它们。
WENDELL DARE CUDA ROFL1024×1536 569 KB
温德尔·戴尔 CUDA ROFL1024×1536 569 KB
(I’m really very sorry, I had to…)
（我真的非常抱歉，我不得不……）
I started with the official BF16 checkpoint of Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU at tensor parallelism 1. The KV cache was BF16, with no weight/activation or KV-cache quantization. The software was a pinned nightly vllm build. I used eager execution, disabled CUDA graphs, prefix caching, and MTP, and used 2k-token chunked prefill.
我在 RTX PRO 6000 Blackwell GPU 上以张量并行度 1 开始使用 Qwen3.6-27B 的官方 BF16 检查点。KV 缓存为 BF16，没有进行权重/激活或 KV 缓存量化。软件使用的是固定版本的每晚构建 vllm。我使用了 eager 执行模式，禁用了 CUDA 图、前缀缓存和 MTP，并使用了 2k-token 的分块预填充。
Qwen3.6-27B is dense, not an MoE, but it is still a hybrid model. 64 layers repeat in a pattern of three Gated DeltaNet/linear-attention layers followed by one full-attention layer. Only those 16 full-attention layers use the selectable attention backend in this experiment; the Gated DeltaNet path remained fixed.
Qwen3.6-27B 是稠密模型，不是 MoE，但它仍然是一个混合模型。64 层按照三个门控 DeltaNet/线性注意力层后跟一个全注意力层的模式重复出现。在本实验中，只有那 16 个全注意力层使用了可选的注意力后端；门控 DeltaNet 路径保持固定。
The workload replayed here is “Prompt 2”, a roughly 100k token context captured from a real Turnstone lab workstream containing multiple tool calls and real work products. It was selected to resemble what a local agent actually does rather than a synthetic needle-in-a-haystack test. And maybe more importantly, it doesn’t appear in any benchmark or training dataset in the wild today. Nobody could have benchmaxed for this, or calibrated their quant to accommodate it.
这里重放的工作负载是“提示词 2”，这是一个大约 100k token 的上下文，捕获自 Turnstone 实验室的一个真实工作流，其中包含多个工具调用和实际工作成果。选择它是为了模拟本地代理实际执行的任务，而不是那种合成式的“大海捞针”测试。而且更重要的是，它目前并未出现在任何公开存在的基准测试或训练数据集中。没有人能针对此进行过基准测试优化，也没有人校准过量化以适应它。
There are three available full attention backends to select from in vllm for this workload: FlashAttention 2, Flash Inference, and Triton Attention. This was the only change made between executions, the rest of the hardware and software stack remained stable.
对于此工作负载，vllm 中有三个可用的全注意力后端可供选择：FlashAttention 2、Flash Inference 和 Triton Attention。这是执行之间唯一做出的更改，其余硬件和软件栈保持稳定。
I also performed a same-backend cross-GPU repeatability control. For this graph, I captured the full-vocabulary logits in BF16 every 32 prompt tokens. Distribution comparisons such as KLD were calculated afterward in FP64 from those stored logits.
我还执行了同后端跨 GPU 的可重复性控制。对于此图表，我每 32 个提示词 token 捕获一次全词汇表的 logits（BF16）。随后使用存储的 logits 以 FP64 精度计算分布比较，如 KLD。
Top-1 agreement is whether the token with the highest logit, the greedy argmax, was the same. All three backends were evaluated against the same forced token history. A “top-1 flip” therefore means a backend would have chosen a different greedy next token at that position. We did not let that choice alter the remaining history. This keeps the mathematical comparison controlled, but it does not show how far an unconstrained generation would branch or whether a tool call would eventually fail… that comes in test 2 ;D
Top-1 一致性是指具有最高 logit 的 token（即贪婪 argmax）是否相同。所有三个后端均针对相同的强制 token 历史进行评估。因此，“top-1 翻转”意味着某个后端在该位置会选择不同的贪婪下一个 token。我们并未让该选择改变剩余的历史记录。这保持了数学比较的可控性，但它无法展示无约束生成会分支多远，或者工具调用最终是否会失败……这些将在测试 2 中涉及 ;D
The following graph shows % of sampled logits resulting in token flips:
以下图表显示了导致 token 翻转的采样 logit 百分比：
triton-baseline-top1-disagreement-lines-8k-prompt2-96k3334×1331 241 KB
triton-baseline-top1-disagreement-lines-8k-prompt2-96k3334×1331 241 KB
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力