Unsloth发布DeepSeek-V4.1 GGUF基准测试及量化对比
Unsloth vs GSQ-RCO - 512K context DeepSeek-V4.1 GGUF benchmarks Terminal Bench
本地部署玩家关注DeepSeek大模型的量化效果，这份报告提供了详尽的长上下文基准数据和对竞品量化方案的统计学证伪，对选型极具参考价值。
We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD version 3.1 GGUFs and show we're >10-20% better than GSQ-RCO
我们在 DeepSeek UD 3.1 GGUF 模型上，使用 512K 上下文运行了 DeepSWE、Terminal Bench、Automation Bench、HLE 和 SciCode，结果显示我们比 GSQ-RCO 高出 10-20%
We ran all other benchmarks (14 in total) to 128K. AIME2025 64x, DeepSWE/TB 3x. We addressed some issues people mentioned with these new benchmarks:
我们将所有其他基准测试（共 14 个）运行至 128K。AIME2025 运行 64 次，DeepSWE/TB 运行 3 次。我们解决了人们针对这些新基准测试提出的一些问题：
- 32K context was too short - we did 512K context for agentic + 128K for general tasks
- Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench
- Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See this section where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.
- Our graders are wrong - we used Math Verify, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See graders and infra fixes
- 32K 上下文太短——我们对代理任务使用 512K 上下文，对通用任务使用 128K 上下文
- 饱和的基准测试——新增未饱和的代理风格基准测试，包括 Terminal Bench、DeepSWE 和 Automation Bench
- 基准测试存在噪声——我们进行了 351 亿 token 的基准测试，而之前仅为 13 亿，数据量增加了 27 倍。请参阅本节，其中我们进行了 64 次 AIME 2025 运行、3 次 DeepSWE 运行、3 次 TB 运行等。
- 我们的评分器有误——我们使用了 Math Verify、改进了评分器，在 HLE 中使用了 gpt-6-luna xhigh，并在 Terminal Bench (Harbor)、DeepSWE 及其他测试中使用了官方评分器。详见评分器和基础设施修复部分
For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).
对于 DeepSWE v1.1，我们使用了 mini-swe-agent，并将最大步数设为 500（我们观察到的最大运行步数为 334 步，因此没有超过限制）。
Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues
Terminal Bench 2.1 也使用了 mini-swe-agent，我们将最大步数设为 150，且无最大限制。由于在我们的基础设施上无法通过，我们排除了 89 个任务中的 7 个——详见 https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues
We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.
我们还重新检查了 ISTA 的数据，因为我们进行了更多次运行（数据量增加 27 倍）。GSQ-RCO IQ1_M 声称恢复率为 98.7%，但我们测得的值为 97.24 [96.23, 98.21]，p 值为 0.0013。
GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. So this is possible.
GSQ-RCO IQ2_XS 声称保持 100% 恢复率是可能的，其 p 值为 0.21。我们测得的值为 98.38% [97.52, 99.22]。因此这是可能的。
GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. So this is nearly impossible.
GSQ-RCO IQ2_XXS 声称恢复率为 99.80%，但我们测得的值为 98.38 [97.52, 99.22]，p 值为 0.0001。因此这几乎不可能。
Paired difference to the official base (our runs, same questions, sign-flip permutation test, two-sided):
与官方基线（我们的运行结果，相同问题，符号翻转置换检验，双尾）的配对差异：
| AIME25 | GPQA-D | LCB v6 | |
|---|---|---|---|
| GSQ-RCO IQ1_M | -1.61 (p=0.02) | -2.89 (p=0.004) | -3.24 (p=0.00002) |
| GSQ-RCO IQ2_XXS | -0.47 (p=0.25) | -2.07 (p=0.02) | -2.00 (p=0.01) |
| GSQ-RCO IQ2_XS | -0.42 (p=0.48) | -0.51 (p=0.42) | -0.38 (p=0.52) |
The claimed recovery tested directly against our measurement (the card's definition: sum of the 3 scores / the base's sum, vs official FP8/FP4): p (no loss) = paired sign-flip permutation test of recovery = 100%; p (recovery >= claim) = one-sided paired bootstrap over questions.
直接针对我们测量值进行测试的声称恢复率（卡片定义：3 个分数之和 / 基线之和，与官方 FP8/FP4 对比）：p (无损失) = 恢复率等于 100% 的配对符号翻转置换检验；p (恢复率 >= 声称值) = 基于问题的单尾配对自助法。
| Claimed recovery | Measured recovery [95% CI] | p (no loss) | p (recovery >= claim) | |
|---|---|---|---|---|
| GSQ-RCO IQ1_M | 98.70 | 97.24 [96.23, 98.21] | 0.00002 | 0.0013 |
| GSQ-RCO IQ2_XXS | 100.00 | 98.38 [97.52, 99.22] | 0.0003 | 0.0001 |
| GSQ-RCO IQ2_XS | 99.80 | 99.54 [98.88, 100.18] | 0.16 | 0.21 |
| 声称的恢复率 | 测量的恢复率 [95% CI] | p (无损失) | p (恢复 >= 声称) | |
|---|---|---|---|---|
| GSQ-RCO IQ1_M | 98.70 | 97.24 [96.23, 98.21] | 0.00002 | 0.0013 |
| GSQ-RCO IQ2_XXS | 100.00 | 98.38 [97.52, 99.22] | 0.0003 | 0.0001 |
| GSQ-RCO IQ2_XS | 99.80 | 99.54 [98.88, 100.18] | 0.16 | 0.21 |
If we can get more details on how the benchmarks were run from GSQ-RCO, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026
如果我们能从 GSQ-RCO 获得有关基准测试运行方式的更多详细信息，我们将非常乐意进行更多测试。我们在博客中提供了所有设置和方法：https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026
All details and data and methods at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026.
所有详细信息、数据和方法见 https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026。
Our new Unsloth Dynamic 3.1 GGUFs for DeepSeek are at https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF
我们针对 DeepSeek 的新版 Unsloth Dynamic 3.1 GGUF 文件位于 https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力