Unsloth发布Qwen3.8-27B量化版,Dynamic V3.1精度提升
Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
提供了一手量化工程实践与完整调优参数,对本地部署Qwen3.8有直接参考价值,建议开发者收藏对照压测。
For the complete documentation index, see llms.txt. This page is also available as Markdown.
完整文档索引请参阅 llms.txt。此页面也可作为 Markdown 格式获取。
Qwen3.8-27B is a 27B parameter dense model Qwen, featuring vision and reasoning capabilities, a 256K context window, and runs locally on 17GB RAM/VRAM setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and Unsloth Desktop.
Qwen3.8-27B 是一个拥有 270 亿参数的密集模型 Qwen,具备视觉和推理能力,支持 256K 上下文窗口,可在配备 17GB RAM/VRAM 的设备上本地运行。Qwen3.8 在代理式编码、视觉和聊天任务方面表现出色,现在可通过 Unsloth GGUFs、NVFP4 和 Unsloth Desktop 运行。
Nov 11 Update: Qwen3.8-27B GGUFs now use Unsloth Dynamic V3.1 which improves average accuracy by 5-10% accuracy at comparable sizes including Terminal Bench, DeepSWE and more benchmarks. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and future models.
11月11日更新:Qwen3.8-27B GGUFs 现采用 Unsloth Dynamic V3.1,在同等参数量下(包括 Terminal Bench、DeepSWE 及其他基准测试),平均准确率提升了 5-10%。我们正在开发更优的动态量化版本,该版本将应用于 Qwen4 及后续模型。
Run Qwen3.8 GuideDownload Unsloth
运行 Qwen3.8 指南下载 Unsloth
Thank you Qwen for day zero access. Unsloth quants also include:
感谢 Qwen 提供首日访问权限。Unsloth 量化版本还包括:
- Developer Role Support for agentic tools like Codex
- MTP enabled for fast inference
- Tool calling: Improved parsing nested objects to make tools succeed more
- 为 Codex 等代理工具提供支持的角色功能
- 启用 MTP 以实现快速推理
- 工具调用:改进了嵌套对象的解析,以提高工具调用的成功率
Unsloth quants: Qwen3.8-27B-GGUF
Unsloth 量化版本:Qwen3.8-27B-GGUF
Dynamic 4-bit Qwen3.8-27B in Unsloth Desktop
Unsloth Desktop 中的动态 4-bit Qwen3.8-27B
Dynamic V3.1 Update (November 11th 2026)
Dynamic V3.1 更新(2026年11月11日)
We're excited to release new Qwen3.8-27B GGUFs using our updated Unsloth Dynamic V3.1 (preview) algorithm which improves Unsloth quants by 5-10% accuracy at the same size on most benchmarks and is the best in class for most tasks including coding, agentic workloads and general chat.
我们很高兴发布基于更新的 Unsloth Dynamic V3.1(预览版)算法的新版 Qwen3.8-27B GGUFs。该算法在大多数基准测试中,以相同参数量实现了 5-10% 的准确率提升,并在包括编码、代理工作负载和通用聊天在内的多数任务中处于同类最佳水平。
Most quants from 2bit until 6bit are updated. We found 3bit UD3 quants to do on par with UD3.1, so we do not update them. We are actively working on making all quants better!
从 2bit 到 6bit 的大多数量化版本均已更新。我们发现 3bit UD3 量化的表现与 UD3.1 相当,因此未对其进行更新。我们正在积极致力于优化所有量化版本!
We ran agentic benchmarks including Terminal Bench 2.1, DeepSWE v1.1, and general benchmarks like LiveCodeBench v6, GPQA Diamond, MMLU-Pro, AIME25 and many others. We used 256K context lengths for all agentic benchmarks, and 128K for the rest. We followed the same methodology for Qwen3.8-27B as DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next which also utilized our V3.1 preview algorithm.
我们运行了包括 Terminal Bench 2.1、DeepSWE v1.1 在内的代理基准测试,以及 LiveCodeBench v6、GPQA Diamond、MMLU-Pro、AIME25 等通用基准测试。所有代理基准测试均使用 256K 上下文长度,其余测试使用 128K。我们在 Qwen3.8-27B 上遵循的方法论与 DeepSeek-V4.1-Flash 和 Qwen3.8-Flash-Next 相同,后者也使用了我们的 V3.1 预览算法。
Dynamic V3.1 Full Breakdown
Dynamic V3.1 详细分解
Click below to see full details of benchmarking and more:
点击下方查看基准测试详情及其他内容:
Click to expand Full details and breakdown
点击展开完整详情和分解说明
Exllama-v3 considerations
Exllama-v3 注意事项
To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.
为了比较 GSQ-RCO 和 Exllama-v3,我们必须从 x 轴中排除词元嵌入层,因为 Exllama-v3 对其使用 BF16。
If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, the below plot is the true reflection of actual usage.
如果你考虑 Exllama-v3 的完整下载大小和总 RAM 使用量,那么添加 BF16 嵌入表并包含所有嵌入大小会改变图表。我们的主图的 x 轴从所有量化提供商中移除了嵌入层大小。然而,在纯 RAM 使用量和磁盘空间使用量与准确率的对比中,下图才是实际使用情况真实反映。
We have not investigated why Exllama-v3 keeps token embeddings in BF16, but as the lm_head is still quantized heavily, the embedding layer might have "absorbed" most of the quantization damage from it. We would be glad to benchmark new Exllama-v3 non BF16 token embedding quants like using 8-bit for example, and see if benchmarks change.
我们尚未调查为什么 Exllama-v3 将 token 嵌入保持在 BF16,但由于 lm_head 仍然被重度量化,嵌入层可能已经“吸收”了来自它的大部分量化损伤。我们将很高兴对新的 Exllama-v3 非 BF16 token 嵌入量化(例如使用 8 位)进行基准测试,看看基准测试结果是否发生变化。
GSQ-RCO results are misleading
GSQ-RCO 的结果具有误导性
For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well.
对于 LiveCodeBench,GSQ-RCO 显示 Unsloth UD3 量化始终比 ISTA 的表现差。然而我们发现相反的情况——UD3 量化(我们的旧量化)在 LCB 上除最小尺寸外,在所有尺寸上都优于 ISTA。UD3.1 也在 LCB 上进一步提升了表现。
We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.
我们估算 GSQ-RCO 使用的上下文预算为 96K,因为这最符合我们所进行的运行数据。在 96K 时,GSQ-RCO 的表现略好于 UD3,但在 128K 时,UD3 的表现要好得多。
On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse.
在另一个基准测试中,针对 ISTA GSQ-RCO 的 AIME25 基准测试,他们的量化达到了 100%、100%、100% 和 96.7% 的准确率恢复。并且他们声称我们的量化表现更差。
Through out investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do BETTER on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.
经过调查,我们估计 ISTA 只进行了 1 次试验——我们进行了 14 次试验,平均恢复率为 93% 到 96%——并非 100% 的准确率恢复。事实上,在我们的 14 次运行与他们的 1 次运行相比,我们的 UD3 旧量化平均表现更好,因此如果你进行更多次试验,我们会得出相反的结论。
We also checked and graded our answers using Math Verify Dynamic 3.1 GGUFs and also LightEval and we find exactly the same conclusions. We also tried greedy decoding (temperature of 0), and this could not get the 100% accuracy recovery ISTA found across multiple greedy runs, as GPUs themselves are non deterministic. That is why you benchmarks should be run multiple times. Greedy decoding is also not representative of real world use cases.
我们还使用 Math Verify Dynamic 3.1 GGUFs 以及 LightEval 检查和评估了我们的答案,发现结论完全相同。我们还尝试了贪婪解码(温度为 0),这在多次贪婪运行中无法获得 ISTA 找到的 100% 准确率恢复,因为 GPU 本身是非确定性的。这就是为什么你的基准测试应该运行多次的原因。贪婪解码也不能代表现实世界的使用场景。
We also checked 96K (which is most likely ISTA's budget) and 128K, and our quants still do on average better.
我们还检查了 96K(这很可能是 ISTA 的预算)和 128K,我们的量化平均表现仍然更好。
KLD of Unsloth v3.1 vs v3
Unsloth v3.1 与 v3 的 KLD
We did KLD for multiple domains for Unsloth UD3.1 quants and UD3 quants - we reduce KLD throughout most quants and most domains - we did not replace the 3-bit ones, as we found UD3 to do mostly great, and the accuracy recovery on benchmarks was not significant enough with UD3.1 to warrant an update, so we did not update them.
我们为 Unsloth UD3.1 量化和 UD3 量化在多个领域进行了 KLD(KL 散度)分析——我们在大多数量化和大多数领域中降低了 KLD——我们没有替换 3-bit 模型,因为我们发现 UD3 的表现总体很好,且 UD3.1 在基准测试中的精度恢复并不显著到足以值得更新,因此我们未对其进行更新。
Benchmark details and setups
基准测试详情与设置
We follow the same procedure for all benchmarks as for Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash as well as discussed in Dynamic 3.1 GGUFs. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning for Qwen3.8-27B.
我们对所有基准测试遵循与 Qwen3.8-Flash-Next、DeepSeek-V4.1-Flash 以及 Dynamic 3.1 GGUFs 中讨论的相同流程。我们使用 temperature 1.0、top_p 0.95、top_k 20、min_p 0,无 presence 或 repetition penalty,并对 Qwen3.8-27B 启用 preserve_thinking 和 xhigh reasoning。
⚙️ Usage Guide
⚙️ 使用指南
Qwen3.8-27B Requirements:
Qwen3.8-27B 要求:
Qwen3.8-27B 4-bit quants work on 16-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM. Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory)
Qwen3.8-27B 4-bit 量化可在 16-19GB VRAM 的设备上运行,如 RTX 5080、4090 或配备 24GB RAM 的 Mac。表格:硬件需求(单位 = 总内存:RAM + VRAM,或统一内存)
1-bit
2-bit
3-bit
4-bit
6-bit
8-bit
BF16
7-8 GB
9-11 GB
12-14 GB
16-19 GB
23-26 GB
31 GB
56 GB
If you want to use MTP for faster inference, prepare to have 1-2GB extra headroom.
如果您希望使用 MTP 进行更快的推理,请预留 1-2GB 的额外余量。
Recommended Settings
推荐设置
Qwen3.8-27B is a hybrid thinking model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can adjust the thinking effort:
Qwen3.8-27B 是一个混合思维模型,针对思维模式和非思维模式有不同的默认设置。Extra high 为默认启用状态,因此如果您希望缩短思维轨迹,可以调整思维强度:
Parameter
参数
Thinking Mode
思考模式
Instruct (non-thinking) Mode
指令(非思考)模式
temperature
1.0
0.7
top_p
0.95
0.80
top_k
20
20
min_p
0.0
0.0
presence_penalty
0.0
1.5
repetition_penalty
1.0
1.0
- Maximum context window: 262,144 (can be extended to 1M via YaRN)
- Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
- Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
- 最大上下文窗口:262,144(可通过 YaRN 扩展至 1M)
- 思考模式:temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
- 指令(或非思考)模式:temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
💡 Thinking + Preserve Thinking
💡 思考 + 保留思考
Qwen3.8 has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):
Qwen3.8 支持“保留思考”功能,可保留上一轮对话的思考痕迹。这会增加使用的 token 数量,但可能提高连续对话的准确性。Unsloth 为 Qwen3.8 提供了“思考”和“保留思考”的开关(见右侧):
Qwen3.8-27B comes with support for reasoning_effort, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:
Qwen3.8-27B 原生支持 reasoning_effort,可用于调整推理深度并控制成本。这些开关在 Unsloth 中会自动启用:
- xhigh (default): for complex tasks demanding thorough analysis
- medium: balancing accuracy and speed
- low: efficient reasoning optimizing for speed and cost
- none
- xhigh(默认):适用于需要深入分析的复杂任务
- medium:平衡准确性与速度
- low:高效推理,优化速度与成本
- none
To change thinking / reasoning effort in unsloth run or llama-server, use --chat-template-kwargs '{"reasoning_effort":"medium"}'
要在 unsloth run 或 llama-server 中更改思考/推理强度,请使用 --chat-template-kwargs '{"reasoning_effort":"medium"}'
If you're on Windows Powershell, use: --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"
如果您使用的是 Windows PowerShell,请使用:--chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"
Change medium to your desired reasoning level.
将 medium 更改为您期望的推理级别。
Run Qwen3.8 Guide
运行 Qwen3.8 指南
You can now run Qwen3.8-27B in llama.cpp and Unsloth Desktop. Feel free to change quantization type.
您现在可以在 llama.cpp 和 Unsloth Desktop 中运行 Qwen3.8-27B。请随意更改量化类型。
- Hugging Face: Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4
- ModelScope: Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4
- Hugging Face: Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4
- ModelScope:Qwen3.8-27B-GGUF • Qwen3.8-27B-NVFP4
Run in Unsloth DesktopRun in llama.cppNVFP4 GuideQwen3.8-2.4T-A95B Guide
在 Unsloth Desktop 中运行 | 在 llama.cpp 中运行 | NVFP4 指南 | Qwen3.8-2.4T-A95B 指南
🦥 Run Qwen3.8 in Unsloth Desktop
🦥 在 Unsloth Desktop 中运行 Qwen3.8
Qwen3.8 can run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux and:
Qwen3.8 可以在 Unsloth Desktop 中运行,这是一款用于本地 AI 的开源 UI 应用。Unsloth 会自动卸载到 RAM 并检测多 GPU 设置。使用 Unsloth Desktop,您可以在 MacOS、Windows、Linux 上本地运行模型,并且能够:
- Search, download, run GGUFs, MLX and safetensor models
- Self-healing tool calling + web search
- Code execution (Python, Bash)
- Automatic inference parameter tuning (temp, top-p, etc.)
- Fast CPU + GPU inference via MLX and llama.cpp
- 搜索、下载并运行 GGUF、MLX 和 safetensor 模型
- 自我修复的工具调用 + 网络搜索
- 代码执行(Python、Bash)
- 自动推理参数调整(温度、top-p 等)
- 通过 MLX 和 llama.cpp 实现快速的 CPU + GPU 推理
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力