Qwen3.8-27B单卡RTX 3090加速引擎实测：140
UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp
本地部署Qwen的同学必看，这篇给出了完整的性能对比数据和复现细节，单卡跑27B模型还能保持精度无损，直接去GitHub看怎么配置。
This is a follow-up to my post from yesterday (https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/) detailing the CUDA megakernel. Thank you all for the positive feedback and PRs!
这是我昨天帖子（https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/）的后续，详细介绍了 CUDA megakernel。感谢大家积极的反馈和 PR！
RECAP: The megakernel is a CUDA engine for Qwen3.8-27B that runs 1.4-1.9x faster than llama.cpp on a single 3090. The most common questions were about accuracy, the baseline and context size, so here are the numbers.
回顾：megakernel 是 Qwen3.8-27B 的一个 CUDA 引擎，在单张 3090 上比 llama.cpp 快 1.4-1.9 倍。最常见的问题是关于准确性、基线和上下文大小，以下是数据。
Code and benchmarks: https://github.com/L-Forster/open-jet/tree/master/megakernel
代码和基准测试：https://github.com/L-Forster/open-jet/tree/master/megakernel
Firstly, I benchmarked the engine accuracy against llama.cpp. No measurable loss in quality:
首先，我针对 llama.cpp 对引擎的准确性进行了基准测试。质量没有可测量的损失：
Decode accuracy vs llama.cpp
解码准确性与 llama.cpp 对比
| Context | Positions | MK top token | MK KL | LC top token | LC KL |
|---|---|---|---|---|---|
| 1K | 13,299 | 99.08% | 0.0009 | 98.68% | 0.0019 |
| 30K | 1,999 | 99.65% | 0.0007 | 99.35% | 0.0009 |
| 96K | 1,999 | 99.60% | 0.0008 | 99.45% | 0.0014 |
| 上下文 | 位置 | MK top token | MK KL | LC top token | LC KL |
|---|---|---|---|---|---|
| 1K | 13,299 | 99.08% | 0.0009 | 98.68% | 0.0019 |
| 30K | 1,999 | 99.65% | 0.0007 | 99.35% | 0.0009 |
| 96K | 1,999 | 99.60% | 0.0008 | 99.45% | 0.0014 |
MK = megakernel, LC = llama.cpp batched. Top token = how often the top token matches the reference. KL = mean KL divergence in nats.
MK = megakernel，LC = llama.cpp batched。Top token = 最高概率 token 与参考值匹配的频率。KL = nats 单位的平均 KL 散度。
- Perplexity (1K run only): megakernel 2.4274, llama.cpp batched 2.4283, reference 2.4263.
- Log-loss vs the reference: +0.0005 +/- 0.0004 at 1K, +0.0018 +/- 0.001 at 30K, +0.0012 +/- 0.001 at 96K. Not significant.
- The 30K and 96K rows are one sequence of code each & the 1K row is code and prose.
- Speculative decoding is exact: greedy output is the main model's greedy choice, and sampling keeps the same distribution as without drafts.
- 困惑度（仅 1K 运行）：megakernel 2.4274，llama.cpp batched 2.4283，参考值 2.4263。
- 与参考值的对数损失：1K 时为 +0.0005 +/- 0.0004，30K 时为 +0.0018 +/- 0.001，96K 时为 +0.0012 +/- 0.001。不显著。
- 30K 和 96K 行各包含一个代码序列，而 1K 行包含代码和散文。
- 推测解码是精确的：贪婪输出即主模型的贪婪选择，采样保持与无草稿时相同的分布。
Speed (same 3090, megakernel vs llama.cpp with MTP)
速度（同一张 3090，megakernel 与带 MTP 的 llama.cpp 对比）
- Writing code: 140 vs 73 tok/s
- Reasoning mode: 78 vs 51 tok/s
- With 30K tokens of code in context: 73 vs 52 tok/s
- Prompt processing (prefill): ~1,600 vs ~1,100 tok/s
- With speculative decoding off they are about equal (42.9 vs 40.4). The gain comes from checking 4-5 drafted tokens per pass for about the cost of one. llama.cpp ran at 2 drafts, the megakernel at 4.
- 编写代码：140 vs 73 tok/s
- 推理模式：78 vs 51 tok/s
- 上下文中包含 30K token 的代码时：73 vs 52 tok/s
- 提示处理（prefill）：约 1,600 vs 约 1,100 tok/s
- 关闭推测解码时，两者大致相等（42.9 vs 40.4）。增益来自于每次迭代检查 4-5 个草稿 token，成本约等于一个。llama.cpp 运行了 2 个草稿，megakernel 运行了 4 个。
Since the first post
自第一篇帖子以来
- Two PRs have been merged: the loader now checks every tensor's type, and the server warms up before accepting requests.
- Model download fixed after unsloth removed the Q4_K_M from their repo.
- It also loads the Q4_K_M from lmstudio-community and mradermacher.
- openjet setup shows which engine each model will use.
- 已合并两个 PR：加载器现在会检查每个张量的类型，服务器在接受请求前会进行预热。
- 模型下载已修复，因为 unsloth 从其仓库中移除了 Q4_K_M。
- 它也从 lmstudio-community 和 mradermacher 加载 Q4_K_M。
- openjet 设置显示每个模型将使用哪个引擎。
Quantisation support
量化支持
This table details the quantisation support, with unsloth Q4_K_M being the native model and what I benchmarked, lmstudio and mradermacher being semi-compatible, not fully benchmarked, tentative results show similar speed to native after type conversions. Other quants fall back to llama.cpp
下表详细说明了量化支持情况，unsloth Q4_K_M 是原生模型且是我进行基准测试的版本，lmstudio 和 mradermacher 是半兼容的，未完全进行基准测试，初步结果显示经过类型转换后速度与原生版本相似。其他量化版本回退到 llama.cpp
| Qwen3.8-27B file | Runs on |
|---|---|
| unsloth's original Q4_K_M (what openjet setup downloads) | megakernel |
| Q4_K_M from lmstudio-community, mradermacher | megakernel, semi-compatible |
| Other quants (unsloth UD-*, bartowski, GSQ-RCO and others) | llama.cpp |
| Qwen3.8-27B 文件 | 在 unsloth 的原始 Q4_K_M（openjet 设置下载的内容）上运行 | lmstudio-community、mradermacher 提供的 megakernel Q4_K_M | megakernel，半兼容 |
|---|---|---|---|
| 其他量化版本（unsloth UD-*、bartowski、GSQ-RCO 及其他） | llama.cpp |
Limits
限制
- Single GPU. RTX 30/40/50 series with 24 GB or more. MK is tuned on a 3090 only.
- 147K context runs on a 3090 (fp16 KV cache to ~80K, using q8 above).
- No task-level pass-rate benchmark yet.
- 单 GPU。RTX 30/40/50 系列且显存 24 GB 或以上。MK 仅在 3090 上调优。
- 147K 上下文可在 3090 上运行（fp16 KV 缓存至约 80K，使用 q8 及以上）。
- 尚无任务级通过率基准测试。
Contributions that would help
有助于改进的贡献
- Tunes for specific hardware: RTX 4090, 5090 and other 24 GB+ cards. It has only been tuned on a 3090.
- Support for other quantisations of Qwen3.8-27B.
- Please include benchmarks with the PR: your GPU, and bench/bench.py or bench/code_bench.py numbers before and after.
- 针对特定硬件的调优：RTX 4090、5090 及其他 24 GB+ 显卡。目前仅在 3090 上调优。
- 支持 Qwen3.8-27B 的其他量化版本。
- 请在 PR 中包含基准测试结果：你的 GPU 型号，以及 bench/bench.py 或 bench/code_bench.py 的前后对比数据。
How to contribute: see CONTRIBUTING.md in the repo.
如何贡献：请参阅仓库中的 CONTRIBUTING.md。
Comment the quant you use for Qwen3.8-27B so I know what to add support for next! (also planning for Qwen 4 when that drops).
评论你使用的 Qwen3.8-27B 量化版本，以便我知道下一步需要添加对哪种的支持！（同时也计划在新版 Qwen 4 发布时提供支持）。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力