Betelgeuse引擎在RTX 3060上实现Qwen极速推理
Betelgeuse: Running Qwen 3.8 Flash Next at 22 tok/s sustained (spiking to 48 tok/s) on an RTX 3060 12GB + 16GB DDR4 RAM 100% bit-exact!
硬核工程实践，详细拆解了如何在低配硬件上通过自定义调度与缓存策略实现MoE模型的高性能推理，参数与架构细节清晰可复现，值得本地部署玩家收藏参考。
About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.
大约两个月前，我在此发帖讨论过尝试预测下一个 token 会使用哪些 MoE 专家，以加速 Qwen 3.6 35B 的 CPU/GPU 卸载。
Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.
在说其他事情之前，先对那篇旧帖做个快速坦白：我最终把那个原型扔掉了。为了获得我当时看到的那些大数字，我基本上是在通过门控剪枝（当路由器的权重较低时丢弃冷专家）作弊。这让 tok/s 计数器疯狂飙升，但在已经低比特量化的情况下，它悄无声息地降低了输出质量和推理能力。我不想发布一个只是为了在截图中看起来很快而让模型“脑叶切除”的东西，所以我把它废弃了。
When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.
当 Qwen 3.8 Flash Next 发布时，我想看看我们是否真的能在普通的预算游戏 PC 上以实际可用的速度运行 68GB GSQ-RCO-IQ2_XS 版本——而不丢弃任何专家。
For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):
作为背景，这是我测试的确切配置（我实际上特意拔出了多余的内存条，以测试普通 16GB PC 的表现）：
- GPU: RTX 3060 12GB
- RAM: 16GB DDR4, single-channel (~19 GB/s)
- OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)
- Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)
- GPU: RTX 3060 12GB
- RAM: 16GB DDR4，单通道（约 19 GB/s）
- OS: CachyOS / Arch Linux（在严格的 MemoryMax=12G cgroup 内测试）
- Storage: 消费级 PCIe NVMe SSD（约 2.1 GB/s）
If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.
如果你尝试在默认的 llama.cpp 中用 12GB VRAM + 16GB RAM 加载一个 68GB 的 MoE 模型，你已经知道会发生什么：mmap 会把操作系统的页缓存折腾到死，每个 token 你会遇到约 1,500 次主要页面错误，生成速度缓慢至 1.47 tok/s。
Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.
然后我测试了 Strata。Strata 非常酷，如果你有 48GB–64GB 的 RAM 来将 24GB+ 的专家锁定在系统内存中，它可以达到 40+ tok/s。但在 16GB RAM 的设置下，除非你强制使用 --low-ram mmap，否则它拒绝加载；在这种情况下，由于受限于冷 SSD 读取速度，它会降至 8.11 tok/s。
So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).
所以在过去的一周里，我构建了 Betelgeuse（bel-cli / bel-server，它是 llama.cpp 的一个分支，而且我也正在为 llama.cpp 提交一个干净的上游 PR，尽管它可能不会被合并 lol）。
What it actually does under the hood:
它在后台实际做了什么：
- Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).
- Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).
- In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.
- RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.
- No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.gguf n-gram shard).
- 零门控剪枝 / 100% 位精确：所有 48 层中每层的 10 个路由专家都在全精度下计算。我们验证了 28,080 条路由器记录和 576 个生成的 token，与黄金标准默认输出逐字节一致（差异为 0.000e+00）。
- 到达松弛 VRAM 槽位池（VRAM 中约 5,000 个槽位）：所有 49 个密集层 + GDN + QSA + LM head 保持在约 3.2 GB 的 VRAM 中。剩余的 VRAM 持有约 5,000 个专家槽位，分布不均匀（早期层 0–4 上有 240 个槽位/层，因为这些层有 0ms 的计算余量；中间层有 92 个；尾部层有 60 个）。
- 流内 GPU 门铃 + 1-拆分 CUDA 图重放：与其让 ggml_backend_sched 对每个 token 分裂图 48 次以询问 CPU 哪些专家未命中（这会在 cudaStreamSynchronize 中消耗约 360ms/token），自定义的流内 CUDA 内核可在设备端以约 2 µs 解析命中，仅在专家缺失时触发映射的主机门铃。每层 token 嵌入（per_layer_token_embd）也暂存至 CUDA0，使得整个解码过程作为单次 CUDA 图重放运行。
- RAM 受害者缓存 + io_uring 页面调度器：当专家从 VRAM 池中驱逐时，它会进入专用的主机 RAM 受害者缓存，而不是被丢弃。VRAM 和 RAM 保持 100% 互斥，而不是在 OS 页面缓存中重复相同的热门专家。物理 NVMe 读取量从约 71 MB/token 降至 12.43 MB/token。
- 磁盘上无重复的 35GB 专家膨胀：与其在 Shard 1 .gguf 内部保留 35GB 的死专家张量，同时在 experts.bin 中再保留另一个 35GB，我们将专家载荷从 Shard 1 中剥离，因此主 GGUF 仅为 3.2 GB（Dense.gguf）+ 35.3 GB（experts.bin，4KB 对齐的单次读取块）+ **26.8 GB（**PLE.gguf n-gram 分片）。
Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G)
测量结果（RTX 3060 12GB + 16GB DDR4，MemoryMax=12G）
| Engine | Cold Decode (Run 1) | Warm Decode (Run 2) | Peak Bursts | Prompt Processing (PP) | Physical SSD Reads | Output Parity |
|---|---|---|---|---|---|---|
| Stock llama.cpp (mmap) | 1.47 tok/s | 1.63 tok/s | ~2 tok/s | 14–20 tok/s | 208–312 MiB/tok | Baseline |
| Strata (--low-ram mmap) | 8.11 tok/s | 8.11 tok/s | ~9 tok/s | 34.6 tok/s | ~105 ms pool wait | ~0.022 KL |
| Betelgeuse (bel-server) | 16.80 tok/s | 22.03 tok/s | Up to 48 tok/s | 16.2 – 19.3 tok/s | 12.43 MiB/tok | 100% Byte-Exact |
| 引擎 | 冷解码（第 1 轮） | 热解码（第 2 轮） | 峰值突发 | 提示处理 (PP) | 物理 SSD 读取 | 输出奇偶校验 |
|---|---|---|---|---|---|---|
| Stock llama.cpp (mmap) | 1.47 tok/s | 1.63 tok/s | ~2 tok/s | 14–20 tok/s | 208–312 MiB/tok | 基线 |
| Strata (--low-ram mmap) | 8.11 tok/s | 8.11 tok/s | ~9 tok/s | 34.6 tok/s | ~105 ms 池等待 | ~0.022 KL |
| Betelgeuse (bel-server) | 16.80 tok/s | 22.03 tok/s | 高达 48 tok/s | 16.2 – 19.3 tok/s | 12.43 MiB/tok | 100% 字节精确 |
On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.
在热运行期间，它在长代码提示下保持 21.8–22.03 tok/s 的持续速度；当生成遇到所有 10 个路由专家都驻留在 VRAM/受害者池中时，它会飙升至 48 tok/s。
Links & What's Next
链接 & 下一步
- GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]
- HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS
- Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]
- Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
- Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.
- GitHub（Betelgeuse Engine — bel-cli & bel-server）：[https://github.com/Betelgeuse-AI/Betelgeuse]
- HuggingFace（预打包 Alpha-Orionis-XS 3.2GB Dense + Experts 捆绑包）：https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS
- 上游 llama.cpp PR：[https://github.com/ggml-org/llama.cpp/pull/31469]
- 即使在巨大的 Q8_K_XL 量化模型（总计约 190GB）上，bel-server 在同一台 12GB 3060 + 16GB DDR4 机器上也能以 8 tok/s 冷启动速度和 12 tok/s 热速度运行（突发可达 21 tok/s），而 stock llama.cpp 会立即 OOM 或爬行于 <0.1 tok/s。
- 没有专家被丢弃或截断（在28,080条路由器记录和576个token上，与未修改的原始版本相比差异为0.000e+00）。性能提升源于消除了每层48次的CPU/GPU调度器分割（围绕共享专家的KICK/WAIT门铃机制），使RAM victim cache严格独立于VRAM（MOE_DROP_MMAP），并通过升级后的oracle + 4KB对齐的io_uring包预加载冷缺失。
What I'm working on next:
我接下来要做的事情：
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.
我们目前正将这种流式处理 + victim cache 架构移植到 DeepSeek V4.1 Flash (1017)——早期开发中的测试在同一硬件类别上已达到约25 tok/s的速度。
Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)
欢迎提问或在基准测试中挑刺！请给我们点个Star :)
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力