跳到主内容
@wquguru
精选88r/LocalLLaMA（Reddit）技巧与观点

利用二手BC-250矿卡集群运行本地大模型的性能优化实践

Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards

原文
发到 X
推荐理由

极具复用价值的硬核工程实践，详细拆解了低成本异构算力下的推理加速技巧，适合有闲置硬件或追求极致性价比的本地部署玩家参考。

This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the future. I am running 4 BC-250 boards connected by 2.5gb ethernet adapters to a 2.5gb switch using llama with vulkan and RPC. All the boards are inside an asrock 4u12g bc250 case that they originally came in. After starting out at 30 tok/s on flash next Iq2 and 80 tok/s with 3.6 35B I pointed Claude Code at the cluster and we are now running a better quant next flash as well as maintaining 60-70 tok/s (~150 ppt) at context. The 4 boards can run 2 instances of 3.6 35B at 145tok/s (~450 ppt) starting and going below 100 tok/s at 150k context. These boards cost me less than $100usd each when i bought them but i have seen deals recently on aliexpress for $150usd. I now have outlet power monitors and under load with next flash it pulls 600 - 700 W with the 2 instances of 3.6 35B it pulls 800 - 900 W. The next step for me is to use AIO coolers for each board as now I have been seeing some thermal throttling as the boards are better utilized . I will also 3d printing and building a AIO cooler lid for the case (upgrading from the cardboard plenum lol) now that I am going to keep the cluster. Before anyone suggests upgrading to a m.2 networking solution to get more bandwidth the limiting constraint is latency as its not sending a lot of data between boards at a time.

这将是我对 bc-250 集群的第三次更新，也是我首次涉足本地 AI，整个过程非常有趣。这已经从一次周末实验变成了我认为未来会持续使用的工具。我使用 llama 配合 Vulkan 和 RPC，通过 2.5Gb 以太网适配器将 4 块 BC-250 主板连接到一个 2.5Gb 交换机上。所有主板都放在它们原装的 ASRock 4U12G BC250 机箱内。起初在 Flash Next IQ2 模型下速度为 30 tok/s，而在 3.6 35B 模型下达到 80 tok/s。随后我将 Claude Code 指向该集群，我们现在运行着性能更好的量化版 Flash Next，同时在上下文窗口下保持 60-70 tok/s（约 150 ppt）的速度。4 块主板可以以 145 tok/s（约 450 ppt）的速度运行两个 3.6 35B 实例，并在 150k 上下文时降至 100 tok/s 以下。我购买这些主板时每块花费不到 100 美元，但最近我在 AliExpress 上看到有 150 美元的优惠。我现在安装了插座功率监测器，在负载下运行 Flash Next 时功耗为 600-700 W，而运行两个 3.6 35B 实例时功耗为 800-900 W。我下一步的计划是为每块主板使用一体式水冷散热器，因为随着主板利用率提高，我观察到了一些热节流现象。既然我决定保留这个集群，我还将 3D 打印并制作一个用于机箱的一体式水冷散热盖（从原来的纸板风道升级 lol）。在任何人建议升级到 M.2 网络方案以获得更高带宽之前，需要指出的是限制因素是延迟，因为板间同时传输的数据量并不大。

BC-250 Info: https://elektricm.github.io/amd-bc250-docs/

BC-250 信息：https://elektricm.github.io/amd-bc250-docs/

Here is the Claude summary of optimizations:

以下是 Claude 对优化措施的总结：

Next Flash

Flash Next

The current build runs a 100K context:

当前构建支持 100K 上下文：

  • 59.4 / 65.2 tok/s on that request (first / second run)
  • 66.6-68.3 tok/s behind a 5K-token prompt (sampled, 512 tokens)
  • 72.6 tok/s on one browser generation of 50,357 tokens
  • 在该请求中，第一次运行/第二次运行的速度分别为 59.4 / 65.2 tok/s
  • 在 5K token 提示词后（采样，512 tokens），速度为 66.6-68.3 tok/s
  • 在一次浏览器生成 50,357 tokens 的过程中，速度为 72.6 tok/s

Most of the work was done on the Q2_0 file first (27.9 → 68-69 tok/s on a 5K prompt) and then carried over to IQ3:

大部分工作首先在 Q2_0 文件上进行（在 5K 提示词下从 27.9 提升至 68-69 tok/s），然后迁移到 IQ3：

  • Speculative decoding with the model’s own MTP draft head, run as a chain of passes. Pass A verifies the known token. Passes of two drafts each follow one board behind (six drafts, four passes). A round ends at the first rejected draft. Adding the third pass took Q2_0 from 61 to 68 tok/s, and six drafts took an IQ3 temperature-0 edit request from 84.0 to 90.1-90.5.
  • Recorded and replayed graphs. Workers record each stage graph’s Vulkan commands once and replay them (saves 3.7-4.5 ms of host time per graph, +2%). Remote graphs’ nodes are regrouped so a layer’s independent products share a barrier group (+5.7%).
  • Exact kernels for this architecture (routed-expert top-k, two-token expert passes, recurrent-state fusion) and for the parts that grow with context: +7.3% at 62K, then 66.1 → 68.9-69.6 tok/s.
  • Context handling. Context checkpoints are copied on the boards instead of through the coordinator (665 → 18 ms each, prompt speed +60%). Prompt batches are pipelined through the boards. Context went from 65,536 to 100,096 tokens with 32-token prompt batches.
  • 使用模型自身的 MTP 草稿头进行投机解码，以一系列传递链的形式运行。传递 A 验证已知 token。随后的每次传递包含两个草稿，每个草稿落后于前一块主板一步（共六个草稿，四次传递）。一轮传递以第一个被拒绝的草稿结束。增加第三次传递使 Q2_0 的速度从 61 提升至 68 tok/s，而六个草稿使 IQ3 在温度设为 0 的编辑请求中从 84.0 提升至 90.1-90.5。
  • 录制与重放图。Worker 对每个阶段图的 Vulkan 命令仅录制一次并重放（每图节省主机时间 3.7-4.5 毫秒，提升 +2%）。远程图的节点被重新分组，使同一层的独立产品共享一个屏障组（+5.7%）。
  • 针对此架构的精确内核（路由专家 top-k、双令牌专家传递、循环状态融合）以及随上下文增长的部分：在 62K 时提升 +7.3%，随后从 66.1 提升至 68.9-69.6 tok/s。
  • 上下文处理。上下文检查点直接在板卡上复制，而非通过协调器（每个从 665 毫秒降至 18 毫秒，提示速度提升 +60%）。提示批次通过板卡流水线传输。上下文容量从 65,536 增加到 100,096 个令牌，使用 32 个令牌的提示批次。

Qwen3.6-35B-A3B Q4_K_M (HauhauCS GGUF), two boards, K/V cache q8_0

Qwen3.6-35B-A3B Q4_K_M (HauhauCS GGUF)，两块板卡，K/V 缓存 q8_0

.

  • Step one (the Flash-Next kernels, graph replay and transport, two drafts): +39% to +66% over the baseline from 120 to 99.6K prompt tokens.
  • Chain of passes with the MTP head (four drafts), plus attention that decodes a K/V tile once into shared memory: first request 105.8 → 116.0 tok/s at a short prompt, 73.0 → 95.2 at 44K.
  • Longer context. The attention mask is one limit per token instead of a cells × tokens tensor (768 MiB less on the coordinator), and prompt batches read the q8_0 K/V in place. Usable context went from about 100K to 258K (of 262,144).
  • Replay on the coordinator too. The coordinator’s own half is replayed from recorded command buffers (+2 to +4%). Draft-head checkpoints keep positions only, so a repeated request’s prompt time at 99.6K went from 932-953 ms to 142-163 ms.
  • Startup and saving. A pair starts in 32-35 s instead of 105 s, and a conversation can be saved and restored bit for bit after a restart.
  • Latest (
  • 第一步（Flash-Next 内核、图重放与传输、两个草稿）：相比基线，在 120 到 99.6K 提示令牌范围内提升 39% 至 66%。
  • 带有 MTP 头的传递链（四个草稿），加上将 K/V 图块解码一次到共享内存的注意力机制：短提示下首次请求从 105.8 提升至 116.0 tok/s，44K 时从 73.0 提升至 95.2。
  • 更长上下文。注意力掩码改为每个令牌一个限制，而非 cells × tokens 张量（协调器上减少 768 MiB），且提示批次就地读取 q8_0 K/V。可用上下文从约 100K 增加到 258K（总计 262,144）。
  • 协调器上也进行重放。协调器自身的半部分从录制的命令缓冲区中重放（提升 +2 到 +4%）。草稿头检查点仅保留位置信息，因此重复请求在 99.6K 时的提示时间从 932-953 毫秒降至 142-163 毫秒。
  • 启动与保存。一对设备启动时间从 105 秒缩短至 32-35 秒，并且重启后可以逐位保存和恢复对话。
  • 最新版本（

verified

已验证

  • ). The draft head’s catch-up pass now only writes its K/V (it was computing attention nobody read), and RPC sends are batched. At 258K: 51.1 → 53.5 tok/s on a repeated request, 51.7 → 54.4 sampled. Plain decoding at that depth is 36.4.
  • ）。草稿头的追赶传递现在仅写入其 K/V（此前正在计算无人读取的注意力），且 RPC 发送已批量处理。在 258K 时：重复请求下从 51.1 提升至 53.5 tok/s，采样值为 51.7 → 54.4。在该深度下的纯解码为 36.4。

Exactness

精确性

Weights, quantization and sampler settings are untouched. At temperature 0, the speculative rounds are verified bit for bit against plain decoding of the same build by hashing every logits row (five depths up to 99.6K on the 35B).

权重、量化和采样器设置保持不变。在温度 0 时，投机轮次通过与相同构建版本的纯解码逐位比对 logits 行的哈希值进行验证（在 35B 模型上测试了最多 99.6K 的五种深度）。

Against stock llama.cpp, the logits differ at float-rounding level (about 0.03) where kernels were rewritten. On the 35B that includes summing attention in fixed-size chunks, so a result no longer depends on how many tokens a pass carries. Stock’s own drafted rounds differ from its plain decoding by the same order.

与原版 llama.cpp 相比，在内核重写处，logits 存在浮点舍入级别的差异（约 0.03）。在 35B 模型中，这包括以固定大小块累加注意力，因此结果不再取决于单次传递携带的令牌数量。原版自身的草稿轮次与其纯解码之间的差异也处于同一数量级。

更进一步：量化金融体系

看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力

进入量化体系 →
话题#LocalLLaMA#BC-250#llama.cpp#性能优化

相似阅读

关联信息，但可能不是同一事件