从单卡3090到20张DGX Spark:家庭本地大模型集群搭建与调优复盘
From 1x3090 to 20 DGX Sparks: my house fuses were the first bottleneck
极具参考价值的一手工程实践,完整展示了家庭环境突破算力与电力限制、搭建大规模本地推理集群的路径与踩坑经验,适合有类似需求的开发者参考。
From the first LLaMA 33B I knew I wanted that magic-like intelligence locally, mine, so nobody could take it away when I needed it. I bought a 3090 for my home PC. Then LLaMA 65B appeared and I was dazzled, it looked like it had all the knowledge in the world. I made two copies, one local and one on my Synology NAS RAID, so I'd never lose it, and bought a second 3090 to run it. I was happy for a year with small coding tasks on LLaMA and Qwen models.
从第一个 LLaMA 33B 开始,我就渴望那种如魔法般的智能能本地运行、属于我,这样在我需要时没人能夺走它。我为家用 PC 买了一块 3090。随后 LLaMA 65B 出现,令我眼花缭乱,它看起来仿佛拥有全世界的知识。我做了两份副本,一份在本地,一份放在我的 Synology NAS RAID 上,以防丢失,并购买了第二块 3090 来运行它。在 LLaMA 和 Qwen 模型上进行小型编码任务的一年中,我很满意。
Then DeepSeek 671B MoE appeared. Wow, frontier level at home. I upgraded to a Threadripper with 512GB DDR4 and ran it at 8 t/s with experts offloaded to RAM, or Qwen 235B at 10-12 t/s when I wanted speed. I used these for real coding at my job, in OpenWebUI.
接着 DeepSeek 671B MoE 出现了。哇,这是家庭环境下的前沿水平。我升级到了配备 512GB DDR4 的 Threadripper,并将专家模块卸载到 RAM 中以 8 t/s 的速度运行它;或者当我追求速度时,以 10-12 t/s 的速度运行 Qwen 235B。我在 OpenWebUI 中使用这些模型进行工作中的实际编码。
Then agentic coding took off and this was too slow. At 100k context generation speed halved and prefill made it a beautiful yet agonising experience. So: 16x3090 across P620-based nodes on a 100Gbit network. It ran MiniMax M2, Qwen 235B and even Qwen 397B, as good as anyone could desire. I built an entire paid project with 397B in OpenCode. But bigger models were out of reach, and the house circuit said no: the fuses blew whenever the rig and the electric oven ran together. Heat and stability were issues too.
随后代理式编码(agentic coding)兴起,这太慢了。在 100k 上下文下生成速度减半,预填充过程带来了一种美丽却令人煎熬的体验。于是:在基于 P620 的节点上部署了 16x3090,通过 100Gbit 网络连接。它运行 MiniMax M2、Qwen 235B 甚至 Qwen 397B,效果达到任何人所能期望的最佳水平。我在 OpenCode 中用 397B 构建了一个完整的付费项目。但更大的模型遥不可及,而且家里的电路不允许:当服务器集群和电烤箱同时运行时保险丝就会熔断。散热和稳定性也是问题。
Next came 4x ASUS GB10, after I read they can be linked (3 was the biggest supported config). 397B at 30 t/s on 400W, versus 50-60 t/s at 6kW, rock solid and almost silent. A dream come true. I built two more projects with it. Then MiMo 2.5 Pro and Kimi 2.6 appeared, smarter and more productive. I found no published solution for an 8-node cluster, but I still bought four more GB10s and made it work. 397B ran at FP8 instead of INT4, and 20% faster. I posted the first MiMo 2.5 Pro and Kimi 2.6 solutions on 8xSparks on the NVIDIA forum. I liked the result so much that I talked my older brother into buying his own 8x GB10, so he could run the best open models locally too, in privacy, without depending on API availability and rising costs.
接下来是 4x ASUS GB10,在阅读到它们可以链接后(3 是当时支持的最大配置)。400W 功耗下 397B 达到 30 t/s,而之前 6kW 功耗下为 50-60 t/s,极其稳定且几乎无声。梦想成真。我用它构建了另外两个项目。随后 MiMo 2.5 Pro 和 Kimi 2.6 出现,更聪明、更高效。我没有找到针对 8 节点集群的已发布解决方案,但我还是又买了四台 GB10 并让它跑通了。397B 以 FP8 而非 INT4 运行,速度快了 20%。我在 NVIDIA 论坛上发布了首批在 8xSparks 上运行 MiMo 2.5 Pro 和 Kimi 2.6 的解决方案。我非常喜欢这个结果,以至于说服了我的哥哥也购买了一套自己的 8x GB10,以便他也能在隐私保护下本地运行最好的开源模型,而不依赖 API 的可用性和不断上涨的成本。
His house is a 5-minute walk from mine. When Kimi K3 (2.8T) appeared, biggest and smartes open weights model, we joined the clusters: two 8x clusters for daily use, or one 16x when we want the biggest model at home. After some work I published the first working solution for Kimi K3 on 16x Sparks on the NVIDIA forum. Through multiple iterations, it went from an unusable 7 t/s at 100k context to a fairly usable 20 t/s at 300k.
他的家离我家只有 5 分钟步行路程。当 Kimi K3 (2.8T) 出现时——这是最大、最聪明的开源权重模型——我们合并了集群:日常使用两个 8x 集群,或者当我们想要在家运行最大模型时使用一个 16x 集群。经过一些工作,我在 NVIDIA 论坛上发布了首个在 16x Sparks 上成功运行 Kimi K3 的解决方案。经过多次迭代,它在 100k 上下文下的速度从无法使用的 7 t/s 提升到了在 300k 上下文中相当可用的 20 t/s。
Now we're adding 4 more Sparks, so a smaller, faster model (GLM 5.3 Flash) runs 24/7 while the big cluster runs either GLM 5.3 on 8x plus MiMo 2.6 Pro on the other 8x, or 16x Kimi K3, or Qwen 3.8 2.4T.
现在我们额外增加了 4 个 Spark,因此一个更小、更快的模型(GLM 5.3 Flash)全天候运行,而大型集群则运行 8x GLM 5.3 加上另外 8x 的 MiMo 2.6 Pro,或者 16x Kimi K3,或者 Qwen 3.8 2.4T。
I'm always tuning speed on the big models and rebuilding vLLM/SGLang images, so always-on smaller cluster made sense, why? Because for all my work projects and my vllm/sglang personal projects, I chose to use only local hosted models, I never paid a comercial model subscription, not because of the cost, but, because of my strong confidence in local models future. They arrive October 2, along with 4 more Sparks for my younger brother, who got caught by the same local AI microbe :)
我一直在调整大型模型的速度并重建 vLLM/SGLang 镜像,因此保持小型集群始终在线是有道理的,为什么呢?因为在我的所有工作项目以及我的 vllm/sglang 个人项目中,我选择仅使用本地托管模型,从未购买过商业模型订阅,并非因为成本原因,而是出于我对本地模型未来的强烈信心。它们将于 10 月 2 日到来,同时还有 4 个额外的 Spark 留给我的弟弟,他也染上了同样的本地 AI 病毒 :)
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力