GLM-5.2 744B MoE模型在25GB内存消费级机器上运行

GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine 👌

Very slow, but possible.

MoE changes the memory requirement because each token activates about 40B parameters, not the full model.

Colibrì keeps the dense tensors resident in RAM, including attention, embeddings, and shared components.

Those resident tensors take about 9.9GB after int4 quantization.

The routed experts stay on local NVMe storage as a roughly 370GB int4 model.

During decoding, the router selects experts for each MoE layer and token.

The runtime then loads only those selected expert weights from disk.

This works because inactive experts do not need to occupy RAM.

The main drawback is throughput, not correctness or model size.

A cold token can require about 11GB of expert reads across 75 MoE layers.

That creates very low cold decoding speed, around 0.05 to 0.1 tokens per second.

The system reduces this cost with LRU expert caching, pinned hot experts, and OS page cache reuse.

Compressed MLA KV cache also lowers memory pressure during long context generation.

MTP speculative decoding can improve tokens per forward once the cache is warm.

The same speculation can hurt cold-cache latency because draft verification may load extra experts.

So the design is only possible because MoE sparsity reduces active parameters per token.

The cost is that SSD bandwidth, cache hit rate, and CPU matmul speed dominate latency.