HuggingFace发布MoE训练手册：基于Megatron-LM的实操指南
The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting [P]
MoE训练是当前的工程热点，这份手册提供了基于Megatron-LM的可复现参数、代码片段和避坑指南，直接引用了80万GPU小时的实战数据，对做底层训练优化的同学极具参考价值。
Handbook: https://huggingface.co/spaces/huggingface/moe-blogpost
手册：https://huggingface.co/spaces/huggingface/moe-blogpost
Hello, I'm Nouamane, one of the authors. I'm at Hugging Face and also worked on the Ultra-Scale and Smol Training Playbooks, if you've read those.
你好，我是 Nouamane，本文作者之一。我在 Hugging Face 工作，也曾参与编写《超大规模训练指南》和《轻量级训练指南》，如果你读过那些内容的话。
This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline. Chapter 3 takes the MoE layer apart (router, routing, load balancing, experts, block layout, training, systems). For each part we give the Megatron-LM code, what to watch, our recommendation, and what frontier MoEs chose. Chapter 4 follows the model into serving and RL. Chapters 3 and 4 need no GPU.
本手册面向那些已经训练过稠密模型并打算接下来训练 MoE（混合专家）的人，或者对 MoE 训练原理感到好奇的人。第 2 章是 Megatron-LM 的运行手册：测试硬件、构建镜像、训练一个稠密基线模型，然后在单个节点上训练第一个 MoE 模型，并与该基线进行对比。第 3 章拆解了 MoE 层（路由器、路由策略、负载均衡、专家、块布局、训练、系统）。对于每个部分，我们提供了 Megatron-LM 代码、注意事项、我们的建议以及前沿 MoE 模型的选择。第 4 章将模型延伸至推理服务和强化学习阶段。第 3 章和第 4 章无需 GPU。
When a result comes from our runs, we say so. Claims about frontier models cite their paper, config or code, and the toy simulations are labelled as such. The lessons come from about 800k GH200 GPU-hours allocated to MoE training on the Alps cluster at CSCS.
当结果来自我们的实验运行时，我们会明确说明。关于前沿模型的声明均引用其论文、配置或代码，玩具模拟则标注为“玩具模拟”。这些经验教训来源于在 CSCS Alps 集群上分配给 MoE 训练的约 80 万小时 GH200 GPU 时间。
Released: the handbook, its 55 figures as a gallery (https://huggingface.co/spaces/huggingface/moe-gallery), the scripts (https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139), and every recommended flag in one Megatron-LM block.
已发布：手册本身、55 张图表组成的画廊（https://huggingface.co/spaces/huggingface/moe-gallery）、脚本（https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139），以及所有推荐参数在一个 Megatron-LM 代码块中汇总。
Happy to answer questions. 🤗
欢迎提问。🤗
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力