MoE推理延迟增加的原因与优化策略
A good technical LLM interview question:
做LLM部署的同学必看，清晰拆解了MoE推理中通信开销如何抵消计算优势，并给出了具体的工程优化思路，非常硬核。
A good technical LLM interview question:
一个好的技术大模型面试问题：
You replace a dense feed-forward layer with a top-2 MoE.
你用 top-2 MoE（混合专家）替换了一个稠密的前馈层。
The profiler confirms that the MoE executes fewer FLOPs per token.
性能分析器确认，MoE 每个 token 执行的浮点运算次数（FLOPs）更少。
However, end-to-end inference latency has increased instead of decreasing.
然而，端到端推理延迟反而增加了，而不是减少。
Why did this happen?
为什么会发生这种情况？
(answer below)
（答案见下文）
A standard Transformer applies the same feed-forward network to every token in a layer.
标准 Transformer 对层中的每个 token 应用相同的前馈网络。
An MoE layer replaces that network with multiple experts. Each expert is a feed-forward network with its own weights.
MoE 层用多个专家替换了该网络。每个专家都是一个具有自己权重的前馈网络。
A router reads each token's hidden-state vector, scores the available experts, and selects the top two. It also produces a routing weight for each selection.
路由器读取每个 token 的隐藏状态向量，对可用的专家进行评分，并选择前两个。它还为每次选择生成一个路由权重。
Only those selected experts process the token. The model therefore contains many expert weights while executing only a small subset for each token.
只有被选中的专家处理该 token。因此，模型包含许多专家权重，但为每个 token 只执行其中的一小部分。
The serving system must move activations when the selected experts are stored on different GPUs.
当被选中的专家存储在不同的 GPU 上时，服务系统必须移动激活值（activations）。
The visual follows a batch that begins on GPU 0.
图示遵循一个从 GPU 0 开始的批次。
The router returns two expert IDs and two routing weights per token. One selected path is shown for each example token to keep the diagram readable.
路由器为每个 token 返回两个专家 ID 和两个路由权重。为了保持图表的可读性，每个示例 token 仅显示一条选中路径。
Token T1 selects E1 on GPU 0. Its activation remains in local GPU memory, where E1 processes it.
Token T1 在 GPU 0 上选择了 E1。其激活值保留在本地 GPU 内存中，由 E1 进行处理。
T2 selects E4 on GPU 1 in the same server. GPU 0 transfers the activation through NVLink, a high-bandwidth GPU-to-GPU connection. NVSwitch connects several GPUs through this fabric.
T2 在同一服务器的 GPU 1 上选择了 E4。GPU 0 通过 NVLink（一种高带宽 GPU 间连接）传输激活值。NVSwitch 通过这种互连结构连接多个 GPU。
T3 selects E7 on GPU 2 in another server. Its activation crosses the cluster network through InfiniBand or Ethernet.
T3 在另一台服务器的 GPU 2 上选择了 E7。其激活值通过 InfiniBand 或以太网跨越集群网络传输。
To be clear, the system transfers activation vectors, which are the numerical representations produced for a token by the preceding Transformer operations.
明确地说，系统传输的是激活值向量，即前序 Transformer 操作为 token 生成的数值表示。
Each destination GPU runs its expert and returns the output to GPU 0.
每个目标 GPU 运行其专家，并将输出返回给 GPU 0。
GPU 0 multiplies both selected expert outputs by their routing weights, adds them together, and restores the original token order.
GPU 0 将两个选中专家的输出分别乘以其路由权重，将它们相加，并恢复原始的 token 顺序。
Different tokens can select different expert pairs. Across a batch, those assignments may involve every GPU storing experts.
不同的 token 可以选择不同的专家对。在一个批次中，这些分配可能涉及所有存储专家的 GPU。
Each GPU may therefore send activations to several GPUs while receiving activations for its own experts. This exchange is called all-to-all communication.
因此，每个 GPU 可能向多个 GPU 发送激活值，同时接收为其自身专家提供的激活值。这种交换称为 all-to-all 通信。
And there are several ways to optimize this.
并且有多种方法可以优化这一点。
For instance, keeping frequently selected experts within the same server reduces remote traffic.
例如，将频繁选中的专家保持在同一服务器内可以减少远程流量。
Balanced routing prevents one GPU from delaying the layer.
平衡的路由可以防止某个 GPU 延迟整个层的处理。
Communication overlap allows local expert computation to continue while remote activations move.
通信重叠允许在远程激活传输的同时继续本地专家计算。
In this setup, sparse routing reduces expert FLOPs, but token dispatch and network communication erase part of that saving.
在这种设置下，稀疏路由减少了专家的浮点运算次数（FLOPs），但令牌分发和网络通信抵消了部分节省。
If you want to dive deeper, I recently covered MoE inference engineering, including routing, expert placement, token dispatch, load balancing, communication overlap, memory, quantization, and offloading.
如果你想深入了解，我最近讨论了 MoE 推理工程，包括路由、专家放置、令牌分发、负载均衡、通信重叠、内存、量化和卸载。
Read my article below.
阅读我的下文文章。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力