MoE推理生产现实：低并发下的GEMV优化与Qwen-3.5实践

A lesser-known production reality of MoE inference:

(must-know for technical LLM interviews)

MoE models reduce expert computation by routing each token to a small subset of experts.

But routing also divides a batch of token rows into several expert-specific batches.

Their sizes depend on how many tokens enter the layer and which experts those tokens select.

This creates a systems problem that is not immediately evident from architecture diagrams.

Let's walk through it.

1) Dense transformer execution

Assume a decode batch contains 32 token rows.

A dense Transformer sends all 32 rows through the same feed-forward network. For each feed-forward projection:

> 32 token rows x the same weight matrix

The accelerator performs one 32-row matrix multiplication and reuses that projection's weights across all 32 rows.

2) MoE execution

A simplified top-1 MoE router may divide those rows like this:

• Expert 1 receives 8 rows - Expert 2 receives 3 rows - Expert 3 receives 0 rows - Expert 4 receives 11 rows - All other experts collectively receive 10 rows

Each 32-row feed-forward projection has become several smaller expert projections with different row counts.

The engine must group rows by expert, run each expert, apply routing weights, and restore the original token order.

With top-k routing, each token appears in multiple expert batches.

Sparse routing reduces expert computation, but it also fragments the original batch.

3) How grouped GEMM helps

GEMM is General Matrix Multiplication.

Each active expert has its own weight matrices. Grouped GEMM schedules several expert matrix multiplications together:

• Expert 1 rows x Expert 1 weights - Expert 2 rows x Expert 2 weights - Expert 4 rows x Expert 4 weights

This reduces launch overhead and helps the accelerator process differently sized expert batches together.

It works best when each active expert receives several rows.

4) During low concurrency

During ordinary autoregressive decoding, each active request contributes one current token to a model step.

With 32 active requests, some experts may receive several token rows.

With only one to three requests, the selected experts may barely overlap. Most active experts can receive a single row.

Each operation now resembles GEMV, or General Matrix-Vector Multiplication.

The engine loads an expert's weights to process one token vector. That leaves little opportunity to reuse those weights across rows.

Grouped GEMM can combine the launches, but it cannot turn one-row expert batches into large matrix multiplications.

5) An example

Researchers encountered this while optimizing Qwen-3.5-397B-A17B-FP8 on Ironwood TPUs.

Each token selected ten routed experts from a pool of 512. With three concurrent requests, they found that only 5% of the selected experts were duplicated across requests.

Most routed expert batches therefore contained one row.

So they built a custom kernel that prefetched selected expert weights into the TPU's fast VMEM and processed these GEMV-like operations directly.

This resulted in a 3.6x faster MoE block at concurrency one on a TPU configuration.

This does not mean grouped GEMM is inefficient. Instead, it implies that the best execution strategy depends on the expert-batch shape:

• Several rows per expert → grouped GEMM can work well - One row per expert → hardware-specific weight streaming may work better

The architecture stays the same, and concurrency and routing overlap change how the work is assigned to each expert.

To dive deeper, I wrote a full article covering this and the other engineering problems behind MoE inference, including memory, cross-GPU dispatch, expert placement, load imbalance, quantization, and offloading.

Read it below.