SemiAnalysis：推理计算与数据移动的工程解析

Mixture of Experts, now widely used in frontier models, has changed both the structure of serving and the economics of useful inference. It did more than increase parameter count. It changed which tensors are active for each token, what must remain close together, which transfers need strong local bandwidth, which can tolerate a weaker network link, and how memory movement, storage, and scheduling contribute to useful throughput.

The best place to begin is the service as a whole. Inference runs inside a cluster coordinated by an orchestration layer such as NVIDIA Dynamo, Mooncake, or a custom scheduler. These work closely with inference servers like vLLM or SQlang, and those also have their own orchestration features. This article will not go into the details of how you work with the orchestration software or which you should choose. The article aims to provide an overview of the process and reasons for various features you may use.

A user (or their agent) starts a conversation with a request (query), and the conversation may continue after answers with more requests. The request- answer is a “turn”. In modern AI systems the user is often running a in a client application (whether GUI or command-line oriented) and some of the answers coming back from the AI are intercepted by that application as instructions for it to run, things like editing your code or searching your company guidelines on an HR question.

The AI system can also do some of those things itself from its data center, for example it may search the web. The results from these intercept tool actions are also returned to the AI as requests, creating more turns. The server keeps a context (often referred to as KV cache or just cache) which is the distillation of the session, allowing each new request, from user or from tool, to be properly interpreted for overall forward progress. There can be thousands of turns per hour when the user launches an agent on a long-running task. The conversation can also pause and resume hours or even days later.

When a request arrives at the inference servers the orchestration puts it into a queue, while will feed into an input-fill worker, along with any context from system prompts and prior turns in the conversation. The query is then converted into new context appended to the conversation. That new context state is then moved to a decode worker. Decode repeatedly reads the accumulated state, generates answer tokens, and those answer tokens are also appended to the conversation state. The process may loop through tools, agents, users, and further input-fill work. A model worker running on GPUs is only one part of this token factory; storage, networking, and orchestration connect its stages.

Inside the workers, four operating regimes are worth distinguishing from the beginning:

• Prefill, where the initial block of new tokens is processed together.
• Midfill, where a continuation request is appended to an existing cached context.
• Decode attention, where the context is used to generate the basics of newly generated tokens.
• Decode experts, where each basic new token is refined by a selection of experts a large total set of possible experts.

These regimes place different demands on compute, memory, and networking. Prefill and midfill are closely related, with Prefill being a Midfill with zero prior context. However, Prefill is a common special case, corresponding to one-shot queries like classic “”chatbot” use, and can be optimized a bit differently than Midfill. Prefill reaches high arithmetic intensity (ratio of computation to data movement) and does not need to wait for context to be located and read. Midfill begins from an existing KV cache for prior state and appends a new request input sequence. This generally has a more moderate arithmetic intensity than prefill because there are fewer new tokens and more existing data to move per token.

Decode attention and decode expert work are generally low arithmetic intensity since there are few new tokens compared to the prior context or expert weights, which at decode state are data that needs to be moved. Meanwhile, across all of them, the tensors for attention and for experts are the same regardless of token value, so there is benefit to sharing a single reading of a tensor with as many tokens as can be queued to use it, even tokens coming from unrelated requests by other users which simply happen to be running at the same time and which are networked closely enough to share. Treating all four mode as the same workload gives away much of the structural advantages that MoE models offer in terms of different intensity and different sharing patterns.

In this document we will treat the 4 stages separately, and sometimes assume they are disaggregated. There are tradeoffs between aggregated (where the models stay on one server, which reconfigures as the request progresses between the stages) and disaggregated where the orchestrator may find a free slot on another machine already configured appropriately. Generally we will talk about the stages as disaggregated, with aggregation being a special case of always making the current node available when a previous stage finishes. There will be a section on the trade-offs at the end, after the work to be done has been more completely introduced.

A transformer model is a deep stack of repeating layer groups. A group may contain one kind of layer, or it may contain a full-attention layer together with several linear, local, selective, or otherwise optimized-attention layers (this is a hybrid group of layers). Within each layer, attention, shared transformations, routing, selected experts, and recombination occur in sequence. Those steps are known in advance and repeat continuously, allowing the same hardware resources to serve different parts of the flow at different moments.

The machines can also be described equally simple recurring units. A node is a group of accelerators, typically a tray in a rack, each accelerator pairs compute with local fast memory. A node is coordinated by a CPU and connected to other nodes through NICs (network interface controller). Nodes stack into racks. A rack may act as one scale-up domain when very large tensors are , several smaller scale-up islands, or a set of pipeline stages. Datacenter networking then joins bounded workers to storage and orchestration rather than participating in every inner tensor operation.

This essay follows that progression. It begins with the global service and the operating regimes, then opens the model into wide layer-group slices and maps them onto accelerator nodes and racks. From there it develops pipeline, tensor, and expert parallelism; the limit imposed by KV-cache memory; the different value of batching in prefill, midfill, and decode; and the role of scheduling in a many-rack token factory.

Figure 1. Overview of model stages in an inference service.

1. Orchestration delivers requests via queues into pools of workers

An inference service is a cluster where prior requests and immutable state are stored when idle and then queued for delivery to specialized workers when a new request pulls in context or a prior request reawakens for continuation. The incoming queue is not yet a batch. Each request carries prompt tokens, references to reusable context, service objectives, and often an existing conversation or agent state. An orchestrator may place it into a prefill, midfill, or decode batch, and the request can join or leave that batch independently as it completes, making that capacity available for new work. Continuous batching works best when the scheduler knows which workers have slots that match the request’s context length and service objective.