SemiAnalysis深度解析GLM-5.3稀疏注意力与推理成本优化

How Sparse Attention Affects DRAM/NAND Memory

How does sparse attention affect the TAM of memory, including HBM and NAND? Sparse attention selects top-k most relevant tokens to attend to, reducing the memory consumption and bandwidth requirements during the core Scaled Dot-Production Attention (SDPA) operation. However, the efficiency improvement doesn’t directly translate to overall memory savings in practice. Concretely, the top-k selection operation typically requires the full context to be in HBM, so sparse attention doesn’t eliminate the memory capacity bottleneck.

Sparse attention throughput is bottlenecked by memory capacity. Source: HiSparse

To overcome this limitation, the SGLang team designed HiSparse, a hierarchical memory system that proactively offloads KV cache entries from device HBM to host DRAM. HiSparse behaves like an LRU (Least Recently Used) cache, where it loads tokens from DRAM to HBM upon top-k selection cache miss, and it offloads tokens from HBM to DRAM based on the LRU eviction policy. To reduce the cache miss latency, HiSparse overlaps KV cache loading of layer N with the execution of layer N-1 (layer-wise overlapping), introduced in HiSparse’s prior work HiCache.

Layer-wise overlapping. Source: CachedAttention

With HiSparse, SGLang greatly boosts throughput at high concurrency and long context scenarios, at the cost of top-k cache miss I/O overhead.

Sparse attention reduces KV cache memory and bandwidth requirements at the SDPA operation, but it does not reduce the overall memory capacity usage. In addition, HiSparse shows that system optimizations can overcome sparse attention’s memory capacity limitations, so sparse attention memory profile alone cannot sufficiently portray the full picture of system KV cache efficiency. To provide a more holistic picture of serving sparse attention models, here we explain the design of Z.ai’s GLM-5 model series, and how they are served in practice.

GLM5.3 Agentic Inference Serving

In our realtime benchmark board InferenceX, GB300 delivers the lowest modeled serving cost in our comparison at a response speed of 150 tokens per second. GB300 handles more tokens per GPU, and its higher hourly cost still leaves it ahead of GB200 on cost efficiency. The GB300 result uses Dynamo-TRT-LLM, while the GB200 result uses Dynamo-SGLang.

We use the September 17 AgentX snapshot results to help illustrate the serving implications of GLM-5.3. We estimate costs using InferenceX’s model of owning and operating infrastructure at hyperscaler scale. Total tokens include input and generated output, including cached input history reused across turns. Streaming speed is measured using p90 interactivity; time to first token is evaluated separately.

At 150 tokens per second, GB200 costs approximately $0.044 per million total tokens, compared with $0.238 for MI355X running ATOM. That is about 82% lower cost at the same streaming-speed target. These values are estimated from the measured curves, rather than from a separate test at exactly 150 tokens per second.

For this comparison, we use InferenceX’s estimated cost of owning and operating the infrastructure at hyperscaler scale. Total tokens include input and generated output, including input history reused from cache.

Source: SemiAnalysis InferenceX current InferenceX dashboard; interpolation method; GB300 source run; MI355X source run.

Among the Dynamo-SGLang results, GB300 serves roughly 13950 total tokens per second per GPU here, a 17.5% throughput advantage versus 11873 total tokens for GB200. However, we assume $2.31 per GB300 GPU hour versus $1.86 for GB200. The higher hourly cost more than offsets GB300’s throughput lead at this target.

In the comparison with AMD, GB200 costs about 52% less than ATOM at 100 tokens per second, 62% less at 125, and 82% less at 150. Across these three targets, the gap widens as response speed increases.

Counting only generated output, GB200 costs $5.92 per million output tokens at a response speed of 150 tokens per second, which is 79% less than $28.52 for MI355X ATOM. This metric is useful to analyze cost under agentic workloads, where agents repeatedly reuse long input histories while generating relatively little new text.

Source: SemiAnalysis InferenceX; current InferenceX dashboard

The GB200 measurements on either side of the 150 token per second target have p90 TTFT (time to first token) of 14-19 seconds, compared with 1.3-1.7 seconds for MI355X ATOM. Some of the GB200 lowest cost runs have much longer waits for the first token.

We therefore make a second comparison using only configurations that were actually tested and achieved at least 150 tokens per second with p90 TTFT below two seconds. The chart below shows B200 with Dynamo-SGLang at $0.0666 per million total tokens versus $0.1265 for MI355X with SGLang. B200 still costs about 47% less with the first-token limit.

Source: SemiAnalysis InferenceX B200 run 442071; GB300 run 441597; MI355X run 442075

Optimizations

In GLM’s 5.2 / 5.3 model, its attention architecture reduces memory demands, and long running agents still need to retain earlier conversation history. GLM reduces the cost of handling this history in two ways: KV compression reduces the cached state stored per token, and sparse attention reduces how much of that state each attention operation reads. Then the serving engine would determine where to store the cache and how to retrieve it efficiently.

These B200 results show how much reuse can happen outside of GPU memory. When concurrency increases from 8 to 16 requests, the share of prompt tokens reused from GPU memory falls from 90.3% to 54.8%. Much of that reuse shifts to host memory, which rose from 6.0% to 40.3%. Much of the decline in GPU cache is offset by reuse from host memory, keeping the overall cache hit rate above 95% at all concurrency levels.

Source: SemiAnalysis InferenceX; InferenceX B200 rows 440962, 440961, and 440959; workflow run 33683520699

Beyond cache management, changes to how the serving engine executes prefill and decoding can also change performance. Two separate GLM-5.2 studies illustrate this:

• vLLM keeping the first decoding step on a consistent CUDA graph execution path reduced average TPOT (time per output token) from about 40 ms to 22 ms in an NVFP4 study.
• On a different 8 MI355X long context workload, ATOM engine processing prefill in chunks across pipeline stages delivered 98% higher total throughput and reduced median TTFT from 28.6 to 8.7 seconds.

For GLM5.3 on TileRT, AMD MI355X was supported first. TileRT is a low-latency LLM inference engine for that compiles decoding into a single persistent kernel, reducing launch overhead and overlapping computation, memory access, and communication.

It prioritizes faster token generation per user, rather than maximum batched throughput, while vLLM handle prefill.

On AgentX, FP8 TileRT MI355X achieves 2x the P90 interactivity than the best FP4 MI335X config. Comparing to GB300 NVL72, it brings a 40% uplift in interactivity. These are serving metrics that used to require specialized hardware.

However, TTFT is suboptimal and there is still room for development, such as optimizing KV transfer, FP4 support, and larger batch sizes.

DeepSeek Sparse Attention

GLM-5 (and GLM-5.x models) is a 744B total, 40B active-parameter mixture-of-experts model. For every token, it has 1 shared expert and is routed through 8 out of 256 experts, which is sparsity 32. It features DeepSeek Sparse Attention, which we will discuss in this section.

DeepSeek Sparse Attention (DSA), introduced in DeepSeek V3.2, consists of two components: A lightning indexer that selects top K tokens, and a sparse Multi-Latent Attention (MLA).

Lightning Indexer