LLM推理瓶颈：内存带宽而非算力

A tricky LLM interview question:

You're serving an LLM in FP16 on an A100.

Token generation is quite slow so you upgrade to an H100, which has over 3x the FP16 compute.

But upgrading the GPU didn't improve the token generation speed much.

Why did this happen?

(answer below)

LLM token generation is a memory bandwidth problem disguised as a compute problem.

An A100 can do about 300 trillion FP16 operations per second. But it can only move ~2 TB per second between HBM and its compute units.

During generation, every new token requires reading the model weights and the entire KV cache from memory. While the compute finishes almost instantly, but the chip still have to spend time waiting for data.

This is why the highest-impact serving optimizations all try to tackle the same two things:

• how much data moves - and how often the GPU sits idle

The visual below covers five of them.

> Flash attention:

Standard attention writes the full attention matrix in HBM, and it grows quadratically with sequence length.

Flash Attention splits Q, K, and V into tiles, computes attention inside on-chip SRAM, and writes only the final output back.

The output is mathematically identical, but the quadratic intermediate never exists in memory, so long sequences become relatively manageable.

> Paged attention:

Naive serving reserves one contiguous memory chunk per request, sized for the maximum possible sequence length.

vLLM paper measured 60-80% of KV cache memory wasted this way.

Paged attention is inspired by virtual memory from OS concepts.

The cache stays in small fixed-size blocks mapped through a block table, allocated as generation proceeds and freed on completion, which reduces waste to under 4%.

> Continuous batching:

Static batching waits for the slowest request in a batch before admitting anything new. So one long generation keeps every finished slot idle until it completes.

Continuous batching makes scheduling decisions after every decode step instead of after every batch.

When a request finishes, a queued request takes its slot at the next decode step, and the GPU stays saturated.

> Speculative decoding:

A small draft model proposes K tokens cheaply, then the large target model verifies all K in a single forward pass, accepting matches and correcting the first mismatch.

The rejection sampling scheme guarantees an output distribution identical to running the large model alone, so the speedup costs zero quality.

> Kernel fusion:

Every GPU operation runs as its own kernel by default, and each one writes its result to HBM for the next kernel to read back.

Kernel fusion merges LayerNorm, MatMul, and activations into one kernel that keeps intermediates in registers and SRAM.

Compilers like TensorRT-LLM and torch(.)compile do this automatically from the model graph.

That said, all five techniques above are built for one model owning the whole GPU.

Paged attention and continuous batching fit more concurrent requests when more memory is available for KV cache blocks.

So vLLM pre-allocates nearly all GPU memory at startup and turns it into one big block pool.

But production stacks rarely run just one model.

A typical pipeline also runs several small specialized models like embedders, rerankers, and extractors alongside the LLM.

Serving each one the standard way requires a separate vLLM or TEI process, and every process claims its memory upfront with no visibility into the others.

So a 600M-parameter embedder reserves a GPU it will never fully use, and models that could comfortably share one card end up spread across many.

This is also why moving to small yet better models does not lead to substantial cost savings alone.

Superlinked's open-source inference engine (SIE) solves this by running all of those models behind one API with shared GPU memory and on-demand model loading, so the card is actually shared instead of claimed.

GitHub repo → https://github.com/superlinked/sie

(don't forget to star it ⭐ )

I wrote a full breakdown of this small-model serving problem and how SIE solves it.

It covers why switching to small specialized models doesn't reduce inference costs unless those models share GPUs, and why vLLM's design makes that impossible.

Read it below.