LLM 首字延迟优化：算力翻倍为何无效

A good technical LLM interview question:

Your LLM chatbot takes 12s before it generates the first token, and the users are complaining.

So you move the model onto a GPU with 3x the computing power.

The time to first token barely improves.

Why did this happen?

(answer below)

Latency in an LLM app is a placement problem disguised as a model problem.

If you profile the 12 seconds, the model's prefill itself may only account for around 1.5 seconds of it.

So halving the prefill step saves just 750ms out of 12000, which is under 7%.

The rest is spread across stages that never touch the GPU.

The request first travels to whatever region the app runs in, and a cross-continent round trip could cost over a second before any code executes.

Then the request handler starts. On a container-based serverless platform under load, this adds several seconds of cold start, paid before auth, rate limiting, or prompt assembly even begins.

Retrieval adds its own hop, and the response streams back across the same distance.

Optimizing a stage that was already fast cannot alter the latency that's majorly affected by other stages.

Those other stages are slow for a structural reason.

An LLM app runs two workloads that want opposite machines.

• The request path is short, spiky, and needs to sit close to users - Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not.

So the actual decision is not which model to run, but where each of these two workloads runs.

There are three options, each with its own tradeoffs:

> A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request

> Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them.

> Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model.

So the answer is not to pick one, but to split the app across two of them.

The request path runs close to users, and inference runs on a dedicated GPU it calls into.

That also explains the failed upgrade. More compute made a stage that was already fast faster, and left the 10.5 seconds around it untouched.

To actually learn how it's done in practice, Akamai's GitHub has a reference implementation for each half.

• vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply.
• akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss.

Both are available on Akamai’s new Developer Hub, alongside their tutorials and code samples.

It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday.

If you create a new Akamai Cloud account, you can also get $300 in credits for joining.

Join here: https://fandf.co/4w7id0z

That said, this post treats generation as a single 1.5s block, but that block has its own structure, and knowing it well tells you whether a model is slow to start or slow to stream.

I wrote a first-principles walkthrough of it, covering the prefill and decode split, KV caching, and where the time actually goes inside each one.

Read it below.

Thanks to Akamai Cloud for partnering today!