Google与Anthropic共识：模型加速与硬件绑定

Google and Anthropic agree on one thing about LLM inference:

The faster a model runs, the fewer machines can run it.

This sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.

And those choices only fit the hardware they were made for.

That is why they build the same model more than once, one version per chip or runtime it has to run on.

The visual below covers the six formats used to run LLMs in production:

1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.

A .pt file is a list of instructions that torch.load executes and loading a checkpoint can also run a payload (possibly malicious).

Safetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.

Both still need Python, the model code, and a tokenizer alongside.

3) GGUF is llama.cpp's format, built so a model can run on a machine with no ML framework installed.

To achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.

4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.

It decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.

5) MLX is Apple's framework for its own silicon, where the CPU and GPU share a single pool of memory.

It decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.

6) TensorRT is NVIDIA's compiler, which turns a model into machine code for one specific GPU.

Everything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.

The safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.

That said, picking the right format only solves how ONE model behaves.

But production systems run a fleet of models, usually an embedding model, a reranker, an extractor, and a generator, so the bottleneck moves from the file to the GPU that they all have to share.

Except they mostly cannot share it.

For instance, each vLLM process claims 90% of GPU memory at startup and holds it, so teams end up provisioning a GPU per model, no matter how carefully each file was built.

SIE is an open-source inference server to solve this problem.

Models load when a request needs them and get evicted when they go cold, so one GPU serves embeddings, reranking, extraction, and generation behind a single API instead of running one process per model.

GitHub repo: https://github.com/superlinked/sie

(don't forget to star it ⭐ )

I wrote a full breakdown of this small-model serving problem and how SIE solves it.

It covers why switching to small specialized models doesn't reduce inference costs unless those models share GPUs, and why vLLM's design makes that impossible.

Read it below.