自托管AI模型成本降低75%：SIE推理引擎实现多模型共享GPU

Massive breakthrough here!

Self-hosting AI models just got ~75% cheaper.

For narrow tasks, most of a frontier model's parameters are dead weight, storing knowledge the task never touches.

For instance, a reranker does not need world knowledge to order ten passages by relevance. A small and specialized model does the task just as well in practice but at a fraction of the cost.

So production teams are now moving the routine steps onto small task-specific models.

Karpathy emphasized the same logic at the level of the model itself, arguing that a "cognitive core" of roughly a billion parameters is enough for the thinking part of most work.

Practically, a single request in today's agentic pipelines goes through 4-5 such models, like an embedder for retrieval, a reranker for precision, an extractor for entities, and often a small LLM for generation.

The catch is that using cheaper and smaller models does not imply a cheaper system, because each of those models still has to be served somewhere.

And the standard way to serve them is one server per model:

• vLLM holds the LLM. - TEI holds the embedder - and everything else may get a custom FastAPI wrapper.

None of these processes knows the others exist. Each claims its slice of GPU memory at startup and holds it whether traffic arrives or not.

For instance, vLLM's gpu-memory-utilization defaults to 0.9 per instance, and running two on one card requires computing the split by hand before any traffic shows up.

Since GPUs are billed by the hour, not by the work done on them, an idle card costs exactly what a busy card costs.

So four small models end up holding four cards, and most of that hardware sits idle between calls.

If one card could carry all four models' traffic, three of them could be switched off, and the bill would drop by 75%, with nothing about the work changing.

Traffic spikes don't alter the math either.

When the load exceeds one card, you add replicas of the shared server and drop them when the burst passes, which is still cheaper than four per-model fleets, where each one has to be sized for its own worst case.

Regardless, this argument explains why switching to small models rarely reduces the bill on its own because the cost was never in the model calls but rather in the servers.

Fixing this would require an inference engine with one server that holds every model type (LLMs, reranker, embedding models, etc.), owns the card's memory, sees every request, and loads or evicts models as traffic moves.

The approach is now actually implemented in SIE (Superlinked Inference Engine), an open-source inference engine that runs 85+ models behind one API.

Four calls cover the whole pipeline:

• encode() returns vectors - score() returns relevance scores - extract() returns entity spans - and generate() runs small open LLMs.

Models load on first request and are evicted least-recently-used, so one GPU serves a rotating set of models instead of sitting siloed behind one.

It runs anywhere from a laptop to a Kubernetes cluster, and it plugs into Qdrant, Weaviate, Chroma, LanceDB, LangChain, and LlamaIndex.

You can find the repo here: http://github.com/superlinked/sie

(don't forget to star it ⭐️)

To dive deeper, I published a full breakdown of this exact problem.

It covers how multi-step agentic pipelines are actually built today and why serving stacks designed around vLLM struggle to run them economically.

It then builds the entire pipeline on a single L4, with real code for all four calls (encode, score, extract, generate) running an invoice-processing example through one SIE server.

It also goes under the hood, showing how models load on first request, how least-recently-used eviction frees GPU memory as traffic shifts, and how pooling requests across callers keeps one GPU busy where separate servers would sit idle.

Read it below.