LLM路由与混合代理架构对比解析

LLM Routing vs Mixture of Agents, clearly explained.

When you’re building a system that needs to handle diverse queries, two architectural patterns are considered.

One picks the best model for each task, and the other gets multiple models to collaborate on the same task.

Both involve multiple LLMs, but what they do with those models is completely different.

The visual below explains how each one works.

In LLM routing, an intent classifier reads the prompt, and a cost ranker weighs latency, token price, and capability, so only one model gets the call while everything else stays idle.

• The prompt is classified by domain and action. - A cost ranker picks the best model given latency and price constraints. - One model responds, the rest never see the query.

In Mixture of Agents, a set of consulting models each analyzes the query independently and hands their output to a single aggregator, which uses every perspective to write the final answer and make the tool calls.

The consulting models never see each other’s analysis, only the aggregator does.

On HermesBench, Nous Research reports a MoA preset scored higher than either model alone by roughly six points and eight to eleven percent, and no individual model in that setup reached the same result on its own.

LLM routing and MoA are not competing approaches but rather two different tools for two different problems.

Routing picks the right model for the task, while MoA combines multiple models to exceed what any one can do alone.

Routing looks like a simpler pattern of the two on paper, but most of the engineering in a routing layer has nothing to do with picking the best model.

The main problem is cache hit. An agent task invokes several calls in sequence, and each call resends the accumulated context, which providers bill at about 90% off once the model has already seen it.

But that cache is tied to a specific model. If the router switches models mid-task, the warm cache is no longer valid, and the full context gets re-billed at cold rates.

To solve this, you need to route once per task instead of once per call.

The first call routes normally, the best model gets pinned to the session, and every call in the task goes to the same model, so the cache stays warm.

Beyond that, a production routing layer also needs a guardrail check before the router, a routing model small enough that the decision costs less than the call it routes, and a cost policy with a fallback for when the primary model fails.

All of this is already implemented in Plano, an open-source proxy that runs locally between the agent and the model providers.

The whole routing setup lives in one YAML config, so swapping models or changing policies never touches agent code. It also has a dashboard that shows which model answered each request and what it cost.

Here's the repo: https://github.com/katanemo/plano

(don't forget to star it ⭐ )

I put this exact pipeline in front of my Hermes agent, and the bill dropped 2x, without changing a line of agent code. I have also written an article with the complete walkthrough about it.

Read it below.