生产环境模型路由：按任务固定模型，而非按调用切换
How to think about model routing in production:
做模型路由或 Agent 编排的同学必看，按任务固定模型能省下大量缓存重建成本，Plano 已开源可直接落地，赶紧拿你的链路试试。
How to think about model routing in production:
如何在生产中思考模型路由：
Traditional routers classify by intent, and inside one task the intent keeps changing. Each call gets scored on its own, usually with no context that it's the middle of something.
传统路由器按意图分类，而在同一任务中，意图不断变化。每次调用独立评分，通常没有上下文表明这是某个过程的中间环节。
So if the routing layer switches the model midway, the new model has to build the whole cache again, at cold rates.
因此，如果路由层中途切换模型，新模型必须以冷启动速率重新构建整个缓存。
Routing per call is the problem, not routing itself, so one common solution is to route once per task, not once per call.
问题在于按调用路由，而非路由本身，所以一个常见解决方案是每个任务路由一次，而不是每次调用路由一次。
First LLM call routes normally, and that model gets pinned to a session ID. Every subsequent call goes to the same model.
首次LLM调用正常路由，该模型被固定到会话ID。后续每次调用都指向同一模型。
When the task changes, the pin resets, and the router picks again.
当任务变化时，固定重置，路由器再次选择。
This still gives you routing, but it's just that it happens at the point where switching is actually (or mostly) free.
这仍然提供路由，只是发生在切换实际（或大部分）免费的时刻。
Easy tasks get routed to the cheap model and hard ones to the expensive model, and the model stays pinned during the same task.
简单任务路由到廉价模型，困难任务路由到昂贵模型，且在同一任务期间模型保持固定。
The idea is called model affinity, and the visual below explains this.
这个想法称为模型亲和性，下图对此进行了解释。
To use this in practice, Plano (open-source) already implements it behind an OpenAI-compatible endpoint.
要在实践中使用，Plano（开源）已在OpenAI兼容端点后实现了这一点。
You can integrate it by just changing the base URL in the actual code while the whole model routing config is defined in a YAML file.
您只需更改实际代码中的基础URL即可集成，而整个模型路由配置在YAML文件中定义。
Basically, you generate one ID per task and pass it as a header on every call in the loop:
基本上，您为每个任务生成一个ID，并在循环中的每次调用中作为头部传递：
affinity_id = str(uuid.uuid4())
response = client(.)chat(.)completions(.)create(
...,
extra_headers={"X-Model-Affinity": affinity_id},
)affinity_id = str(uuid.uuid4())
response = client(.)chat(.)completions(.)create(
...,
extra_headers={"X-Model-Affinity": affinity_id},
)The pin lasts 10 minutes by default, and you can back it with Redis if you run more than one replica. When the task changes, you can swap in a new ID, and it routes again.
默认固定持续10分钟，如果您运行多个副本，可以用Redis支持。当任务变化时，您可以换用新ID，它会再次路由。
Here's the repo: http://github.com/katanemo/plano
仓库地址：http://github.com/katanemo/plano
Btw, there's also early research trying to remove the KV transfer constraint.
顺便说一句，也有早期研究试图消除KV传输限制。
NVIDIA published a method recently that converts one model's KV cache into the format another model expects, so the target skips prefill entirely, and the conversion runs about 2.7x-25x faster than reprocessing the context.
NVIDIA最近发布了一种方法，将一个模型的KV缓存转换为另一个模型期望的格式，因此目标模型完全跳过预填充，转换速度比重新处理上下文快约2.7倍至25倍。
It's closed-form and training-free, which is new. But every pair they tested is within one family, Qwen to Qwen and Llama to Llama, and cross-family is listed as future work.
这是封闭形式且无需训练，这是新的。但他们测试的每一对都在同一家族内，Qwen到Qwen和Llama到Llama，跨家族列为未来工作。
So it's not production-ready yet. But if cross-family transfer gets solved, you could switch models mid-task without repaying for the cache.
所以它尚未准备好用于生产。但如果跨家族转移得到解决，您可以在任务中途切换模型而无需重新支付缓存费用。
NVIDIA paper: https://arxiv.org/abs/2608.03893
NVIDIA论文：https://arxiv.org/abs/2608.03893
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力