单卡部署百个微调模型：vLLM共享端点与动态加载实践
How to serve 100 fine-tuned models on one GPU:
提供了一手工程实践，详细拆解了多适配器共享部署的架构权衡与性能数据，做 LLM 服务优化的同学可直接参考其 vLLM 配置思路。
How to serve 100 fine-tuned models on one GPU:
如何在单张 GPU 上服务 100 个微调模型：
A straightforward deployment would assign a separate endpoint to every fine-tuned model.
一种直接的部署方式是为每个微调模型分配一个独立的端点。
That layout is clean at the model level, but it creates a scheduling problem.
这种布局在模型层面很清晰，但会带来调度问题。
Traffic gets split across independent queues, and each queue controls its own workers.
流量被分散到独立的队列中，每个队列控制自己的 Worker。
One endpoint can therefore be waiting for a GPU while another has idle capacity. Low traffic per endpoint also means more cold starts and fewer requests available for continuous batching.
因此，一个端点可能在等待 GPU，而另一个端点却有闲置容量。每个端点的低流量也意味着更多的冷启动，以及可用于连续批处理的请求更少。
Ultimately, the endpoint layout determines whether requests can share the same workers. There are four options:
最终，端点布局决定了请求是否可以共享相同的 Worker。有四种选择：
1) A merged deployment combines each adapter with the base model. Every fine-tune becomes a complete model with its own copy of the base weights.
1) 合并部署将每个适配器与基础模型结合。每次微调都成为一个完整的模型，拥有自己的一份基础权重副本。
2) A dedicated deployment assigns one endpoint to each adapter. Workloads remain isolated, but every endpoint has its own queue, workers, and scaling policy.
2) 专用部署为每个适配器分配一个端点。工作负载保持隔离，但每个端点都有自己的队列、Worker 和扩缩容策略。
3) A shared deployment registers several adapters behind one endpoint. Requests enter the same queue and select the required adapter during inference.
3) 共享部署在一个端点背后注册多个适配器。请求进入同一个队列，并在推理过程中选择所需的适配器。
4) A dynamic deployment fetches adapters when requests arrive. This supports a larger catalog but requires download management, caching, eviction, validation, and access control.
4) 动态部署在请求到达时获取适配器。这支持更大的目录，但需要管理下载、缓存、驱逐、验证和访问控制。
The shared layout (number 3) works well when the adapters are known in advance and receive irregular traffic. Requests from every fine-tune contribute to the same queue, and any available worker can process them.
当适配器事先已知且接收不规则流量时，共享布局（第 3 种）效果良好。来自所有微调模型的请求都会汇入同一个队列，任何可用的 Worker 都可以处理它们。
I built a visual implementation of this layout using Qwen2.5-1.5B-Instruct and three rank-16 LoRA adapters for extraction, SQL generation, and routing.
我使用 Qwen2.5-1.5B-Instruct 和三个 rank-16 LoRA 适配器（分别用于提取、SQL 生成和路由）构建了该布局的可视化实现。
Each box in the video below represents a tenant request, and its color identifies the adapter it was routed to. The lower panel reports concurrency, throughput, memory consumption, and GPU utilization.
下面视频中的每个方框代表一个租户请求，其颜色标识了它被路由到的适配器。下方面板报告了并发数、吞吐量、内存消耗和 GPU 利用率。
I used Runpod Serverless because the implementation required a custom vLLM container and control over worker scaling without manually managing GPU instances.
我使用 Runpod Serverless，因为该实现需要一个自定义的 vLLM 容器，并需要在不手动管理 GPU 实例的情况下控制 Worker 扩缩容。
I packaged the base model and three adapters into one vLLM worker (the same mechanisms would hold for 100 adapters as well).
我将基础模型和三个适配器打包到一个 vLLM Worker 中（对于 100 个适配器的情况，机制也是一样的）。
Runpod Serverless ran it on an RTX 4090 and exposed it through an OpenAI-compatible endpoint.
Runpod Serverless 在 RTX 4090 上运行了它，并通过一个 OpenAI 兼容的端点进行暴露。
The endpoint could scale to zero between traffic bursts and report queue delay (delayTime) separately from model execution time (executionTime).
该端点可以在流量突发之间缩容至零，并单独报告队列延迟（delayTime）和模型执行时间（executionTime）。
Use Runpod here: http://fandf.co/4zTAjq2
在此使用 Runpod：http://fandf.co/4zTAjq2
The recorded workload below completed a 100 request burst with zero failures. It reached 27.9 requests per second and 795.7 tokens per second at peak, while GPU utilization reached 93%. I worked with the Runpod team to put this together as part of our paid partnership.
下面记录的负载测试完成了 100 次请求的突发，零失败。峰值达到每秒 27.9 次请求和每秒 795.7 个 token，同时 GPU 利用率达到 93%。我与 Runpod 团队合作完成了这项工作，这是我们付费合作的一部分。
If you want to dive deeper, I also wrote an article that explains how the same architecture extends to 100 fine-tuned models, including memory calculations, adapter routing, cold starts, and endpoint trade-offs.
如果你想深入了解，我还写了一篇文章，解释了相同的架构如何扩展到 100 个微调模型，包括内存计算、适配器路由、冷启动以及端点权衡。
Read it below.
请在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力