哈佛与芝加哥大学研究：一年生产数据揭示LLM服务负载规律
New Harvard+Chicago study.
New Harvard+Chicago study.
哈佛+芝加哥新研究。
A year of production data shows that efficient LLM serving depends less on a single clever scheduler than on understanding how traffic changes and repeats.
一年的生产数据显示，高效的LLM服务更依赖于理解流量变化和重复模式，而非单一的智能调度器。
The researchers studied 6.12B requests across 9,174 models for 1 year and found repeatable patterns that serving systems can use.
研究人员对一年内9,174个模型上的61.2亿次请求进行了研究，发现了服务系统可利用的可重复模式。
Users often come back to the same model with the same growing context. 99% of the reuse they measured came from requests returning within 15 minutes.
用户经常带着相同的增长上下文回到同一模型。他们测量的99%的复用来自15分钟内返回的请求。
So the GPU may already have much of the previous prompt computed and can reuse it instead of doing that work again.
因此，GPU可能已经计算了先前提示的大部分内容，可以复用而不是重新执行该工作。
A load balancer that only spreads traffic evenly can weaken this advantage by sending the next turn to another GPU.
仅均匀分配流量的负载均衡器可能会削弱这一优势，因为将下一轮请求发送到另一个GPU。
The traffic itself also changes over months: popular models turn over, outputs get shorter, and newer users tend to send larger inputs.
流量本身也会在数月内变化：热门模型更替，输出变短，新用户倾向于发送更大的输入。
That makes short or synthetic workload benchmarks a poor picture of long-running production.
这使得短期或合成工作负载基准无法准确反映长期生产环境。
Overall, routing, caching, and capacity planning should use the history and structure of real traffic, rather than treating every request as an isolated job.
总体而言，路由、缓存和容量规划应利用真实流量的历史和结构，而不是将每个请求视为孤立任务。
– arxiv. org/abs/2608.13573
– arxiv. org/abs/2608.13573
Title: "A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing"
标题：“LLM服务的一年：工作负载演变、缓存与负载均衡”
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力