跳到主内容
@wquguru
精选90GitHub 新爆发仓库(GitHub)技巧与观点

GitHub热榜:Agent自动化调优本地LLM推理速度的完整工作流

sybil-solutions/inference-optimisation-skill: Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill,…

原文
发到 X
推荐理由

做本地部署的同学必看,这套带理论上限计算和严格对照实验的优化工作流非常硬核,直接拿你的链路压测一遍。

Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

针对任意硬件和量化方案,提升本地大语言模型推理速度(解码、推测解码、预填充、KV缓存、启动时间、首字延迟)的代理技能

GitHub repository created 2026-11-16 · 1029 stars in 1.8 days (about 578 a day) · Python

GitHub 仓库创建于 2026-11-16 · 1.8 天内获得 1029 颗星(日均约 578 颗)· Python

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:

一种用于在任意硬件和任意量化方案下提升本地大语言模型服务器速度的代理技能。它涵盖:

  • single-user and concurrent decode - speculative decoding - prefill - KV cache size - startup time and time to first token
  • 单用户与并发解码 - 推测解码 - 预填充 - KV 缓存大小 - 启动时间和首字延迟

It works under a memory cap, and every result has to survive a full-context stability check.

它在内存上限约束下运行,且每个结果都必须通过全上下文稳定性检查。

It encodes how we actually got these speedups. It is not a list of flags. The workflow:

它编码了我们实际获得这些加速的方法。这不是一份标志位列表。工作流程如下:

1. Compute the theoretical ceiling. 2. Change one thing at a time. 3. Measure against interleaved controls. 4. Log every result, including the failures. 5. Gate on long-context retrieval. 6. Qualify a clean, pinned build before publishing.

1. 计算理论上限。 2. 每次只改变一个变量。 3. 使用交错对照进行测量。 4. 记录所有结果,包括失败案例。 5. 以长上下文检索作为关卡验证。 6. 在发布前确保构建干净且版本固定。

It works with Claude Code and any agent that loads `SKILL.md`-style skills. Any model can also follow it from `PROMPT.md`.

它与 Claude Code 以及任何加载 `SKILL.md` 风格技能的代理兼容。任何模型也可以从 `PROMPT.md` 中遵循该流程。

What we've been able to accomplish

我们所取得的成果

Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.

本技能所提炼出的爬坡优化结果。每个数值均在标注的硬件上测量得出;方法和原始数据请参阅链接来源。

HardwareModel / quant / engineBefore → after
Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s)Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTPDecode, 1 user: 43.6 → 53 tok/s (median)Decode, 4 users: 84.7 → **103 tok/s**Prefill 32K: 1,037 → **1,660 tok/s**Prefill 256K: 297 → 628 tok/s (2.1×)KV cache: 264K tokens → **4 × 256K**Startup: 113 → **43 s**Registry Lab gates: decode 28.6 → 41.0, prefill 255 → 667
Intel Arc Pro B70Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernelsDecode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)Same card with llama.cpp Q4_K_M: 25.0Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → **2,421/2,259/1,415/911**Agent-session replay: 193 → 147 s
Intel Arc Pro B70Same model, SGLang + the same kernelsDecode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day
RTX 3090Qwen3.8-27B EXL3 3 bpw + MTP96.2 tok/s prose / 141.1 code, 1 user
RTX 3090Qwen3.6-35B-A3B EXL3 3 bpw + MTP251.7 / 357.9 tok/s, 1 user
RTX 3090Flash-Next EXL3, 256K contextMTP: 45.5 → 65.6 tok/s
4× H200 NVLDeepSeek-V4.1-Flash, vLLM + speculative decodingCrashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass
硬件模型 / 量化 / 引擎优化前 → 优化后
Intel Arc B390 iGPU (笔记本, 64 GB 统一 LPDDR5X, ~115 GB/s)Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP解码, 1 用户: 43.6 → 53 tok/s (中位数)解码, 4 用户: 84.7 → **103 tok/s**预填充 32K: 1,037 → **1,660 tok/s**预填充 256K: 297 → 628 tok/s (2.1×)KV 缓存: 264K tokens → **4 × 256K**启动: 113 → **43 s**Registry Lab 关卡: 解码 28.6 → 41.0, 预填充 255 → 667
Intel Arc Pro B70Qwen3.8-27B, EXL3 4 bpw, vLLM + 自定义 XPU 内核解码: 1 用户时 91.2 tok/s, 16 用户时 365 (散文模式, 思维链开启)同卡使用 llama.cpp Q4_K_M: 25.0预填充 4K/32K/128K/254K: 1,654/1,434/928/650 → **2,421/2,259/1,415/911**代理会话回放: 193 → 147 s
Intel Arc Pro B70相同模型, SGLang + 相同内核解码, 1 用户: 82.6 tok/s, 对比同日同卡的 vLLM 75.7
RTX 3090Qwen3.8-27B EXL3 3 bpw + MTP96.2 tok/s 散文 / 141.1 代码, 1 用户
RTX 3090Qwen3.6-35B-A3B EXL3 3 bpw + MTP251.7 / 357.9 tok/s, 1 用户
RTX 3090Flash-Next EXL3, 256K 上下文MTP: 45.5 → 65.6 tok/s
4× H200 NVLDeepSeek-V4.1-Flash, vLLM + 推测解码负载下崩溃 → 解码 229.1 tok/s, 预填充 17,546 tok/s; 全部六个关卡通过

Sources:

来源:

  • local-ai-registry: - Ornith B390 recipe, with the full hill-climb log in PR #453 - the RTX 3090 and H200 recipes - trellis-serve: - `xpu/README.md`, `xpu/docs/PROGRESS.md` and `xpu/docs/SGLANG.md` for B70 - `cuda/README.md` for the RTX 3090 - qwen-serve
  • local-ai-registry: - Ornith B390 配方,附带 PR #453 中的完整爬山日志 - RTX 3090 和 H200 的配方 - trellis-serve: - `xpu/README.md`、`xpu/docs/PROGRESS.md` 和 `xpu/docs/SGLANG.md`(用于 B70) - `cuda/README.md`(用于 RTX 3090) - qwen-serve

Reading the B390 result

阅读 B390 的结果

That run is the one this skill grew out of. Its ceiling calculation said:

这次运行是该技能的基础。其上限计算指出:

  • Decode: 100 tok/s for one user was only reachable with two useful speculative drafts. This model's draft head accepts ~14% at depth 2, so the practical target was ~88–97 tok/s. We reached ~55% of it; the rest is GEMV kernel efficiency. - Prefill at 256K: we reached 628 tok/s against a ~670–684 tok/s ceiling.
  • 解码:对于单个用户,仅在使用两个有效的推测草稿时才能达到 100 tok/s。该模型的草稿头在深度 2 处接受约 ~14%,因此实际目标约为 ~88–97 tok/s。我们达到了约 55%;其余部分取决于 GEMV 内核效率。 - 256K 预填充:我们以 628 tok/s 的速度运行,而上限约为 ~670–684 tok/s。

The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.

该技能会提前让智能体进行此计算,以便在目标超出物理极限时尽早告知你。

How it works

工作原理

The skill has three parts:

该技能包含三个部分:

`SKILL.md`: the procedure. - Inventory, ceilings, ground rules, the loop and lever order. - When to stop, how to qualify and publish, and how to report.

`SKILL.md`:操作流程。 - 库存、上限、基本规则、循环与杠杆顺序。 - 何时停止、如何验证和发布以及如何报告。

`references/`: loaded only when needed. - `ceilings.md`: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example. - `methodology.md`: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting. - `levers.md`: every lever with measured evidence. It is tagged general vs stack-specific. - `gotchas.md`: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware. - `publishing.md`: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.

`references/`:仅在需要时加载。 - `ceilings.md`:如何计算解码、预填充、TTFT、启动时间和 KV 最大值,并附有示例。 - `methodology.md`:测量陷阱。按块计数的 token、尺寸错误的提示词、接受计数器、漂移、统一内存记账。 - `levers.md`:每个带有实测证据的杠杆。标记为通用或特定于堆栈。 - `gotchas.md`:vLLM、SGLang、llama.cpp 和 ExLlama 中的已知陷阱,以及 Intel、NVIDIA、Apple 和 AMD 硬件上的陷阱。 - `publishing.md`:固定版本、干净构建、六个验收关卡、证明构建与最佳版本一致、结果卡片。

**`scripts/`: a small, engine-agnostic harness.** It is standard-library Python plus bash. - `bench.py` benchmarks any OpenAI-compatible server. It measures: - startup and short-prompt TTFT; - c1 decode (median of N) and cN aggregate decode; - cold prefill at several lengths; - long-context needle retrieval; - peak memory, with a cap watchdog.

**`scripts/`:一个小型、引擎无关的测试框架。** 它由标准库 Python 和 bash 组成。 - `bench.py` 对任何兼容 OpenAI 的服务器进行基准测试。它测量: - 启动时间和短提示词的 TTFT; - c1 解码(N 的中位数)和 cN 聚合解码; - 多种长度下的冷预填充; - 长上下文针检索; - 峰值内存,并带有容量看门狗。

It flags looping output, failed requests and runs on battery. Token counts come from server usage stats, never from streamed chunks, and everything lands in `ledger.jsonl`. - `ceiling.py` computes the theoretical ceilings from a JSON of measured bandwidth and FLOPs plus component sizes. - `queue.sh` is a serialized, detachable run queue, for interleaving control/candidate/control runs. - `mem.sh` reports a container's real footprint. That includes driver residency on unified-memory GPUs, which cgroups don't see.

它会标记循环输出、失败的请求以及在电池供电下运行的情况。Token 计数来自服务器使用统计信息,绝不来自流式分块,所有内容均存入 `ledger.jsonl`。 - `ceiling.py` 根据测量的带宽和 FLOPs JSON 以及组件大小计算理论上限。 - `queue.sh` 是一个序列化的、可分离的运行队列,用于交错执行控制/候选/控制运行。 - `mem.sh` 报告容器的实际占用空间。这包括统一内存 GPU 上的驱动程序驻留,而 cgroups 无法看到这一点。

Install

安装

Claude Code (user-level):

Claude Code(用户级):

Then ask, for example:

然后询问,例如:

  • "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap" - "what's the theoretical max tok/s for this model on this machine?"
  • "在 :8000 端口上对模型进行 hill-climb(爬山式)解码和预填充,显存上限为 40 GiB" - "该模型在此机器上的理论最大 tok/s 是多少?"

Other agents: point them at `skills/inference-optimisation/SKILL.md`, or paste `PROMPT.md` as the task prompt.

其他智能体:将它们指向 `skills/inference-optimisation/SKILL.md`,或将 `PROMPT.md` 粘贴为任务提示。

Quick start without an agent

无需智能体的快速入门

Principles that carried the most weight

权重最高的原则

  • Ceilings first. If you don't know the maximum, you can't tell whether 53 tok/s is great or bad, or whether "100 tok/s" is possible. - Noise is ±5–10%. Most single-run "wins" are drift. Interleave controls and use medians. - Measure what matters: - real tokens, not stream chunks; - real prompt lengths, from the tokenizer; - real memory, including GPU residency on unified memory. - Speculative decoding is about per-step cost, not acceptance. - Prune the draft head's vocabulary. - Sweep draft length per concurrency and context. - Keep sampling on the GPU. - Prove full-context stability with exact-match needles and a concurrent full-KV soak before calling anything "best". - Write down failures. Most of the value in `PLAN.md` is the list of things that did *not* work and why.
  • 先确定上限。如果你不知道最大值,就无法判断 53 tok/s 是好是坏,也无法判断“100 tok/s”是否可能。 - 噪声约为 ±5–10%。大多数单次运行的“优势”只是波动。应交替控制变量并使用中位数。 - 测量关键指标: - 真实 token 数,而非流分块; - 来自分词器的真实提示长度; - 真实内存,包括统一内存中的 GPU 驻留部分。 - 投机解码关注的是每步成本,而非接受率。 - 剪枝草稿头的词汇表。 - 针对并发数和上下文长度扫描草稿长度。 - 保持采样在 GPU 上进行。 - 在使用精确匹配的针测试以及并发的全 KV 负载测试证明全上下文稳定性之前,不要称任何结果为“最佳”。 - 记录失败案例。`PLAN.md` 中大部分价值在于列出那些*未*起作用的方法及其原因。

Contributing

贡献指南

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件