LLM微调12种核心技术与OpenPipe ART实战
I have been fine-tuning LLMs for over 2 years now!
系统梳理了从PEFT到RLVR的完整微调技术栈，并给出了OpenPipe ART解决无监督任务奖励函数的具体落地方案，适合做Agent和垂直模型优化的工程师收藏参考。
I have been fine-tuning LLMs for over 2 years now!
我微调大语言模型（LLM）已经超过两年了！
Here are the top 12 techniques I'd learn if I were to fine-tune them:
如果我要微调它们，以下是我会学习的 12 种顶级技术：
Bookmark this.
收藏这篇。
1. LoRA > Freezes the base weights and trains two low-rank matrices as the update, resulting in ~95-99% fewer params to fine-tune.
1. LoRA > 冻结基础权重，训练两个低秩矩阵作为更新，从而减少约 95-99% 的微调参数量。
2. QLoRA > LoRA on top of a 4-bit quantized base model.
2. QLoRA > 在 4-bit 量化基础模型之上应用 LoRA。
3. Prefix tuning > Prepends trainable vectors to keys and values at every layer, weights frozen.
3. Prefix tuning > 在每个层的键（keys）和值（values）前添加可训练向量，权重冻结。
4. Adapter tuning > Inserts small trainable modules between transformer layers.
4. Adapter tuning > 在 Transformer 层之间插入小型可训练模块。
5. Instruction tuning > Supervised tuning on (instruction, response) pairs so the model follows directions instead of just continuing text.
5. Instruction tuning > 对（指令，响应）对进行监督式微调，使模型遵循指令，而不仅仅是续写文本。
6. P-tuning > Optimizes continuous prompt embeddings through a small encoder, mainly for NLU tasks where discrete prompts are unstable.
6. P-tuning > 通过小型编码器优化连续提示嵌入，主要用于离散提示不稳定的自然语言理解（NLU）任务。
7. BitFit > Trains only the bias terms, ~0.08% of params, and still rivals full fine-tuning on small-to-medium datasets.
7. BitFit > 仅训练偏置项，约占参数的 0.08%，在小到中等规模数据集上仍能媲美全量微调。
8. RLHF/RLAIF > RLHF trains a reward model on human preference rankings, then PPO against it. The pipeline behind the first ChatGPT. > RLAIF swaps the human labeler for an LLM judging. RLHF-level quality at a fraction of the cost.
8. RLHF/RLAIF > RLHF 基于人类偏好排名训练奖励模型，然后使用 PPO 对其进行优化。这是首个 ChatGPT 背后的流水线。 > RLAIF 用 LLM 评判者取代人类标注员。以极低的成本实现 RLHF 级别的质量。
9. DPO (Direct Preference Optimization) > Skips the reward model and optimizes preference pairs directly with a classification-style loss. Simpler than PPO.
9. DPO (直接偏好优化) > 跳过奖励模型，直接使用分类风格的损失函数优化偏好对。比 PPO 更简单。
10. GRPO (Group Relative Policy Optimization) > Samples a group of responses per prompt and normalizes their rewards within the group. DeepSeek R1 ran on it.
10. GRPO (组相对策略优化) > 为每个提示采样一组响应，并对组内的奖励进行归一化。DeepSeek R1 就是基于此运行的。
11. RLVR (Reinforcement Learning with Verifiable Rewards) > Replaces the learned reward model with a checker or compiler returning verifiable scores. The free signal behind R1's math and code.
11. RLVR (带可验证奖励的强化学习) > 用返回可验证分数的检查器或编译器替换学到的奖励模型。这是 R1 在数学和代码方面获得免费信号的关键。
12. Federated fine-tuning > Tunes across decentralized clients that share only weight updates, never raw data. For when data can't leave the device.
12. 联邦微调 > 在去中心化客户端上进行微调，仅共享权重更新，从不共享原始数据。适用于数据无法离开设备的情况。
GRPO needs exactly one scalar reward per response. RLVR (13) produces that for free on math and code by running the answer through a checker or compiler.
GRPO 需要每个响应恰好一个标量奖励。RLVR (13) 通过将答案输入检查器或编译器，在数学和代码任务中免费生成该奖励。
But tasks like a RAG answer, a support reply, or a summary have no gold label to match against.
但像 RAG 回答、客服回复或摘要这样的任务，没有黄金标签可供比对。
The usual fallback is a hand-written reward function scoring faithfulness, hallucination, and completeness.
通常的回退方案是手写奖励函数，用于评估忠实度、幻觉和完整性。
It takes days to calibrate, rewards the wrong behavior when the weights are off, and breaks every time you add a tool or edit the system prompt.
校准需要数天时间，当权重偏差时会奖励错误的行为，并且每次添加工具或编辑系统提示时都会失效。
RULER, implemented in OpenPipe's ART (open-source), solves this.
RULER，由 OpenPipe 的 ART（开源）实现，解决了这个问题。
During training, it passes the N sampled trajectories to a judge LLM, which ranks them relative to each other against the agent's system prompt and returns the scores.
在训练期间，它将 N 个采样的轨迹传递给评判 LLM，后者根据智能体的系统提示对这些轨迹进行相对排名并返回分数。
Relative ranking is more stable than absolute scoring, and GRPO normalizes within the group anyway, so the rankings feed straight into the pipeline like with RLVR.
相对排名比绝对评分更稳定，而且 GRPO 无论如何都会在组内进行归一化，因此排名可以直接像 RLVR 那样接入流水线。
Here's the GitHub Repo: https://github.com/OpenPipe/ART
这里是 GitHub 仓库：https://github.com/OpenPipe/ART
(don't forget to star it ⭐ )
（别忘了给它点个星 ⭐）
I wrote a full breakdown recently on how exactly this works, with the training loop and code.
我最近写了一篇详细的解析，介绍了其工作原理，包括训练循环和代码。
Read it below.
请在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力