SFT、DPO与RLVR对比及Qwen3 GRPO实战验证
SFT vs. DPO vs. RLVR, clearly explained:
提供了一手工程实践与具体参数配置，包含完整的实验数据对比，做模型微调的同学可直接参考其选型逻辑与GRPO调优细节。
SFT vs. DPO vs. RLVR, clearly explained:
SFT、DPO 与 RLVR，清晰解析：
All three methods customize an LLM, but they learn from different forms of supervision.
这三种方法都对大语言模型（LLM）进行定制，但它们从不同类型的监督信号中学习。
Consider a math problem with a known answer.
考虑一个有已知答案的数学问题。
1) Supervised fine-tuning
1) 监督微调（Supervised Fine-Tuning）
SFT needs the prompt and an ideal completion. The model learns to reproduce that response.
SFT 需要提示词（prompt）和理想的补全内容。模型学习复现该响应。
It works well when high-quality responses already exist. The expensive part is writing and validating one for every prompt.
当已存在高质量响应时，这种方法效果很好。昂贵的部分在于为每个提示词编写并验证一个理想响应。
2) Direct preference optimization
2) 直接偏好优化（Direct Preference Optimization）
DPO needs a preferred and rejected response for the same prompt.
DPO 需要针对同一提示词提供一个优选响应和一个被拒绝的响应。
Training makes the preferred response more likely without requiring a separate reward model.
训练过程使优选响应的可能性增加，而无需单独的训练奖励模型。
It is useful when comparing two responses is easier than writing an ideal response from scratch.
当比较两个响应比从头编写一个理想响应更容易时，这种方法很有用。
3) Reinforcement learning with verifiable rewards
3) 基于可验证奖励的强化学习（Reinforcement Learning with Verifiable Rewards）
RLVR needs the prompt, a reference answer, and a programmatic verifier.
RLVR 需要提示词、参考答案以及一个程序化验证器。
The model samples several solutions. A verifier compares each final answer with the reference and returns a reward.
模型采样多个解决方案。验证器将每个最终答案与参考答案进行比较，并返回奖励。
GRPO compares rewards within the group and increases the probability of higher-scoring solutions.
GRPO 在组内比较奖励，并提高高分解决方案的概率。
> SFT learns from what the model should say. DPO learns which response is better. RLVR learns from whether the result passes a test.
> SFT 学习模型应该说什么。DPO 学习哪个响应更好。RLVR 学习结果是否通过测试。
That makes RLVR useful when success can be checked automatically. A verifier can run code against tests, validate structured output against a schema, or compare a numerical answer with the ground truth.
这使得 RLVR 在成功可以自动检查的情况下非常有用。验证器可以对代码运行测试，根据模式验证结构化输出，或将数值答案与真实值进行比较。
I tested this on Amazon SageMaker AI by training Qwen3 0.6B on 1,500 GSM8K problems using a numeric verifier.
我在 Amazon SageMaker AI 上对此进行了测试，使用数值验证器在 1,500 个 GSM8K 问题上训练 Qwen3 0.6B 模型。
The base model solved 736 of 1,319 test problems, reaching 55.80% exact-match accuracy.
基础模型解决了 1,319 个测试问题中的 736 个，精确匹配准确率达到 55.80%。
After 100 GRPO steps, the RLVR model solved 774 problems and reached 58.68%.
经过 100 步 GRPO 训练后，RLVR 模型解决了 774 个问题，准确率达到 58.68%。
Training corrected 133 baseline failures but broke 95 previously correct answers. The net improvement was 38 problems, or 2.88 percentage points.
训练纠正了 133 个基线失败案例，但破坏了 95 个之前正确的答案。净提升为 38 个问题，即 2.88 个百分点。
I would treat this as a proof of signal, not the ceiling.
我会将其视为一种信号证明，而非性能上限。
This was a 100-step run on a 0.6B model. The next experiments would train for longer, use more verified examples, and tune the learning rate and GRPO group size.
这是在 0.6B 模型上进行的 100 步运行。接下来的实验将进行更长时间的训练，使用更多经过验证的示例，并调整学习率和 GRPO 组大小。
These methods can also be combined. SFT can first teach the model a stable solution format, after which RLVR can optimize correctness against an executable verifier.
这些方法也可以结合使用。SFT 可以先教会模型稳定的解法格式，随后 RLVR 可以利用可执行验证器来优化正确性。
The method should follow the supervision that is cheapest to produce reliably:
该方法应遵循最易于可靠生成的监督类型：
- Use SFT when ideal completions exist. - Use DPO when preferences are easier to label. - Use RLVR when success can be checked programmatically.
- 当存在理想补全时使用 SFT。 - 当偏好更容易标注时使用 DPO。 - 当成功可以通过编程检查时使用 RLVR。
If you want to try this in practice, AWS has released a hands-on serverless model customization workshop that makes it easy to get started with RLVR on Amazon SageMaker AI.
如果您想在实践中尝试，AWS 发布了一个动手操作的服务端模型定制研讨会，使您能够轻松地在 Amazon SageMaker AI 上开始使用 RLVR。
It covers the dataset, custom reward function, training job, evaluation, and deployment.
它涵盖了数据集、自定义奖励函数、训练任务、评估和部署。
Workshop: http://fandf.co/4jL2MbR
研讨会：http://fandf.co/4jL2MbR
I also wrote a detailed engineering walkthrough with the code and experiment results.
我还撰写了一篇详细的工程指南，包含代码和实验结果。
The full article is below and thanks to AWS for collaborating with me on this issue.
完整文章如下，感谢 AWS 与我合作完成本期内容。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力