RLHF、DPO、GRPO 可视化对比

RLHF vs. DPO vs. GRPO, explained visually:

(a popular LLM interview question)

These methods are often grouped together as variations of the same algorithm.

But they take completely different approaches to teaching a model to behave the way you want, and those differences show up in the training setup, the data you need, and what it costs you at scale.

The visual below summarizes how they work:

Let’s understand this in detail today!

RLHF (Reinforcement Learning from Human Feedback) is the simplest formulation, which also went into building ChatGPT.

You sample prompts, run them through the policy, and score the outputs with a reward model. Then you compute advantages, apply a KL penalty to keep the policy close to the reference, and update via PPO.

This requires four live models running simultaneously: - The policy you’re training - A frozen reference copy - A reward model - And a critic.

This means running four sets of forward passes per training step.

At production scale, this is expensive to run and hard to stabilize. And the reward model needs its own human-preference training before any of this can even begin.

DPO (Direct Preference Optimization) takes a more direct route.

Instead of training a separate reward model, DPO derives the reward signal implicitly from the policy itself.

It works from preference pairs, a winner and a loser response to the same prompt. It computes log-probability ratios between the current policy and a frozen reference, and uses that ratio as the implicit reward.

This removes the explicit reward model and the critic entirely, bringing the pipeline down to just policy and frozen reference.

The tradeoff is that you need labeled preference pairs upfront. And since there’s no online exploration, DPO can be brittle if the preference data doesn’t cover the failure modes you care about.

GRPO (Group Relative Policy Optimization), introduced by DeepSeek in 2024, takes a different path.

It keeps the RL framing but eliminates the critic by replacing the value baseline with group statistics.

For each prompt, the model generates a group of outputs, and a verifier scores each one.

Advantages are then computed from the mean and standard deviation of those scores within the group.

The critic is a key bottleneck in RLHF since it’s slow to converge, adds instability, and requires a regression head on top of the policy.

GRPO removes all of that by using the group itself as the baseline, keeping the RL loop intact but at lower overhead than PPO.

Here’s a useful way to understand all three at once. - RLHF uses external scores and an internal critic to measure advantage. - DPO encodes preference implicitly and skips both. - GRPO generates its own baseline from a group and keeps the RL loop without the critic.

All three produce aligned models, but they differ on how much machinery the alignment requires.

But the progression doesn't stop at GRPO.

GRPO with verifiable rewards works great when you can check an answer programmatically, like math or code.

If you move to open-ended agentic tasks that give you nothing to verify against, the whole setup breaks down.

I wrote about how top AI labs are solving this in 2026 with RULER, an LLM-as-judge reward function that extends RL training to non-verifiable tasks without writing custom reward functions.

Read it below.