LLM 训练的四个阶段：预训练、指令微调、偏好微调、推理微调

4 stages of LLM training, clearly explained:

(a popular LLM interview question; bookmark this)

→ Pre-training → Instruction fine-tuning → Preference fine-tuning → Reasoning fine-tuning

The visual covers all of them.

A base model treats a question as text to be continued, not a request to be answered. Questions in its training data usually have more questions next to them, so that's what the model replies.

The stages after pre-training allow changing that behaviour, and each one uses a different training signal to do it.

0) Randomly initialized model

The weights are random numbers, so token selection is close to uniform, and the LLM outputs gibberish text.

1) Pre-training

The model reads a large text corpus and repeatedly predicts the next word.

This stage teaches the model grammar and facts about the world.

Also, most of its raw reasoning ability comes from this stage, and it's where nearly all of the training compute goes.

The model becomes good at continuing text, but nothing in the objective connects a question to an answer.

2) Instruction fine-tuning

Here, it trains on pairs of instructions and good responses, so it picks up the convention that a question should get an answer and a request to summarise should get a summary.

OpenAI used roughly 13,000 human-written examples on GPT-3, against a pre-training corpus of hundreds of billions of words.

The knowledge was already in the weights, and this stage mostly taught a response format.

3) Preference fine-tuning

A lot of requests have no single correct answer, and supervised fine-tuning needs a specific target to train against.

In this setup, two responses go to a human who picks the better one. You may have already seen this in ChatGPT, where it asks which of the two responses is better.

Those picks train a separate model that predicts which response a human would have chosen. The main model then gets updated to score higher against that predictor.

4) Reasoning fine-tuning

For maths and code, there's usually one correct answer, and a program can check it. Human preference isn't needed here, since correctness is already a usable reward signal.

The model attempts a problem, the answer gets checked, and it's rewarded when it gets it right.

DeepSeek trained R1 this way with GRPO, and behaviours like working through intermediate steps and going back to fix its own mistakes showed up during training without any examples demonstrating them.

That said, stage 4 only works when a program can check the answer. Maths and code inherently provide that.

But a support reply, a RAG answer, or a summary has no correct output to check against, so the reward has to come from somewhere else.

I wrote a full breakdown of how labs are handling that problem, from RLHF through GRPO, and what replaced the checker.

Read it below.