新书《RLHF：对齐与后训练LLM》发布，作者分享5大核心收获

Housekeeping: No voiceover on another quick “launch” post. More essays soon!

After a few long years of finding time to document my lessons from training open models, my post-training book is done! It’s published by Manning, under the title Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs.

Telling the story of the book is a useful way to explain why you may want a copy.

The book started as a website where I wanted to document key methods of post-training that had potentially no online material explaining them. If there was something, I couldn’t find it. This existed for more topics than you would expect, given post-training was already popular in 2024 (when I bought the domain), and continues to this day. Topics like rejection sampling, outcome reward models, and character training are prime examples. This has helped make the website fairly popular, as it’s still one of the few places discussing these topics at a foundational, intuitive way.

Otherwise, most of the book is about communicating intuitions and history. Much of the LLM industry is defined by core techniques that haven’t changed much in the last few years. This book was my attempt to explain in simple terms why post-training works, what trade-offs people need to make to get it right, and what misconceptions people often get stuck on. To do this, some of the older, foundational blog posts on Interconnects were reworked to stitch together the story behind key mathematical topics. For this reason, a lot of the explanatory text is likely higher voice than your average textbook.

This is the book I wanted to read when I was getting started a few years ago! With how many people still ask me basic post-training questions, in fact a population that’s accelerating in size, I suspect this book will be very well received. I still use the book regularly and hear from established researchers all over the industry that they do too. So, it’s not a beginner book — it’s more tailored to someone who has already finished a bachelor’s degree in CS — but if you master it you will be far ahead in your post-training worldview.

Share

A discount to readers!

The book is also freely available online and comes with a full 12 hour course (slides + video on YouTube), a simple code-base with suggested exercises, and model completion comparisons. It’s 50% off until August 19th on Manning with the code PBLambert.

Buy my book from Manning

Buy my book on Amazon

Yes, the title of this book is a little outdated — I’ve learned some lessons and fought some battles with the publishing process — but I can guarantee the content is very fresh. I regularly reference the book for my research work, and hear from friends that do the same. I added a section on on-policy distillation at the last possible moment! The book is shipping from Manning and Amazon US now, and from Amazon UK in October.

1. Intuitions for how RL algorithms change the outputs of models

By word or page count, the book is about 25% RL. This seems appropriate. If there’s one thing the book is doing it’s teaching people how to think about various RL algorithms. This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).

Below is an example intuition you should be able to follow after reading.

For example, here’s a fun figure that we’ve iterated on for the PPO clipping understanding. At the end of the day, PPO’s surrogate objective reduces to six regions. These can be seen as two gradients, when the advantage for a token is positive or negative, depending on the current value of the policy ratio. For an individual sample in a completion, it lives somewhere on this plot. If it was the first gradient step in the batch, it starts at 1 on the x axis (gradient always flows), then depending how the ratio updates after changing the RL policy behavior, the gradient is either the same or becomes 0 (which is what the clipping arguments are for).

This intuition filters very closely into how the systems are designed, in order to manage the gradients and numerical issues they tend to cause. The math-focused policy-gradient section is pretty thorough, covering all the algorithms you’ve likely heard about in the last 3 years:

• Deriving the Policy Gradient
• Vanilla Policy Gradient
• REINFORCE
• REINFORCE Leave One Out (RLOO)
• Proximal Policy Optimization (PPO)
• Understanding the PPO Objective
• Value Functions and PPO
• Group Relative Policy Optimization (GRPO)
• Group Sequence Policy Optimization (GSPO)
• Clipped Importance Sampling Policy Optimization (CISPO)
• Comparing Algorithms

2. An understanding of the crucial factors facing new RL systems and algorithms

Most of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput. The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.

Agentic tasks are only adding more infrastructure on top of these fundamentals. The book is designed to be the simplest resource to start from roughly 0 LLM RL knowledge and be ready to tinker with the systems. It starts with basics, such as explaining the general form of implementing an RL algorithm:

It continues with teaching you about loss aggregation — the idea that spawned DAPO and Dr. GRPO as some of the seminal, early GRPO variants — and truncated importance sampling — the technique used to make PPO work in early RL experiments.

• Policy-Gradient Basics
• Loss Aggregation Tradeoffs
• Asynchronous RL Systems
• Truncated Importance Sampling
• Example: PPO
• Example: GRPO

Share

3. The histories that lead to modern post-training

Knowing how a field came to be has always been fascinating to me. As you become an expert, knowing the history of your field better than anyone is what lets you make the best predictions (Bill Gurley gives similar advice in his recent book). In a time when the foundations of deep learning were being built, like the transformer, the core of modern post-training was also born in the alignment field. The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT. Much like those early to scaling LLMs, the people who created this field a decade ago deserve incredible credit for how modern progress has unfolded.

Chapter 2 of the book is a crash course on this, but the book is littered with this type of thinking.

4. Dispelling the magic of “distillation”

It’s really nice to have a boring textbook chapter on distillation given the broader AI policy discussions ongoing. This one, chapter 12, explains the various industry-standard ways that outputs from an LLM are used to train downstream models. When the technique is often described in nefarious ways, and as a tool of geopolitical competition, it’s a deescalatory action to break out a 300 page textbook to explain to someone how broad the term they’re attacking is.

With this, chapters 10 through 12 are all about making some opaque practices of the data industries clearer to readers.

The distillation chapter continues the theme I outlined above, explaining the key changes that needed to be made to transition the early knowledge distillation literature of 2015 to 2-3 key insights that got us to the multi-teacher on-policy distillation (MOPD) of models like Xiaomi MiMo-V2-Flash and DeepSeek V4.

5. A survey of all the other little headaches you encounter when trying to do post-training right