Eric Jang 详解如何用现代工具从零构建 AlphaGo

Eric Jang walks through how to build AlphaGo from scratch, but with modern AI tools.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.

Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.

Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.

Watch this one on YouTube so you can see the chalkboard.

And check out the flashcards I wrote to retain the insights.

Sponsors

• Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh
• Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh

Timestamps

(00:00:00) – Basics of Go

(00:08:17) – Monte Carlo Tree Search

(00:32:04) – What the neural network does

(01:00:33) – Self-play

(01:25:38) – Alternative RL approaches

(01:45:47) – Why doesn’t MCTS work for LLMs

(02:01:09) – Off-policy training

(02:12:02) – RL is even more information inefficient than you thought

(02:22:16) – Automated AI researchers

Transcript

00:00:00 – Basics of Go

Dwarkesh Patel

Today, I’m here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You’ve been on sabbatical for the last few months. One of the things you’ve been doing is rebuilding, improving, and hacking on AlphaGo.

Today, you’re going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang

I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning. That was quite mysterious to me, and I’ve always wanted to understand that phenomenon a little better.

My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It’s always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree.

Eric Jang

If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. I’m not certain if it’s stronger than AlphaGo Zero, AlphaZero, or MuZero, but it’s very strong, and this is what most Go practitioners today train against when they’re playing an AI.

Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

Dwarkesh Patel

We should first discuss how Go works. How does the game work?

Eric Jang

Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. I might start by putting down a black stone. Black always goes first. Go ahead. The way you capture an opponent’s stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it’s cut off from oxygen, if you will, and it’s a dead stone. Now I control these four stones as well as this empty intersection here.

There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you’re actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it’s actually fine. You put it down, and it immediately resolves to death, so the outcome is the same.

Let’s start over and play a few stones, and then I’ll explain some more. I’ll just start there.

Dwarkesh Patel

I’m basically playing randomly here, but I’m trying to get around your stones and see if I can surround them.

Eric Jang

This move exposes one empty neighbor for your white stone. It’s akin to a check in chess. If you don’t respond immediately by putting one here, then I can immediately capture this.

Dwarkesh Patel

I see. Because it’s the diagonals that determine whether you’re surrounded or not.

Eric Jang

The cross-section, not the diagonals. This one is surrounded on three sides, so you’re at threat of losing that stone if you don’t play one immediately there.

Now you can see that I’m starting to pressure you, because by putting a stone here, you’re forced to put one here.

Dwarkesh Patel

Otherwise, you would have this two-block to yourself.

Eric Jang

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I’ll do in response once you do that.

Dwarkesh Patel

You have a lot of confidence in my abilities, but I’m guessing you’d put the black here.

Eric Jang

That’s right, and then I would capture all three of these stones.

Dwarkesh Patel

So I should just assume that this little block is gone.

Eric Jang

Yes. In Go, it’s actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

Dwarkesh Patel

Presumably you’d put one here.

Eric Jang

So now I would capture this entire group, and this would be mine.