Rebuilding AlphaGo from Scratch: What Search, Self-Play, and Supervision Reveal About RL for LLMs

Today, I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics.

You've been on sabbatical for the last few months. One of the things you've been doing is rebuilding, improving, and hacking on AlphaGo. Today, you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development.

Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning.

That was quite mysterious to me, and I've always wanted to understand that phenomenon a little better. My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It's always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree.

If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. I'm not certain if it's stronger than AlphaGo Zero, AlphaZero, or MuZero, but it's very strong, and this is what most Go practitioners today train against when they're playing an AI.

Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

We should first discuss how Go works. How does the game work?

Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. I might start by putting down a black stone. Black always goes first. Go ahead.

The way you capture an opponent's stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it's cut off from oxygen, if you will, and it's a dead stone. Now I control these four stones as well as this empty intersection here.

There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you're actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it's actually fine. You put it down, and it immediately resolves to death, so the outcome is the same.

Let's start over and play a few stones, and then I'll explain some more. I'll just start there. I'm basically playing randomly here, but I'm trying to get around your stones and see if I can surround them. This move exposes one empty neighbor for your white stone. It's akin to a check in chess. If you don't respond immediately by putting one here, then I can immediately capture this.

I see. Because it's the diagonals that determine whether you're surrounded or not.

The cross-section, not the diagonals. This one is surrounded on three sides, so you're at threat of losing that stone if you don't play one immediately there. Now you can see that I'm starting to pressure you, because by putting a stone here, you're forced to put one here.

Otherwise, you would have this two-block to yourself.

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I'll do in response once you do that.

You have a lot of confidence in my abilities, but I'm guessing you'd put the black here.

That's right, and then I would capture all three of these stones.

So I should just assume that this little block is gone.

Yes. In Go, it's actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting. Presumably you'd put one here. So now I would capture this entire group, and this would be mine.

There's one more case I want to demonstrate, which I actually had a bug in my code for recently. Let's consider a formation like this, with other pieces on the board in play. Let's talk about how the game ends. In this territory, who controls these areas? Is it white or black?

White.

It's actually black, because I've surrounded this whole area. Assuming I have other black stones here, it's very hard for you to break this out of the control of these stones.

And when the final score is tallied, would these ones also count as being in...

Great question. This is where different rule sets have different ways of scoring. We should talk about how you resolve scores between humans and how you resolve scores in computer Go, because there's some ambiguity in how humans evaluate this. Most humans would look at this board configuration and conclude that black has totally surrounded white, and white has no chance of life. We could play out more here, but at the end I would capture everything. However, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip.

This is where it's a little bit hard for a computer to decide these kinds of things. How do humans do it? It's worth thinking about how humans resolve this, because this will map later to how we think about the deep neural network. Humans basically say, "I think the game is done," and then you have to also say, "I think the game is done." Then we'll say, "I think these are my stones," and you have to agree. If you don't agree, we keep playing. Essentially, once two humans—their so-called value function—agree on a consensus, then the Chinese rules resolve that.

In Tromp-Taylor scoring, it's perfectly unambiguous, so it can be decided algorithmically by a computer. If you have this at the endgame, the way you score it is that you first count how many stones you control, and that's unambiguous. Then you count how many empty intersections are not touched by your opponent's stones. These intersections would not count for either player, because all of these intersections are connected to both white stones and black stones. If this were like this, then white would get three points. This is a little odd because a human would know that white is actually losing these points. But Tromp-Taylor scoring would consider white to have all of these points as well as these points. So that is a very big difference in how computer Go scores things and how humans score things.

How does the game end?

The game ends when either a player chooses to resign or both players pass consecutively. Those are the rules.

Now help me crack this with AI. Let's understand how AlphaGo actually works and how somebody in the audience might be able to implement it.

Let's start with an intuition about the underlying search process used to make moves, and we'll layer on ideas from deep learning to make it much more efficient and tractable.

Go is a game with just two players. We're going to draw a person here, and we're going to draw an AI here. Let's say this person is playing black, so they go first. They go here. Now the AI is going to make a move based on what it sees here.

There's a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. You would need at least three different values. Maybe you could use zeros, ones, and twos. The AI might see something like zero, zero, zero, zero, one.

This is the input to the AI on its turn. The AI can choose. Let's just pick three possible random moves it can make, and I just drew these at random. Which move is best here? Well, we don't know until the game ends. Go doesn't have any kind of local reward for which move here is good. This is what makes Go a very difficult game. You don't actually know who won until you really get to the end.

How deep is this tree? On a 19x19 Go board, there are roughly on the order of 361 moves on any given move, and of course, as it fills up, you have fewer moves. The number of steps in the game can be somewhere from 250 to 300 moves. Experts might decide to end the game well before that, but under Tromp-Taylor scoring, you actually have to play things all the way to the end. So this could be 300 moves, a depth of 300 in the tree.

If you keep on expanding possible moves—here the AI goes, then here the human would go, and so forth—you find that you end up with an enormous explosion in the possible game outcomes originating from just this one state. This is something on the order of 361³⁰⁰, which is far more than the number of atoms in the universe. Of course, there are redundancies and symmetries, so it's not actually that, but if you were to do a naive tree with no merging of children, you end up with a tree about this big.

What do you mean by "merging of children"?

Let me use this board here. If we start here, and then you play here, and then I play here, and then you play here, that's equivalent to me starting here, you playing here, me playing here, and then you playing here. Both arrived at the same spot, but through different paths. So this child node can be thought of as a shared ancestor.

Got it. And I guess it starts at 361 but it decreases by one each time.

Yes, the branching factor decreases by one each time.

In any case, this is a very large tree. This is also why computer scientists for many years thought that Go was not a tractable problem this century. The amount of compute you would need to exhaustively search every possibility is just too large. If you could, Go is actually a deterministic game. In any given state, you can compute the best possible strategy you can make in order to win the game. You can search all the possible futures where you win, and then just make sure you always stay in that set of futures.

AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable. Before we get into how neural networks are involved, let's talk about how we can, assuming we had a powerful enough computer, search this tree to find the best move.

In the beginning, you're not going to build out the whole tree, because storing it would be very expensive. Instead, you might interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see what else is there. There are some early algorithms in the bandit literature like UCB1, which is not exactly appropriate for a sequential game like Go, but very much inspired the action selection algorithm used in AlphaGo.

UCB1 looks like: on every move, we're going to take the best action—the argmax over a that maximizes Q of a, and I'll explain what Q of a is in a moment—plus some sort of exploration bonus.

On every node, we're going to track a few quantities. Let's consider each of these a node. This is the root node where you're making decisions from, and these are the children of the root node. Each node is basically a data structure that stores a visit count of this child node.

How often the parent visited this node.

Yes. One thing that's easy to trip on if you come from robotics or other kinds of reinforcement learning is, where are the actions? I'm only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you actually can just infer the action based on the child. If I go here, that implies an action, and this is the state we resolve to.

If you ask LLMs to vibe-code an MCTS implementation, they will most likely design the right data structure here. But it's sort of a chef's choice. You can rewrite the tree structure however you like. This is what Claude 4.6 wrote for me when I asked it, and it was a very reasonable choice.

Qₐ represents the mean action value of this action. I'll use a subscript a to denote that this corresponds to taking a specific action to get here from the root node. Taking a gets us to this node here. We're also going to store the probability of taking this action.

Again, from the parent?

From the parent, yes. What are the odds that we sample this one? This will become relevant later. We've talked about a deterministic tree for now, so I'll bring probabilities into this later. Finally, we have a dictionary of children, which is just more of these nodes in a classic linked list-style reference tree. This is the basic data structure to implement a tree.

In AlphaGo, they use a slightly different action-selection criterion called PUCT, short for Predicted Upper Confidence with Trees. When you select which child to take, you do argmax a of Q(s,a) plus a constant. The equation forms are pretty similar.

These are both scoring criteria. You want to argmax this quantity and you want to argmax this quantity to determine which action to take. Let's break down the intuition of how you select actions here. Q(s,a) is the mean action value, so how good a given child is on average. If you actually knew the whole tree, this is all you need to select the best action. You don't really need to do more than that. But if you're interactively building this tree as you're figuring out what the Q values should be, then occasionally you have to try some other actions as an explore-versus-exploit trade-off.

In both UCB and PUCT, there's this term that basically rewards taking actions you haven't taken before. As we mentioned, each node stores the visit count of taking that specific action. Everything is initialized to zero. For a given action, let's call it action a, initially it's zero. As n is increasing, if we've already made 10 action selections from that root node but we haven't picked a yet, then this term starts to become quite large for a. Conversely, if we've chosen a 10 times out of 10, then this term is quite small. It diminishes very quickly. The same thing is true here.

Just to make sure I'm understanding, let me put it in my own words. Let's focus on UCB. Conceptually, you can think of it as two different things: the Q and the exploration term. Let's be clear about what Q is. Q is basically saying, once we do these rollouts—you're actually running all these simulations—you go down the tree and figure out: if I end up at the terminal value of this tree, do I win this game or not? You average whether I win or not across all the leaves of this tree starting from this node. That average, you put in Q.

So Q represents the probability that I'll win this game starting at this node. That is your sort of exploit. It's saying: I've run these simulations, and I think this is a good move or not. The other term is saying: have I explored this branch enough yet relative to the other actions I could be exploring, or have already explored?

If I haven't explored this branch yet, maybe I think it has a low score, but I just haven't explored that many leaves down this node in this tree. So I should try this even though Q, the exploit term, is telling me it's not that valuable.

Because ln(n) grows slower than n, over time you will move from the argmax being dominated by the exploration term, which is the second term here, to the argmax being dominated by the Q term,

which is when you've done enough simulations and are confident that this is the branch to go down.

Yes, that's right. The motivation for UCB was to come up with an algorithm where, if you don't know the payoff of the different actions you can select, this strategy, given some exploration term here, bounds your regret in terms of how wrong you can possibly be. I don't know the proof. I also don't know if this one is proved to have logarithmically or square-root-bounded regret, but I think the algorithm was derived to look something like this. You can tell these terms grow a little differently, and this is to account for the fact that Go has many more actions for any given move compared to your standard bandit problem.

One small clarification: you talked about simulations and probabilities. We should remember that Go fundamentally is a deterministic game. Where does the notion of probability come from here? If you had a very powerful computer, there are no probabilities. You can just compute the true average of the mean action value. So where does the probability come in?

In computer Go before AlphaGo, we've always done some sort of Monte Carlo method where we take the expected Q value averaged over a randomly selected tree. That randomly selected tree is where probabilities come in. The interpretation of Q is: what is the expected action value under the random distribution induced by some random search process?

Where does the random search process come in? That's where Pₐ, of action, comes in. If we assume a naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves. You would be taking this average over a very diffuse tree. This is a valid integral, but it's very slow because you're going to consider a lot of trees that have very low value. It's essentially almost like an importance sampling problem. Only a few actions and paths contribute high value, and almost everything else is low value. So that's a tricky problem here.

This is the action selection criterion for how you decide which moves to go down. As you move down in tree search, you will eventually run into a node where it's quite clear you've won or lost. At the very end of the game, when there are no valid moves left to play under Tromp-Taylor scoring, you can decide whether you won or lost. This is the final return of the whole game.

We can assign a value, U, to a terminal leaf node of the tree, but how do we assign values to the nodes prior to that, the parents? You take the mean action value, which is essentially your average. Suppose these were all leaf nodes. The mean action value of this node is just the average of whether you won or lost at the leaf nodes. Correspondingly, you can walk up the chain and say the mean action value of this node—let's call it Qb—is just a weighted average of these ones here. The weighted average could depend on whether you have a different sampling distribution. But the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can go backwards—this is called the backup step—and assign values to these nodes or actions corresponding to the average over the final terminal leaf.

If you were to do this without neural networks, it would still be intractable. You would have trouble finding which actions to sample. A lot of the actions would contribute very low value, especially if you're trying to fight your way out of a losing position. Only a few actions give you high value, so the search in practice is still very expensive. But the idea is that because Go follows a tree structure, you can inform a very good estimate of the value of this node based on the downstream values, assuming they're all correct and you've searched deep enough.

Your earlier explanation—about the sorts of states where it's obvious to a human who's going to win, but deterministically you still have to play it out—actually drew upon the intuition for why 1) the value function is both trainable and 2) necessary in order to learn this game effectively.

Maybe it's worth defining value in the first place. We talked about U being your final resolution of whether you won or lost. This is a terminal leaf node condition. Humans don't play all the way to the leaves of the tree. They stop dozens of moves before, maybe even 100 moves before in high-level play. How do they know? You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates p(win). The human glances at the board and knows, "I'm probably going to lose." They're essentially running a neural network that looks at a board and implicitly amortizes a huge number of possible game playouts. They take that average and decide whether the board is winnable or not, and whether they should concede or keep playing.

This is remarkable. If you think about the beauty of something like this, a neural network in a human can somehow do all of this simulation at a glance. They just know within a few seconds, without actually playing every single game logically, based on crystallized knowledge and experience. They can do this. This gives us a hint that in games like Go, there are ways to radically speed up the search process. This is one of the fundamental intuitions behind why AlphaGo works. You can train a value function to look at a board and quickly resolve the game without playing out all of these trees to a very deep search depth.

Makes sense. I will say for the audience, for previous episodes when I was prepping and it seemed relevant to understand how AlphaGo works, I would find it very confusing. But it's the kind of thing where once you understand the problem in this way and then build the next few pieces, it is actually a lot more understandable and it makes a lot of sense. It's okay to be confused right now, but it's probably simpler to understand, by the end of this lecture, than you anticipate. I’ll just make that note for the audience.

The important intuition at a high level—to step back about where we're going with all this—is that classically, for games like Go, you could build a tree, but we don't have computers powerful enough for that. Estimating the value of every action you could possibly take is also hard because you don't know until the end of the game. You could take averages by playing them to the end, but that's also hard because you don't know which actions to take to sample these averages.

Conceptually, there are two problems: the breadth of the tree and the depth of the tree. AlphaGo gives us a way to shrink both of those to be tractable. That's essentially the core idea behind it.

We take the idea that humans can glance at a board and instantly predict whether we win. That maybe gives us the opportunity to truncate how deep we search. We also know that humans can intuitively, at a glance, decide what moves might be good on a Go board. These are two things we can use deep neural networks for, to accelerate the search process.

Before we talk about neural nets, let's go back to how this playout works. We've only talked about making one move. The AI looks at this encoded Go board. It has a tree. It searches deeply into the tree to find out which of its actions might be the best, takes that action, and goes back to the human. Now the human sees a Go board that looks like this, and they make their move. Maybe they put their stone here. Now we go back to the AI, which now looks at a new encoded board. I've used 2 to denote the AI playing as white, 1 to denote the human playing as black, and 0 as empty. On the AI's turn, it does the MCTS tree search all over again from scratch. It throws away the old tree that it searched last round. Now there's a new root node and it begins to search anew. You can basically think about MCTS as a search algorithm that decides which moves to play best, aided by neural networks, and it's done on every move. Let's talk about the neural network part of this.

While you're erasing, another thing that was important for me to understand was about the MCTS data structure with nodes and children of nodes. This is done per move and reinstantiated once a move is made. A human makes a move, then the AI looks at this and runs a bunch of simulations to figure out what move to make next. A simulation is exploring one more node in the MCTS tree. Once you run 1,000 simulations, that informs the probability of what move to make next, as you'll explain. That’s what you store. You choose the best move given those probabilities. You discard all of that, the next player makes a move, and you restart this process at the beginning of every move.

Correct. One small addendum: you don't discard all of that. You keep one thing behind that we'll use later.

Just like I did for Reiner, I wanted to make flashcards for this episode so that people could retain these concepts. And ideally, an LLM could generate some candidates for me to then refine. But to actually get high quality suggestions, I needed to design a whole pipeline where the AI could take and ingest screenshots of the blackboard at the right timestamps and then make SVG diagrams in case visuals were helpful, and then run the writing and drawing through a critic and then revise the card in response to this feedback. It's very hard to accomplish this just by stacking LLM calls. This sort of step-by-step recipe works much better if you have a durable agent that's been engaging with the task across all the previous stages.

So I used the Cursor SDK to spin up an agent for each card. The Cursor harness saved me a bunch of work in designing some custom context scaffold or figuring out how to design tool calls for taking screenshots or making animations. These agents all run in the cloud, so I don't have to worry about leaving my laptop open. I just get an email when I have candidates to review. You can check out my cards at flashcards.dwarkesh.com. You can start building with the agents SDK at cursor.com/dwarkesh.

Now that we have a basic intuition of how moves are made with search, we're going to talk about how neural networks can speed this up by providing an analog to human intuition.

There are two networks. There is the value network, which takes in a state and predicts, am I going to win or lose? It's a binary classification problem. Then we have a policy network, which induces a distribution over good actions to take. I'm going to draw a one-dimensional flattened move distribution, but this is really a square grid. These are the probability distributions over good actions. Both of these are categorical classification problems. You can train this like any classifier with deep learning, cross-entropy loss, that kind of stuff.

The specific architecture does not matter too much. I tried a few different architectures. Transformers work, ResNets work. For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. But this may not always be true.

Wait, why is that?

They provide the inductive bias of local convolutions. Generally, transformers start to outperform residual convolutional networks when you want more global context.

One interesting finding from the KataGo paper was that they found it quite useful to pool together and aggregate global features throughout the network, to give the network a global sense of how to connect value from one side of the board to the other.

What does it mean to aggregate global features?

If you have a very large 19x19 Go board, and you've got some battles going on here and some battles going on there, when you pass this through a convolutional neural network, the receptive fields are going to be good at computing local things and making that invariant. But they won't be able to connect these two features easily. They need to be pooled together and attend to each other somehow. The argument for why transformers are good for computer vision tasks—vision transformers and so forth—is that because they have global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can learn the invariant local features through data.

I've tried very hard to make transformers work for this problem because I was curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But try as I might, I haven't figured out a way to make transformers better than ResNets for now.

One more tangential question. It makes sense why transformers, with their global pooling of information, would be better if you need to consider information that is not just spatially… CNNs give you a bias that the things next to you are especially relevant. And then they're aggregated up slowly. For games where it isn't that relevant what is happening locally—you just have to consider the whole thing—you're saying transformers would work better. We’re talking about the spatial dimension. How about the temporal dimension? Right now we're only considering the previous move because it is a deterministic full-information game. What if it were something like poker or Diplomacy, where a bluff they made a while back is relevant to understanding now? And to decide your next move, you need to consider all those previous states. Would that change the consideration of what inductive bias and which architecture is most relevant?

Great question. Go is a perfect information game. In perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy. If you know your opponent has a particular bias, like they love to play aggressively, you can in principle counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there exists a single Nash equilibrium that can be decided solely using the current state. That is a design choice AlphaGo made, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman. No human strategy seems to be able to beat it.

There are variations of this where you would need to consider temporal history. This is a very exciting research area, and I'd encourage people to fork my repo and try it out. If you were to play 2v2 Go, you actually need to model your partner's behavior. You may not have information on how they play, so you need to aggregate information on how they play so you can respond accordingly. These are situations where it's no longer a perfect information game. In games of imperfect information or partial observability, you do need context to build a model. That's a place where things can get very exciting in terms of self-play or Diplomacy-style games.

Returning to the neural network, the architecture again is not super important. You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use Karpathy-style AutoResearch hyperparameter tuning to make your architecture pretty good. You don't have to worry too much about that. You just need to set up the problem so that you have a target optimization.

We're going to pick a somewhat arbitrary architecture that worked for what I did. But again, this part is not super important. You have your encoded board state and, similar to an RGB, we're going to have three channels. One channel to encode black, one to encode white, and one to encode empties or a masked region if you want to train on multiple board sizes.

I'm actually not going to talk about multiple board sizes for now. That's a bit too complicated. We have this two- or three-channel RGB-like image, and we feed it into a ResNet.

Then we have two branching heads. One head predicts the value function, which is a single logit, let's just call it R¹. Then we have the policy, which is R³⁶¹.

This is the architecture. We're going to train this to predict the outcomes of games given the board state, and we're also going to train it to predict what are good moves.

The original AlphaGo paper, called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later, they removed this restriction by having the model teach itself how to play well. I find it super nice for implementation, for your audience, to always initialize your experiments to something easy and get the problem working before trying to bite off the whole thing and learn tabula rasa.

In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible, especially if you're doing something new that you haven't done before. Always pick something that works and then get it to do something better, rather than start from something that doesn't work at all and try to make it work. Under that philosophy, it's a great idea to start with something that has a good initialization.

We're going to take human expert plays and train this model to predict good actions. We're going to take all the moves in which an expert won and predict those actions. Regardless of board state, whether you won or lost, you're going to predict the outcome.

You might be wondering, for some of the early boards where only one stone has been put down, how could you possibly know who the winner of this game is? Well, if you have hundreds of thousands of games, on average you'll probably see that boards starting like this have half of the games branch off and win, and half branch off and lose. So that'll actually be fine. When you train this model to predict those outcomes, the logit will sort of converge to 0.5.

For these things, it's expected that once you train the model, a starting board state will look like 0.5, and then as you progress towards the end of the game, the win probability will either go up or down. This is your move number. As you get hundreds of steps into the game, it becomes much clearer who's more likely to win or lose under your expert data distribution.

I didn't understand the significance of why this way of thinking about value is especially relevant to the expert data.

It is not relevant to the expert data. It's true for any data that you trained it on. If you were to learn tabula rasa, you would also expect this to fall out.

Imagine you're vibe coding AlphaGo. You gather some expert datasets from KataGo online, or you have a dataset of human players, and you train this model. It turns out this model is already a pretty good Go player. It will most likely beat most human players. If you just take this policy recommendation and take the argmax over these probabilities—if you take the argmax and just take this action as your Go play—it'll be a very fast Go player that doesn't think in terms of reasoning steps. It just shoots from the hip, and it'll be a very strong Go player.

This is already quite miraculous if you think about how ten neural network layers, maybe under 3 million parameters, can already do something that impressive. You can start this way. It's important when implementing this to verify that this is probably true. It's good to verify that your Go rules are implemented correctly and that you can run these simulations relatively quickly. Just as a checkpoint, you want to make sure you can actually do this basic step before you try to layer on more complex things like search.

But we can do a lot better than taking the raw neural network and playing the moves. So let's apply the neural network to improve Monte Carlo tree search.

We start with our root node, and we now have a four-step iterative process to do MCTS. This tripped me up when I was first reading the paper and trying to understand it. Essentially what we're going to do is choose a number of simulations, and this number varies. It can be somewhere between 200 and 2,048. I believe in the AlphaGo vs. Lee match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible. But in training, you don't actually need too many. KataGo, I think, uses something in this order as well.

Do you know if they used… If you watch the documentary, they had a laptop out during the game. They didn't use the laptop itself. It was on some—

It was on some TPU pod, I think.

Honestly, kind of unfair. Lee is not using 1E22 FLOPS to do a move.

Fair enough. Interestingly, modern Go bots don't need that much compute at test time. What we'll find out, as we talk about how the MCTS policy improvement works, is that over time the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network. You can do all of that work with one neural network forward pass. The TPU pod will always add the extra oomph on top, and that's what they wanted for the match.

We're going to pick this number of simulations thing, and for every simulation, we're going to do several things simultaneously. We're going to see which moves are the best in the current tree. We're going to add extra leaves to the tree if we get to a point where we need to add a leaf, and we're going to update the action values for the tree. Every simulation involves this four-step process: selection, expansion, evaluation, and backup.

At the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node, or our current board that our AI wants to play at. We're basically going to select the best action for this. When this root node is created, we also know that we can evaluate it under our neural network and get the quantities Vθ, as well as our probability over actions. And I'm going to say root. For all of the actions here, we can create a bunch of children. In this case, I'm drawing a 3x3 board with one missing, so there are eight possible children associated with this root node. Each of these has an associated probability of taking that action, so there's P₈, P₁, ... P₂, and so on.

So at the beginning of our Monte Carlo tree search, we have our root node, and we can initialize it with some children. The policy network evaluated on the root node gives us—on a 3x3 board with one existing stone placed—eight possible children that this AI could take. With each of the children, the policy network also gives us the probability of selecting that child.

The first step is to do the selection of the tree. Again, this is a very shallow tree. All we have so far is essentially a tree of depth one. Our first move is to select by maximizing, or argmaxing, the PUCT criterion, which is basically Q(s,a) + CPUCT x Pₐ x (√N / (1 + Nₐ)). For each of these, Nₐ is zero for all the actions initially. N is zero. So we're going to pick according to this. Initially, the chosen action is most likely going to be biased towards the highest-likelihood action, because these are uniform for every node.

Let's suppose P₁ was the highest-probability node, so you selected this one here. Now you get to this node and you realize it's not a leaf node. It's not a terminal game, so you cannot resolve the final resolution. The next step is expansion. You will then run this board state through the policy network. Note that this is the AI's move. The AI is making this move. When we expand this tree, we're now thinking about what the human, or any opponent, might do.

When we evaluate the node here, we're going to evaluate it from the perspective of this player. This node has possible actions that we could take, and we expand the leaf nodes here. For each of these nodes that we could arrive at, we're going to now check how good they are. From here, the human could play here, here, or here. We're going to store Vθ for each of these, so Vθ of Node₁’, Vθ of Node₁’’, and so on.

We're basically using our neural network to make an intuitive guess of how good this board is from the perspective of this player. Fortunately, because it's a zero-sum game, it's easy to deduce that the value for this player at this step is just one minus the value from the other perspective. It's easy to flip the search process depending on which player you're at.

This is the expansion step. You've taken a non-leaf node, expanded it, and evaluated the value. This is essentially a quick guess as to whether I'm going to win or not if I were to play to the end. You can almost think about Vθ as a shortcut for searching to the end of the tree, for any given simulation. This is essentially the evaluation step. We're evaluating the quality of each of these boards.

In the original AlphaGo Lee, they did something kind of interesting. They took this value and averaged it with the value of a real Go playout. They actually played a real game from here all the way to the end. I'm just going to draw this squiggly line to indicate some path. They play this all the way to Tromp-Taylor resolution of a full board. So this is a zero or one. They took this value and averaged it with this one here. The formula was α x Vθ of some node plus (1 - α) of a truly randomly sampled playout. You might be wondering how they play this out. It would be very costly to do another search on this playout, almost like a tree within a tree. They don't do this. Instead, they just take the policy network and play it against itself. They use it as both players, and they just play it all the way to the end. This helps ground the estimates here in reality because you can get a single-sample estimate of whether you win or not. In the endgame, where the board is almost resolved, this becomes quite useful because the play according to the policy will most likely decide a pretty reasonable guess of the game, so you're not facing a problem where this becomes untethered from reality.

It turns out this is totally unnecessary. In all subsequent papers after AlphaGo Lee, they got rid of it. In my implementation, I did the same, and it speeds things up a lot because you don't have to roll these games out on every single simulation.

Okay, just to reinforce my own understanding and re-explain it. For the audience, by the way, in case it's not obvious, the P there in the select, that is the probability coming from the network in this case.

Correct. The policy network here.

Fundamentally, a simulation, just think of it as rolling out one more node in the search process.

Almost. A simulation is easy to think about when the whole tree already exists. You just walk down the tree using the PUCT selection criterion, and then you keep going. In AlphaGo, the data structure is such that we begin with a tree that basically only has depth one—its only children—and you want to iteratively build out the tree as you're also selecting actions down the tree. That's the core thing here. Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.

Let me finish up with the last step, which is the backup. Once you've scored these things, the Q value assigned to the node here for taking this action is just the average across your evaluated values. You take a running mean over all the simulations you've taken, averaging the values of the children nodes. That's the backup step, and once you evaluate this, you can recursively go back up. If you know the action value of this node, you can then take the average on its parent, and so on.

You have this four-step process: 1) You're choosing the best action you know of so far. 2) You may run into a node you haven't been to before, so you need to grow the tree a bit. 3) You run it through the network to guess whether you're going to win or not. 4) You walk all the way back up to the root node to update your values on what the best moves are. As you do this iteratively, because you're always selecting according to this criterion, you're always going to be selecting the best action you think at any given branch. The final visit counts of how often you chose these things will reflect your correct policy distribution as induced through this search process. The visit count we stored in the node earlier actually becomes the vote for which action we should finally select.

As a test of understanding, it's worth thinking a little bit about whether we could make this even simpler. Could we actually maybe even get rid of this one and still make the thing work? Recall that when you do an expansion and then an evaluation at this node, you are checking the win probability of each of the child nodes. So if this one is one and these are zero, you do know something about which action might be better to take. Why would you still need this, right? Why not just normalize the values into a distribution and call that your policy distribution?

This is fine. You can do this, and it probably does work. But in practice, having a single forward pass that gives you a pretty good guess is how the breadth is pruned out. There is a duality here. It would be weird if the policy recommended an action that disagreed with the value. If a policy said this was very high probability, but this one said it was a low value, then there's something fundamentally wrong between your policy head and your value head. They are linked, and you probably could get rid of this if you came up with a different way to recover it from just the value evaluations.

Right. Just to make sure I understand, the reason you don't do that is so you don't have to do 360 independent forward passes to say, "Okay, here's the value of everything. Let's argmax over it." Instead, you can just do one forward pass and get the probabilities of all of them.

You can usually batch these somewhat efficiently, so it probably isn't a huge computational burden in practice. But yes, you would have to pass up to 361 boards into a single mini-batch update to evaluate all the values, then normalize them. There's actually a more important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself and recursively improve its own predictions and search capabilities. That's where having this as an explicit entity you're modeling, rather than an implicit normalization over your value, is a good idea.

Makes sense. Okay.

We talked about the simulations. What you end up with as you roll out the number of simulations is a tree that looks like… I'm drawing a very low-dimensional version of this. Of course, in the real game, it's much more high-dimensional. You'll end up with a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed to be too low. But along one path, there will be a set of actions with very high visit counts that gravitate towards that one set of decisions as you increase N. This is the mental picture of what the tree in Monte Carlo tree search looks like. You should contrast this with an exhaustive tree like in tic-tac-toe, where there are nine actions, then eight, then seven and six, and so it's a nine-factorial-sized tree. The Monte Carlo tree search in Go is very sparse. It only considers the paths that you've expanded children nodes on.

Now that we have the search algorithm that applies the value function as well as the policy function, we can talk about how the Monte Carlo tree search algorithm can act as an improvement operator on top of these guys here.

Twenty years ago, Jane Street's data center fit in the corner of an office. Ron Minsky, who co-leads the tech group there, told me about how it all got started.

One of our compute clusters we called The Hive, and I remember the first version of the Hive was literally like six Dell boxes stacked on top of each other at the end of the row. And the trading systems themselves we also had there because we actually wanted the ability to make sure we could turn the damn thing off. I mean there were ups and downs, like literally at some point,

one of the people who was cleaning the office unplugged one of the trading systems in the middle of the day as they were vacuuming. So in the end it is in fact better to have it all in a data center. Jane Street's data centers have come a long way since those sixth Dells, and I got to tour one of them in Texas with Ron and Dan Pontecorvo, who leads Jane Street's physical engineering team. These cabinets, these GB300 cabinets, consume at peak about 140 kW each. Compare that to traditional air cooled, you're talking about 10 to 40 kW? It's a lot more.

We got deep into the details of running one of these data centers. Things that I had never considered before. It's filled with a liquid, a mix of deionized water and propylene glycol, 25% of propylene glycol. That's to inhibit any bacteria or algae growth. I don't love the world where we have to worry about bacteria growing in our servers.

I got to see way more of what actually happens in a data center than I've ever seen before. Jane Street was willing to literally pull up the floorboards and take out the racks and take me to the back where all the chillers are. You can check all of this out at janestreet.com/dwarkesh where we posted the full tour.

We now talk about the RL part of how this thing gets stronger by playing itself.

Let's say we play a game. You make a move. The AI will compute the search, and this is the visit count distribution. Let's say this is your initial policy recommendation at this node. After MCTS, it gets more confident about one of these actions. Maybe the distribution looks a bit more peaky like this based on the search. Of course, you can tune the search process so that it ends up more diffuse, but that's probably not a good idea. MCTS should get more confident about specific actions over others. It might place a lot of weight on other actions initially, and then as you increase the number of sims, it should converge to a very peaky distribution. Let's call this π.

Let's wrap this in an MCTS operator of (a | s). After applying the MCTS process, your policy recommended distribution looks like this. It's a bit more peaky than the previous one. Then you take the argmax, or maybe you just sample from this. It doesn't have to be the argmax. You make your move, throw away the tree, and begin anew on the next move. Again, you compute a new distribution. Initially, maybe your guess looks like this, and then you refine it through MCTS.

There should be one more X on the board, right?

I'm sorry, that's correct, yes. Something that looks like this. On every move, you have your initial guess from your policy network. Then the search process, which combines your policy network and your value network, arrives at a more confident action that you take, and so on and so forth. Then the game ends, and one person wins and one person loses.

The beauty of how AlphaGo trains itself is that it can actually take this final search process—the outcome of the search process—and tell the policy network, "Hey, instead of having MCTS do all this legwork to arrive here, why don't you just predict that from the get-go? Why don't you not use this guess and predict this to begin with?" If you have this guess to begin with in your policy network, then MCTS has to do a lot less work to get things to work. If we draw a test-time scaling plot…

Let's say this is the number of simulations. At zero simulations, your implicit win rate is here. If you just take this raw action, this is what your win rate is. Let's say we increase the number of sims, maybe you have a win rate curve that looks like this. When you search for 1,000 simulation steps, that gets you to a policy here that gets you to here, which is great.

But if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here. Let's say this was zero, by distillation, if you then spend another 1,000 sim steps, you actually get to here. It's almost as if you could amortize the first 1,000 steps into the policy network instead of the search process, then you could begin at a much better starting point and get a much better result for the number of sims that you play.

The sigmoid type nature of test-time scaling: as the number of simulations increases, the increase in win rate gets smaller. Is that true even for the distilled network? That is to say, is there some gain where we start from the distilled network and get these early gains again, or is that just inherent to the nature of MCTS?

To be honest, I actually don't know the test-time scaling behavior of MCTS simulations. I believe it might be quite sensitive to how strong this one is in practice. I'm just drawing a monotonically increasing function that gets to one. Don't pay too much attention to the shape of the curve. Just know that it's monotonic with respect to sims. The idea of MCTS is very brilliant.

We got something better by applying search. Now, on our next iteration of updating this network, we're going to train it to approximate the outcome of 1,000 steps of search. Instead of starting here, we get to have our neural network start here, and then the play gets stronger once we then apply another 1,000 steps on top of it. You can keep going.

The training algorithm for AlphaGo is to basically take the games where you've applied the search on every move that the policy encountered—whether you won or lost, and that's quite important—and just train the model to imitate the search process.

There's an analogy to robotics, which is the DAgger algorithm. First I'm going to draw a schematic of the states: S₀, S₁, S₂, S₃. Let's say we took a series of actions in an MDP to get a trajectory. These actions may be suboptimal. Maybe we lost at the end of this game.

There is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. Maybe a better action here would have been to take A₀'. A better action here would have been A₁', and yet another one, A₂', A₃'.

What MCTS is doing is saying: you played this game where you eventually lost, but on every single action, I'm going to give you a strictly better action that you should have taken instead. It does not guarantee that you are going to win, but it does guarantee that if you take these tuples as training data and retrain your policy network to predict these actions instead of the original ones, you're going to do better.

This is very related to DAgger in robotics and imitation learning, where you want to collect an intervention here. Even if you're in a not-great state—for example, a self-driving car that veers off the side of the road—there is still a valid action that corrects you and brings you back.

Pedantic question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network, that when the value network hasn't been well-trained on finished games, MCTS is worse than a randomly initialized policy. Is it just a heuristic that MCTS is better than the policy, or is there some guarantee?

In practice, it is a heuristic. It does work in practice, but let me illustrate an example where MCTS can give you a worse distribution than your policy network.

This can often happen if your self-play algorithm has trained to a good point, but then somehow it collapses because it's not trained on diverse data or something.

Let's say we have a board state where the policy recommendations are very good. So Πθ (a | s) is great. But somehow, maybe because we're playing a lot of games where the bots just resign instead of playing all the way to the Tromp-Taylor resolution, they forget how to evaluate those late-stage plans. Like in the case we showed with the corner play, maybe 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states. You might end up in a scenario where your terminal value is very bad.

If the terminal values of the leaves are not good, then this will propagate all the way up and cause your PUCT selection criteria and your backups to be off. And then you end up visiting a very different distribution than what your policy initially recommended. Also, if your number of sims is low, you might have a variance issue where you just don't explore enough. It's only guaranteed to converge when you take N to infinity.

Variance in your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy recommendation. That's why it's not a guaranteed improvement.

That is why I suspect AlphaGo Lee had the playouts to the end in their training algorithm, so that they could ground this thing in real playouts. In practice, what you could also do is, for 10% of the games, prevent the bots from resigning and just say, "Resolve it to the end." That way you get some training data in your replay buffer to really resolve those late-stage playouts that normal human players would not play to.

If you assume that the value functions are correct, this is why MCTS gives you a better policy. It's a very critical chain of assumptions. Assuming this is accurate, your search process should give you a better recommendation than your initial guess.

If you have a cold-started policy—if you have an AlphaZero-type thing—really what's happening for the first few epochs is that the policy is kind of useless. What you're really just doing is, "Hey, let's play full games, and once we have played full games, for the preceding moves, we'll have labeled who won and who didn't win." The loss for AlphaZero has two components: how good is the policy relative to MCTS, and how good is the value prediction relative to who actually won the game from this move. You can think of this being applied to every single action or every single move. Really what's happening at the beginning of AlphaZero training is just that we're trying to get the value function to actually predict who will win the game if you find yourself in this state and you're this player. Functionally, that's all that's happening. Later on, once that's well trained, now the policy is also improving.

Correct. There's one trick I found to be pretty useful. This is not a peer-reviewed claim, so take it with a grain of salt. I found it useful in my own implementation to do the following. You want to first make sure that this is good before you invest a lot of cycles doing MCTS. It doesn't really make a lot of sense to do search on garbage value predictions. You want to start at a good place where this works.

AlphaGo Lee does a very good thing where it just takes human games, you train on them, and it just works. It totally works. You can also take an open-source Go bot, play it against itself, generate data, and that also works. If you have some offline dataset that has realistic, good play, you can easily learn the late-stage value functions pretty well. That's what you need to start the search process.

Sorry, can you just repeat that sentence one more time?

Sure. It's quite easy to evaluate a late-stage Go game. When almost all the pieces are on the board, it's almost like a decidable problem because there's lower and lower uncertainty as to the depth of the tree. So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree.

Then, as you play more games, the search will back up good values into the intermediate nodes of the tree. As you increase the amount of data, your value head gets a good intuition of what is a healthy board state versus a not-healthy board state. Those are much more subtle to judge in the mid-game than at the beginning or the end. The most difficult part to score is not the beginning, because the beginning is obviously 0.5, and at the end it's pretty obvious who's winning. The hard part that you want to learn in the value function is who is winning in the middle.

So this is actually very analogous to TD learning.

Yes. There's a beautiful connection to TD learning that we can talk about in a bit, contrasting it with Monte Carlo tree search. You first want to get good value functions, and expert data can give you a quick shortcut. I recommend for practitioners, just do that first to initialize to a good starting point.

If you want to do the AlphaZero thing or KataGo tabula rasa learning, what you can try to do is play random games on a small board. Just take a random agent. If you play 50,000 games, you'll actually learn a pretty good value function as well. On a 9x9 board, you can see enough of the common patterns with random play. If you train on both 9x9 and 19x19 data—and KataGo proposed one of these architectures—there's pretty good transfer learning from the value head evaluated at 9x9 to 19x19.

Right, because this, unlike other games, has very much a sense… There's not like a new kind of piece introduced when you increase the size or something.

If we take it to its limit and consider a very tiny 4x4 Go board, if you play 50,000 games, you're going to have a lot of end states that look like human play. It's just Tic-Tac-Toe at that point. If you broaden this a little bit to 5x5 or 9x9, it's not unrealistic to imagine that purely random play will generate pretty reasonable-looking boards. You can score those pretty easily. That is what gives you the bootstrapping to then improve your policy with search. But it's very critical that MCTS has accurate value estimates. You need to ground the value. Ultimately, MCTS will fall apart if you don't have a grounding function for the value.

I'd be curious how much compute you save by training the value and policy on the same network. Because they share the same representations, how much more efficient is learning? We've just talked about how they're making similar predictions, or they should be in line with each other. So I'd be curious if you're halving the amount of compute you have to do by keeping them in the same network.

AlphaGo Lee, the original AlphaGo paper, had two separate networks. In all subsequent papers, they merged them into two heads, and presumably this saves compute. But answering that question in a rigorous scientific way is a simple question that in practice, if you really want to chase the question down to its limit, takes quite a bit of work to really resolve. Intuitively, yes, they share a lot of representations. As we mentioned, your policy network and your value network, when doing evaluations, should agree. There really should be this consistency between them.

Tell me if this is the wrong way to think about it. When I learn how an LLM works and how simple RLVR is as an algorithm, I'm stunned by the kinds of things it can do. It can learn how to build very complicated code repositories simply from getting a yes or no. Here, if you understand it more deeply, just predicting MCTS, AlphaGo seems less impressive in retrospect the more you understand it. You're putting in a lot of bias by telling it how it should titrate exploration as things go on. You're building this very explicit tree search for it. I don't know if you share that intuition where the more you understand it, the less impressive the accomplishment in 2017 seems.

I personally disagree. I think they're profound for different reasons. I don't understand the LLM RL enough to comment on it on your podcast. But why is AlphaGo a profound accomplishment? It's worth stepping back a little bit. It is different from modern RL, and we can talk a little bit about some of the algorithmic choices there.

I think the most profound thing here is that a 10-layer neural network pass, basically 10 steps of reasoning… Of course, the reasoning is not just one trail of thought. It could be distributed representations and a lot of thoughts going on at the same time. But by construction, let's say a 10-layer neural network can only do 10 sequential

steps of thinking. 10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem. This was a breakthrough that I think most people don't even fully comprehend today, how profound that accomplishment is.

This is what also girds AlphaFold, for example, where you have a very, very difficult physical simulation process where you would need to roll out so many microscale simulations, and yet 10 steps of a somewhat small neural network can somehow capture what feels like an NP-class problem into a single problem.

It actually makes me wonder if our understanding of problems like P=NP, or these fundamental computational hardness problems, is incomplete. Obviously, it's not a proof of P=NP, but there's something to it that is very disturbing, where what felt like a very hard problem can fall to a very simple macroscopic solution.

That is a very interesting insight, that a lot of problems which are proven to be NP-hard—I don't know if Go is proven to be NP-hard, but protein folding, et cetera—neural networks can solve.

They're NP-hard in the worst case, but we're usually not concerned about the worst case. These problems usually have a lot of structure to them. I think the question we should be asking ourselves is about how we've been formulating solutions to NP-hard problems in worst-case complexity. I wouldn't say this solves Go. It doesn't give us an exact solution of the optimum, but in practice, it is extremely useful. The same thing has been shown in AlphaTensor and AlphaFold. Yes, there is a very hard problem that, in the worst case, seems intractable, and yet we're able to make almost arbitrary amounts of progress.

In the limit, what might this look like? If you want to simulate something very complex like weather, or predict the future—do we live in a simulation or not—the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute.

I feel totally not qualified on the computational complexity or the math to comment on this, but I wonder if there's an important role of chaos here. What is the problem with weather, and why does it take 10x the amount of resources to predict weather a day out, and continually so for every additional day out? It's because it's a chaotic system, so small perturbations can totally change the final estimate as time goes on. I guess you would expect that for Go and protein folding as well.

Here's an analogy to weather that might be relevant in Go. Here's our current board state. Given what we know about both players, what is the exact board state in the future? This is extremely sensitive to initial conditions. A single stone placed here can disrupt the entire prediction. This is hard. Intuitively, this is the chaotic problem. Yet somehow, we can predict who's going to win. This captures a lot of possibilities.

There's a more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macrostructure over a lot of possible futures. In weather, it could be the same thing. We don't exactly care what the velocity of wind 6,000 feet above a specific latitude and longitude is. We care where the hurricane is, and things like that.

In chaos, there's the classic Lorenz attractor. If you start anywhere on the Lorenz attractor, you don't know where you're going to end up, but you do know that the thing looks like this. There's a kind of beauty here. Sometimes we don't necessarily care about the microscale things. We actually care about the macroscopic structure, and these things can be predictable.

Contrast that to something like a hash function, which is also incredibly dependent on initial conditions but doesn't have a macrostructure, or at least hopefully doesn't, if the algorithms work.

One would hope. There's no equivalent of a value function or broadly how the weather is going to be that's interesting there. It's really just about what the board is going to look like exactly 100 moves from now exactly.

Intuitively, that seems correct. This is also out of my area of expertise, but I find it interesting that the tools of cryptography and hashing have also not been able to prove that you cannot come up with fast approximations. You cannot come up with fast approximations.

If they were able to do that, then you could prove P is not equal to NP. In fact, we know that there's structure in many cryptographic protocols, obviously RSA cryptography. There is structure, and that structure is what quantum computers exploit to break them.

Reiner has a very interesting blog post, which we've talked about in the episode, where he points out that if you look at a high level at what cryptographic protocols look like and what neural networks look like, they're extremely similar. You have sequential layers of jumbling information together. There's a convergent evolution in the algorithms. In cryptography, you want the final state to be incredibly sensitive to initial conditions, so that it comes out looking jumbled if you change anything. In neural networks, you similarly want everything to be dependent on all the information, because you want to process all the information and consider how it relates to itself.

You have the maximum power of a neural network at the edge of chaos. I think there are some research papers from Jascha Sohl-Dickstein on this. There's something quite fundamental about chaos that's not just hopeless noise. There's something useful in chaotic systems, at least at that boundary. But this is just thinking about it as a philosophy. I don't know the math well enough to comment on it.

We'll talk about LLM RL in a little bit, because there's some connection there, but let's go back to MCTS. What is it doing? Crucially, it is not saying that we're going to increase the probability of winning directly. It's not going to upweight all actions that won and downweight all actions that didn't win. Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we're going to make every action that we took better by having the policy network predict that outcome instead.

This is a very nice idea because you have one supervision target for every single action. The variance of your learning signal is very low compared to the alternative naive RL thing.

Let's consider a very naive algorithm that looks a lot more like modern LLM RL today, where we do something like take the winner of a self-play game and encourage it to do more of that.

It's worth thinking a little bit about what alternatives we could use to train self-play agents instead of MCTS. We use a lot of LLM-style RL these days. Is that relevant? Could we do that instead? Suppose we have a very naive algorithm where we take a league of agents of different checkpoints and play them against each other. For the games where a single player wins, we reinforce those actions up and retrain the policy network to imitate those winners, instead of the MCTS objective.

Let's say you have a chain of actions that led to a win, and you have a matchup between two agents that are basically the same. Assume policy a (πₐ) and policy b (π_b) are evenly matched, so their true win rate is 50%. Let's say you play 100 games, and each game lasts 300 moves.

You're doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don't, and you just play them against each other to see that occasionally one might have a better strategy than the other.

Let's say policy a wins 51 games and policy b wins 49. This is just due to random luck, or maybe you perturbed policy a in some way that let it do this. Just to have a very simple model, let's pretend that for 50 of the games, they played exactly equally. In the one game where a won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or random noise, it happened to make a smarter move than it did previously.

So you have one true supervision signal for your policy network, and then you have 99 games times 300 moves for which imitating those actions gives you exactly the same policy you had before. The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions where you want… Actually, let me clarify a little bit.

We're talking about how the good move, the out-of-distribution move, is a small fraction of all the moves that are played across all the games on which you'd want to train.

This, of course, reminds me of how LLMs are trained with policy gradient methods. When Karpathy was on the podcast, he called it "sucking supervision through a straw". It's interesting that this thing you're saying—which would be intractable and prevents you from actually getting beyond a certain level in Go—is just by default how LLMs are trained?

Right. This is not to say it doesn't work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision samples, so long as you find a way to mask out the supervision from these guys. This is where things start to get pretty related to RL in terms of advantage and baselines.

Let's look at the gradient variance of a very naive approach—I'm going to call it gradient RL—that's basically the sum of rewards. The sum of rewards is the return. In our naive setup here, we only have an indicator variable for the return, where either you won or lost. In the case where you lost, your gradient is zero, so you don't train on those examples, and when you won, you try to predict those things. You can think about this setup as a special case of this general formula here.

The trouble here is that this is very high variance. When you multiply these terms out to compute the variance of the gradient, it's equal to the expectation of… Just for simplicity, we can pretend this is on average zero or something if you're centering it at no signal. The variance here basically means that you're taking the square of this product term. You end up with a term that grows quadratically with T. When you have a setup like this, this thing acts as a coupling effect on top of these terms here.

Let's actually map this to an LLM case, and we can answer why LLMs only do one-step RL instead of a multi-step RL scenario. In LLMs, you have a decoder that might predict some words like "hello world". In current LLM RL, they treat this entire sequence as a single action, aₜ, and big T is just one.

It's true that because of how transformers are formulated through the product of conditional probabilities, the log probability of the whole sequence is equal to the sum of the probabilities of individual tokens. In this case, I would say something like log(hel) + log(lo) + log(world). This is true, and if this term were one, then they would be the same thing. However, when you're sampling, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms. The problem becomes, how do you ascribe the credit associated with every episode to all these different terms?

The thing I'm confused on is what it would even look like to do it that way in LLMs, because you only get a reward at the end of the episode.

You could imagine a reward that says, "I'm going to give you some process supervision where you get a reward for each of these actions on every step."

Okay, so you're saying if instead of doing it that way… Well, the way you've written it, it would be a sum at the end anyway, so they wouldn't have to be multiplied. But you're saying instead of doing it that way, you would just add up these process rewards at the end and then treat that as one single reward signal?

Correct, for one single log prob action.

But isn't that how it's written to begin with anyways? The sum of the rewards?

The thing that's a little bit hidden here in the math is that we're assuming that when you decompose the problem into a multi-step problem, you're now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance of this one. In the case where you don't separate it out, if you just have T=1, you have a single estimate of log prob and a single estimate of reward. This term still shows up. In LLMs, the naive REINFORCE estimator looks a bit like the return of the single action… It looks kind of like this. This is the basic form here, but it's still a contributor to variance. You want to make sure that, similar to how in this case we were training on a lot of neutral labels, you're sort of penalizing the labels that don't help and only rewarding the ones that actually make you better.

Intuitively, the analogy is: can we find a term in our training objective such that it's actually discouraged from doing this, or these don't have any effect on the gradient, and this has an effect on the gradient?

If you applied that there, the only thing you could do is eliminate 49 of the games. At least the way you have it written there, it would be 51 times…

Actually, the optimal case is to discard all of these moves, and only get a gradient on that single move that you got better on.

How would you do that? This is a pretty tricky problem in practice. This is where advantage estimation happens in reinforcement learning. You want to subtract a term from your multiplier. Instead of an indicator function of one and zero, you want something that behaves like a zero for all of these guys, and then a one for these ones. You could do that if you can say, "Hey, I won this game, so this is slightly above baseline performance."

Well, you won a lot of games. But you don't know which ones let you win because they were truly better versus winning by accident. How would you design a baseline where it's truly better?

This is where in RL people use things like TD learning to better approximate the quality function, the Q that we mentioned earlier. You can try to subtract that from your return. Ideally, what you really want to do in RL is push up the actions that make you better than average and push down the actions that make you worse than average. They call this advantage. There are multiple ways to compute it. I highly recommend John Schulman's "Generalized Advantage Estimation" paper as a good treatment on how to think about various ways to compute it. At the end of the day, you want to reduce variance by trying to make this smaller, so it doesn't magnify the variance of this one.

That makes sense. This requires you to have a very good estimate of what average performance from a state would look like. This gets us back to the value function thing we were talking about earlier.

Keep in mind that this model-free RL setting is trying to solve a credit assignment problem where you don't know which actions were actually good and which ones were bad. Monte Carlo tree search is doing something very fundamentally different. It's not trying to do credit assignment on wins. It's trying to improve the label for any given action you took.

We can actually think about a completely different algorithm called neural fictitious self-play, which was used to great effect in systems like AlphaStar and OpenAI's Dota. Let me talk a little bit about how you can unify some of these RL ideas in the model-free setting as well as the self-play setting.

What happens if you don't have the ability to easily search a tree? In Go, it's a perfectly observable game.

You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft where you don't have complete control over the binary, it's a little bit hard to do this, and I'm not even sure if it's a deterministic game. That makes this difficult from a data structures perspective.

What is done instead is that the basic idea of supervising your actions with a better teacher is still there. We're going to talk a little bit about how neural fictitious self-play works. It's the same idea. We're going to come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we perform search. Assuming we have a good value function, the search will give us a better result than our initial guess. In a game where you can't easily simulate a search process, what they do instead is train what's known as a best response policy.

You fix your opponent. Let's say you're currently training πa against a strong opponent, πb. In StarCraft, maybe these are the Zergs and you're playing Protoss or something.

You fix your opponent, and you treat this as a classic model-free RL algorithm, where your goal is just to beat this guy. Here you use your standard TD-learning-style tricks, or use PPO or any model-free RL algorithm to try to hill climb to winning against this player. You have a reward function where the return is one if it wins against πb, and zero otherwise. This is no longer a self-play problem. It's just a fixed opponent, and you're trying to maximize a score against them. You have a fixed environment where all you care about is beating this guy.

Once you have a good policy that you train with your favorite model-free RL algorithm—PPO or SAC or V-MPO—you now have a good policy that gives you a good label for what this one should do when playing against that player. When you train multiple best response policies, you can basically distill the RL algorithms into the labels for a given opponent.

You might have a best response policy against πb, and then maybe you have a league of opponents like πb, πc, πd. You're going to take the best response policy that you train against each of these fixed opponents, and supervise them with the label that this one would provide. This is almost like a proxy for your MCTS teacher. Instead of an MCTS teacher, you use a model-free RL algorithm to find the best search action that you could do to beat your opponent. Then you're distilling the policy here into what's known as a mixed strategy, where it's trying to average across all possible opponents you could play against. This is what gives you something that can do no worse than an averagely selected opponent from the league.

This gets around the problem of having to derive a teaching signal from MCTS, but fundamentally it's still about relabeling your states with better actions so that they improve your policy.

Just to make sure I understand, if you win a game against this other policy, you reinforce all the actions on that trajectory. Yes. Here you can use a number of algorithms like PPO, V-MPO, or even Q-learning if you want. The specific algorithm here is usually a model-free thing because you don't have search, but there's an interesting connection between MCTS and Q-learning that I want to bring up. In MCTS, you have a tree, and through the resolution of your value function at the approximate leaves of the tree, you can back up through many sequences and obtain some sort of mean value estimate. Your Q is derived from the average of a bunch of simulations.

In model-free algorithms, there's often a component of estimating a Q value. Q values are often learned through TD learning, although in PPO, the way they do advantage estimation is not necessarily through a Bellman backup.

In Q-learning, there's this very cool trick where Q(s,a) is backed up as r plus some discount factor times the max over a, Q of your next step. Intuitively, if you have an MDP and this is terminal, it's saying that the best action you can take at this state is equal to the reward you get for taking this action, plus the best that you can do at the next state.

There's a recursive, dynamic-programming property of MDPs, and you can train neural networks to basically enforce this consistency. You can say, "Well, once I know the Q value of this action, I can then use that to compute something about the Q value for the next."

Earlier I was like, "Hey, why are we training policy? Why don't we just train the value alone?" That is what this is. This is an algorithm for recovering value estimates of intermediate steps when you don't have the ability to do forward search. You must collect a trajectory first of n steps before you're able to do this trick. But the intuition is kind of the same.

Knowing something about the Q value here can tell you something about the Q value here. And indeed, you can recover a policy from a Q value. You don't need to explicitly model the policy distribution. You can recover the policy distribution by doing argmax over your Q values.

Q-learning, or approximate dynamic programming, propagates what you know about the future Qs backward like this. You can see that there's a similar structure going on here, where in one case you're planning over trajectories your agent hasn't been to yet, whereas in the other you're planning over trajectories your agent has visited.

Importantly, why was Q-learning a big deal? It's because historically we just haven't had the ability to do search on fairly high-dimensional problems like robotics or whatever. For a long time, we made the assumption that if we can't model the dynamics with a world model, we're going to instead just collect trajectories and then plan with respect to the only number that really matters, which is reward.

This is very interesting. To unify this with our discussion of LLMs, you don't have Q values, but you're doing this backwards learning where you find the trajectories that pass some unit test in some coding environment and then reinforce those trajectories. There's a huge difference between that and this forward approach with MCTS. The reason it's much more preferable to do MCTS is because you can do it per move and make each move better, rather than having to learn per trajectory and hope, as Karpathy said, to learn this…

Through a straw. Right, you get supervision through a straw. You basically just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right. The reason you can do this much more sample-efficient, favorable thing with Go is that because MCTS works in Go, you basically know that if you just do the search locally here—and this search is truncated at the end by a value function that works even if you haven't unfolded your whole trajectory—you can just say, "This is my new policy," and improve in a more iterative, local way, rather than having to unfold all these trajectories. There was some research from Google in 2023 or 2024 where they tried to apply tree structures to reasoning.

The jury is still out as to whether this can ever work. We probably will see a revisiting of this idea of forward search in the future.

But there are two things that make MCTS very simple for Go. Value estimation is concrete. You can determine it for real, and then you can use it to truncate depth, as you said. The breadth is also determined. What's critical is that the action selection algorithm, where you iteratively visit and grow the tree, is well suited for the size and depth of problem that Go is.

But for something like LLM reasoning, PUCT might not be a good enough heuristic. It might be too greedy with local tokens, and it might only give you obvious thoughts that are correct but don't really solve your final problem. I would say the jury is probably still out on what the final instantiation of reasoning for LLMs will look like. I wouldn't rule out that this stuff could come back, but it's a bit hard.

Don't LLMs natively learn to do MCTS, where they'll try an approach and be like, "Oh, that doesn't work. Let's back up. Let's try this other thing," and then go in the direction that proves to be more fruitful? Certainly, LLMs manage to do something that looks like real human reasoning without having to do an explicit tree structure.

That being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even if not in exactly the same instantiation as AlphaGo.

Just to make sure I understand the crux of it: the breadth comes from the number of legal actions being wider, and the depth comes from not being able to train a value function as easily, because… Here's an example where LLMs break down.

The CPUCT rule involves the √N over 1+Nₐ. In an LLM, you're most likely never going to sample the same child more than once. If you have multiple steps of thinking, because language is so broad and open-ended, a discrete set of actions is not really an appropriate choice for an LLM. Even though they're discrete tokens, it's just such a large number that this type of exploration heuristic is probably not the right thing to guide how to search down a tree.

I guess the crux comes down to the fact that in Go, you know that MCTS is almost certainly better than your current policy, even though you haven't explored the end of any trajectory. In normal reasoning for LLMs or robotics, there's no way to just locally evaluate and improve your next move in a way that's independent of actually solving the problem.

"No way" is a strong word. Lots of people have thought about how to apply MCTS or its successors like MuZero to continuous control spaces, and I'm sure very cool research work is still ongoing to try to crack that problem.

But yes, the apparent challenge right now is that most problems in much higher-dimensional action spaces, or something combinatorially much bigger like language, don't seem as amenable to the discrete action-selection heuristics and game-evaluation type stuff that Go has.

That's not to say the idea of thinking into the future along multiple parallel tracks might not give you some information about which way to search. If you think about mathematics, it often occupies more of a logical search procedure where you can back up and see which paths seem good or not. There's more of a rigid structure there, whereas in a business negotiation it's less of a tree and maybe something a bit different.

We're now seated, so I can ask you some more questions about AlphaGo and AI research more generally. In 2021, Andy Jones had a paper called "Scaling Scaling Laws with Board Games", and he basically anticipated inference compute, or inference scaling, by showing that you can trade off test-time compute and training compute. That is to say, you can spend more compute searching through MCTS, and if you do that, you can get the equivalent performance to having spent more time training the model.

If you see this pattern, you might think that with LLMs you might do something similar in the future, and in fact that's what ended up happening. What is a fun exploration one could do now to explore other axes of scaling in toy settings, which will be important to understanding what AI development might be like in a few years? Test-time scaling and reasoning, and how they interact with model size, are quite profound when it comes to how much needs to actually be done as explicit search versus how much can be packed into the forward pass of a neural network.

How does a forward pass of a neural network learn to do something that should be sequential and recursive? That's quite interesting. The Andy Jones "Scaling Scaling Laws with Board Games" paper is quite cool. There was another really nice result from that paper. Not only can you predict scaling laws of the LLM variety—where as you increase parameters you can decrease the amount of compute for search or vice versa—but he also showed that you can predict how much compute is needed to solve a larger version of the board game.

For example with Go, which can scale from 3x3 to an infinitely sized Go board, you might be able to revisit this question and try to reproduce whether this shows up. I actually started this project with the motivation of asking: does the "Bitter Lesson", or our knowledge of scaling laws, allow us to execute a lot better on a compute-optimal Go bot? Can we build a strong Go bot without all of the KataGo tricks, just by really focusing on the Bitter Lesson and scaling laws?

I have not been successful so far, but usually when you want scaling laws to work, you want to be in the regime where the recipe already works and the data sets are good, rather than trying to figure out how to do scaling while also trying to figure out what the right data sets are.

The scientific understanding component in research often follows a step where you get something to work first. Then you use that system to collect data that then helps you build a mental model of how things work, such as scaling laws. Usually, if you want to build a strong Go bot using scaling laws, you actually have to make a strong Go bot first and then use the scaling laws to extrapolate a bit further into the future.

Say more. Just so I understand, are you saying scaling laws did not work? There was no scaling-laws pattern that you could see in your Go bot?

A mistake I made initially, when I had some bugs around how MCTS labeling was working, was that I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert data sets. You can indeed plot things that look like this, but if you're in a regime where your policy is not working well, you might just be studying scaling laws on bad data.

One important implementation detail is that if you want to study a scaling-laws problem, you have to have a problem for which the data is good, the architecture is good, and there are no bugs, and then you solve it from there.

Ex ante, I wasn't able to apply scaling laws to direct what to look at until I had the rest of the system working. This sounds obvious to researchers. Of course you want a working, bug-free system before you study scaling. But as advice for practitioners, on where I tripped up when I started this project, you don't necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.

Speaking of compute, you can look at these charts of compute used to train the best AI model in the world over time going back 10 years. It's a very smooth line in log space that grows exponentially year over year. Except there's this huge aberration, and that aberration is AlphaGo Zero, which was trained on way more compute than any other AI model at the time. It was 3E23 FLOPS. It's comparable to a frontier LLM, orders of magnitude off, but still.

The question is, especially with you being able to get something off the ground, did you train it on your own? I got a donation from Prime Intellect for about $10K. I spent maybe the first $4K doing exploratory research, and then about $3K on the final run. Some of it remained for serving the model.

Is your sense that they did a bad job training it, if you can do it in $10K now?

The compute required to be the first to do something is always much larger than the compute it takes to catch up. It's the same story playing out in LLMs. Once someone else has done it, you can use tricks like distillation. You can use all sorts of crutches to bootstrap your way to success.

With my own bot that I've hosted online, I actually used best-response training against the KataGo models to get a strong level of performance.

As of the time of recording, I'm validating whether I can do that first step, which is to do the tabula rasa play. Importantly for research, you often want to start from a good init. The simple thing I did first was train

best-response agents against KataGo. The AlphaZero team did not have any policy they could train against, because they were trying to do everything tabula rasa. Being the first to do it means you're prioritizing getting the thing working rather than the most compute-efficient possible implementation.

This plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatter plot is all over the place, and there isn't a very clean line the way there is for frontier LLMs. That's because the folks training these models often are not at the scale where every FLOP counts and they need to squeeze out the performance of every single FLOP as the deciding factor in pre-training. Instead, their focus is more, "We want a certain capability to show up, so we optimize the training setup to make it easy to derive that capability." Once you have that capability… Invariably if you scale up the compute, you are forced to make it compute-efficient, because this is hundreds of millions of dollars we're talking about. But in the past, when compute for experiments was more plentiful—or not accounted for in a way the researcher was really responsible for—you end up with people optimizing for things besides being on the compute-optimal Pareto frontier.

I see, like speed or something?

Yeah, time to result, or just getting it to work. The first AlphaGo probably had lots of compute, and they didn't need to worry too much about making it the most compute-optimal thing.

How much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017 but didn't?

Great question. Going into this project, I knew in the back of my mind that things always get easier to do over time. I wanted to see where Go was at, given that there hadn't seemed to be any major open-source strong bot after KataGo in 2020. Reading the KataGo paper, there were a lot of clever ideas. I was thinking, "Let’s see if the Bitter Lesson had happened, where a lot of these tricks just go away because Nvidia made faster GPUs. Roughly, where are we on that?"

Again, this is not a peer-reviewed claim. It's just my preliminary vibe guess based on what I’ve seen with my own experiments. It seems like architecture choices don't matter that much. Transformer versus ResNet… We're at the speed of GPU where the size of the model is not so big that this really matters.

You can simplify the setup quite a lot. Instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can do a dumb synchronous thing where you collect, train a supervised learning model, and then collect again. There are opportunities to simplify infrastructure. Nvidia GPUs have indeed gotten faster. Whereas KataGo was trained on V100s, you can train on half the number of desktop Blackwell GPUs and it still works.

Some of the auxiliary supervision objectives that KataGo developed aren't really necessary if you have a strong initialization. If you're initializing best-response training against KataGo itself, your own model needs none of the tricks that KataGo needs. The core thing is getting as quickly as possible to some strong opponents. That matters a lot more than the specific architectural innovations.

But there are still some nice compute multipliers. I found that training on 9x9 boards was very nice for resolving endgame value functions. If you can co-train that on an architecture that can transfer between 9x9 and 19x19, you can really cut down the warm-start time to learn from scratch. AlphaGo Zero's plot showed the first 30 hours or so spent basically catching up to the supervised learning baseline. You can cut down that time a lot by pre-training on a small board and then warm-starting that into your 19x19 board play.

There was some other stuff, like varying the number of sims between episodes. This turns out to be not that sensitive. You can fix it or increase it. It doesn't matter too much. Anyway, it's nice from a scientific perspective, revisiting an old paper and seeing what really matters.

This is a tangential question, but why is it okay to have a buffer in AlphaGo? Every time I talk to an AI researcher, they're telling me about how bad it is to be off-policy. But the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step, or in a batch of backward steps, would not be among the ones made by the most recently trained model. So why is that okay?

Great question. This gets into the fundamental off-policy versus on-policy reinforcement learning questions. As you'll recall, in MCTS, you take actions that you took, and you relabel them to take different actions on the same states.

The off-policy part comes in where, what if you're relabeling states that your new policy would never visit? What's the point? You're wasting capacity. In the extreme limit, imagine the distribution of states in your training buffer are all states you would never visit. Then you're supervising them to take good actions on states you would never achieve. Therefore, your policy can get really bad. This is where off-policy can really hurt AlphaGo.

However, if you interpret this from the DAgger perspective—which is basically a way to correct yourself back to the optimal trajectory given some data—what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories. Any of those states are given a supervision target to funnel you back into your optimal trajectory.

Maybe I can just draw real quickly here. In a DAgger-style setup, your optimal training data distribution is your optimal states and actions. You want to be in this state, you want to be in this state, you want to be in this state, and then you win here. These are your optimal policy actions. These are the things you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can funnel yourself back in.

But why isn't this a fully general argument for off-policy training?

This is actually why you want to do off-policy training sometimes. You don't want a compounding error where, if you make a mistake, you don't have the data for how to return to your optimal distribution. Optimal control doesn't really say too much about how to not accidentally get here, because it's making the assumption that once you learn the policy, you're going to get to here. But in applications like robotics, a gust of wind blows you slightly off and now you need to correct. Or the friction on one of your tires is slightly lower than the other wheel, and now your car is drifting and you have to correct it. These kinds of things often happen in more real environments.

There was a funny quote about chess and Go. The problem with Go and chess is that the other player is always trying to do some shit. Things can drift off, and you always want to be able to correct back to your winning condition.

Your replay buffer really should have the states your policy would visit, plus some distribution of states you might drift to, and then how to return back to your optimal states.

Now, if you take this to the extreme and you say, "We don't have any of this data, and we're just going to be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here…" Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. You're training your model on states you would never reach. This is where off-policy can really hurt.

As part of this project, I did try an experiment where I took a bunch of trajectories, and to try to saturate the GPU as much as possible, I took random states from the dataset and reran MCTS on just those states. Instead of playing a whole game where I'm doing MCTS on every move, I ignore the causality of moves and pick random board states, and I label those with my current network. I might revisit old states I've labeled before and relabel them again with my current network.

In practice, this actually does work. You can take some states that are reasonable and constantly be relabeling them while training.

This starts to converge on a very robotics-like setup, which is very common. You have your dataset of trajectories, and then you have something like a replay buffer pusher. These are off-policy offline trajectories. Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have a job that's continuously replanning what the best action you should have done instead of the action you took. In robotics, it's very common to minimize TD error. Your Bellman updater is constantly pulling things from here and trying to satisfy Q(s,a). From here you have your trainer, which is trying to fit the Q to the Qᵗᵃʳᵍᵉᵗ.

You can think about this as a sort of planner. You visit old states you've been to, and you take your current model and rethink. What could I have done better if I visited this? This is how off-policy robotic learning systems are usually trained. These days there's a simpler recipe, but in the Google QT-Opt days, we did things like this.

So what is the trainer?

The trainer is where you try to minimize Q(s,a) and Qᵗᵃʳᵍᵉᵗ.

Can you explain the whole setup again at a high level?

You have your off-policy data that came from various policies. You're constantly pushing transitions you saw before to a replay buffer. Then you've got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at s to have a better value? The way you enforce that is by trying to minimize the TD error. Given this, you have s’. You compute Q(s’,a) and you find the action that should go with s’ that makes this Q value as high as possible. Then you add that to the reward here, and that gives you your actual target. For this current s and a, your Qᵗᵃʳᵍᵉᵗ is this. Now you send back the Qᵗᵃʳᵍᵉᵗ to this transition. With this tuple, you pair a Qᵗᵃʳᵍᵉᵗ. Then on the trainer side, you simply use supervised learning and minimize your current network's Q(s, a) with its target.

Got it. So in the background you're basically thinking through how valuable all these actions actually were.

Yeah. In a more optimal policy where you're trying to maximize this, what is the Qᵗᵃʳᵍᵉᵗ of this transition?

It's sort of like daydreaming.

Exactly. You can think about it like you're going back in hindsight. Given what I've seen in the historical buffer, was there a better action I could have taken?

The connection to Go here—I tried and it was moderately successful, but too complex to open source—was replacing this with an MCTS relabeler. Instead of doing this target network computation, you run MCTS on your transition. In this case, you have your state, your action, and whether you won or not at the game. Actually, you can just toss those two. You don’t care about these ones. You just take your state and plan MCTS to get your best policy, Π, on your current network. Not the network that took this action, but your current best policy network. You rerun your search offline on these transitions.

If these are transitions your policy can get to, this actually acts as a very nice stabilizing effect. Another benefit is that you can fully saturate your GPU better because you're not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel. Here the trainer would just predict the MCTS label as best as possible.

This kind of works, and it is quite relevant in robotics where you have a lot of offline data and you can't simulate things like MCTS. But in practice, it does run into the problem that if the current model is looking at states it would never reach, it's wasting capacity. You have to be a little bit careful here.

Much of RL has converged to a much more on-policy setup where they don't really try to directly train on off-policy data. At best, they use off-policy data as a way to reduce variance, but not to directly influence the objective.

Why have they converged to that?

It's just more stable. You might use the off-policy Q as a way to do advantage computation, like Q minus the sum of Q. If there are N actions, this is your value, and these are your current Q values. Your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here. If there's a problem in these dynamics, it doesn't blow up your loss as much. In robotics, there's a convergence toward using off-policy data to shape your rewards but not actually be directly here.

I'm reminded of our earlier conversation about why MCTS is so favorable compared to the REINFORCE or policy gradient thing LLMs do. This might be totally wrong, but I wrote a blog post a few months ago about how policy gradient RL is even more inefficient than you might think. The inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer—as an agent has to do two days' worth of work to figure out if it did a project correctly, rather than just completing the next word in a sentence—the amount of information per FLOP decreases. You have to unroll two days' worth of thinking to see if you implemented a feature correctly, so the amount of samples per FLOP has been decreasing.

You're trying to maximize as you’re learning, bits per FLOP. You can think of bits per FLOP as samples per FLOP times bits per sample.

What I mentioned a second ago is that the samples per FLOP go down as RL becomes more long-horizon. But this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning.

Early on in training, let's say you have a vocabulary size for an LLM that is 100K long, so there are 100K possible tokens that one could answer. You have a totally untrained model and a prompt like, "The sky is…" With supervised learning, the model would have some probability distribution over all the things it could say. There's a label that says the term here is "blue," and it would learn through cross-entropy loss exactly how far its distribution is from correctly saying "blue."

If you were doing this through RL, the model would try, "The sky is halcyon." Nope, that's wrong. "The sky is told." Nope, that's wrong. This is a totally untrained model. You would have to do this on the order of 100K times just to stumble on "blue" and get some learning signal.

In the supervised learning regime, you have your distribution of probabilities, you get told that it's "blue," and you figure out how far off you were. The amount you learn is a function of your pass rate. The further away you are from "blue," the more you learn to go toward "blue" using cross-entropy loss. You can think of it as like your pass rate, your prior probability of having said "blue." As a function of that, in supervised learning, through cross-entropy loss, you would learn negative log(p), p being pass rate, bits, once you get this label. Whereas in RL, if you're just randomly guessing and seeing if it works, that's basically just the entropy of a binary random variable.

What's also tough here is that the distribution you're sampling under is your policy's distribution. If your policy has no chance of sampling "blue," then you will never get a signal.

Exactly. That's modeled by the fact that your probability of sampling "blue" is extremely low. If you sample it, you do learn as much as you would have in supervised learning. In all other cases—99.99% of the time in an untrained model—you're learning incredibly little from seeing that "halcyon" or "told" is not the correct word. That's what happens most of the time. You

learn very little. If you put your pass rate on the X-axis and the bits you're learning from a sample on the Y-axis, with 0%, 50%, and 100%, so at the end of training you're here. If you have supervised learning, the negative log pass rate would look something like this. The entropy of a binary random variable would look like this, depending on whether you're doing nats or bits. If you do bits it's one here at the peak. This is like a coin flip. You learn the most from a coin flip. This is supervised learning. This is RL. However, the problem is you spend most of training in this regime, in the low pass rate regime. How fast you're learning is a function of how many bits per sample you're getting, and you're getting very little signal here.

If you chart the pass rate on a log scale—where at the beginning of training with a vocab size of 100K the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100—what this graph looks like here, supervised learning would look like this, and then RL would look like that. And arguably you spend all your time here, potentially never even getting a single success. It's a depressing plot in the sense that once you're here, it's not at all obvious how you get to there. Once you're here you have something, but you actually spend all the time here in many RL problems. There's a question of how you initialize so you're at least at a non-zero pass rate.

One more thing I'd like to add about bits per sample that's very relevant to any machine learning problem is that there's a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets the entropy of this distribution is far higher than the one-hot. There's way more information in bits per sample in a soft label. That's why distillation is so effective per sample. It's giving you way more information per sample.

I wonder what the equation would be, but obviously it's…

It would just be the entropy of this distribution. The entropy of this is zero. The entropy of this is the entropy equation. This is also why AlphaGo is quite beautiful. In AlphaGo, you don't train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Both of these are valid, and if you wanted to do a scientific experiment of how important this soft label is, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft target.

Earlier I was stumbling around. Intuitively, why is this ability to do iterative search—where you don't necessarily need to be able to win the game in the beginning, you just need to be able to improve your current policy—so powerful a capability in learning compared to how LLMs currently learn RL?

It's exactly this thing of considering your pass rate of the entire trajectory. I just don't know a formal way to think about this. Maybe you should help me out here. Why is AlphaGo an elegant RL algorithm?

The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate. This is what allows you to hill-climb this beautiful supervised learning signal. If you look at the actual implementation of AlphaGo, every step of the way, there's actually no TD error learning or dynamic programming, at least explicitly. It's just supervised learning on a value classification as well as a policy KL minimization.

It's just a supervised learning problem on improved labels. The training is very stable. You can train as big of a network as you want. You can retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. You don't need a complex distributed system to keep everything on policy. At the end of the day, you're just saying, "I have some improved labels. Let's retrain my supervised model on these targets." You're always in this beautiful regime where you're just trying to improve the policy, rather than escape local minima where every signal is flat all around you.

One way to draw the curve is if you draw the win rate of an MCTS policy versus the raw network… Let's say this dotted line is the raw network. The MCTS policy looks like this. Every step of the way, this supervision signal is very clean. You're never in a situation where the MCTS is giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts.

That's a great way to explain it. Maybe we sit down and I ask some questions about automated research.

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this automated LLM coding assistant loop. There's an idea that if you fully automated AI research, you could have some sort of singularity. Obviously we're not there yet, but to the extent that we have early indications of what this process might look like, I'm curious about your observations about what the AI is good at, what it's not good at, what you think about this scenario's likelihood eventually, and what thoughts you have about this in general.

I think automated scientific research is one of the most exciting skills that the frontier labs are developing right now. It's important for everyone who's doing any kind of research to get a good intuition of what it can do now and what it can't, and how the science process might work in the future once we have AIs automating a lot of this investigation. In brief, I mostly used Opus 4.6 and 4.7 while working on this.

What works is that the models can do a very good job of hyperparameter optimization. In the past, people would come up with a search space of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network. They would do a grid search or a Bayesian hyperparameter optimization approach, and it would find some tuned parameters.

The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, "I've identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let's try to find the best way to fit the constraints of the optimization problem." You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric. This can squeeze out quite a lot of performance. On a fixed data set with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like LLMs or Go.

It is also fantastic now at basically executing any experiment. I have a Claude Skill that I wrote called Experiment where I give it a description of what I want it to plot. I just describe, "Here's the x-axis I want, here's the y-axis. Answer this question for me." It'll run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it and so forth. That's what works quite well today, and I think we can expect these abilities to get better in the future. But it's also useful to know what it's not doing so well today.

In the blog version of this tutorial, I have a plot of all the experiments I did grouped in a tree, where every node represents a failed, successful, or mixed experimental result. From there, it branches off into a child representing the follow-on experiment. Occasionally, I'll rabbit-hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it's probably not worth it. So then I'll jump to a completely different track. I call these things rows. What I find is that the current closed models the public can access today don't seem to be that great at selecting what the next experiment should be in a given track. They don't seem to be able to step back and do the lateral thinking of, "Wait a minute, this track doesn't really make sense. Let's go back to first principles and think about what the bottleneck might be, or what we are trying to achieve."

Often I had to catch infra bugs myself by prompting the right question to Claude to investigate what's causing the discrepancy, and then it'll answer the question. With Mythos-class models or Mythos++ models coming online, maybe this just completely changes and these problems fall to improved scaling. But at the same time, I think there's a rich opportunity to develop RL environments that might incentivize this kind of lateral thinking.

One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it's very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily. The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work or not, and predicting the difference a particular modification to your training algorithm might make. I think there's a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics.

Or automating AI research.

Or automating AI research. Which is the real crux, the scary/incredible thing of making AIs make future versions of AIs.

You're suggesting the outer loop here could just be your win rate against KataGo?

That's one of them. I think there are a lot of deeper questions that one could tackle. For example, let's say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn't necessarily "achieve the best Go bot ever". The outcome might just be, "Can I predict what the win rate of my Go bot will be?" Or, "Can I predict the scaling-law plots that emerge from my idea?" But then you can verify that you haven't reward-hacked anything by using a very verifiable game like Go on the outer loop.

I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn't working as a result of a bug, or is it the result of the idea itself being wrong? Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He's able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that "this idea should work, so therefore there has to be a bug", versus the other way around. Why don't we start with that question? How locally verifiable are things which are good ideas?

As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don't have an answer to. Ultimately, to play a very strong Go bot, you probably did need to discover deep learning. Having a challenging game that cannot be cheated easily on the outer loop could be used as an outer-loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable—and this is where research taste really matters—you have to come up with ways to initialize your problem so that you don't try to solve a very intractable problem. Maybe you can leverage LLMs as a universal grammar in the middle to give you some sort of local feedback. The fact that LLMs are a universal grammar means that they can move at almost any level of the stack. They can think very locally, as well as step back and think in very broad steps. I think that's where a lot of the lateral thinking ability of humans comes from: knowing when the track you're pursuing or the objective you're pursuing is not right, and you should be asking a different question.

The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I've heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don't end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas. Having a single top-down vision of how things should work is very important. Having worked at different AI labs and also played around with parallel agents trying different ideas, what's your sense of how parallelizable AI innovation is?

Great question. I think the research taste for executing well on the Bitter Lesson is that you need to know how much the Bitter Lesson can buy you and how much is too much to ask for, at any given moment. Of course, in the fullness of time compute is the single most important determinant of how things work. It's almost inevitable that as you scale up energy and compute and parameters, intelligence will just fall out of that. That's super beautiful, super profound. No algorithmic detail really matters beyond that. But in the present day, we don't have infinite compute and parameters and arbitrarily good initialization, so we have to come up with heuristics that give us that. These heuristics are probably somewhat redundant. That's probably why you see this effect where a lot of these compute multipliers don't necessarily stack. They might have some correlated benefit. And then three years down the line, when the Nvidia GPUs have gotten even stronger, maybe they stack even less well. Maybe at any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the KataGo paper. There were many algorithmic ideas applied, and then you can see that with modern Blackwell GPUs and Ada-class GPUs—which are much better than the V100-grade GPUs that that paper used—some of these algorithmic tricks to speed up convergence just don't matter so much compared to something else. I think that's a matter of taste in the present time.

How about the outer loop? How verifiable for making AI smarter? With Go, you do have this outer loop of win rate against the best open-source model out there. Even there, as you were saying, there are other outer loops of whether you discovered a new phenomenon, which is actually very hard to…. If you didn't know scaling laws were important… When were Chinchilla or Kaplan scaling laws released?

2018.

So if you're back in 2015, there's not an automated procedure one can easily imagine for knowing which paper is the scaling laws paper versus which is just another random plot. Even in the Go case, it's a hard-to-verify outer loop, and the whole idea of an outer loop is to have some backstop on improvement, let alone for general AGI, where of course we have a bunch of these benchmarks. But there's a problem. We know the things we can measure, and we improve on the things we can measure. We care about this broader ability to do economically useful work, which is not super easy to measure, at least until you automate everything.

There's a question of how good the outer verification loop is for AI self-improvement, and does that matter?

I'm going to give a non-rigorous argument, but one that I intuitively believe. DeepMind started with a focus on games. They used games as their outer loop, and then their researchers learned from the experience of solving games, and now they're working on LLMs. Presumably, there was some positive transfer from

their time working on games and Atari and Go and StarCraft that now helps them make good LLMs. I assume that there's positive transfer in some regard, whether it's coding or general research ability or project management. All these things probably help them do well.

If that's the case, why wouldn't it also be true for automated AI researchers? They should be able to positively transfer experience tackling quick-to-verify, quick-to-iterate environments to something more ambitious and economically useful, like automating drug discovery.

I don't know. Isn't the issue… Historically, until Gemini 3 or whatever a couple years ago, people were saying, "Look, Google isn't catching up in LLMs because they're too tied to the old approach." There are gains, but there are also ways in which it actively hinders you. It's not obvious to me.

The jury's still out. Let's say currently Google's doing quite well. Who knows if the initialization of training on games is ultimately going to hobble their ability to be the winner in the long term. It's hard to say for sure.

Likewise, who knows if the seeming late start was really just them pre-training for longer on how to scale up TPUs. They invested all their tech tree in getting TPUs to be good, which seemed not that useful in the short term but in the long term it becomes… So it's even hard for humans to reason about what the optimal research strategy should be, even with the data we have today.

Okay, we should let people know how they can find out more about this project, whether to fork it themselves, or check out your blog post where you do an excellent job explaining many of these ideas. Where do people go next?

My website is evjang.com. There's a blog post that links to an interactive version of this tutorial. On my GitHub, which is the username ericjang, there's an AutoGo repo that people can fork and reproduce the training results.

I also highly recommend people check out this blog post, "As Rocks May Think." We touched on some of the ideas in this conversation, but it's this grander thesis of what happens when you have thinking as a primitive in—

Computer science.

Exactly. I highly recommend people check out that blog post as well.

I encourage the audience to think about the relationship between thinking and Go via MCTS and search, and how it relates to LLMs. I think there's something quite profound there, and probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs.

It's not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them. You can actually do a lot of research on Go, MCTS, and reasoning with very small budgets. So that's very exciting.

Awesome, Eric. Thanks for doing this.

Thank you. It's an honor to be on the podcast.