What Are We Scaling? Dwarkesh Patel on RL, Continual Learning, and Why AGI Isn't Imminent
Dwarkesh Patel
This video is a narrated essay from Dwarkesh Patel's blog. Its central question is a tension the author sees in current AI forecasting. Many people expect AGI within a few years, and they are also enthusiastic about scaling reinforcement learning on top of large language models. Patel argues these two positions don't fit together. If a humanlike learner were close, the labs' current way of building skills into models would be pointless. Patel expects transformative AI eventually, but thinks the missing ingredient is continual, on-the-job learning, and that solving it will take years rather than arrive as a sudden breakthrough.
The contradiction in pre-baking skills
Patel starts with how labs currently train their models. Through mid-training and reinforcement learning on verifiable outcomes, they try to build specific skills directly into the model. A whole supply chain of companies now builds RL environments that teach models to navigate a web browser or use Excel to build financial models.
Patel frames this as a dilemma. Either models will soon learn on the job in a self-directed way, which would make all this pre-baking pointless, or they won't, which means AGI is not imminent. Humans don't go through a special training phase where they rehearse every piece of software they might need at work. Patel quotes Beren Millidge, who wrote that when frontier models improve on benchmarks, we should consider not only increased scale and clever ML research, but also "the billions of dollars that are paid to PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities."
Robotics as the clearest case
Patel says the tension is most visible in robotics. In his view, robotics is fundamentally an algorithms problem rather than a hardware or data problem. A human can learn to teleoperate current hardware to do useful work with very little training. If we had a humanlike learner, robotics would be largely solved. Because we don't, companies have to go into a thousand different homes and practice a million times on picking up dishes or folding laundry.
The "automated researcher" counterargument
Some people who expect a takeoff within five years answer that the clunky RL work serves one goal: building a superhuman AI researcher. A million copies of this "automated Ilya" would then figure out robust, efficient learning from experience.
Patel compares this to the old joke, "we're losing money on every sale, but we'll make it up in volume." He finds it "super implausible" that an automated researcher would crack the algorithm for AGI, a problem humans have worked on for the better part of a century, while lacking the basic learning abilities children have.
He also argues that this story doesn't match what the labs are actually doing. You don't need to pre-bake a consultant's skill at making PowerPoint slides in order to automate Ilya. In Patel's reading, the labs' behavior points to a worldview in which models will keep generalizing poorly and keep failing to learn on the job. That is why the labs build economically useful skills in ahead of time.
The efficiency counterargument, and the macrophage example
A second counterargument is that even if models could learn skills on the job, it is more efficient to build them in once during training than to teach them again for every user and company. Patel agrees this makes sense for fluency with common tools like browsers and terminals. He also notes that one key advantage of AGIs will be their ability to share knowledge across copies. But he thinks people underrate how much company-specific and context-specific skill most jobs require, and he says there is currently no robust, efficient way for AIs to pick up those skills.
He illustrates this with a dinner conversation between an AI researcher and a biologist. The biologist had long AGI timelines. Asked why, she described part of her recent lab work: looking at slides and deciding whether a dot is actually a macrophage or just looks like one. The AI researcher replied that image classification is a textbook deep learning problem, exactly the kind of thing models can be trained to do.
Patel finds this exchange revealing because it shows the key disagreement between him and people who expect transformative economic impact within a few years. In his view, human workers are valuable precisely because nobody has to build custom training loops for every small part of their job. Building a pipeline to recognize macrophages as this particular lab prepares its slides is not net productive. Neither is building another loop for the next lab-specific microtask, and the one after that. What's needed is an AI that can learn from semantic feedback or self-directed experience and then generalize the way a human does.
Patel says every day involves a hundred tasks that require judgment, situational awareness, and skills and context learned on the job. These tasks differ across people and even from day to day for the same person. He concludes that you cannot automate even a single job by baking in a predefined set of skills, let alone all jobs.
He adds that people are underestimating how big a deal real AI will be, because they imagine more of the current regime. They are not picturing billions of humanlike intelligences on servers that can copy and merge everything they learn. He says he expects actual brain-like intelligences within the next decade or two.
Why "diffusion lag" is cope
Some argue that AI isn't yet widely deployed across firms, or providing much value outside coding, because technology takes a long time to diffuse. Patel calls this cope. He thinks the argument hides the fact that current models lack the capabilities needed for broad economic value.
If models really were like humans on a server, he argues, they would diffuse very quickly, because they would be much easier to onboard than a human employee. They could read a company's entire Slack and Drive within minutes and immediately absorb the skills of the company's other AI employees. He also points to adverse selection in hiring. The market for human workers is like a lemons market: it's hard to tell in advance who is good, and a bad hire is costly. Spinning up another instance of a vetted AI model avoids that problem. Since companies hire people all the time, Patel expects integrating real AI labor to be easier than hiring.
He then makes a revenue argument. If capabilities were at AGI level, people would spend trillions of dollars a year on model tokens. Knowledge workers worldwide collectively earn tens of trillions of dollars a year in wages. Patel says labs are orders of magnitude below that figure because the models are nowhere near as capable as human knowledge workers.
Why some goalpost-shifting is justified
Patel expects the objection that demanding tens of trillions in revenue is itself a moved goalpost. Until recently, people asked whether models could reason, had common sense, or were just pattern-matching. He agrees that AI bulls are often right to criticize bears for moving goalposts, and that it's easy to underestimate the past decade's progress.
Still, he argues that some goalpost-shifting is justified. If he had been shown Gemini 3 in 2020, he says he would have been certain it could automate half of knowledge work. We keep solving what we thought were the sufficient bottlenecks to AGI: models now have general understanding, few-shot learning, and reasoning, yet we still don't have AGI. He thinks the rational response is to conclude that intelligence and labor involve more than previously realized. By his earlier definition, we are close to AGI or have even surpassed it in many ways. But because model companies aren't earning the trillions AGI would imply, he concludes his earlier definition was too narrow.
He expects this to continue. His prediction is that by 2030, labs will have made significant progress on continual learning, which he calls his "hobby horse," and models will earn hundreds of billions of dollars a year. They still won't have automated all knowledge work, and he imagines himself saying then that more capabilities are still needed. He sums it up this way: models keep getting more impressive at the rate short-timelines people predict, but more useful at the rate long-timelines people predict.
Pre-training scaling vs. RL scaling
Next comes the question in the title: what exactly are we scaling? Pre-training produced an extremely clean, general trend of falling loss across multiple orders of magnitude of compute. Patel notes that this trend followed a power law, which is "as weak as exponential growth is strong." His complaint is that people borrow the prestige of pre-training scaling, which is almost as predictable as a physical law, to justify bullish predictions about reinforcement learning from verifiable reward. For RL, he says, there is no well-fit public trend.
When researchers do try to estimate RL scaling from scarce public data, Patel says the results are fairly bearish. He cites Toby Ord, who connected data points across OpenAI's o-series benchmarks and concluded that "we need something like a millionx scale up in total RL compute to give a boost similar to a single GPT level."
A broadly deployed intelligence explosion
Patel then turns to intelligence explosion scenarios. These include a software singularity, where models write the code for smarter successors, and a software-plus-hardware singularity, where AIs also improve their successors' computing hardware. He argues both neglect what he thinks will be the main driver of further improvement: continual learning. Humans become more capable at anything mostly through experience in that domain.
He describes a suggestion Beren Millidge made in conversation. In the future, continual-learning agents might go out and do different jobs, generate value, and bring their learnings back to a "hive mind" model that performs some kind of batch distillation across all the agents. The agents themselves could be fairly specialized, containing what Andrej Karpathy has called the "cognitive core" plus the knowledge and skills relevant to their deployment.
Continual learning will look like in-context learning did
Patel predicts that solving continual learning won't be a single, one-time achievement. He compares it to in-context learning. GPT-3 showed in 2020 that in-context learning could be very powerful, which is why its paper was titled "Language Models are Few-Shot Learners." But in-context learning wasn't solved when GPT-3 came out, and Patel says much progress is still needed, from comprehension to context length.
He expects a similar path for continual learning. Labs will probably release something next year that they call continual learning, and it will count as real progress. But human-level on-the-job learning may take another five to ten years to refine.
For this reason, Patel doesn't expect runaway gains for whichever lab first gets traction on continual learning. If a full solution appeared out of nowhere, it might be "game, set, match," as a recent podcast guest put it when he asked about this possibility. He doesn't think that's how it will happen. More likely, one lab will get initial traction, people using the feature will work out how it was implemented, and other labs will soon replicate and slightly improve it.
Why competition will likely stay fierce
Patel ends by explaining his prior that competition among model companies will stay intense. He notes that earlier supposed flywheels, such as user engagement on chat or synthetic data, have done little to reduce competition. Roughly every month, the big three model companies rotate places at the top, and other competitors aren't far behind. He says some force has so far neutralized any runaway advantage a single lab might gain. He offers three possible explanations without settling on one: talent poaching, the San Francisco rumor mill, or ordinary reverse engineering. If that force persists, he suggests, the first lab to make progress on continual learning is unlikely to pull permanently ahead.
I'm confused why some people have super short timelines yet at the same time are bullish on scaling up reinforcement learning atop LLMs. If we're actually close to a humanlike learner, then this whole approach of training on verifiable outcomes is doomed.
Currently the labs are trying to bake in a bunch of skills into these models through mid-training. There's an entire supply chain of companies that are building RL environments which teach the model how to navigate a web browser or use Excel to build financial models. Either these models will soon learn on the job in a self-directed way, which will make all this pre-baking pointless, or they won't, which means that AGI is not imminent. Humans don't have to go through this special training phase where they need to rehearse every single piece of software that they might ever need to use on the job.
Beren Millidge made an interesting point about this in a recent blog post he wrote. He writes, "When we see frontier models improving at various benchmarks, we should think not just about the increased scale and the clever ML research ideas, but the billions of dollars that are paid to PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities."
You can see this tension most vividly in robotics. In some fundamental sense, robotics is an algorithms problem, not a hardware or a data problem. With very little training, a human can learn how to teleoperate current hardware to do useful work. So if you actually had a humanlike learner, robotics would be in large part a solved problem. But the fact that we don't have such a learner makes it necessary to go out into a thousand different homes and practice a million times on how to pick up dishes or fold laundry.
One counterargument I've heard from the people who think we're going to have a takeoff within the next five years is that we have to do all this kludgy RL in service of building a superhuman AI researcher. And then the million copies of this automated Ilya can go figure out how to solve robust and efficient learning from experience. This just gives me the vibes of that old joke: we're losing money on every sale, but we'll make it up in volume. Somehow, this automated researcher is going to figure out the algorithm for AGI, which is a problem that humans have been banging their head against for the better half of a century, while not having the basic learning capabilities that children have. I find it super implausible.
Besides, even if that's what you believe, it doesn't describe how the labs are approaching reinforcement learning from verifiable reward. You don't need to pre-bake in a consultant's skill at crafting PowerPoint slides in order to automate Ilya. So clearly, the labs' actions hint at a worldview where these models will continue to fare poorly at generalization and on-the-job learning, thus making it necessary to build in the skills that we hope will be economically useful beforehand into these models.
Another counterargument you can make is that even if the model could learn these skills on the job, it is just so much more efficient to build in these skills once during training rather than again for each user and each company. And look, it makes a ton of sense to just bake in fluency with common tools like browsers and terminals. And indeed, one of the key advantages that AGIs will have is this greater capacity to share knowledge across copies. But people are really underrating how much company- and context-specific skills are required to do most jobs. And there just isn't currently a robust, efficient way for AIs to pick up these skills.
I was recently at a dinner with an AI researcher and a biologist. And it turned out the biologist had long timelines. And so we were asking about why she had these long timelines. And then she said one part of work recently in the lab has involved looking at slides and deciding if the dot in that slide is actually a macrophage or just looks like a macrophage. And the AI researcher, as you might anticipate, responded, "Look, image classification is a textbook deep learning problem. This is dead center in the kind of thing that we could train these models to do."
And I thought this is a very interesting exchange because it illustrated a key crux between me and the people who expect transformative economic impact within the next few years. Human workers are valuable precisely because we don't need to build in the schley training loops for every single small part of their job. It's not net productive to build a custom training pipeline to identify what macrophages look like given the specific way that this lab prepares slides, and then another training loop for the next lab-specific microtask, and so on. What you actually need is an AI that can learn from semantic feedback or from self-directed experience and then generalize the way a human does.
Every day you have to do a hundred things that require judgment, situational awareness, and skills and context that are learned on the job. These tasks differ not just across different people but even from one day to the next for the same person. It is not possible to automate even a single job by just baking in a predefined set of skills, let alone all the jobs.
In fact, I think people are really underestimating how big a deal actual AI will be because they are just imagining more of this current regime. They're not thinking about billions of humanlike intelligences on a server which can copy and merge all the learnings. And to be clear, I expect this, which is to say I expect actual brain-like intelligences within the next decade or two, which is pretty [ __ ] crazy.
Sometimes people will say that the reason that AIs aren't more widely deployed right now across firms and already providing lots of value outside of coding is that technology takes a long time to diffuse. And I think this is cope. I think people are using this cope to gloss over the fact that these models just lack the capabilities that are necessary for broad economic value.
If these models actually were like humans on a server, they'd diffuse incredibly quickly. In fact, they'd be so much easier to integrate and onboard than a normal human employee is. They could read your entire Slack and Drive within minutes. And they could immediately distill all the skills that your other AI employees have. Plus, the hiring market for humans is very much like a lemons market where it's hard to tell who the good people are beforehand. And then obviously hiring somebody who turns out to be bad is very costly. This is just not a dynamic that you would have to face or worry about if you're just spinning up another instance of a vetted AI model.
So for these reasons, I expect it's going to be much easier to diffuse AI labor into firms than it is to hire a person. And companies hire people all the time. If the capabilities were actually at AGI level, people would be willing to spend trillions of dollars a year buying tokens that these models produce. Knowledge workers across the world cumulatively earn tens of trillions of dollars a year in wages. And the reason that labs are orders of magnitude off this figure right now is that the models are nowhere near as capable as human knowledge workers.
You might be like, "Look, how can the standard have suddenly become labs have to earn tens of trillions of dollars of revenue a year? Until recently people were saying, can these models reason? Do these models have common sense? Are they just doing pattern recognition?" And obviously AI bulls are right to criticize AI bears for repeatedly moving these goalposts. And this is very often fair. It's easy to underestimate the progress that AI has made over the last decade. But some amount of goalpost shifting is actually justified.
If you showed me Gemini 3 in 2020, I would have been certain that it could automate half of knowledge work. And so we keep solving what we thought were the sufficient bottlenecks to AGI. We have models that have general understanding. They have few-shot learning. They have reasoning. And yet we still don't have AGI. So what is a rational response to observing this? I think it's totally reasonable to look at this and say, "Oh, actually there's much more to intelligence and labor than I previously realized." And while we're really close and in many ways have surpassed what I would have previously defined as AGI in the past, the fact that model companies are not making the trillions of dollars in revenue that would be implied by AGI clearly reveals that my previous definition of AGI was too narrow.
And I expect this to keep happening into the future. I expect that by 2030, the labs will have made significant progress on my hobbyhorse of continual learning and the models will be earning hundreds of billions of dollars in revenue a year, but they won't have automated all knowledge work. And I'll be like, "Look, we made a lot of progress, but we haven't hit AGI yet. We also need these other capabilities. We need X, Y, and Z capabilities in these models." Models keep getting more impressive at the rate that the short timelines people predict, but more useful at the rate that the long timelines people predict.
It's worth asking, what are we scaling? With pre-training, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude in compute, albeit this was on a power law, which is as weak as exponential growth is strong. But people are trying to launder the prestige that pre-training scaling has, which is almost as predictable as a physical law of the universe, to justify bullish predictions about reinforcement learning from verifiable reward, for which we have no well-established publicly known trend.
And when intrepid researchers do try to piece together the implications from scarce public data points, they get pretty bearish results. For example, Toby Ord has a great post where he cleverly connects the dots between the different o-series benchmarks, and this suggested to him that "we need something like a million-times scale-up in total RL compute to give a boost similar to a single GPT level."
So people have spent a lot of time talking about the possibility of a software-only singularity where AI models will write the code that generates a smarter successor system, or a software-plus-hardware singularity where AIs also improve their successor's computing hardware. However, all these scenarios neglect what I think will be the main driver of further improvements atop AGI: continual learning. Again, think about how humans become more capable at anything. It's mostly from experience in the relevant domain.
Over conversation, Beren Millidge made this interesting suggestion that the future might look like continual learning agents who are all going out and they're doing different jobs and they're generating value, and then they're bringing back all their learnings to the hive mind model, which does some kind of batch distillation on all of these agents. The agents themselves could be quite specialized, containing what Karpathy called the cognitive core plus knowledge and skills relevant to the job they're being deployed to do.
Solving continual learning won't be a singular one-and-done achievement. Instead, it will feel like solving in-context learning. GPT-3 already demonstrated in-context learning could be very powerful in 2020. Its in-context learning capabilities were so remarkable, the title of the GPT-3 paper was "Language Models are Few-Shot Learners." But of course, we didn't solve in-context learning when GPT-3 came out. And indeed, there's still plenty of progress that still has to be made, from comprehension to context length.
I expect a similar progression with continual learning. Labs will probably release something next year which they call continual learning and which will in fact count as progress towards continual learning. But human-level on-the-job learning may take another five to ten years to iron out. This is why I don't expect some kind of runaway gains from the first model that cracks continual learning that's getting more and more widely deployed and capable.
If you had fully solved continual learning drop out of nowhere, then sure, it might be game, set, match, as Satya put it on the podcast when I asked him about this possibility. But that's probably not what's going to happen. Instead, some lab is going to figure out how to get some initial traction on this problem, and then playing around with this feature will make it clear how it was implemented, and then other labs will soon replicate the breakthrough and improve it slightly.
Besides, I just have some prior that the competition will stay pretty fierce between all these model companies. And this is informed by the observation that all these previous supposed flywheels, whether that's user engagement on chat or synthetic data or whatever, have done very little to diminish the greater and greater competition between model companies. Every month or so, the big three model companies will rotate around the podium, and the other competitors are not that far behind. There seems to be some force, and this is potentially talent poaching, it's potentially the rumor mill in SF, or just normal reverse engineering, which has so far neutralized any runaway advantage that a single lab might have had.
This was a narration of an essay that I originally released on my blog at dwarkesh.com. I'm going to be publishing a lot more essays. I found it's actually quite helpful in ironing out my thoughts before interviews. If you want to stay up to date with those, you can subscribe at dwarkesh.com. Otherwise, I'll see you for the next podcast. Cheers.
Article published · Updated