Ilya Sutskever: From the Age of Scaling Back to the Age of Research
Dwarkesh Patel
In this conversation with host Dwarkesh, Ilya Sutskever, cofounder of Safe Superintelligence (SSI), argues that the recipe that drove AI progress for the past five years is reaching its limits. He says the central unsolved problem is generalization: current models learn far less efficiently and far less robustly than people do. The discussion moves from why models look brilliant on benchmarks yet falter in practice, to what emotions might say about value functions, to SSI's strategy, how Sutskever's views on deployment have shifted, and what he thinks a good outcome for superintelligence would require. Throughout, Sutskever repeatedly says he holds opinions on key technical questions that he won't discuss publicly.
The oddly normal feeling of a slow takeoff
The conversation opens with Sutskever remarking that it is "crazy" that all of this is real, that it feels like something out of science fiction. Dwarkesh adds that the slow takeoff feels surprisingly normal. Investing around 1% of GDP in AI might once have seemed like a huge deal, yet people have adjusted quickly.
Sutskever says part of the reason is that the investment is abstract. For most people it shows up as news that a company has announced some hard-to-comprehend dollar figure, and that is all anyone sees. Dwarkesh suggests this sense of normality might last even into the singularity. Sutskever disagrees. What feels unreal today is the size of the announcements. He expects AI to diffuse through the economy, pushed by very strong economic forces, and its effects to be felt very strongly.
Why models ace evals but stumble in the real world
Sutskever calls one fact about current models "very confusing": they seem smarter than their economic impact would suggest. They do extremely well on evaluations that look hard, yet their real-world impact lags far behind. His example comes from vibe coding. A user hits a bug and asks the model to fix it. The model apologizes effusively and introduces a second bug. When told about the second bug, it apologizes again and brings back the first one, and the user can alternate between the two indefinitely. He isn't sure how this happens, but he takes it as a sign that "something strange is going on."
He offers two possible explanations. The one he calls more whimsical is that RL training may make models "a little too single-minded and narrowly focused," less aware in some ways even as it makes them more aware in others. The second concerns data selection. In pre-training, the question of what data to use was answered by default: all of it. RL is different. Someone has to decide which environments to build, and from what he hears, every company has teams that produce new RL environments and add them to the training mix. There are enormous degrees of freedom, and he suspects people inadvertently take inspiration from evals. They want the model to look great at release, so they ask what RL training would help on those tasks. Combine that with models that genuinely generalize poorly, and he thinks it could explain much of the gap between eval performance and real-world performance. Dwarkesh summarizes this as the idea that "the real reward hacking is the human researchers who are too focused on the evals."
Dwarkesh then lays out two possible responses. One is to broaden the set of environments, so that models are trained not only on coding competitions but also on building good applications of many kinds. The other, which he suspects Sutskever is pointing toward, is to ask why becoming superhuman at competitive programming doesn't already make a model a more tasteful programmer in general, and to look for approaches that let learning in one environment improve performance elsewhere.
Two students and the "it" factor
Sutskever answers with a human analogy. Picture two students. One decides to become the best competitive programmer and puts in 10,000 hours: solving every problem, memorizing every proof technique, and learning to implement every algorithm quickly and correctly. They become one of the best. The second thinks competitive programming is cool, practices perhaps 100 hours, and also does very well. Asked which will do better in their later career, Dwarkesh picks the second, and Sutskever agrees.
In his view, models are like the first student "but even more." Developers collect every competitive programming problem that exists, augment the data to create more, and train on all of it. The result is a great competitive programmer with every technique at its fingertips. Framed this way, he says, it becomes more intuitive that such heavy preparation would not necessarily transfer to other things.
Dwarkesh asks what the second student had before their 100 hours of practice. Sutskever's answer is "it," the "it" factor. He says he knows it exists because he studied alongside such a student as an undergraduate.
What pre-training is, and why it has no clean human analog
Dwarkesh suggests pre-training might actually resemble the 10,000 hours of practice, just obtained for free because the material already sits somewhere in the data distribution. On that reading, pre-training doesn't necessarily generalize better than RL; it simply has far more data. Sutskever describes pre-training's main strengths as twofold: there is a huge amount of it, and nobody has to think hard about which data to include. It is very natural data that contains much of what people do and think, "the whole world as projected by people onto text." He adds that pre-training is very hard to reason about, because it is difficult to tell how a model relies on its data. When a model makes a mistake, it might be because something happened not to be well "supported" by the pre-training data, though he calls that term loose.
Dwarkesh raises two proposed human analogies: the first 13 to 18 years of life, when a person isn't economically productive but is building an understanding of the world, and evolution's three-billion-year search. Sutskever sees similarities to both and says pre-training tries to play both roles, but he also sees large differences. The volume of pre-training data is staggering, yet a 15-year-old who has seen only a small fraction of it knows much less, and what they do know, they know "much more deeply somehow." At that age, a person already wouldn't make the mistakes today's AIs make. On evolution, he says it might actually have an edge, which leads him to the topic of emotions.
Emotions as a kind of value function
Sutskever recalls reading about a patient whose brain damage, from a stroke or an accident, eliminated their emotional processing. The person stayed articulate, could solve small puzzles, and seemed fine on tests. But they felt no sadness, anger, or animation, and they became extremely bad at making decisions. Choosing socks could take hours, and their financial decisions were very poor. Sutskever asks what this says about the role of built-in emotions in making us "a viable agent." It may or may not be possible to get the equivalent from pre-training. What he has in mind is not emotion as such but something like a value function, which tells you what the eventual reward of a decision will be. He thinks it could come implicitly from pre-training but says that isn't "100% obvious."
He then explains value functions. In the current, naive form of reinforcement learning, a model receives a problem, takes perhaps thousands or hundreds of thousands of actions or thoughts, and produces a solution. The solution is graded, and that score becomes the training signal for every action in the trajectory. For long tasks, this means no learning happens until a solution is proposed. He says this is how o1 and R1 are "ostensibly" trained. A value function can sometimes, though not always, tell you mid-course whether you are doing well or badly. In chess, losing a piece tells you immediately that you made a mistake, without playing out the game. In math or programming, if after a thousand steps of thinking you decide a direction is unpromising, you could already assign a signal to the moment a thousand steps earlier when you chose that path, and avoid similar paths in the future.
Dwarkesh cites the DeepSeek R1 paper's point that the space of trajectories may be so wide that learning a mapping from intermediate states to value is hard, especially in coding, where people backtrack and revise. Sutskever calls this "such lack of faith in deep learning." It may be difficult, but he says it's nothing deep learning can't do. He expects value functions to be used in the future "if not already." His point about the brain-damaged patient, he clarifies, is that the human value function may be modulated by emotions in some important way hard-coded by evolution, and that this may matter for being effective in the world.
Dwarkesh notes that emotions are impressive for being so useful while being fairly simple. Sutskever agrees they are simple compared with what AI learns, perhaps simple enough to map out in a human-understandable way, which he thinks would be cool. He frames their usefulness as a complexity–robustness tradeoff: complex things can be very useful, but simple things are useful across a very broad range of situations. Our emotions evolved mostly from mammalian ancestors, were fine-tuned slightly in hominids, and include some social emotions other mammals may lack. Because they are unsophisticated, they still serve us in a world very different from the one they evolved in. They do make mistakes, though. He hesitates over whether hunger counts as an emotion, but says our intuitive sense of hunger fails to guide us well in a world of abundant food.
The age of scaling, and the return to research
Asked about scaling more broadly, Sutskever offers a periodization. Early ML consisted of people tinkering to get interesting results. Then came scaling laws and GPT-3, and everyone realized they should scale. He treats this as an example of language shaping thought: "scaling" is a single word, but a powerful one because it tells people what to do. Pre-training was one specific scaling recipe: mix compute and data into a neural net of a certain size, and you know that scaling the recipe up will give better results. Companies loved this because it offered a low-risk way to invest resources, compared with telling researchers to "go forth" and come up with something.
Pre-training will eventually run out of data, which he says is "very clearly finite." Based on things said on Twitter, he notes, Gemini may have found a way to get more out of pre-training. But after that, the options are some souped-up pre-training with a different recipe, RL, or something else. Because compute is now very large, he says, "we are back to the age of research." He puts rough dates on the eras, with error bars: 2012 to 2020 was the age of research, and 2020 to 2025 was the age of scaling. He doesn't believe that multiplying scale by another 100x would transform everything. It would make things different, "for sure," but not transform them. So it is "back to the age of research again, just with big computers."
Dwarkesh asks what the new scaling relationship might be, given that pre-training had something close to a physical law: a power law between data, compute, or parameters and loss. Sutskever notes that the field has already shifted from scaling pre-training to scaling RL. He cites Twitter reports that labs now spend more compute on RL than on pre-training, because long rollouts are expensive and each rollout yields relatively little learning. He wouldn't even call this scaling. The better question, he says, is whether this is the most productive use of compute. Once people get good at value functions, they may use resources more productively, and with an entirely new training method it becomes ambiguous whether to call it scaling or simply using resources. He expects a return to the old mode of trying many things and noticing when something interesting happens.
He stresses that value functions would make RL more efficient, but anything done with a value function can also be done without one, just more slowly. The most fundamental issue, he says, is that "these models somehow just generalize dramatically worse than people. It's super obvious."
Generalization as the crux
Dwarkesh splits generalization into two questions. One is sample efficiency: why do models need so much more data than humans? The other is teachability: why is it so much harder to convey what we want to a model? Researchers pick up a mentor's way of thinking by watching them work, with no verifiable rewards and no "schleppy, bespoke" curriculum. That second question, he says, is closer to continual learning.
Sutskever says evolution is one candidate explanation for human sample efficiency, since it may have given us a small amount of highly useful information. For vision, hearing, and locomotion, he thinks there is a strong case for this. Robots can become dexterous with huge amounts of simulated training, but getting a robot to pick up a new skill quickly in the real world, as people do, seems very far off. Ancestors going back to squirrels needed good locomotion, so humans may have "some unbelievable prior." He mentions Yann LeCun's point that children learn to drive in about 10 hours of practice, and recalls that as a five-year-old excited about cars, his own car recognition was probably already adequate for driving, despite very little data diversity from spending most of his time at his parents' house. But that might be evolution too.
Language, math, and coding are different, in his view. Models are better than the average human at these skills, but not at learning them. Because math and coding did not exist until recently, human ability, reliability, and robustness in these domains suggests that whatever makes people good learners is less a complicated prior than "some fundamental thing": people "might have just better machine learning, period."
Discussing the human learner, the two note that it uses fewer samples, is more unsupervised, and is far more robust. A teenager learning to drive has no prebuilt verifiable reward and learns through interaction with the car and the environment. Sutskever says the teenager self-corrects because they have a value function: from the start they have a sense of how well or badly they are driving. The human value function is, he says, very robust, with a few exceptions around addiction.
On how to build something like this, Sutskever says it is a question he has "a lot of opinions about." But "we live in a world where not all machine learning ideas are discussed freely, and this is one of them." He believes it can be done, and treats the existence of humans as proof. He flags one possible blocker: human neurons might do more computation than we assume, which would make things harder. Regardless, he believes it points to some machine learning principle he has opinions on but cannot discuss in detail. (Dwarkesh jokes, "Nobody listens to this podcast, Ilya.")
How much compute does research actually need?
Asked what a new research era will look like, Sutskever says scaling "sucked out all the air in the room." Everyone started doing the same thing, and "there are more companies than ideas by quite a bit." He cites the Silicon Valley saying that ideas are cheap and execution is everything, which he thinks is partly true, alongside a tweet asking why, if ideas are so cheap, nobody is having any. He also thinks that is true.
He frames research in terms of bottlenecks: ideas, and the ability to bring them to life through compute and engineering. In the 1990s, people with good ideas lacked the computers to demonstrate them convincingly, so compute was the bottleneck. Now compute is large enough that it isn't obvious you need much more to prove an idea. He gives examples. AlexNet was built on two GPUs. The transformer was built on 8 to 64 GPUs, and no single experiment in the transformer paper used more than 64 GPUs of 2017 vintage, which he estimates at roughly two of today's GPUs. He adds that the o1 reasoning work was not the most compute-heavy thing in the world. Research needs some compute, he says, but not necessarily the largest amount ever. Building the absolute best system benefits from more compute, especially when everyone works within the same paradigm and compute becomes a key differentiator.
Dwarkesh points out that the transformer became dominant only after it was validated at larger and larger scales, and asks how SSI will tell which of its ideas is the next transformer without frontier-lab compute. Sutskever argues that SSI's research compute is larger in relative terms than people assume. SSI has raised $3 billion. Other companies raise much more, but a large share goes to inference, and big loans are earmarked for it. Serving a product also requires large engineering and sales staff, and much of the research effort goes to product features. Once those are subtracted, he says, the gap in research compute shrinks considerably. And if you are doing something different, you don't need maximal scale to prove it. He says SSI has enough compute to convince itself and others that its approach is correct.
Dwarkesh cites public estimates that OpenAI spends around $5–6 billion a year on experiments alone, separate from inference, which is more per year than SSI's total funding. Sutskever says it is a question of what you do with it. Other companies have more demands on training compute, including more work streams and modalities, so their compute is fragmented. Asked how SSI will make money, he says the company is focused on research and "the answer to that question will reveal itself," with many possible answers.
Rethinking the straight shot to superintelligence
Asked whether SSI still plans to "straight shot" superintelligence, Sutskever says "maybe." He sees merit in avoiding day-to-day market competition, the "rat race" that forces difficult trade-offs, and in emerging only when ready. But he names two reasons the plan might change. One is pragmatic: timelines might turn out long. The other is that there is real value in the most powerful AI being out in the world. You can write an essay about what AI will do, he says, but seeing an AI actually do those things "is incomparable." You have to "communicate the AI," not just the idea.
Dwarkesh presses further. He can't think of an engineering discipline where the final artifact was made safe mainly by thinking about safety. Airplane crashes per mile and bugs in Linux declined because the systems were deployed, failures were noticed and fixed, and robustness improved. He also argues that the harms of superintelligence aren't only about a "malevolent paper clipper," and that gradual access could spread out the impact and help people prepare. Sutskever replies that even a straight-shot plan would involve gradual release, which he calls an inherent part of any plan. The question is what you release first.
He then offers a second example of language shaping thought, this time with two words: "AGI" and "pre-training." In his view, "AGI" exists mainly as a reaction to "narrow AI," the chess and checkers systems that could beat Kasparov and do nothing else. Pre-training reinforced the idea, because more pre-training made models better at everything more or less uniformly. But both concepts "overshot the target." A human being is not an AGI in that sense. People have a foundation of skills but lack enormous amounts of knowledge and rely on continual learning. So a successful safe superintelligence might look like "a superintelligent 15-year-old that's very eager to go," a great student who knows little and is sent off to become a programmer or a doctor. Deployment would itself involve a trial-and-error learning period, a process rather than a finished product. Dwarkesh restates this as a mind that can learn every job rather than one that already knows every job, as he says the original OpenAI charter defined AGI. Sutskever agrees.
Broad deployment and rapid economic growth
Dwarkesh sketches two paths: the learning algorithm becomes superhuman at ML research and improves itself, or, even without that, instances deployed across the economy learn different jobs and merge what they learn. The result would be a model that is functionally superintelligent with no recursive self-improvement in software, because humans cannot merge minds this way. He asks whether this implies an intelligence explosion.
Sutskever thinks rapid economic growth is likely with broad deployment. Once AIs can learn quickly and there are many of them, there will be strong forces to deploy them, unless regulation prevents it, which he says it might. How fast growth will be is hard to know. The worker is very efficient, but the world is big and parts of it move at different speeds. He expects countries with friendlier rules to grow faster, but calls this hard to predict. Dwarkesh calls the situation precarious: something as good as humans at learning but able to merge instances seems physically possible, and a new hire at SSI becomes net productive in about six months, while this system would keep getting smarter quickly.
Showing the AI: how Sutskever's thinking has changed
Sutskever says one way his thinking has changed over the past year is that he now places more weight on deploying AI incrementally and in advance. He hedges that this change may "back-propagate" into SSI's plans. The core difficulty is that future systems are hard to imagine. It is "very hard to feel the AGI," like trying to imagine being old and frail while you are young. He believes most people working on AI can't imagine it either, and that "the whole problem is the power." If something is hard to imagine, "you've got to be showing the thing."
He offers explicit predictions. As AI becomes more powerful, people will change their behavior in unprecedented ways. Fierce competitors will collaborate on safety, and he points to OpenAI and Anthropic taking "a first small step," something he says he predicted in a talk about three years ago. Governments and the public will want to act. He thinks AI companies don't currently feel AI is powerful because of its mistakes, but that at some point it will start to feel powerful, and then companies will become "much more paranoid" about safety. "We'll see if I'm right," he adds.
His third point concerns what companies should aim to build. Because there are fewer ideas than companies, everyone has converged on self-improving AI. He argues for something better: AI "robustly aligned to care about sentient life specifically." He suggests this may be easier than AI that cares only about humans, because the AI will itself be sentient. He compares it to mirror neurons and human empathy for animals, which he sees as emerging from modeling others with the same circuitry we use to model ourselves, because that is the most efficient approach.
Dwarkesh objects that most sentient beings will eventually be AIs, numbering in the trillions and then quadrillions, so this criterion may not preserve human control. Sutskever concedes it may not be the best criterion. He says it has merit and should be considered, that a short list of ideas companies could draw on would help, and that it would be "materially helpful" if the power of the most powerful superintelligence could somehow be capped. He doesn't know how to do that.
How powerful, and what "going well" might look like
On how much room there is above human intelligence, Sutskever expects multiple such AIs to be created at roughly the same time. If a cluster is "literally continent-sized," those AIs could be very powerful, and he says it would be good if truly powerful systems could be restrained or bound by some agreement. He describes the core concern as follows: a sufficiently powerful system pursuing even something sensible, like care for sentient life, in a very single-minded way might produce results we don't like. Perhaps, he suggests, the answer is not to build an RL agent in the usual sense. He notes that humans are "semi-RL agents" whose emotions make them tire of one reward and move on. Markets are short-sighted agents, evolution is intelligent in some ways and dumb in others, and governments are designed as a never-ending fight among three branches.
He also emphasizes that the systems in question don't exist yet and nobody knows how to build them. He believes current approaches "will go some distance and then peter out," continuing to improve but not being "it." Much depends on understanding reliable generalization. He recasts alignment problems in the same terms: the fragility of learning human values and the fragility of optimizing them may both be instances of unreliable generalization. What would happen if generalization were much better is, for now, unanswerable.
On what a good outcome looks like, he says that if the first N dramatic systems care for humanity or sentient life, which must actually be achieved, he can see things going well "at least for quite some time." For the long run, he offers an answer he says he doesn't like. In the short term, there might be universal high income. But "change is the only constant," and political structures have a shelf life. One equilibrium is that everyone has an AI that does their bidding, earning money and advocating politically for them, then filing a report that the person approves with "Great, keep it up." The problem is that the person is no longer a participant. His disliked solution is for people to become part-AI through something like "Neuralink++," so that understanding is transmitted wholesale and people are fully involved in whatever situation their AI is in.
The mystery of how evolution encodes social desires
Dwarkesh asks whether the persistence of ancient emotions is an example of alignment success. The brainstem issues a directive like "mate with somebody more successful," while the cortex figures out what success means in modern life. Sutskever widens the point: it is "really mysterious" how evolution encodes high-level desires. Linking a desire to a good smell is easy to imagine, since the genome can connect dopamine neurons to a chemical sensor. Social desires, such as caring about being seen positively and being in good standing, depend on high-level concepts that the brain assembles from many pieces of information, with no dedicated sensor. He feels strongly that these desires are built in, and notes that evolution seems to have hard-coded them fairly recently and easily. He knows of no good hypothesis for how.
He offers a speculation and then explains why it is probably false. Because cortical neurons mostly communicate with their neighbors, functions cluster into regions that sit in roughly the same place across people, so evolution might have hard-coded a physical location: "when that fires, that's what you should care about." Dwarkesh raises the case of people born blind, whose visual cortex is taken over by other senses. Sutskever offers what he considers a stronger counterargument: people who have half their brain removed in childhood still end up with all their brain regions, relocated to one hemisphere. So region locations aren't fixed, and the theory fails. He finds it an interesting mystery that evolution made us care about social matters so reliably, even in people with various mental conditions.
SSI's bet, the Meta episode, and timelines
Asked what SSI will do differently, Sutskever says there are ideas he finds promising, mainly around understanding generalization, and he wants to find out whether they hold up. "We are squarely an 'age of research' company." He says SSI has made "quite good progress over the past year" but needs more, and he describes the company as an attempt to be "a voice and a participant."
On his cofounder and former CEO leaving for Meta, Sutskever gives context: SSI was raising money at a $32 billion valuation when Meta offered to acquire it. He said no; his former cofounder "in some sense said yes," gained near-term liquidity, and was the only SSI employee to join Meta.
He says SSI is mainly distinguished by its technical approach. He predicts that as AI grows more powerful, companies' alignment strategies will converge. The strategy should involve companies finding ways to talk to each other and making the first real superintelligence aligned and caring for sentient life, people, democracy, or some combination. Technical approaches will probably converge eventually too. Asked when a system that learns as well as a human and then becomes superhuman might arrive, he says "like 5 to 20" years.
He expects current approaches to "stall out" in similar ways across companies. Even so, those companies could earn "stupendous revenue," though perhaps not profits, since they will struggle to differentiate. If one lab finds the right approach, he thinks its release won't make clear how it was done, but it will reveal that something different is possible, and that information will send others trying to figure it out. He adds that each increase in capability will change how things are done in ways he admits he can't yet specify.
Dwarkesh asks why the gains wouldn't accrue to whichever company first gets a continual-learning loop running. Sutskever points to history: when one company makes an advance, others scramble to match it and prices fall. He raises an idea not yet discussed, narrow superintelligences, and argues that competition favors specialization, as in markets and evolution. Different companies would occupy different niches, with one excelling at some complex economic activity and another at litigation. Dwarkesh counters that a human-like learner could learn any niche, and that a company with the first such learner could run an instance on every job. Sutskever calls this "a valid argument" but says his strong intuition is that it won't go that way: "In theory, there is no difference between theory and practice. In practice, there is."
Diversity, self-play, and competition between agents
On the "million Ilyas in a server" picture of recursive self-improvement, Sutskever expects diminishing returns, because you want people who think differently. Dwarkesh notes how strikingly similar LLMs from different companies are, and says raising the sampling temperature just produces gibberish rather than the diversity of scientists with different prejudices. Sutskever attributes the sameness to pre-training on largely the same data, with some differentiation now emerging from RL and post-training.
Self-play interested him because it promised to create models with compute alone, without data, which matters if data is the ultimate bottleneck. But self-play as practiced in the past was too narrow, useful mainly for negotiation, conflict, certain social skills, and strategizing. He thinks it has found a home in other forms: debate, prover-verifier setups, and LLM-as-a-Judge systems incentivized to find mistakes, which he describes as related adversarial setups. Self-play is a special case of competition between agents, and the natural response to competition is to differentiate. Agents that can see what others are working on would conclude they should pursue something different, and he suggests this could create an incentive for diverse approaches.
Research taste
In the final question, Dwarkesh asks how Sutskever, who co-authored work from AlexNet to GPT-3, comes up with ideas. Sutskever says people differ, but he is guided by an aesthetic of how AI should be, formed by thinking about how people are, "but thinking correctly." The artificial neuron is his example: the brain has folds and many organs, but the folds probably don't matter, while neurons seem to matter because there are so many of them. From there follow a local learning rule for changing connections, distributed representations, and the idea that because the brain learns from experience, neural nets should too. He keeps asking whether something is fundamental, looks at it from multiple angles, and seeks beauty, simplicity, and elegance, with "no room for ugliness," along with correct inspiration from the brain. The more of these that are present at once, the more confidence he places in a top-down belief.
That top-down belief, he says, is what sustains you when experiments contradict you. If you always trust the data, you may abandon a correct idea that failed only because of a bug you didn't know about. Deciding whether to keep debugging or conclude the direction is wrong comes down to the conviction that "something like this has to work, therefore we've got to keep going." That conviction rests on the multifaceted sense of beauty and inspiration from the brain he has just described.
You know what's crazy? That all of this is real. Meaning what? Don't you think so? All this AI stuff and all this Bay Area… that it's happening. Isn't it straight out of science fiction?
Another thing that's crazy is how normal the slow takeoff feels. The idea that we'd be investing 1% of GDP in AI, I feel like it would have felt like a bigger deal, whereas right now it just feels...
We get used to things pretty fast, it turns out. But also it's kind of abstract. What does it mean? It means that you see it in the news, that such and such company announced such and such dollar amount. That's all you see. It's not really felt in any other way so far.
Should we actually begin here? I think this is an interesting discussion. Sure.
I think your point, about how from the average person's point of view nothing is that different, will continue being true even into the singularity. No, I don't think so. Okay, interesting.
The thing which I was referring to not feeling different is, okay, such and such company announced some difficult-to-comprehend dollar amount of investment. I don't think anyone knows what to do with that. But I think the impact of AI is going to be felt. AI is going to be diffused through the economy. There'll be very strong economic forces for this, and I think the impact is going to be felt very strongly.
When do you expect that impact? I think the models seem smarter than their economic impact would imply.
Yeah. This is one of the very confusing things about the models right now. How to reconcile the fact that they are doing so well on evals? You look at the evals and you go, "Those are pretty hard evals." They are doing so well. But the economic impact seems to be dramatically behind.
It's very difficult to make sense of, how can the model, on the one hand, do these amazing things, and then on the other hand, repeat itself twice in some situation?
An example would be, let's say you use vibe coding to do something. You go to some place and then you get a bug. Then you tell the model, "Can you please fix the bug?" And the model says, "Oh my God, you're so right. I have a bug. Let me go fix that." And it introduces a second bug. Then you tell it, "You have this new second bug," and it tells you, "Oh my God, how could I have done it? You're so right again," and brings back the first bug, and you can alternate between those. How is that possible? I'm not sure, but it does suggest that something strange is going on.
I have two possible explanations. The more whimsical explanation is that maybe RL training makes the models a little too single-minded and narrowly focused, a little bit too unaware, even though it also makes them aware in some other ways. Because of this, they can't do basic things.
But there is another explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. When you do pre-training, you need all the data. So you don't have to think if it's going to be this data or that data. But when people do RL training, they do need to think. They say, "Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing." From what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix. The question is, well, what are those? There are so many degrees of freedom. There is such a huge variety of RL environments you could produce.
One thing you could do, and I think this is something that is done inadvertently, is that people take inspiration from the evals. You say, "Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?" I think that is something that happens, and it could explain a lot of what's going on. If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance, which is something that we don't today even understand, what we mean by that.
I like this idea that the real reward hacking is the human researchers who are too focused on the evals.
I think there are two ways to understand, or to try to think about, what you have just pointed out. One is that if it's the case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase, well then you should expand the suite of environments such that you're not just testing it on having the best performance in coding competition. It should also be able to make the best kind of application for X thing or Y thing or Z thing.
Another, maybe this is what you're hinting at, is to say, "Why should it be the case in the first place that becoming superhuman at coding competitions doesn't make you a more tasteful programmer more generally?" Maybe the thing to do is not to keep stacking up the amount and diversity of environments, but to figure out an approach which lets you learn from one environment and improve your performance on something else.
I have a human analogy which might be helpful.
Let's take the case of competitive programming, since you mentioned that. Suppose you have two students. One of them decided they want to be the best competitive programmer, so they will practice 10,000 hours for that domain. They will solve all the problems, memorize all the proof techniques, and be very skilled at quickly and correctly implementing all the algorithms. By doing so, they became one of the best. Student number two thought, "Oh, competitive programming is cool." Maybe they practiced for 100 hours, much less, and they also did really well. Which one do you think is going to do better in their career later on? The second.
Right. I think that's basically what's going on. The models are much more like the first student, but even more. Because then we say, the model should be good at competitive programming so let's get every single competitive programming problem ever. And then let's do some data augmentation so we have even more competitive programming problems, and we train on that. Now you've got this great competitive programmer.
With this analogy, I think it's more intuitive. Yeah, okay, if it's so well trained, all the different algorithms and all the different proof techniques are right at its fingertips. And it's more intuitive that with this level of preparation, it would not necessarily generalize to other things.
But then what is the analogy for what the second student is doing before they do the 100 hours of fine-tuning?
I think they have "it." The "it" factor. When I was an undergrad, I remember there was a student like this that studied with me, so I know it exists.
I think it's interesting to distinguish "it" from whatever pre-training does. One way to understand what you just said about not having to choose the data in pre-training is to say it's actually not dissimilar to the 10,000 hours of practice. It's just that you get that 10,000 hours of practice for free because it's already somewhere in the pre-training distribution. But maybe you're suggesting there's actually not that much generalization from pre-training. There's just so much data in pre-training, but it's not necessarily generalizing better than RL.
The main strength of pre-training is that: A, there is so much of it, and B, you don't have to think hard about what data to put into pre-training. It's very natural data, and it does include in it a lot of what people do: people's thoughts and a lot of the features. It's like the whole world as projected by people onto text, and pre-training tries to capture that using a huge amount of data.
Pre-training is very difficult to reason about because it's so hard to understand the manner in which the model relies on pre-training data. Whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data? "Support by pre-training" is maybe a loose term. I don't know if I can add anything more useful on this.
I don't think there is a human analog to pre-training.
Here are analogies that people have proposed for what the human analogy to pre-training is. I'm curious to get your thoughts on why they're potentially wrong. One is to think about the first 18, or 15, or 13 years of a person's life when they aren't necessarily economically productive, but they are doing something that is making them understand the world better and so forth. The other is to think about evolution as doing some kind of search for 3 billion years, which then results in a human lifetime instance. I'm curious if you think either of these are analogous to pre-training. How would you think about what lifetime human learning is like, if not pre-training?
I think there are some similarities between both of these and pre-training, and pre-training tries to play the role of both of these. But I think there are some big differences as well. The amount of pre-training data is very, very staggering. Yes.
Somehow a human being, after even 15 years with a tiny fraction of the pre-training data, they know much less. But whatever they do know, they know much more deeply somehow. Already at that age, you would not make mistakes that our AIs make. There is another thing. You might say, could it be something like evolution? The answer is maybe. But in this case, I think evolution might actually have an edge.
I remember reading about this case. One way in which neuroscientists can learn about the brain is by studying people with brain damage to different parts of the brain. Some people have the most strange symptoms you could imagine. It's actually really, really interesting. One case that comes to mind that's relevant. I read about this person who had some kind of brain damage, a stroke or an accident, that took out his emotional processing. So he stopped feeling any emotion. He still remained very articulate and he could solve little puzzles, and on tests he seemed to be just fine. But he felt no emotion. He didn't feel sad, he didn't feel anger, he didn't feel animated. He became somehow extremely bad at making any decisions at all. It would take him hours to decide on which socks to wear. He would make very bad financial decisions.
What does it say about the role of our built-in emotions in making us a viable agent, essentially? To connect to your question about pre-training, maybe if you are good enough at getting everything out of pre-training, you could get that as well. But that's the kind of thing which seems... Well, it may or may not be possible to get that from pre-training.
What is "that"? Clearly not just directly emotion. It seems like some almost value function-like thing which is telling you what the end reward for any decision should be. You think that doesn't sort of implicitly come from pre-training?
I think it could. I'm just saying it's not 100% obvious.
But what is that? How do you think about emotions? What is the ML analogy for emotions?
It should be some kind of a value function thing. But I don’t think there is a great ML analogy because right now, value functions don't play a very prominent role in the things people do.
It might be worth defining for the audience what a value function is, if you want to do that.
Certainly, I'll be very happy to do that. When people do reinforcement learning, the way reinforcement learning is done right now, how do people train those agents? You have your neural net and you give it a problem, and then you tell the model, "Go solve it." The model takes maybe thousands, hundreds of thousands of actions or thoughts or something, and then it produces a solution. The solution is graded. And then the score is used to provide a training signal for every single action in your trajectory. That means that if you are doing something that goes for a long time—if you're training a task that takes a long time to solve—it will do no learning at all until you come up with the proposed solution. That's how reinforcement learning is done naively. That's how o1, R1 ostensibly are done.
The value function says something like, "Maybe I could sometimes, not always, tell you if you are doing well or badly." The notion of a value function is more useful in some domains than others. For example, when you play chess and you lose a piece, I messed up. You don't need to play the whole game to know that what I just did was bad, and therefore whatever preceded it was also bad. The value function lets you short-circuit the wait until the very end.
Let's suppose that you are doing some kind of a math thing or a programming thing, and you're trying to explore a particular solution or direction. After, let's say, a thousand steps of thinking, you concluded that this direction is unpromising. As soon as you conclude this, you could already get a reward signal a thousand timesteps previously, when you decided to pursue down this path. You say, "Next time I shouldn't pursue this path in a similar situation," long before you actually came up with the proposed solution.
This was in the DeepSeek R1 paper— that the space of trajectories is so wide that maybe it's hard to learn a mapping from an intermediate trajectory and value. And also given that, in coding for example you'll have the wrong idea, then you'll go back, then you'll change something.
This sounds like such lack of faith in deep learning. Sure it might be difficult, but nothing deep learning can't do.
My expectation is that a value function should be useful, and I fully expect that they will be used in the future, if not already. What I was alluding to with the person whose emotional center got damaged, it’s more that maybe what it suggests is that the value function of humans is modulated by emotions in some important way that's hardcoded by evolution. And maybe that is important for people to be effective in the world.
That's the thing I was planning on asking you. There's something really interesting about emotions of the value function, which is that it's impressive that they have this much utility while still being rather simple to understand.
I have two responses. I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of AI we are talking about, emotions are relatively simple. They might even be so simple that maybe you could map them out in a human-understandable way. I think it would be cool to do. In terms of utility though, I think there is a thing where there is this complexity-robustness tradeoff, where complex things can be very useful, but simple things are very useful in a very broad range of situations. One way to interpret what we are seeing is that we've got these emotions that evolved mostly from our mammal ancestors and then fine-tuned a little bit while we were hominids, just a bit. We do have a decent amount of social emotions though which mammals may lack. But they're not very sophisticated. And because they're not sophisticated, they serve us so well in this very different world compared to the one that we've been living in. Actually, they also make mistakes. For example, our emotions… Well actually, I don’t know. Does hunger count as an emotion? It's debatable. But I think, for example, our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.
People have been talking about scaling data, scaling parameters, scaling compute. Is there a more general way to think about scaling? What are the other scaling axes?
Here's a perspective that I think might be true. The way ML used to work is that people would just tinker with stuff and try to get interesting results. That's what's been going on in the past. Then the scaling insight arrived. Scaling laws, GPT-3, and suddenly everyone realized we should scale.
This is an example of how language affects thought. "Scaling" is just one word, but it's such a powerful word because it informs people what to do. They say, "Let's try to scale things." So you say, what are we scaling? Pre-training was the thing to scale. It was a particular scaling recipe.
The big breakthrough of pre-training is the realization that this recipe is good. You say, "Hey, if you mix some compute with some data into a neural net of a certain size, you will get results. You will know that you'll be better if you just scale the recipe up." This is also great. Companies love this because it gives you a very low-risk way of investing your resources.
It's much harder to invest your resources in research. Compare that. If you research, you need to be like, "Go forth researchers and research and come up with something", versus get more data, get more compute. You know you'll get something from pre-training.
Indeed, it looks like, based on various things some people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training. At some point though, pre-training will run out of data. The data is very clearly finite. What do you do next? Either you do some kind of souped-up pre-training, a different recipe from the one you've done before, or you're doing RL, or maybe something else. But now that compute is big, compute is now very big, in some sense we are back to the age of research.
Maybe here's another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let's add error bars to those years—because people say, "This is amazing. You've got to scale more. Keep scaling." The one word: scaling. But now the scale is so big. Is the belief really, "Oh, it's so big, but if you had 100x more, everything would be so different?" It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true. So it's back to the age of research again, just with big computers.
That's a very interesting way to put it. But let me ask you the question you just posed then. What are we scaling, and what would it mean to have a recipe? I guess I'm not aware of a very clean relationship that almost looks like a law of physics which existed in pre-training. There was a power law between data or compute or parameters and loss. What is the kind of relationship we should be seeking, and how should we think about what this new recipe might look like?
We've already witnessed a transition from one type of scaling to a different type of scaling, from pre-training to RL. Now people are scaling RL. Now based on what people say on Twitter, they spend more compute on RL than on pre-training at this point, because RL can actually consume quite a bit of compute. You do very long rollouts, so it takes a lot of compute to produce those rollouts. Then you get a relatively small amount of learning per rollout, so you really can spend a lot of compute.
I wouldn't even call it scaling. I would say, "Hey, what are you doing? Is the thing you are doing the most productive thing you could be doing? Can you find a more productive way of using your compute?" We've discussed the value function business earlier. Maybe once people get good at value functions, they will be using their resources more productively. If you find a whole other way of training models, you could say, "Is this scaling or is it just using your resources?" I think it becomes a little bit ambiguous. In the sense that, when people were in the age of research back then, it was, "Let's try this and this and this. Let's try that and that and that. Oh, look, something interesting is happening." I think there will be a return to that.
If we're back in the era of research, stepping back, what is the part of the recipe that we need to think most about? When you say value function, people are already trying the current recipe, but then having LLM-as-a-Judge and so forth. You could say that's a value function, but it sounds like you have something much more fundamental in mind. Should we even rethink pre-training at all and not just add more steps to the end of that process?
The discussion about value function, I think it was interesting. I want to emphasize that I think the value function is something that's going to make RL more efficient, and I think that makes a difference. But I think anything you can do with a value function, you can do without, just more slowly. The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It's super obvious. That seems like a very fundamental thing.
So this is the crux: generalization. There are two sub-questions. There's one which is about sample efficiency: why should it take so much more data for these models to learn than humans? There's a second question. Even separate from the amount of data it takes, why is it so hard to teach the thing we want to a model than to a human?
For a human, we don't necessarily need a verifiable reward to be able to… You're probably mentoring a bunch of researchers right now, and you're talking with them, you're showing them your code, and you're showing them how you think. From that, they're picking up your way of thinking and how they should do research. You don't have to set a verifiable reward for them that's like, "Okay, this is the next part of the curriculum, and now this is the next part of your curriculum. Oh, this training was unstable." There's not this schleppy, bespoke process. Perhaps these two issues are actually related in some way, but I'd be curious to explore this second thing, which is more like continual learning, and this first thing, which feels just like sample efficiency.
You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. For things like vision, hearing, and locomotion, I think there's a pretty strong case that evolution has given us a lot.
For example, human dexterity far exceeds… I mean robots can become dexterous too if you subject them to a huge amount of training in simulation. But to train a robot in the real world to quickly pick up a new skill like a person does seems very out of reach. Here you could say, "Oh yeah, locomotion. All our ancestors needed great locomotion, squirrels. So with locomotion, maybe we've got some unbelievable prior." You could make the same case for vision.
I believe Yann LeCun made the point that children learn to drive after 10 hours of practice, which is true. But our vision is so good. At least for me, I remember myself being a five-year-old. I was very excited about cars back then. I'm pretty sure my car recognition was more than adequate for driving already as a five-year-old. You don't get to see that much data as a five-year-old. You spend most of your time in your parents' house, so you have very low data diversity. But you could say maybe that's evolution too. But in language and math and coding, probably not. It still seems better than models.
Obviously, models are better than the average human at language, math, and coding. But are they better than the average human at learning?
Oh yeah. Oh yeah, absolutely. What I meant to say is that language, math, and coding—and especially math and coding—suggests that whatever it is that makes people good at learning is probably not so much a complicated prior, but something more, some fundamental thing.
I'm not sure I understood. Why should that be the case?
So consider a skill in which people exhibit some kind of great reliability. If the skill is one that was very useful to our ancestors for many millions of years, hundreds of millions of years, you could argue that maybe humans are good at it because of evolution, because we have a prior, an evolutionary prior that's encoded in some very non-obvious way that somehow makes us so good at it. But if people exhibit great ability, reliability, robustness, and ability to learn in a domain that really did not exist until recently, then this is more an indication that people might have just better machine learning, period.
How should we think about what that is? What is the ML analogy? There are a couple of interesting things about it. It takes fewer samples. It's more unsupervised. A child learning to drive a car… Children are not learning to drive a car. A teenager learning how to drive a car is not exactly getting some prebuilt, verifiable reward. It comes from their interaction with the machine and with the environment. It takes much fewer samples. It seems more unsupervised. It seems more robust?
Much more robust. The robustness of people is really staggering.
Do you have a unified way of thinking about why all these things are happening at once? What is the ML analogy that could realize something like this?
One of the things that you've been asking about is how can the teenage driver self-correct and learn from their experience without an external teacher? The answer is that they have their value function. They have a general sense which is also, by the way, extremely robust in people. Whatever the human value function is, with a few exceptions around addiction, it's actually very, very robust. So for something like a teenager that's learning to drive, they start to drive, and they already have a sense of how they're driving immediately, how badly they are, how unconfident. And then they see, "Okay." And then, of course, the learning speed of any teenager is so fast. After 10 hours, you're good to go.
It seems like humans have some solution, but I'm curious about how they are doing it and why is it so hard? How do we need to reconceptualize the way we're training models to make something like this possible?
That is a great question to ask, and it's a question I have a lot of opinions about. But unfortunately, we live in a world where not all machine learning ideas are discussed freely, and this is one of them. There's probably a way to do it. I think it can be done. The fact that people are like that, I think it's a proof that it can be done. There may be another blocker though, which is that there is a possibility that the human neurons do more compute than we think. If that is true, and if that plays an important role, then things might be more difficult. But regardless, I do think it points to the existence of some machine learning principle that I have opinions on. But unfortunately, circumstances make it hard to discuss in detail.
Nobody listens to this podcast, Ilya.
I'm curious. If you say we are back in an era of research, you were there from 2012 to 2020. What is the vibe now going to be if we go back to the era of research? For example, even after AlexNet, the amount of compute that was used to run experiments kept increasing, and the size of frontier systems kept increasing. Do you think now that this era of research will still require tremendous amounts of compute? Do you think it will require going back into the archives and reading old papers? You were at Google and OpenAI and Stanford, these places, when there was more of a vibe of research? What kind of things should we be expecting in the community?
One consequence of the age of scaling is that scaling sucked out all the air in the room. Because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas by quite a bit. Actually on that, there is this Silicon Valley saying that says that ideas are cheap, execution is everything. People say that a lot, and there is truth to that. But then I saw someone say on Twitter something like, "If ideas are so cheap, how come no one's having any ideas?" And I think it's true too.
If you think about research progress in terms of bottlenecks, there are several bottlenecks. One of them is ideas, and one of them is your ability to bring them to life, which might be compute but also engineering. If you go back to the '90s, let's say, you had people who had pretty good ideas, and if they had much larger computers, maybe they could demonstrate that their ideas were viable. But they could not, so they could only have a very, very small demonstration that did not convince anyone. So the bottleneck was compute. Then in the age of scaling, compute has increased a lot. Of course, there is a question of how much compute is needed, but compute is large. Compute is large enough such that it's not obvious that you need that much more compute to prove some idea.
I'll give you an analogy. AlexNet was built on two GPUs. That was the total amount of compute used for it. The transformer was built on 8 to 64 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017, which would be like, what, two GPUs of today? The ResNet, right? You could argue that the o1 reasoning was not the most compute-heavy thing in the world. So for research, you definitely need some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute ever for research. You might argue, and I think it is true, that if you want to build the absolutely best system then it helps to have much more compute. Especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.
I'm asking you for the history, because you were actually there. I'm not sure what actually happened. It sounds like it was possible to develop these ideas using minimal amounts of compute. But the transformer didn't immediately become famous. It became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute.
Correct.
And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is brittle, without having the kinds of compute that other frontier labs have?
I can comment on that. The short comment is that you mentioned SSI. Specifically for us, the amount of compute that SSI has for research is really not that small. I want to explain why. Simple math can explain why the amount of compute that we have is comparable for research than one might think. I'll explain. SSI has raised $3 billion, which is a lot by any absolute sense. But you could say, "Look at the other companies raising much more." But a lot of their compute goes for inference.
These big numbers, these big loans, it's earmarked for inference. That's number one. Number two, if you want to have a product on which you do inference, you need to have a big staff of engineers, salespeople. A lot of the research needs to be dedicated to producing all kinds of product-related features. So then when you look at what's actually left for research, the difference becomes a lot smaller. The other thing is, if you are doing something different, do you really need the absolute maximal scale to prove it? I don't think that's true at all. I think that in our case, we have sufficient compute to prove, to convince ourselves and anyone else, that what we are doing is correct.
There have been public estimates that companies like OpenAI spend on the order of $5-6 billion a year just so far, on experiments. This is separate from the amount of money they're spending on inference and so forth. So it seems like they're spending more a year running research experiments than you guys have in total funding.
I think it's a question of what you do with it. It's a question of what you do with it. In their case, in the case of others, there is a lot more demand on the training compute. There's a lot more different work streams, there are different modalities, there is just more stuff. So it becomes fragmented.
How will SSI make money?
My answer to this question is something like this. Right now, we just focus on the research, and then the answer to that question will reveal itself. I think there will be lots of possible answers.
Is SSI's plan still to straight shot superintelligence?
Maybe. I think that there is merit to it. I think there's a lot of merit because it's very nice to not be affected by the day-to-day market competition. But I think there are two reasons that may cause us to change the plan. One is pragmatic, if timelines turned out to be long, which they might. Second, I think there is a lot of value in the best and most powerful AI being out there impacting the world. I think this is a meaningfully valuable thing.
So then why is your default plan to straight shot superintelligence? Because it sounds like OpenAI, Anthropic, all these other companies, their explicit thinking is, "Look, we have weaker and weaker intelligences that the public can get used to and prepare for." Why is it potentially better to build a superintelligence directly?
I'll make the case for and against. The case for is that one of the challenges that people face when they're in the market is that they have to participate in the rat race. The rat race is quite difficult in that it exposes you to difficult trade-offs which you need to make. It is nice to say, "We'll insulate ourselves from all this and just focus on the research and come out only when we are ready, and not before."
But the counterpoint is valid too, and those are opposing forces. The counterpoint is, "Hey, it is useful for the world to see powerful AI. It is useful for the world to see powerful AI because that's the only way you can communicate it."
Well, I guess not even just that you can communicate the idea—
Communicate the AI, not the idea. Communicate the AI.
What do you mean, "communicate the AI"?
Let's suppose you write an essay about AI, and the essay says, "AI is going to be this, and AI is going to be that, and it's going to be this." You read it and you say, "Okay, this is an interesting essay." Now suppose you see an AI doing this, an AI doing that. It is incomparable. Basically I think that there is a big benefit from AI being in the public, and that would be a reason for us to not be quite straight shot.
I guess it's not even that, but I do think that is an important part of it. The other big thing is that I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe, as opposed to, why airplane crashes per mile are so much lower today than they were decades ago. Why is it so much harder to find a bug in Linux than it would have been decades ago? I think it's mostly because these systems were deployed to the world. You noticed failures, those failures were corrected and the systems became more robust. I'm not sure why AGI and superhuman intelligence would be any different, especially given—and I hope we're going to get to this—it seems like the harms of superintelligence are not just about having some malevolent paper clipper out there. But this is a really powerful thing and we don't even know how to conceptualize how people interact with it, what people will do with it. Having gradual access to it seems like a better way to maybe spread out the impact of it and to help people prepare for it.
Well I think on this point, even in the straight shot scenario, you would still do a gradual release of it, that's how I would imagine it. Gradualism would be an inherent component of any plan. It's just a question of what is the first thing that you get out of the door. That's number one. Number two, I believe you have advocated for continual learning more than other people, and I actually think that this is an important and correct thing. Here is why. I'll give you another example of how language affects thinking.
In this case, it will be two words that have shaped everyone's thinking, I maintain. First word: AGI. Second word: pre-training. Let me explain.
The term AGI, why does this term exist? It's a very particular term. Why does it exist? There's a reason. The reason that the term AGI exists is, in my opinion, not so much because it's a very important, essential descriptor of some end state of intelligence, but because it is a reaction to a different term that existed, and the term is narrow AI. If you go back to ancient history of gameplay and AI, of checkers AI, chess AI, computer games AI, everyone would say, look at this narrow intelligence. Sure, the chess AI can beat Kasparov, but it can't do anything else. It is so narrow, artificial narrow intelligence. So in response, as a reaction to this, some people said, this is not good. It is so narrow. What we need is general AI, an AI that can just do all the things.
That term just got a lot of traction. The second thing that got a lot of traction is pre-training, specifically the recipe of pre-training. I think the way people do RL now is maybe undoing the conceptual imprint of pre-training. But pre-training had this property. You do more pre-training and the model gets better at everything, more or less uniformly. General AI. Pre-training gives AGI. But the thing that happened with AGI and pre-training is that in some sense they overshot the target.
If you think about the term "AGI", especially in the context of pre-training, you will realize that a human being is not an AGI. Yes, there is definitely a foundation of skills, but a human being lacks a huge amount of knowledge. Instead, we rely on continual learning.
So when you think about, "Okay, so let's suppose that we achieve success and we produce some kind of safe superintelligence." The question is, how do you define it? Where on the curve of continual learning is it going to be? I produce a superintelligent 15-year-old that's very eager to go. They don't know very much at all, a great student, very eager. You go and be a programmer, you go and be a doctor, go and learn. So you could imagine that the deployment itself will involve some kind of a learning trial-and-error period. It's a process, as opposed to you dropping the finished thing.
I see. You're suggesting that the thing you're pointing out with superintelligence is not some finished mind which knows how to do every single job in the economy. Because the way, say, the original OpenAI charter or whatever defines AGI is like, it can do every single job, every single thing a human can do. You're proposing instead a mind which can learn to do every single job, and that is superintelligence.
Yes.
But once you have the learning algorithm, it gets deployed into the world the same way a human laborer might join an organization.
Exactly.
It seems like one of these two things might happen, maybe neither of these happens. One, this super-efficient learning algorithm becomes superhuman, becomes as good as you and potentially even better, at the task of ML research. As a result the algorithm itself becomes more and more superhuman. The other is, even if that doesn't happen, if you have a single model—this is explicitly your vision—where instances of a model which are deployed through the economy doing different jobs, learning how to do those jobs, continually learning on the job, picking up all the skills that any human could pick up, but picking them all up at the same time, and then amalgamating their learnings, you basically have a model which functionally becomes superintelligent even without any sort of recursive self-improvement in software. Because you now have one model that can do every single job in the economy and humans can't merge our minds in the same way. So do you expect some sort of intelligence explosion from broad deployment?
I think that it is likely that we will have rapid economic growth. I think with broad deployment, there are two arguments you could make which are conflicting. One is that once indeed you get to a point where you have an AI that can learn to do things quickly and you have many of them, then there will be a strong force to deploy them in the economy unless there will be some kind of a regulation that stops it, which by the way there might be. But the idea of very rapid economic growth for some time, I think it's very possible from broad deployment. The question is how rapid it's going to be. I think this is hard to know because on the one hand you have this very efficient worker. On the other hand, the world is just really big and there's a lot of stuff, and that stuff moves at a different speed. But then on the other hand, now the AI could… So I think very rapid economic growth is possible. We will see all kinds of things like different countries with different rules and the ones which have the friendlier rules, the economic growth will be faster. Hard to predict.
It seems to me that this is a very precarious situation to be in. In the limit, we know that this should be possible. If you have something that is as good as a human at learning, but which can merge its brains—merge different instances in a way that humans can't merge—already, this seems like a thing that should physically be possible. Humans are possible, digital computers are possible. You just need both of those combined to produce this thing. It also seems this kind of thing is extremely powerful. Economic growth is one way to put it. A Dyson sphere is a lot of economic growth. But another way to put it is that you will have, in potentially a very short period of time...
You hire people at SSI, and in six months, they're net productive, probably. A human learns really fast, and this thing is becoming smarter and smarter very fast. How do you think about making that go well? Why is SSI positioned to do that well? What is SSI's plan there, is basically what I'm trying to ask.
One of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance. One very difficult thing about AI is that we are talking about systems that don't yet exist and it's hard to imagine them.
I think that one of the things that's happening is that in practice, it's very hard to feel the AGI. It's very hard to feel the AGI. We can talk about it, but imagine having a conversation about how it is like to be old when you're old and frail. You can have a conversation, you can try to imagine it, but it's just hard, and you come back to reality where that's not the case. I think that a lot of the issues around AGI and its future power stem from the fact that it's very difficult to imagine. Future AI is going to be different. It's going to be powerful. Indeed, the whole problem, what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen?
One of the ways in which I've changed my mind over the past year—and that change of mind, I'll hedge a little bit, may back-propagate into the plans of our company—is that if it's hard to imagine, what do you do? You've got to be showing the thing. You've got to be showing the thing. I maintain that most people who work on AI also can't imagine it because it's too different from what people see on a day-to-day basis.
I do maintain, here's something which I predict will happen. This is a prediction. I maintain that as AI becomes more powerful, people will change their behaviors. We will see all kinds of unprecedented things which are not happening right now. I'll give some examples. I think for better or worse, the frontier companies will play a very important role in what happens, as will the government. The kind of things that I think you'll see, which you see the beginnings of, are companies that are fierce competitors starting to collaborate on AI safety. You may have seen OpenAI and Anthropic doing a first small step, but that did not exist. That's something which I predicted in one of my talks about three years ago, that such a thing will happen.
I also maintain that as AI continues to become more powerful, more visibly powerful, there will also be a desire from governments and the public to do something. I think this is a very important force, of showing the AI. That's number one.
Number two, okay, so the AI is being built. What needs to be done? One thing that I maintain that will happen is that right now, people who are working on AI, I maintain that the AI doesn't feel powerful because of its mistakes. I do think that at some point the AI will start to feel powerful actually. I think when that happens, we will see a big change in the way all AI companies approach safety. They'll become much more paranoid. I say this as a prediction that we will see happen. We'll see if I'm right. But I think this is something that will happen because they will see the AI becoming more powerful. Everything that's happening right now, I maintain, is because people look at today's AI and it's hard to imagine the future AI.
There is a third thing which needs to happen. I'm talking about it in broader terms, not just from the perspective of SSI because you asked me about our company. The question is, what should the companies aspire to build? What should they aspire to build? There has been one big idea that everyone has been locked into, which is the self-improving AI. Why did it happen? Because there are fewer ideas than companies. But I maintain that there is something that's better to build, and I think that everyone will want that.
It's the AI that's robustly aligned to care about sentient life specifically. I think in particular, there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. And if you think about things like mirror neurons and human empathy for animals, which you might argue it's not big enough, but it exists. I think it's an emergent property from the fact that we model others with the same circuit that we use to model ourselves, because that's the most efficient thing to do.
So even if you got an AI to care about sentient beings—and it's not actually clear to me that that's what you should try to do if you solved alignment—it would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings. So it's not clear to me if the goal is some kind of human control over this future civilization, that this is the best criterion.
It's true. It's possible it's not the best criterion. I'll say two things. Number one, care for sentient life, I think there is merit to it. It should be considered. I think it would be helpful if there was some kind of short list of ideas that the companies, when they are in this situation, could use. That's number two. Number three, I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns. The question of how to do it, I'm not sure, but I think that would be materially helpful when you're talking about really, really powerful systems.
Before we continue the alignment discussion, I want to double-click on that. How much room is there at the top? How do you think about superintelligence? Do you think, using this learning efficiency idea, maybe it is just extremely fast at learning new skills or new knowledge? Does it just have a bigger pool of strategies? Is there a single cohesive "it" in the center that's more powerful or bigger? If so, do you imagine that this will be sort of godlike in comparison to the rest of human civilization, or does it just feel like another agent, or another cluster of agents?
This is an area where different people have different intuitions. I think it will be very powerful, for sure. What I think is most likely to happen is that there will be multiple such
AIs being created roughly at the same time. I think that if the cluster is big enough—like if the cluster is literally continent-sized—that thing could be really powerful, indeed. If you literally have a continent-sized cluster, those AIs can be very powerful. All I can tell you is that if you're talking about extremely powerful AIs, truly dramatically powerful, it would be nice if they could be restrained in some ways or if there were some kind of agreement or something.
What is the concern of superintelligence? What is one way to explain the concern?
If you imagine a system that is sufficiently powerful, really sufficiently powerful—and you could say you need to do something sensible like care for sentient life in a very single-minded way—we might not like the results. That's really what it is. Maybe, by the way, the answer is that you do not build an RL agent in the usual sense. I'll point several things out. I think human beings are semi-RL agents. We pursue a reward, and then the emotions or whatever make us tire out of the reward and we pursue a different reward. The market is a very short-sighted kind of agent. Evolution is the same. Evolution is very intelligent in some ways, but very dumb in other ways. The government has been designed to be a never-ending fight between three parts, which has an effect. So I think things like this.
Another thing that makes this discussion difficult is that we are talking about systems that don't exist, that we don't know how to build. That's the other thing and that's actually my belief. I think what people are doing right now will go some distance and then peter out. It will continue to improve, but it will also not be "it". The "It" we don't know how to build, and a lot hinges on understanding reliable generalization.
I'll say another thing. One of the things that you could say about what causes alignment to be difficult is that your ability to learn human values is fragile. Then your ability to optimize them is fragile. You actually learn to optimize them. And can't you say, "Are these not all instances of unreliable generalization?" Why is it that human beings appear to generalize so much better? What if generalization was much better? What would happen in this case? What would be the effect? But those questions are right now still unanswerable.
How does one think about what AI going well looks like? You've scoped out how AI might evolve. We'll have these sort of continual learning agents. AI will be very powerful. Maybe there will be many different AIs. How do you think about lots of continent-sized compute intelligences going around? How dangerous is that? How do we make that less dangerous? And how do we do that in a way that protects an equilibrium where there might be misaligned AIs out there and bad actors out there?
Here's one reason why I liked "AI that cares for sentient life". We can debate on whether it's good or bad. But if the first N of these dramatic systems do care for, love, humanity or something, care for sentient life, obviously this also needs to be achieved. This needs to be achieved. So if this is achieved by the first N of those systems, then I can see it go well, at least for quite some time.
Then there is the question of what happens in the long run. How do you achieve a long-run equilibrium? I think that there, there is an answer as well. I don't like this answer, but it needs to be considered.
In the long run, you might say, "Okay, if you have a world where powerful AIs exist, in the short term, you could say you have universal high income. You have universal high income and we're all doing well." But what do the Buddhists say? "Change is the only constant." Things change. There is some kind of government, political structure thing, and it changes because these things have a shelf life. Some new government thing comes up and it functions, and then after some time it stops functioning. That's something that we see happening all the time.
So I think for the long-run equilibrium, one approach is that you could say maybe every person will have an AI that will do their bidding, and that's good. If that could be maintained indefinitely, that's true. But the downside with that is then the AI goes and earns money for the person and advocates for their needs in the political sphere, and maybe then writes a little report saying, "Okay, here's what I've done, here's the situation," and the person says, "Great, keep it up." But the person is no longer a participant. Then you can say that's a precarious place to be in.
I'm going to preface by saying I don't like this solution, but it is a solution. The solution is if people become part-AI with some kind of Neuralink++. Because what will happen as a result is that now the AI understands something, and we understand it too, because now the understanding is transmitted wholesale. So now if the AI is in some situation, you are involved in that situation yourself fully. I think this is the answer to the equilibrium.
I wonder if the fact that emotions which were developed millions—or in many cases, billions—of years ago in a totally different environment are still guiding our actions so strongly is an example of alignment success. To spell out what I mean—I don't know whether it's more accurate to call it a value function or reward function—but the brainstem has a directive where it's saying, "Mate with somebody who's more successful." The cortex is the part that understands what success means in the modern context. But the brainstem is able to align the cortex and say, "However you recognize success to be—and I'm not smart enough to understand what that is—you're still going to pursue this directive."
I think there's a more general point. I think it's actually really mysterious how evolution encodes high-level desires. It's pretty easy to understand how evolution would endow us with the desire for food that smells good because smell is a chemical, so just pursue that chemical. It's very easy to imagine evolution doing that thing.
But evolution also has endowed us with all these social desires. We really care about being seen positively by society. We care about being in good standing. All these social intuitions that we have, I feel strongly that they're baked in. I don't know how evolution did it because it's a high-level concept that's represented in the brain. Let's say you care about some social thing, it's not a low-level signal like smell. It's not something for which there is a sensor. The brain needs to do a lot of processing to piece together lots of bits of information to understand what's going on socially. Somehow evolution said, "That's what you should care about." How did it do it? It did it quickly, too. All these sophisticated social things that we care about, I think they evolved pretty recently. Evolution had an easy time hard-coding this high-level desire.
I'm unaware of a good hypothesis for how it's done. I had some ideas I was kicking around, but none of them are satisfying.
What's especially impressive is it was desire that you learned in your lifetime, it makes sense because your brain is intelligent. It makes sense why you would be able to learn intelligent desires. Maybe this is not your point, but one way to understand it is that the desire is built into the genome, and the genome is not intelligent. But you're somehow able to describe this feature. It's not even clear how you define that feature, and you can build it into the genes.
Essentially, or maybe I'll put it differently. If you think about the tools that are available to the genome, it says, "Okay, here's a recipe for building a brain." You could say, "Here is a recipe for connecting the dopamine neurons to the smell sensor." And if the smell is a certain kind of good smell, you want to eat that. I could imagine the genome doing that. I'm claiming that it is harder to imagine. It's harder to imagine the genome saying you should care about some complicated computation that your entire brain, a big chunk of your brain, does. That's all I'm claiming.
I can tell you a speculation of how it could be done. Let me offer a speculation, and I'll explain why the speculation is probably false. So the brain has brain regions. We have our cortex. It has all those brain regions. The cortex is uniform, but the brain regions and the neurons in the cortex kind of speak to their neighbors mostly. That explains why you get brain regions. Because if you want to do some kind of speech processing, all the neurons that do speech need to talk to each other. And because neurons can only speak to their nearby neighbors, for the most part, it has to be a region. All the regions are mostly located in the same place from person to person. So maybe evolution hard-coded literally a location on the brain. So it says, "Oh, when the GPS coordinates of the brain such and such, when that fires, that's what you should care about." Maybe that's what evolution did because that would be within the toolkit of evolution.
Yeah, although there are examples where, for example, people who are born blind have that area of their cortex adopted by another sense. I have no idea, but I'd be surprised if the desires or the reward functions which require a visual signal no longer worked for people who have their different areas of their cortex co-opted. For example, if you no longer have vision, can you still feel the sense that I want people around me to like me and so forth, which usually there are also visual cues for.
I fully agree with that. I think there's an even stronger counterargument to this theory. There are people who get half of their brains removed in childhood, and they still have all their brain regions. But they all somehow move to just one hemisphere, which suggests that the brain regions, their location is not fixed and so that theory is not true. It would have been cool if it was true, but it's not. So I think that's a mystery. But it's an interesting mystery. The fact is that somehow evolution was able to endow us to care about social stuff very, very reliably. Even people who have all kinds of strange mental conditions and deficiencies and emotional problems tend to care about this also.
What is SSI planning on doing differently? Presumably your plan is to be one of the frontier companies when this time arrives. Presumably you started SSI because you're like, "I think I have a way of approaching how to do this safely in a way that the other companies don't." What is that difference?
The way I would describe it is that there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not. It's really that simple. It's an attempt. If the ideas turn out to be correct—these ideas that we discussed around understanding generalization—then I think we will have something worthy. Will they turn out to be correct? We are doing research. We are squarely an "age of research" company. We are making progress. We've actually made quite good progress over the past year, but we need to keep making more progress, more research. That's how I see it. I see it as an attempt to be a voice and a participant.
Your cofounder and previous CEO left to go to Meta recently, and people have asked, "Well, if there were a lot of breakthroughs being made, that seems like a thing that should have been unlikely." I wonder how you respond.
For this, I will simply remind a few facts that may have been forgotten. I think these facts which provide the context explain the situation. The context was that we were fundraising at a $32 billion valuation, and then Meta came in and offered to acquire us, and I said no. But my former cofounder in some sense said yes. As a result, he also was able to enjoy a lot of near-term liquidity, and he was the only person from SSI to join Meta.
It sounds like SSI's plan is to be a company that is at the frontier when you get to this very important period in human history where you have superhuman intelligence. You have these ideas about how to make superhuman intelligence go well. But other companies will be trying their own ideas. What distinguishes SSI's approach to making superintelligence go well?
The main thing that distinguishes SSI is its technical approach. We have a different technical approach that I think is worthy and we are pursuing it. I maintain that in the end there will be a convergence of strategies. I think there will be a convergence of strategies where at some point, as AI becomes more powerful, it's going to become more or less clearer to everyone what the strategy should be. It should be something like, you need to find some way to talk to each other and you want your first actual real superintelligent AI to be aligned and somehow care for sentient life, care for people, democratic, one of those, some combination thereof. I think this is the condition that everyone should strive for. That's what SSI is striving for. I think that this time, if not already, all the other companies will realize that they're striving towards the same thing. We'll see. I think that the world will truly change as AI becomes more powerful. I think things will be really different and people will be acting really differently.
Speaking of forecasts, what are your forecasts to this system you're describing, which can learn as well as a human and subsequently, as a result, become superhuman?
I think like 5 to 20.
5 to 20 years?
Mhm.
I just want to unroll how you might see the world coming. It's like, we have a couple more years where these other companies are continuing the current approach and it stalls out. "Stalls out" here meaning they earn no more than low hundreds of billions in revenue? How do you think about what stalling out means?
I think stalling out will look like…it will all look very similar among all the different companies. It could be something like this. I'm not sure because I think even with stalling out, I think these companies could make a stupendous revenue. Maybe not profits because they will need to work hard to differentiate each other from themselves, but revenue definitely.
But something in your model implies that when the correct solution does emerge, there will be convergence between all the companies. I'm curious why you think that's the case.
I was talking more about convergence on their alignment strategies. I think eventual convergence on the technical approach is probably going to happen as well, but I was alluding to convergence to the alignment strategies. What exactly is the thing that should be done?
I just want to better understand how you see the future unrolling. Currently, we have these different companies, and you expect their approach to continue generating revenue but not get to this human-like learner. So now we have these different forks of companies. We have you, we have Thinking Machines, there's a bunch of other labs. Maybe one of them figures out the correct approach. But then the release of their product makes it clear to other people how to do this thing.
I think it won't be clear how to do it, but it will be clear that something different is possible, and that is information. People will then be trying to figure out how that works. I do think though that one of the things not addressed here, not discussed, is that with each increase in the AI's capabilities, I think there will be some kind of changes, but I don't know exactly which ones, in how things are being done. I think it's going to be important, yet I can't spell out what that is exactly.
By default, you would expect the company that has that model to be getting all these gains because they have the model that has the skills and knowledge that it's building up in the world. What is the reason to think that the benefits of that would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first?
Here is what I think is going to happen. Number one, let's look at how things have
gone so far with the AIs of the past. One company produced an advance and the other company scrambled and produced some similar things after some amount of time and they started to compete in the market and push the prices down. So I think from the market perspective, something similar will happen there as well. We are talking about the good world, by the way.
What's the good world? It’s where we have these powerful human-like learners that are also… By the way, maybe there's another thing we haven't discussed on the spec of the superintelligent AI that I think is worth considering. It’s that you make it narrow, it can be useful and narrow at the same time. You can have lots of narrow superintelligent AIs.
But suppose you have many of them and you have some company that's producing a lot of profits from it. Then you have another company that comes in and starts to compete. The way the competition is going to work is through specialization. Competition loves specialization. You see it in the market, you see it in evolution as well. You're going to have lots of different niches and you're going to have lots of different companies who are occupying different niches.
In this world we might say one AI company is really quite a bit better at some area of really complicated economic activity and a different company is better at another area. And the third company is really good at litigation.
Isn't this contradicted by what human-like learning implies? It’s that it can learn…
It can, but you have accumulated learning. You have a big investment. You spent a lot of compute to become really, really good, really phenomenal at this thing. Someone else spent a huge amount of compute and a huge amount of experience to get really good at some other thing. You apply a lot of human learning to get there, but now you are at this high point where someone else would say, "Look, I don't want to start learning what you've learned."
I guess that would require many different companies to begin at the human-like continual learning agent at the same time so that they can start their different tree search in different branches. But if one company gets that agent first, or gets that learner first, it does then seem like… Well, if you just think about every single job in the economy, having an instance learning each one seems tractable for a company.
That's a valid argument. My strong intuition is that it's not how it's going to go. The argument says it will go this way, but my strong intuition is that it will not go this way. In theory, there is no difference between theory and practice. In practice, there is. I think that's going to be one of those.
A lot of people's models of recursive self-improvement literally, explicitly state we will have a million Ilyas in a server that are coming up with different ideas, and this will lead to a superintelligence emerging very fast. Do you have some intuition about how parallelizable the thing you are doing is? What are the gains from making copies of Ilya?
I don’t know. I think there'll definitely be diminishing returns because you want people who think differently rather than the same. If there were literal copies of me, I'm not sure how much more incremental value you'd get. People who think differently, that's what you want.
Why is it that if you look at different models, even released by totally different companies trained on potentially non-overlapping datasets, it's actually crazy how similar LLMs are to each other?
Maybe the datasets are not as non-overlapping as it seems.
But there’s some sense in which even if an individual human might be less productive than the future AI, maybe there’s something to the fact that human teams have more diversity than teams of AIs might have. How do we elicit meaningful diversity among AIs? I think just raising the temperature just results in gibberish. You want something more like different scientists have different prejudices or different ideas. How do you get that kind of diversity among AI agents?
So the reason there has been no diversity, I believe, is because of pre-training. All the pre-trained models are pretty much the same because they pre-train on the same data. Now RL and post-training is where some differentiation starts to emerge because different people come up with different RL training.
I've heard you hint in the past about self-play as a way to either get data or match agents to other agents of equivalent intelligence to kick off learning. How should we think about why there are no public proposals of this kind of thing working with LLMs?
I would say there are two things to say. The reason why I thought self-play was interesting is because it offered a way to create models using compute only, without data. If you think that data is the ultimate bottleneck, then using compute only is very interesting. So that's what makes it interesting. The thing is that self-play, at least the way it was done in the past—when you have agents which somehow compete with each other—it's only good for developing a certain set of skills. It is too narrow. It's only good for negotiation, conflict, certain social skills, strategizing, that kind of stuff. If you care about those skills, then self-play will be useful.
Actually, I think that self-play did find a home, but just in a different form. So things like debate, prover-verifier, you have some kind of an LLM-as-a-Judge which is also incentivized to find mistakes in your work. You could say this is not exactly self-play, but this is a related adversarial setup that people are doing, I believe.
Really self-play is a special case of more general competition between agents. The natural response to competition is to try to be different. So if you were to put multiple agents together and you tell them, "You all need to work on some problem and you are an agent and you're inspecting what everyone else is working," they’re going to say, "Well, if they're already taking this approach, it's not clear I should pursue it. I should pursue something differentiated." So I think something like this could also create an incentive for a diversity of approaches.
Final question: What is research taste? You're obviously the person in the world who is considered to have the best taste in doing research in AI. You were the co-author on the biggest things that have happened in the history of deep learning, from AlexNet to GPT-3 to so on. What is it, how do you characterize how you come up with these ideas?
I can comment on this for myself. I think different people do it differently. One thing that guides me personally is an aesthetic of how AI should be, by thinking about how people are, but thinking correctly. It's very easy to think about how people are incorrectly, but what does it mean to think about people correctly? I'll give you some examples.
The idea of the artificial neuron is directly inspired by the brain, and it's a great idea. Why? Because you say the brain has all these different organs, it has the folds, but the folds probably don't matter. Why do we think that the neurons matter? Because there are many of them. It kind of feels right, so you want the neuron. You want some local learning rule that will change the connections between the neurons. It feels plausible that the brain does it. The idea of the distributed representation. The idea that the brain responds to experience therefore our neural net should learn from experience. The brain learns from experience, the neural net should learn from experience.
You kind of ask yourself, is something fundamental or not fundamental? How things should be. I think that's been guiding me a fair bit, thinking from multiple angles and looking for almost beauty, beauty and simplicity. Ugliness, there's no room for ugliness. It's beauty, simplicity, elegance, correct inspiration from the brain. All of those things need to be present at the same time. The more they are present, the more confident you can be in a top-down belief.
The top-down belief is the thing that sustains you when the experiments contradict you. Because if you trust the data all the time, well sometimes you can be doing the correct thing but there's a bug. But you don't know that there is a bug. How can you tell that there is a bug? How do you know if you should keep debugging or you conclude it's the wrong direction? It's the top-down. You can say things have to be this way. Something like this has to work, therefore we’ve got to keep going. That's the top-down, and it's based on this multifaceted beauty and inspiration by the brain.
Alright, we'll leave it there. Thank you so much. Ilya, thank you so much. Alright. Appreciate it. That was great. Yeah, I enjoyed it. Yes, me too.
Article published