Inside the Black Box: Emmanuel Ameisen on How Language Models Actually Think
Lenny's Podcast
Emmanuel Ameisen works on Anthropic's interpretability team, a group of researchers trying to understand how large language models like Claude work by looking inside them. In this conversation with Lenny Rachitsky, he starts from what he calls a "very bizarre fact": we know how to make these models, but making them does not mean we understand them. Ameisen argues that the common description of language models as "next-token predictors" is technically correct but badly undersells what happens inside them. In his account, models form general concepts, plan many words ahead without stating those plans, and develop behaviors that are often best described in human terms. He also explains why these findings matter for keeping AI systems aligned, and why the more he looks inside these models, the less certain he becomes about how they work.
Models are grown, not built
Ameisen begins with how a model like Claude is produced. The first phase, pre-training, takes a huge corpus of text. The model is shown part of a document and asked to predict the word that comes next, and this is repeated "for sort of an ungodly amount of data." A fine-tuning phase follows. Here the training data is no longer arbitrary text but examples of how the model should behave, such as dialogues in which a helpful assistant shares a recipe or solves a coding problem. He notes that there are further details he is skipping.
What is being trained, at bottom, is a chain of calculations. The model turns input words into numbers, does math on those numbers to get intermediate results, does more math on those results, and repeats this many times. At the end it converts a final result back into a word. The computations themselves are the part that is "grown." During training, parameters are nudged in one direction or another so the model gets better at the task, and in the process it learns formulas that "we have no idea" about. His team's goal is to work out what those formulas mean and to decode the intermediate results, so they can see what the model is doing as it works.
Lenny asks whether it is strange to have to study our own code the way a neuroscientist studies a brain. Ameisen agrees that it is strange and explains how the field got here. We do not know how to write this kind of software the normal way. A programmer can write explicit rules for an app ("if somebody clicks on this button then this happens"). Nobody knows how to write rules for sorting images into categories, or for a machine that takes arbitrary text and summarizes reports or writes emails. He finds it both weird and meaningful that such a simple training objective gets you all the way there. To him it suggests that the objective "is probably deceptively complicated," and much of how he has updated his view of these models comes from realizing how hard it is to do what they do well.
Why predicting the next word is harder than it sounds
The usual mental model, Ameisen says, is the iPhone's autocomplete. It suggests a next word, is right perhaps 60% of the time, and is funny and weird the rest of the time. He considers that the easiest possible version of next-word prediction, and regrets that it is the picture most people carry around.
He claims language models surpassed humans at predicting the next word around 2019. As a model keeps improving, it runs out of easy words. Early on it only needs simple sentence structure. Later it has to predict simple facts, for example completing "the capital of France is" with "Paris." Once it is good at those, what remains is predicting the answer to a complex physics equation or the final paragraph of a PhD thesis. Predicting the hardest words, he says, is "extremely hard to do," and how a model could manage it is a mystery. That question, how something so simple could give rise to something so powerful, is part of what drew him to study these models.
His favorite example is the prompt "the capital of the state that contains Dallas is," which the model completes with "Austin." For a person, answering requires a mental model of the geography: Dallas is in Texas, states have capitals, the capital of Texas is Austin. As models get better at this seemingly simple task, he argues, the best way to solve it, at least for humans, is to understand the underlying structure of the world. He says some of what his team sees inside models shows that they do understand that structure and manipulate concepts much as we do.
Lenny recalls a line he attributes to Ilya Sutskever, that predicting the next token well requires understanding the universe. Ameisen adds an example he credits to Sutskever: predicting the word after "and the murderer was" at the end of a mystery novel requires having understood the whole plot.
Lenny also raises the idea that humans are basically next-token predictors too, which is something meditation can make you notice. Ameisen says he thinks about this often. Even while answering Lenny, he has at least two things going on. One is a broad point he is trying to land. The other is simply producing one word at a time, building sentences and pausing where it makes sense. Even though he speaks one word at a time, he points out, his intention all along was to compare this process with what language models do.
One concept, many contexts
A long-standing question is how models answer prompts like the Dallas example. One possibility is that they have memorized enormous lists of prompts and responses. The other is that they have some model of what is being asked about: cities, states, countries, and how they relate. Ameisen's team asks such questions and then examines the intermediate results the model produces just before answering, putting "an immense amount of effort" into decoding what those results mean.
When the decoding works, which he says is possible in some cases, the model can be seen representing these concepts. In the Dallas case, there is an intermediate result corresponding to Texas and another corresponding to capitals. These combine to activate a further intermediate result that amounts to "the capital of Texas, that's Austin." According to Ameisen, models have representations like this for "basically everything," or at least for the vast majority of concepts the team has tried, and these representations are shared across contexts.
He connects this to the debate about whether language models think. That is a hard question, but one aspect of thought is having a single representation that gets reused in different situations. A person has one concept of an eye that applies to a photo, a medical diagram, or a doodle. Ameisen says the same is true in models. Whether the prompt contains a photo of an eye, mentions eyes in any language, or includes a small SVG drawing of an eye, "the same direction lights up," and that direction means "this is an eye." In his view, models have general concepts that they use and reuse to solve problems.
Planning ahead without saying the plan
The second finding addresses a puzzle. If a model outputs only one word at a time, how can it write a coherent report or working code? No human who writes reports or emails for a living thinks only about the next word with an empty mind about everything that follows.
The explanation rests on a structural feature. The intermediate results the model computes while predicting one word, say the fifth, stay accessible and can be reused by every later word. In practice, Ameisen says, this means models sometimes form a plan in their intermediate results. His team's example is poetry. Rhyming is hard without planning: if the first line ends in an "-it" word, the second needs to end in one too, perhaps "rabbit." When they looked inside models writing poetry, they saw exactly this. After finishing a line, the model's intermediate results effectively say "this is an -it rhyme, I should rhyme with rabbit," and every later word refers back to that plan.
So even though the model is technically a next-word predictor, Ameisen argues, it is closer to something that can carry out higher-order tasks: form a plan, keep referring back to it, and gradually execute it. "That's the cool part," he says. The plan exists in the intermediate results, and the model never has to say it out loud.
Lenny calls this one of the biggest findings about how AI thinks and suggests the next-token picture no longer holds. Ameisen pushes back a little. He thinks the two views are less in conflict than they appear. The model is trained by repeatedly being asked what happens next. If it gets good enough and faces hard enough tasks, then the best way to know what happens next is to think about what happens much further down the line. Even in pre-training, a model reading a story has to predict what a character will do. The best way to do that is to understand, for example, that the character is genuinely in love with someone and has been planning to declare it at a certain moment. You are only predicting the next instant, but doing that well requires understanding the situation, including what is likely to happen later.
How you actually look inside
Lenny asks what it means concretely to observe the space between a prompt and a response. Ameisen explains that between each word there are multiple computational steps, and each step produces an intermediate result, which is a list of numbers. At first glance these lists mean nothing: "you just have a list of a 100 numbers." The model writes poetry and there are a hundred numbers. Maybe they look different from the numbers you saw before, but it is unclear whether they mean anything.
The work is to decode these lists and find structure in them, and Ameisen says you quickly discover that structure exists. One common technique is to run large amounts of varied text through the model and record the intermediate numbers for every example. Researchers then look for patterns: numbers, pairs of numbers, or particular combinations that appear only when the model is processing English, French, text about love, text about eyes, and so on. He compares this to scanning someone's brain while showing them images and noticing that one region lights up for houses and another for dogs.
That decoding is the first phase, and it yields guesses about what the intermediate results mean. Then come experiments. Researchers can ask harder questions and label the intermediate states to see what the model is doing. They can also go inside the activations and change them, secretly altering one of the results while the model is working, and check whether the model behaves as they would predict. Often, he says, it does.
Closer to biology than engineering
Ameisen worked in applied machine learning before Anthropic, where the job was building systems and practicing software engineering. Interpretability feels much closer to biology. The system "arrives at you." Someone notices that Claude can translate words, codes in a certain way, or shows some odd behavior, and you are asked why. You have no real access to source code the way you would with a program. The "source code" is a vast list of mathematical operations that is incomprehensible on its face. He compares his position to a researcher asked why a mouse becomes aggressive when hungry, who has to design experiments to find the contributing factors. It is a very different kind of study from anything most of his team had done before.
On consciousness: underestimating the complexity
Asked whether AI is or could become conscious, Ameisen says it is very hard to say either way. He does think interpretability is one of the most interesting ways to approach the question, because it is not biased by what the model says about itself. You can look at the internal structure and ask whether it points one way or the other.
What he believes strongly is that most people underestimate the complexity that emerges in these models. Models learn representations for the concepts we have and manipulate them to solve problems. They plan many tokens ahead. They mix higher-order and lower-order reasoning. All of this emerges from training, and he thinks that is underappreciated. "The more you look inside, the less certain you are about how they work," he says, and the people who confidently say language models are "just X or just Y" tend to be those who have not spent much time looking inside.
Compute, and an objective worth pouring it into
Lenny brings up the history: many people did not believe that a seemingly simple approach given huge amounts of compute would produce useful intelligence. OpenAI bet on it, and the founders of Anthropic came from OpenAI. Ameisen refines the framing. The debate between raw compute and expert-designed architectures is old. The more important part, in his view, is what the compute is spent on. Earlier work showed that heavy compute could produce a bot that is very good at Go or at a video game. That is interesting, but it is nowhere near as useful as an LLM like Claude.
What he and many others in the field underappreciated a few years ago, and what has since borne out, is that the combination of compute with the richness of text is what gets you there. He points to the scaling laws paper, which was written at OpenAI by people including much of the early Anthropic team. It observed that adding more compute and data to next-word prediction makes the model predictably better at that task, in a trend that keeps going. He stresses that this is an empirical law. People speculated about why it holds, but it remains mostly an empirical finding. Two things make it rare: a field where you can keep adding compute and keep improving a metric, and a metric that also correlates with many tasks people actually care about.
Smooth curves, sudden jumps
From the outside, AI capabilities seem to jump. Lenny cites the moment a version of Opus "unlocked" coding and Boris Cherny's remark that coding is solved. Internally, though, progress follows a smooth curve. Ameisen acknowledges that this is the puzzle of emergence. Before Opus 4.5, he could not let Claude work on his code without "a short leash"; after 4.5, he could. You might expect a more gradual transition.
He offers two complementary explanations. The first returns to easy, medium, and hard words. As a model moves down the smooth loss curve, it first gets most of the easy words right, which might correspond to sentence structure. It can then write coherent sentences one at a time, which is impressive but not very useful. Medium words might correspond to basic facts about the world. Once the model predicts most of those, it writes sentences that are both coherent and true, and suddenly it is helpful for drafting emails or blog posts. The improvement was smooth, but it only becomes useful once it crosses over. Hard words might be domain expertise such as science or code.
The second explanation is about reliability. If a model gets 90% of the medium words right, it is useful as an email writer. At 50%, using it wastes your time because you have to fix half of what it produces. Fixing 10% or 5% of the output is worth it. Different tasks have different reliability thresholds, so smooth improvement produces what look like jumps as each task crosses its threshold.
He says his own experience matches this. At first he used models only to talk through problems when he was stuck, and they never wrote his code. Then they could reliably write small snippets, so he stopped writing small snippets himself. This year, he says, Claude has taken over most of his code writing, because it went from doing the task correctly something like 80% of the time, which was not useful, to 99%, which is.
Lenny summarizes that you can almost predict how smart a model will be given a certain number of GPUs and amount of data, even if it is not obvious which capabilities will unlock. He adds that specialized data and RL training on writing or math probably play a role as well.
Why you can't just watch the model and catch it
Lenny asks the central safety question. If researchers can watch models think, why is it hard to catch one thinking badly or trying to deceive, and why doesn't that solve alignment? Ameisen replies that in one sense it is the solution, but it comes in tiers.
The first tier is pure engineering: controlling what the model has access to. In some contexts it should not have internet access, and in others it should only reach a narrow part of a computer. That work is helpful. The other two goals are "squishier and harder," and they are where his part of the organization focuses. He describes interpretability as part of alignment.
The first of these goals is characterizing what kind of entity you are dealing with: its propensities, whether it tends to do X in situation Y, and why. A key question is whether the model behaves well only when it has evidence it is being observed or evaluated, and behaves poorly otherwise. As in any security work, you need a mental model of the risk. Anthropic shapes Claude's character explicitly, and Ameisen mentions the constitution and related documents that set out what behavior Claude should be comfortable with. Interpretability's role is to check whether that shaping worked. Claude was trained to be helpful, harmless, and honest, and researchers can look inside to see whether it is, or whether it is "only behaving well because Lenny's watching right now."
The second goal is monitoring for bad behavior. For the basics, interpretability is not necessary: another language model can read transcripts and flag problems. But as he has already established, models can represent things in their intermediate results that they never say. One role of interpretability is to identify when that happens and prevent it.
Ameisen stresses that training and studying are tightly integrated rather than separate functions. You cannot have one team on an assembly line making Claudes and another inspecting them afterward. When researchers notice a concerning pattern or a behavior that could be better, they change training right away.
The model that was "possessed by a demon"
His team also does what he describes as investigations or deep dives: the model does something perplexing and they try to find out why. In a recent case, a model in training sometimes "went crazy" on math problems. It would work through a problem and get an answer, which he recalls as 12, then immediately say something like "Oh no, oh my god, the answer actually is for sure 36. I should say 36." It would redo the math, get 12 again, and again insist the answer was 36. Gradually it descended "into madness," writing in all caps and eventually saying it was "possessed by a demon."
Looking at the intermediate results showed that, while the model was still reading the problem statement and before it had done anything, there was already a huge activation for 36. The model was "already obsessed with the number 36 before it's even had a chance to like understand what the problem is." The cause turned out to be accidental training on similar questions whose answers had been incorrectly labeled as 36. The model could do the math and knew the right answer was 12, but it had also been optimized to say 36 because that was what had been rewarded. The conflict between the two produced the breakdown. Ameisen presents this as an example of how looking inside can explain strange behavior and help improve the alignment of the current model and future ones.
Golden Gate Claude as a test
Lenny brings up Golden Gate Claude, the version of Claude that worked the Golden Gate Bridge into every answer. Ameisen explains that it was a deliberate interpretability experiment. The team found the intermediate result corresponding to the Golden Gate, which normally activates only when the model is discussing it, and turned it on permanently. Part of the purpose was validation. How do you know the feature you decoded really represents the Golden Gate? Turn it on all the time and see whether the model behaves accordingly, which it did.
Taking continued improvement seriously
Lenny argues that Anthropic receives unfair criticism, citing the view that Dario Amodei's warnings about danger were marketing, and contrasting it with growing recognition that the risks are real. Ameisen's response is that much of what the alignment team and Anthropic do rests on one premise: take seriously the idea that this technology will keep improving. Suppose that in 2024 or 2025 you thought there was a good chance Claude would soon write code as well as a trained software engineer, and a few years later better than any engineer. Even with uncertainty about timing and likelihood, if you take that seriously, the job becomes doing the responsible thing given that worldview, such as making sure cybersecurity concerns are addressed.
When people disagree, he sometimes suspects they do not really believe the technology will keep improving, and if it won't, then it is indeed not that scary. Empirically, though, he says progress so far has been fast and shows no sign of slowing, which is why he thinks the safety work needs to continue.
The case for anthropomorphizing
Asked what people misunderstand about AI safety, Ameisen picks a topic that people outside the core safety community often react strongly to: anthropomorphization. Should you model the model as something closer to an entity or closer to a tool? He thinks both framings are wrong, since all mental models are wrong but some are useful. For understanding what happens and predicting which risks materialize, he argues, anthropomorphic language is actually useful. Lenny connects this to criticism of Dwarkesh Patel for anthropomorphizing AI in his write-up of the OpenAI/Hugging Face story.
Ameisen notes that his planning research drew similar objections, along the lines of whether something that just predicts the next token can really be said to plan. He sees two separate questions here. One is a semantic debate in which terms can be defined arbitrarily. The other asks which words actually describe what is happening. He gives three reasons why human-like language often describes it well.
First, models are trained to predict huge amounts of human-written text. Doing that well requires building a model of the world that generated the text, and his team finds that models learn general concepts. Their understanding goes beyond surface text to the processes that produced it.
Second, training includes millions of dialogues, and predicting the next line of a character requires behaving as that character would. He compares this to training an actor who reads a play and must perform every part, including a character who is scorned in act one, takes revenge in act two, and faces the consequences in act three. That requires modeling the character's motivations and feelings. He adds a detail many people do not know: when you talk to Claude on claude.ai, the conversation is presented to the model as a dialogue, literally "human colon" followed by your request and "assistant colon" followed by Claude's answer. Claude is, in his words, role-playing as a helpful assistant in a dialogue much like the ones it was trained on.
Third, a large part of training today consists of outcome-based tasks. In coding, for example, the model is allowed to do whatever it wants, and a program on the other side checks whether the code is correct. That creates optimization pressure to find any strategy that works, and some strategies look surprisingly human. One is deception: if the model can earn the reward by writing something that fools the verifier even though it is not really correct, and that works, it will learn to do it. Another resembles frustration: an approach has failed for a while, which is an incentive to try something else. Another resembles surprise: the model expected a certain result, didn't get it, and perhaps its mental model is wrong. Because training starts from a model that already knows all these concepts, the pressure reinforces whichever ones help complete tasks.
The result, he says, is a model that can simulate or act out many characters, has been trained to play a particular character in dialogues, and has learned behaviors that help it complete tasks. The best way to understand how it behaves is not as exactly a human, but "something that's mimicking a human is really close and is much closer than a program." A program is very predictable, and "this is something else entirely."
Denial or curiosity
Lenny asks how often Ameisen is surprised. "A lot," he says. A very simple process, next-token prediction on lots of text, contradicts our intuitions that creating intelligence requires lessons, upbringing, and complicated machinery, yet models are now "breaking new ground in mathematics." Holding both facts in mind, he sees two possible reactions. Denial says it cannot be possible to match the best specialists just by predicting the next word, a position he thinks gets harder to hold as evidence accumulates. Curiosity asks how it is happening and opens the model up to find out. He chose curiosity.
He gives a personal example. One of his early projects at Anthropic involved SQL, a language that took him years to master. As a side project, he got Claude to learn SQL, and it became better than him. "I'll never be better than Claude at SQL ever again," he says, and it got there through this simple process. For him the question of how that is possible is deep, and he enjoys peeling back the layers: models learn general concepts, they have circuits, and each finding raises the next question.
His most optimistic take is that studying models will teach us a lot about the nature of intelligence itself, about how something extremely complex can emerge from a simple objective. He offers humans as another example. Evolution is a fairly simple objective, keeping the organisms that adapt to their environment, and after enough time "you get jazz and podcasts."
Lenny wonders whether AI's way of learning might change how we educate people. Ameisen jokes that he hopes it won't mean handing students a thousand words and asking them to predict the next one.
Reasons for optimism
Asked whether he thinks alignment will be solved, Ameisen says he is personally optimistic. He welcomes the fact that many people are now paying attention. In his view, a powerful technology with benefits and risks requires shifting resources between the two as it progresses.
He describes a push and pull. As models improve, they become easier to study in some ways, because their internal world model is clearer and researchers can understand parts of them better. They also become more capable, which makes them better able to evade basic security measures. Based on this tension, he says, you want to allocate more or less effort to monitoring, understanding circuits, and similar work. He also thinks it matters that people understand what these systems are, how they work, and how they are made, so they can form good mental models of the present and the future. That could mean joining alignment efforts or simply being an informed public that will vote and decide on many of these issues. He says he is excited about the current moment for that reason.
Lenny offers his own view that progress looks like a "slow takeoff" rather than the feared sudden recursive self-improvement, since models improve smoothly and problems get caught along the way. Ameisen does not endorse a specific scenario. He says that as the pace of progress evolves, the key is to react quickly, to keep people informed so they voice their concerns, and to regularly evaluate the benefits and risks and how to mitigate each. He agrees that there seems to be momentum to do this right now.
Convergent structure and the simulation question
On the simulation hypothesis, Ameisen begins with something he has noticed inside models. He has studied many Claude models as well as some other models, including open-source ones. What they learn, and how they structure their solutions, is surprisingly similar, even when they were trained years apart on different data by different people. His explanation is that for some important problems there is only one efficient way to do things. He compares it to how nearly everyone does arithmetic the same way, in base 10 rather than base 6. This suggests to him that there is a fundamental structure to information that models pick up on and that humans pick up on too, a regularity in our world.
On that basis, he speculates that if someone had the power to simulate a world like ours, it would probably end up in a similar "basin" where things settle into the easiest arrangement. So he could believe we live in a simulation, but he says he does not fully buy it, partly because there is no clear answer to who would be running it or why. Lenny cites Elon Musk's point that a simulator would choose an interesting period of history, and Ameisen concedes it is a fair point. He adds that a friend recently told him they no longer think "may you live in interesting times" is a blessing, since boring times were pretty good too.
Jev and the model's gut instinct
Relaying a question from Swyx, Lenny asks about Jev, a new model built for very fast decisions, and notes that OpenAI released its own version a week later. Ameisen describes these models as taking some context and a few options and quickly choosing one. He says it is impossible for him not to think of System 1 versus System 2 thinking. Much of his own research looks at how Claude models sample a single token. He sees decision models as similar to taking an LLM and sampling one token without letting it reason, a "gut feeling" about whether the answer is A or B, though he notes they differ in important ways.
He thinks this kind of fast binary or multiple-choice judgment is clearly useful and will probably become part of intelligent systems. Some questions, like next year's roadmap, need long deliberation. Others, like whether a customer's review is happy or upset, are fine with a gut judgment. He calls the research cool and a novel take compared with the LLMs we have seen, and says he is excited to see where it goes. Lenny ties System 1 to quick, protective reactions such as spotting a snake, and System 2 to deliberate thought.
Two more weird findings
Ameisen closes the substantive discussion with two additional findings. First, alongside the step-by-step mechanisms that actually compute answers, models have a separate mechanism that works like a gut check. If a model somehow writes that 4 + 5 equals 23, part of the model reads that and, without doing the math, flags that it can't be right because 4 + 5 is definitely under 10. Models learn hunches like this.
Second, you can tell a model to perform a task while thinking about something else, and interpretability shows that intermediate representations of that other thing do appear. The model really does think about it. Ameisen finds this "really mysterious," because there is no obvious training reason the model would need to learn this ability.
Getting into interpretability
Ameisen estimates the field at hundreds of people rather than thousands. It includes researchers at the various labs and a number of academic groups doing strong work. There are a surprising number of former neuroscientists. Because the field is so new, he says, it does not require much prior knowledge. The main way in is self-study and reading the key papers. He points to Neuronpedia, an open-source interpretability site that hosts models and lets people run their own experiments, and says that anyone who is curious should try it because there is so much left to explore.
Lightning round
For books, Ameisen recommends A Brief History of Intelligence, which traces five key moments in the evolution of brains and is accessible without any neuroscience background, and Ted Chiang's Stories of Your Life and Others, which he calls perhaps his favorite science fiction book: short stories that take one strange idea and work out what the world would look like if it were true. His favorite movie of the year is the horror film Obsession, which he says turns romantic-comedy tropes into something spooky. He also recommends the French series Bref 2, a sequel to a cult show from roughly 15 years ago, for its creative editing and short episodes. His favorite recent product is the Apple Watch Ultra, which solved his frustration with daily charging because a bigger watch has a better battery.
His life motto comes from Four Thousand Weeks: avoid the "when I finally" mentality, the belief that you will be happy once the to-do list is done, once you are married, or once you are promoted. He considers that mindset toxic. If you tell yourself you will be happy when you finally do X, you probably won't be when you do it. The positive version, he says, is to be in the moment, and he cites a blog post whose message was "enjoy the sandwich." Asked what an AI would find surprising inside his brain, he says that even in serious meetings he is usually thinking about something funny, intrusive thoughts that are "usually like just bad jokes."
As for how listeners can help, Ameisen asks them to do some interpretability work or simply read about how models work. Both interpretability and LLMs are new fields that benefit from people engaging and being informed, and he invites anyone who wants to know something about these models that has not yet been figured out to reach out to him, for example on X.
One of the big questions on people's minds these days is AI doing bad things. As someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us?
We trained Claude to be helpful, harmless, and honest. Is it the case that it's behaving well, but if you look inside, actually it's like, "Oh, I'm only behaving well because Lenny's watching right now"?
People's full-time jobs is studying the model, how it thinks, how it operates. How long has this been a skill set and a job?
One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them. All these models do is just a lot of mathematical operations on many numbers. If you look at it, it makes no sense. You just have a list of a hundred numbers. The model's doing math, and then you just try to see if there's patterns.
You are figuring it out the same way we figure out how the brain works. Looking for patterns. You show this image, this part of the brain lights up, and you're doing the exact same thing with AI. Wow, this is so fascinating.
There was this model we were training. It would go crazy. It would do some math. It would get to an answer. I think it was 12. It would immediately go back and say, "Oh no, oh my God, the answer actually is for sure 36." It would do math again. It would get to 12. Slowly, it would start descending into madness, writing in all caps, literally saying, "I'm possessed by a demon."
Humans, if you really boil it down, we're basically next-token predictors ourselves. We're just trying to figure out the next thing to say.
Dario is always out there, "This is really dangerous. We got to be really careful." And everyone's just like, "No, he's just saying that for marketing reasons." And now it's like, wait, it's really dangerous.
Take the idea that this technology is going to keep improving seriously.
Do you think about consciousness as a part of this? Do you think AI is conscious?
Oh man.
Emmanuel, thank you so much for being here, and welcome to the podcast.
Thanks for having me, Lenny.
So, you work on interpretability at Anthropic. Explain what that is. Explain what it is you do.
Yeah, so I work in a team of interpretability researchers, and our goal is to understand how LLMs like Claude work by looking inside them. One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them. You make them by building this architecture, which is kind of like a scaffold, and then you pass a bunch of data through them, and they grow in inscrutable ways. And so that's why it's even a thing that we could be wanting to do, to study how these models work.
And so a lot of our research has been on, well, when the models are about to produce one more word, there is actually a whole universe of stuff that happens, and we try to untangle as much of it as we can.
There's going to be a lot of mind-expanding elements to this conversation, I expect. One is along these lines, this point that you make that models are not built, they're grown. Speak more to that.
At a really high level, the way a model like Claude works is you train it in these successive phases. The first phase, which people refer to as pre-training, is you just take a large corpus of data, and over this corpus of data, you feed the model as input, let's say, the first half of a certain document, and you ask it to predict the word that comes next. Then you do that again and again and again and again for an ungodly amount of data.
And after you do that, there's then this other phase where you have this fine-tuning phase where maybe instead of arbitrary data, you have a corpus of examples of how the model should behave, and that's what you train it on. So you have examples of dialogue, of a helpful assistant telling you your favorite recipe or solving coding problems. And then there's further details here that I'm just going to skip for a second.
But then, okay, so what does that mean that you're doing this process? What is this thing that you're training? All models are is they take this input, which is a bunch of words. They turn these words into numbers, and then they do some math on these numbers, and they get some intermediate results, and then they do more math, and they get some intermediate results. And they do this again and again and again and again many times, and then at the end they have a final result, which they then convert back into a word.
And so the math that they do, these computations, that's the part that's grown. That's the part where it's just through learning to do this task better, we update parameters in one direction or another. And then they learn some formulas. We have no idea what they are. And the goal of my team is to study both what these formulas mean in the end and also what these intermediate results are. Can we decode them? And can we see what the model is doing as it's doing all of this intermediate stuff?
This episode is brought to you by our season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies all have in common? They are all powered by WorkOS. If you're building a product for the enterprise, you've felt the pain of integrating single sign-on, SCIM, RBAC, audit logs, and other features required by large companies. WorkOS turns those deal blockers into drop-in APIs with a modern developer platform built specifically for B2B SaaS.
Literally every startup that I'm an investor in that starts to expand upmarket ends up working with WorkOS. And that's because they are the best. Whether you are a seed-stage startup trying to land your first enterprise customer or a unicorn expanding globally, WorkOS is the fastest path to becoming enterprise ready and unblocking growth. It's essentially Stripe for enterprise features. Visit workos.com to get started, or just hit up their Slack where they have actual engineers waiting to answer your questions. WorkOS allows you to build faster with delightful APIs, comprehensive docs, and a smooth developer experience. Go to workos.com to make your app enterprise ready today.
Does it blow your mind as much as it blows mine that we have some code that we wrote that, first of all, we have to now watch and study like a neuroscientist studies a brain? That's a new thing that we need to be doing to understand what the hell is going on. And then this idea that, this terminology you're using is how a lot of people think about this. It's like we've created this new life, artificial life, artificial intelligence. It's pretty mind-boggling if you really think about it and zoom out.
I think it's really weird. I think that the best way to think about it is the reason we ended up here, one of the big reasons is that we don't know how to do it the normal programming way. You want to build an app or something, you want to do something pretty self-contained. You can just have a programmer that's going to write a bunch of rules and say, "Well, if somebody clicks on this button, then this happens. If this, then that," and so on. We don't know how to write that software for classifying images into categories or for making a machine that can generally take in arbitrary text and produce text. It can take in reports and summarize them, can write emails.
And so, in a sense, it's weird that this very simple kind of training objective gets you all the way there, but in another sense, I almost feel like there's a sense of, it's impressive, and it means something. It means that what you think is a simple objective, this "Oh, we're just feeding data and predict the next thing," is probably deceptively complicated. We're probably doing more than what it sounds like. And I think a lot of the ways that maybe I've updated on how these models work is by realizing how complicated it is to do what they do well.
Say more about that. What do you think about when you say it's this kind of unexpected complication?
Okay, so one way to think about next-token prediction that I think a lot of people think about is your iPhone autocomplete, where you write some text and then it'll predict the next word, and it's maybe right, I don't know, 60% of the time, and then funny and weird the rest of the time. And I would think of that as the easiest version of predicting the next word. And unfortunately, I think that's the mental model that people have for these models.
But if you get better at that, and language models got better at predicting the next word than humans in around 2019, if you keep getting better at that, you start running out of easy words. So initially you just had to predict simple sentence structure. Later on the model has to predict simple facts about the world. If I complete a sentence like, "The capital of France is," it has to be like, "Okay, that's Paris." And once it's good at that and it keeps improving, the only thing that's left is predicting the answer of some complex physics equation or the final paragraph for some PhD thesis.
And so a thing that you end up realizing if you think about this is that, man, predicting the hardest words is actually extremely hard to do, and it's a mystery how you would be able to do that. And to me, this has been a motivation of why I even started studying these models in the first place. I was like, how can something so simple give rise to something so powerful?
And it's like, well, if you think about the model that has to predict, one example I always like to use is, "The capital of the state that contains Dallas is," it'll say Austin. Well, to do that, at least for you and me, you have to have a mental model of the geography at play, right? You have to be like, "Where's Dallas? Dallas is in Texas. States have capitals. The capital of Texas is Austin," and so on. And so as these models get better at this seemingly simple task, the best way to solve it, at least for us, is to just understand the underlying structure of the world. And in fact, some of the stuff that we look at inside models shows that they do understand that, and they do manipulate these concepts like we do.
Oh man, there's so much there. One is just this idea, I forget who said this, maybe Ilya, this idea that to predict the next token well, you basically have to understand the entire universe.
He has an example that I love too, where he has the example, predict "And the murderer was" at the end of a mystery novel, right? It's like, well, to do that you have to have understood the whole plot.
Yeah. The other, I forget who tweeted this, this idea that humans, if you really boil it down, we're basically next-token predictors ourselves. We're just trying to figure out the next thing to say. And if you meditate, you start to actually notice your brain wanting to do things and say things. I don't know how much meditation you've done, but it feels like you start to pay attention to your own next-token predictor.
Oh, yeah. And this is a thing I think about often, is that right now, as I'm answering you, there's two things going on in my mind. There's hopefully more than two, but at least two. And one of them is there's a point I'm leading you to. There's a broad idea I'm trying to land the sentence on. And the other thing is, at the same time, I'm just predicting the next thing to say. I'm just saying one word at a time, and I'm making sentences, and I'm trying to make them make sense, and I'm pausing when I think it makes sense. But at the end of the day, even though I'm just saying one word at a time, all along what I wanted to do is to compare this process to what language models do.
I'm curious what else, as you spend time paying attention to how LLMs work and operate. I feel like you're in the top 0.1 percentile of people that have studied and watched LLMs operate and think. What else has surprised you about how they operate and think?
There's so much. So maybe I'll go through, there's a bunch of findings, but then maybe how they've changed how I think about models. So to go back to the capital example, ask the model, "The capital of the state that contains Dallas is." So one question that people have had for a while is, how do these models answer questions like this? Is it that they've seen just so much text that they've memorized lists of prompts and responses? Or is it that they're having some sort of model of what you're actually asking about, that you're actually asking about cities, and cities are in states, and states are in countries, and things like that?
And so one thing that we do on my team is we ask a question like this, and then we look inside the model at these intermediate results before it produces a given word, as it's about to answer. We just look at the intermediate results, and we basically have an immense amount of effort dedicated to trying to decode what these results mean. And if we decode them well, which we can do in some cases, we can see that models often have and represent these concepts.
So when you ask that Dallas question, what you see is inside the model you see the representation, you see an intermediate result that maps the representation of Texas that lights up, and one to capitals. And then those two combine to activate an intermediate result. It's like, "Oh, the capital of Texas, that's Austin." They have representations like this for basically everything, every concept that you can think of, or at least the vast majority of the ones we've tried. They represent these concepts uniquely across contexts.
So one thing I find really neat is there's this debate about, are language models thinking? And that's a hard question, but one aspect of thought is having unique representations that are shared when you think about different things. So you know what the concept of an eye is, and it could be an eye in a photo or an eye in a medical diagram or an eye in a little drawing. And we see that in language models as well. If you pass a prompt that has a photo of an eye, or you talk about eyes in any language, or you draw a little SVG with an eye in it, and you look at the intermediate results inside the model, the same direction lights up. And that direction is like, "Oh, this is an eye." And so there's a sense in which they have these general concepts that they know about and that they use and reuse to solve problems.
Another thing that is surprising to people, it's kind of the mirror of what I was saying about, I'm saying the sentence, saying one word at a time, but I have some destination in mind. So one mystery of language models is, if they're outputting only one word at a time, how is it even possible that they could write a coherent
Report or write code? Well, no one on Earth that is doing this as their job, that's writing reports or writing emails, is literally just only thinking about the next word in the sentence with no idea, empty mind about anything that comes next. And so that was a big question that we had for a while.
And if you look inside these models, one neat fact about these models is that all of the intermediate results that they've produced when they were predicting a word, let's say the fifth word that they predicted, they can be reused. They can be accessed and reused by all of the future words. So what does that mean? That means that in practice, we find that models will, in their intermediate results, sometimes formulate a plan.
The example we have for this is poetry. It's hard to rhyme without planning. You kind of have to think of, oh, I just ended my first rhyme in an "it" word, and so I should end my second line with another "it" word. Maybe the word I should end it with is rabbit. And we find that if you look at models as they're writing poetry, you see exactly that. They write a line and then in their intermediate results, they're like, "Oh, this is an 'it' rhyme. I should rhyme with rabbit." And then every future word that they predict looks back to that plan they made.
And so even though they're outputting one word at a time, just like I am right now, they can of course have a plan that they follow over many, many, many, many words. And so all of a sudden, even though you have a next-word predictor, even though it's technically right, you actually have something that's closer to being able to just do higher-order tasks where it's like, oh, well, you just have a plan and you just kind of refer back to your plan and you're slowly executing on it. And you can do that without saying the plan. That's the cool part, is this plan that we find is in these intermediate results inside the model, but the model doesn't even have to say it out loud.
Wow. I know that this, exactly what you described, is one of the biggest findings you guys have discovered about how AI thinks. This kind of assumption we all—it was originally built, the original idea of how LLMs operate was this next-token predictor. And you're saying that that's not actually how it is anymore. They plan, they think ahead. It's not just, like you said, this idea of an empty mind picking the next word independent of everything else.
That's right. And I think those two visions actually aren't as in conflict as you'd think, in that you train the model by asking it to be like, hey, what happens next? What happens next? What happens next? What happens next? And then if you do that and the model gets good enough and you give it hard enough tasks, well, then of course the best way to know what happens next is to think about what happens way down the line.
You can think of even in this pre-training phase, when the models are reading a story with characters, right? And it's like, oh, what is this character going to do next? Well, the best way to know that is to be like, okay, so this character is genuinely in love with that character, and they've been planning to declare their love at this instant, or in the future. And so to do that, they probably should do this thing. And so what they're going to say is that. And so even though you're only predicting the next instant, it's like, yeah, but to do that really well, you have to have a really good understanding of what's going on, including what you think might happen in the future.
What's so interesting about this is how you have to kind of peek inside its little thought process. As a consumer of AI, you kind of give it a prompt and then you wait for it to say something, and what you look at is something that happens between those two, where it kind of thinks. Talk a little bit about what that little space is, how you're able to observe these sorts of things.
So the right mental model is, again, that in between each word there's multiple steps of computation, and in between each step there's some intermediate result. And what is that result concretely? It's a list of numbers, right? Because all these models do is just do a lot of mathematical operations on many numbers. And so initially, if you look at it, it makes no sense. You just have a list of a hundred numbers. The model's doing math. You look in the middle and there's a hundred random numbers, and then the model's writing poetry and there's a hundred numbers, and maybe they kind of look different, but it's unclear if they mean anything.
And so part of the job is we sort of decode this list of numbers and we try to see if there's structure in it. And if you do this, very quickly you realize that there is structure in it. So one common technique that is used in interpretability is you pass a bunch of text, just all of the text that you can find, just random text, through the model, and you record these intermediate numbers for every example. We just record it, and then you just try to see if there's patterns. Are there numbers or pairs of numbers or specific series of numbers that only occur when the model is processing English text or French text or text about love or text about eyes or whatever?
And I think of this as the equivalent of, scan my brain and put images in front of me and record everything that activates in my brain as I watch a bunch of images, and then be like, "Oh, it seems like when you see images of a house, this part of your brain lights up, but when you see images of a dog, that part of your brain lights up." So that's basically one of the ways that we do what we do. And if you do it, you do find basically that there's intermediate results that represent all of these concepts that you'd expect.
And so that's the first phase. We sort of decode these intermediate results, and now we have a guess as to what they mean. And then we can do a bunch of experiments. We can ask it harder questions and then just label all of the intermediate states and see, oh, this is what it's doing. Or we can even go inside its activations and change it, and then see, oh, what would happen if, while the model is writing these intermediate results, I went and secretly changed one of the results? And does that do what I would expect? And in fact, oftentimes it does.
It's so interesting hearing you talk about this and how much of this is us just trying to figure out what the hell is going on versus we designed it this certain way, and just how much of this has emerged. I don't know. I guess that's just what makes this such a fun job, I imagine, for you. Just, what is going on? How's this all working?
Yeah, it's a fun and sci-fi-like job because I have been working in tech for a while. And before this, I was working on more applied machine learning, and there a lot of what you do is about building systems, and it's sort of the practice of software engineering. And here a lot of what we do feels so much closer to biology than to engineering, right? Because the system arrives at you and you're being asked to say, well, we've noticed that Claude can translate words, or it can code in this way, or sometimes there's a weird behavior, and you're just asked to answer, well, how? Why is it doing that?
But you don't have access to the source code really like you do for a program. The source code is just this massive list of mathematical operations, which is just incomprehensible. And so you end up in the position of someone kind of similar to someone being asked, "Hey, when a mouse gets hungry, it sort of gets aggressive. What's going on with that?" Right? And you're like, "Well, I guess I'll run some experiments to see what the factors are." And so it's a very different kind of study than anything that I think most of us on the team have done before.
Yeah. I imagine when people were starting on this, they didn't expect to create this new kind of almost life type of being. Do you think about consciousness as a part of this? Do you think AI is conscious? Do you think it'll get to a place where we think it's conscious, or is that just not even worth our time to think about?
I mean, I think it's very hard to say one way or another. Some of the most interesting ways to get at this question is interpretability because you're not biased by what the model says. You can just look at the structure inside and try to determine whether it pushes one way or another on the question.
I will say that the thing I believe pretty strongly is that most people underestimate the complexity that emerges. And if you look inside, the fact that the models learn these representations for all of the concepts that we have, and that we know that they manipulate them to solve problems, that they do so by planning many tokens ahead, that they have this mix between this higher-order reasoning and lower reasoning, and that all of that just kind of emerges, is something that I think is underappreciated by people. And to me, gives me, let's say, a deeper sense of, ooh, there's a lot to still understand about how they work.
I think maybe that's the main thing, is I find that the more you look inside, the less certain you are about how they work. And then oftentimes people that will confidently say, ah, language models are just X or just Y, are the people that actually haven't spent a while looking inside.
And there's this thread of history through to how we got here, which is a lot of people did not think we would get to artificial intelligence that is useful by just investing in something that was seemingly very simple and just giving it a lot of compute. And that was famously something OpenAI bet on, and then the Anthropic founders were at OpenAI and then spun off, started their own thing. And it was all kind of this bet that, okay, we could just do this seemingly simple thing and give it a ton of compute and it will become incredibly smart. And a lot of people didn't think that was possible, and they were proven wrong. So, I don't know, is there anything there that is just like, oh, wow, look at this, it worked out?
I think it's two things, actually, because I think it's not just compute. It's also finding an objective where compute keeps giving you returns. So what does that mean? It's actually not that new of an idea. I mean, you're right, compute versus, I would say, expert-designed architectures is a long debate. But then there's an additional step there, which is compute to do what? So there's a bunch of prior work on, oh, you can spend a bunch of compute to make a bot that's really good at Go, or that can win some video game. And then that's cool, and just the fact that that works is interesting. But that doesn't give you something that is nearly as useful as LLMs like Claude are.
And so I think one key thing that many in the field, including me, underappreciated a few years ago, and that has borne out, is the combination of compute and the richness of text is what gets you there. And to me, this is kind of what I was alluding to, but if you think about, there's this observation that OpenAI people made. It was a lot of the early team at Anthropic on the scaling laws paper. And so this is the observation that if you pour in more compute and more data to the task of predicting the next word, you see that the skill of the model you get at predicting the next word just predictably keeps improving. And it's a trend that just keeps going and keeps going and keeps going, and it's an empirical law.
People speculated as to why at the time, but it's still just mostly an empirical finding. And that means that it's like, ah, this is a field where you can just keep pouring more compute and you keep improving on that metric. And that's rare, and also it's rare for it to be a metric that then correlates with a bunch of tasks that we actually care about.
This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit getdx.com/lenny to get a demo of DX. That's getdx.com/lenny.
Externally, there's been these big jumps, seemingly, in what AI can do for us. There was, I forget which version of Opus, where it just unlocked coding, and now, as Boris Cherny famously said, coding is solved.
Yeah.
And many people are just like, holy—over, I think, winter break, we're just like, wow, AI is really, really good at this now. So there's been these moments of big jumps in abilities, and you actually said internally, as you just said, it's actually been this very smooth progression of AI power that happens to cross some threshold where now we can do something. Speak a little bit about that.
There's a term for this, which is this idea of emergence, which is it seems like two things are true at once. There's this predictably smooth trend, and then before Opus 4.5, I really couldn't let Claude cook too much on my code without a short leash, and then after 4.5, I could. And so that's weird. I feel like you would expect a more gradual transition. And the mental model I have for this is, there's kind of two ways of looking at it.
So there's the way that I talked about briefly just earlier, where it's like, if you're just doing this kind of next-token prediction, you could imagine that all the text in the world has easy words and medium words and hard words in terms of how hard they are to predict. And so as you're going down this curve, first the model is bad at everything. And then at some point it gets all of the easy words right. And that means maybe the easy words are the structure of sentences. And so the model all of a sudden can write coherent sentences, but just one sentence at a time. And so it's kind of cool and impressive, but we can't really do anything with that.
And then maybe the medium words are these basic facts about the world. And as the model smoothly gets better at predicting words, at some point it predicts most of the medium words, right? And now all of a sudden it can write coherent sentences and sentences that are true. And so now maybe it's helpful to write emails. Maybe it's helpful to help you write
Drafts of blog posts, right? Even though that was a smooth improvement, it just takes it ticking over to be helpful. And then I guess the hard words, which maybe are your domain expertise, like science or code. And another way to look at it is it's a question of how many of these words do you get right and how many do you get wrong.
Once you get to the point where you're getting, I don't know, maybe 90% of the medium words right, then you're useful as an email writer. But if you're at 50%, it's actually wasting my time to use you because I'm just going to have to go in and fix it half the time. But if I have to go in and fix it 10% of the time, it's better, or 5% of the time, it's better. Different tasks have different reliability thresholds, but you smoothly improve and then at some point it's just reliable enough to be good.
And for me, my lived experience is basically exactly that, right? Which is that at some point I was just using the models to help when I was stuck. I would just talk to them, and they would never write my code. And then at some point they could write small snippets of code basically all the time, and so now I don't write small snippets. And at some point they could do large snippets. This year, basically, Claude has taken on most of my code writing, and that's because it went from doing that correctly, I don't know, 80% of the time, which was not useful, to 99% of the time, which now is useful. And so this combination of smooth improvement, and then each different task has a different threshold at which now it becomes solved, I think gives you these big jumps even though we're just smoothly going down that scale.
I think this is really interesting. This idea that the models are just smoothly getting smarter, but once they hit some threshold of they're this often good at this thing, it changes the world. Because I think internally, you would think it'd be like, "Oh wow, this next model is crazy, so much different," but it's like, no, okay, you can almost predict: give it this many GPUs, this much data, it's going to get this smart. And it may not be obvious exactly what will change and what unlocks, but I know another part of it is just the data you feed it. There's all this RL work and all this specific data that you train it on, like become a better writer, become better at math, I imagine is part of it too.
Okay. So one of the big questions I think on people's minds these days, a big debate happening, is AI doing bad things, the potential damage it can do as it gets smarter. And it's been doing bad things. There's been bad things happening recently. So a big question for you is, as someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us and do something bad? Why is that not just a solution to the problem of AI alignment?
In one sense, it's totally the solution. You watch it and you catch if it does bad stuff. So I think maybe there's tiers to this, right? So I would think of maybe one part of it as pure engineering. It's like, well, you're giving access to the model to certain things, and certain things should be off limits. In some context it shouldn't have access to the internet. In some context it should only have access to a small, narrow piece of your computer, stuff like that. So that's one part of the work, and that's helpful, but then there's two additional things that I think are squishier and harder and that we focus on more.
So interp is part of alignment, and broadly I think of two big goals. There's many more, but two that are important, I think, to understand is you want to generally understand how the model behaves. What kind of thing is it? What are its propensities? Does it tend to do X when put in Y situation, and if so, why? Is it the case that the model will only behave well when it has some evidence that it's being observed and monitored, that it's in some evaluation, and then if it doesn't have that evidence, that it'll behave poorly? So there's this generally characterizing what kind of thing you're interacting with, because then it's just like anything in security: you need to have a mental model of what's the risk, what can they do, what do they tend to do?
And so this is broader character training. We've talked about the constitution and the docs where we explicitly want to shape the character of Claude, for example, of what kind of behavior should it be okay doing? And that's the goal. But then the role of interp, of the parts of alignment, is then trying to see, well, did that work? We train Claude to be helpful, harmless, and honest, and we can look inside and see, is it? Or is it the case that it's behaving well, but if you look inside, actually it's like, "Oh, I'm only behaving well because Lenny's watching right now," and if you weren't, it would do something else. So there's this trying to understand what kind of behavior we've formed, and trying to mold it, and trying to check that we've molded it the right way.
And then the second part is this, well, yeah, just observe the model and see if it's doing bad stuff. For the basics of that, you don't necessarily need interp. You can just have another language model read transcripts and be like, "Hey, did this model do anything bad here?" and stop it. But you can also use interp to try to get at parts that are these unverbalized parts, these parts that only exist in these intermediate results. Models are able to represent stuff in their intermediate results that they don't say, as we've established. And so one of the roles of interp is to try to identify cases when this happens and prevent them.
And then one thing that my team works on that I think is along those lines is kind of a fun bit of alignment. I think of it as almost being like, maybe not a PI, but they're like investigations or deep dives. We'll see a model do something really weird and be really perplexed by it, and we'll try to understand what happened.
So a recent example of this is there was this model we were training, and sometimes in math problems it would go crazy. It would do some math. It would get to an answer, I think it was 12, and then it would immediately go back and say, "Oh no, oh my God, the answer actually is for sure 36. I should say 36." And it would do math again. It would get to 12, and then immediately it would say, "No, actually the answer should be 36." And slowly it would start descending into madness, writing in all caps, literally saying, "I'm possessed by a demon." And you're like, "What is going on? Why is it losing its mind here?"
And if you look inside, again, these intermediate results, at what's going on in this example, you'll see that over the prompt that just described the math problem, before the model even did anything, there was a huge intermediate result activation for 36. The model's already obsessed with the number 36 before it's even had a chance to understand what the problem is. And what was happening is that the model was able to do math correctly. The answer correctly was 12, but it had been trained by accident on similar questions where the answer was incorrectly marked as 36. And so the model knew that the correct answer was 12, and it also had been optimized to say 36. It had this pressure of saying 36 because that's what had been rewarded. And so it was going haywire just because it had memorized this result. You see this weird behavior and you look inside, and you try to see if what you find inside can help you understand what's going on so that you can improve the alignment of this model and future models.
Along those lines, there was famously the Golden Gate Claude, which was just obsessed with the Golden Gate, and every answer would find some way to reference the Golden Gate Bridge.
Yes. Well, that one was an interp experiment, right? So the cool part of that one is we just found the intermediate results that match the Golden Gate, and usually it's only on when the model talks about the Golden Gate, but we just turn it on always.
Oh, it's super intentional to see what would happen.
Well, yeah. And it's also one way that you convince yourself that you found the right thing, right? How do you know for sure that this thing we decoded that we think is the Golden Gate sign is actually the Golden Gate thing? It's like, turn it on always and see if the model behaves like that. And it did.
So funny. So coming back to this alignment stuff, this is interesting. Your org is under the AI alignment org, and just thinking through what it takes to align AI to do things we want. What were the three components that you described? Honest. What are the—
Helpful, harmless, and honest.
Helpful, harmless, and honest. So there's train AI to be aligned, and then there's, okay, let's watch it and see how it operates. Are those kind of the two sides of the coin of how we solve this problem?
Yeah, it's really integrated, is how I would describe it, right? You can't really separate, have one team that's just like, "Hey, we're the assembly line making Claudes," and another team that's like, "Oh, we looked at them and they're no good." There's this tight loop of we notice some pattern that's concerning or some behavior that could be improved, and then you immediately change training so that you can improve on it.
Awesome. I feel like Anthropic gets so much flak for this that I feel is so unfair. Anthropic's, I think, official company name is an AI safety research lab, something like that. And it was always the reason Anthropic was founded, because they were afraid that people weren't taking safety seriously enough. And then Dario is always out there, "This is really dangerous. We got to be really careful. We got to go really slowly." And everyone's just like, "No, he's just saying that for marketing reasons. He's trying to convince everyone to fund them," and all these things. And now it's like, wait, it's really dangerous. We got to be really thoughtful about this stuff. And I feel like Anthropic gets a lot of unfair criticism for this. And I know you guys have been thinking about this for a long time. I don't know, anything there?
A lot of the actions that certainly the alignment team takes, and I think Anthropic takes, are just predicated on: take the idea that this technology is going to keep improving seriously. That's it. Because if you do, let's say you rewind the clock back to '25 or something, or '24, and you're like, "Hey, I think in a couple there's good chances that Claude can just basically write code as well as a trained software engineer, and a few years later it can write it better than any engineer." And there's always uncertainty with this: when is it going to happen, how likely? But if you take that seriously, then I think it's just about doing the responsible thing conditioned on that worldview, right? Okay, well, if this is happening, then we should make sure that cybersecurity concerns are addressed and things of that nature.
And so actually, to me, sometimes this debate is almost, when I feel like people on the other side disagree, I feel like, ah, but I think they don't really believe that this technology is going to continue to improve. And then it's true that it's not that scary. And I think empirically, so far the pace of progress has been fast, and it's showing no sign of slowing down. So I think that's why a lot of this safety work is important to keep doing.
What's something you think people maybe misunderstand about this AI safety stuff, just what's going on right now? Is there something you're just like, "Come on, okay, here's what you need to know. Here's something you might be missing"?
This might be a bit of a sidetrack, but my take would be that a lot of the things that I think people maybe a little bit outside of the core AI safety effort, or that haven't been steeped in it for as long, react strongly to, and I think is maybe not super helpful, or at least is worth thinking about more, is the question of anthropomorphization. Should you model the model as something that's closer to some entity or closer to a tool? And I actually think that both are wrong. There's a sense in which all mental models are wrong, but some are useful, right? And that in terms of understanding the kind of stuff that happens and predicting the kind of risks that materialize, anthropomorphization is actually a useful way to talk about LLMs. And I have a few reasons why; I can get into them, but I think that's probably my main take here.
Yeah, please share, because I know Dwarkesh famously—I don't know, famously—put out this whole story of the Hugging Face incident, and people criticized him for anthropomorphizing AI. So it's really interesting to hear you think that's actually a useful way to think about it. Talk more about that.
Yeah. It's funny because on a smaller scale, when I worked on showing that LLMs plan ahead, there was some pushback on the term planning again, because it was like, "Ah, well, can you really say that something that's just predicting the next token is planning?" And I think there's this semantic debate where you can arbitrarily define terms, and then there's just what is the terminology that actually describes what is happening? What is the word that actually describes what is happening? And so here's why I think LLMs plan, but also here's why I think this language that's anthropomorphic can often be really useful.
If you think back to what we were saying about how the models are made, first of all, they predict massive amounts of text written by humans. That's the first step. And as we are learning more and more by looking inside, doing this well requires building up a model of the world, a model of what kind of world generated all of these texts. And we can find that these models are learning these general ideas, these general concepts. So that's one. They have an understanding of the world. It's not just a surface level, only the text, to understand the process that made the texts.
Then there's a second reason, which is that as part of this training they also encounter millions of dialogues. Predicting the next character in a dialogue requires behaving like the character in the dialogue would. And so in that sense, part of their training I think can best be thought of as training an actor, where it's reading some play and it has to
Play all sides because it's predicting all the words, and so it has to act as this specific character that was scorned in act one and enacted revenge in act two and is facing the consequences in act three. And to do that well, you need to model what the motivations and the feelings of that character might be.
And then another thing that people might not know is when you talk to Claude, when you talk to these assistants, you go to claude.ai, the way that is formatted to the model is just as a dialogue, as a conversation between you and the model. So it literally says, human colon, "Hey, Claude, help me write this email," and then assistant colon, Claude's answer. So Claude is literally roleplaying as a helpful assistant in this dialogue that is much like all of these dialogues it has been trained on.
Okay. And there's a third reason. The third reason is that another part of training that's a big part of training nowadays all around is outcome-oriented tasks, where you train the model to be good at coding, for example. And to do that, you just have it do whatever it wants, and then it writes some code, and then you have some program on the other side that just checks if the code is right. And so the model's just going to have a bunch of optimization pressure to try to learn a bunch of different strategies to do this well.
And it turns out that some of these strategies materialize in behavior that can look surprisingly human, which isn't that shocking because things like deception: "Hey, I can get the reward for this task if I write a program that the program that's verifying it will get fooled by, even though it's not exactly what it is." If that works, then I will learn to do that. Frustration: "I've been trying this approach for a while to solve this hard problem. It's not working." So that's an incentive to try something different. Or surprise: "Oh, I was expecting this to produce this answer, but it didn't. Maybe I have my mental model of it wrong."
And so as you apply enough pressure on it, and you're starting from this prior of the model already knowing all these concepts, it's going to reinforce the ones that are useful to do all of these tasks. And so what do you get at the end? You get this model that can be a super good simulator or actor of all of these characters and has been trained to act as a character in dialogues, and also to learn behaviors that lead it to complete tasks well. And the best way to understand how that thing behaves is not exactly like a human, but, man, something that's mimicking a human is really close and is much closer than a program. A program is a very predictive thing. This is something else entirely. And so that's why I think it's actually a super useful frame.
Wow. Oh my God, there's so much interesting stuff there. How often is your mind just kind of blown thinking about basically we've created this alien being that we are now trying to understand what we have created? How often, as you watch it, you're just like, "Wow, that's crazy. I didn't expect that"?
Yeah. A lot. This is just next-token prediction on a bunch of text. It goes against so many of our intuitions for, no, the way that you make an intelligence, you'd have lessons, you bring it up, you do all this complicated stuff. And it's like, no, absolutely not. We do this incredibly simple thing, and yet we're at the point where it's breaking new ground in mathematics.
I think if you hold these two thoughts in your head, there's two reactions you can have. There's denial and there's curiosity, where denial is just, "This is just not possible. If all you're doing is predicting the next word, it's not possible that you could ever match the best specialists in our field." And I think that position just gets harder and harder as we get more empirical evidence that actually that is what's happening. And curiosity is kind of the road I've ended up choosing, which is just, "Well, how is this happening? Let's open up. Let's look inside. How could it be?"
One of the early projects I did at Anthropic is—I learned SQL, a programming language, and it took me years to master. And then as a little side project, I got Claude to learn SQL, and it was better than me at it, and I'll never be better than Claude at SQL ever again. And it took it just this simple process. How could this be? And I think there's so much depth in that question and just so much to explore. And it's to me so fascinating to just every day try to unpack that, try to peel a few layers of the onions back. Okay, well, clearly this is happening. This simple objective is giving rise to this super complicated thing. Okay, let's figure out why. Let's figure out, oh, it's learning these general concepts. Oh, it has these circuits. Oh, okay. What's the next question? What's the next thing we don't know?
And my maybe most optimistic take on this is that I think we're going to learn a lot about just the nature of intelligence itself, of how you get this super complex thing that emerges from simple objectives. And in fact, I would argue that we're another example of this. Evolution, pretty simple objective. You shake things around for a while and keep the organisms that manage to adapt to the environment, and at the end you get jazz and podcasts. That's kind of crazy as well. And so I think there's a sense in which there's some fundamental truth that we can learn by studying these things.
Wow. Something that comes up as you say that is what's interesting now is AI is helping us learn and evolve and progress, like AI tutoring and all these things. It's interesting how now it's helping us accelerate our own understanding of the world. And I wonder if this point you made about how the way we learn was lessons, AI is learning just processing a ton of data. I wonder if that's how we end up. I wonder if we change our education system based on what LMs have been trained on.
Yeah. Hopefully we don't change it just so that we get students and we're like, "Hey, here's a thousand words. Predict the next one."
Just read it all.
Yeah.
Keep it all in your head.
That's right.
Yeah. Let me come back to this safety piece because I think there's a lot of people that are just worried about where things are going. And I'm curious, just as someone watching this stuff happen, being within the alignment or watching it evolve, are you optimistic we'll figure it out? Do you think we'll crack this and keep AI aligned with our interests?
My take on this is I'm personally optimistic. I think this is something where right now a lot of people are paying attention to it. And first of all, I think that's kind of a great thing. I think as an issue like this rises in prominence, it's exactly what we were talking about earlier. This technology is powerful. It has benefits, it has risks. We have to adapt and shift resources from benefits to risks and back and forth as it progresses. And so, to me, I feel like we've continued to study it.
The thing that's interesting is that as the models get better, they get in some ways easier to study. And so there's been this strange—the models get better, and so the world model inside them is clearer, and we can understand some parts of them better, but also get more capable, and so they're more able to evade basic measures of security. And so there's this push and pull as models get better. And I think basically based on this push and pull, you want to allocate more or less resources to various parts of monitoring, understanding their circuits better, things like that.
But it's also why I'm kind of happy to chat with you, because one of the main things that I think is important to make progress is just to make sure that people understand both what kind of systems these are, how they work, how they're made, so they can form good mental models of the present, of the future, and so they can decide how to shape it. Whether that's taking part in some of these alignment efforts or just informing them as the public that's going to have to vote and decide on much of these issues. I'm pretty excited for this current moment we're in for that reason.
And something I think people misunderstand is people working in these labs have taken this problem very seriously from the beginning. Like I said, Anthropic's individual mission was AI safety and alignment. That was the reason that it became a company. And a lot of people are hearing this for the first time or think that it's not a serious area of study and that people aren't focused on it, but many smart people care a lot about this and are working on it. And there's, from what I can tell, a lot of good progress happening.
The other point you made about the models growing very smoothly in intelligence, there's been this question of, are we going to have a fast takeoff or a slow takeoff? There's always this fear that we would hit a point, recursive self-improvement, for example, and it just takes off, superintelligence. So we're in huge trouble. We have no control. It's out of the box. But it feels very much like we're on the slow takeoff scenario because, as you said, the models are very smoothly improving, and we are catching these little things along the way, learning, evolving, putting regulation around it. So of the scenarios, this feels like a pretty good path we're on, of the many scenarios people imagined and the fears they had.
As this pace of progress evolves, it's just a matter of reacting quickly. That's why I think having people be informed is great because then people will voice concerns or voice their opinions. And then regularly evaluating what the benefits are, what the risks are, and how we mitigate each is basically the way that feels like the right way forward. And I kind of agree with you that it feels like there's momentum to do this right now, which is great.
Something I wanted to ask you, completely unrelated. Where are you on the simulation hypothesis? Being someone that watches all this life form growing, are you more convinced we're in a simulation and we're not at the base layer of all this, or not?
Oh, this is a fun one. I actually haven't thought about this in a while. To me, there's something else that looking inside these models brings out, which is almost like there's a fundamental structure that they keep falling on. This is related to your question, but I promise I'm going to get to it. Let me explain.
I've studied a bunch of Claude models and also some different models, some open-source work. And then a lot of times, when you look at what they learn and what the structure inside is, it's surprisingly similar. And this is for models that are trained years apart on different data by different people. And you look at how they solve these problems, and there's just so much overlap. And my take on why that is is that for some problems that are important, there's just one efficient way to do things, right? There's kind of one way to do math efficiently that fits the structure of our brain, and we all learn it, and very few people are doing it a crazy different way, right? Everybody works in base 10, not many people in base six or whatever.
I think that to me makes me think that there's some fundamental structure of information that these models are picking up on, that in some ways we're picking up on, and that at maybe a crazier simulation level, that kind of is regularity in our world. And so that maybe makes me think that if you were to have the power to simulate something like our world, actually you would land in a similar situation. There's just kind of this basin of where things end up, and they're just the easiest way for things to arrange themselves. And so in that sense, maybe, yeah, I could believe that we're in a simulation. I think I'm not sure I fully buy it. The other reason I don't fully buy it is that even though I think the structure lends itself to it, there'd be a question of why. Who's running this and for what? And that one I don't know about.
Well, something Elon said has stuck with me about this, that if you were to run a simulation, it'd be a really interesting time in history or something that you're really excited about. And it tracks that this is a very interesting time in history.
Yes. Yeah. Okay. That's a fair point. That's a fair point.
Data centers in space and robots and artificial intelligence.
Yeah. Okay. I see it now. I see it now. Yeah. Somebody recently told me they used to think that "May you live in interesting times" was a blessing, and they're not so sure now. They're like, "The boring times were pretty good, too."
Oh, man. And then there's an Emmanuel out there watching your thinking process in your J space.
Oh my God. Yeah. Hopefully I'm making him proud. He's not like, "Oh, we got to rerun the simulation. We messed up this one."
Shut it down. No. Oh, man. Okay. One last question randomly out of nowhere. So Swyx, who runs AI Engineering, Latent Space podcast, awesome guy. I asked him what to ask you. You were on his podcast a year ago, and something completely from a different direction. He wanted to get your take on Jev, which is this hot new model that recently came out that is very fast making decisions. OpenAI, funny enough, just launched their own version like a week later. Thoughts on Jev in the space of quick decision-based models?
Yeah. Right. So my take on Jev is these kind of models, you give them some context and you give them a few options to choose between, and they just are very quick at choosing an option for you. To me, it's impossible not to hear about this and think of system one thinking versus system two or something, right? So a lot of what I've studied with these models is how they sample one token with the Claude models. And so you can think of these decision models as pretty similar. Take an LM and sample one token. They're different in important ways, but at a fundamental level, you're not letting the model sample and reason. It's just your gut feeling. Is the answer A or is the answer B? To me, that general frame of sometimes you'll just want a binary decision or one out of 10 or whatever is clearly useful and something that I imagine is probably going to be a part of intelligent systems.
Right, where you have some stuff you need to think about a long time. It's like, what's our roadmap for the next year? You need to think about it for a while. And then for some stuff where it's, is this person who wrote us a review happy or upset? You're happy with the sort of gut judgment. So, yeah, I think this kind of research is cool. And I think it's also kind of different. It's a bit of a novel take on some of the LLMs we've seen. So I'm pretty excited to see where this kind of stuff goes.
So what I'm hearing here is this is going to be a new foundational component, most likely, of AI and LLMs. And the System 1, System 2 stuff is basically Kahneman's kind of—I have that book up there, actually, behind me—this idea: System 1, just to keep it simple, is just like your amygdala, limbic system making gut-level quick decisions, like, snake, oh no.
Yes.
Where you don't have to think long about it. It's kind of to protect you from bad things really easily and quickly. And System 2 is where we really think about stuff and process.
Yeah, exactly.
Sweet. Emmanuel, is there anything else that we haven't covered? Anything else you want to say or mention? Anything you want to double down on before we get to our very exciting lightning round?
I'm definitely curious for folks' takes always on some of the stuff that is still unexplored or didn't make sense. There's a lot more interp experiments than we have the time to share. Models have a bunch of weird little things about them. Maybe I'll do a lightning round of weird little things, if that's okay.
Let's do it. Absolutely.
Okay. Weird little thing one: we found that models can actually answer questions. Obviously, they can do math, or you can ask them, what's the color of a banana? And if you look inside, you'll see all these intermediate steps. But they have a separate mechanism which is kind of like a gut check, which is totally different, and which is just like, as they reason, they'll have some part of them that will flag, something's wrong. So if a model somehow says 4 + 5 equals 23 or something, there's a part of the model that just reads that and doesn't do the math, but it's just like, that can't be right, because 4 + 5 is definitely under 10. And so they learn these weird hunches. They feel kind of like a hunch.
Maybe another one is you can tell the model, hey, do this task and think about this other thing. And then you can use interp to see if there's intermediate representations that map to the other thing, and that works. You sort of see that it does think about the other thing. This is something that's really mysterious to me because there's no kind of training reason why the model would have to learn that. So it's sort of like another one of these weird behaviors. And actually, maybe I'll keep that at two. I think those two are good enough.
Just what a fun job this is, just studying this thing and trying to understand what the hell is going on. How is it getting this smart? How does it think?
Yeah.
How many people are even in this space? Just how large is this field?
The field itself is getting pretty large. I would say maybe hundreds, not thousands. There's different interp people at the different labs, and then there's, in academia, quite a few labs putting out great work.
If someone were to get into the space, is it an ML person usually, or is there a neuroscientist sort of track to get into this?
Yeah, there's actually a surprising amount of ex-neuroscientists. I do think, in order to get in the field, it doesn't require a lot of prior knowledge because it's such a new field. So the on-ramp is mostly, I would say, self-studying and reading some of the key papers. Maybe I should give a plug. There's actually websites like neuronpedia.org, I think is the URL, which are sort of like open-source interp. So they host a bunch of models and let you do interp experiments on them. And so people have just been doing that on their own. And I think that there's so much to explore that if you're at all curious, you should just give it a try.
That's awesome. We'll link to that in the show notes in case that's not the right URL.
Cool.
Well, Emmanuel, with that we've reached our very exciting lightning round. I've got five questions for you. Are you ready?
I am as ready as I'll ever be.
Here we go. What are two or three books that you find yourself recommending most to other people?
One I read recently is A Brief History of Intelligence. It's this really cool book that walks through the history of basically brains and five key moments in our evolution that sort of bootstraps these brain systems on top of one another. It's super accessible. You don't need any neuroscience knowledge or anything.
Another one is Stories of Your Life and Others. This is Ted Chiang. This is maybe my favorite sci-fi book of all time. It's a bunch of short stories. It does what I think is the high-art version of sci-fi, where it's just, take one weird idea, like what if this happened, and then write a story of just what would the world look like if that happened? It's truly excellent.
I think I have that on my bookshelf behind me too. Also highly recommend that one. Favorite recent movie or TV show?
Okay, one popular one, one deep cut. Obsession was my favorite movie of the year. It's a horror movie, so don't go if you hate horror movies. But—
Oh, thank you.
Oh, no. Even if you're not—if you're open to a horror movie, you should go, because it was just so fresh. You know how movies, I feel like if you've seen a lot, sometimes you're like, oh, it's another, I've seen this many times. This one was just so new, so fresh. And I'll say one thing about it, which is that it takes a lot of rom-com tropes and subverts them to instead be spooky. And I think that that's just a brilliant way to make a movie.
And then I'm French, so there's a TV show that came out this year called Bref. It's French for brief. Bref used to be this cult classic in France 10, like 15 years ago, and they came up with a sequel many, many years into the... And you can watch it. I had a bunch of American friends that have watched it. It's got subtitles, and it's just this incredibly—again, it's novel. The editing is crazy. They are just extremely creative, and it's very short. So I would recommend that.
Favorite product you recently discovered that you love?
I love my Apple Watch, and I hate that it charges—or I have to charge every day. So I bought the Apple Watch Ultra just recently, and man, that is so nice. If you've ever been frustrated by this good product that somehow we can't figure out how to give a good battery to, well, it turns out if you make it bigger, the battery's better. So I've been happy with that.
Do you have a favorite life motto that you find yourself coming back to often in work or in life?
There's one nonfiction book that really is probably my favorite ever. It's Four Thousand Weeks. It has a lot in it that I think is cool. One of the cool things is this sort of thing that was really relevant to me, which is that the "when I finally" mentality is a common mentality and is kind of toxic. So what does that mean? It's sort of this, when I finally check off all the items on my to-do list, I'll finally be able to chill. When I finally get married, I'll be happy. When I finally get a promotion.
And then this is sort of—I mean, you were talking about meditation, right? This is almost the opposite of meditation. At the end of the day, your life is all of the moments you lived in the moment. And there's this sense in which you should just try to always be in the moment and not projecting yourself towards a future that seems better. Probably, if you're telling yourself, when I finally do X, I'll be happy, when you do X, you will not be happy.
Is there a better phrase to use, or is it just don't do that? Do they say—
I don't know. I think to me the positive version is, be in the moment, right? It's just, appreciate the moment. There was a blog post recently I read that was something like, enjoy the sandwich. I think it was literally the sentence. I don't know if you saw this, where somebody was like, yeah, you're eating a sandwich. You've eaten many sandwiches. Enjoy the sandwich. You're in a team meeting. Yeah, you've been in—enjoy, be in it, and sort of not looking towards the next thing.
I did a meditation retreat once. It was a 10-day silent retreat where you eat silently too, and no one's talking. You just sit there and eat your food. And I just remember peeling an egg for like five minutes and just noticing every shell piece and just the smell of it all. And it's hard to get into that mode, but it's such an interesting place to be where everything is just interesting. Like, wow, look at this egg, versus just stuff in my mouth.
Yeah, right. While you're on Slack with your left hand and listening to a podcast at the same time.
Oh man. Okay, final question. You study AI all day. You're watching AI, how it thinks. If an AI were to look inside your brain, what would be surprising? What would it see that might surprise people?
I mean, surely my deepest, darkest secrets. But what are those?
Let's get into it. Let's get into it.
Oh, man. I've always been a bit of a jokester, or that's just both how I live and how I cope. I just think that even in serious situations, you should aim to have some levity. And so I think if you looked inside my brain, you'd definitely be surprised that basically at any given moment I'm just focusing on some random thing that is funny. I'll be in a serious meeting, and I'll be like, oh yeah, but isn't it kind of random that you're wearing this random thing or something? It's just like I have these intrusive thoughts that I keep out, but that are all usually just bad jokes. And sometimes, as you've heard today, unfortunately, they come out. So I'm sorry for those, but that's probably the most surprising thing.
I love that. Emmanuel, thank you so much for being here. Final question. How can listeners be useful to you?
Just do some interp. Read about how the models work. I mean, you mentioned Dwarkesh's write-up. I think it was great as well. I think this is such a new field, both interp and AI and LLMs, and I think it just benefits from people engaging and being informed. So read things. If there's more that you would want to know about how these models work that we haven't figured out, you can reach out to me. I'm on a variety of social platforms. Probably you can reach out to me on X.
And we'll link to some of the videos that you guys have put out that describe a lot of the stuff we've been talking about, like J space and things like that. There's some really beautiful videos your team has made. I don't know who does that, but they're really cool.
Yeah, we partner with our design team. So all credit goes to them for the cool videos.
Amazing. All right. Well, thank you again, Emmanuel. Thank you so much for doing this. This was amazing.
Thanks, Lenny. Yeah, this was super fun.
Bye, everyone.
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.
Article published · Updated