Adam Marblestone on the Brain's Hidden Advantage: Reward Functions, Not Architecture
Dwarkesh Patel
Dwarkesh Patel opens with what he calls his "million-dollar question": LLMs are trained on far more data than any human ever sees, yet they have only a fraction of human capabilities. How does the brain do it? Adam Marblestone, CEO of Convergent Research and formerly a research scientist on Google DeepMind's neuroscience team, says he doesn't know the answer. He also doesn't expect the answer to come from smart people thinking harder. His view is that neuroscience needs to become a far more powerful field, technologically and in other ways, before questions like this can be settled.
He still offers a working hypothesis. The field has neglected the brain's loss functions. Evolution, he suspects, built enormous complexity into what the brain is trained to do. That idea, developed largely through the work of AI safety researcher Steve Byrnes, ends up tying together several threads in the conversation: how the genome can specify intelligence with so little information, how evolution could wire abstract social desires, and why neuroscience might need connectomes to answer any of it.
Four components, and the neglected one
Marblestone uses the vocabulary of modern deep learning to break the problem into parts. There is the architecture, including hyperparameters such as the number of layers. There is the learning algorithm: backprop and gradient descent, or something else. There is initialization. And there are cost functions: what the system is trained to do, what its reward signals, loss functions, and supervision signals are.
His hunch is that the last component is where the field has looked least. Machine learning favors mathematically simple objectives such as next-token prediction and cross-entropy. Evolution, he suggests, may have built "many different loss functions for different areas turned on at different stages of development." He describes this as "a lot of Python code, basically," generating a specific curriculum for what each part of the brain needs to learn. Evolution has seen many times what worked and what didn't, so it could encode knowledge of the learning curriculum itself.
The cortex as an omnidirectional prediction engine
On the common claim that the cortex holds a universal learning algorithm, Marblestone first says plainly that nobody knows. He notes that the cortex has six physical layers of tissue at any one location, which is a different sense of "layer" from a neural network's. The connections between cortical areas look more like network layers. Some models try to explain how cortical circuitry approximates backprop, but that leaves open what the cost function would be. Is it next-token prediction, image classification, or something else?
One possibility he finds appealing is that each cortical area is a very general prediction engine. It can learn to predict any subset of the variables it sees from any other subset, which he calls "omnidirectional inference." He contrasts this with an LLM, which natively computes one conditional probability: the next token given everything before it. An LLM can fill in the middle of "the quick brown fox blank blank the lazy dog" only as an emergent behavior of the context window. A cortex built for omnidirectional prediction could, by design, infer any missing pattern from any clamped subset of its inputs.
He says this closely resembles what Yann LeCun argues with energy-based models. Such a model represents the joint distribution over all variables. You clamp some variables to known states and sample the rest, and at test time you can choose an entirely different subset to clamp. On this picture, association areas might predict vision from audition. Other areas might predict what the innate, "lizard" part of the brain is about to do: whether a muscle will tense, whether you are about to laugh, whether your heart rate will rise.
Ilya's question: how does evolution encode abstract desires?
Marblestone connects this to something Ilya Sutskever said on Patel's podcast: that he knew of no good theory of how evolution encodes high-level desires. Marblestone thinks the question is profound and closely tied to the brain's loss functions.
He gives a self-deprecating example. Suppose he feels embarrassed because he imagines Yann LeCun listening and objecting that he described energy-based models badly. That triggers innate embarrassment and shame. But evolution never saw Yann LeCun, energy-based models, podcasts, or important scientists. Somehow the genome has to ensure that whatever neurons in the learned world model represent "Yann LeCun is upset with me" get robustly wired to the innate shame response, and then drive further learning. The reward function has to use learned information that evolution could not have anticipated.
Byrnes's answer divides the brain into a Learning Subsystem (cortex and other learning structures) and a Steering Subsystem (hypothalamus, brainstem, and related areas that contain innate responses and innate reward and bootstrapping functions). The Steering Subsystem even has its own sensory apparatus. The superior colliculus is a subcortical visual system with innate abilities to detect things like faces or threats. Parts of the amygdala or cortex learn to predict what the Steering Subsystem will do. Byrnes calls these predictors "Thought Assessors."
Patel presses on generalization. The cortex could learn that a literal image of a spider is bad because the innate system flags it. But how does it come to treat "somebody says there's a spider on your back" as bad without supervision on that case? Marblestone walks through the mechanism. An innate circuit might fire when something small, dark, high-contrast, and fast moves toward your body, and you flinch. Some group of hypothalamic neurons represents "I just flinched," and that is a negative contribution to reward. Taken alone, this signal barely generalizes. You might avoid that exact situation, and perhaps the actions that led to it, which Byrnes calls generalization "downstream of the reward function." Meanwhile, part of the amygdala learns a classifier: could I have predicted the flinch a few hundred milliseconds or seconds earlier? The brain trains predictors like this for every important Steering Subsystem variable. Am I about to flinch? Am I talking to a friend? Should I laugh? Is this friend high-status? Am I about to taste salt?
These predictors generalize because their inputs are completely different. They read from the cortex's abstract world model, where the word or concept "spider" can activate in many situations, from a book about spiders to a room known to contain them. The Steering Subsystem can only use what the superior colliculus and a few other sensors provide. The cortical predictor can fire on abstract concepts. As Marblestone jokes, this conversation is probably activating the audience's "skittering insect" hypothalamic neurons purely through abstract language.
This also answers how to locate the "social status" neurons in cortex. They are whichever neurons predict the Steering Subsystem's innate social-status heuristics. Train a predictor, and the neurons that belong to it are, by construction, the ones now wired to that reward.
Marblestone notes that Byrnes, a former physicist working as an AI safety researcher, assembled this by synthesizing the existing neuroscience literature. He links this to academic incentives: the work is speculative, and it's hard to say what the next experiment would be, how to publish it, or how to train a grad student on it. Still, he thinks Byrnes "has an answer to Ilya's question essentially."
Could masking or better encoders get us there?
Patel asks whether you could get omnidirectional inference by training models to map every token to every token, or by adding more cross-modal labels. Marblestone says that might be the way, but he isn't sure. Some people believe the brain uses a different form of probabilistic inference or a learning rule other than backprop. Another version is that the brain runs "crappy versions of backprop" through a few layers and resembles a multimodal foundation model. Either way, he stresses flexibility. A model trained to fill in one blank doesn't know how to fill in a blank it was never trained on, because that prediction was never amortized into the network. A powerful inference system could choose, at test time, which variables to clamp and which to infer.
Patel suggests the gap might be in the encoder: maybe modalities aren't represented at a shared level of abstraction, which might also explain why LLMs seem weak at drawing connections between ideas. Marblestone replies that these questions are all tangled together. Without knowing whether the brain does backprop-like learning, whether it uses energy-based models, or even how areas are connected, it is hard to reach ground truth. He mentions a few relevant directions. His friend Joel Dapello used a model of early visual cortex (V1) as the input stage to a convnet and saw some improvements. The retina does motion detection and filtering, so sensory preprocessing may matter. Astera, which also employs Byrnes, launched a project based on Doris Tsao's work on vision systems that need less training because assumptions like "objects are bounded by surfaces" and rules about occlusion are built into the architecture. Evolution may have put some changes into architecture, he allows, but he still thinks cost functions may be a key thing it contributes.
Amortized inference, test-time compute, and what evolution chose to bake in
Patel lays out his understanding of amortized inference. True Bayesian inference means considering possible causes of an observation and finding the best explanation. That is intractable, so it requires sampling. A feedforward network skips this and maps observation directly to the most likely cause. Marblestone agrees and notes that Monte Carlo methods, Boltzmann machines, and probabilistic programming all rely on this kind of slow sampling.
Patel suggests that test-time compute in reasoning models is doing that sampling again. You can read a chain of thought trying one approach, rejecting it, and trying another, and those capabilities later get distilled back into the model. He speculates that digital minds, because they can be copied, should amortize more than biological minds do, and asks what future AI might amortize that evolution didn't.
Marblestone gives the probabilistic-AI view first: inference is inherently hard, amortization is a crude approximation, and stochastic neurons suggest the brain may be doing real sampling. But he adds that perception works in milliseconds and doesn't seem to use much sampling, so the brain must also be baking things into approximate forward passes. On evolution, he points out that everything must pass through the genome, and the human environment is very dynamic. If the Learning Subsystem has little pre-initialization and learns within a lifetime, then evolution didn't amortize much into that network. It amortized instead into innate behaviors and bootstrapping cost functions.
Why the genome can be so small
Patel says this framework resolves a puzzle he has raised with other guests. If evolution is like pretraining, how does a roughly 3-gigabyte genome, only a small fraction of which concerns the brain, specify an intelligence? If much of learning efficiency comes from well-designed reward functions, the puzzle eases: a reward function in Python can be a single line, and a thousand of them take little space. Marblestone agrees and adds that the Thought Assessor mechanism lets evolution get generalization cheaply. The genome doesn't have to anticipate the future. It has to specify which variables matter, heuristics for finding them, a compact learning algorithm and architecture, and then all the bespoke "Python code" about spiders, friends, mothers, mating, social groups, and joint eye contact.
As evidence, he points to single-cell atlases from researchers including Fei Chen and Evan Macosko, part of the scaling-up of neuroscience under the BRAIN Initiative. RNA sequencing can count cell types across regions, especially in the mouse brain, even without knowing the circuits. The finding he highlights is that subcortical, Steering Subsystem regions contain many more "weird and diverse and bespoke cell types" than the cortex. Cortex seems to have enough cell types to build a learning algorithm and set some hyperparameters. The Steering Subsystem has thousands of unusual cells, which "might be like the one for the spider flinch reflex and the one for I'm-about-to-taste-salt."
Why would each reward function need its own cell type? Marblestone's reasoning is that in the Learning Subsystem, all the action is in synaptic plasticity within a repeating architecture. The code for an eight-layer transformer is barely longer than for a three-layer one. Innately wired reward circuits instead need specific genetic wiring rules, such as "this neuron must synapse onto that one without any learning." He thinks that most likely happens through cells expressing distinct receptors and proteins that trigger synapse formation on contact, which requires distinct cell types. Patel notes that the Steering Subsystem still looks complicated, with its own vision, touch, and navigation systems. Marblestone agrees; that is precisely why so much genomic real estate would go there. Even a fly brain has innate circuits for orientation and optical flow relative to wind direction. In mammals, he would group such things into the Steering Subsystem.
Asked for numbers on how much of the genome this involves, Marblestone says he doesn't know and defers to biologists. He notes that much of the genome is shared even with yeast because it goes toward having a working cell at all. He adds that the differences between humans and chimpanzees, including social instincts and cortical differences, involve a tiny number of genes.
How the human brain grew
Patel proposes that this explains the rapid expansion of the hominid brain. If social learning raised the returns to learning ("the elder told me this is how you make a spear"), a bigger cortex became worth having, and growing one requires few genes because it replicates what a mouse already has. Marblestone agrees, with caveats about possible tweaks.
On whether omnidirectional inference is ancient or newly unlocked in primates, he says he isn't sure there's agreement. Language may involve macro-wiring changes, such as connecting auditory regions to memory regions and social instincts, and he mentions Broca's and Wernicke's areas and their links to the hippocampus and prefrontal cortex. Such changes could also take few genes. He leans toward the view that the potential was largely already there and what changed was the incentive to expand it and wire it to social instincts. He notes Suzana Herculano-Houzel's work showing that neuron count scales better with brain weight in primates than in rodents, which could suggest improved scalability. He says he isn't deep on this and doesn't rule out special features of human architecture.
Drawing on A Brief History of Intelligence, he says some form of learning goes back to anything with a brain, and primitive reinforcement learning goes back at least to vertebrates. Birds may have reinvented something cortex-like without six layers. Even the fly's mushroom body uses specific dopamine signals to train subgroups of neurons to associate sensory input with "Am I going to get food now?" or "Am I going to get hurt now?", which he says looks somewhat like a Thought Assessor.
Patel recalls a Beren Millidge post noting that visual and auditory cortex scaled disproportionately in humans compared with olfactory areas, attributed there to scaling-law properties of the data. Patel offers another reading: social reward functions needed to make use of seeing elders and hearing them. Marblestone connects this to the human eye's visible white sclera, which he says is designed for joint eye contact, and describes the first couple of years of life as bootstrapping toward detecting eye contact and communicating through language.
What kind of RL does the brain do?
Marblestone stresses that his answers are directional; what he wants is to map the whole mouse brain and make neuroscience a ground-truth science. He picks up Ilya's remark that it's strange LLMs don't use value functions. Current LLM RL upweights every token of a successful trajectory. He calls this conceptually "a really dumb form of RL," even compared with Q-learning for Atari a decade ago, while acknowledging how well it's being made to work on GPUs.
In the brain, he says, parts of the striatum and basal ganglia are thought to do something like model-free RL over a small, finite action space. Those actions might be motor commands to the brainstem or cognitive actions like telling the thalamus to let one cortical area talk to another, or releasing a memory from the hippocampus. Peter Dayan's work and related research show dopamine carrying a reward prediction error, consistent with learning value functions. He suggests this neuroscience helped motivate DeepMind's temporal-difference work. On top of that, the cortical world model can model when rewards will arrive, predicting the Steering Subsystem or the basal ganglia.
He also describes "RL as inference": clamp reward to high and sample plans that would lead to it. A general model-based system that includes plans and rewards gets this "for free." Patel likens this to a value head; Marblestone reframes it as a value input, with reward as one of the quasi-sensory variables the cortex predicts.
Patel brings up Joseph Henrich's work on cultural knowledge, such as complicated multi-step processes for making a poisonous bean edible, as model-free RL at civilizational scale. The two sketch a hierarchy: evolution, model-free; basal ganglia, model-free; cortex, model-based; culture, possibly model-free, though Marblestone notes culture also stores some of the model. Evolution shows that a completely unforesighted outer loop can produce all of this, which suggests simple algorithms can get you anything if run enough.
Is biological hardware a handicap?
Patel lays out the tradeoffs: the brain runs at about 200 hertz on 20 watts, yet fingers are far more dexterous than current motors, which might hint at a "cognitive dexterity" from unstructured sparsity and co-located memory and compute. Marblestone expects we'll eventually get the best of both worlds. The obvious downside of the brain is that it can't be copied and has no external read-write access to its synapses. Otherwise it may have many advantages, and it argues for co-designing algorithms with hardware: slow, low-voltage switches, co-located memory and compute, and tolerance for stochasticity. If the brain really does sampling-based inference, stochastic neurons are a natural fit, since they generate samples without a random number generator and can learn their probabilities.
Patel jokes that the takeaway may be that Yann LeCun and Beff Jezos were right. Marblestone says that is one reading. He notes he hasn't worked on AI since LLMs took off and is impressed by scaling, but thinks they are "kind of onto something" about probabilistic models, or at least possibly, and adds that this was what neuroscientists and AI people broadly thought until about 2021.
On whether cellular machinery beyond synapses does substantial computation, Marblestone doesn't believe the radical claims that memory mostly isn't in synapses or that learning is mostly genetic change. Much cellular activity, such as weight normalization or recycling a neuron after its memory has been consolidated elsewhere, is likely implementation overhead. A computer just edits a number; a cell has to do it "with molecular machines itself without any central controller." He does cite one more convincing case: in the cerebellum, which learns the delay between a flash and an air puff to the eyelid, the cell body appears to store the time constant rather than the delay being built from chains of synapses.
Different reward functions, different minds
Patel asks how different AGI might be if it has a general world model but different drives. Marblestone reframes this as whether a paperclip maximizer can actually be smart. Channeling Byrnes, he says the concern is that the minimal Steering Subsystem needed for capability is much smaller than what's needed for human-like social instincts and ethics. LLMs show that language can be learned without eye contact, given the right starting point. So he thinks it's probably possible to build powerful model-based RL systems lacking most human reward functions, and that is a concern. Patel notes such a system would still need drives like curiosity. Marblestone agrees but calls that set "pretty minimal," especially for a system that starts from a pretrained LLM. Most of the reason to understand the Steering Subsystem, in his reading of Byrnes, is alignment.
Do we even have the right vocabulary?
Patel observes that AI's borrowings from neuroscience mostly flow the other way: we invent backprop, CNNs, or TD learning, and then find something similar in the brain. Why assume our concepts are adequate? Marblestone says the evidence that we're onto something is that the resulting AIs work surprisingly well, and there is empirical support. CNNs, even pretrained on cat pictures, predict visual-cortex activity better than other computational neuroscience models, as measured by benchmarks like Jim DiCarlo's lab's Brain-Score.
He presents the opposing view fairly. György Buzsáki's The Brain from the Inside Out argues that our psychological and AI concepts are made up and that we must discover the brain's own primitives, with much of that work focused on oscillations. Marblestone thinks there's a case for this. He wants the research portfolio to include fully bottom-up efforts, simulating a worm or zebrafish as a physical dynamical system from connectome, molecules, and activity, alongside reverse engineering using AI vocabulary. His guess is that reverse engineering will "work-ish," citing TD learning, which Richard Sutton invented and which dopamine appears to implement in part.
What would a connectome actually tell us?
Patel challenges him: we have full access to LLM weights and still can't explain why they're intelligent. Why would a perfect brain map help? Marblestone partly disputes the premise. We can describe what an LLM fundamentally is (an architecture, a learning rule, hyperparameters, initialization, and training data), even if we know this because we built it. He wants to describe the brain in those terms rather than hunting for a "Golden Gate Bridge circuit." He cites Konrad Kording and Tim Lillicrap's paper "What does it mean to understand a neural network?", which argues that a network trained on something like the digits of pi or cellular automata will contain computations interpretability can never fully capture. What you can still say is how it got that way.
For the bottom-up approach, a connectome is essential raw material for simulation. For the reverse-engineering approach, it supplies a huge number of constraints for choosing among hypotheses: energy-based model or amortized VAE-style model, backprop or not, local or global learning rules. Even basic facts are unknown. How many kinds of dopamine or Steering Subsystem signals are there? Is prefrontal-to-auditory wiring the same as prefrontal-to-visual? Answering each through bespoke experiments takes a very long time, while a connectome gets them all at once. Patel notes that interpretability researchers can already study LLMs. Marblestone replies that you can't do interpretability on the hypothetical brain-like model-based RL system that AGI might eventually converge to.
Timelines, cost, and the Human Genome Project analogy
Marblestone concedes this research is not very relevant in an "AI 2027" scenario; in that world, the outcome of that scenario, not a connectome, determines science ten years from now. But he doesn't put much probability on that. His timelines are "more in the 10-year-ish range," partly because current AI "is weirdly different from all this brain stuff." If AlphaZero-style model-based RL had produced GPT-5-level capabilities, his prior and data would agree. As it is, the data looks good but his prior finds it odd, so he holds no strong opinion on whether LLMs can get there.
In a ten-year world, he thinks it matters whether we have connectomes on hard drives and comparisons of reward functions and architecture across mouse, shrew, and small primate. That would take "low billions-dollar scale funding in a very concerted way." E11 Bio, Convergent's main connectomics effort, aims to make brain mapping several orders of magnitude cheaper. He says a Wellcome Trust report a year or two ago estimated the first mouse connectome as a several-billion-dollar project, and E11 and others in the field are trying to bring that to low tens of millions. A human brain is about 1,000 times bigger, so naive scaling still means billions for one human brain. He isn't sure every neuron is needed. The targets he describes include a full mouse brain, a human Steering Subsystem, and whole brains of several mammals with different social instincts, which he thinks is feasible at hundreds of millions to low billions with focused effort.
Standard connectomics uses electron microscopy on very thin slices and mostly shows membranes. E11 and others have moved to optical microscopy, which doesn't damage tissue, so it can be washed and probed for fragile molecules. That yields a "molecularly annotated connectome": not only who connects to whom, but which molecules are at each synapse and which cell type each neuron is. He is careful to say this is not the same as synaptic weights or a simulation, though activity mapping could be added and an ML model trained to predict activity from structure.
On the Human Genome Project, Marblestone cites his PhD advisor George Church: the first genome cost about $3 billion, roughly $1 per base pair. The National Human Genome Research Institute then structured funding so companies competed to lower costs, and the cost fell about a million-fold in ten years as sequencing moved from macroscopic chemistry to massively parallel imaging of individual DNA molecules. E11, as a Focused Research Organization (FRO), deliberately started with technology development rather than brute force, though hundreds of millions would still be needed for data collection. Funding is "very TBD." E11 has been philanthropic, the National Science Foundation has announced an FRO-inspired "Tech Labs" call, and he's heard rumors of connectomics companies forming. He tried pitching AI labs seven or eight years ago without much interest; maybe now there would be. His argument: the questions discussed here, such as why the brain is energy-efficient or whether it does real or amortized inference, are "all answerable by neuroscience," and low billions is small next to trillions in GPUs. He sees moonshot startups as on a continuum with FROs, but warns that many "upload the brain" companies could skip the ground-truth science entirely.
Training AI on brain data
Patel raises an idea from a talk Marblestone gave five years ago, which Marblestone credits to a Gwern blog post. When training an image classifier, you would also predict the human neural activity recorded while viewing the image, as an auxiliary loss. Would that shape the network to represent cats and dogs the way the brain does, carry richer information than a one-hot label, generalize better, and perhaps resist adversarial examples? The obstacle is data. Labeled images are cheap, while brain activity is not. He calls this a technological accident: "we got GPUs before we got portable brain scanners." He distinguishes this regularization approach from distillation, mentioning Andreas Tolias's work on neural-network surrogate models of visual cortex as a closer analog to distilling part of the brain. He connects it back to Byrnes: the Learning Subsystem already predicts the Steering Subsystem as an auxiliary task, which then lets the Steering Subsystem build reward functions on top of that predictor.
Lean and automating mathematics
Marblestone sits on the board of Lean, which he says was developed for years at Microsoft and elsewhere and became one of Convergent's FROs. In Lean, a proof is written as a program, and the system checks whether the conclusions follow from the assumptions. That helps collaboration, since Terry Tao need not trust an amateur's result if Lean verifies it, and it makes proof correctness a perfect RLVR signal. He mentions Harmonic, which he describes as having at least a billion-dollar valuation, AlphaProof, and a few other emerging companies. He expects "RLVRing the crap out of math proving" to work, with proof search that resembles AlphaGo's search in Go.
That doesn't solve math. Conjecturing, judging what is interesting, and choosing high-level proof strategies remain. He mentions a theoretical paper by Yoshua Bengio and others on whether a loss function for good conjectures exists. A powerful theorem might be one that compresses many other results into short derivations, something like its effect on the Kolmogorov complexity of the rest of the proof network. Whether AI will mechanically prove the Riemann hypothesis, he says he doesn't know, and compares it to the unpredictability of when Go would fall.
He expects applied payoffs in formally verified software: proving that code cannot be hacked in certain ways or that user-accessible memory cannot affect other memory, and eventually asking an LLM to synthesize provably correct software. On why this hasn't taken off yet, he names the specification problem. Mathematicians know which theorems they want, but power-grid engineers may not know the formal security spec for their code, and there are few tools for writing one. Convergent is incubating a potential FRO on this. Formal methods have been an academic backwater, apart from efforts like a DARPA program that built a provably secure quadcopter, but he thinks current trends could "flip the tide."
On whether mathematicians will find this satisfying, he compares the change to the end of mandatory assembly programming: less grinding, fewer failures unrelated to whether your concept was right, and easier collaboration. He acknowledges the worry that skipping mechanical work might stunt intuition, draws the parallel to vibe coding, and says the answer isn't obvious, but his hunch is that it is "super positive." He also expects more outsiders to contribute, the way Byrnes has in neuroscience, perhaps even outsider string theorists.
Patel sketches a scenario: if AGI is a decade away, LLMs offer what Terence Tao called "automated cleverness but not automated intelligence," and formal proof could filter their output so it builds reliably, perhaps becoming the medium through which many AIs verify one another, including checking whether a message is trying to manipulate them. Marblestone mentions davidad's ARIA program in the UK on safeguarded AI, which relies heavily on provable safety properties, and the idea of world models written in equations rather than neuron activations, a move back toward symbolic methods powered by automated proving. He calls it an interesting vision without predicting it will play out within ten years, and notes Tao's work on proving theorems en masse to study the landscape of which ones get proved.
Open questions: representation, experience, continual learning
A round of questions yields mostly honest uncertainty. On whether the brain's world model is symbolic or more like a hidden state, Marblestone mentions face-patch neurons, hippocampal place cells, cognitive maps, and the variable-binding problem. His hunch is that "it's going to be a huge mess," probably not very symbolic, and that the productive path is studying architecture, loss functions, and learning rules. Others disagree, he notes. On the unity of conscious experience, he says he is "pretty much at a loss," mentions talks by Max Hodak, and says that nobody has any idea and it might even involve new physics.
On continual learning, he points to hippocampal replay and systems consolidation into cortex, possible multiple timescales of plasticity, and synapses with many states. He hasn't seen anything that clearly explains it. On whether the brain has an analog to a transformer's split between parameters and key-value activations, he says the brain certainly has weights and activations and several kinds of attention. He wonders whether the thalamus, which relays and gates cortico-cortical traffic, might do some key-value-like matching. His answer to all of this is "maybe, I don't know." He describes his own role as mostly "really unbiased data collection so all the other people can figure out these questions."
The Gap Map
Finally, Marblestone explains the Gap Map. While incubating FROs, Convergent talked with many scientists. Some described their next research project. Others described a missing piece of infrastructure that no combination of grad students and traditional grants could produce, a "miniature equivalent of the Hubble Space Telescope" that would lift the whole field without itself being a discovery. The Gap Map lists these; he says it is really more of a fundamental capabilities map.
What surprised him was the overall shape: a few hundred fundamental capabilities, which at deep-tech-startup or Series A scale would total a few billion dollars, not a trillion. He cautions that the map is not comprehensive and is really a summary of conversations. The bigger surprise was Lean. Coming from neuroscience and biology, needs like connectomics alongside genomics were obvious to him, but he hadn't realized math-proving infrastructure was a gap. Convergent is even finding unexplored gaps in astronomy, perhaps because projects above a certain size fall into more bureaucratic federal processes.
His conclusion is that small-scale creative work, serendipity, and students pursuing independent directions remain key, but "some amount of scalable infrastructure is missing in essentially every area of science, even math." He had assumed mathematicians just needed whiteboards; it turns out they also need verifiable programming languages.
The big million-dollar question that I have, that I've been trying to get the answer to through all these interviews with AI researchers: How does the brain do it? We're throwing way more data at these LLMs and they still have a small fraction of the total capabilities that a human does. So what's going on?
This might be the quadrillion-dollar question or something like that. You can make an argument that this is the most important question in science. I don't claim to know the answer. I also don't think that the answer will necessarily come even from a lot of smart people thinking about it as much as they are. My overall meta-level take is that we have to empower the field of neuroscience to just make neuroscience a more powerful field technologically and otherwise, to actually be able to crack a question like this.
Maybe the way that we would think about this now with modern AI, neural nets, deep learning, is that there are certain key components of that. There's the architecture. There's maybe hyperparameters of how many layers you have or properties of that architecture. There is the learning algorithm itself. How do you train it? Backprop, gradient descent, is it something else? How is it initialized? If we take the learning part of the system, it still may have some initialization of the weights. And then there are also cost functions. What is it being trained to do? What's the reward signal? What are the loss functions, supervision signals?
My personal hunch within that framework is that the field has neglected the role of these very specific loss functions, very specific cost functions. Machine learning tends to like mathematically simple loss functions. Predict the next token, cross-entropy, these simple computer scientist loss functions. I think evolution may have built a lot of complexity into the loss functions actually, many different loss functions for different areas turned on at different stages of development. A lot of Python code, basically, generating a specific curriculum for what different parts of the brain need to learn. Because evolution has seen many times what was successful and unsuccessful, and evolution could encode the knowledge of the learning curriculum. In the machine learning framework, maybe we can come back and we can talk about where do the loss functions of the brain come from? Can different loss functions lead to different efficiency of learning?
People say the cortex has got the universal human learning algorithm, the special sauce that humans have. What's up with that?
This is a huge question and we don't know. I've seen models where the cortex… The cortex typically has this six-layered structure, layers in a slightly different sense than layers of a neural net. Any one location in the cortex has six physical layers of tissue as you go in layers of the sheet. And those areas then connect to each other and that's more like the layers of a network.
I've seen versions of that where what you're trying to explain is just, "How does it approximate backprop?" And what is the cost function for that? What is the network being asked to do, if you are trying to say it's something like backprop? Is it doing backprop on next token prediction or is it doing backprop on classifying images or what is it doing? And no one knows. But one thought about it, one possibility about it, is that it's just this incredibly general prediction engine. So any one area of the cortex is just trying to predict… Basically can it learn to predict any subset of all the variables it sees from any other subset? Omnidirectional inference, or omnidirectional prediction.
Whereas an LLM is just seeing everything in the context window and then it computes a very particular conditional probability which is, "Given all the last thousands of things, what are the probabilities for the next token." But it would be weird for a large language model to say "the quick brown fox blank blank the lazy dog" and fill in the middle versus doing the next token, if it's doing just forward. It can learn how to do that stuff at this emergent level of the context window and everything, but natively it's just predicting the next token.
What if the cortex is natively made so that any area of cortex can predict any pattern in any subset of its inputs given any other missing subset? That is a little bit more like "probabilistic AI".
A lot of the things I'm saying, by the way, are extremely similar to what Yann LeCun would say. He's really interested in these energy-based models and something like that is like, the joint distribution of all the variables. What is the likelihood or unlikelihood of just any combination of variables? If I clamp some of them and I say that definitely these variables are in these states, then I can compute, with probabilistic sampling for example—conditioned on these being set in this state, and these could be any arbitrary subset of variables in the model—can I predict what any other subset is going to do and sample from any other subset given clamping this subset? And I could choose a totally different subset and sample from that subset. So it's omnidirectional inference.
And so there could be some parts of the cortex, there might be association areas of cortex that predict vision from audition. There might be areas that predict things that the more innate part of the brain is going to do. Because remember, this whole thing is riding on top of a lizard brain and lizard body, if you will. And that thing is a thing that's worth predicting too. You're not just predicting do I see this or do I see that. Is this muscle about to tense? Am I about to have a reflex where I laugh? Is my heart rate about to go up? Am I about to activate this instinctive behavior?
Based on my higher-level understanding… Like I can match somebody has told me there's a spider on my back to this lizard part that would activate if I was literally seeing a spider in front of me. You learn to associate the two so that even just from somebody hearing you say "There's a spider on your back"…
Well, let's come back to this. This is partly having to do with Steve Byrnes' theories, which I'm recently obsessed about. But on your podcast with Ilya, he said, "Look, I'm not aware of any good theory of how evolution encodes high-level desires or intentions."
I think this is very connected to all of these questions about the loss functions and the cost functions that the brain would use. And it's a really profound question, right? Let's say that I am embarrassed for saying the wrong thing on your podcast because I'm imagining that Yann LeCun is listening and he says, "That's not my theory. You described energy-based models really badly." That's going to activate in me innate embarrassment and shame, and I'm going to want to go hide and whatever. That's going to activate these innate reflexes. That's important because I might otherwise get killed by Yann LeCun's marauding army of other…
The French AI researchers are coming for you, Adam.
So it's important that I have that instinctual response. But of course, evolution has never seen Yann LeCun or known about energy-based models or known what an important scientist or a podcast is. Somehow the brain has to encode this desire to not piss off really important people in the tribe or something like this in a very robust way, without knowing in advance all the things that the Learning Subsystem of the brain, the part that is learning cortex and other parts… The cortex is going to learn this world model. It's going to include things like Yann LeCun and podcasts. And evolution has to make sure that those neurons, whatever the Yann-LeCun-being-upset-with-me neurons, get properly wired up to the shame response or this part of the reward function.
And this is important, right? Because if we're going to be able to seek status in the tribe or learn from knowledgeable people, as you said, or things like that, exchange knowledge and skills with friends but not with enemies… We have to learn all this stuff. It has to be able to robustly wire these learned features of the world, learned parts of the world model, up to these innate reward functions, and then actually use that to then learn more. Because next time I'm not going to try to piss off Yann LeCun if he emails me that I got this wrong. We're going to do further learning based on that. In constructing the reward function, it has to use learned information. But how can evolution, which didn't know about Yann LeCun, do that?
The basic idea that Steve Byrnes is proposing is that part of the cortex, or other areas like the amygdala that learn, what they're doing is they're modeling the Steering Subsystem. The Steering Subsystem is the part with these more innately programmed responses and the innate programming of these series of reward functions, cost functions, bootstrapping functions that exist. There are parts of the amygdala, for example, that are able to monitor what those parts do and predict what those parts do.
How do you find the neurons that are important for social status? Well, you have some innate heuristics of social status, for example, or you have some innate heuristics of friendliness that the Steering Subsystem can use. And the Steering Subsystem actually has its own sensory system, which is crazy. We think of vision as being something that the cortex does. But there's also a Steering Subsystem, subcortical visual system called the superior colliculus with innate ability to detect faces, for example, or threats. So there's a visual system that has innate heuristics and the Steering Subsystem has its own responses. There'll be part of the amygdala or part of the cortex that is learning to predict those responses.
What are the neurons that matter in the cortex for social status or for friendship? They're the ones that predict those innate heuristics for friendship. You train a predictor in the cortex and you say, "Which neurons are part of the predictor?" Those are the ones that, now you've actually managed to wire it up.
This is fascinating. I feel like I still don't understand… I understand how the cortex could learn how this primitive part of the brain would respond to… Obviously it has these labels on, "here's literally a picture of a spider, and this is bad, be scared of this." The cortex learns that this is bad because the innate part tells it that. But then it has to generalize to, "Okay, the spider's on my back. And somebody's telling me the spider's on your back. That's also bad."
Yes.
But it never got supervision on that. So how does it…?
Well, it's because the Learning Subsystem is a powerful learning algorithm that does have generalization, that is capable of generalization. The Steering Subsystem, these are the innate responses. You're going to have some built into your Steering Subsystem, these lower brain areas: hypothalamus, brainstem, et cetera. Again, they have their own primitive sensory systems. So there may be an innate response.
If I see something that's moving fast toward my body that I didn't previously see was there and is small and dark and high contrast, that might be an insect skittering onto my body. I am going to flinch. There are these innate responses. There's going to be some group of neurons, let's say, in the hypothalamus, that is the I-am-flinching or I-just-flinched neurons in the hypothalamus. When you flinch, first of all, it's a negative contribution to the reward function. You didn't want that to happen, perhaps.
But that's a reward function that doesn't have any generalization in it. I'm going to avoid that exact situation of the thing skittering toward me. Maybe I'm going to avoid some actions that lead to the thing skittering. That's a generalization you can get, what Steve calls downstream of the reward function.
I'm going to avoid the situation where the spider was skittering toward me, but you're also going to do something else. There's going to be a part of your amygdala, say, that is saying, "Okay, a few milliseconds, hundreds of milliseconds or seconds earlier, could I have predicted that flinching response?" It's going to be a group of neurons that is essentially a classifier of, "Am I about to flinch?" And I'm going to have classifiers for that for every important Steering Subsystem variable that evolution needs to take care of. Am I about to flinch? Am I talking to a friend? Should I laugh now? Is the friend high status? Whatever variables the hypothalamus, brainstem, contains… Am I about to taste salt? It's going to have all these variables and for each one it's going to have a predictor. It's going to train that predictor.
Now the predictor that it trains, that can have some generalization. The reason it can have some generalization is because it just has a totally different input. Its input data might be things like the word "spider", but the word "spider" can activate in all sorts of situations that lead to the word "spider" activating in your world model. If you have a complex world model with really complex features that inherently gives you some generalization. It's not just the thing skittering toward me, it's even the word "spider" or the concept of "spider" is going to cause that to trigger.
This predictor can learn that. Whatever spider neurons are in my world model, which could even be a book about spiders or somewhere, a room where there are spiders or whatever that is…
The amount of heebie-jeebies that this conversation is eliciting in the audience…
Now I'm activating your Steering Subsystem, your Steering Subsystem spider hypothalamus subgroup of neurons of skittering insects are activating based on these very abstract concepts in the conversation.
If you keep going, I'm going to put in a trigger warning.
That's because you learned this. The cortex inherently has the ability to generalize because it's just predicting based on these very abstract variables and all these integrated information that it has. Whereas the Steering Subsystem only can use whatever the superior colliculus and a few other sensors can spit out.
By the way, it's remarkable that the person who's made this connection between different pieces of neuroscience, Steve Byrnes, is a former physicist. For the last few years, he's been trying to synthesize—
He's an AI safety researcher. He's just synthesizing. This comes back to the academic incentives thing. I think that this is a little bit hard to say. What is the exact next experiment? How am I going to publish a paper on this? How am I going to train my grad student to do this? It's very speculative. But there's a lot in the neuroscience literature and Steve has been able to pull this together. And I think that Steve has an answer to Ilya's question essentially, which is, how does the brain ultimately code for these higher-level desires and link them up to the more primitive rewards?
Very naive question, but why can't we achieve this omnidirectional inference by just training the model to not just map from a token to next token, but remove the masks in the training so it maps every token to every token, or come up with more labels between video and audio and text so that it's forced to map one to each one?
I mean, that may be the way. It's not clear to me. Some people think that there's a different way that it does probabilistic inference or a different learning algorithm that isn't backprop. There might be other ways of learning, energy-based models or other things like that, that you can imagine that is involved in being able to do this and that the brain has that. But I think there's a version of it where what the brain does is crappy versions of backprop to learn to predict through a few layers and that it's kind of like a multimodal foundation model. LLMs are maybe just predicting the next token. But vision models maybe are trained in learning to fill in the blanks or reconstruct different pieces or combinations. But I think that
it does it in an extremely flexible way. If you train a model to just fill in this blank at the center, okay, that's great. But what if you didn't train it to fill in this other blank over to the left? Then it doesn't know how to do that. It's not part of its repertoire of predictions that are amortized into the network. Whereas with a really powerful inference system, you could choose at test time, what is the subset of variables it needs to infer and which ones are clamped?
Okay, two sub-questions. One, it makes you wonder whether the thing that is lacking in artificial neural networks is less about the reward function and more about the encoder or the embedding… Maybe the issue is that you're not representing video and audio and text in the right latent abstraction such that they could intermingle and conflict. Maybe this is also related to why LLMs seem bad at drawing connections between different ideas. Are the ideas represented at a level of generality at which you could notice different connections?
Well, the problem is these questions are all commingled. If we don't know if it's doing a backprop-like learning, and we don't know if it's doing energy-based models, and we don't know how these areas are even connected in the first place, it's very hard to really get to the ground truth of this. But yeah, it's possible. I think that people have done some work.
My friend Joel Dapello actually did something some years ago where he put a model—I think it was a model of V1, specifically how the early visual cortex represents images—as an input into a convnet and that improves some things. It could be differences. The retina is also doing motion detection and certain things are getting filtered out. There may be some preprocessing of the sensory data. There may be some clever combinations of which modalities are predicting which or so on, that lead to better representation. There may be much more clever things than that.
Some people certainly do think that there's inductive biases built in the architecture that will shape the representations differently or that there are clever things that you can do.
Astera, which is the same organization that employs Steve Byrnes, just launched this neuroscience project based on Doris Tsao's work. She has some ideas about how you can build vision systems that basically require less training. They build into the assumptions of the design of the architecture things like objects are bounded by surfaces and surfaces have certain types of shapes and relationships of how they occlude each other and stuff like that.
It may be possible to build more assumptions into the network. Evolution may have also put some changes of architecture. It's just I think that also the cost functions and so on may be a key thing that it does.
I want to talk about this idea that you just glanced off of which was amortized inference. Maybe I should try to explain what I think it means, because I think it's probably wrong and this will help you correct me.
It's been a few years for me too.
Right now, the way the models work is that you have an input, it maps it to an output, and this is amortizing a process, the real process, which we think is what intelligence is. It's that you have some prior over how the world could be, what are the causes that make the world the way that it is. And then when you see some observation, you should be like, "Okay, here's all the ways the world could be. This cause explains what's happening best." Now, doing this calculation over every possible cause is computationally intractable. So then you just have to sample like, "Oh, here's a potential cause. Does this explain this observation? No, forget it. Let's keep sampling." And then eventually you get the cause, then the cause explains the observation, and then this becomes your posterior.
That's actually pretty good. Bayesian inference in general is of this very intractable thing. The algorithms that we have for doing that tend to require taking a lot of samples, Monte Carlo methods, taking a lot of samples. And taking samples takes time. This is like the original Boltzmann machines and stuff. They're using techniques like this, and still it's used with probabilistic programming, other types of methods often. The Bayesian inference problem, which is basically the problem of perception, given some model of the world and given some data, how should I update my… What are the missing variables in my internal model?
And I guess the idea is that neural networks are hopefully… Obviously, mechanistically, the neural network is not starting with, "Here is my model of the world, and I'm going to try to explain this data." But the hope is that instead of starting with, "Hey, does this cause explain this observation? No. Did this cause explain this observation? Yes." What you do is just like observation… What's the cause that the neural net thinks is the best one? Observation to cause. So the feedforward goes observation to cause to then the output that…
You don't have to evaluate all these energy values or whatever and sample around to make them higher and lower. You just say, approximately that process would result in this being the top one or something like that.
Exactly. One way to think about it might be that test-time compute, inference-time compute is actually doing this sampling again. You literally read its chain of thought. It's actually doing this toy example we're talking about where it's like, "Oh, can I solve this problem by doing X? Nah, I need a different approach." This raises the question. Over time it is the case that the capabilities which required inference-time compute to elicit, get distilled into the model. So you're amortizing the thing which previously you needed to do these rollouts, these Monte Carlo rollouts, to figure out.
In general, maybe there's this principle that digital minds which can be copied, have different tradeoffs which are relevant, from biological minds which cannot. So in general, it should make sense to amortize more things because you can literally copy the amortization, or copy the things that you have sort of built in. This is a tangential question where it might be interesting to speculate about. In the future, as these things become more intelligent and the way we train them becomes more economically rational, what will make sense to amortize into these minds, which evolution did not think was worth amortizing into biological minds? You have to retrain every time.
First of all, I think the probabilistic AI people would be like, of course you need test-time compute, because this inference problem is really hard and the only ways we know how to do it involve lots of test-time compute. Otherwise it's just this crappy approximation that's never going to… You have to do infinite data or something to make this. I think some of the probabilistic people will be like, "No, it's inherently probabilistic and amortizing it in this way just doesn't make sense." They might then also point to the brain and say, "Okay, well the brain, the neurons are stochastic and they're sampling and they're doing things. So maybe the brain actually is doing more like the non-amortized inference, the real inference."
But it's also strange how perception can work in just milliseconds or whatever. It doesn't seem like it uses that much sampling. So it's also clearly doing some baking things into approximate forward passes or something like that to do this.
In the future, I don't know. Is it already a trend to some degree that things that people were having to use test-time compute for, are getting used to train back the base model? Now it can do it in one pass.
Maybe evolution did or didn't do that. I think evolution still has to pass everything through the genome to build the network and the environment in which humans are living is very dynamic. So maybe, if we believe this is true, there's a Learning Subsystem per Steve Byrnes, and a Steering Subsystem, that the Learning Subsystem doesn't have a lot of pre-initialization or pretraining. It has a certain architecture, but then within lifetime it learns. Then evolution didn't actually amortize that much into that network. It amortized it instead into a set of innate behaviors in a set of these bootstrapping cost functions, or ways of building up very particular reward signals.
This framework helps explain this mystery that people have pointed out and I've asked a few guests about, which is that if you want to analogize evolution to pretraining, well how do you explain the fact that so little information is conveyed through the genome? So 3 gigabytes is the size of the total human genome. Obviously a small fraction of that is actually relevant to coding the brain.
Previously people made this analogy, that actually evolution has found the hyperparameters of the model, the numbers which tell you how many layers there should be, the architecture, basically, how things should be wired together. But if a big part of the story is that increased sample efficiency aids learning, generally makes systems more performant, is the reward function, is the loss function—and if evolution found those loss functions that aid learning—then it actually makes sense how you can build an intelligence with so little information. Because the reward function, in Python the reward function is literally a line. So you just have a thousand lines like this, and that doesn't take up that much space.
Yes. It also gets to do this generalization thing with the thing I was describing where we were talking about the spider, where it learns just the word "spider" which triggers the spider reflex or whatever. It gets to exploit that too.
It gets to build a reward function that actually has a bunch of generalization in it just by specifying these innate spider stuff and the Thought Assessors, as Steve calls them, that do the learning. That's potentially a really compact solution to building up these more complex reward functions too, that you need.
It doesn't have to anticipate everything about the future of the reward function. It just has to anticipate what variables are relevant and what are heuristics for finding what those variables are. And then it has to have a very compact specification for the learning algorithm and basic architecture of the Learning Subsystem. And then it has to specify all this Python code of all the stuff about the spiders and all the stuff about friends, and all the stuff about your mother, and all the stuff about mating and social groups and joint eye contact. It has to specify all that stuff.
So is this really true? I think that there is some evidence for it.
Fei Chen and Evan Macosko and various other researchers have been doing these single-cell atlases. One of the things that scaling up neuroscience technology—again, this is one of my obsessions—has done through the BRAIN Initiative, a big neuroscience funding program, is they've basically gone through different areas, especially of the mouse brain, and mapped where the different cell types are? How many different types of cells are there in different areas of cortex? Are they the same across different areas? Then you look at these subcortical regions, which are more like the Steering Subsystem or reward-function-generating regions. How many different types of cells do they have? And which neuron types do they have? We don't know how they're all connected and exactly what they do or what the circuits are or what they mean, but you can just quantify how many different kinds of cells there are with sequencing the RNA.
And there are a lot more weird and diverse and bespoke cell types in the Steering Subsystem, basically, than there are in the Learning Subsystem. Like the cortical cell types, it seems like there's enough to build a learning algorithm up there and specify some hyperparameters. And in this Steering Subsystem, there's a gazillion, thousands of really weird cells, which might be like the one for the spider flinch reflex and the one for I'm-about-to-taste-salt.
Why would each reward function need a different cell type?
Well, this is where you get innately wired circuits. In the learning algorithm part, in the Learning Subsystem, you specify the initial architecture, you specify a learning algorithm. All the juice is happening through plasticity of the synapses, changes of the synapses within that big network. But it's a relatively repeating architecture, how it's initialized.
It's just like how the amount of Python code needed to make an eight-layer transformer is not that different from one that makes a three-layer transformer. You're just replicating. Whereas all this Python code for the reward function, if superior colliculus sees something that's skittering and you're feeling goosebumps on your skin or whatever, then trigger spider reflex, that's just a bunch of bespoke, species-specific, situation-specific crap. The cortex doesn't know about spiders, it just knows about layers.
But you're saying that the only way to write this reward function is to have a special cell type?
Yeah, well, I think so. I think you either have to have special cell types or you have to somehow otherwise get special wiring rules that evolution can say this neuron needs to wire to this neuron, without any learning. And the way that that is most likely to happen, I think, is that those cells express different receptors and proteins that say, "Okay, when this one comes in contact with this one, let's form a synapse." So it's genetic wiring, and those need cell types to do it.
I'm sure this would make a lot more sense if I knew 101 neuroscience, but it seems like there's still a lot of complexity, or generality rather, in the Steering Subsystem. So if the Steering Subsystem has its own visual system that's separate from the visual cortex, different features still need to plug into that vision system. So the spider thing needs to plug into it and also the love thing needs to plug into it, et cetera, et cetera. So it seems complicated.
It's still complicated. That's all the more reason why a lot of the genomic real estate on the genome, and in terms of these different cell types and so on, would go into wiring up the Steering Subsystem, pre-wiring it.
Can we tell how much of the genome is clearly working? So I guess you could tell how many are relevant to producing the RNA that manifest or the epigenetics that manifest in different cell types in the brain. Right?
Yeah. This is what the cell types help you get at. I don't think it's exactly like, "Oh, this percent of the genome is doing this", but you could say, "Okay, in all these Steering Subsystem subtypes, how many different genes are involved in specifying which is which and how they wire? And how much genomic real estate do those genes take up versus the ones that specify visual cortex versus auditory cortex?" You're just reusing the same genes to do the same thing twice. Whereas the spider reflex hooking up… Yes, you're right. They have to build a vision system and they have to build some auditory systems and touch systems and navigation-type systems.
Even feeding into the hippocampus and stuff like that, there's head direction cells. Even the fly brain has innate circuits that figure out its orientation and help it navigate in the world. It uses vision, figures out its optical flow of how it's flying and how its flight is related to the wind direction. It has all these innate stuff that I think in the mammal brain we would all lump that into the Steering Subsystem. There's a lot of work. So all the genes that basically go into specifying all the things a fly has to do, we're going to have stuff like that too, just all in the Steering Subsystem.
But do we have some estimate of like, "Here's how many nucleotides, here are many megabases it takes to—"
I don't know. I think you might be able to talk to biologists about this. We have a lot in common with yeast from a genes perspective.
Yeast is still used as a model for some amount of drug development and stuff like that in biology. And so much of the genome is just going towards you having a cell at all, it can recycle waste, it can get energy, it can replicate. And then what do we have in common with a mouse?
So we do know at some level that the differences between us and a chimpanzee or something—and that includes the social instincts and the more advanced differences in cortex and so on—it's a tiny number of genes that go into this additional amount of making the eight-layer transformer instead of the six-layer transformer or tweaking that reward function.
This would help explain why the hominid brain exploded in size so fast. Presumably, tell me if this is correct, but under this story, social learning or some other thing increased the ability to learn from the environment. It increased our sample efficiency. Instead of having to go and kill the boar yourself and figure out how to do that, you can just be like, "The elder told me this is how you make a spear." Now it increases the incentive to have a bigger cortex, which can learn these things.
Yes and that can be done with a relatively few genes, because it's really replicating what the mouse already has, making more of it. It's maybe not exactly the same and there may be tweaks, from a genome perspective, you don't have to reinvent all this stuff.
So then how far back in the history of the evolution of the brain does the cortex go back? Is the idea that the cortex has always figured out this omnidirectional inference thing, that's been a solved problem for a long time? Then the big unlock with primates is that we got the reward function, which increased the returns to having omnidirectional inference?
It's a good question. Or is the omnidirectional inference also something that took a while to unlock?
I'm not sure that there's agreement about that. I think there might be specific questions about language. Are there tweaks, whether that's through auditory and memory, some combination auditory memory regions? There may also be macro-wiring where you need to wire auditory regions into memory regions or something like that, and into some of these social instincts to get language, for example, to happen. But that might also be a small number of gene changes to be able to say, "Oh, I just need from my temporal lobe over here, going over to the auditory cortex, something." There is some evidence for the Broca's area, Wernicke's area. They're connected with the hippocampus and so on and prefrontal cortex. So there's like some small number of genes maybe for enabling humans to really properly do language. That could be a big one.
But is it that something changed about the cortex and it became possible to do these things? Or is that that potential was already there, but there wasn't the incentive to expand that capability and then use it, wire it to these social instincts and use it more?
I would lean somewhat toward the latter. I think a mouse has a lot of similarity in terms of cortex as a human.
Although there's Suzana Herculano-Houzel's work on how the number of neurons scales better with weight with primate brains than it does with rodent brains. So does that suggest that there actually was some improvement in the scalability of the cortex?
Maybe, maybe. I'm not super deep on this. There may have been changes in architecture, changes in the folding, changes in neuron properties and stuff that somehow slightly tweak this.
But there's still a scaling either way.
That's right. So I'm not saying there isn't something special about humans in the architecture of the Learning Subsystem at all. But yeah I think it's pretty widely thought that this is expanded. But then the question is, "Okay, well, how does that fit in also with the Steering Subsystem changes and the instincts that make use of this and allow you to bootstrap using this effectively?"
But just to say a few other things, even the fly brain has some amount, even very far back… I mean, I think you've read this great book, A Brief History of Intelligence, right? I think this is a really good book. Lots of AI researchers think this is a really good book it seems.
You have some amount of learning going back all the way to anything that has a brain. Basically you have something like primitive reinforcement learning, going back at least to vertebrates. Imagine a zebrafish. Then you have these other branches. Birds may have reinvented something cortex-like. It doesn't have the six layers, but they have something a little bit cortex-like. So some of those things after reptiles, in some sense birds and mammals both made a somewhat cortex-like, but differently organized thing.
But even a fly brain has associative learning centers that actually do things that maybe look a little bit like this Thought Assessor concept from Byrnes, where there's a specific dopamine signal to train specific subgroups of neurons in the fly mushroom body to associate different sensory information with, "Am I going to get food now?" or "Am I going to get hurt now?"
Brief tangent. I remember reading in one blog post that Beren Millidge wrote that the parts of the cortex which are associated with audio and vision have scaled disproportionately between other primates and humans, whereas the parts associated, say, with odor have not. And I remember him saying something like that this is explained by that kind of data having worse scaling law properties. Maybe he meant this, but I think another interpretation of actually what's happening there is that these social reward functions that are built into the Steering Subsystem needed to make use more of being able to see your elders and see what the visual cues are and hear what they're saying. And in order to make sense of these cues which guide learning, you needed to activate the vision and audio more than odor.
I mean, there's all this stuff. I feel like it's come up in your shows before, actually. But like even the design of the human eye where you have the pupil and the white and everything, we are designed to be able to establish relationships based on joint eye contact. Maybe this came up in the Sutton episode. I can't remember. But yeah, we have to bootstrap to the point where we can detect eye contact and where we can communicate by language. That's like what the first couple years of life are trying to do.
Okay, I want to ask you about RL. So currently, the way these LLMs are trained, if they solve the unit test or solve a math problem, that whole trajectory, every token in that trajectory is upweighted. What's going on with humans? Are there different types of model-based versus model-free that are happening in different parts of the brain?
Yeah, I mean, this is another one of these things. Again, all my answers to these questions, any specific thing I say, it's all just saying that directionally we can explore around this. I find this interesting, maybe I feel like the literature points in these directions in some very broad way. What I actually want to do is go and map the entire mouse brain and figure this out comprehensively and make neuroscience a ground-truth science. So I don't know, basically.
But first of all, I think with Ilya on the podcast, he was like, "It's weird that you don't use value functions, right?" You use the dumbest form of RL basically. Of course these people are incredibly smart and they're optimizing for how to do it on GPUs and it's really incredible what they're achieving. But conceptually it's a really dumb form of RL, even compared to what was being done 10 years ago.
Even the Atari game-playing stuff was using Q-learning, which is basically a kind of temporal difference learning. The temporal difference learning basically means you have some kind of a value function of what action I choose now doesn't just tell me literally what happens immediately after this. It tells me what is the long-run consequence of that for my expected total reward or something like that. So you would have value functions like… The fact that we don't have value functions at all in the LLMs is crazy. I think because Ilya said it, I can say it. I know one one-hundredth of what he does about AI, but it's kind of crazy that this is working.
But in terms of the brain, I think there are some parts of the brain that are thought to do something that's very much like model-free RL, that's parts of the striatum and basal ganglia. It is thought that they have a certain finite relatively small action space. The types of actions they could take, first of all, might be like, "Tell the brainstem and spinal cord to do this motor action? Yes or no." Or it might be more complicated cognitive-type actions like, "Tell the thalamus to allow this part of the cortex to talk to this other part," or "Release the memory that's in the hippocampus and start a new one or something." But there's some finite set of actions that come out of the basal ganglia, and that it's just a very simple RL. So there are probably parts of other brains and our brain that are just doing very simple naive-type RL algorithms.
Layering one thing on top of that is that some of the major work in neuroscience, like Peter Dayan's work, and a bunch of work that is part of why I think DeepMind did the temporal difference learning stuff in the first place. They were very interested in neuroscience. There's a lot of neuroscience evidence that the dopamine is giving this reward prediction error signal, rather than just reward, "yes or no, a gazillion time steps in the future." It's a prediction error and that's consistent with learning these value functions.
So there's that and then there's maybe higher-order stuff. We have the cortex making this world model. Well, one of the things the cortex world model can contain is a model of when you do and don't get rewards. Again, it's predicting what the Steering Subsystem will do. It could be predicting what the basal ganglia will do. You have a model in your cortex that has more generalization and more concepts and all this stuff that says, "Okay, these types of plans, these types of actions will lead in these types of circumstances to reward." So I have a model of my reward.
Some people also think that you can go the other way. So this is part of the inference picture. There's this idea of RL as inference. You could say, "Well, conditional on my having a high reward, sample a plan that I would have had to get there." That's inference of the plan part from the reward part. I'm clamping the reward as high and inferring the plan, sampling from plans that could lead to that. So if you have this very general cortical thing, it can just do. If you have this very general model-based system and the model, among other things, includes plans and rewards, then you just get it for free, basically.
So in neural network parlance, there's a value head associated to the omnidirectional inference that's happening in the—
Yes, or there's a value input.
Oh, okay. Interesting.
Yeah and it can predict. One of the almost sensory variables it can predict is what rewards it's going to get.
By the way, speaking about amortizing things, obviously value is like amortized rollouts of looking up reward.
Yeah, something like that. It's like a statistical average or prediction of it.
Tangential thought. Joe Henrich and others have this idea for the way human societies have learned to do things like, how do you figure out that this kind of bean, which actually just almost always poisons you, is edible if you do this ten-step incredibly complicated process, any one of which if you fail, the bean will be poisonous? How do you figure out how to hunt this seal in this particular way, with this particular weapon, at this particular time of the year, et cetera? There's no way but just like trying shit over generations. And it strikes me this is actually very much like model-free RL happening at a civilizational level.
No, not exactly. Evolution is the simplest algorithm in some sense. If we believe that all of this can come from evolution, the outer loop can be extremely not foresighted.
Right, that's interesting. Just hierarchies of… Evolution: model-free…
So what does that tell you? Maybe the simple algorithms can just get you anything if you do it enough.
Right. Yeah, I don't know.
So, evolution: model-free. Basal ganglia: model-free. Cortex: model-based. Culture: model-free potentially.
I mean you pay attention to your elders or whatever. Maybe there's like group selection or whatever of these things is like more model-free. But now I think culture, well, it stores some of the model.
Stepping back, is it a disadvantage or an advantage for humans that we get to use biological hardware, in comparison to computers as they exist now? What I mean by this question is, if there's "the algorithm", would the algorithm just qualitatively perform much worse or much better if inscribed in the hardware of today? The reason to think it might… Here's what I mean. Obviously the brain has had to make a bunch of tradeoffs which are not relevant to computing hardware. It has to be much more energetically efficient. Maybe as a result it has to run on slower speeds so that there can be a smaller voltage gap. So the brain runs at 200 hertz, it has to run on 20 watts. On the other hand, with robotics we've clearly experienced that fingers are way more nimble than we can make motors so far. So maybe there's something in the brain that is the equivalent of cognitive dexterity, which is maybe due to the fact that we can do unstructured sparsity. We can co-locate the memory and the compute.
Yes.
Where does this all net out? Are you like, "Fuck, we would be so much smarter if we didn't have to deal with these brains." Or are you like—
I think in the end we will get the best of both worlds somehow. I think an obvious downside of the brain is it cannot be copied. You don't have external read-write access to every neuron and synapse, whereas you do. I can just edit something in the weight matrix in Python or whatever and load that up and copy that. In principle. So the fact that it can't be copied and random-accessed is very annoying.
But otherwise maybe it has a lot of advantages. It also tells you that you want to somehow do the co-design of the algorithm. It maybe even doesn't change it that much from all of what we discussed, but you want to somehow do this co-design. So yeah, how do you do it with really slow low-voltage switches? That's going to be really important for energy consumption. Co-locating memory and compute. I think that hardware companies will probably just try to co-locate memory and compute.
They will try to use lower voltages, allow some stochastic stuff. There are some people that think that all this probabilistic stuff that we were talking about—"Oh, it's actually energy-based models, so on"—it is doing lots of sampling. It's not just amortizing everything. The neurons are also very natural for that because they're naturally stochastic. So you don't have to do a random number generator in a bunch of Python code basically to generate a sample. The neuron just generates samples and it can tune what the different probabilities are and learn those tunings. So it could be that it's very co-designed with some kind of inference method or something.
It'd be hilarious… I mean the message I'm taking from this interview is that like all these people that folks make fun of on Twitter, Yann LeCun and Beff Jezos and whatever, I don't know maybe they got it right.
That is actually one read of it. Granted, I haven't really worked on AI at all since LLMs took off, so I'm just out of the loop. But I'm surprised and I think it's amazing how the scaling is working and everything. But yeah, I think Yann LeCun and Beff Jezos are kind of onto something about the probabilistic models or at least possibly. In fact that's what all the neuroscientists and all the AI people thought until 2021 or something.
Right. So there's a bunch of cellular stuff happening in the brain that is not just about neuron-to-neuron synaptic connections. How much of that is functionally doing more work than the synapses themselves are doing versus it's just a bunch of kludge that you have to do in order to make the synaptic thing work? So with a digital mind, you can nudge the synapse, sorry, the parameter, extremely easily. But with a cell to modulate a synapse according to the gradient signal, it just takes all this crazy machinery. So is it actually doing more than it takes extremely little code to do?
I don't know, but I'm not a believer in the radical, "Oh, actually memory is not synapses mostly, or learning is mostly genetic changes" or something like that. I think it would just make a lot of sense, I think you put it really well, for it to be more like the second thing you said. Let's say you want to do weight normalization across all the weights coming out of your neuron or into your neuron. Well, you probably have to somehow tell the nucleus of the cell about this and then have that send everything back out to the synapses or something. So there's going to be a lot of cellular changes.
Or let's say that you just had a lot of plasticity and you're part of this memory. Now that's got consolidated into the cortex or whatever. Now we want to reuse you as a new one that can learn again. There's going to be a ton of cellular changes, so there's going to be tons of stuff happening in the cell. But algorithmically, it's not really adding something beyond these algorithms. It's just implementing something that in a digital computer is very easy for us to go and just find the weights and change them. In a cell, it just literally has to do all this with molecular machines itself without any central controller. It's kind of incredible. There are some things that cells do, I think, that seem more convincing.
One of the things the cerebellum has to do is predict over time. What is the time delay? Let's say that I see a flash and then some number of milliseconds later, I'm going to get a puff of air in my eyelid or something. The cerebellum can be very good at predicting what's the timing between the flash and the air puff, so that now your eye will just close automatically. The cerebellum is involved in that type of reflex, learned reflex. There are some cells in the cerebellum where it seems like the cell body is playing a role in storing that time constant, changing that time constant of delay, versus that all being somehow done with, "I'm going to make a longer ring of synapses to make that delay longer." No, the cell body will just store that time delay for you.
So there are some examples, but I'm not a believer out of the box in essentially this theory that what's happening is changes in connections between neurons and that that's the main algorithmic thing that's going on. I think there's very good reason to still believe that it's that rather than some crazy cellular stuff.
Going back to this whole perspective of how our intelligence is not just this omnidirectional inference thing that builds a world model, but really this system that teaches us what to pay attention to, what the important salient factors are to learn from, et cetera. I want to see if there's some intuition we can derive from this about what different kinds of intelligences might be like. So it seems like AGI or superhuman intelligence should still have this ability to learn a world model that's quite general, but then it might be incentivized to pay attention to different things that are relevant for the modern post-singularity environment. How different should we expect different intelligences to be?
I think one way to think about this question is, is it actually possible to make the paperclip maximizer or whatever? If you try to make the paperclip maximizer, does it end up just not being smart or something like that because the only reward function it had was to make paperclips? I'd say, can you do that? I don't know. If I channel Steve Byrnes more, I think he's very concerned that the minimum viable things in the Steering Subsystem that you need to get something smart is way less than the minimum viable set of things you need for it to have human-like social instincts and ethics and stuff like that. So a lot of what you want to know about the Steering Subsystem is actually the specifics of how you do alignment essentially, or what human behavior and social instincts is versus just what you need for capabilities.
We talked about it in a slightly different way because we were sort of saying, "Well, in order for humans to learn socially, they need to make eye contact and learn from others." But we already know from LLMs that depending on your starting point, you can learn language without that stuff. So I think that it probably is possible to make super powerful model-based RL optimizing systems and stuff like that that don't have most of what we have in the human brain reward functions and as a consequence might want to maximize paperclips. And that's a concern.
But you're pointing out that in order to make a competent paperclip maximizer, the kind of thing that can build spaceships and learn physics and whatever, it needs to have some drives which elicit learning, including say curiosity and exploration.
Yeah, curiosity, interest in others, interest in social interactions. But that's pretty minimal, I think. And that's true for humans, but it might be less true for something that's already pretrained as an LLM or something. So most of why we want to know the Steering Subsystem, I think, if I'm channeling Steve, is alignment reasons.
How confident are we that we even have the right algorithmic conceptual vocabulary to think about what the brain is doing? What I mean by this is that there was one big contribution to AI from neuroscience, which was this idea of the neuron in the 1950s, just this original contribution. But then it seems like a lot of what we've learned afterwards about what the high-level algorithm the brain is implementing, from backprop, to if there's something analogous to backprop happening in the brain, to "Oh, is V1 doing something like CNNs," to TD learning and Bellman equations, actor-critic, whatever… It seems inspired by this dynamic where we come up with some idea, maybe we can make AI neural networks work this way, and then we notice that something in the brain also works that way. So why not think there's more things like this?
There may be. I think the reason that I think that we might be onto something is that the AIs we're making based on these ideas are working surprisingly well. There's also a bunch of just empirical stuff. Convolutional neural nets and variants of convolutional neural nets. I'm not sure what the absolute latest is, but compared to other models in computational neuroscience of what the visual system is doing, they are just more predictive. You can just score, even pretrained on cat pictures and stuff, CNNs, what is the representational similarity that they have on some arbitrary other image compared to the brain activations measured in different ways? Jim DiCarlo's lab has this Brain-Score and the AI model is actually… There seems to be some relevance there. Neuroscience doesn't necessarily have something better than that.
So yes, that's just recapitulating what you're saying, that the best computational neuroscience theories we have seem to have been invented largely as a result of AI models and finding things that work. So find backprop works and then saying, "Can we approximate backprop with cortical circuits?" or something. There's been things like that. Now, some people totally disagree with this.
György Buzsáki is a neuroscientist who has a book called The Brain from Inside Out where he basically says all our psychology concepts, AI concepts, all this stuff is just made-up stuff. What we actually have to do is figure out what is the actual set of primitives that the brain actually uses. And our vocabulary is not going to be adequate to that. We have to start with the brain and make new vocabulary rather than saying backprop and then try to apply that to the brain or something like that. He studies a lot of oscillations and stuff in the brain as opposed to individual neurons and what they do. I don't know. I think that there's a case to be made for that.
And from a research program design perspective, one thing we should be trying to do is just simulate a tiny worm or a tiny zebrafish, almost as biophysical or as bottom-up as possible. Like get connectome, molecules, activity and just study it as a physical dynamical system and look at what it does. But I don't know, it just feels like AI is really good fodder for computational neuroscience. Those might actually be pretty good models. We should look at that. I both think that there should be a part of the research portfolio that is totally bottom-up and not trying to apply our vocabulary that we learn from AI onto these systems, and that there should be another big part of this that's trying to reverse engineer it using that vocabulary or a variant of that vocabulary. We should just be pursuing both.
My guess is that the reverse engineering one is actually going to work-ish or something. Like we do see things like TD learning, which Sutton also invented separately.
That must be a crazy feeling to just like— Yeah, that's crazy. This equation I wrote down is like in the brain. It seems like the dopamine is doing some of that, yeah.
So let me ask you about this. You guys are funding different groups that are trying to figure out what's up in the brain. If we had a perfect representation, however you define it, of the brain, why think it would actually let us figure out the answer to these questions? We have neural networks which are way more interpretable, not just because we understand what's in the weight matrices, but because there are weight matrices. There are these boxes with numbers in them. Even then we can tell very basic things. We can kind of see circuits for very basic pattern matching of following one token with another. I feel like we don't really have an explanation of why LLMs are intelligent just because they're interpretable.
I think I would somewhat dispute it. We have some description of what the LLM is fundamentally doing. What that's doing is that I have an architecture and I have a learning rule and I have hyperparameters and I have initialization and I have training data. But those are things we learned because we built them, not because we interpreted them from seeing the weights. The analogous thing to connectome is like seeing the weights. What I think we should do is we should describe the brain more in that language of things like architectures, learning rules, initializations, rather than trying to find the Golden Gate Bridge circuit and saying exactly how this neuron actually… That's going to be some incredibly complicated learned pattern.
Konrad Kording and Tim Lillicrap have this paper from a while ago, maybe five years ago, called "What does it mean to understand a neural network?" What they say is basically that you could imagine you train a neural network to compute the digits of pi or something. It's like some crazy pattern. You also train that thing to predict the most complicated thing you find, predict stock prices, basically predict really complex systems, computationally complete systems. I could train a neural network to do cellular automata or whatever crazy thing. It's like, we're never going to be able to fully capture that with interpretability, I think. It's just going to be doing really complicated computations internally. But we can still say that the way it got that way is that it had an architecture and we gave it this training data and it had this loss function. So I want to describe the brain in the same way. And I think that this framework that I've been laying out is that we need to understand the cortex and how it embodies a learning algorithm. I don't need to understand how it computes "Golden Gate Bridge."
But if you can see all the neurons, if you have the connectome, why does that teach you what the learning algorithm is?
Well, I guess there are a couple different views of it. So it depends on these different parts of this portfolio. On the totally bottom-up, we-have-to-simulate-everything portfolio, it kind of just doesn't. You have to make a simulation of the zebrafish brain or something and then you see what are the emergent dynamics in this and you come up with new names and new concepts and all that. That's the most extreme bottom-up neuroscience view.
But even there the connectome is really important for doing that biophysical or bottom-up simulation. But on the other hand you can say, "Well, what if we can actually apply some ideas from AI?" We basically need to figure out, is it an energy-based model or is it an amortized VAE-type model? Is it doing backprop or is it doing something else? Are the learning rules local or global? If we have some repertoire of possible ideas about this, just think of the connectome as a huge number of additional constraints that will help to refine, to ultimately have a consistent picture of that.
I think about this for the Steering Subsystem stuff too, just very basic things about it. How many different types of dopamine signal or of Steering Subsystem signal or thought assessor or so on… How many different types of what broad categories are there? Even this very basic information that there's more cell types in the hypothalamus than there are in the cortex, that's new information about how much structure is built there versus somewhere else. How many different dopamine neurons are there? Is the wiring between prefrontal and auditory the same as the wiring between prefrontal and visual? The most basic things, we don't know. The problem is learning even the most basic things by a series of bespoke experiments takes an incredibly long time. Whereas just learning all that at once by getting a connectome is just way more efficient.
What is the timeline on this? Presumably the idea of this is, first, to inform the development of AI. You want to be able to figure out how we get AIs to want to care about what other people think of its internal thought pattern. But interp researchers are making progress on this question just by inspecting normal neural networks. There must be some feature…
You can do interp on LLMs that exist. You can't do interp on a hypothetical model-based reinforcement algorithm like the brain that we will eventually converge to when we do AGI.
Fair. But what timelines on AI do you need for this research to be practical and relevant?
I think it's fair to say it's not super practical and relevant if you're in an AI 2027 scenario. And so what science I'm doing now is not going to affect the science of ten years from now, because what's going to affect the science of ten years from now is the outcome of this AI 2027 scenario. It kind of doesn't matter that much probably if I have the connectome, maybe it slightly tweaks certain things. But I think there's a lot of reason to think maybe that we will get a lot out of this paradigm. But then the real thing, the single event that is transformative for the entire future, that type of event, is still more than five years away or something.
Is that because we haven't captured omnidirectional inference, we haven't figured out the right ways to get a mind to pay attention to things in a way that makes sense?
I mean, I would take the entirety of your collective podcast with everyone as showing the distribution of these things. I don't know. What was Karpathy's timeline, right? What's Demis's timeline? So not everybody has a three-year timeline.
But there are different reasons and I'm curious which ones are yours. What are mine?
I don't know, I'm just watching your podcast. I'm trying to understand the distribution. I don't have a super strong claim that LLMs can't do it.
But is the crux the data efficiency or…?
I think part of it is just that it is weirdly different from all this brain stuff. So intuitively it's just weirdly different than all this brain stuff and I'm kind of waiting for the thing that starts to look more like brain stuff. I think if AlphaZero, and model-based RL and all these other things that were being worked on 10 years ago, had been giving us the GPT-5 type capabilities, then I would be like, "Oh wow, we're both in the right paradigm and seeing the results a priori. So my prior and my data are agreeing." Now it's like, "I don't know what exactly my data is. Looks pretty good, but my prior is sort of weird so I don't have a super strong opinion on it."
So I think there's a possibility that essentially all other scientific research that is being done is somehow obviated. But I don't put a huge amount of probability on that. I think my timelines might be more in the 10-year-ish range. If that's the case, I think there is probably a difference between a world where we have connectomes on hard drives and we have an understanding of Steering Subsystem architecture. We've compared even the most basic properties of what are the reward functions, cost function, architecture, et cetera, of a mouse versus a shrew versus a small primate, et cetera.
Is this practical in 10 years?
I think it has to be a really big push.
How much funding, how does it compare to where we are now?
It's like low billions-dollar scale funding in a very concerted way I would say.
And how much is on it now?
So if I just talk about some of the specific things we have going on with connectomics… E11 Bio is our main thing on connectomics. They are trying to make the technology of connectomic brain mapping several orders of magnitude cheaper. The Wellcome Trust put out a report a year or two ago that said to get one mouse brain, the first mouse brain connectome would be a several billion dollars project.
E11 technology, and the suite of efforts in the field, is trying to get a single mouse connectome down to low tens of millions of dollars. That's a mammal brain. A human brain is about 1,000 times bigger. If with a mouse brain you can get to $10 million or $20 million, $30 million, with technology, if you just naively scale that, a human brain is now still billions of dollars, to just do one human brain. Can you go beyond that? Can you get a human brain for less than a billion? But I'm not sure you need every neuron in the human brain. We want to, for example, do an entire mouse brain and a human Steering Subsystem and the entire brains of several different mammals with different social instincts. So with a bunch of technology push and a bunch of concerted effort, real significant progress if it's focused effort can be done in the hundreds of millions to low billions scale.
What is the definition of a connectome? Presumably it's not a bottom-up biophysics model. So is it just that it can estimate the input-output of a brain? What is the level of abstraction?
You can give different definitions and one of the things that's cool… So the standard approach to connectomics uses the electron microscope and very, very thin slices of brain tissue. It's basically labeling. The cell membranes are going to show up, scatter electrons a lot and everything else is going to scatter electrons less. But you don't see a lot of details of the molecules, which types of synapses, different synapses of different molecular combinations and properties.
E11 and some other research in the field has switched to an optical microscope paradigm. With optical, the photons don't damage the tissue, so you can wash it and look at fragile gentle molecules. So with E11's approach, you can get a "molecularly annotated connectome." That's not just who is connected to who by some synapse, but what are the molecules that are present at the synapse? What type of cell is that?
A molecularly annotated connectome, that's not exactly the same as having the synaptic weights. That's not exactly the same as being able to simulate the neurons and say what's the functional consequence of having these molecules and connections. But you can also do some amount of activity mapping and try to correlate structure to function. Train an ML model basically to predict the activity from the connectome.
What are the lessons to be taken away from the Human Genome Project? One way you could look at it is that it was a mistake and you shouldn't have spent billions of dollars getting one genome mapped. Rather you should have just invested in technologies which have now allowed us to map genomes for hundreds of dollars.
Well, George Church was my PhD advisor and he's pointed out that it was $3 billion or something, roughly $1 per base pair for the first genome. Then the National Human Genome Research Institute basically structured the funding process right. They got a bunch of companies competing to lower the cost. And then the cost dropped like a million-fold in 10 years because they changed the paradigm from macroscopic chemical techniques to these individual DNA molecules which would make a little cluster of DNA molecules on the microscope and you would see just a few DNA molecules at a time on each pixel of the camera. It would give you a different, in parallel, look at different fragments of DNA. So you parallelize the thing by millions-fold. That's what reduced the cost by millions-fold.
With switching from electron microscopy to optical connectomics, potentially even future types of connectomics technology, we think there should be similar patterns. That's why E11, the Focused Research Organization, started with technology development rather than starting with saying we're going to do a human brain or something and let's just brute force it. We said let's get the cost down with new technology. But then it's still a big thing. Even with new next-generation technology, you still need to spend hundreds of millions on data collection.
Is this going to be funded with philanthropy, by governments, by investors?
This is very TBD and very much evolving in some sense as we speak. I'm hearing some rumors going around of connectomics-related companies potentially forming. So far E11 has been philanthropy. The National Science Foundation just put out this call for Tech Labs, which is somewhat FRO-inspired or related. You could have a tech lab for actually going and mapping the mouse brain with us and that would be philanthropy plus government still in a nonprofit, open-source framework. But can companies accelerate that? Can you credibly link connectomics to AI in the context of a company and get investment for that? It's possible.
I mean the cost of training these AIs is increasing so much. If you could tell some story of not only are we going to figure out some safety thing, but in fact once we do that, we'll also be able to tell you how AI works… You should go to these AI labs and just be like, "Give me one one-hundredth of your projected budget in 2030."
I sort of tried a little bit seven or eight years ago and there was not a lot of interest. Maybe now there would be. But all the things that we've been talking about, it's really fun to talk about, but it's ultimately speculation. What is the actual reason for the energy efficiency of the brain, for example? Is it doing real inference or amortized inference or something else? This is all answerable by neuroscience. It's going to be hard, but it's actually answerable. So if you can only do that for low billions of dollars or something to really comprehensively solve that, it seems to me, in the grand scheme of trillions of dollars of GPUs and stuff, it actually makes sense to do that investment.
Also, there's been many labs that have been launched in the last year where they're raising on the valuation of billions for things which are quite credible but are not like, "Our ARR next quarter is going to be whatever." It's like we're going to discover materials and—
Yes, moonshot startups or billionaire-backed startups.
Moonshot startups I see as on a continuum with FROs. FROs are a way of channeling philanthropic support and ensuring that it's open source public benefit, various other things that may be properties of a given FRO. But yes, billionaire-backed startups, if they can target the right science, the exact right science. I think there's a lot of ways to do moonshot neuroscience companies that would never get you the connectome. It's like, "Oh, we're going to upload the brain" or something, but never actually get the mouse connectome or something. These fundamental things that you need to get to ground truth the science. There are lots of ways to have a moonshot company go wrong and not do the actual science. But there also may be ways to have companies or big corporate labs get involved and actually do it correctly.
This brings to mind an idea that you had in a lecture you gave five years ago about. Do you want to explain behavior cloning?
Actually this is funny because the first time I saw this idea, I think it might have been in a blog post by Gwern.
There's always a Gwern blog post.
There are now academic research efforts and some amount of emerging company-type efforts to try to do this.
Normally, let's say I'm training an image classifier or something. I show it pictures of cats and dogs or whatever and they have the label "cat" or "dog". And I have a neural network that's supposed to predict the label "cat" or "dog" or something. That is a limited amount of information per label that you're putting in. It's just "cat" or "dog".
What if I also had, "Predict what is my neural activity pattern when I see a cat or when I see a dog and all the other things?" If you add that as an auxiliary loss function or an auxiliary prediction task, does that sculpt the network to know the information that humans know about cats and dogs and to represent it in a way that's consistent with how the brain represents it and the kind of representational dimensions or geometry of how the brain represents things, as opposed to just having these labels? Does that let it generalize better? Does that let it have richer labeling?
Of course that sounds really challenging. It's very easy to generate lots and lots of labeled cat pictures. Scale AI or whatever can do this. It is harder to generate lots and lots of brain activity patterns that correspond to things that you want to train the AI to do. But again, this is just a technological limitation of neuroscience. If every iPhone was also a brain scanner, you would not have this problem and we would be training AI with the brain signals. It's just the order in which technology has developed is that we got GPUs before we got portable brain scanners.
What is the ML analog, what you'd be doing here? Because when you distill models, you're still looking at the final layer of the log probs across all—
If you distill one model into another, that is a certain thing. You are just trying to copy one model into another. I think that we don't really have a perfect proposal to distill the brain. To distill the brain you need a much more complex brain interface. Maybe you could also do that. You could make surrogate models. Andreas Tolias and people like that are doing some amount of neural network surrogate models of brain activity data. Instead of having your visual cortex do the computation, just have the surrogate model. So you're distilling your visual cortex into a neural network to some degree. That's a kind of distillation.
This is doing something a little different. This is basically just saying I'm adding an auxiliary… I think of it as regularization or I think of it as adding an auxiliary loss function that's smoothing out the prediction task to also always be consistent with how the brain represents it. It might help you with things like adversarial examples, for example.
But what exactly are you predicting? You're predicting the internal state of the brain?
Yes. So in addition to predicting the label, a vector of labels like yes cat, not dog, yes, not boat, one-hot vector or whatever of yes, it's cat, instead of these gazillion other categories, let's say in this simple example. You're also predicting a vector which is all these brain signal measurements. So Gwern, anyway, had this long-ago blog post of like, "Oh, this is an intermediate thing. We talk about whole brain emulation, we talk about AGI, we talk about brain-computer interface. We should also be talking about this brain-data-augmented thing that's trained on all your behavior, but is also trained on predicting some of your neural patterns."
And you're saying the Learning System is already doing this through the Steering System?
Yeah, and our brain, our learning system also has to predict the Steering Subsystem as an auxiliary task. That helps the Steering Subsystem. Now, the Steering Subsystem can access that predictor and build a cool reward function using it.
Separately, you're on the board of Lean, which is this formal math language that mathematicians use to prove theorems and so forth. Obviously there's a bunch of conversation right now about AI automating math. What's your take?
Well, I think that there are parts of math that it seems like it's pretty well on track to automate. First of all, Lean was developed for a number of years at Microsoft and other places. It has become one of the Convergent Focused Research Organizations to kind of drive more engineering and focus onto it.
So Lean is this programming language where instead of expressing your math proof on pen and paper, you express it in this programming language Lean. And then at the end, if you do that that way, it is a verifiable language so that you can click "verify" and Lean will tell you whether the conclusions of your proof actually follow perfectly from your assumptions of your proof. So it checks whether the proof is correct automatically.
By itself, this is useful for mathematicians collaborating and stuff like that. If I'm some amateur mathematician and I want to add to a proof, Terry Tao is not going to just believe my result. But if Lean says it's correct, it's just correct. So it makes it easy for collaboration to happen, but it also makes it easy for correctness of proofs to be an RL signal in very much RLVR. Formalized math proving—so formal means it's expressed in something like Lean and verifiable—is now mechanically verifiable. That becomes a perfect RLVR task.
I think that that is going to just keep working, it seems like there is at least one billion-dollar valuation company, Harmonic, based on this. AlphaProof is based on this. A couple other emerging really interesting companies. I think that this problem of RLVRing the crap out of math proving is going to work and we will be able to have things that search for proofs and find them in the same way that we have AlphaGo or what have you that can search for ways of playing the game of Go. With that verifiable signal, it works.
So does this solve math? There is still the part that has to do with conjecturing new interesting ideas. There's still the conceptual organization of math of what is interesting. How do you come up with new theorem statements in the first place? Or even the very high-level breakdown of what strategies you use to do proofs. I think this will shift the burden of that so that humans don't have to do a lot of the mechanical parts of math. Validating lemmas and proofs and checking if the statement of this in this paper is exactly the same as that paper and stuff like that. That will just work.
If you really think we're going to get all these things we've been talking about, real AGI would also be able to make conjectures. Bengio has a paper, more like a theoretical paper. There are probably a bunch of other papers emerging about this. Is there a loss function for
good explanations or good conjectures? That's a pretty profound question. A really interesting math proof or statement might be one that compresses lots of information and has lots of implications for lots of other theorems. Otherwise you would have to prove those theorems using long complex paths of inference. Here, if you have this theorem, this theorem is correct, and you have short paths of inference to all the other ones. And it's a short compact statement. So it's like a powerful explanation that explains all the rest of math. And part of what math is doing is making these compact things that explain the other things.
It's like the Kolmogorov complexity of this statement or something.
Yeah, of generating all the other statements, given that you know this one or stuff like that. Or if you add this, how does it affect the complexity of the rest of the network of proofs? So can you make a loss function that adds, "Oh, I want this proof to be a really highly powerful proof"? I think some people are trying to work on that. So maybe you can automate the creativity part.
If you had true AGI, it would do everything a human can do. So it would also do the things that the creative mathematicians do. But barring that, I think just RLVRing the crap out of proofs, I think that's going to be just a really useful tool for mathematicians. It's going to accelerate math a lot and change it a lot, but not necessarily immediately change everything about it.
Will we get mechanical proof of the Riemann hypothesis or something like that, or things like that? Maybe, I don't know. I don't know enough details of how hard these things are to search for, and I'm not sure anyone can fully predict that, just as we couldn't exactly predict when Go would be solved or something like that.
I think it's going to have lots of really cool applied applications. So one of the things you want to do is you want to have provably stable, secure, unhackable software. So you can write math proofs about software and say, "This code, not only does it pass these unit tests, but I can mathematically prove that there's no way to hack it in these ways, or no way to mess with the memory", or these types of things that hackers use, or it has these properties.
You can use the same Lean and same proof to do formally verified software. I think that's going to be a really powerful piece of cybersecurity that's relevant for all sorts of other AI hacking the world stuff. And if you can prove the Riemann hypothesis, you're also going to be able to prove insanely complex things about very complex software. And then you'll be able to ask the LLM, "Synthesize me a software that I can prove is correct."
Why hasn't provable programming language taken off as a result of LLMs?
I think it's starting to. One challenge—we are actually incubating a potential Focused Research Organization on this—is the specification problem. So mathematicians know what interesting theorems they want to formalize. Let's say I have some code that is involved in running the power grid or something and it has some security properties, well what is the formal spec of those properties? The power grid engineers just made this thing, but they don't necessarily know how to lift the formal spec from that. And it's not necessarily easy to come up with the spec that is the spec that you want for your code. People aren't used to coming up with formal specs and there are not a lot of tools for it. So you also have this user interface plus AI problem of what security specs should I be specifying? Is this the spec that I wanted? So there's a spec problem and it's just been really complex and hard.
But it's only just in the last very short time that the LLMs are able to generate verifiable proofs of things that are useful to mathematicians, starting to be able to do some amount of that for software verification, hardware verification. But I think if you project the trends over the next couple years, it's possible that it just flips the tide. Formal methods, this whole field of formal methods or formal verification, provable software. It's this weird almost backwater of the more theoretical part of programming languages and stuff, very academically flavored often. Although there was this DARPA program that made a provably secure quadcopter helicopter and stuff like that.
Secure against… What is the property that is exactly proved? Not for that particular project, but just in general. Because obviously things malfunction for all kinds of reasons.
You could say that what's going on in this part of the memory over here, which is supposed to be the part the user can access, can't in any way affect what's going on in the memory over here or something like that. Things like that.
So there's two questions. One is how useful is this? Two is, how satisfying, as a mathematician, would it be? The fact that there's this application towards proving that software has certain properties or hardware has certain properties, if that works, that would obviously be very useful. But from a pure… Are we going to figure out mathematics? Is your sense that there's something about finding that one construction cross-maps to another construction in a different domain, or finding that, "Oh, this lemma, if you redefine this term, it still satisfies what I meant by this term. But a counterexample that previously knocked it down no longer applies." That kind of dialectical thing that happens in mathematics. Will the software replace that? Yeah. How much of the value of this sort of pure mathematics just comes from actually just coming up with entirely new ways of thinking about a problem, mapping it to a totally different representation? Do we have examples?
I don't know. I think of it maybe a little bit like when everybody had to write assembly code or something like that. The amount of fun cool startups that got created was just a lot less or something. Fewer people could do it, progress was more grinding and slow and lonely and so on. You had more false failures because you didn't get something about the assembly code, rather than the essential thing of was your concept right. Harder to collaborate and stuff like that. And so I think it will be really good.
There is some worry that by not learning to do the mechanical parts of the proofs that you fail to generate the intuitions that inform the more conceptual parts, the creative part. It's the same with assembly.
Right. So at what point is that applying? With vibe coding, are people not learning computer science or actually are they vibe coding and they're also simultaneously looking at the LLM that's explaining these abstract computer science concepts to them and it's all just all happening faster? Their feedback loop is faster and they're learning way more abstract computer science and algorithm stuff because they're vibe coding. I don't know, it's not obvious. That might be something about the user interface and the human infrastructure around it. But I guess there's some worry that people don't learn the mechanics and therefore don't build the grounded intuitions or something. But my hunch is it's super positive.
Exactly, on net, how useful that will be or how much overall math breakthroughs, or math breakthroughs even that we care about, will happen? I don't know. One other thing that I think is cool is the accessibility question. Okay, that sounds a little bit corny. Okay, yeah, more people can do math, but who cares? But I think there's lots of people that could have interesting ideas. Like maybe the quantum theory of gravity or something.
Yeah, one of us will come up with the quantum theory of gravity instead of a card-carrying physicist.
In the same way that Steve Byrnes is reading the neuroscience literature and he hasn't been in the neuroscience lab that much. But he's able to synthesize across the neuroscience literature and be like, "Oh, Learning Subsystem, Steering Subsystem. Does this all make sense?" He's an outsider neuroscientist in some ways. Can you have outsider string theorists or something, because the math is just done for them by the computer? And does that lead to more innovation in string theory? Maybe yes.
Interesting. Okay, so if this approach works and you're right that LLMs are not the final paradigm, and suppose it takes at least 10 years to get the final paradigm in that world. There's this fun sci-fi premise where you have… Terence Tao today had a tweet where he's like, "These models are like automated cleverness but not automated intelligence." And you can quibble with the definitions there. But if you have automated cleverness and you have some way of filtering—which if you can formalize and prove things that the LLMs are saying you could do—then you could have this situation where quantity has a quality all of its own. So what are the domains of the world which could be put in this provable symbolic representation?
So in the world where AGI is super far away, maybe it makes sense to literally turn everything the LLMs ever do, or almost everything they do, into super provable statements. So LLMs can actually build on top of each other because everything they do is super provable. Maybe this is just necessary because you have billions of intelligences running around. Even if they are super intelligent, the only way the future AGI civilization can collaborate with each other is if they can prove each step. They're just brute force churning out… This is what the Jupiter brains are doing.
It's a universal language, it's provable. It's also provable from the perspective of, "Are you trying to exploit me or are you sending me some message that's trying to hack into my brain effectively?" Are you trying to socially influence me? Are you actually just sending me just the information that I need and no more for this? So davidad, who's this program director at ARIA now in the UK, he has this whole design of an ARPA-style program, a sort of safeguarded AI that very heavily leverages provable safety properties.
Can you apply proofs to… Can you have a world model? But that world model is actually not specified just in neuron activations, but it's specified in equations. Those might be very complex equations, but if you can just get insanely good at just auto-proving these things with cleverness, auto-cleverness… Can you have explicitly interpretable world models as opposed to neural net world models and move back basically to symbolic methods just because you can just have insane amount of ability to prove things?
Yeah, I mean that's an interesting vision. I don't know in the next 10 years whether that will be the vision that plays out, but I think it's really interesting to think about. Even for math, I mean, Terence Tao is doing some amount of stuff where it's not about whether you can prove the individual theorems. It's like let's prove all the theorems en masse and then let's study the properties of the aggregate set of proved theorems. Which are the ones that got proved and which are the ones that didn't? Okay, well that's the landscape of all the theorems instead of one theorem at a time.
Speaking of symbolic representations, one question I was meaning to ask you is, how does the brain represent the world model? Obviously nets out in neurons, but I don't mean extremely functionally. I mean conceptually, is it in something that's analogous to the hidden state of a neural network or is it something that's closer to a symbolic language?
We don't know. There's some amount of study of this. There's these things like face patch neurons that represent certain parts of the face that geometrically combine in interesting ways. That's with geometry and vision. Is that true for other more abstract things? There's this idea of cognitive maps. A lot of the stuff that a rodent hippocampus has to learn is place cells and, where is the rodent going to go next and is it going to get a reward there? It's very geometric. And do we organize concepts with an abstract version of a spatial map?
There's some questions of can we do true symbolic operations? Can I have a register in my brain that copies a variable to another register regardless of what the content of that variable is? That's this variable binding problem. Basically I don't know if we have that machinery or is it more like cost functions and architectures that make some of that approximately emerge, but maybe it would also emerge in a neural net? There's a bunch of interesting neuroscience research trying to study this, what the representations look like.
But what's your hunch?
Yeah, my hunch is that it's going to be a huge mess and we should look at the architecture, the loss functions, and the learning rules. I don't expect it to be pretty in there.
Which is that it is not a symbolic language type thing?
Yeah, probably it's not that symbolic. But other people think very differently.
Another random question speaking of binding, what is up with feeling like there's an experience? All the parts of your brain which are modeling very different things, have different drives, and at least presumably feel like there's an experience happening right now. Also that across time you feel like…
Yeah, I'm pretty much at a loss on this one. I don't know. Max Hodak has been giving talks about this recently. He's another really hardcore neuroscience person, neurotechnology person. The thing I mentioned with Doris maybe also sounds like it might have some touching on this question. But yeah, I don't think anybody has any idea. It might even involve new physics.
Here's another question which might not have an answer yet. Continual learning, is that the product of something extremely fundamental at the level of even the learning algorithm? You could say, "Look, at least the way we do backprop in neural networks is that you freeze the weight, there's a training period and you freeze the weights. So you just need this active inference or some other learning rule in order to do continual learning." Or do you think it's more a matter of architecture and how memory is exactly stored and what kind of associative memory you have basically?
So continual learning… I don't know. At the architectural level, there's probably some interesting stuff that the hippocampus is doing. People have long thought this. What kinds of sequences is it storing? How is it organizing, representing that? How is it replaying it back? What is it replaying back? How exactly does that memory consolidation work? Is it training the cortex using replays or memories from the hippocampus or something like that? There's probably some of that stuff.
There might be multiple timescales of plasticity or clever learning rules that can simultaneously be storing short-term information and also doing backprop with it. Neurons may be doing a couple things: some fast weight plasticity and some slower plasticity at the same time, or synapses that have many states. I mean, I don't know. From a neuroscience perspective, I'm not sure that I've seen something that's super clear on what causes continual learning except maybe to say that this systems consolidation idea of hippocampus consolidating cortex. Some people think it is a big piece of this and we still don't fully understand the details.
Speaking of fast weights, is there something in the brain which is the equivalent of this distinction between parameters and activations that we see in neural networks? Specifically in transformers we have this idea that some of the activations are the key and value vectors of previous tokens that you build up over time. There's the so-called fast weights that whenever you have a new token, you query them against these activations, but you also obviously can't query them against all the other parameters in the network which are part of the actual built-in weights. Is there some such distinction that's analogous?
I don't know. I mean we definitely have weights and activations.
Whether you can use the activations in these clever ways, different forms of actual attention, like attention in the brain… Is that based on, "I'm trying to pay attention"...
I think there's probably several different kinds of actual attention in the brain. I want to pay attention to this area of visual cortex. I want to pay attention to the content in other areas that is triggered by the content in this area. Attention that's just based on reflexes and stuff like that. So I don't know. There's not just the cortex, there's also the thalamus. The thalamus is also involved in somehow relaying or gating information. There's cortico-cortical connections. There's also some amount of connection between cortical areas that goes through the thalamus. Is it possible that this is doing some sort of matching or constraint satisfaction or matching across keys over here and values over there?
Is it possible that it can do stuff like that? Maybe. I don't know. This is all part of the architecture of this corticothalamic system. I don't know how transformer-like it is or if there's anything analogous to that attention. It'd be interesting to find out.
We've got to give you a billion dollars so you can come on the podcast again and tell me how exactly the brain works. Mostly I just do data collection. It's really unbiased data collection so all the other people can figure out these questions.
Maybe the final question to go off on is, what was the most interesting thing you learned from the Gap Map? Maybe you want to explain what the Gap Map is.
In the process of incubating and coming up with these Focused Research Organizations, these nonprofit startup-like moonshots that we've been getting philanthropists and now government agencies to fund, we talked to a lot of scientists.
Some of the scientists were just like, "Here's the next thing my graduate student will do. Here's what I find interesting. Exploring these really interesting hypothesis spaces, all the types of things we've been talking about." Some of them were like, "Here's this gap. I need this piece of infrastructure. There's no combination of grad students in my lab or me loosely collaborating with other labs with traditional grants that could ever get me that. I need to have an organized engineering team that builds the miniature equivalent of the Hubble Space Telescope. If I can build that Hubble Space Telescope, then I will unblock all the other researchers in my field or some path of technological progress in the way that the Hubble Space Telescope lifted the boats and improved the life of every astronomer."
But it wasn't really an astronomy discovery in itself. It was just that you had to put this giant mirror in space with a CCD camera and organize all the people and engineering and stuff to do that. So some of the things we talked to scientists about looked like that. The Gap Map is just a list of a lot of those things and we call it a Gap Map. I think it's actually more like a fundamental capabilities map. What are all these things, like mini Hubble Space Telescopes? And then we organized that into gaps for helping people understand that or search that. What was the most surprising thing you found?
I think I've talked about this before, but one thing is just the overall size or shape of it or something like that. It's a few hundred fundamental capabilities. So if each of these were a deep tech startup-size project, that's only a few billion dollars or something. If each one of those were a Series A, that's only… It's not like a trillion dollars to solve these gaps. It's lower than that. So that's one thing. Maybe we assumed that, and that's what we got. It's not really comprehensive. It's really just a way of summarizing a lot of conversations we've had with scientists.
I do think that in the aggregate process, things like Lean are actually surprising because I did start from neuroscience and biology and it was very obvious that there's these -omics. We need genomics, but we also need connectomics. We can engineer E. coli, but we also need to engineer the other cells. There's somewhat obvious parts of biological infrastructure. I did not realize that math proving infrastructure was a thing and that was emergent from trying to do this.
So I'm looking forward to seeing other things where it's not actually this hard intellectual problem to solve it. It's maybe slightly the equivalent of AI researchers just needing GPUs or something like that and focus and really good PyTorch code to start doing this. Which are the fields that do or don't need that? So fields that have had gazillions of dollars of investment, do they still need some of those? Do they still have some of those gaps or is it only more neglected fields?
We're even finding some interesting ones in actual astronomy, actual telescopes that have not been explored. Maybe because if you're getting above a critical mass-size project, then you have to have a really big project and that's a more bureaucratic process with the federal agencies. I guess you just need scale in every single domain of science these days. Yeah, I think you need scale in many of the domains of science. That does not mean that the low-scale work is not important. It does not mean that creativity, serendipity, etc., and each student pursuing a totally different direction or thesis that you see in universities is not also really key. But I think some amount of scalable infrastructure is missing in essentially every area of science, even math, which is crazy. Because mathematicians I thought just needed whiteboards, but they actually need Lean. They actually need verifiable programming languages and stuff. I didn't know that.
Cool. Adam, this is super fun. Thanks for coming on. Thank you so much. My pleasure. Where can people find your stuff?
Pleasure. The easiest way now… My adammarblestone.org website is currently down, I guess. But convergentresearch.org can link to a lot of the stuff we've been doing. And then you have a great blog, Longitudinal Science. Longitudinal Science, yes, on WordPress. Cool. Thank you so much. Pleasure.
Article published