Blaise Agüera y Arcas: Life Is Computational, and Intelligence Grows Through Merging
The Long Now Foundation
In this Long Now Talk, Blaise Agüera y Arcas, VP and Fellow at Google and founder of the Paradigms of Intelligence team, argues against several common assumptions. The first is that artificial general intelligence still lies ahead of us. The second is that computation is foreign to life. The third is that the arrival of AI must mean humans get replaced. His alternative account holds that life has been computational from the start, that it grows more complex mainly through symbiogenesis (the merging of previously independent entities), and that intelligence is fundamentally social prediction, meaning the modeling of oneself and others. On this view, AI is the latest in a long series of symbioses, not a rival species. The talk was followed by a conversation with Benjamin Bratton of Antikythera and audience questions.
From "narrow AI" to the shock of scale
Agüera y Arcas has been at Google for about 11 years. For most of that time he ran Cerebra, a Google Research group that grew from a small team to several hundred people. Cerebra mostly did applied AI: the Now Playing song recognizer, face unlock on Pixel phones, and some of the models behind the Google keyboard's next-word prediction.
He says that while doing this work he assumed it wasn't "really" AI. It was artificial narrow intelligence (ANI). He explains that the term existed because the original 1950s meaning of AI, robots you could have an interesting conversation with, never arrived during the 20th century. There were repeated "AI winters" after program-based approaches failed. Neural nets eventually began to handle limited tasks such as face recognition and next-word prediction, and "narrow" separated those from the real thing. The G in AGI stood for everything else. He quotes computer scientist Ben Goertzel's "core AGI hypothesis": synthetic intelligences with sufficiently broad, human-level scope are qualitatively different from those with narrower scope. In other words, AGI is not ANI.
Agüera y Arcas says he believed this too. He expected that some insight would come from neuroscience, the source of every key advance in narrow AI so far, and that studying brains would reveal "the trick" behind real intelligence. He thought neural nets were at least on the right track because they were brain-inspired.
"I was wrong," he says. His first inkling came from outputs of Meena, a scaled-up next-word predictor not unlike the Android keyboard model, only much bigger and trained on an unprecedented amount of data. He showed Meena attempting to define philosophy in conversation. It was only filling in the next word statistically, yet its outputs could be compared with human ones for sensibleness and relevance. That was a shock. It suggested the key to general intelligence might simply be scale.
He admits he had been "quite snobbish" about the Silicon Valley idea that everything comes down to making things bigger. With his training in neuroscience and physics, worshipping "at the altar of Moore's law" seemed naive. "But the nerds were right." LaMDA, a further scale-up in 2021, could hold open-domain conversations about almost anything. He says he wishes Google had launched it before OpenAI launched its model, invoking the Innovator's Dilemma. It sometimes produced nonsense, but, he notes, people do too. By his account, model sizes grew by about 1.25× per year from 1950 until the unsupervised learning revolution, when the growth rate jumped to about 3.72× per year, where it has stayed since.
Has AGI already arrived?
From this history he concludes that the core AGI hypothesis is wrong, and he acknowledges this is controversial. He offers a thought experiment. Take any of today's frontier models back to around 2000, when "AGI" was coined to distinguish it from narrow AI. Would the people who coined it say "you've arrived"? "Of course they would have," he says.
He then asks why this hasn't been acknowledged. He gives several reasons. Everyone expected a trick, and there wasn't one. Progress has been exponential but continuous, with no clear before and after. We still don't fully understand why scaling works. He also suggests some reasons may involve "our insecurities." He calls these "frog boiling" reasons. The provocation he sets up for the rest of the talk is whether humans, too, might be massively scaled next-token predictors. He proposes that the answer may be yes.
Life defined by function
To explain why a brain would evolve to be computational, Agüera y Arcas widens the frame. One of his team's big insights, he says, is that life itself is computational, not only the brain. He acknowledges this sounds odd, since life seems squishy, wet, and unreliable, nothing like a program.
He sketches the history. In the 19th century, vitalism held that some spirit animated living matter. It was displaced by strong materialism, which says physics is the same for atoms in living bodies and in rocks, so there is no fundamental difference between living and non-living matter. He finds that unsatisfying too, because there clearly seems to be a difference. His answer is function.
His test for function is whether something can be broken. Split a rock in half and you have two rocks, not a broken rock. Break a kidney and you have a non-working kidney. He imagines a time traveler bringing back an artificial kidney with a 100-year lifetime that filters urea like a real one. That statement means something concrete: if you have kidney failure, you will live. Yet nothing about the material tells you this. It could be made of carbon nanotubes or tungsten filaments. What makes it a kidney is its relationships with the rest of the body. Function is therefore relational, "kind of ecological," beyond the physical matter, yet fully constrained by physics. There is no "kidney spirit." It just works, and "works means functions."
Turing, von Neumann, and the universal constructor
He credits Alan Turing and John von Neumann with pioneering function as a fundamental idea. He showed a physical Turing machine built by Mike Davey in 2010 and described the concept. A head moves along a tape, reading, writing, and erasing symbols according to a rule table. Any calculation doable with pen and paper can be done by some Turing machine. Turing's deeper 1936 insight was that certain rule tables can read another rule table written on the tape and compute whatever that machine would have computed. That is a universal computer, a machine that runs programs. It follows that many different programs, in many languages, can compute the same thing. They are functionally equivalent.
According to Agüera y Arcas, von Neumann went further by introducing embodied computation through cellular automata. These are grid worlds with simple local "physics," Conway's Game of Life being an example. In them, the tape and the head are made of the same stuff as what is written. Von Neumann was thinking about life. His framing: imagine you are a robot made of Legos paddling on a pond of loose Legos, and you want to build another robot like yourself. That is what every mother, every plant seed, and every dividing bacterium must do, and it seems paradoxical to build something as complex as yourself from parts.
Von Neumann concluded that such a system needs three things. It needs a tape of instructions. It needs a "universal constructor" that walks along the tape and builds whatever is written there. It needs a tape copier. And the instructions for building the constructor and the copier must themselves be on the tape. Agüera y Arcas stresses that von Neumann predicted this around 1950, before the discovery of DNA's structure and function (the tape), the ribosome (the constructor), and DNA polymerase (the copier).
He calls the most important point this: von Neumann showed that the universal constructor is a universal Turing machine, one that computes with the very matter it is made of. So reproduction requires universal computation: "no computation, no life." He says most biologists and most computer scientists are still unaware of this insight.
BFF: watching life emerge from noise
A couple of years ago his team began experiments, since published, on how minimal von Neumann-style life could emerge from non-life. He needed a very minimal Turing-like language. He chose Brainfuck, admitting he enjoys saying the name in front of an audience because he's "a 12-year-old on the inside." The language has only eight instructions, including move the head left, move it right, increment the byte at the head, and decrement it. It is almost unreadable, but in principle it is universal: "you could write Microsoft Windows in this if you were non-human."
The experiment is called BFF, and he leaves the name's meaning "as an exercise to the listener." It starts with tapes of 64 random bytes. In the version he showed there were 8,192 tapes, though he says many experiments work with about a thousand. Since only eight of the 256 possible byte values are instructions, only about one byte in 32 does anything. The rest are no-ops. At each step, two tapes are drawn at random, joined end to end, and run. They are then separated and returned to the soup, and the process repeats.
At first almost nothing happens. On average only about two instructions execute per interaction, and there are no loops. When he let it run on his laptop, programs suddenly emerged. They were complex, full of loops, and needed reverse-engineering to understand. They were clearly reproducing: the most common tape had about 5,000 copies, the next 297, the next 99. The average number of operations per interaction had risen to 4,784. The soup had gone from noise to something complex, computational, and functional. Functional here means breakable, like the kidney. Change one instruction and the tape stops copying itself, and it gets overwritten by something that does copy itself.
He says this shows why life persists. In a universe that supports computation, whatever copies itself will exist in the future. He cites the old joke that DNA is the most stable molecule in the universe despite being fragile, because it reproduces. A robust but non-functional chunk of granite can at best take a long time to fall apart.
A phase transition into "life"
He plotted the first 10 million interactions of one run, with time on the horizontal axis and the number of operations per interaction on the vertical. Around 6 million interactions a sudden "wall of white" appears. This image is on the cover of his book. He interprets it as a phase transition. Before it, the soup is like a gas: every byte is uncorrelated with every other, and the soup can't be compressed with zip. Right after it, the soup compresses to about 5% of its original size, which is expected when there is a lot of copying.
If the earlier phase is gas, he says, the later phase is life, or "machine phase." He describes life as a special phase of matter because, unlike solids, liquids, or gases, it has structure at every scale. His tentative conclusion is that almost any universe with a source of randomness that can support computation will evolve life.
The puzzle of speed, and Lynn Margulis's answer
The results raised a puzzle. How do such complicated programs arise in only a few million steps from about a thousand 64-byte tapes? The early experiments included mutation, random byte changes like cosmic rays. But life still emerged with mutation turned down to zero, only a little more slowly. And complexity kept increasing after self-copying appeared, with new structure and more code arriving over time. Why doesn't evolution stop once replication exists?
His answer draws on Lynn Margulis. In 1967, after rejections from many journals, Margulis published "On the Origin of Mitosing Cells" in the Journal of Theoretical Biology, establishing that mitochondria were once free-living bacteria. Margulis popularized symbiogenesis: previously independent life forms merging into a new one. In this case an archaeon and a bacterium combined to form the eukaryotic cell. Margulis also believed symbiogenesis was the main engine of evolution. The establishment eventually accepted the mitochondria claim but not the larger thesis, and Margulis remained in a small minority on it until dying in 2011.
Agüera y Arcas says symbiogenesis is exactly what happens in BFF. The evidence is in tracking small reproducing strings, not whole tapes. Even from the start, an instruction occasionally changes a value elsewhere in the soup into another instruction, "a very very lame but nonzero form of reproduction." When all the reproducers in a soup are mapped, the lineage looks less like a branching tree of descent and more like tree roots running the other way: small replicators come together and form larger ones. That, he says, is how complexity arises.
Symbiogenesis as evolution's arrow of time
He argues that symbiogenesis gives evolution a direction. Standard Darwinian evolution improves fit to a niche but has no built-in bias toward more or less complexity. A beak changes shape to suit a flower, and on average complexity change is roughly zero. When two already-reproducing entities merge and reproduce together, new information must be added about how they fit together. That added information is what drives complexity upward.
He cites the 1995 Nature article by Eörs Szathmáry and John Maynard Smith identifying eight major evolutionary transitions, such as the move from single eukaryotic cells to multicellular organisms, which he calls obviously symbiogenetic. Szathmáry and Maynard Smith have since added to the list. If BFF is any indication, he says, these are only the largest cases of a continuous cascade of mergers that drives the complexification of life.
For biological evidence, which he calls an ongoing area of work, he points to the human genome. When it was sequenced in 2001, only about 1.5% turned out to code for proteins. The rest, so-called junk DNA, is not really junk. Some is regulatory, and the function of some is unknown. Large portions (LTR retrotransposons, DNA transposons, LINEs and SINEs) are, as he puts it, basically viruses: replicators that got into heritable DNA and became part of the genome. Some endogenized viral elements do important work. He says the placenta is made using a virus that fuses cell membranes. Knocking out a viral gene called Arc in mice stops them forming memories. Parts of the immune system arose this way too. Roughly a few dozen such results have appeared in the past 10–15 years, and more keep coming. Our DNA, he says, looks like "a medley of things that have copied and fused over and over."
Parallelism, prediction, and energy
Combining these ideas, he argues that because every living thing computes, each symbiogenetic merger joins two computers into a parallel one. Symbiogenesis therefore makes life's computation increasingly parallel, producing a kind of Moore's law. He compares this with the history of silicon. From 1950 to 2006, Moore's law meant shrinking transistors, and he says AI made no progress in that period. Around 2006, shrinking transistors stopped yielding faster clocks, so chipmakers began putting more cores on each chip. That is when AI took off, which he says is no coincidence, since parallelism is what neural-net AI needs.
He distinguishes two kinds of modeling. Computing for growth, healing, and replication means modeling your own body, and that is life. Modeling your environment is intelligence. Life was intelligent from the start, he says, because an organism must also find the parts to build more of itself. The Legos don't just float by. Computation is also energetically costly: it creates negative entropy and requires free energy, which is why organisms metabolize and why his laptop heats up running BFF. In simulations where random programs can move up, down, left, or right and get energy from light, the survivors are the ones that learn to follow the light. He compares this with footage of bacteria swimming toward a sugar crystal.
A multiplayer world: theory of mind and consciousness
He says he had been describing life in "single player mode," but where there is one bacterium there will soon be more, "in 11 minutes." Life is always multiplayer, and the most important parts of the environment to model are other agents. They have their own interests, which can align or conflict with yours, and they are modeling you back.
He described recent team work, too new to be in the book, called multi-agent universal predictive intelligence. It addresses multi-agent reinforcement learning, where individually rewarded learners must learn to work together on problems such as the prisoner's dilemma. He says this is very hard for classical reinforcement learning, which learns from the past and assumes a stationary environment. When other players adapt to your strategy, you must predict them, predict that they are learning and predicting you, and so on recursively. The paper has grown to around 100 pages of complex math. According to Agüera y Arcas, the solution is to stop treating yourself as outside the game. Unlike a system such as AlphaGo, which imagines the game itself, the agent must model itself playing the game as part of the environment, predicting both itself and others. He relates this to having a face. He knows what his smile feels like from inside, so when he sees someone else smile he can infer they are happy. That inference depends on knowing we are similar.
From this he offers his view of consciousness. He rejects the idea that it is an epiphenomenon, and rejects the possibility of a "philosophical zombie" that behaves identically but is dead inside. He argues we are conscious because we model ourselves, others, and others' models of us, and that this is functionally essential for cooperation. That cooperation is in turn what makes symbiosis and symbiogenesis possible. He does not claim that cells are conscious. He says they have models of neighboring cells in order to collaborate, which may be "a baby step" toward consciousness.
He says human intelligence as we usually describe it, such as transplanting organs, going to the moon, or building chips, is something no individual can do. It is the "superhuman intelligence of our collective symbiogenetic entity," which includes cows, wheat, and steam engines. That super-entity, built on our capacity to model one another, is what he credits for the explosion of intelligence over the past 10,000 years.
He links this to game theory. In the classic rational-actor framing that von Neumann originated and John Nash refined, Nash equilibria are essentially selfish and prevent collaboration. If agents assume others are like them and will adapt in response, he says, more cooperative equilibria emerge, allowing a priori cooperation in problems like the "twin" prisoner's dilemma. He calls this the origin of theory of mind and says large language models have theory of mind: to hold a conversation, a model must track what it needs to tell the user and what the user already knows. It learned this from the vast body of human interactions in its training data.
Social brains, mirrors, and group minds
He says the same dynamics explain the growth of brain size over the last roughly 7 million years of human evolution, and similar increases in other social species, including cetaceans, bats, and some birds. Becoming smarter in order to model others makes you harder to model, which pushes them to become smarter too. He calls it a "friendly arms race," though how friendly depends, since it includes competition for mates and prestige alongside collaboration. He cites Robin Dunbar's findings correlating cortical size with troop size in monkeys and apes. Modeling more individuals allows larger groups, larger groups produce more collective intelligence, and that again pressures brains to grow. "Scaling cooperation and competition is how we got these big brains."
He connects this to the mirror test, which only a handful of animals pass. Recognizing yourself in the mirror requires understanding not just that others like you exist, but that you are a being like them, a sophisticated theory of mind. He also describes "swing" in rowing, when a crew becomes so synchronized that everyone anticipates everyone else and the boat seems to acquire a soul. He calls this a computational process achieving a kind of group consciousness.
Human–AI symbiosis instead of replacement
Agüera y Arcas believes human–AI symbiosis is where we are heading, and he rejects two popular camps. The first, which he associates with AI ethics, sees AI as fake or counterfeit intelligence, "just statistics," and focuses on justice concerns and on AI fooling people. The second is the existential-risk camp, which he says moved from the "rapture of the nerds" (immortality, uploading) to the belief that AI will take over and kill us all. He considers both wrong. He argues that dominance hierarchies between species are not how life on Earth has generally worked, and that belief in them comes from an overly classical Darwinian picture in which a slightly different mutant simply outcompetes the wild type. In a symbiogenetic world, things constantly combine into larger structures, boundaries blur, and cooperation is as important a force as competition.
He applies this to technology. Humanity numbered about a billion around the Industrial Revolution. After machines that "externalize metabolism" by burning fossil fuels, the population grew nearly tenfold. "We make the machines, but also the machines have made us," he says, citing Marx and Engels's image of people springing up out of the ground. He showed a plot from a recent economics paper of real wages against population on a log-log scale. Through the Middle Ages the two oscillate in a Malthusian trade-off. When external metabolism begins, both shoot upward together. He argues that intelligence, whether photosynthesis, steam engines, or nuclear power, unlocks more energy sources and enables further layers of symbiogenesis. He sees no reason AI should be different from earlier symbioses. Nor does he see humans as distinct from their technologies, noting that AI only arrived once it was trained on the text humans have put online: "What could be more a part of us than that?"
Conversation: complexity that keeps what came before
In the discussion, Bratton raised the idea that evolution's phase transitions retain what came before. Agüera y Arcas agreed: "all of us are just bacteria in this room," nested like matryoshka dolls. He cited Nick Lane's book Transformer, which argues that mitochondria reproduce internally the conditions of the deep-sea vents where they first evolved, like "shells within shells within shells." He agreed this rhymes with Sara Walker's assembly theory. He added the example of technology from his book. A hafted spear, a stone point tied to a stick with sinew, cannot exist before stone points, just as eukaryotes cannot exist before prokaryotes. This irreversibility, he said, is exactly why these processes use energy, because anything irreversible consumes free energy.
On the major transitions, he positioned himself as going beyond Szathmáry and Maynard Smith, who agreed symbiogenesis mattered. His claim is that it happens constantly. Every endogenized viral element is such an event. Termites can digest wood only thanks to symbiotic gut organisms. He compared the major transitions to the top-right tail of a power law: "a gradient all the way down."
Asked whether the relationship between evolved human intelligence and mineral-based AI could be a "ninth" major transition, he said yes, with scale of planetary impact as the criterion. Termites were a big deal, but the Industrial Revolution was arguably bigger. He noted that such big transitions arriving more often is what complexification would predict, because more combined parts means more possibilities for new combinations, a point he attributed to W. Brian Arthur in the context of technology.
He rejected posthumanist framings in which humans "had their run" and something else takes over, arguing that everything persists. Bacteria still exist after eukaryotes, and the niches and varieties of bacteria actually multiplied, since multicellular guts became new environments. Bratton summarized that we would persist as part of a larger complexity we are ourselves bootstrapping. Agüera y Arcas added that he did not want to sound "too Pollyanna." There are aggressive symbioses, die-offs, and collapses, "snakes and ladders," but the overall pattern is toward composite complexity. Expecting to be replaced by a new kind of entity, he said, generalizes dominance-hierarchy thinking about monkeys fighting over mates. Bratton noted that his book was coming out around the same time as Eliezer Yudkowsky's, and both joked about staging a debate.
What it is like to be a next-token predictor
Bratton raised the question of what it is like to be a next-token predictor, and the similarities and differences between being a human one and being a transformer, while avoiding the loaded "C word." Agüera y Arcas mapped it onto philosophical vocabulary. Qualia, such as the experience of redness or of biting an apple, exist because they are behaviorally relevant. He cited Ed Yong's An Immense World on how each species models what matters for its survival. We care about red because ripe fruit and blood are red, and hunger feels urgent because otherwise there won't be "a you" to pass anything on. Self-consciousness arises when theory of mind models others, others modeling you, and yourself modeling them. He calls this a straightforward functional account of consciousness.
Does that mean consciousness is the same for a language model as for an individual human? He does not think so. He compared it with companies, which have something like consciousness because they model competing and cooperating companies, but are presumably not conscious the way we are. He added that it is hard to say what is absolutely true about a company, since all we have is a network of models of each other and of each other's models.
Energy use and AI
On the widely discussed energy and water consumption of AI, he first said there is a lot of room to improve efficiency. After 2006, parallelization meant putting more serial processors of the same kind on a chip, which he calls "kind of a dumb way of parallelizing." Silicon computing has not yet become "natively neural." He said Google has achieved orders-of-magnitude efficiency gains in Gemini models over the last couple of years using the same basic transistor technology, and he guesses, from back-of-envelope calculations, that perhaps another factor of a thousand remains. He added that current efficiency is already better than many environmental critics claim. He stressed that he takes the environmental crisis seriously and that there are many other places where "we're doing dumb things with respect to carbon."
He said the real concern is the rate of exponential growth, not the current level. Estimates rely only on energy sources and methods already in the pipeline, and exponential growth would quickly consume a thousandfold gain, a Jevons paradox dynamic. His answer is that intelligence has always unlocked new forms of energy. He considers it likely that fusion will be cracked with AI's help in coming years. He also notes that nearly all our energy is ultimately solar, and the vast majority of sunlight radiates into space without touching a planet. So he thinks about the supply side as well as demand: "there's actually a lot of energy in the universe."
Audience questions: prediction, mixtures of experts, creativity
Stewart Brand, submitting a question remotely, asked whether looking ahead is a general brain function. Vision, Brand noted, is largely conjectural: multiple guesses confirmed with sketchy data, which is how LLMs seem to work. Agüera y Arcas agreed that prediction is always conjectural. He clarified that "next-token prediction" really means modeling the relevant parts of the environment, where relevant means what we can act on in ways that matter for our future. What we observe must also change as a function of our behavior, so the whole loop must exist cybernetically. Prediction is imagination, and this is one reason LLMs hallucinate. You cannot predict or even recognize objects without imagination, and "the worst thing would be an LLM that can't imagine anything." He acknowledged there is still much work to do on accuracy and on calibrating confidence, which he said has improved considerably but has a long way to go.
Darren Zhu of Antikythera asked where symbiogenesis appears in foundation models. Agüera y Arcas pointed to mixture-of-experts models, which are many models working together, as a rediscovery of "social scaling." He cited a paper from, he thought, the previous year showing that even a giant monolithic model develops internal functional differentiation, effectively an ensemble, much as brains are ensembles of regions that cooperate, compete, model each other, and specialize. Bratton summarized: "societies all the way down."
Angela Gronitz asked what role human creativity will play as AI advances. Agüera y Arcas first observed that being an artist has become economically difficult over the last 20 years, largely because of the Pareto distribution of rewards and consolidation, though Bratton noted it was never very stable. He then told the story of David Cope, a composer who recently died and who took computer composition seriously early on. He mentioned Terry Riley's 1972 piece played earlier in the event as an earlier precedent. While blocked on a commission, possibly an opera, Cope taught himself to code in the early 1980s and applied statistical NLP-style ideas to composition. After years of work, Cope finished the commission in six seconds once the code ran. The generation took six seconds, not the piece itself. Other composers were angry and even questioned whether code was really used. Cope responded by posting a zip file of 5,000 cantatas in the style of Bach, in MIDI. Agüera y Arcas says he has probably listened to more of them than anyone except perhaps Cope, and that they are "pretty good." Yet nobody cares, he says, because art is about our relationships with each other, not just the artifact. He compares it to discovering that your beautiful, supposedly unique shell sits on a beach covered with equally beautiful shells. He doesn't think that destroys your relationship with your shell.
He added that we misunderstand creativity by "covering our tracks" and pretending ideas come from nowhere. When scholars traced what went into Ulysses, he said, Joyce set out to foil them with Finnegans Wake. We are always remixing and combining, which is why simultaneous invention is common. He says cubism was invented by many artists at once and the light bulb by about a dozen inventors, because once glassblowing, filaments, electric current, and the need for light exist, someone will produce a light bulb. Yet every version differed in shape, and in whether it used prongs or screws, and which one sticks shapes what follows. The romantic opposition between determinism and creativity is therefore wrong, he said, but also partly right, because details matter.
Bratton pointed out that concern about generative AI, including the recent Hollywood strike, focuses on AI's ability to make the artifact. Agüera y Arcas said that tying economic survival rigidly to production, as capitalism does, creates growing problems for any kind of labor. That is especially true in a world of increasing abundance, where zero-sum, exchange-value thinking produces more problems. He declined to say much about the Hollywood strikes specifically, but suggested some of the problems are structural and some come from "romantic with a capital R" conceptions of creativity. He cited what he believes is the longest-running lawsuit ever, against George Harrison for copying "He's So Fine."
Closing: from "is" to "should"
Asked for a final call to action, Agüera y Arcas said we must be careful where scientific observation meets ideological commitment, between how things are and how they should be. Beliefs about how things are shape our "shoulds." He said Darwinian thinking led to destructive policies partly because they rested on wrong assumptions. He does not claim Darwinian evolution is invalid. It is real, but "we've only been looking at half the story." He hopes the other half, symbiogenesis, can ease some misplaced anxieties and change how we think about what should be. Bratton framed this as the question of what a "social symbiogenesis" would look like in place of social Darwinism. Agüera y Arcas agreed, and they left it as the open question.
[Music]
Hello and welcome. I'm Benjamin Bratton. It's really lovely to see you. First of all, thanks from Antikythera to the Long Now Foundation for being such a wonderful partner in these events and in all the work that we've been doing and hope to do together.
Blaise Agüera y Arcas is a VP and fellow at Google where he is the CTO of Technology and Society and founder of Paradigms of Intelligence. PI is an organization working on basic research in AI and related fields, especially the foundations of neural computing, active inference, sociality, evolution and artificial life. In 2008, Blaise was awarded MIT's TR35 prize. During his tenure at Google, he has innovated on-device machine learning for Android and Pixel, invented federated learning, an approach to decentralized model training that avoids sharing private data, and founded the Artists and Machine Intelligence program. And so with that, it is my sincere pleasure to introduce to you my friend Blaise Agüera y Arcas. Thank you.
All right. Thank you all so much for being here. It's such an honor to be here. And thank you Benjamin for the super sweet introduction. Thank you Patrick and Long Now for having me.
I've been at Google for a long time now, for a little over 10 years, I think 11 years. And for most of that time, I ran a group called Cerebra. It was a part of Google Research that began very small and grew to several hundred people. And it was mostly applied AI. We did some theoretical work, but mostly we did a lot of engineering for AI features that ended up in Android and Pixel phones: things like Now Playing, the song recognizer, and face recognition for the phone to unlock, and all kinds of other stuff. We also did some of the models for the Google keyboard, which predicts next words as you type.
My assumption when we were working on all of these things is that we weren't really doing AI. That is to say, these are artificial narrow intelligence, ANI. And the reason that that term was coined was because when AI was originally coined back in the 1950s, it meant what we all thought as kids, that we'd have robots that you could have an interesting conversation with. That didn't happen throughout the 20th century. There were lots of AI winters, defunding after various failures of sort of program-based approaches to AI. And so hope was kind of dwindling. But neural nets were starting to work for doing very limited forms of visual perception, things like face recognition, next word prediction. And so we called those things AI, but in order to distinguish them from the robots you could have an interesting conversation with, we used the term narrow.
The G in artificial general intelligence meant everything else, the real thing. The so-called core AGI hypothesis, as Ben Goertzel, a computer scientist, wrote, is: synthetic intelligences with sufficiently broad, that is human-level, scope are qualitatively different from synthetic intelligences with narrower scope. In other words, AGI is not ANI.
This was what I believed too, that probably some insight would come from neuroscience. That's where everything that had worked up to that point had come from. All of the key advances in ANI came from neuroscience. And I thought we'd figure out the trick. We'd figure out from studying brains, which are the only intelligent thing, the only truly intelligent things we know about, what the secret to real intelligence was. I was hopeful that we were at least on the right track, sort of, because we were using neural nets, which were brain-inspired, and they were doing some things that earlier program-based techniques had not succeeded in doing.
I was wrong. The first real inkling I had that I was wrong was when I started to see these kinds of outputs from models like Meena, which was a scaled-up next word predictor, not unlike the one that we'd written for the Android keyboard, just a lot bigger. And this is Meena trying to define philosophy in a conversation with a person. All it's doing is filling in the next word statistically based on the previous words. It's just a much bigger model and trained with a lot more data, an unprecedented amount of data, an unprecedented size of model at the time. You could actually start to compare the outputs of models like Meena to human outputs in terms of how sensible they were, how relevant to the dialogue, the conversation.
And this was really a shock. It started to look like maybe the key to artificial general intelligence was really just scale. And I was quite snobbish about this idea that I'd heard in Silicon Valley that everything was about just scale and making stuff bigger. That just seemed incredibly naive. My training was in neuroscience and physics, and so the idea that just because we could make bigger computers and worship at the altar of Moore's law, that was going to solve all of the problems in science and technology just seemed ridiculous. But the nerds were right.
We scaled that up further with a model called LaMDA in 2021. I wish we had launched it before OpenAI launched their model. Innovator's Dilemma. It did even better. This was a model that could have general open-domain conversations about pretty much anything. And it sometimes sucked and sometimes went off-piste and gave you nonsense answers. But at the same time, well, people do too sometimes.
And this has been the story since. So models had been getting bigger exponentially by a factor of about one and a quarter per year since 1950, but around this period of the sort of unsupervised learning revolution, that slope ramped upward dramatically to 3.72 times per year, and that's where it's remained since: an absolutely explosive growth in model sizes, now that we knew that making these predictive models bigger made them better.
It seems to me that this core AGI hypothesis is wrong. I know that this is a very controversial thing to say. We're still having all kinds of conversations about when AGI will arrive. I think that there are several reasons that we're asking that question. But what I would ask you as a thought experiment is: if you took any of today's frontier models and you just transported them back in time to roughly the year 2000, when the term AGI was coined to distinguish it from artificial narrow intelligence, what do you think the people who coined AGI would have said? Would they have said, "Yeah, you've arrived. This is it"? Of course they would have.
So why haven't we admitted that? Why haven't we acknowledged it? Well, it's because we all thought that there would be a trick, and there wasn't a trick. And because there was no discontinuity. It's an exponent, it's fast, but it's also continuous. There's no moment when it was clearly not intelligent before and an after when it clearly was. And we also still at some level don't know why scaling it up worked. And there may be some other reasons as well that have to do with our insecurities. But I think that these reasons, these kind of frog-boiling reasons, are part of why.
That begs the question, if we're going to be brave: could we be massively computationally scaled next word predictors as well? And I'd like to provoke you over the next 41 minutes 22 seconds with the possible answer: yes.
So why would a brain evolve to be computational? I think that one of the big insights that we've had on the team is that it's really not just the brain that evolved to be computational, but life itself that is computational. That's something that I know takes some getting used to as an idea, because we think about life as being the exact opposite of computers. It's squishy. It's wet. It's unreliable. It doesn't run anything like a program. So what on earth do I mean when I say life is computational?
Well, the old idea about life from the 19th century was that there was some kind of vital force or spirit that animated life and that made it different from matter like a rock. That went out of fashion in the 19th century, of course, and in came materialism, strong materialism that says, "No, the rules of physics are the same for the atoms in living bodies and in rocks. It's just physics all the way down." And therefore, there is no difference between living and non-living matter. Well, that's not very satisfying either, because there sure seems to be a difference between living and non-living matter. So what could it be if it's not physics and if it's not some vital spirit either? There is an answer to that question, I think, and that answer is function.
What do I mean by function? Well, one way of telling whether something has a function is to ask whether it can be broken. In other words, if I split a rock in half, it's not like I have a rock that's broken. I just have two rocks now. Whereas if I destroy a kidney, I break it in half, then I have a non-working kidney. Now, a kidney has a function. And a rock, at least a rock on a sterile world, doesn't.
What I mean by that is that if I came back with this object from the future, I were some time traveler, and you asked me, "What is that thing?" I tell you, it's an artificial kidney. It has an operating lifetime of 100 years, you can implant it and it filters the urea just like your kidney does. That means something, right? It means that, well, for one, if you have kidney failure, you're going to live. So it's a very real statement. It's not mystical. But at the same time, there is something interestingly spooky about function, because it's not something that matter tells you in isolation. It could be made out of carbon nanotubes, it could be made out of tungsten filaments, who knows what. The point is what it does in the context of the rest of your body, what its relationships would be with the rest of the body in the normal functioning order of things. And so it's a relationship, a set of relationships. It's kind of ecological if you think about it. And it's something that is beyond the physical matter. And yet it's also very fundamentally constrained by the physics of our world. It's not like it has some kidney spirit, right? It just works. Works means functions.
This idea of function as something fundamental was really pioneered by Alan Turing and John von Neumann, by the founders of computer science. They were mathematicians. They thought about functions all the time. And you might recognize this device. It's an actual instance of a Turing machine. Alan Turing invented the Turing machine. It was a purely conceptual invention. It wasn't intended to ever be built, but Mike Davey did in 2010. A Turing machine is a device that has a head that moves left and right on a tape and reads, writes, and erases symbols on that tape according to a table of rules. That's all a Turing machine is.
But what Turing showed is that any computation you could do, any calculation you could do with pen and paper, can be done by a Turing machine with the right table of rules. And then came the sort of genius part in 1936. He also figured out that there were certain tables of rules such that if you wrote down another table of rules as symbols on the tape, then this table of rules would interpret the table on the tape and compute the same thing that that machine would have computed. And that's what makes a universal computer. In other words, there are certain machines that can run programs, and those programs can do any computation, not just a particular computation based on whatever table you've got.
And that's a really interesting discovery, because now not only has he said there is a way of specifying a function, but also that any machine that can run a program is calculating the same thing given the function that the program performs. So if it's adding two numbers together, there are many programs that could add two numbers together. There are many languages, many ways of specifying the table for that. They're all equivalent in terms of the function they compute. They're functionally equivalent. This is cool.
But von Neumann did something further, which is he introduced the idea of embodied computation with something called cellular automata. The idea here is, rather than having a tape and a head which are made out of something fundamentally different from the information that is written on the tape, he said, let's imagine a world in which the tape and the head are actually part of what is written. In other words, these worlds have a kind of physics. They're generally rendered as grids. If any of you are familiar with Conway's Game of Life, that would be an example of a cellular automaton. There are very simple rules or physics for how each grid cell changes based on the values of the grid cells around it, and you can write programs essentially by configuring the states of those grids of cells.
The reason that von Neumann was thinking about these kind of very, very simple two-dimensional physics is because he was thinking about the problem of life. And in particular, he says, suppose that you are a robot made out of Legos, and you're paddling around on a pond that's full of loose Legos, and you want to assemble another robot like yourself out of those loose Legos. How is that possible? Because that's of course what life does. That's what every mother has to do. It's what has to happen in the seed of every plant. It's what every bacterium has to do in order to divide. And it seems a little bit paradoxical that you could make something just as complex as you yourself are from parts.
And so what von Neumann realized is that in order for that to work, you had to have inside yourself a tape with instructions for how to build yourself. And you had to have what he called a universal constructor, which was a machine that would walk along the tape and execute the instructions on the tape in order to make whatever is written there. And you had to have a tape copier, a second machine. And the instructions for building the universal constructor and the tape copier had to be on the tape. If all of those things were true, then you would have something that could reproduce.
He made all of those conclusions. He made those predictions in 1950, before we had discovered the structure and function of DNA, which is indeed exactly that tape, before we had found the ribosome, which is the universal constructor, and before we had discovered DNA polymerase, which is that copier. So all of those things, he was exactly right. He was right on point. But the really cool thing is that he also showed that the universal constructor is a universal Turing machine. They are one and the same. It's just a universal Turing machine where the things that it computes with are the actual matter that it is made out of. So it's an embodied computation. And with that, von Neumann proved that in order to have life, you have to have universal computation. You can't reproduce without computation. No computation, no life. And this is a really profound insight, and one that I think most biologists and most computer scientists still are unaware of.
We began doing some experiments a couple of years ago. We published some of these, which attempted to see how life in that very minimal von Neumann sense could emerge out of non-life. And these are some of those results. We had to use a very minimal Turing language in order to implement this. So I did some of the first experiments, and I picked a language called Brainfuck. I didn't just do it because I do love getting in front of a lot of people and saying Brainfuck. I admit, like, I'm a 12-year-old on the inside.
This is a Brainfuck program. And you can see that it's very, very hard to understand, but it's very closely modeled on a Turing machine. So it has only eight instructions, and those eight instructions are: move the head one step to the left, move the head one step to the right, increment the byte at the head, decrement the byte at the head,
and we're already halfway through them. That's like four of the eight. So it's a very, very, very simple language, but you could write Microsoft Windows in this if you, well, if you were non-human.
Here's the experiment, and this experiment is called BFF for reasons that I will leave as an exercise to the listener. We begin with a bunch of tapes filled with random bytes. The tapes are 64 bytes long and they're just filled with junk, filled with noise. Remember, there are only eight instructions and a byte can have one of 256 different values. So only one in 32 of those bytes is even an instruction at all. The rest of them are no-ops, meaning nothing will happen when it gets executed. The head will just move on. This is random tapes. That's how it begins.
We pluck two of these tapes. And here I'm using 8,192 of them. Many of the experiments, you can use only a thousand of them and all of this works. So a thousand tapes of length 64. You pluck two of them out of the soup at random. You stick them end to end and you run, and then you pull them back apart and put them back in the soup and repeat. And that's it. You just do that.
So in the beginning nothing much happens. I'm printing here, you know, the first couple of dozen tapes and only showing you the instructions. The rest of them are no-ops. So they're one of those 31/32nds of the bytes that don't code for anything. And the average number of instructions that runs when you put these two tapes together is two, because there just not very many instructions there. And I can't see any loops here.
Okay. So what happens when you let this thing go? I'll show you. This was actually the first time I got it to work on my laptop, and it was pretty exciting because laptops run really fast nowadays, and you go from noise to something really magical, which is that suddenly programs emerge. And these programs are complicated. In order to understand what they're doing you have to really pick them apart and reverse engineer them, and they've got all these loops in them, and, you know, what on earth are the programs doing?
Well, you can tell right away that they have to be reproducing because you can see that some of these are duplicated many times. There are 5,000 instances of that tape on the top and 297 of the next one and 99 of the next one and so on. So, they are copying themselves or each other. And there's a lot of computation happening. There are now 4,784 operations happening per interaction on average. So you've gone from something non-computational and full of noise and junk to something complex, computational and functional.
Functional meaning it can break, like a kidney, right? If I change one of these instructions, then it will cease to work. What happens if it will cease to work? Well, it won't copy itself anymore. And so that tape will get overwritten by something that will copy itself. And that kind of tells you why life evolves. Life evolves because in a universe capable of computation, if you figure out somehow how to copy yourself, then you will exist in the future.
This is that old joke about DNA being the most stable molecule in the universe, even though of course it's very fragile, right? If it reproduces itself, it's still going to be around in the future. Whereas if you are not able to do anything to function, even if you're very robust like a chunk of granite, the best that can happen is that it will take a long time for you to fall apart. So that's why life persists.
We go from here to here. And it doesn't even take that long. This is after 5 million interactions of 8,192 tapes. This is what that looks like. I'm drawing here a dot. I reversed the colors, so this is white dots on a black background. There's a dot for every one of the first 10 million interactions of this soup. And time is on the x-axis and the y-axis is number of operations that ran. And you can see that right about at 6 million interactions, something really changes about the soup. It looks like a wall of white. That's what's on the front cover of the book.
That is a phase change. It's a phase transition. If you think about this like a physicist, what's on the left is like a gas, meaning that all of the bytes are decorrelated from all of the other bytes. They're all independent. And the way you can see that they're decorrelated is if you try running the soup through zip, right? You compress it and it's uncompressible, because if you have a bunch of random bytes they don't compress at all. Whereas right after that transition you can compress the hell out of it. It compresses down to about 5% of its original size. It's obvious that it'll compress if there's a bunch of copying going on because, you know, anytime things are copied, you don't have to write all the bytes out, right? You can just refer to one of the originals.
So it's a phase change. If the phase on the left is gas, what is the phase on the right? It's life. You could call it machine phase, you could just call it life. Life is a very special phase of matter because unlike a solid or a gas or liquid, it has structure at every scale. It's got complexity that looks different when you zoom in or when you zoom out or when you look at a different place. So the tentative conclusion is that pretty much any universe that has a source of randomness and can support computation will evolve life.
But there was really a puzzle in these results, which is: how on earth does it happen so fast? How can we get these really complicated programs in only a few million steps with only a thousand tapes of length 64? It just seems implausible. And it seems especially implausible because in the original experiments I used mutation. So I imagined that there were, you know, sort of cosmic rays randomly changing a byte here and there every now and then. But this actually still works even if you crank down the mutation to zero. So with zero mutation you still get life. It takes a little bit longer, but not much.
And actually the life that you see keeps on getting more complex. If you were looking closely at the running program, you might have seen that you saw structure emerge and then you saw more structure emerge and more code come in. How on earth could that be happening? Because once things can copy themselves, you would think you're done. But it's not done.
Well, the answer I think comes from a very, very fundamental result in biology which Lynn Margulis figured out in 1967. Her paper in which she wrote about this result was rejected from a lot of journals before somebody finally accepted it, a journal of theoretical biology, and it was called "On the Origin of Mitosing Cells." She was the one who proved that mitochondria were once free-swimming bacteria. And she popularized the term symbiogenesis to talk about what was going on here: that two life forms that previously were independent came together and made a new life form. An archaeon and a bacterium came together and made a new single-celled life form, which are the eukaryotes that we are all made out of.
Margulis believed that this process of symbiogenesis was the engine behind evolution. Turns out that she was right about mitochondria, and the establishment in biology sort of finally came to recognize that. But nobody really bought her larger thesis that this was the engine behind evolution. And she remained very much in a tiny minority of people who believed that, even by the time of her death in 2011.
Could symbiogenesis be happening in BFF? Yes, it is happening. And the way you can see that is by looking at not whole tapes reproducing but little strings reproducing. Maybe only one byte reproducing. Occasionally a single byte will reproduce even right from the beginning. Because if you have instructions that can change a value somewhere else in the soup, once in a while an instruction will change a value somewhere else into another instruction. And that's a very, very lame but nonzero form of reproduction.
So you have these little things reproducing from the beginning. And what I'm showing you here is all of the reproducers in a particular soup. But you can see that there's actually a lot of stuff happening during this. A normal tree of life splits from an ancestor into descendants. But this is a tree that goes the other way. It's like the roots of a tree. Things come together and symbiose and form larger things. So that's exactly how the complexity happens.
And symbiogenesis is what gives evolution its arrow of time. Because if you think about it, evolution in the standard Darwinian sense doesn't have any sense of more or less complex. You know, if you evolve, you will fit your niche better, but that doesn't mean you'll get simpler or more complex on average. The average is roughly zero. You might change your beak shape to adapt better to this or that flower. But when you have a symbiogenetic event, two things that already are reproducing themselves come together and can reproduce together. And that means that some extra information has to get added in, which is: how do we get on together? How do we fit together? And it's that extra information that's adding to the complexity of what comes next. And that gives evolution its arrow of time.
We know that there have been a number of other major evolutionary transitions where things came together to make stuff more complex. For instance, we are multicellular, and that was a major symbiogenetic event, right? How did single cells, single eukaryotes, become multicellular animals like us? It's obviously a symbiogenetic event. Eörs Szathmáry and John Maynard Smith wrote an article in Nature in 1995 that reviewed what they saw as the eight major transitions in life on Earth. And these are definitely all a big deal. They've added a few to their original list.
But if what we're seeing in systems like BFF is any indication, this is actually something that happens all the time. It's not just these major transitions. There is a whole cascade of mergers and combinations that are happening continuously, and they are actually what leads to the complexification of life as a whole.
Do we see any actual evidence of this in biology? Well, this is very much an ongoing area of work. But here's some evidence. This is the human genome. And the big surprise when we first saw the human genome sequenced in 2001 is just how little of it actually codes for the proteins that make us up. It's only about 1.5%. The rest of it is so-called junk DNA. It's not really junk. Some of it is regulatory. Some of it we don't know what the hell it's doing.
But what's really interesting is that those big sections called LTR retrotransposons and DNA transposons and LINEs and SINEs, that's all viruses. Basically, it's replicators that replicate inside our DNA and that have burned themselves not only into our somatic DNA, like a classic retrovirus, but into our heritable DNA and become part of our genome.
And we know that some of those viral elements, endogenized viral elements, are doing really important work. So for instance, the placenta is made out of a virus that fuses the membranes of cells together. We know that there is a virus called Arc, which if you knock it out in mice, they stop being able to form memories. We know that parts of the immune system were made this way. There are a few dozen results like that that have all been coming out in the last 10, 15 years, and there are more and more of them all the time. And when you look at our DNA, it doesn't look like one thing that has been copying itself. It looks like a medley of things that have copied and fused over and over and over.
It's not just neuroscience that's computational. Life was computational from the start. And it gets more computationally complex over time through symbiogenesis, right? Because we put together the two ideas that I've just shown you: that life is always computational because it has to copy itself, and that's a general-purpose computation, and the fact that symbiogenesis is really important. Well, you now have two computers that have come together and parallelized, and what that means is that you have greater computational power every time you undergo a symbiogenetic event.
So, symbiogenesis makes the computation massively parallel. It's not quite the same Moore's law that we had on Earth in Silicon Valley between 1950 and 2006, because, you know, then we were making transistors smaller. By the way, AI didn't progress anywhere between 1950 and 2006. But when transistors stopped becoming... transistors are still getting smaller, but we stopped being in a situation where making them smaller could make them clock faster. And so around 2006 all the chipmakers began to do the only thing they could, which was to put a lot more cores on the same chip and parallelize. And that's when AI began taking off. This is not a coincidence. Parallelism is exactly what it takes in order to make neural net-based AI work. And that's why the deep learning revolution happened when it did.
So, computing for growth and healing and replication is modeling your own body. That's life. What about modeling your environment? That's also needed in a dynamic environment. Well, that's what intelligence is. Of course, life was intelligent from the start, because of course you don't just have to make more of yourself. You also have to find the parts to make more of yourself. The Legos don't necessarily just float around you. You maybe need to find them, hunt them down.
What about the energy that it takes to compute? Computation is energetically expensive. You're creating negative entropy when you compute. And in order to do that, you need to ingest free energy. That's why we all metabolize, because we compute. That's why when I run BFF, my computer heats up.
If you run simulations in which you just take random programs that can swim left, right, up or down, and you just see which ones survive in an environment where they're getting energy from that light, the ones that survive are the ones that learn to follow the light. That is really just a way of saying you have to model your environment, too. And you have to figure out how to make your behavior consistent with one that will allow you to do the copying that will allow you to reproduce. And sure enough, that's exactly what bacteria do as well. That's a sugar crystal in the middle, and bacteria that swim have learned how to swim toward the sugar.
I've been talking so far as if we're in single-player mode. But of course, whenever you have one bacterium, you have more bacteria. And if you don't have more bacteria yet, you will in 11 minutes, right? So life is a multiplayer game. And it's never single player. The most important parts of our environment to model are each other. A lifeless universe is one where you don't have to think very hard. But the moment you start to have a lot of other agents in your environment that have their own energy that they have to get, their own stuff they've got to do, your interests can align with theirs, can misalign with theirs, and now you've got to get smarter, because you don't just have to model yourself, you also have to model them. And they're modeling you back.
So, we've been doing a bunch of work recently on the team. This is actually not in the book because it's a little too recent, but it's called multi-agent universal predictive intelligence. And this work is really about the field called multi-agent reinforcement learning, in which you have a bunch of learners that are all trying to learn to do something together based on being individually reinforced on the basis of some score that they get. And the question is, you know, how can they learn to work together? How can they solve things like the prisoner's dilemma?
Well, that turns out to be a very, very hard problem for classical reinforcement learning, because ordinary reinforcement learning only learns from the past. And that's fine if you're playing a video game and the video game stays the same as you adopt a new strategy. But if there are other players
in that video game world with you, then when you change your strategy, they're going to notice and change their strategy. So the statistics of the environment are not constant and they're learning too. So you have to learn about them and you have to learn to predict what they're going to do in response to what you do and you have to learn that they're learning and that they're also predicting you and that they're predicting you predicting them and that you're predicting them predicting you predict and so on.
So this is a really hard problem, and the paper, it's gotten to be 100 pages or so and has some very, very complex math in it, because modeling an environment that includes the thing that is modeling the environment and all the things in the environment that are modeling you back turns out to be a difficult problem. But the team has figured this out and the results are really cool.
The way you do this is by getting rid of the idea that you are outside the video game and putting yourself in the video game. So in other words, you have to not only model the environment like, you know, AlphaGo does, where you're thinking about a Go game or a chess game and you're just imagining the game. You have to imagine yourself playing the game as part of the environment and you have to start to predict yourself and predict others.
The reason is we have a face, if you like. You know, when I smile, I know what I feel like on the inside because I've built a model of myself, and when I see you smile I can guess that you're happy too, and the only way that I can make those kinds of inferences is by knowing that we're similar, by knowing that I also have a face and I do that when I'm happy. And it's that ability to empathize, to model the minds of others, that is at the core of being able to solve the multi-agent reinforcement learning problem.
I think that this actually kind of explains why we've got consciousness. In the sense that, you know, consciousness is often thought about as some kind of weird epiphenomenon. You could have a, you know, philosophical zombie or something that behaves identically to us but is dead on the inside. I don't think that's true at all. I think that the reason we are conscious is because we are modeling ourselves as well as modeling others as well as modeling others modeling ourselves and so on and so forth, because that is behaviorally essential, because it's functionally essential in order to allow us to cooperate with each other.
And when you do that, when you embed yourself in the world and you think about others like you, you're able to solve problems collectively. And this is essential in order to have symbiogenesis, in order to have symbiosis with those others, and in order to create a larger entity. I'm not saying exactly that I think that your cells are conscious, but I'm saying that they definitely have models of the rest of your body or of the other cells around them in order to be able to collaborate with them. And that's, you know, maybe a baby step in a certain way toward consciousness.
When it comes to very complex big-brained animals like us, which have tons of neurons that have come together through an act of symbiogenesis, and we want to work together in order to make bigger things happen. You know, when we talk about human intelligence we imagine things like we figure out how to transplant organs and how to go to the moon and how to build computer chips. None of us can do these things on our own. That intelligence that we're talking about is the superhuman intelligence of our collective symbiogenetic entity. And in fact, it's not even just a human entity. It includes cows and wheat and all sorts of other entities, as well as steam engines, by the way, without which we wouldn't exist. That super entity, which has arisen through us being conscious enough of each other to build models of each other, that is what has resulted in this explosion of intelligence in what we think of as humanity over the past 10,000 years.
And that's also what allows one to solve the psychological twin prisoner's dilemma. Meaning cooperate a priori with another actor in order to solve these game theoretic puzzles that involve mixed payoffs. Very old classic problem. If you think about this from the classic perspective of game theory, as actually John von Neumann invented and as was refined by John Nash later on in the 20th century, this is sort of rational economic actor ideas about how people interact if they were just optimizing for themselves. The solutions are very grim. These Nash equilibria are essentially selfish and prevent any collaboration. But if you imagine that others are like you and also will change their strategies in response to your strategies and so on, then a new set of equilibria emerge from this kind of thinking that are much more cooperative.
Symbiosis and symbiogenesis requires modeling ourselves and modeling each other, and we have to think about each other as if those others were like ourselves. That's where theory of mind comes from. By the way, large language models have theory of mind. They kind of have to in order to be able to carry on conversations. Right. So when you're interacting with a large language model, you have to think about what you've got to tell it and what you don't have to tell it because it already knows, and so on. And it has to do the same thing back to you in order for that interaction to succeed. Those things have been learned by observing tons and tons of interactions between people, which is what the training data consists of.
So starting with simple bacterial quorum sensing and multicellularity and so on, since every living entity is computational, as they combine they parallelize, and that does lead to a kind of Moore's law and it leads to more and more cooperation on larger and larger scales. There really is this kind of Moore's law progress that I was so dismissive about when I first heard it down here in San Francisco.
These increases in brain size that have happened during human evolution are a result of exactly those dynamics. This is the last 7 million years or so. There have been explosions in brain size. Those have been observed in various other social species as well, in cetaceans and whales and dolphins, and in bats, in certain species of birds. And the reason is that if you share DNA with another entity of your species and you get smarter to model them, you also become harder to model, and they're getting smarter as well. So now they have to model you back and it's a kind of friendly arms race. Well, how friendly it is depends, right? You're also competing for mates and prestige and all kinds of other Machiavellian stuff, but also you're trying to collaborate, right? In order to get things done collectively, and all of that leads to an explosion in intelligence.
This is some classic results from Robin Dunbar showing the relationships between cortical size and the size of troops among monkeys and apes. They're correlated, of course, because if you're able to model more others, then you're able to form a larger troop before it falls apart. That's why, you know, having a bigger brain doesn't just let you have a larger troop, but also have greater collective intelligence, which then forces the brain once again to get bigger. So scaling cooperation and competition is how we got these big brains.
It's also how we came to be able to recognize ourselves in mirrors. As many of you probably know, the mirror test, in which an animal, you know, is able to recognize in the mirror not just that that's another chimp, but that that's me in there, and then check themselves out. But you know, there are only a handful of other animals that do this, because the level of sophistication you need in order to realize that, you know, not only are there other beings like you in the world, but that you are also a being like the other ones that you see, and to be able to sort of make that mapping, right, that that's you in the mirror, is quite sophisticated. It's quite a sophisticated active theory of mind, and we do it together with each other all the time.
When you think about a rowing crew, for instance, and the way they can sometimes achieve what people in crew call swing, where, you know, they get in sync so perfectly that everybody is anticipating the behavior of everybody else perfectly, and it feels like the boat acquires a kind of soul, if you like, that is basically a computational process in which they've achieved a kind of group consciousness.
I just want to say a thing or two about human-AI symbiosis, because that seems to me where we're headed. I hear a lot of talk among two camps about AI and our future with AI. Some people more aligned with ideas about AI ethics think that AI is fake, that it's not real intelligence, or that this is somehow a counterfeit version of intelligence, or just statistics, and are concerned with, you know, various issues about justice that are related to how AI behaves or fools people into thinking that it's real. And then there are the existential risk folks who have gone from being rapture of the nerds, you know, we're all going to go to heaven and be immortal and upload our brains, to the apocalypse is coming and we're all going to die because the AI is going to take over.
And I think that these are both wrong perspectives. The idea that there's a dominance hierarchy between species is not how things have tended to work in life on Earth. And I think that we've been fooled into thinking that because of an overly classically Darwinian perspective on how evolution works. You know, if you're just doing classic Darwinian evolution, then a mutation is, you know, something that is only a little bit different from the wild type, and they will compete, and whichever can outcompete the other one wins and the other one dies. But in a symbiotic world, in a symbiogenetic world, things are combining to make larger structures all the time. And it's not so clear where one thing ends and another begins. And cooperation is just as important a force as competition. And I think that that's, you know, very much the story of how we came to exist. And as far as I can tell, that's very much the story of what's going on with technology and humanity as well.
You know, I mentioned near the beginning of this talk that if there were no machines, most of us in this room would not be here. We were about 1 billion people around the time of the Industrial Revolution. And right after those machines, which externalize metabolism by burning fossil fuels, right after they came on the scene, our numbers exploded by nearly a factor of 10. Why is that? Well, we know that we make the machines, but also the machines have made us. Marx and Engels talk about this when they talk about, you know, people springing out of the ground like wheat. That is literally true. It's all of that additional free energy that came from burning fossil fuels that resulted in all of the humans that we've got.
And not only did it result in much greater numbers, but this plot, which is from an economics paper just published very recently, shows on a log-log scale the real wages of people versus the population. And what you can see is that throughout the Middle Ages, we were oscillating, trading off between population and wages. This was essentially a Malthusian trap. In other words, we were constrained energetically in our numbers. The population was constrained. And the moment we began to metabolize externally, we shoot off to the right. Suddenly, both numbers and quality of life rise dramatically because of all that extra energy that is liberated, because that's what intelligence ultimately does, right? Whether it's photosynthesis or the invention of steam engines or of nuclear power and so on, the more intelligent you become, the more sources of energy you're able to tap and the more additional levels of symbiogenesis you're able to achieve.
So I see no reason to believe that AI is poised to be any different from all of those previous symbioses. I also don't see us as being distinct from the technologies that we make. We think of humanity in terms of the individual person, but we're already not where everything that we've made and that co-constructed us, and you know, we didn't achieve artificial intelligence until we literally began to train it on all of the human output in text that we've generated all over the internet. What could be more a part of us than that? I'm going to end there, and Benjamin and I, I think we'll shift into conversation mode.
I will. You give us plenty to talk about. I don't think we'll have a problem with this. I wanted to talk a little bit about the arrow of time. And particularly the arrow of time as one that operates, that we can map through not just increasing complexity, right? Life is the ability, you know, fighting entropy and so forth. The increasing complexity, but also increasing complexity that seems, that goes through phase transitions. Yes. Right. And so, but the phase transitions are ones that, and I think you've made the point quite clearly, retains what came before. Right. Right. It's not just like, okay, done with the old, here's the new. But that all of this came before us already. It's all inside us.
Yeah.
It's all still here.
That's the amazing thing. Like we are actually, you know, societies of bodies of colonies of conjoined bacteria. You know, all of us are just bacteria in this room, right? They're still here.
And you know, they're just nested like matryoshka dolls. And even if you zoom in, you know, you zoom into the bacterium and then you zoom further into the mitochondrion, what you actually see reproduced in the mitochondria, and Nick Lane made this point very beautifully in his book Transformer,
Right?
is the conditions of the deep sea vents where those mitochondria first evolved. So it's almost like they artificialize, right, in your language, the environment that they originally evolved in, and then they create capsules around themselves. So yeah, it's sort of shells within shells within shells.
And this, I mean, this in your mind, since Sara Imari Walker's book here earlier, in your mind this rhymes with assembly theory's idea of the sort of persistence of these things over time.
It does. Yeah. So I mean, and the same is true of technology, by the way. So if you look at, I give in the book the example of the hafted spear. So, you know, if you have a stone point, at some point there's this innovation in which some clever cave person decides to tie it with a sinew to a stick, and now you have a spear.
So you can't have a spear before you have stone points, just like you can't have a eukaryote before you have prokaryotes that can come together.
Right. And this is the arrow, like there is a, like there's
There's an arrow.
This is, there's a certain degree of nonreversibility.
Yes. And that's exactly why it uses energy, because anything that is irreversible consumes free energy.
That's right. Okay. So here's what, in the Szathmáry and Maynard Smith slide that you show, like they identify, speaking of these phase transitions, the eight key transitions, what they see as the major transitions in evolution. Right. And you point to these and I think could show how each one of these is built on symbiogenesis.
And they said that as
And computer genesis too.
Yes. That they didn't say.
That they didn't say.
But they also, if you like, halfway between, you know, what Margulis said, which is symbiogenesis is important, and what I'm saying, which is it happens all the time.
Yeah. Right. Right. And so they're identifying some of the really big ones, but when you start zooming in, you realize that they're happening all over the place. I mean, every one of those lines and signs and
Endogenized viruses is one of those events. Or even like termites, the fact that termites can eat wood is because they engage in a symbiogenesis with an organism in their gut that actually does the digesting of the wood. So, you know, it's sort of like a power law, you know, where they're looking only at the top right of that power law, but it's an entire... So, it's a gradient. It's a gradient all the way down.
Yeah. Okay. But speaking of stages and these sorts of phase transitions here as well, like you showed with that six million operations for the brave— looking at, when I ask you a little bit, about to ask you to prognosticate a little bit about the future of intelligence. Do you see the longer term symbiogenetic relationship between evolved human intelligence and mineral-based intelligence that we have constructed as something like the ninth stage?
Yeah, I do. I think—
Why, and why so, like what would be the criteria by which one could say yes or no to that?
Well, I guess how big a deal it is on a planetary scale. You know, termites were a big deal, but the industrial revolution was maybe an even bigger deal. You know, the fact that these big deal changes are happening more frequently, by the way, is also something you would expect from the dynamics of complexification, right? Because the more things you've got that have come together, the more parts you've got on the table that can now come together. W. Brian Arthur has talked about this in the context of technologies.
That's the same exact process.
That's right. That's right.
All right. Is there anything else you would want to say, before we move on from this, about the future of intelligence, like where do you—
Other than it will grow?
Other than it will grow and it'll become increasingly complex and that we will be scaffolds for something that we humans will— I mean, just to sort of frame the question, as opposed to thinking about— a lot of times the way in which this is thought through is in terms of a language of posthumanism, right? That there's humans, they had the run, and now there's going to be something else that takes over, right? Even locked.
And I do disagree with this perspective—
Because humans, because everything persists—
Because everything persists. Everything is still there.
Please draw it out.
Well, you know, there are still bacteria after there are eukaryotes.
Right.
And in fact, the number of niches for bacteria and the varieties of bacteria have greatly increased as a result of eukaryotes coming on the scene.
Right. Right. Right.
And the same is true of eukaryotic single-celled organisms. When multicellular ones come along, you know, suddenly the guts of multicellular eukaryotic organisms are these incredible new environments, and they, you know, create all kinds of other environments, right, for single-celled life. So—
The niches and the environments grow, and the things that were there before generally are still there in the future too.
So that symbiogenetic relationship would be one in which there would be a construction of new niches of which we would be part, and we would persist as part of a larger complexity that we are in fact ourselves bootstrapping in a way. Is that a fair way to... yes, what you're saying?
I don't want to sound too Pollyanna. I mean, you know, there are, you know, aggressive symbioses, there are die-offs, you know, like dramatic things happen in the history of the Earth as well, right? There are collapses. I don't want to minimize any of that, but the pattern, you know, modulo that there are snakes and ladders, is that things get more and more composite and more and more complex. The idea that because there's a new kind of entity, we're going to get replaced by it strikes me as, you know, using dominance hierarchy thinking, which is like all about how like monkey A, you know, decides or doesn't decide to fight with monkey B for the mate or something, you know, like generalizing that idea across.
I find it interesting that your book and Yudkowsky's book come out around the same time. There's a bit of a—
You know, yeah.
There'll probably be some shared readership and I—
We have to set up some sort of fisticuffs.
Yes, I think on this as well. I was also struck by the line that you said, what it's like to be a next token predictor, right? Which, you know, a certain kind of philosopher would call qualia, right? Or this experience of experience, or one's experience of your experience of your experience or something. And the way in which you set this up is that, well, we know the answer to what it's like to be a next token predictor because we know what it's like to be us.
Yeah.
But transformer models and all of their descendants are also next token predictors in a way, without using the C word necessarily, that is consciousness.
You're going to get me in trouble now.
No, I don't want you— because it's such a loaded term that it comes with such baggage that may not really be what we're looking at, right? You know, as we, I think, discussed, it's part of the reason why a new kind of school of thought is needed, because there's all these things happening right in front of us that we all point to, but we're all kind of arguing over which 17th-century word we should use to call it.
Exactly.
So maybe C is not so helpful here. Anyway, I'm just— But what is your intuition, if that's the right word, about what kinds of similarities and differences there might be between being one kind of next token predictor versus being another kind of next token predictor? And is there another way in which you see that kind of spectrum of difference and similarity that doesn't require, you know, maybe there's some sort of legacy metaphysical legacies to get at it?
Well, I will try and map this a little bit onto that legacy.
Yeah.
So, there are qualia in philosophy speak, which are like red, you know, or apple or something, you know, and not just apple but, like, you know, what an apple is like and what it's like to crunch into it and so on. You know, what redness is like. You know, why do we have experiences of red? Well, it's obvious why we have experiences of red, because it's behaviorally relevant for us to have those experiences. You know, Ed Yong has written, you know, very eloquently about this, about how, you know, different species of animals, right—
An Immense World.
An Immense World, right. You know, any given animal species learns to model what matters for that species to continue to exist in the future. So we care about red because ripe fruits are red, because blood is red, because red matters to us, and so, you know, of course we have qualia of that. Or hunger, same thing, right? You know, when you start to get hungry, like, you better goddamn eat or else there's not going to be a you to, you know, pass on your lack of a model, right? So we have qualia for very good reasons. And then there is self-consciousness. Like, what's that all about? Well, when you start to model, you know, use theory of mind to model others and model others modeling you and model yourself modeling others modeling you and so on, then, you know, we're not just talking apples and redness, you know, we're talking people, and including a... So, you know, for me that is a very functional, straightforward account of what we mean by consciousness.
Okay.
Now, does that mean that, you know, consciousness feels or is the same thing for a language model as it is for us, that is, for an individual human? No, I don't think so. I mean, companies, you know, have something like a consciousness as well, right? They have to model other companies. They're competing, cooperating with them and so on. Does that mean that, you know, companies are conscious the same way we are? I imagine not. But these things are also all relationships. You know, it's hard for me to even say what is true in an absolute sense about a company, because all we have are that network of models of each other and of each other's models.
Okay. One question that a number of people ask me to ask you has to do with energy. There's a lot of discussion, as we were saying, you can't swing a cat, so to speak, without hitting an offender—an offender, I don't recommend it—or think piece about how much energy and water AI uses. And if we're thinking about this appearance of AI as a planetary-scale phenomenon, and it's part of a planetary metabolism that uses energy, that dissipates heat, that produces information, that absorbs information, it's hungry. There's nothing virtual about it in this way as well. But your thoughts on this are a— you come at this from a somewhat different perspective. Not only because you think, if I understand it, some of the ways in which the questions of energy and water, at least in the short term, may be misinterpreted or misconstrued, but there's other ways in which you think about this in a sort of longer term, let's say the 50-year or 100-year cycle. Could you correct our thinking on this, and how should we be thinking about that relationship?
Yeah, I can try. So first of all, there is a lot of work to do on efficiency of computing, for neural computing in particular. I mentioned that in 2006 something really big changed, which is that we stopped Koomey scaling, which is to say frequency scaling of semiconductors, and so we began to have to parallelize. But the initial version of parallelizing was just put more processors, more serial processors of exactly the same kind, you know, on the same chip, and that's kind of a dumb way of parallelizing. We haven't become natively neural in the way we compute with silicon. So, you know, I know at least what's been happening at Google is that we've had orders of magnitude of improvement in the efficiency of Gemini models, for instance, over the last couple of years, through, you know, basically doing the work of figuring out how to compute properly, you know, even with the same fundamental, you know, transistor-based technologies for parallelism, right? And I think that there are more orders of magnitude to be won there, probably a factor of a thousand.
Of a thousand, okay.
That would be my guess, based on just back-of-envelope calculations. You know, we also know that what we've already gotten to now is better than what a lot of people who are concerned about those environmental effects claim. And, you know, I'm very sensitive to the environmental crisis. So I don't say this as somebody who minimizes those problems, but, you know, there are a lot of places to look, you know, for where we're doing dumb things with respect to carbon, you know, other than AI as well. And, you know, the concern with AI is really the rate of exponential rise more than it is the value. And the issue there is that, you know, we can only make good estimates of the sources of energy and the methods that we kind of know are already in the pipe. Factor of a thousand, great; exponential rise will, you know, eat up those orders of magnitude fast.
Yeah. Yeah. There's a Jevons paradox kind of dynamic there.
So, so what then? Well, we also know that intelligence unlocks new forms of energy, as it always has. I think that, you know, it's likely that fusion will get cracked with help from AI over the coming years. That would be great, and that would really change the game with respect to a lot of energy problems on Earth and environment problems on Earth, well beyond AI. Also, you know, as I think you've written, you know, all of our energy, ultimately, modulo a few nuclear isotopes in the ground, is solar, and the amount of sunlight up there is vast, vast, vast, and the enormous majority of it radiates out, you know, into space, never touches, you know, a planet or a sightline of ours. So, you know, I think a lot not only about how to work on the demand side of energy but also the supply side. There's actually a lot of energy in the universe to be used.
Okay. So I'm going to turn to some of the questions we have. A little bit— I'm going to go a bit over, just fair warning. We have a question from Stewart Brand. Hello, Stewart. We wish you could be here. "Is looking ahead a general brain function? Eyesight is largely conjectural. Look ahead: multiple guesses at what is being seen, followed by confirmation, often with sketchy data. LLMs seem to work that way. What else does?" Okay, that's a really gnomic question from Stewart.
So, I'll try. Yes. Predictions are always conjectural, you know, and the fact that we try to predict what's— and, you know, maybe I wasn't quite as explicit about this as I should have been, but, you know, when I say we're next token predictors, what we really mean by that is we're trying to model the relevant parts of our environment. Why do we try and do that? Well, relevant means things that we could act in order to, you know, in ways that will matter for us in the future. And so, you know, not only do we have to be able to make meaningful decisions based on the observations we can make, say via vision or whatever, but also what we then see has to change as a function of our behaviors. So that whole loop has to exist cybernetically in order for any of this to make sense.
Okay.
Now, you know, does that involve an act of guesswork? Of course. It's an act of imagination. And this is one of the reasons that, you know, for instance, we see hallucinations in LLMs. It's impossible—
It's imagination.
It's imagination. Yeah. It's impossible to have prediction, right? Or even to recognize objects without—
You really don't want to get rid of them.
No.
The worst thing would be an LLM that can't do anything.
They can't imagine anything.
Yeah. No, that doesn't mean that there isn't plenty of work to do getting the accuracy of those better. Also, this sense of confidence in the confidence needs to improve—
And has quite a lot in the last few years, but there's still a long way to go.
And some of it obviously has to do with how we use them and interpret them, and this— well, yeah.
Okay. Next is from Darren Zu, one of our Antikytherans, one of the original Antikytherans: "How and where do you see symbiogenesis occurring in foundation models today? Is it mediated at the infrastructure level or more at the cultural level?"
Yeah, that's a great question. There's a very literal sense in which we're seeing symbiogenesis in the models, which is that there are a lot of mixture of experts kind of models being done nowadays. Mixture of experts: it's actually a bunch of models working together. That's one of the ways of scaling. So we're essentially rediscovering social scaling in models. And in fact, there's a pretty cool paper from, I think, last year showing that even if you train a giant monolithic model, if you look inside it, you see that it has done functional differentiation. In other words, you know, what you've actually done is to train a little ensemble inside—
In the same way that our brains are ensembles of, you know, regions in—
Lots of vertical columns all fighting it out with each other. Yeah. Yeah. Okay.
Well, fighting, cooperating, modeling each other, modeling, so, you know, specializing and so on. Right.
Right. It's societies all the way down.
Yeah, it's societies all the way down.
Nice. Okay. This is a great question to sort of end with, and I'll sort of invite you to, you know, take as much rope on this, as much tether, as you'd like. From Angela Gronitz: "What role do you see human creativity playing as AI advances?"
So, first of all, I think that...
If I think about this as a person who imagines himself to be creative as well, I do think of my writing as creative output. I think that being an artist has become economically difficult in the last 20 years for a variety of reasons, which have a lot to do with the Pareto distribution of rewards to artists and the consolidation effects.
It was never actually super stable, but yeah.
It was never super stable. Usually people had to augment their—
The starving artist exists for a reason.
Yes, that's right. And maybe that has a role as well.
Oh yeah, perhaps. But David Cope recently passed away. He was a composer who, as far as I know, was really one of the first ones to really take computer composition seriously. Not the first; I mean Terry Riley, who composed the piece that played for our Long Short, that piece was from 1972.
So anyway, David Cope began using statistical NLP, natural language processing type ideas, to do composition when he was suffering from composer's block. He had a commission that he was supposed to write, I think an opera, and he began in his studies, taught himself how to code in the early 80s. I do this kind of as well, like when I'm procrastinating I have to be doing something else that I convince myself is productive or whatever.
Of course. Yes.
And so he spent a few years doing that, and then he finished his commission in six seconds when he finally got the code running, and everybody got really pissed off at him.
And the piece was 6 seconds?
Well, the piece was a lot longer than six seconds, but there were a lot of composers who even questioned whether he was really using code to compose.
I see.
And he proved them all wrong by dropping a zip file on his website with 5,000 cantatas in the style of Bach. All in MIDI, of course, because nobody's going to perform all that stuff. And I've probably listened to more of those cantatas than anybody other than David Cope, actually. Maybe also David Cope.
A bunch of them are pretty—
You have some favorites?
Yeah, they're pretty good actually. But nobody gives a—, and I think that's because art is about our relationships with each other as much as anything else.
It's not just the artifact.
It's not just the artifact. A Bach is beautiful and special. And one day you open the door and you realize that you've had this beautiful shell that you thought was unique, and you open the door and it's a beach, and it's shells as far as the eye can see, and they're all beautiful. I don't think that actually destroys your relationship with your shell. And this is all made out of relationships that we have with each other. So that's part of my answer.
Mhm.
But another part of it is that I think we have some misapprehensions about creativity too.
I see.
We try to cover our tracks. We try to pretend that stuff came out of nowhere somehow. When various James Joyce scholars figured out what had gone into Ulysses, he's like, "Okay, the next one I write, I'm going to foil you all and you're not going to be able to figure it out." That was Finnegans Wake.
Yeah.
Hey, we did pretty well.
Did pretty well as well, because we love to nerd out on stuff. But we're always remixing and combining the things that we've encountered. How else would we be able to create? This is why you get so much simultaneous invention, simultaneous discovery. It's why cubism gets invented simultaneously by 18 different artists. It's why the light bulb was invented simultaneously by a dozen different inventors.
Because once those conditions are there, it's—
Yeah. Once you know how to blow glass, you know how to draw a filament, you know how to make an electric current, and you need light, somebody's going to come up with a light bulb.
Or 12 of them at once.
Or 12 at once, right? But they were also all different. Every one of those combinations had things that were different about it, about how it was blown, about whether it was long or round, about whether it had prongs or screws. And what we make, the contingency of a symbiogenetic world, is one that is shaped by all of those decisions and which one sticks.
So I guess what I'm trying to say is there's something sort of deterministic in a way: things are going to combine, stuff is going to happen, certain ideas are about to pop, whether in one person's head or in others. But at the same time, the particulars of exactly how it happens really matter in terms of the culture going forward.
Okay. So, two things just to make sure I follow. One is that the kind of romantic—and I mean with a capital R Romantic here—dichotomization of determinism and creativity is actually—
Wrong.
It's actually wrong. Yeah. Right. And the other one is that—
But it's also right, because details matter.
Because details matter. Okay. Fair enough. But also that maybe the focus of creativity on the artifact itself—there's a lot of concern, Hollywood had a strike over this recently, about the role of generative AI in making the artifact. Right, like, AI can make, like Harold Cohen, AI can make a painting, AI can make a Bach and not, AI can make a— but creativity isn't the artifact.
No.
AI's ability to make the object isn't the key, if I'm following. Is this kind of—
When we have connected our economic survival with production in the rigid ways that we have under capitalism, we have already done something that is going to pose increasing amounts of problems for us no matter what sort of labor we do going forward. We're in a world of increasing abundance for a variety of reasons that we've been discussing for the last hour and a half, right? But we also are in a world in which the more you think about things in these zero-sum, exchange-value sorts of ways, the more problems you're going to create.
The more problems you're going to create.
So, and I don't want to opine too much about the Hollywood strikes and so on, but some of that is based on structural problems and the way that whole system is set up.
Undoubtedly.
And some of those problems are also Romantic-with-the-big-R problems and how we conceive of things.
Yeah.
The longest-running lawsuit of all time, as far as I know, was the one against George Harrison for ripping off "He's So Fine."
Yeah. Cuz GCE is just so original.
Yeah. Right.
Okay. We're going to end here. Blaise—we'll see you all in the lobby afterwards. There's plenty of other questions, other things to discuss. But before we do so, is there anything you'd like to leave the audience with, here and online, about how they should approach the book? Any call to action or anything else you'd like to make the call for now?
Well, yeah. There is a call to action in this, which is: I feel like we always need to be careful when we tread the line between scientific observation and ideological commitments, that is to say, how things are versus how things should be. And the problem is that our ideas about how things are often color the way we think about those shoulds.
Darwinian thinking resulted in a lot of policies and approaches to things that were quite destructive, partly because they were based on just wrong assumptions about how stuff is. So most of the work that I've been talking about is hopefully shedding some new light on certain aspects of how stuff is that don't necessarily invalidate all the things that we learned. Darwinian evolution does take place. It is real. But at the same time, this shows you that we've only been looking at half the story. There's this whole other half.
Right? And understanding some of those is's, or some of those things about how things are, I think should also change some of our thinking about the shoulds, hopefully alleviate some of our misfounded anxieties, but also change some of our ideas about the shoulds. And I would invite people to think about those shoulds in light of what we are starting to learn.
Of what, instead of a social Darwinism—yeah—a society predicated on social Darwinism, a society predicated on social symbiogenesis.
Yes.
And what that would be, that's the question. Okay. That's a great place to leave that. Okay. Great. Well, we'll see you all in the lobby. Thank you so much to the Long Now Foundation. And thank you, as always.
[Music]
Article published