Peter Mattis on B-Trees, Distributed Databases, and Writing Database-Grade Code with AI
The Pragmatic Engineer
Peter Mattis co-created GIMP in college, helped build Gmail's original storage backend, was on the founding team of Google's Colossus file system, and co-founded Cockroach Labs, where he is CTO. By his own account he has been a prolific coder for 30 years: about 100,000 lines of production code in his peak pre-AI years. He then largely stopped coding between 2022 and 2024, and has returned with AI coding agents. In this conversation he says his current output is "a bit insane," and that it is database-quality code rather than throwaway prototypes.
The conversation runs from his early projects through Google's storage infrastructure and the design of CockroachDB, then turns to how AI changed his work: why agents need firm direction on testing, what he expects to happen to code review, and why he thinks domain experts gain the most from these tools.
From Mechanical Engineering to GIMP
Mattis got into computers through games, and through a house that always had machines like the Apple II Plus and Apple IIGS; his mother had done programming at IBM. He typed in BASIC programs from books and magazines without really understanding them. He still started college as a mechanical engineering major, following his father, because he didn't think "there was any money in computers." In his first semester the engineering homework ran to six pages per problem, while a CS course he took at the same time seemed easy even though almost everyone else was failing it. He switched majors.
GIMP began because he and his roommate, Spencer (later his Cockroach Labs co-founder), were bored with a course, possibly a compilers course. Mattis had done journalism in high school, knew some computer graphics, and wanted to build something like Photoshop. They worked it out as they went, reading papers where they could. Mattis also wrote much of the graphics library, GTK, which has since evolved enormously. He argues you need some naivety to start a project like that: "if you know how hard it's going to be, you would never start it." Once you begin, the work snowballs.
He tells one story as a lesson for founders. A couple of weeks before their first public release, someone posted on a graphics Usenet group about their own graphics program, which apparently did everything GIMP did and more. The two students decided to release anyway. They never heard from that person again. His advice now: if you have an idea, a dozen other people have probably had it and a couple may be working on it, but most never follow through. Don't be put off by a pre-announcement, and treat competition as something to enjoy, not fear. He says the same applies to what Cockroach Labs is building now.
GIMP also shaped his career. The very first Google logo was made in GIMP, according to Mattis, and in 2001 Larry Page or Sergey Brin (he isn't sure which) looked him up and invited him to interview. Google was three years old. He turned it down because he lived in San Francisco and didn't want to commute to Mountain View. After a year at a startup that wasn't going anywhere, Google called again. He joined on April 1, 2002, with smaller stock options than the first offer, though he says he did fine. He also remembers people telling him they learned to program by reading GIMP's source, code he wrote 30 years ago when he "wasn't nearly as good a software engineer."
Building Gmail's Backend
At Google he joined an internal email project code-named Caribou, which became Gmail. A prototype built on Google's existing search and retrieval systems already existed and was then completely rewritten. Mattis worked on the rewrite: the backend for message threading, message storage, and indexing. He was on the project for about three years, through its launch on April 1, 2004. The date made the product look like an April Fool's joke.
The host recalled that free email accounts then offered around 10 MB. Mattis thought Hotmail or Yahoo Mail was at about 4 MB. Gmail offered roughly a gigabyte, and search came back almost instantly. Internally, he says, the launch was described as "a shock and awe campaign for the industry." Feasibility came from building on GFS, the Google File System, together with Google's search and retrieval expertise. Threading was a deliberate decision from the start, which he says was unusual for email then.
B-trees were part of the threading design. When a message arrived, the system used the search index to match its subject or message ID to a thread ID, and B-trees also tracked things like per-thread unread counts. He thinks it was his second B-tree implementation; he now counts about a dozen. He says all that code has since been rewritten.
On economics, the team estimated a per-user cost of "a couple dollars per year" and asked whether it could still be offered free. The answer took "a little bit of a leap of faith." Mattis credits Paul Buchheit, whom he says was not on his part of the work, with building ads into Gmail in one night by reusing existing ad functionality, which grew into a large business. Mattis wasn't involved in the invite-only launch either, but he recalls concern about load and says the invite system did two things: it throttled growth and it made invites something people wanted.
Google3, BUILD Files, and the Road to Bazel
Around the same time Mattis worked part-time on Google's build system. Google had a monorepo. The first version predated him, and by the time he arrived it was "Google 2," built around a single huge Makefile (possibly with some sub-makefiles) that was showing strain. Mattis laid some of the groundwork for Google3, alongside many others.
His idea was that Makefiles are like assembly language for dependencies: they can express everything, but writing them by hand is error-prone, and it's easy to miss a dependency. Google3 introduced BUILD files in a stripped-down Python dialect. A generator compiled them into a monolithic Makefile that nobody had to write by hand. Later, engineers realized the Makefile step could be dropped by implementing the dependency-update engine directly. That became Blaze internally, then Bazel externally, and engineers who went from Google to Facebook built Buck. The host noted that these systems matter for large codebases because they open up caching and other performance levers, and Mattis agreed.
Colossus and Erasure Coding
His storage work on Gmail, plus an unsuccessful research storage project, got him invited to the founding team of Colossus, GFS's successor. As far as he knows, it still exists at Google in some later form.
He compares Colossus to something like S3 from the outside: a flat namespace with little hierarchy, not a POSIX file system, and at first without full directories or permissions. Files are stored on a fleet of servers writing to hard drives (now SSDs), and replicated so that crashes don't lose data.
He calls erasure coding "a big breakthrough" for Colossus. Colossus used Reed-Solomon, though Mattis notes better codes now exist. GFS stored three full copies of everything. With Reed-Solomon, he recalls overhead around 2x, and he wasn't certain of the exact parameters. The result used less space and offered more redundancy. For a mental model he cites a scheme he thinks S3 has discussed publicly: split data into nine chunks such that any five can reconstruct it, so any four can be lost. Often the first five chunks are the data itself and the other four are parity, though he said he had forgotten some details. He admits he never fully understood the math, Galois fields and so on, and regrets not taking more math. But the math was proven decades earlier for communication networks, so the team's work was the engineering needed to make it run in a storage system.
The other big problem was scale. Mattis recalls GFS topping out around a thousand machines per cluster, and Google wanted ten thousand. The single GFS master holding metadata (file names, and for each file its list of 64 MB chunks, which the master scanned periodically for repair work) was a bottleneck. The team chose to put Colossus metadata in Bigtable. But Bigtable was GFS's biggest user and was supposed to run on Colossus, so there was a circular dependency. The solution was layered: a foundational Bigtable that didn't use Colossus, Colossus storing its metadata in that Bigtable, and ordinary Bigtable running on Colossus. Mattis says he is proud of this design and "a little bit embarrassed by" it. It ran for years before being replaced at some point he doesn't know. His lesson: use what you already have to get off the ground and remove it later. Building a proper metadata layer from scratch would have delayed Colossus.
Latency, Hardware, and "Speed of Light Numbers"
The host asked how distributed file systems deliver high throughput, high availability, and low latency together. Mattis's answer was that latency in these systems isn't actually very low. He says services like S3 and Google Cloud Storage (built on Colossus) have excellent throughput, but first-read latency is around 20–30 ms when data sits on hard disks. SSDs bring that down, though not to what state-of-the-art SSDs can do.
He is struck by how fast hardware has improved. By his figures, a hard-drive read is 5–10 ms, and an SSD read over NVMe is 30–50 microseconds. Network round trips were milliseconds when Colossus was built; within a zone at Google or Amazon he puts them around 100 microseconds now. "The hardware guys have really done a good job."
The host brought up Simon Eskildsen of Turbopuffer and "napkin math": design from what the hardware can theoretically do, not from what a human will notice. Mattis calls these "speed of light numbers." Human-facing design can think about 100 ms of tolerable delay, or about 4 ms per frame in games. Storage systems have to be designed for machine speeds. Sometimes the limit really is the speed of light, as with cross-zone and cross-region latency over fiber.
That led to a tangent. Mattis says the fastest way to send data across the globe is to send it up to space, bounce it between Starlink satellites, and bring it down on the other side, because light is slower in fiber than in near-vacuum. For shorter distances, such as New York to Chicago in high-frequency trading, he says the distance isn't long enough for satellites to pay off, and firms use microwave links instead.
He then turned to single machines. Getting top performance from multithreaded programs means avoiding synchronization, partly with lock-free programming but mainly by arranging work so locks aren't needed. It also means paying attention to the whole cache hierarchy: registers, L1, L2, L3, memory, disk. High-frequency trading firms think about this constantly. In his view most software ignores it and gradually gets slower.
Rethinking Standard Data Structures: B-Tree Maps and Swiss Tables in Go
Mattis loves data structures, and two of his side projects improved widely used libraries.
The first was at Google. A colleague working on Gaia, the internal system that maps logins and user IDs to user metadata, noticed that STL maps, which are balanced binary trees, were using a lot of memory and showing up in profiles. Every node traversed in a binary tree is a different cache line, and for a map from a 4-byte integer ID, two pointers per node is a lot of overhead. The colleague considered skip lists, which Mattis also recommends studying. Mattis thought the problem looked like a B-tree and built one that matched almost all of STL map's semantics. The exception was pointer stability, since inserting into a B-tree node shifts elements. If you don't need that property, the B-tree is both faster, thanks to spatial locality, and smaller, because it has fewer pointers.
His short explanation of B-trees: the best way to store about eight sorted items is a sorted array, not a tree, and a linear scan is often faster than binary search. When a ninth item arrives, split into two nodes of four and five and add a parent pointing to both. Recurse from there. He jokes that everyone should implement one, then says nobody should anymore, because tools can now write it with all the many optimizations included.
The host said they had assumed standard libraries are already as fast as they can be. Mattis's reply was that these improvements came from engineers who saw something odd in a profile and asked why. At the time, Google didn't have a dedicated team for internal libraries; engineers contributed to a shared codebase from their own projects.
The second project came years later, from outside Google, in Go. Mattis follows research on data structures. Hash tables are usually taught with chaining (linked lists per bucket) and open addressing (probing further slots). He read about Swiss tables, which he believes came from Google's Swiss office, though he wasn't sure. Several people had tried Swiss-table implementations for Go's built-in map, but Go's runtime map was already heavily optimized and hard to beat. Mattis worked on it on and off, then made real progress on a long flight to Bangalore for a business trip, getting some benchmarks to run faster. He combined ideas from others on the Go issue tracker, got help from the runtime team, and reached an implementation that was faster on most benchmarks, though not all. Cockroach Labs used it. Mattis stresses that putting it into the runtime was "a whole other level." The Go team took it over and finished it. He also mentions a colleague who contributed a fast CRC implementation based on Intel papers, as an example of outsiders contributing to the language.
Leaving Google: Viewfinder and an Acqui-hire
After Colossus, Mattis worked for a while on Google Goggles, which he calls a technology "before its time"; the Android phones then weren't powerful enough. He felt he might be stagnating and left to start Viewfinder, a mobile photo-sharing startup around 2012, alongside Instagram. He says they never worked out go-to-market or viral growth. Square acqui-hired the team for its engineers and got the IP, which he doesn't think Square ever used. The company had raised little money and repaid its investors roughly in full, maybe with a small haircut. That mattered to him: investors expect losses, "but you kind of feel bad."
From Spanner to CockroachDB
Mattis explains the difference between Colossus and Spanner, a sister project. Colossus is a distributed storage system for large, append-only files from 64 MB to gigabytes, which can't be updated in place. A database stores huge numbers of small, typed rows and columns, which is a different engineering problem. Distributed databases usually sit on a distributed storage system, as Spanner sits on Colossus. Colossus's append-only design pushed Spanner toward immutable files and log-structured merge trees. Google didn't invent LSM trees but popularized them with LevelDB, which came out of the Bigtable and Spanner work and led to RocksDB. Mattis later wrote Cockroach Labs' own LSM engine, Pebble.
The idea for CockroachDB came up at Viewfinder. Mattis, Spencer, and Ben Darnell (also a Google alum who joined Viewfinder) disliked the databases available compared with Google's internal Bigtable and Spanner. He mentions HBase and Riak. They sketched a design but set it aside: "guys, we're doing a mobile photo sharing site, we shouldn't build a distributed database." He adds that perhaps they should have pivoted. At Square they saw the same data-storage problems, and Spencer persuaded management to let him work on it part-time, then brought in Mattis and Darnell. Once it drew outside attention, they spun it out.
Viewfinder had avoided VC money, and Mattis says he wouldn't recommend that. The VCs he has worked with are "some of the sharpest people" he has met and offer market perspective that technical founders especially lack. Cockroach Labs raised money soon after a short Bay Area roadshow.
The name was Spencer's. Cockroaches are thought to be unkillable and to survive a nuclear apocalypse, and that was the goal for the database. (GIMP's name, Mattis adds, came from Pulp Fiction.) He describes a recent campaign called "performance under adversity," which he says showed an existing capability: run a workload, kill a node or sometimes a whole region, and the system keeps going. He says customers have reported data-center fires in which everything else went down and CockroachDB kept running.
Who Needs a Distributed Database, and Why Sharding Belongs in the Database
Mattis says companies often call after a disaster. He describes a large bank, which he didn't think he could name, that suffered a serious region-wide outage from a weather event. The CEO then issued a mandate that such failures couldn't happen again, and it flowed down the organization. Other customers hit capacity limits: one early customer on AWS reached the largest instance available. The standard next step is manual sharding into 10, 20, or 100 shards, which Google itself did for a while. Cockroach Labs' view is that the application developer then becomes a database developer "and they're doing it poorly," reimplementing distributed transactions and indexes. The company believes that burden belongs to the database.
On sharding, the simplest form works like a hash table: hash each user into one of a fixed number of shards. When a shard fills up, resharding may mean remapping everything, much like doubling a hash table and copying every entry. Consistent hashing, which he associates with Cassandra, reduces this so a new node takes only part of each shard's data. CockroachDB follows Bigtable, Spanner, and HBase instead. It treats all keys as one ordered keyspace, splits it into contiguous ranges, and keeps an index mapping ranges to nodes. "You squint," Mattis says, "and everything's either a B-tree or a hash table," and that range index looks a lot like a B-tree. B-trees also sit under most secondary indexes in databases. He cites the 1970s-era paper "The Ubiquitous B-Tree" and says nearly every data system he has worked on has contained B-trees somewhere.
Transactions, Consistency, and Raft
Mattis explains ACID. Durability means committed data survives crashes. Atomicity means a set of changes either all commit or all abort, which makes errors much easier for applications to handle. Isolation is where the real difficulty lies: transactions should run as concurrently as possible but appear to have happened in some serial order. Linearizability is the gold standard, and serializability is the step below it. Weaker levels allow anomalies such as the textbook bank example, where a $100 balance check lets the same $100 be withdrawn twice. He says these produce "very egregious bugs" in balances, shopping carts, and many other places.
Eventual consistency, where a write to a primary may not yet show up on a secondary, is faster but moves the burden to the application developer. CockroachDB offers strong consistency: you read what you wrote, whichever node you read from. When the host asked whether that means writing to every replica, Mattis said yes, and the engineering challenge is doing it fast. He sees this as a pattern across computing, with systems growing more sophisticated so applications stay simple, and compares it to graphics, which went from drawing individual pixels to GPUs rendering billions of triangles.
On consensus, he calls Raft an alternative to Paxos, which was notoriously hard to implement. Consensus is impossible with two replicas: after a crash, neither side can tell whether a write reached the other. It needs at least three. Writes go to all replicas, and reads normally come from one. The majority "consensus read" happens during recovery after a failure. CockroachDB typically uses three replicas, five for some system tables, and customers can choose five or seven, with more replicas costing more storage and more time. Spreading replicas across regions adds speed-of-light latency of tens to hundreds of milliseconds or more. Mattis's general rule for distributed databases is to avoid chatty round trips: issue reads in one parallel batch, then write. Alternating reads and writes accumulates latency.
"Mission critical" to him means "tier zero" systems: trading systems, banking, and also DoorDash, which he names as a customer, and shopping carts where downtime can cost hundreds of thousands to millions of dollars an hour. At Google, he says, the rule was that Gmail and Search could not go down. The bigger cost is reputational: "grandma won't be able to pay her rent." The company has "some hundreds" of people, perhaps 110 engineers and about 150 in R&D, and revenue he describes as growing steadily. Building a distributed database takes years of stabilization, quality work, performance work, and enterprise integration.
Stepping Away from Code, Then Coming Back
In CockroachDB's early years all three co-founders wrote a great deal of code. Mattis's peak was about 100,000 lines a year. Around 2019 he decided to replace RocksDB, and the Pebble rewrite was 40,000–50,000 lines from him before others joined. He compares that with an industry figure he cites of about 3,000 lines a month per engineer, roughly 36,000 a year. He also says there is a limit to how much anyone can hold in their head.
For years he was told that CTOs should stop coding and lead from above. He resisted, then agreed at some point, and says the advice was right at the time. From about 2022 to 2024 his output fell. The Go Swiss table was from that period, but he did little on the core product. Being in meetings all day as an executive leaves no room for coding.
AI brought him back. He started with GitHub Copilot and tried Cursor, which he calls "glorified autocomplete," surprisingly good at finishing functions. The company wanted engineers to use these tools, and someone argued that leaders had to use them to guide others: if you don't know how to be a good engineer, it's very hard to manage engineers. The early models weren't quite good enough, but he watched them improve, first Sonnet, then Opus.
The turning point was Thanksgiving last year. For a long time he had wanted a tool to benchmark CockroachDB across a large matrix of configurations: CPUs per node, disks per node, nodes per cluster. It never seemed important enough to prioritize, though he suspected something interesting was there. Over four days, working with an Opus model (4.5 or 4.7, he wasn't sure), "the code just materialized." He stopped typing. He compares it to films where hackers type at impossible speed, except now it works. Last month, he says, he implemented another B-tree: about 10,000 lines of highly optimized Rust in about 30 minutes.
That tool led to more. In the office of the CTO, which is charged with innovation, he was looking at better autoscaling for CockroachDB clusters. Around January he reached what he calls "kind of a research breakthrough," through design discussions with models as well as coding. He describes them as sparring partners that aren't always right but are encyclopedic, and that let you explore ideas very fast. He dislikes their sycophancy: "I hate that sycophancy, it kills me." Since then the company has been working toward a new launch, much of it built by agent-assisted engineers. He estimates close to 100% of engineers use AI to some extent and considers the code very high quality, with caveats he explains below.
He first compared working with agents to management. The host objected that management involves conflict, performance reviews, and other human work. Mattis agreed and refined it: it's more like being a tech lead, or a "hands-on architect," for a 30- to 40-person organization, without the management overhead. You can keep correcting the agents until they get it right, faster than with people. His conclusion is that "you just have to be more ambitious about everything you do."
AI in the Product and Across the Company
Cockroach Labs' thesis, as Mattis puts it, is that nearly all future applications will be written by AI. Some handcrafted software will remain, as people still write assembly, but the share of AI-generated code will approach 100%. Humans may keep supplying agency and vision for many years. He expects a surge in the number of applications, and says the company is building multiple things with that in mind.
Inside the company, an internal platform for mini-applications launched earlier this year, and within a couple of months non-engineers had built 500 to 1,000 apps. HR builds small tools, and the CFO builds dashboards that previously couldn't be made. He adds that his wife needs software she can't build herself, that he has never helped her build it ("my own failing"), and that he expects everyone to get custom software eventually.
His Setup
Mattis used Emacs for 20-plus years, moved to VS Code at a colleague's urging, and ended up with about eight terminal tabs in VS Code running Claude Code. About six weeks before the recording, another colleague pushed him to try the Claude desktop app. It felt awkward at first, and then: "holy crap, this is awesome." He also keeps the Codex desktop app for a second model on critical work. One model writes a design, and the other is asked to adversarially review "what my colleague produced." It isn't perfect, but he finds it worthwhile. He praises Cursor, Factory, and Cognition as well, and says everyone is moving quickly.
He usually uses the best model available and names Fable and Astra. He isn't sure it adds much intelligence, but it saves him deciding case by case, and he would switch only when he needs speed. With the launch close, he tells senior engineers to use the best model. He finds his limit is about five to ten concurrent sessions, and each may run sub-agents: three for some research on the train ride over, sometimes 20, sometimes maybe 100 through features like Anthropic's dynamic workflows. At other times he runs none because he's thinking.
Quality, Testing, and the Future of Code Review
The host noted that quality seems to be declining across the industry and asked how a database company avoids more regressions from more output. Mattis agreed that more lines usually mean more defects. What offsets it, he says, is a firm hand with the agents, and he hopes model providers are listening. Agents "get a little bit lazy on the testing side." You need comprehensive tests and a full range of techniques: property-based testing, metamorphic testing, deterministic simulation testing, and more. Humans are lazy about testing too, himself included; at some point you decide it's enough and ship. He finds it easier to enforce that discipline with agents than with people.
He applies the same thinking to security and performance. Every commit can be reviewed adversarially by one or more security-focused agents. Given recent hacks and leaks, he points out that software has a finite number of bugs, and he expects security and quality to catch up. For him quality includes UX details, and a wrong UX element is now easy to fix, so there's "no excuse." Cockroach Labs' designer skips Figma, works directly in HTML, CSS, and JavaScript, and sometimes opens pull requests to the production UI (not the core database). He says she loves it, and so does everyone else, with no waterfall process.
Mattis was an early member of Google's C++ readability reviewers. He thinks code review was essential before AI. He still looks at most agent-generated code but with less scrutiny, the way you review a senior engineer's PR more lightly than a junior's, and agents are moving toward the senior end. Remaining risks are incomplete tests, missed security practices, and performance regressions. His answer is to add constraints so the wrong thing is hard to do. He suspects, without committing to this year or next, that people will largely stop reading code, as they stopped reading compiler output. Even where someone has to check the assembly, he asks why it should be a human. His example: he wanted a zero-overhead test-time abstraction that compiles out in production, and the model built a check that inspects the decompiled output to confirm only a few extra instructions appear. He would not have built that himself, and now it catches regressions on every change. He thinks models like working inside guardrails like these.
On flow, he says he still gets into the zone, though it is less intense and involves more parallel threads. He starts several experiments that would each have taken a week, reviews progress, and redirects when something looks off. He compares it to being a professor with a swarm of research assistants reporting back in hours, not weeks. Many experiments fail, and "you just got to let go."
The Analogies: Cars, F1, and Sorcerers
Mattis offers several analogies. Coding used to be walking, with some people running efficiently, but everyone moving on their own power. Then a car pulled up. You have to get in without knowing the controls, and you might hit the gas and crash into a tree, but you have to learn to drive. He thinks we are in "the era of the F1 driver," with a large gap between experts who know where the tools break and first-time users. On the idea of the "100x engineer," his sense is that AI stretches the existing spectrum and amplifies prior ability, though he says that isn't exactly true and it helps some people more than others. The host mentioned Boris Cherny and an OpenAI engineer (Tibo) as strong engineers before AI, and Mattis agreed.
Pair programming, with an android at the keyboard, captured how it felt a year ago. His current analogy is domain experts as sorcerers. He cites a ChatGPT session posted by Terence Tao on what he believes was the Jacobian conjecture. He couldn't follow the math, but Tao was clearly talking to the model as a peer and exploring ideas quickly, and it read almost like a foreign language. Knowing the right incantations in the right order produces something real; not knowing them produces "sparkle stuff." Asking a model to "build me a distributed database like CockroachDB" without expertise gets you something "hollow inside." An expert who specifies the storage layer, networking layer, data structures, and runtime concerns can get "something quite magical very, very rapidly."
Advice for Engineers
For mid-level and senior engineers, Mattis says you now have an excellent tutor always available. At Google he learned by reading changes from Jeff Dean (fast, high-performance code) and Sanjay Ghemawat (consistently elegant code), picking things up "via osmosis." Now you can have AI take apart expert code and explain it at any level: a diagram, like you're five, like you're ten, in French. He calls AI, in some sense, a translation tool, and says to keep asking until you understand, then move from learning to contributing by asking why something was built a certain way and what would happen if you changed it.
The host suggested using AI to learn the business domain as well, such as payments at a payments company. Mattis agreed and stressed curiosity: at Google he worked on Gmail but studied how search and Bigtable worked. He stressed personal agency even more. Anyone offering to teach you AI will be out of date, so you have to use the tools constantly yourself. He says he has learned more in the past year than in the previous five combined, and that he got noticeably better as a coder over this one year, as he did each decade before.
He ends with mixed feelings: the moment is "a little bit exhausting right now, but it's very exciting." He got into software to build things, and now he can build faster and drop some of the compromises he used to accept. Visible improvements like better UX and web animations, in his view, are only the surface of changes that go much deeper.
Met Larry and Sergey from Google. Somehow one of them looked me up, asked me to come interview at Google.
When Google was three years old.
Three years old. And I said, "No."
You did not.
I've always been a very prolific coder. I look back at my GitHub output. It's like peak years, maybe 100,000 lines of code in a year. Pre-AI, right? This is when you're back doing this manually. I look at that, it's like there's a max that you can hold in your head at a time.
We did erasure coding in Colossus. That was a big breakthrough. You have nine chunks of data, but any five of those chunks can be used to reconstruct it. And what this means is you can lose any four copies and you can still reconstruct your data.
What do you think good software engineering looks today compared to four years ago before we had AI?
I think the ambition has to increase.
He built an image editor while at college, designed the original storage system behind Gmail, and built many more large, complex, and widely used systems. This is Peter Mattis, co-founder and CTO of Cockroach Labs. Before we sat down to talk, Peter told me, "For the last 30 years, I've always been a prolific coder, but my current output is a bit insane. And this isn't vibe-coded junk, but database-worthy, high-quality, high-performance code thanks to working with strong coding models."
Today, we cover why B-trees are so important when building databases, and why Peter kept reaching for the data structure over and over throughout his career. How he wrote 100,000 lines of production code per year pre-AI, stopped coding between 2022 and 2024, and why he is back now. Why he thinks AI agents are lazy about testing and how this is an easier fix than it looks, and many more. If you're interested in distributed databases, distributed storage systems, or knowing how Peter manages to use AI to produce unusually high-quality and production-ready code, this episode is for you.
This episode is presented by turbopuffer, a ridiculously scalable, fast, and cheap hybrid search engine built on top of object storage by an engineering team I've really grown to like after spending time with them. The Turbo engineering team is doing something really, really cool. They've completely redesigned their storage architecture from first principles to make storage faster, cheaper, and more reliable at scale. If you follow turbopuffer, you know that their storage architecture was a massive part of their early success.
Redesigning a winning architecture is a big deal. It's one thing for your query plans to pass unit tests. It's another to bring each query plan to performance par while maintaining correctness and reliability in production. Here's a cool part. Turbopuffer is documenting the whole thing. Their new storage architecture, which they're calling Tuff V3, demands some hardcore systems engineering, and they're building it in public, sharing design decisions and benchmark results as they ship. They're keeping a worklog of their journey, and the first post just dropped today. Follow along at turbopuffer.com/v3. That is turbopuffer.com/v3.
Peter, welcome to the podcast.
I'm happy to be here. This is awesome.
I wanted to get into how did you get into tech originally? When did you figure out computers are interesting?
I figured that out in elementary school, high school. Gaming was a little bit of a gateway drug for me as a software engineer, as many other people. I remember early on my mom did programming at IBM at some point. I'm never even quite sure what she did, but we had computers always around our house, like Apple II Plus, Apple IIGS. I'm from that era, on up. I'd go to the bookstore, I'd find a book on BASIC or a magazine, type in programs, no idea what it was doing, but I was addicted. You could put stuff into these computers and you get interesting stuff out.
And then I got to college and I didn't think there was any money in computers. I didn't know anything about it. I started as a mechanical engineer, following the footsteps of my dad.
You started mechanical engineering as your specialization?
As my major, yeah. And I got in there and I'd done some computer stuff before. It was a real foolish move to do this. But first semester of doing these homework assignments in mechanical engineering, and they were god-awful, like six pages for a single problem. And I happened to take a CS course at the same time, and it was so easy, and then everybody almost was failing it. I'm like, "I'm in the wrong field. Let me switch."
And then you switched.
Then I switched. Yeah.
What was the first software that you built, either at college—it must have been at college—that you were like, "All right, this is a piece of software that I'm kind of proud of. That's a complete piece of software."
Well, the big thing that I did along with my roommate in college, we had this course. Was it a compilers course? I'm not even quite sure anymore. This is like 30 years ago. And we were kind of bored with it. So we wanted to do something kind of fun on the side. And I'd done journalism in high school, my junior, senior year, and I knew stuff about computer graphics and wanted to do something like Adobe Photoshop. So started just kicking the tires, and we built up this program that a lot of people know of called the GIMP. And along with the GIMP, I did a lot of the graphics library, GTK. This has since evolved just massively since then.
It's kind of interesting because after college, I stepped away from it. Didn't really stay involved much past my first year out of college, but definitely people still know me. And it led to some interesting events in my career.
And starting it was literally just you saying, "All right, I want to do something like Photoshop. How hard could it be?" And then this wasn't through what you learned in college, right? This was you figuring out how to build a graphical, I guess, engine, rendering, drawing, data structures, all of that stuff, right?
All that stuff. Figured it all out. I remember trying to look at some papers back then. My roommate was looking at papers. We're just figuring it all out. And it's one of these things you almost need to be a little bit naive to start anything like that, because if you know how hard it's going to be, you would never start it. So I think you have to have this level of, if you're really truly wise about the effort, you never get into it. But then as you get going, it just keeps on snowballing. It gets further and further along, and then at some point you're like, "Wow, this is really awesome."
But there's an interesting little tidbit associated with our work on the GIMP, which is a little bit known. I think we've talked about this before, but it's just fascinating. We were getting to the point where we're like, "We should release this to the public." And back then there were Usenet newsgroups where people would post stuff, and there's one on graphics. And I remember just a couple weeks before we were going to release our first version of the GIMP, someone else came on there. They were like, "I've been working on this graphics program," and it did everything the GIMP did, every last thing and then some. And we're just like, "Well, that sucks. I guess we'll just keep on working on it. It's been fun." And then we released the GIMP. Never heard from this other guy again.
And I just took a little lesson with that. There's always going to be someone else working on your idea. You can't get dissuaded if they pre-announce it. Nothing ever comes of it. And a lot of the marketing behind some of that stuff that you might hear is like—I don't know if we stole his thunder. I'm not even sure what happened. Never traced down what happened. But there's a real lesson there.
So there's a possible future where you read this announcement of someone saying, "I'm going to build all of this thing," and you go like, "Someone did it," and you go back and just do something else, and the GIMP never happens.
That's right.
Wow. Yeah. I guess especially today with startups, this sounds like just do your thing. Put it out there at the very least, right?
And the advice I give people is you might have a unique idea, but most likely there's a dozen people out in the world who've had the same idea. And there might be a couple of them working on it, but a lot of people don't even work on it. They're just like, whatever, they can't get going on their idea. So I wouldn't be concerned at all if you hear someone else working on your same idea. That's probably the case. We're working on some cool stuff now at my current company. I guarantee you there's other competitors out there working on the same thing. And just know it's a competition. You got to enjoy that aspect of it, not be afraid of it.
And what were some fun things that the GIMP led to?
Well, I got kind of tired with graphics, and that's why I moved away from it after college. I got into storage systems, and I bounced around. I worked at one of the early search engines, Inktomi, and then I got over to another startup. And through that time period, in that first startup I was doing, met Larry and Sergey from Google, because it turns out that Larry—I think it was maybe Sergey, I'm not quite sure—the very first version of the Google logo was done in the GIMP. So they knew of us. Somehow one of them looked me up, asked me to come interview at Google. I did, back in 2001.
This is when Google was three years old.
Yeah. Three years old. And I said no.
You did not.
I said no. Yeah. Here's the calculation in my mind. I was like, Google's down in Mountain View. I was living up in San Francisco, and I didn't want to do the commute. So I went and worked at another startup for a year. And that one, after a year, I saw it wasn't going anywhere. And they called me back up and they said, "Hey, do you want to interview again?" I was like, "Sure." "We're not going to be able to offer you the same stock options you got before." I'm like, "Okay." I can't remember what it was, and I still can't remember what they offered me the first time. But I probably would have made a lot more money if I had taken that first. But I did all right. I'm not complaining. But it's one of these things, now I think back about that. I'm like, a lot of good stuff came out of it.
Some point along the line, Red Hat was going public. They actually offered friends and family stock during their IPO to a lot of people. We got offered friends. I made a little bit of money off that. Not a lot, but it was just after college. It was actually quite significant at the time. So there's a number of good things that came out of that.
And I also used the GIMP. I was looking for, "Oh, I can't afford Photoshop, and here's the GIMP," and it did so many things. So I'm sure there's so many people you had a positive impact on, being able to use this thing. And the nice thing, what I really liked about it, is it was free, but I wasn't stealing any software from anyone. You see what I mean? This was at a time where free software was not as common as today. Free and open source was not as mainstream as it is today.
Yeah. Yeah. And then the other cool thing I've heard from a lot of people, because people still, I mention this, like, "Oh, I've used it." And then some people, software engineers, like, "I learned how to program by looking at your code." And I'm just like, "Wow, that was the code I wrote 30 years ago, and I wasn't nearly as good as a software engineer as I am now."
And then you got into Google second time around. You said yes. What did you start to work on?
Yeah. Well, I got in there. This was early days, 2002. It was April 1st, auspicious date.
Your start date?
Yeah. April 1st, 2002. And I got in there and they're like, "Oh, well, we're going to actually build email, Google email." And it wasn't called Gmail at the time. There's a code name internally. It's called Caribou.
Caribou. Yeah. And got in there, started working on it, and I was tasked with working on the backend threading and message storage and indexing system. And worked on that pretty hardcore for the first year and a half. Actually, I was on, maybe it was three years on it, through the launch, which actually happened to be April 1st, 2004. Do you remember this? It was considered an April Fool's joke.
Yeah, I remember. So correct me if I'm wrong, but the launch said Gmail with one gigabyte of storage, or something, infinite storage and like, star, one gigabyte. I'm not sure what it was, but back then most email providers would give you about 10 megabytes of storage, the free email providers, and then you could pay for maybe 50 megabytes or 100 megabytes, but that was really expensive. This launch, it looked like an April Fool's joke, because who could possibly give you a free email service with 20 to 50 times or 100 times more storage?
We were talking about this internally. It was like a shock and awe campaign for the industry. I think it was actually four megabytes for Hotmail or Yahoo Mail. And then you had this, and not only that we had so much more storage, but it was indexed really fast. So you could do a search and it would come back almost instantaneously.
But can you tell me internally, when the project started, "Okay, we're going to do email for Google," how did you and the team arrive at the point of, "Okay, we will offer all this storage and we'll do it fast"? Because this was at a time, if you can take us back, but what I remember is hard drives were still expensive. They were relatively slow. We're talking HDD. We're not talking necessarily about SSD, if I remember. But can you take us back, what it was like, what the constraints were like, and then how you innovated to actually do something that's never been done before?
Yeah. So at the time, Google internally had this large distributed file system called GFS, Google File System. And we were basically looking at numbers and being like, "Yeah, we think we can build on top of this." They had a lot of knowledge about how to do search and retrieval. It ended up being that some of the existing systems they had for search and retrieval, they started building the prototype on there, and then that was completely rewritten. That was actually what I got involved in, because I got in there and there was already a prototype, and then it was completely rewritten. So I was on that threading side. We decided to have message threading right from the get-go, which is also kind of innovative. That wasn't common in email systems.
And threading, to have the message thread, which I assume needed data structures on the back, and it needed storage and figuring out read intensity, those kind of things.
Yeah. There's some B-trees involved in that. I think that might have been the second time I implemented B-trees, and I think I've implemented them like a dozen times now.
How do B-trees relate to email threads?
It's just like you have to have some storage there where you have a thread ID. A message comes in, you have to look up. It was using the search index to match actually the subject or the message ID into the thread. And there are some other things taking place in the B-tree as well, because we were keeping track of the unread counts of threads and whatnot. So I can't even remember all the details now. This is ancient history. This is 2004. What are we in, 2026? Twenty-two years ago. So it's left my memory, but there were definitely B-trees. There was also this inverted index taking place there. All this code has since been completely rewritten.
What were the economics like in the team? You must have done the economics of being able to offer free email, which, of course, I'm sure you did some maths of how it could be subsidized, but to not make a terrible, terrible loss.
Yeah. We were kicking around ideas, and one of the ideas that had come up at some point was, hey, maybe we can put ads on this. And it was one of these crazy things. I wasn't involved in this. This is Paul Buchheit, who went on to do some cool stuff at FriendFeed and Facebook and ended up, I think, a partner at Y Combinator at some point. But one night he's like, no, I think I can just take some of our existing ad functionality and incorporate it in there. And it just went like gangbusters, and there's a huge business that grew up from that, which is incredible.
I think there's a lot of things that you have a sense of what can be done. Like, oh, we think this is going to cost per user a couple dollars per year. How do we monetize that? We didn't want to charge for it. Eventually they did charge for it because you have the whole Google Workspace stuff, but at the time it was more like, can we do this for free? Oh, it looks like it can be economical. And it took a little bit of a leap of faith.
And do you remember the launch? The reason I'm asking is because I remember that it was an invite-based system. Not everyone could get in, and I assume that must have been to control the expected demand, because you were offering something for free that was paid before, and it was pretty obvious that there would be massive demand. How did you think about it, monitor demand, decide how many people to onboard?
Honestly, this is a team effort, and I wasn't involved in that. I do remember being worried about the load that would happen, and then someone came up with the idea of, hey, we should do an invite-based system. And this also played dual roles. It constrained the growth, but it also drummed up this excitement: oh, can you get me a Gmail invite? I remember people asking me at the time. I was like, yeah, I can get you as many Gmail invites as you want.
And then after you built the system, where did you move on to? What was your next project? Was it the build system?
Well, there was the build system, and that was all muddled in my mind because I was doing that part-time when I was doing Gmail as well at some point. Google actually had this large monorepo. I think they actually still have the monorepo. And they started out with the google1 repo. That was before my time. Then they moved on to google2. That was what was there when I got to Google, and at some point we saw the strains of google2. And google2 was just one single monolithic Makefile. I think it actually had some sub-Makefiles, but it was this really large, unwieldy Makefile.
Someone came to me and was like, I think we could do something. I'd had some interest in the build systems, and I put the foundation in for google3. There was a bunch of people involved. I was doing the initial work on google3. And google3, the initial insight was, hey, Makefiles suck to write. We introduced this thing called BUILD files, and I decided to do it. It's just this stripped-down Python language, but it was still Python at that point. And what Gconfig spit out at the end was a monolithic Makefile, but one you didn't have to write.
Over time this evolved. It became Blaze internally. There's some other systems associated with it now. I don't even know how complicated it's got. I haven't seen it for a long, long time. But then that became Bazel externally. It became Buck. There's some other people who went to Facebook, and they're like, that was awesome at Google.
What was the reason that Makefiles or Make didn't really work? Were we talking about build performance? Are we talking about maintainability or readability?
Yeah, I think Make itself is okay in declaring dependencies, but it's a little bit like assembly language, and you didn't want to write all your dependencies in assembly language. And if you didn't know what you were doing, it was easy to make a mistake and miss dependencies and whatnot. And so my thought behind the BUILD files is, hey, you need to express these dependencies, but at a higher level and just cleaner semantics associated with it. And from that you can compile down into the assembly language. And then eventually folks were like, oh, you don't have to compile down to that. We can just implement the dependency update engine directly.
And then that's where the performance improvements were able to come in. Because at Google scale, when you have a large repo in general, my understanding is that the reason Bazel and Buck are so popular for large codebases is it can help you improve your build performance. It gives you a lot more levers to play around with, from caching, from being smart about cache generation, to obviously just raw performance.
Yeah, that's exactly it.
So you just dabbled and, okay, I'll make this BUILD file, did that, people took that over. But what was your next main focus?
Well, I mentioned GFS earlier, Google File System, and at some point we realized that there were some limitations in GFS, scalability bottlenecks. Because I'd been working on the storage system for Gmail, I actually dabbled in another storage system, a research thing that never went anywhere. But because we were working on that, I got invited to participate in the founding team of Colossus, which was the successor to GFS. And as far as I know, Colossus still exists. It's gone through multiple iterations at Google, but it's the second-generation distributed file system.
So what is Colossus?
When I say a distributed file system, externally you might think of something like S3, blob storage. It had a flat namespace, kind of like S3. You give names and there's a minor hierarchy there, but it's very limited. But it's not like a POSIX file system, so you don't have the full directory hierarchy. You didn't even have the full permission system. A lot of that stuff got added later, but the files are not stored on your local machine. There is a fleet, a service out there that has all the files. They're writing it down to their hard drives, or now SSDs, and your client can access that, and it's all replicated. So if there's any crashes and whatnot, you're not losing your data.
I don't know when, at some point S3 got erasure coding. We did erasure coding in Colossus. That was a big breakthrough.
Which coding?
We used Reed-Solomon. So for the audience who's not familiar with erasure coding, you might think of, I want to have replicas. And there's this thing in hard disks called RAID where it's like, I don't actually have to have full replicas. If you take A and B, you can XOR them together, and then you can have this third version. And there's more and more complicated versions of that. Reed-Solomon is, I think there's actually better codes now, but it's one of the known ways.
Known ways, yeah.
And we had to pioneer internally, oh, how are we going to actually make this work in a distributed file system?
Were you focused on latency, on being able to store data more efficiently?
Storing it more efficiently. So in GFS, and there's varying costs where it's, well, you're not storing one copy of your data or two copies, you're storing three. Triplication, three times as much storage you're having to use. And with Reed-Solomon, you can get that down quite a bit lower. I can't remember offhand exactly what we used for Reed-Solomon, but I think it was essentially 2x. But you get that with also the redundancy, too. So it's smaller and the redundancy is higher.
Because I guess the naive thing, if you're saying, all right, I want my data to be replicated at three places, you take three nodes, three machines, physical machines, and you say, copy one, copy one, copy one. I have it in three places. Great. If one explodes, I still have two. Wonderful. And then you're saying that the algorithm here is you could take not 3x the data but 2x the data, split it smartly across machines, or maybe you could take it lower, and you still have the thing where, oh, one of them explodes, I still have all my data because it's split in a—
That's exactly right. And the mental model, if you just want to understand at a high level, is essentially you might want to say, I want to have eight replicas of this data, or I can't remember offhand, I think S3 might use nine. They've actually talked about this publicly. You have nine chunks of data, but any five of those chunks can be used to reconstruct it. And what this means is you can lose any four copies and you can still reconstruct your data. And oftentimes it's more like the first five chunks are exact replicas and the other four are parity ones. I've forgotten, some of the details escape my mind, but it's along this line.
But when you come up with an algorithm, you can then prove that this algorithm will work, right? This is a little bit like, I know in software engineering maths and algorithms is a bit out of fashion, but in this case this is really important, because once you can prove that this algorithm works, it will work.
It will work. And the math behind here is Galois fields over GF(2), something like that. I don't even know. I never actually understood the full math behind it. I always regretted not doing more math in college, but you didn't have to. Reed and Solomon proved how this worked. I think it's back in the 1970s, something associated with communication networks. So you just take that and use that expertise, leverage that, and have to do all the engineering behind it to make it work in a storage system.
And then when building a distributed storage system like Colossus, what were other things beyond, okay, you want to store data in a resilient but efficient way? What were other problems that you needed to solve? I'm thinking things potentially like sharding and resharding, or metadata being important, those kinds of things.
Particularly for the scale that Google wanted to operate at, GFS had this scalability limit. I believe it was like you could have a thousand machines in a GFS cluster, and it kind of—
Which I guess sounds big until today it's ridiculously small, right?
It sounds big for— And then Google was like, no, we need to have the scale up to 10,000 machines. And there were some bottlenecks. There was this GFS master as a single node. It was a bit of a bottleneck. We're like, we need to have a distributed master to store the metadata for all the objects. And Google at the time happened to have this system called Bigtable, and so Colossus stored this metadata inside Bigtable. And one of the things that I'm proud of and also a little bit embarrassed by, one of the design choices I went down, but actually worked out, is we wanted to use Bigtable for the metadata for Colossus.
And the metadata, just for those of us not as into distributed systems, what is the metadata in a distributed file system?
It's the names of the files, and for each of the files, the files are broken into chunks. What were they, 64-megabyte chunks? And then you have to have the list of chunks for each file. And then periodically the master has to be scanning over this and doing repair work. There's more metadata than that, but that's it in a nutshell.
So we wanted Colossus to have this scalable service, Bigtable, to store its metadata. The biggest user of GFS is Bigtable, so we want Bigtable to work on top of Colossus. So wrap your head around the circular dependency.
Yeah, well, there's a bootstrapping thing, right?
Bootstrapping, yes.
Yeah, which one starts up? You need to mock something somewhere, right?
Yeah. The way it actually worked at the time, and they've since replaced this, because this is the way to get started, leverage what you have and eventually you get rid of it. But there is a foundational Bigtable. That Bigtable didn't use Colossus. There's Colossus using the Bigtable, and then there's normal Bigtable sitting on top of Colossus, and it all worked for years. So I don't even know when they got rid of that. They got rid of it at some point, but that was—
But I guess it sounds like you can make hacks that go really long, knowing that they're hacks, and they get you off the ground, right?
They get you off the ground, right? Because if we had to implement that Bigtable layer from the get-go, it just delayed how long it took to get Colossus built.
One of the things that strike me about a system like Colossus is it promises, or this was internal to Google, but even distributed file systems that are external, they will promise high throughput, high availability, and low latency. And to me it's always a bit conflicting. It's pretty easy to, I guess, build a distributed file system with my limited knowledge. I could probably do something where I have high throughput but high latency because whenever I write something, I write it out to all the replicas. You already mentioned one technique of doing it, but how did you reconcile, how did you get low latency while you have high throughput, while you also have replication going on in the file system?
Well, these distributed file systems, the latency isn't super low, in particular when they're running on hard disks, which Colossus was doing at the time, which S3 does. You actually notice the latency. So S3 is a high-performance system. GCS, the competitor from Google, which is built on top of Colossus, it's high performance, has incredible throughput, but the latencies are like 20 to 30 milliseconds per first read. And that is bounded by your hard disk latency. If you put it on SSD, it gets down to closer to SSD latencies, but not actually the state-of-the-art SSD latencies, which is this crazy thing that's been happening in our industry. It's just how much faster the hardware has been getting.
Reading from a hard drive, maybe 5 to 10 milliseconds. Nowadays I never touch hard drives. Reading from an SSD over NVMe, 30 microseconds, 50 microseconds. So that's microseconds. There's 1,000 microseconds in one millisecond. So we're talking a huge, huge difference.
Well, there are now startups or infrastructure companies that are starting to take advantage of the fact that they can have an NVMe layer, and they pull things up either predictively or not. But as you say, when the physical reality changes, you can build systems on top of it that should take advantage of it.
Yeah, absolutely. The disks are so much faster with SSDs. The networks are so much faster. It's just crazy how fast the intrazone latencies are in a Google or Amazon center. And it wasn't like that when we were building Colossus. I can't even remember what the numbers were, but milliseconds to do a network round trip, and now it's down in 100 microseconds within a zone. I just look at these things, I'm like, holy crap. The hardware
You guys have really done a good job.
Yeah. Sometimes I feel that our software should feel way more snappy, and there is some snappy software, but sometimes I almost wonder if we're getting too complacent with all the abstractions, or we're not even doing this napkin math. Simon Eskildsen at turbopuffer talks about this napkin math, where he was like, all right, here's the theoretical limitation of the hardware. A read from SSD might be, I don't know, 30 microseconds. How can I build a system that is as close to this as possible, as opposed to the other way around, saying, okay, a human will notice 100 milliseconds. Let's build around that.
Yeah. And sometimes when you're architecting something, you need to think about the human perceptible latencies, but oftentimes when you're dealing at the storage system layer—I know Simon working on turbopuffer, they're doing great stuff over there—you have to think about the machine scale and the machine speed, which is a lot faster than human perception. A human can tolerate maybe 100 milliseconds of delay, or if you're playing a game, maybe you need to have frame rates of every four milliseconds, but the machine wants much, much faster than that.
Someone says napkin math. I call this speed-of-light numbers, and sometimes it's literally the speed of light bottlenecking you. The cross-zone latencies between zones, cross-region latencies, is speed of light in fiber. You want to hear a crazy fact?
I love hearing crazy facts.
The fastest way to send a packet of data across the globe is to send it into space.
Is it because the speed of light is faster in vacuum?
Quite a bit faster.
Or, well, it's not full vacuum. No way. So you cover a higher distance.
Yeah. Well, I believe the way to do this, with Starlink, you send it straight up, you bounce it around between Starlink, you send it down to the other side. So you want to get it out of the atmosphere as quickly as possible.
But now, if we're talking speed of light, this will also go there as a digital transformation happening. So you would need to calculate how long it takes for that system to process and do it. But you're saying that even if you do this really well, it will be faster than beaming it through an optical cable, and an optical cable slows down the speed of light, right?
Yeah. The speed of light is only the speed of light in vacuum. In every other medium, it's slower.
I did a deep dive on the hedge fund industry, and they didn't tell me exactly. They said that they do use satellites and microwaves and some of these things. They will not tell you because this is their thing. But I had a suspicion that they might have found a faster way, and I think this is somewhat well known, but the details they're not going to get into. So they're probably bouncing stuff in space.
Yeah. One of the things that they do in high-frequency trading is, between New York and Chicago, it's actually not far enough distance-wise to make it worthwhile to send it into space. So they were doing microwave beams there. Yeah.
But if you really want to get faster, you need to build a vacuum tube between them and send it. Maybe they're doing that. I don't know. Maybe they're doing that.
Yeah. But the thing that I think is just awesome about performance nowadays is there's so many layers you have to be paying attention to in terms of performance that the rabbit hole just goes so deep there. You're almost certainly running on multithreaded systems, right? And you're like, "Oh, I need to have a multithreaded program. I need to have synchronization in there." Well, you get the best performance if you just avoid the synchronization. And part of this is lock-free programming, but part of it is arranging so you don't need locks at all.
You need to care about your processor caches. And there's this whole setup of caches above the CPU. You can think of registers as a cache, and you have L1, L2, L3, even your memory and onto disk. You have to pay attention to all those levels. And if you do, your performance gets way better. And if you ignore it and you're like, "Oh, I'm not worrying about the cache accesses. I'm just accessing data all over," your program will just be way, way slower. And some folks pay a ton of attention to this. High-frequency trading, they do this all day long. And in so many other places, no one pays attention to it, and you get this gradual degradation in the performance of the hardware or the software.
I did want to talk a bit more about low-level stuff, but not about the speed of light, but low-level data structures and programming language features. You made some contributions to the standard library, right?
Well, I've done a couple, not quite standard libraries. I just have been always fascinated by data structures. It's awesome. I think algorithms in general, sorting algorithms, they're just awesome.
You could probably explain insertion sort and everything. Could you explain quicksort as well?
Quicksort's a little bit harder.
That's the thing.
Yeah. And then—
No, but it's a smart one.
Yeah. And then you get to these levels of, "Oh, wait, someone really smart came up with this." So one of the things I worked on just as a little bit of a side at Google at some point was a colleague came to me and he was like, "You know what? We're using the STL map structure all over the place." And the STL map structure is a balanced binary tree. I can't remember if it was red-black trees or one of these other balancing algorithms that many CS college students implement. And he came to me, he's like, "I think we could do better because there's actually a cache problem here. Every node you're traversing down, you're going to a different cache line." And he was thinking about using something else, a skip list, to do this, which, that's another awesome data structure everybody should look at.
But at some point, I was like, actually, this feels more like a B-tree. So I'd implemented B-trees a couple times before and figured out how to implement a B-tree that implemented almost all the semantics of the STL map. It couldn't quite do it perfectly, and the reason is when you insert into a B-tree node, you have to shift stuff around, so you don't get pointer stability. This is fundamental, but if you don't need that for your use case, you actually can pack more data in.
So the thing about a B-tree, the really easy way to describe this is, you just have a small list of items, like eight items. The best way to store that if you want fast access in sorted order is just to sort the items, right? Literally not to have a tree at all. Yeah, for eight items.
Just have it in a very simple list.
Just an array. Sort the array, and then you can either do a linear scan over it, you can do a binary search, and oftentimes linear scan is faster. And then you think about that, well, if I want to store more than eight items, I can just have one node that has eight items, and then I have another node, and then you have a parent node that connects them together. And that is essentially—you think about building it bottom up. You start with just one node of eight items. Oh, I need to insert the ninth thing. I'll split into two chunks, and the two chunks, one will have four, the other will have five, and then you have a parent node that points to them. And then you just recurse on that. That's the B-tree algorithm in a nutshell.
Everybody go implement it. Actually, nobody should implement this anymore because nowadays we have something else that will implement this and all the optimizations, because there's a crap ton of optimizations that you can do on a B-tree.
But I just want to go back to this. There was already an existing implementation for maps, and then your colleague looked at the code and said, "I think we can do better." What I want to figure out is, in my mind, again, someone sitting outside of—I'm not involved in how some of these libraries or data structures are built—I always thought, and again this might be naive, but really smart people sit down, they look at the state of the art, they implement it, and there's no way it can be faster. In fact, I've had arguments in the past saying, "Oh, let's write a faster sorting thing." Surely it is the fastest. But if you could bring us a little bit of how—you've been inside—how it happens, and how other people like yourself and your colleague can say, "Oh, what if we try something else?"
Yeah. My recollection here is he was working on this big internal system, I think it was called Gaia, that actually had the mapping from—you log in, you have your user ID, and you have to look this up. And they were storing all this map from user ID and email and whatnot to the metadata about the user in STL maps. And you just noticed, well, there's a lot of memory usage here, and it shows up on profiles. And then we're like, well, what can we do to do better? And that was the genesis of it. He happened to be working on it, and they were working with me, and we just started noodling on this problem, like, oh, can we do something better? And it's not one of these things—I think now with Google, they have a whole team working on their internal libraries. At the time, it was more of everybody working on their own systems and contributing to a shared base.
But I guess it still goes back to what you were just saying, of just go down the layers, try to understand, and if something just doesn't add up, like suddenly, oh, there's this big explosion of memory usage, just ask the questions. Why is this? And if you're able to, or you happen to be like, oh, can we do something about it? Right.
Yeah. And one of the things that he observed early on, I think part of one of the things was it was a map from integer ID to something else. And if you look at a red-black tree, every node you have your value that you're storing in the map, and then you have two pointers. You might have an integer ID that's four bytes, and then two—
Waste.
Yeah. And you look at that, you're like, "Ooh, that seems like a lot of overhead."
Just, you could use a bit—
Well, the B-tree actually has better spatial locality, and that's what made it faster. But it was actually smaller as well at the same time because you had fewer pointers involved.
You also contributed to Go, right?
Yeah. Well, that came later.
Yeah, it came a lot later. Can we talk about that?
Yeah. Just one of these other things, I pay attention to when there's research papers coming out about new data structures, like hash tables. Hash tables are one of the earliest things you learn about in college in data structures. How do I map keys to values where the ordering is unimportant? That's when hash tables come in. And there's the very earliest ways to do this—I've implemented hash tables multiple times—you take your key, and it might be a string, and you put it through a function, and it spits out an integer. And then you map that into an array of buckets, and if multiple things map to the same bucket, you have to have a linked list.
You have a linked list. Yeah, this is a naive implementation.
Naive implementation, used quite frequently. And over time people discovered a lot better ways to do hash tables. That's called chaining of your hash. There's another technique called open addressing where, instead of actually having a linked list, you just hash it again and move on, or walk down to subsequent buckets, or subsequent slots, to find out where you should be. And I remember reading about this new technique. It came out of some folks at Google. I believe it came out of their Swiss office. It was called Swiss tables.
I believe that's where the naming—
Yeah. I believe that's where the naming came from. Not 100% sure about that, but I remember reading about it. And then I was working on Go for a long period of time, and Go has this built-in map structure, and it's a hash table. A very highly optimized hash table, because the Go team is very competent, the Go runtime team. And various folks had taken an attempt at putting together a Swiss table implementation for Go. And I tested some of them, and I was like, this is fascinating, what Swiss tables do. And I'll explain how it works in just a second. But I looked at it, it's like, well, it's really hard to beat the performance of the runtime. The runtime was really good. And I noodled on this for a little while, and eventually I ended up having to take this business trip to India, to Bangalore, and so I was on a long flight.
Yeah. It always starts like this.
And I'm just going to try to pull on this. And I pulled on it sufficiently that I could get some of the benchmarks to be faster.
Wow.
And then that is like catnip for an engineer. Can I make it all faster, figure out all the rest of it? Got some help from the runtime folks. There's an issue on the Go issue tracker where other people have been attempting this because people proposed, "Hey, let's use Swiss tables." And the Go folks were like, "Well, we don't quite know all the details. You're going to have to navigate this and that." And there were some ideas there. Combined them together and got to the point where we had a complete implementation that was faster on most benchmarks, not quite all of them, but most of them. And then the Go folks eventually picked this up and pushed it over to the finish line.
And then you came with the idea, you got to the point where you were able to show an implementation that showed how some of the benchmarks were faster. And then you started to work with some folks on the Go team, too.
Well, it wasn't quite that. I came up with an implementation that we ended up using at my company. It was good for our use case, but actually putting it into the runtime is a whole other level.
But then you just showed, here's this implementation, and then they took the inspiration and the ideas.
Yeah, they're like, this is great. They always are looking for ways to make the runtime faster, and there's—
Oh, wait. So you did this contribution a lot after you left Google, right? So this was from the outside.
This is from the outside.
That's awesome.
Yeah. I know that people contribute stuff from the outside as well. We had another colleague, he contributed one of the CRC implementations, adapting some stuff that—
CRC?
Cyclic redundancy checksum. Intel published some papers about, here's how to do the CRC in assembly very fast, and he contributed one of the implementations. You see a number of those things where people are contributing externally. It's not a lot, honestly. I actually don't know the full details, but people are regularly contributing to these things.
Peter just described how the Swiss table work came together using an issue tracker in the Go tracker with
Different people contributing to the work and then the Go team pushing all of this over the finish line. This is where I need to mention our season sponsor, Linear, which is a place to coordinate work between humans as well as agents.
One thing I've noticed about how most of us work with agents is how it's a pretty single-player thing. You open a terminal UI, go back and forth with an agent, and it usually produces a PR, but the rest of your team has no idea what happened in that chat unless you tell them or copy the whole history. And when everyone in the team works like this, a lot of work happens that's invisible to the rest of the team.
Linear's take is that agent work should be teamwork. Even today, teams already use Linear to define the work to be done. Now, Linear can already delegate an issue to a coding agent. This agent could be an AI agent that Linear integrates with, like Codex or Cursor, or Linear's own agent, or a custom agent. Either way, the engineer delegating stays responsible for the outcome.
What I really like about how Linear works is how the work stays visible. Your teammates can follow the session of the agent, check out the PR it produces, and join the review. We've gone from single-player work to multiplayer engineering work with agents.
And one more thing I like about Linear: a focus on costs. Linear agents' auto-routing chooses a model that is best suited for the task. Teams can also inspect usage and set limits, track usage, so you can use capable agents without having costs balloon out of control. Hop on board at linear.app/pragmatic.
Peter also previously mentioned Gaia, Google's internal system that maps logins to users. It's not surprising that Google custom-built all systems, including this one, but most of us won't build our internal Gaia. This brings us to our season sponsor, WorkOS. You can think of WorkOS as something like Gaia for the rest of us. Identity infrastructure you'd otherwise spend quarters building yourself.
WorkOS includes single sign-on, SCIM directory sync, audit logs, role-based access control. Basically everything a big customer's security team asks for, delivered as a handful of clean APIs. It's how companies go from "we have a login" to "we can sell to a Fortune 500" without standing up their own internal identity platform.
And WorkOS is already building for the next version of the agentic authorization problem. Their newest product is Airlock, the authorization layer for AI agents. Think about what happens when you hand an agent a task like "clean up the sales opportunities in our pipeline." The last thing you want is for this thing to have standing permissions to delete whatever it likes.
Airlock sits between your agent and the tools they call. It evaluates every request against the agent's intent and your rules, and then it allows it, denies it, or routes it to a human for approval. You write down the policies in plain language. The agent never sees your credentials, and every call and verdict gets logged. It works with coding agents like Claude Code and Codex and with MCP gateways. So if you're working out how to let agents do real work in production without overpermissioning them, take a look at WorkOS Airlock at workos.com/airlock.
And with this, let's get back to Peter and why he left Google after building Colossus. You were at Google, you're building Colossus distributed file systems. You were at this point probably working on probably the largest system on the planet, honestly. Why did you even consider leaving?
After Colossus, I kind of dabbled in this other project called Google Goggles for a little while. Remember the glassholes?
It keeps coming back, the idea, by the way.
Yeah. Now it's still here and present.
It seems like Google's early.
Yeah. And I think that was a technology before its time. I don't think it was ready to do at that point. And it looks like actually doing the glasses is quite a bit harder. The Android phones we were trying to power it on were not powerful enough. And then I just kind of got wanderlust, like, am I just kind of stagnating here at Google? Which is a strange thing to say, but some other people feel as well. And decided to go off and try my hand at another startup that didn't work out. We got acquihired by Square.
I want to pause for a second. This company, what was the company name?
The company that we founded is called Viewfinder.
Viewfinder. Yeah. It was in the mobile photo-sharing space, which should sound familiar. This is like Instagram. This was in 2012, right? As Instagram and all of them were taking off.
Yeah. We were right there in the play, and we just didn't have the right go-to-market, kind of like how to attract the users, how to get viral growth kind of thing.
Because from the outside, what I read when I checked the story, just like, oh, you co-founded a startup, it got acquired by Square. Hooray. Sounds like you had bigger ambitions, and this was a decent outcome but not the dream, right?
Yeah. It wasn't the dream at all. The term I used was acquihired. Sometimes a company will get acquired, get bought for their IP, for their product, for their business, and other times they get bought just for the talent, the people. And we got bought just for the talent. They acquired the IP, but I don't think Square ever did anything with it. It wasn't kind of like where they were working, but we had built up a strong technical team, and that's what we were hired for.
I can't remember the details. We'd raised a small amount of money. We were able to pay our investors back, make them whole. Maybe they got a little bit of a haircut, but maybe they actually got a little bit of a... But it was okay. Essentially, they got their money back, which is like...
As a founder, investors are big boys. They're used to losing their money, but you kind of feel bad if you lose a lot of money for them. So getting them paid back kind of makes you feel a little bit better.
Yeah. And then you spent a little time at the company that acquired you, which was Square. And then you started itching a little bit again.
Yeah. Because we've been working on these distributed file systems and storage systems. Colossus, one of the sister projects to Colossus is Spanner.
And how is Spanner different to Colossus?
Well, Spanner is essentially a distributed database. Colossus is a distributed storage system, and the way I think about the difference between a distributed storage system and a distributed database, you might think, oh, they're both storing data.
I was about to ask, because a distributed database will at some point be a storage system.
Right.
So what's the difference?
For Colossus, it was targeting large files, large append-only files. You can't update them.
Append-only. Yeah.
Yeah. Large append-only files. So 64 megabytes, maybe up to gigabytes in size. But if you're a database, you want to be storing small... If you're using SQL or relational data, you might have a table with billions or billions of rows. Those rows are broken up into columns. The columns are typed. It just has a very different nature to the engineering challenge for a database than it does for a distributed storage system.
And usually distributed databases are implemented on top of some kind of distributed storage system, and that was the relationship. So Spanner was implemented on top of Colossus, and some of the design decisions in Spanner kind of directly fell out of the append-only nature of the files in Colossus. You can't update a file in place, so you have to make the files immutable in your database.
And this is where log-structured merge trees come into play. They weren't invented at Google, but Google really popularized them with LevelDB, which emerged out of the work on Bigtable and Spanner. That got popularized into RocksDB. I subsequently reimplemented one of these things, and this is what we use at Cockroach Labs. It's called Pebble. So I'm very familiar with the internals of that, but it's all based on this idea that the data is stored in immutable files.
So how did you decide to found Cockroach Labs?
While we were working at Square, my co-founder and I—it was actually three of us—we were all at Square, and one of them is Spencer I mentioned earlier. He was working on the... He's my college roommate, and he was also at Google.
He was also at Google. This other guy, Ben Darnell, who was also at Google, he joined us at Viewfinder, ended up at Square. We were kind of just noodling on a project to do, and we'd actually had this design back in Viewfinder. We looked around for a database to be using in order to build Viewfinder on top of. We didn't really like the things that were out there. The technologies inside Google looked better. We had Bigtable. We had Spanner and whatnot.
We were looking around, and HBase existed, but I wasn't quite happy, and there's some other systems like Riak and others. At one point we just came up with a design for CockroachDB, an initial design. Then I was like, no, no, guys, we're doing a mobile photo-sharing site. We shouldn't build a distributed database. So we put it on the back burner, which I think was absolutely the right thing. Or maybe we should have just pivoted away from doing the mobile photo-sharing site, given the way things worked out.
And then we got to Square, and we saw some of the same problems that they were experiencing with data storage systems. And Spencer is very convincing, managed to convince some of the management that, hey, can we just work on this part-time to see if it had life behind the design? And then kind of conscripted Ben and I into it, and eventually started getting attention externally. And we're like, hey, can we go and spin this off into a company? And that's what ended up happening.
And so you started a company, but I understand you didn't raise VC funding initially, right?
That was the case at Viewfinder. We did it differently at Viewfinder. We kind of eschewed the VC money, and in hindsight I wouldn't recommend that.
You wouldn't recommend?
Yeah, I wouldn't recommend not taking the VC money, because in my experience the VCs are very, very intelligent, and they can help you navigate a lot of challenges. I think sometimes there's this perception the VCs will push you into various areas, and maybe there's some bad ones out there that do. The VCs I've had experience with are just some of the sharpest people I've ever met.
So you kind of have an extra person helping you on the team, pretty much.
Mentoring you, giving you guidance, telling you what they're seeing. They give you advice, seeing what they're seeing in the market, where things are going. That is very hard sometimes for a founder.
Especially as a technical founder, right? You're focused on the engineering part.
Yeah. We actually took money right away for Cockroach Labs. It was almost like as soon as we left, we went out and did a little roadshow in the Bay Area and got some interest and got an investor right away.
I have to ask about the name, though.
Yeah.
How did the name of Cockroach...?
Yeah. So we named the... Yes, that was mine. Pulp Fiction came out in college, and like, oh, what should we name this thing? The new image manipulation program. I think we were thinking image manipulation program initially, and I'm like, oh, it's obvious. It just stuck.
At some point, we're noodling on this new database, and you kind of want to give things a name. You can't just say, oh, we're working on this distributed database. You kind of need something. And Spencer's like, oh, CockroachDB. Cockroaches are unkillable. I want this database to be unkillable. Cockroaches are going to survive the nuclear apocalypse. So that was where the genesis was, and it just stuck.
Yeah. Whenever a bunch of our nodes goes down, this thing will still be up.
Yeah. And this is where we're at today with CockroachDB. It's one of the things that I point out. I'm like, holy crap, this is awesome. You kill a node. We did this whole campaign last year, which was really just to prove out something that had already been present. The campaign was performance under adversity. You can run a workload against it, you can kill a node, you can sometimes kill a whole region, and the system keeps on going.
And it's stories like that, what we did on the marketing side there, but also we hear this from our customers too. They've had fires in data centers, and all the other data systems go down, and CockroachDB keeps on going. I'm like, that's awesome.
When you started out, who were companies, startups, that wanted to use CockroachDB, and how has it changed since? Because just making the case, okay, I'm starting a startup, it's a small startup. I will need a database, and I'll typically choose Postgres, right? It's free, everyone's using it. I'm running it on a node. At what point did you see that typically tech companies are like, okay, this is not enough for me, that it's running on a node, either because it can go down or because I'm outgrowing it? What was it they were outgrowing? I'm trying to get a sense of at what point did companies tell themselves, we need something distributed in a database?
Yeah. Oftentimes we have companies calling us up after they've had a disaster.
So a node went down or a hard drive failed, that kind of stuff.
It's not quite like we're ambulance chasers, but if you see an outage, a big outage from some company, we'll sometimes be trying to look them up, but also they will call us. There is a very big bank who's now a customer. I don't think I can name them, but you can go read. They had a very serious outage due to a weather event, and after that...
Which probably knocked down, I'm assuming, a region or a database or a networking cable, or a tree fell on something.
I think it knocked down a whole region. It was a region-wide power outage. You knocked down the region, and there's a mandate from the CEO. It's like, no, we just have to be able to survive these things. And that gets pushed down all the way.
And you see this in other places where one of our early customers, they were running on AWS, and they just got to the maximum size you can run in an Aurora instance. And then what would typically happen at that point is you have to shard your database. This is very standard practice. You take your single-node database, you create 10 or 20 or 100 shards, and this is what Google had done for some period of time.
And that's a heavy burden on the application developer. The way we always phrase this is, the application developer is becoming a database developer at that point, and they're doing it poorly. They're trying to implement distributed transactions or indexes and whatnot. We felt the burden for that belongs on the database developer.
Can we talk about automatic sharding? I think it's safe to assume most of us will know what sharding is when you're... But actually let's start from manual sharding and then how you can implement automatic sharding, and if you can tell us tactics that a database like CockroachDB can do to actually just take that
Load off of you. Yeah. So I think the very basic form of sharding is a little bit like the hash table. Let's say you have a fixed number of shards, like let's say it's just 100 shards. Your data model is a user with a lot of data associated with the user. You just take the user and you map them to one of the shards, and you kind of just rely on the hash function to get fairly even distribution. The problem with this is, at some point, one of your shards will get full and you have to reshard, and that's a very, very onerous process.
Resharding, I guess a simple way to do it is if it's just a hard drive, I don't know, per node where you write the user data, it gets full and you're like, okay, well, I now need to split it somehow. I need to move it. I need to remap it. I need to reject my metadata which knows where this data lives. That kind of stuff.
Yeah. And it depends on exactly how you're doing that mapping from the user ID or whatever your shard key is to the shard. You might have to remap them all, right? This is very typical. Exactly. And this happens in hash tables where oftentimes, in order to grow the hash table, you just have to essentially create a new hash table, double the size and copy all the data over. Now, that's the very basic, straightforward way. And there's various levels of complexity you add on it. One of them is called consistent hashing. And there's various techniques to do this. It's kind of fascinating how they all work, but in consistent hashing, you can add an additional node and then it only moves a fraction of the data from each shard over there. There's various systems to do that. I believe this is like in Cassandra.
The way CockroachDB does it is more akin to Bigtable, more to Spanner, more to HBase, where instead of actually hashing, we actually take—you can imagine all your keys in a system, and this is always true in a system—that you can imagine just one big contiguous key space. And then you partition contiguous spans of that, and then you have to build up an index on top of those contiguous spans. And what I just described there actually sounds a lot like a B-tree. So there's this index on top that maps you from, like, I need to have this range, which node is it on? And this is a little bit like a B-tree. You kind of squint. I think you squint and everything's either a B-tree or it's a hash table, and that index structure.
It's now starting to make sense, because I remember when I read, it might have been the Wikipedia article on B-trees, it said this is a data structure that is frequently used in databases. It's now coming back to me because I didn't think too much of it. I'm not someone who builds databases. Now that we're talking, we just organically keep touching all the trees again and again.
And the other place that it comes up in databases—so this is where it comes up in distributed databases, and no one ever really calls it a B-tree. I just kind of squint sometimes and I see it's actually kind of a B-tree. But the other place it comes up in databases is for your indexes. So if you have a table and you have an index on your email address, and you want to be able to scan over those email addresses in order, that's a B-tree under the hood in a database. And any kind of index you have, it usually provides sorted order. There are hash indexes, but oftentimes it's the B-tree index, and they are ubiquitous in databases.
There's actually a paper called "The Ubiquitous B-Tree." And basically, I just identified that. I think that paper was written back in the '80s, and they're still ubiquitous today. They are the foundations of single-node databases, and pretty much every data system I worked on has had B-trees at some point, someplace in them.
I want to ask about strong consistency. So CockroachDB offers strong consistency. Now, for people who are a bit more newcomers to distributed systems, can we talk about the consistency models and then why strong consistency is important and why it's hard to implement it?
So there's multiple ways to approach this, but if you've used a database, you've probably heard of transactions. And transactions are a way to perform a whole bunch of mutations atomically. So databases talk about atomicity, consistency, isolation, durability. The durability is really easy. It's like when I write data to the database, it has to be durably written. So if anything crashes, it comes back.
The atomicity is just referring to the fact I want to do a whole bunch of changes. I want them all committed or all aborted at the same time. I don't want to have some kind of partial operation. And why is this important? Why is the atomicity important? Well, the atomicity is what gets you to the point where it's like, as an application, I can do a bunch of operations, and if an error occurs, it all gets rolled back, and it's a much simpler development model to work within.
And then there's the consistency and isolation, which kind of get muddled. The isolation is referring to isolation between transactions. I don't just want to run one transaction at a time. That's easy to do, right? I want to run a lot in parallel. Oh yeah. And make it so that when they're running in parallel, they are running as concurrently as possible. But you want to have the appearance, when they're running as concurrently as possible, that there is a serial order to them. So this is the whole trick. The gold standard for isolation is called linearizability. Don't worry about that. The step down from that is called serializability. And that literally refers to having a serial order of your transactions. But you're having to construct that in a way that you're doing everything as concurrently as possible.
And the benefit of this serializability is, again, it's a very simple model for the application program. They don't have to worry about weird defects occurring in their program. And some of the ones that can occur—the classic description is of a bank, right? Where I might want to have an operation that reads and says, do I have $100 in my bank account to transfer somewhere else? And you could arrange for lesser isolation levels that you might be able to subtract that $100 twice. And that's bad, right? We want to keep accurate track of your bank account or what's in your shopping cart. The use cases are endless there, and you do this wrong and you have very egregious bugs. But now going back to weak consistency and strong consistency.
Anything less than linearizability or serializability might be considered weak consistency. But there's also strong consistency and eventual consistency. So eventual consistency is like sometimes I can do an operation and I might not be able to see all the updates immediately.
The read results will not necessarily be the current update.
But they'll come back at some point. I've written part of it, and the rest of it will show up at some point. And oftentimes when you're doing—
The easiest thing is your credit card balance, right?
Yeah. And it's faster to do it that way. Faster in terms of just the performance you can get out of the system. But again, it's a little bit harder for the application to deal with. One of the places this often comes up in distributed databases or databases that have any sort of replication is that I could write to the primary replica and then I read from the secondary and it's not there yet. That's eventual consistency.
Yeah, that's eventual consistency. And you can often work around this, but it just puts a bigger burden on the application developer because you have to pay attention to that.
And then with CockroachDB, you have strong consistency, meaning as soon as you're writing it, when you're reading from the database, you already get the written value back.
You get the written value back. It's like you read whatever you wrote. It doesn't matter if you're reading from the same node you wrote it to. If you read it from another node, you still actually get the data you just wrote.
Is the trade-off logically not that you would have higher latency? Because clearly, to implement strong consistency, you would somehow need to, in the naive approach, write to all replicas, right? What are you doing inside Cockroach?
Well, we are writing to all the replicas, so—
Well, yeah. You're doing it fast.
We're just doing it fast. You're making that efficient. I think this is one of the things that also fascinates me about the software industry, is we keep on finding ways to be more and more sophisticated in order to do things that make it easier to write the applications, but do it in a very high-performance way. And we've gotten very, very good at this over the years. And this is the area I know about, the databases. This is happening everywhere. I'm just fascinated by how fast graphics have gotten, where when I entered the industry, you were literally writing out each individual pixel, and now you have these GPUs that are doing billions of triangles per second, whatever the current numbers are. And I'm just kind of astounded how much sophistication has gone into every area of computer science, wherever you look at it.
One more thing on CockroachDB. I want to ask about the Raft consensus. What is the Raft consensus?
Yeah, consensus protocols. The original consensus protocol is called Paxos, and it was famously hard to implement. Raft was—you might think of it as a variant of Paxos. It was kind of like an alternative to Paxos, but in my mind today—
And then the consensus algorithm being that you have a number of nodes, like three to a lot more, and then how do you get them to agree? What do you typically agree on?
Yeah. So you agree that the write occurred. So the way to think of consensus: you might think I want to replicate data, and I write it to a primary and I write it to a secondary. And you can't actually have consensus when you only have two replicas. And the reason you can't have consensus is if there's a crash and I come up, if I'm on the secondary, how do I know if something was written to the primary? If I'm on the primary, how do I know it was written on the secondary? Right? And you're always going to be in this confusing place where it's like you either have to roll back a little bit or you lose some data. And consensus requires at least three, but you can have consensus across more than three replicas.
And the idea with consensus is I'm going to write to three places. And normally, when you're doing a read, you don't read from multiple of them. But if there's a crash, I have to do recovery. Then I'm reading, oh, I can read from any two of the three, and I know I can determine what had happened previously. And it's usually just at recovery time that you're actually doing that consensus read. So reads are typically just happening from one replica. You have to write to all three, and on a crash, during that kind of failure, is when the consensus read occurs.
And inside CockroachDB, how many replicas do you choose for the consensus?
It's typically three. For some system tables, it can be five, and then customers also have control of this. At the database level, you can write to five, seven. Five is like, if you're really concerned about the data durability, you might use five. But there's a slowdown the more you're writing to. It's like the more storage space that—
And obviously, we're talking of slowdown with nodes, but of course, if they're between regions, now you have a lot more resilience for, let's say, an earthquake or a power or whatever. But now you will have additional latency. Just speed-of-light basics, right?
Speed-of-light latency, right? And it's tens or up to hundreds of milliseconds, or even higher if you're going across the globe. So you have to be very careful with that in terms of how you architect your queries. One of the things that is just a general truism of distributed databases is you don't want to have a lot of back and forth. You want to do all your reads in one parallel read set, get them back, then do your writes, right? But if you're doing serial operations where I read a row, I write a row, I read a row, I write a row, the latencies just add up.
We talked about founding CockroachDB, but how is the company growing and where are you today?
Yeah. We're powering mission-critical applications. That's our bread and butter.
But by the way, can you elaborate on mission-critical? Because it's like—
Yeah.
You are in the industry where you know what this means, but from the outside, it can feel hard to put a thumb on what is mission-critical. Is my SaaS that is showing ads mission-critical? Probably not.
Yeah. So mission-critical, to my mind, is these—the other term of art is tier-zero applications—the ones that are just the core crown jewels of what's running a company, like a trading system. Your trading system can't go down. If the trading system goes down, this is a critical problem for the firm that is running the trading system. Banking systems, but also we work with DoorDash. Some people might think that delivering a burrito is a mission-critical system, and it certainly is for DoorDash, right? If that goes down, it's problematic. We power shopping carts, other stuff like this where it's like, well, if the shopping cart goes down, that company is losing hundreds of thousands, millions of dollars per hour. So that's the critical you think about.
Yeah. I guess of course they're losing, but their customers are also used to this just working, like running water. And then when it's not, it's the same thing as when your utility breaks, right? Your water, electricity is out. You'll survive, but it's not what you expected.
Yeah. And everybody's like, what age are we living in that the electricity goes out? We kind of had this ingrained into our heads at Google. It's like Gmail cannot go down. People are relying on it. Search cannot go down. If it goes down too long, people are going to move to other systems. And in some ways, search isn't as mission-critical, except, oh wait, every single search has ad dollars behind it. And you actually notice the blip in the revenue. And it's not just the blip in the revenue, it's the blip in reputation as well. I think that's the one that really poisons companies. If your bank is down for a serious amount of time, the reputational damage there will be horrifically bad. And we often talk about, oh, and then Grandma won't be able to pay her rent and she'll get evicted. You've got to take this responsibility really, really seriously.
No, but also just Gmail being mission-critical, just on the way here. We only exchanged numbers later, but we were communicating over email. I was telling you, you were telling me that you're here. I emailed. And I never for a second thought that it could go down. And I think we were responding within 30 seconds, right? And I just know it's
It's there. I didn't bother setting up a secondary communication line.
Yeah. When you have that kind of level of trust with your users, you got to maintain it and invest in it. But then it leads to this kind of freedom for the user as well, where you just don't think about it: I don't have to worry about it. It's just going to work.
And then in terms of the company, how many engineers do you have roughly?
We have some hundreds. I don't actually know the precise engineering number, 110, but there might be 150 in R&D overall. There's other folks besides just engineers. I'm clear, engineering managers, those are engineers as well. And we've been growing steadily. It takes quite a while to build a distributed database.
Not for the faint of heart.
So it took a couple years a couple times.
Yeah. Well, I did a distributed storage system. Now I did a distributed database. It's not for the faint of heart, right? So there's a lot of work getting it to a level of stability, then a level of quality beyond that level of stability. Getting all the bugs out, and then continuing to innovate and put more performance into the system, adding functionality to integrate better within enterprises.
Our revenue has been steadily growing over the years, and it's at this place now where we see a path to future success as well.
And I want to ask about your coding habits. So when you co-founded the company, how much code did you write for the first few years?
I wrote a lot. So I've always been a very prolific coder. I wrote a lot of code early days. We were all technical co-founders, Ben, Spencer and I, and we were all writing a lot of code, and I was no exception. But I look back at my GitHub output, and it's like peak years, maybe 100,000 lines of code in a year, which is a lot.
We're talking pre-AI.
Pre-AI, right. This is when you're back doing this manually, right? At some point, we started out using a system called RocksDB, which is an LSM. At some point, I think it's back in 2019, we ran into limitations with it. I decided we wanted to rewrite it and did a big push to rewrite, and that might have been 40,000, 50,000 lines of code. And then a bunch of other people came up and helped. You look at that output, I'm just like, oh my goodness, that was a lot to keep in your head. It's a lot just to type, 100,000 lines of code.
The average that the industry talks about is 3,000 lines of code from an engineer in a month. And so if you multiply that out, maybe 36,000 in a year, that's good, right? So I was doing a lot. I look at that, it's like there's a max that you can hold in your head at a time. The tools have gotten a lot better since I first entered the industry. We've gotten better debugging techniques, better testing techniques, but still quite significant.
And you were CTO from the beginning, co-founder and CTO, but there was a time sometime around 2022 when you decided to be a bit more hands-off, right?
Yeah.
Can you tell me about that?
The general rule of thumb for engineering leaders is, well, you got to have your team, have to manage your team. And we had a VP of engineering. But I was getting to the point of, okay, are my coding days done? Just direct from a higher level. And I got this advice for a long period of time, and I pushed back on it, but I acquiesced at some point. And I think it was the right advice. I'm not saying that the advice was wrong at the time.
But there was a time period from about 2022 to 2024 where my output declined. I think I actually did the Swiss tables thing in that time period, but I wasn't doing much on the core—
The business.
Yeah, the core business. I would get in there and do some work, but it's really hard, if you're in meetings all day, to also do coding. I think this is the fundamental tension, if you like.
So you took on the meeting burden, the coordination burden, the stuff that was, if I'm reading correctly, before you spent a lot of your head in the code, and now you're spending a lot of your head above the code, the business, the engineering, or whatever, customers, that kind of stuff.
The customers and just being an executive as well. Yeah. So very hard to wear all those hats simultaneously. And then I got back into it because AI started to emerge.
So when did you start using AI? But when did you start to find it useful in terms of coding?
Yeah. Well, it's interesting because those initial versions of glorified autocomplete came out.
We're talking about GitHub Copilot, Cursor.
Copilot. That was the one I had the first exposure to. We dabbled with Cursor at the time, but they're all glorified autocomplete, and it was crazy that you could just start typing something, and the rest of the function—you look at it, I'm like, wait, it got that right. This is crazy, right?
And we're trying to encourage our engineers to use this, and at some point, I can't remember if this is my idea or my co-founders' or someone, basically, in order to really guide people about how to use it, you have to be a user yourself. I think this is true of engineering management in general. You want to manage engineers, you have to know how to be an engineer. If you don't know how to be a good engineer, it's really hard to manage other engineers.
I feel like you'll have a hard time just relating to them, at the very least.
Exactly. So I took it on. It was clear very early on, this is probably going to go somewhere, but it wasn't quite clear how far, how fast it would go. And you start dabbling in this, and I was like, oh, okay, well, it's not quite good enough. It's not quite good enough, but let me start getting back into the coding.
Very rapidly, you start seeing the signs of life, the Opus models coming out. It was Sonnet first, then Opus, and you're looking at these and you're like, oh, wow. Okay. Well, they seem to be able to do quite a lot, but the code is still not great. But then it was just this cadence of continuing improvements. And I was starting to do a lot with Sonnet and then starting to use Opus. And I had that same moment everybody else did. And this was last year, last November.
November, December, winter break, right?
Yeah. It was Thanksgiving. I distinctly remember it because—
You were not chilling, you were coding, weren't you? You were agenting.
I was agenting, just like everyone else. I had this thing I'd wanted to do for a long period of time on CockroachDB. I wanted to test all these configurations of CockroachDB across different vertical scaling: how many CPUs you have on a node, how many stores, disks you have on a node, how many nodes you have in the system, and test across all this huge matrix of it. This is one of these things I couldn't ever quite get prioritized appropriately because it never seemed critical, but I always had this intuition there was something there.
And then over this four-day span, the code just materialized as I was using, I think that was Opus 4.7, or is it 4.5? Whatever the number was. The recollection I had is, I'm a pretty fast typer, and then I just remember having this feeling of, well, the code is just materializing before my eyes. You would ask for these things, and I got out of the habit of actually typing, and you just ask for something, it materializes.
If you've ever seen some of those movies where they have this archetype of a software engineer gets in front of the keyboard and they start typing, you show the screen, it's going way beyond human speed. And it was beyond that, right? I think as software engineers, we used to laugh at these.
I still remember Swordfish, the thing when they're visualizing the things are going, or the code appearing. You're seeing that the person's typing, and then it's a big line, and as software engineers, we're laughing because it's not how it is. But it's crazy, that effect, right?
Yeah, but now it's even better than that effect, right?
Because it actually works this time.
And that's slow in comparison. It materializes faster than that. We've been talking about B-trees a whole bunch. I implemented another B-tree in the past month.
Of course you did.
Yeah. And it took about 30 minutes to implement probably 10,000 lines of highly optimized Rust. It boggles the mind.
You should look up later. 10,000 lines is a crazy amount. You physically cannot type that fast. You started to get back to coding. Are we talking about vibe coding, prototyping, or are we actually talking you started to contribute proper production-ready code that is up to the level of what you're doing at DB?
Well, it started with this tool that was this benchmarking tool that tested this matrix. I'm the CTO. We have an office of the CTO. The office of the CTO's mandate is to innovate, and I was looking for places we can have innovation. And one of the things that we wanted to innovate in was better autoscaling of a CockroachDB cluster.
And in the January time frame, I came out with what I would think is a research breakthrough, you might say. And it came about because I was dabbling in this area and just working with models, trying to understand it. And they're not just good at coding, they're also good at helping you explore design ideas. And I think this is the fascinating thing where you have to get out of the mindset of, I know exactly what I'm going to build, but more like, hey, we have this problem. Talk through it. Be a partner with me.
Be a sparring partner.
Be a sparring partner. There's this advice you might have heard that if you're stuck on a problem, you just go yellow duck it.
Rubber duck.
Rubber duck it. I think I've heard it as yellow duck. Just go talk to something. You don't even need it to respond. Now you can talk to this system, this intelligence, and it will give you back stuff. And it's not always right. This is the thing. Even today, these models are fantastic. Fable's fantastic. Astra is fantastic. And they're not always right, but they know so much. It's just encyclopedic, and you can explore ideas super, super fast and then be pointing out, well, that doesn't sound right to me. And it'll be like, oh yeah, you're absolutely right. I hate that sycophancy. It kills me.
Or, you're right to push back on that. That's absolutely right.
Yeah. And yet, just able to make such fast progress, and I find it absolutely incredible. So it quickly moved from just doing side things, building up some tools and whatnot, to building essentially, over the last eight months since January, we've been building towards a new launch. And a lot of that has been produced by agentically powered engineers.
And the entire company's got on board with this now, where I think it's probably 100% of engineers are using it to a greater or lesser degree, and a lot of code is being produced. I think it's very high-quality code. These models don't test things adequately. They don't look quite intently enough at performance. But you can guide them in the right way.
And I think there's actually a huge advantage to anybody who's done management before. It's a little bit like being a manager of people, where you're a manager of a large group. You're not looking at every line of code, but you're definitely helping architect the system. And I think there's a very strong analogy there.
I sometimes push back on this analogy. The reason being, I was an engineering manager. My take is that working with these agents, it's not like management, because management has so much of the human stuff. As a manager, when I think of all this, a bunch of stuff I dealt with was the people side of things, the conflicts between people, the performance reviews, the meetings, et cetera. And you have none of that.
You do have the orchestration, and this is almost, I guess, a really naive way of management where they don't push back. They start to do it. Sometimes they're unreliable. But I almost like to use orchestration a bit more because I feel management is so much more involved. I think this is the thing where a tech lead who has no management responsibilities, but they have a group of interns, but they don't need to deal with their performance, with their anything. It's a lot closer to that, if you know what I mean.
Yeah, I do agree. I use the engineering manager shorthand, but it's really about being a tech lead for a 30- or 40-person organization, or being an architect for one. I think the term architect gives me a little bit of a distaste, but it's being an architect who's also on the ground.
A hands-on architect.
Hands-on architect. And you don't have any of the management stuff, which is a blessing and curse, but it's remarkable that you can spin these things up. If they make a mistake, you can keep on correcting them until they get it right. We've always been able to do that on the human side, and I can do it faster.
I think the ultimate result for me is you just have to be more ambitious about everything you do. You can produce more, higher performance, higher quality, more secure. So our ambitions have to raise up.
You also mentioned that with AI, everyone's using it and you're building stuff, but also with CockroachDB, you're now building something that is also related to AI. Can you talk about that?
Yeah. We're building multiple things. AI is the future.
Well, it's here to stay. I think that's—
Here to stay. Yeah. One of our theses, which is not crazy, every application in the future is going to be written by AI. I think there will be some bespoke software, handcrafted software. I think we will see that continue to exist, just as people write assembly still, but it's going to be diminishing in size. So asymptotically approaching 100% of software will be written by AI.
Generated, right?
Yeah. It's hard to say when it's going to end, where the humans are the agent, supplying the agency and the vision behind it. I think that might exist for many, many years, but I think the code will fundamentally be written by the AI. And I think we're going to just see this explosion of applications.
And we're seeing that inside Cockroach Labs. We're seeing it elsewhere. Earlier this year, we rolled out this internal platform where non-engineers could write many applications. You're hearing this at other companies. We did the same thing. And over the course of just a couple months, 500 applications, a thousand applications.
By non-engineers?
By non-engineers, primarily by non-engineers. And I thought it was awesome. Our HR team is building these little applications. This is the thing I've always dreamed about doing, and they were never
Serviced. Your CFO is all over the place, and ours is no exception, producing dashboards that they could never create before. And I think it's very empowering. I'm married. I have a wife. She needs software. She cannot produce that software on her own. And I have never actually helped her produce the software, which is my own failing. But I think there's a world in the future where everyone's getting custom software built for them. And you're just going to also see greater and greater applications and higher-quality systems being produced by every company as well.
Today, what is your stack? What do you work with in terms of harness, model? How do you run agents? Is it one agent? Is it multiple? What kind of terminal do you use?
It's evolved over time. When I first started dabbling in AI stuff again, it was GitHub Copilot. I was an Emacs user for 20-some years. I got convinced to move to VS Code, but that's all gone now. At some point I moved to using Claude Code. That was a thing that, for whatever reason, I just got started using. It was Claude Code in the terminal. Of late, I use a mixture. I'll just describe my current setup. I use the Claude desktop app, Claude Code via the Claude desktop app. It's fantastic. Good job, Anthropic. I also sometimes use the Codex desktop app just so I have an alternative model to turn to for some very critical things we're working on.
I will get one of those models, sometimes the best model. Oftentimes I'm using the best model, like Fable. I've recently started using that. Sometimes it's Astra, sometimes it's the other ones. But you have one produce a design, you have the other one being like, "Hey, my colleague produced this. Can you kind of tear it up, adversarially review it?" And it's not always perfect, right? But I think there is utility, especially for something that's super, super critical.
We're pushing towards the launch really soon and rushing towards the finish line. I'm telling folks on the team, especially our very senior folks, you should use the best model right now. It's worthwhile to do that. I'm generally just using the best model. And part of the reason is I don't actually know that I'm getting a lot more intelligence from it, but I don't want to have the cognitive overhead of deciding on a case-by-case basis.
Should I use Sonnet? Should I use Opus? Should I use Fable? Should I use Sol or Astra? And I think maybe if you're like, "I really need speed,"
I would make that decision. But oftentimes I'm doing things in parallel. So you're asking how many agents I'm spinning up. Well, it depends. Oftentimes there's a certain number of sessions you might be using. I find cognitive overhead is about five to ten sessions concurrently, but sometimes those sessions will have many subagents doing things.
Yeah. And they'll be long-running.
Yeah. It'll depend on what I'm doing. As I was coming here, I was on the train. I spun up something to do a little bit of research, and that was probably using three subagents. Other times it might be 20 subagents. Anthropic has dynamic workflows. I can't remember what the Codex thing is called, but they have various ways of spinning up graphs of agents and prosecuting them. I think sometimes I'm probably using 100 subagents; others it's only five. It depends on what's happening, and sometimes I'm not doing any of that agentic stuff because I'm off thinking about something else.
Since you were an Emacs user for 20 years, using the terminal, right? Or, well, Emacs. But how come you moved to a graphical interface with agents? A lot of people still use terminal UIs.
I moved from Emacs to VS Code. A colleague was like, "Hey, these IDEs are really good. You need to try them out." I was like, "Emacs is an IDE." And then I moved, and some of the stuff was just more seamless. Emacs keeps on catching up. It always feels a little bit six months to a year behind, and every time they have a new version, my setup would break, and I just got tired of that. So I moved to VS Code.
But then I'd have many terminals open in VS Code. My setup would always be terminals in VS Code tabs. The terminals and tabs, I'd have eight of them open, and then they started taking over. It used to be just Claude Code in the terminals.
Yeah. And then at some point recently, six weeks ago, a colleague was like, "Well, have you tried out the Claude desktop app recently? It's really good." I was like, "Yeah, no, no, I'm very, very happy." And I made the switch, and it was a little bit awkward at first, and then suddenly I'm like, "Holy crap, this is awesome."
You can manage the agents a bit better, easier. You have the sessions there. The sessions are down the side, and they're kind of like tabs in some regard. But the whole integration that Anthropic's been doing, and OpenAI is doing the same thing now, by the way. I'm not dissing Cursor and Factory and Cognition. They're all pushing the same thing. It's incredible how fast these systems are innovating right now and evolving.
What do you think good software engineering looks like today compared to four years ago, before we had AI? Has it changed, or has that not really changed?
I think the ambition has to increase. I think the quality has to be higher, security has to be higher.
I'm glad you mentioned quality. Can we talk about that? Because I'm seeing across the industry quality declines, which you cannot fully put your finger on AI, but oftentimes it is people pushing out more and more and just not paying attention to small regressions here and there. Again, you're building a database. Have you noticed any, or have you gotten feedback of any quality regressions? Or if not, how come? Because this is just basic. We talk about the law of physics. This is an observed thing that when you start to have more output, you increase your deployment frequency, you often—not always, but you often—have more regressions and more bugs.
If you produce a certain number of lines of code, you're probably going to have a certain number of defects per line of code. And now you can produce more lines of code, so you probably would have more defects, right? But the thing that pushes against that is that you can be telling these agents—you have to give them quite a firm hand at this. This is a thing that I hope that the model providers are listening to, but you need to give them quite a hand. They kind of get a little bit lazy on the testing side, and you have to make sure the tests are comprehensive, but also they're using all the testing techniques.
And there's a lot of testing techniques out there. We have a lot of knowledge about how to do testing well. And the agents are lazy; humans are a little bit lazy. So getting the humans to actually be very disciplined about their testing is also challenging, and I think it's actually easier with the agents. You can instill that. You can get them set up. You have to give them the guidance: use property-based testing, use metamorphic testing, use these advanced testing techniques, deterministic simulation testing. There's technique after technique you can use.
And humans have always—I'm lazy myself. I'm pretty good about being disciplined about testing. At some point you're just like, "Okay, that was enough. We've got to ship." And now you can get a little bit stricter and firmer. I think the same thing applies over on the security side, the performance side. We've always had, at Cockroach and in the industry, known security coding practices.
Yep.
You can literally have every single line of code on every commit reviewed by a security expert.
Doing an adversarial security review by an agent or multiple agents.
Or multiple agents, right. And we're going through a tough time in the industry right now with hacks and security leaks and whatnot. There's only a limited number of bugs that can be in software. I think we'll get it added on the security side and on the quality side.
And then also, when I say quality, it's not just bugs, but it's also little things like, "Oh, that UX element is wrong." And there's no excuse for that right now. Fixing it is so, so easy. And what we're also starting to see evolve is designers using Figma. Well, I think that should be a thing of the past. Right now, our designer just deals with HTML and CSS and JavaScript directly and sometimes is even just producing pull requests, PRs, which she loves. We all love it. Everybody's happy with this. There's not any of this waterfall.
So she's producing pull requests for the production codebase.
Yeah. And this is on the UX side, not on the core database.
Of course. But this is the area that she owns, right?
Yeah. She loves it. Everybody loves it. There's no downside.
This is not new.
Yeah, for sure.
Yeah.
What about code review? What's your take on code review? I think it's starting to get a bit controversial. Is it going to stay or not? Because it's been a practice that's been around. You worked at Google. Google has been so big on code review. As I understand, there are two code reviews, right? The domain expert reviews it, and then there's a language expert who reviews it. So you went through that. How did you think of it, and how are you thinking of it now?
I used to be part of the—I was one of the early members of the C++ coding—
Review, as I'm asking, right? Okay, you will have strong opinions. Tell me.
Yeah. I think code review was fairly essential before the age of AI, but now we're getting to the point where I'm reviewing and looking at code. I look at most of the code that the agents are producing still, and yet you're not giving it the same level of scrutiny, right? And this has always been the case. If you get a pull request from a junior engineer, you have to give it more scrutiny than if you get it from your most senior engineer. And you ask any tech lead, any engineering manager, any software—they'll say, "Yeah, the junior engineer or someone new to the codebase, you just have to give more scrutiny."
And what I'm finding, as the agents are getting better, you're having to give less and less scrutiny. You do have to give scrutiny to some things. Like I said, the testing will be incomplete. Maybe they didn't follow the security stuff. Maybe there was a performance regression. And I think it's assuring there's constraints on the system so it's hard for them to do the wrong thing. I suspect that, I don't know if it's going to be this year or next, we're materially going to stop looking at the code in the same way that we don't look at assembly anymore. You will—
We trust the compiler to produce pretty good assembly.
We trust the compiler to produce good assembly, and most—
Unless you're one of those people where it's really important, and you're maybe a game developer or someone, and you look at it. But there's fewer and fewer of those folks.
Yeah. And even then, I think what we'll be migrating to is: why does the human have to look at the assembly? The AI should look at the assembly. I'm doing something right now where I want to have a zero-overhead abstraction for doing something that is done at test time, and when it's in production, it's compiled out. Fable set up for me a little system where it actually looks at the decompiled code to verify that there's only a few extra instructions put in place. I never would have put that in place, but now it has this guardrail that every time it changes this, it can verify that there's no regression. And I think you're going to see more and more of this where you put guardrails in place. I think the model kind of likes it because it can work within that guardrail.
Now, for 20-plus years, you were writing so much code. I'm sure you were in the zone. Do you remember being in the zone and just turning it out? You wrote a lot of production-ready code. Now that you're coding with AI, do you get into the zone?
Yeah, absolutely.
And how is that zone? Is it the same? Is it different?
It definitely feels a little bit different. It's maybe a little bit less intense, but you're managing more things cognitively.
Can you describe to me what it is right now when you're in the zone?
Yeah. I'm thinking of ideas that normally would have taken me a week to experiment with, and I think of multiple of these experiments, and then I fire them off all simultaneously. And then I'm reviewing what else I should do while that's being completed. Sometimes I'm reviewing—they'll be giving me progress updates of, "Oh, hey, this is coming in. We're seeing this stuff." And I'm being like, "Well, that doesn't sound right. Hey, what about this? Or did we do this correctly?" Maybe I had a design, and it's not implementing the design quite perfectly.
But I have this little feeling that—I haven't been a college professor, but if I was a college professor and I had a whole swarm of research assistants, and they've all gone off doing things, and it's coming back, but it's coming back just really rapidly.
Really rapidly. You're not waiting months or weeks.
I'm not waiting months or weeks. And then I'm iterating. I'm like, "Oh, that one failed. That's fine." You just have to let go. And this is the nature of the software I build. I think there's other pieces where you just whip out a website. You can whip out something that doesn't have this level of scrutiny very quickly. But I talked about this, and I have a whole bunch of analogies about the—
Let's talk about analogies. What analogies do you have about using AI or AI?
The one I was advocating for—I had kind of two that I was advocating for late last year and this year—which is, AI is coming for us. It's here, right? You're producing software, you're walking down the road, and sometimes someone will pass you. They're running, they've got an efficient gait and whatnot, but everyone's under their own locomotion. And these coding agents came, and it was like a car pulled up next to you. Get into the car. You don't know how to drive, you don't understand the controls, but you've got to get in, start. You might forgot the gas pedal, and it takes off and crashes into a tree. But you have to learn how to drive.
I think using all these coding tools doesn't just happen naturally. It's learning how to drive. I think we might be in the era of the F1 driver right now, which is, the really good people can drive these systems a lot harder, a lot faster than the people who are just picking them up. If you've never used an agent coding tool, there's a vast difference between someone who's really expert in them, knows where they break, can pay attention to that, versus someone who's just picking it up for the first time.
One analogy I've heard is we used to talk about the 10x engineer. You remember, this used to be a debate for a very long time. Is it or is it not? But now what I'm hearing is the 100x engineer.
Yeah. And so you're saying that you are
Seeing some folks who, maybe let's not use the term 100x engineering, but this F1 driver who is just really, really good at it. How would you describe a person who you've seen? Is it just rock-solid engineering basics, and they lean into using these tools, or what are they like?
Yeah. It feels like it's directional that if they're good at software engineering before, I sometimes think it's like everyone's in this spectrum of capability of software engineering, and this is just taking that line and spreading it out. It's not quite true. I think it's helped some people more than others, but it feels like it's just stretched it out, so your ability before is now amplified.
I'm always interested to learn that, for example, Boris Cherny, Tibo at OpenAI, they both have been really, really good software engineers. Boris wrote one of the first TypeScript books, the first TypeScript book for O'Reilly. He built some massive systems. Same with Tibo, who built it, and now you're seeing these people are building all these tools and innovating. They were really good before as well.
Yeah. I have two other analogies to give you about what feels like an AI. You've undoubtedly heard about the term pair programming.
Yeah, pair programming.
The idea behind pair programming is it's good to have one keyboard, one module, and two engineers at it, one at the keyboard and the other one sitting beside them, looking over the shoulder and giving guidance. I think there's an aspect of that feeling where I actually got Chat to do a little image of this, where the android is at the computer typing and you're just there giving instructions. But that was maybe the way it felt a year ago. I think it feels a little bit different now.
The one I've just started recently saying is that I feel like the domain experts, the people who were really strong before, are now massively amplified. Have you seen all this mathematical stuff coming out, like the crazy proofs?
Well, I don't understand it, but I...
I don't understand them either.
Yeah. So I told you I wasn't very good at math. I stopped in the freshman year of college, right? But I watch along with these advancements. Terence Tao, he's probably the most famous living mathematician, super genius. He actually posted this session. There was this recent breakthrough, I think it's called the Jacobian conjecture. I don't even know what it meant, but he posted this ChatGPT session where he's interacting. I think it was ChatGPT, and you could see him interacting with this intelligence, and it was crazy because he's talking to it as a peer colleague, and it's responding. I'd encourage everyone to go look this up. It looks almost like a foreign language. His domain expertise is getting amplified by the system he's interacting with, and you can see how he's learning and exploring ideas just really rapidly.
Now, there's all this controversy about AI mathematics, but the analogy that comes to mind to me is the domain experts. They're a little bit like sorcerers in their particular domain. You have the earth sorcerers, the earth wizards, the water ones, whatnot. And if you know the magic incantations, the right words to say in the right order, you actually get something magical. And if you don't, you just get sparkle stuff that doesn't have anything there behind it.
You probably don't know much about distributed databases. If you're going to ask Fable or Astra, build me a distributed database like CockroachDB, you will get something out, but it'll be ultimately hollow inside. If you're an expert in databases and you ask to build a distributed database, and you can point out all the various things you have to know about distributed databases—here's what you have to worry about, the storage layer, the networking layer, here's the various data structures, runtime inside—you can actually get something quite magical very, very rapidly out of it.
So what would your advice be for engineers with mid-level to senior level who have been figuring out how to do coding, to become strong engineers in this age of AI?
Well, first off is you have the most amazing tutor readily at hand. One of the things that I would always do throughout my career, and now I've stopped doing it, but the reason why is going to become obvious, is I would always look at other people's code. I was at Google early on. You've probably heard of Jeff Dean. Jeff Dean was an amazing coder. His colleague Sanjay Ghemawat, also just an incredible coder. I would be looking at their pull requests. I'd be looking at their changes. It wasn't called pull requests at Google. There's a different name for it, but I'd be looking at...
CL, right?
Yeah. CL. This is P4. It's a different version control system. But I'd be looking at their changes, be like, how'd they do what they did, right? Be looking at their code. Oh my goodness, Sanjay's code is really always very elegant. Jeff is high performance. How's he doing that? How's he going about it? It's almost like you acquire via osmosis.
But now what you can do is not just acquire via osmosis, but you can literally—I would be encouraging, if you're a junior engineer and there's a senior engineer nearby, you could ask them how they're doing what they're doing. But you could just ask the AI to dissect what they've done and explain it to you, and explain to you at various different levels. How does this code work? What is it doing? Give me a diagram. Explain it to me like I'm five. Explain it to me like I'm 10. Explain it to me in French. Whatever you want. Fundamentally, to some degree, AI is a translation tool. They translate it from whatever language you're understanding it's in, and keep on interrogating it till it increases your understanding.
You were saying how domain experts are very much amplified. I guess one strategy as a software engineer is obviously become a great software engineer and use it as a tool. You can get a lot faster. You can get good at distributed systems. I'm not a distributed databases expert at all, but I used AI to explain a few things for me to understand up front, which was very helpful, and it would have taken me a lot longer beforehand. So I can use this.
But I wonder if there's another part of, as a software engineer, you can use it to become more of a domain expert wherever you're working. If it's a payments company, use it to learn about payments as well, so you can help the business, you can help your team, and honestly, you'll just learn more, right?
Yeah. I would encourage everyone: you have to have a little curiosity, right? Don't be bound in by the area you're working on. Explore outside of it. I was working on Gmail, but I was fascinated about how the main Google search engine worked. I was fascinated by how the internals of Bigtable worked, even though I wasn't directly working on Bigtable. Just explore and look at those things. And now it's so much easier because you have this super advanced, patient intelligence there to explain it. Why do you think it was done this way? Once you become an expert, you can be interrogating: I see it's done that way. What happens if we change this? Would this be helpful? That's where you go from just learning to actually contributing back.
I think everyone has to have the personal agency to do this. If you're just sitting there waiting for someone to educate you on how to do this, it's going to be really hard right now, because anyone who's coming in explaining how to use AI or explaining how to be a better software engineer, they're going to be out of date, right? You just got to get in there, be using the tools all the time yourself, and using it. You have to learn. I feel like I've learned more in the past, probably even year, than the previous five years combined, which is weird given...
Given your trajectory and given the environment you were working in, right?
Yeah. Everybody's been in the industry for a while. I'm definitely a better coder. I was a better coder 10 years ago than when I first got in the industry. I could look back every decade and realize I got a lot better, and I feel like I just got a lot better over this past year.
And this was also one of the reasons I was really excited to talk to you, because when we started to just exchange messages, the first thing you wrote to me when I asked you, hey, how are things going? You said, you wouldn't believe, but my coding output is insane, and it's high quality, and it's database-quality level. And those were things I don't really usually see. I usually see, okay, I'm now producing more code, but it's slop. But again, to me, this is a bit of an inspiration. You can use these tools to just amplify yourself as a software engineer. You are one example, right?
Yeah. Hopefully one of many.
Yeah. I'm not the only one in Cockroach Labs. We have other people doing this as well. I find it very exciting. It's a little bit exhausting right now, but it's very exciting. I got into software engineering because I like building stuff. You can build stuff faster. The stuff you might have had to compromise on in the past, you can take away some of those compromises. You see this in the UX of software coming out. I think the UX is a lot higher. You see all the fancy web animations and whatnot. That's only just the surface level. It just extends way, way below that.
Peter, this was awesome. Thanks for coming on the podcast.
Yeah, this is wonderful. Thanks for having me.
One reason I was excited to talk to Peter is because he's been a very high-profile and productive engineer pre-AI, building some of the most resilient distributed systems in production. CockroachDB is known for its resilience and how even if several nodes are destroyed, the database still operates without data loss. Basically, it's as hard to get rid of as cockroaches are, hence the name.
One interesting part of our conversation was how Peter built more efficient data structures than the standard coding libraries had, thanks to him and colleagues paying attention to parts of the library that seemed slow. He did it for the C++ STL map and in Go for the Swiss table implementation. I found both stories a good reminder that you can improve the existing library or even the language, especially if you measure which parts feel slow.
Another part of this conversation that I liked was how Peter came a bit of a full circle. He used to write 100,000 lines of code per year, being a very productive engineer and CTO. He did then stop writing code, aiming to coach engineers between 2022 and 2024. And then he started to code again because, with AI tools, he wanted to coach his engineers better. But it's hard to do if you don't use the tools yourself. And now he finds himself being extremely productive, and this time the team around him is productive as well. And we're not talking about vibe-coded software, but database-worthy, high-quality code generated and committed to production.
Peter is convinced that AI amplifies existing expertise. And this is one reason why he probably learned more this last year building with AI than the previous five years combined. I find it a valuable reminder that learning and building deep expertise in software engineering is very valuable.
As closing, I appreciated that Peter said that not only is he excited, but he's also exhausted. There's a lot to learn, but it's tiring, and neither he nor anyone I know is immune to this. So if you're also exhausted with all of the things going on with AI, know that you're not alone.
Check the show notes for more of The Pragmatic Engineer deep dives on Google's engineering culture and on distributed systems. If you like this episode, please make sure you're subscribed in your podcast player, and a special thank you if you leave a rating. Thanks, and I'll see you in the next.
Article published · Updated