Titus Winters on Google's Engineering at Scale and Why the CS Curriculum Should Be "Burned Down"
The Pragmatic EngineerTitus Winters spent more than a decade at Google. He led its readability program, helped build Abseil, and was lead author of Software Engineering at Google. In 2023 he was the only working software engineer on the steering committee that set the ACM/IEEE undergraduate computer science curriculum guidelines. He is now a Senior Principal Scientist at Adobe, focused on developer experience.
In this conversation with Gergely Orosz on The Pragmatic Engineer podcast, Winters covers what Google learned about changing code safely at enormous scale and why teams and psychological safety still matter when AI writes much of the code. He also explains why he thinks today's productivity metrics miss AI's most important value. The thread running through it all is his argument that computer science education, which he describes as largely unchanged since around 2000, needs radical rebuilding.
From a QBasic Book to Google New York
Winters traces his start in programming to a Valentine's Day when he was about 11 or 12. Instead of buying candy, his father, who worked in carpentry and set design but sensed computers would matter, bought him a book on learning QBasic. Programming became his hobby for the next five or six years, self-taught in a pre-internet era "when you had to really fight for it." He later attended a magnet high school in Las Vegas with a computer science program, then Harvey Mudd College.
His graduate path was unusual. He did a PhD at UC Riverside in computer science education, working on applied AI and educational data mining, and published at some of the earliest CS education research conferences. His advisor was a long-serving department chair, so much of his graduate experience consisted of learning "all of the stories of how we got here" and teaching whatever classes needed covering. That background resurfaces later in the conversation.
After the PhD he did not get a professorship. He spent four years at a small consulting group of 30–50 people, mostly working indirectly for DirecTV on embedded set-top box software. In 2010 he interviewed at Google, which was then among the hardest places to get into. He recalls feeling he didn't do well: he flew up from San Diego the morning of the interview to save vacation days, which he now calls a bad plan. In one interview, on camera angles for a Mars rover, he wrote a placeholder function with the right inputs and outputs, said it would work "if I wasn't stupid this morning," and moved on to the interesting parts. He thinks the interviewer appreciated that. He got the offer and moved to Google's New York office, which already had a few thousand people.
Finding the Right Team
His first placement was a poor fit. Because he had a networking background, Google put him on a team monitoring traffic crossing the edge of Google's network, which he describes as really a databases problem, and databases were never his interest. Orosz pushed back: shouldn't a new hire at a hot company just dig in? Winters says he did what he could to learn, but there was also some interpersonal tension, and the fit felt wrong.
About two months in, an email went to New York engineers proposing a new team to maintain old C++ infrastructure code written by early employees "whose employee IDs are like two digits" and who had more important things to do than handle patch requests. Nine or ten people raised their hands. Winters was the only one who showed up to the second meeting, so he joined. He notes this was roughly the last year before Google became formal about headcount and team assignment, though he admits he may not have understood the internals then.
The work suited him. At DirecTV he had built an API for UPnP-style local network RPC, after seeing teams hand-assemble XML for every request. He likes this kind of infrastructure and API design work.
Why Google Made Testing Mandatory
Orosz pointed out that maintaining other people's C++ sounds like a lose-lose: break something and people are angry, do it well and nobody notices. Winters says the safety net was already in place, because Google had already gone through its "yes, you have to write tests" phase.
The story he relates is that around 2006–2007, Search lost the ability to release on a regular cadence. The goal was monthly releases, but they might get one per quarter because of persistent regressions. At the time, the architecture was not well separated, so a change to Gmail could still affect Search. Release engineers banded together and declared that changes without tests would not be merged, regardless of which VP asked or how important the feature was, and would be rolled back. Once that line was drawn, even the engineers who considered tests beneath them had to comply. In 2010, Winters notes, mandatory testing was not yet an obvious industry norm.
For his team, this meant that if they refactored a library and all tests passed, either the change worked or the test coverage had been insufficient. Either outcome was useful, since exposing missing coverage now beats discovering it later at twice the traffic.
Winters states a principle he says has been on the first pages of the book for ten years. A codebase is sustainable if, for any valuable change, you are capable of making that change safely. You may choose not to make it, for reasons of technical debt or priorities, but being incapable of change is a high-risk bet that change won't be needed. "Fear is not an acceptable basis for engineering," he says. He adds that security has gotten "real real bad" in the last few years, so there is nothing that won't need to change.
Why New Engineers Fear Mistakes
Orosz asked whether a new hire who is afraid to push code is working in a poorly engineered system. Winters says it is one or the other: either the system is bad, or onboarding and mentoring didn't happen.
He connects this to education. Undergraduate programs teach that there is a correct answer, and students are graded whenever anyone looks at their work. So new grads easily fall into fear of not knowing whether they "did it right," when in reality engineering is about trade-offs and choosing a point in the design space. Orosz shared his own experience: at his first job he finished work, heard nothing, and assumed he was about to be fired when called into a one-on-one with the CEO. Instead he was praised for onboarding quickly. Years of direct, graded feedback hadn't prepared him for silence.
Winters adds that for undergrads, the only time a human looks at your code is to judge you. Experienced engineers, by contrast, constantly ask for non-judgmental check-ins ("are we going in the right direction?"), and the more of that feedback you get, the more effective you are. He laments that "we've given all of the new grads and all the junior engineers exactly the wrong instinct."
Abseil and "the Largest Refactoring Task in Human History"
Abseil began about ten years ago, after Google's initial pivot to cloud and around the time Google became Alphabet. There was interest in externalizing Google's developer tools and sharing internal components across Alphabet companies that would no longer share one company or one monorepo. The multi-tenant developer-tools idea faded after a year or two. But the low-level piece remained: Abseil was "the layer of common C++ idiom," the vernacular everything at Google is written in, open-sourced as a common substrate. Since some of that vernacular was messy, the question was whether to clean it up before committing to it publicly forever. To Google's credit, Winters says, the answer was yes. Abseil is the public projection of what got cleaned up.
Winters calls that cleanup probably the largest refactoring task in human history, while acknowledging he can't prove it. The codebase was, by his account, 200–250 million lines of first-party, Googler-authored C++, not counting external dependencies. He believes it was the public number at the time. Roughly 10,000 engineers edited it in a given month. Ten to twelve people worked on it full-time. The team's joke: Google has new skyscraper foundation technology, "we're replacing the foundation under the Empire State Building, we're not allowed to close the building or block traffic, and we discovered nuclear waste in the basement."
Orosz offered scale comparisons. Pre-AI, Uber's iOS and Android apps were each about a million lines written by around 200 engineers over years. He rarely hears of pre-AI codebases over 3 million lines, so 250 million is about a hundred times larger than anything else he has encountered.
How the Refactoring Actually Worked
The approach depended on owning the low-level interfaces: caches, containers, string routines, command-line flags. Improving an API with 50,000 call sites or 100,000 required edits was the problem. Winters was friends with, and eventually an "org chart cousin" of, the compiler team working on Clang/LLVM, which was doing abstract syntax tree transformations.
The technique ran as a MapReduce over the codebase. Mappers parse every C++ translation unit and produce parse trees. You then match against those trees, not as a text search but with the guarantee that a specific function is called in a specific context, and run code to rewrite the match. This gave them compiler-assisted, semi-automated generation of text edits.
One surprising obstacle was whitespace: they could generate edits but not place whitespace correctly. "clang-format exists because I wished for it," Winters says. He literally could not accomplish the refactoring until someone built an auto-formatter for C++.
The second obstacle was size. Any codebase has a natural background rate of change, and an edit touching 10,000–20,000 files will never test and resync before something else changes underneath it. Halting a 10,000-person company's commits for an hour doesn't scale. So the technique was:
- Design the refactoring so the old and new APIs can coexist.
- Split the edits, for example by directory, and send them out in parallel batches.
- Once everything is migrated, race to prevent anyone from copy-pasting the old form.
- Delete the old API, make the new one the default, and move on.
"You just do that a few hundred times and your codebase is in much better shape." Initial reactions were often "why are you changing my code?" but quickly turned positive. Winters notes a side effect: when someone is constantly "janitoring" the codebase, copy-and-paste becomes the correct way to learn how to do something, because if a pattern were wrong, it would already have been fixed.
AI for Rewrites: Generate Deterministic Tools, Not Constant Inference
Orosz observed that AI-driven rewrites today feel "janky" compared with Google's deterministic approach, and asked whether that approach would spread. Winters says it already exists. In Java, tools like OpenRewrite let you declare a code pattern and its rewrite, then apply it across all source files. AI turns out to be very good at generating OpenRewrite recipes. That is far cheaper in tokens, more accurate, faster, and auditable. He recalls talks from, he thinks, Netflix and Uber (while saying he could be wrong) that reached the same conclusion: use a deterministic tool that declares the intent of the rewrite, and have the AI learn to produce configurations for it. In C++, friends of his have a startup called BrontoSource focused on high-level declarative rewrites.
He cites a recent Kelsey Hightower talk on "zero token architecture." The thesis, which Winters thinks everyone is converging on, is not to use AI for constant inference. Use it to generate the deterministic tool that does the work, and let AI fill the gaps that the tool can't solve or decide when to invoke it. An architecture built on token consumption forever is probably neither the cheapest nor the most effective.
Orosz described a head of engineering who runs expensive, slow AI code reviews only occasionally. When a review finds something, agents generate a new lint rule, which is cheap, repeatable, and deterministic. The idea was inspired by Uber's pre-AI culture of encoding every lesson, including architectural rules, as lint rules, which made Orosz's first week there painful. Winters agrees: learn hard lessons once, codify them deterministically, apply them fairly, and reserve code review for what humans actually need to do. He thinks not everyone invested enough in linting and static analysis before, and that it is equally true now.
Writing Software Engineering at Google
The book began near the tail end of the idea that Google might ship its developer tools externally. If it did, it would need to explain why those tools are the way they are; otherwise it would feel "like aliens came down" offering something that made no sense. The team's "hubris" to write it came from the refactoring work: having touched everything any engineer thought was a clever solution, Winters' "cadre of weirdos" had seen more "I can see how you thought that was a good idea, don't do that" cases than almost anyone. He stresses these were testing and local-code-design scars, not production scars. "My job ends at merge," he says. That is part of why the book doesn't cover operations, which would have doubled its length.
The book followed the model of the SRE books, gathering an expert on each topic. Winters, Hyrum Wright, and Tom Limoncelli are on the cover. Winters and Wright provided technical leadership; Wright had generated millions of edits that Winters, who had review permissions across nearly the entire codebase, reviewed. Limoncelli, long their tech-writer friend, handled editorial and structural work. Some contributed chapters were good from the start; many needed iteration. The project took almost two years at quarter- to half-time effort, and Winters didn't realize how much they had overshot until he saw the printed book: "Wow, that's huge."
Readability: Possibly the Largest Educational Program in Tech
At Google, every change requires that someone involved, author or reviewer, holds two permissions: ownership of the files, and readability certification in that language, meaning they are trusted to write idiomatic code in that codebase. The goal is consistency and common patterns, because consistency has strategic value.
Originally, employee number three, Craig, sat down with every new employee to walk them through their first change. When that stopped scaling, others acted as proxies, reviewing a big substantive change. But as most work shifted to small edits in existing products, finding a single change meaty enough to demonstrate full competence got hard, and the process felt like hazing. Over the years, starting, Winters believes, with Java, it became a mentoring program. After your team approves a change, a reviewer from a round-robin pool looks for anything substantive worth commenting on. On teams with strong standards there is little to say and you graduate quickly. If you choose not to learn, you graduate slowly, if ever.
Winters says the educational design was deliberate, at least in the parts he influenced. When he left Google almost three years ago, he was the overall owner of the program, and he argues it may have been the largest educational program in tech, or even the world, with tens of thousands of participants at any moment. Some people were "super pissy" about it, since participation wasn't voluntary. Orosz described a progression: addressing comments, wanting fewer, reaching zero, becoming a reviewer, giving feedback. Winters adds that reviewers saw other parts of the codebase and other teams' tech, making it "a huge, subtle, below-the-waterline information-sharing program."
Why Teams Still Matter With AI
Orosz read a passage from the book (a chapter Winters doesn't think he wrote) saying lone craftspeople are rare and great achievements come from heroic team efforts. With talk of "100x engineers" working alone with agents, does the team still hold up?
Winters says yes, regardless of how bullish one is on AI. If agents are the way forward, engineers are still managing a team of agents. He has heard more people say the problem now is turning every engineer into an engineering manager for agents without the training or background for it. If instead AI is a tool like an IDE, then it is still you working with a team of humans. Agents don't have feelings, he says, which changes social dynamics. But clear communication and getting ideas out of your head remain essential. "Software engineering is a team sport, period."
Orosz pointed to the AI labs themselves. Claude Code has a public face in Boris Cherny, but there is a team behind him that clearly gels and trusts each other enough that things keep happening when he is on holiday. Winters picks up on the word trust. He says the labs naturally have something much of the industry lacks: they are learning as they go. Across DORA and other research on productivity and team effectiveness, he says, it boils down to culture, psychological safety, and the permission to learn: admitting you don't know, learning from each other, sharing mistakes so the team only makes them once. Lint rules and SRE postmortem culture are examples. "Every effective structure and organization in tech is actually about information flow and learning." He recommends Cat Hicks's work, and mentions he may have written the education chapter for a forthcoming Platform Engineering at Google book, now in O'Reilly early release.
Psychological Safety as a Predictor
Playing devil's advocate, Orosz asked whether smart engineers really need psychological safety. Winters cites a figure he says has been hard to trace to its source but that he heard from people involved in DORA research: a 15-question assessment of team culture can explain about 30% of the variance in whether a team hits its goals. "Find me anything that's even remotely that powerful a predictor of success. It's a people problem, period."
He then critiques "The Surgical Team" chapter of Fred Brooks's The Mythical Man-Month, his least favorite in the book. It assumes a single project: the best person does the work and everyone else supports them hierarchically. That might make sense for a one-shot task. But engineering is an iterated game, iterated prisoner's dilemma rather than prisoner's dilemma, and that calls for a different strategy: give work to the most junior person who can do it with supervision, so they grow while the outcome remains decent, freeing senior people for problems nobody can yet solve. That requires people to admit what they don't know, which brings you back to psychological safety.
Orosz argued that AI labs get this partly for free. They are in hypergrowth with rising equity, so performance reviews matter less, and since nobody knows the new technology, saying "I don't know" is natural. At companies with flat growth, performance reviews, and annual cuts of, say, 6% of staff, people are incentivized to look smartest, hoard projects, and distrust teammates. Winters agrees. If people believe team outcomes drive their own success, they don't need the one shiny task, and "it might be my turn next." Individualism and competition within a team are "a recipe for stagnation."
Layoffs and Their Hidden Costs
Asked how layoffs change engineers' behavior, Winters describes weeks or months in which every one-on-one starts with "how are you coping?" before any work is discussed. He is very skeptical that executive accounting includes the lost productivity: people updating resumes in case they're next, one-on-ones turned into therapy instead of planning. Layoffs are demotivating because they drive home that the job is "just a business obligation," and if putting in extra doesn't buy safety, "what's the point?"
He praises Adobe on this point. Its long-time CEO, who recently stepped down, saw during the Great Recession how much layoffs hurt morale and productivity. The lesson was that what made numeric sense was "not good for the soul of the company." So, Winters says, Adobe didn't do sweeping layoffs in recent years, though there are local, group-by-group changes as priorities shift.
Measuring Developer Productivity and What AI Breaks
At Adobe, Winters works on developer experience, which he calls "a polite fiction": it is really the business of productivity, "we just know enough not to say that we can measure productivity." Engineering compensation is most companies' biggest expense, so improving tools and culture matters.
Since productivity has no obvious unit, he borrowed from economics: an index metric. Measure everything that might signal things are going well or badly (releases, test coverage, number of PRs, and so on), then weight them. His initial formulation used wisdom of the crowds: experienced engineers repeatedly choose between options like "one more release per week or X more tests," much like a "hot or not" game, to work out linear trade-offs. Remove duplicate stats, keep ones with non-linear relationships, and combine them into a unitless but directionally useful number you can track over time. Adobe is doing something along these lines, he says, slightly smarter than his first version.
But he has realized this won't capture everything after the AI boom. In a recent paper on develop/deploy/operate, he describes four forms of impact in tech:
- Product success: revenue, net promoter score, adoption.
- Compute efficiency: cloud spend, or the Android example of shaving a megabyte off an app download.
- Engineering capacity: productivity gains that aren't dollar-fungible; you just get more engineering capacity.
- Changing the game: strategic innovation. Research makes impossible things possible; security makes very bad things impossible.
AI's problem for metrics lies in the fourth category. Even a clever index can't capture a change in the value of a given piece of work. His example: you can now write a JPEG parser with whatever features you need in a couple of weeks, which would have been "insane" to say three years ago.
Orosz offered a story from a CTO friend at a mid-size niche engineering company. The CTO suspected the company could do the same work with half the staff and become cash-flow positive, but didn't want to make cuts. Weeks later, the company was heading toward cash-flow positive anyway. It had licensed a closed-source, high-performance component from a vendor for a revenue share of something like 5–15%; writing it in-house would have taken a team years. An engineer suggested feeding the AI the specs and the company's extensive test harness. The resulting component was faster and better, built with no prior domain knowledge. When the company considered open-sourcing it, the vendor offered to pay them not to and to take the implementation for re-licensing. Had layoffs happened, Orosz noted, that engineer might have been gone and nobody would have known the game could change.
Winters sees this as bridging a gap. He has not seen research showing post-AI productivity gains above roughly 20–30%, yet people describe 10x experiences, though he "super" doesn't think it's actually 10x. His explanation is that much of AI's value isn't doing the same thing faster but doing harder, more valuable things you wouldn't have attempted. If your metrics count widgets leaving the factory and you start producing deluxe gold-plated widgets, you can't see it. He quotes senior engineers who report being up 70% on code and up 30% on outages with the same number of features. The unknowable part is whether the value of those features went up.
A Curriculum Frozen in the 1990s
On computer science education, Winters establishes his credentials: after the book, he joined the ACM/IEEE/AAAI effort to define undergraduate CS degree requirements, served on the steering committee as its only working engineer, and mostly managed the software engineering requirements.
His blunt assessment, which he also gave in a pre-AI keynote, is that most of the CS curriculum hasn't substantially changed since around 2000, maybe earlier. The average topic is exactly what was needed to build a database or operating system in the '90s. He asks whether topics are taught because they're relevant or because of tradition. Orosz added "or because we know how to teach it." The oldest CS departments are only 50 to 70 years old, far too young to run on tradition. The curriculum has grown by accretion: add topics, cut depth on existing ones "by epsilon." Even before AI, he wanted more practice and drill toward fluent programming and algorithmic thinking rather than "trivia understanding of a million different topics."
Now, he says, a student can complete an entire degree without thinking by feeding assignments to an AI. So he is pushing to "get out of the code" and up-level substantially. Students must still read code, think as systems thinkers, and understand structure and execution. "But no one needs to implement a linked list," and perhaps linked lists shouldn't even be taught, since memory access makes them slow. The focus should shift to design, specification ("precision in saying what you want to happen"), and testing, validation, and verification. His prediction: CS enrollment may plummet while software engineering programs grow, because what matters for getting a job in the AI era is teamwork, communication, design, specification, testing, and validation.
Reps, LeetCode, Boot Camps, and Co-ops
Orosz described his own CS degree: lots of theory, one team-based project course in four years, and a sense of starting from scratch when he reached industry. He had the theory but not the reps. Winters argues LeetCode-style interviews exist because of this gap. They were a proxy for whether someone had spent enough time on task, as programming contest participants had, not because companies needed another breadth-first search.
Orosz noted that boot camps, strong around 2014–2017, took people with no background through three to six intensive months of building a full web application. Graduates arrived knowing version control, pull requests, databases, and browser dev tools, skills four-year grads might or might not have. Winters has done the math: the hours in a CS degree roughly equal 15 months of full-time work, including all the questionably relevant theory, compared to a six-month boot camp laser-focused on immediately relevant skills.
Orosz raised Waterloo, whose grads are common in Silicon Valley, attributing it to built-in placements: about four internships, roughly a year inside companies like Shopify and Meta. Winters says he is "incredibly jealous" of many Canadian programs' co-op structures, even though those degrees often run five years. Some lessons, he admits, were left out of the software engineering requirements because they can't be taught academically. He wants more projects, but worries that a single project course gives the wrong idea of teamwork, since no real team consists of people with identical backgrounds.
What New Grads Should Focus On
Winters calls "learn the basics" a loaded question because nobody agrees on what the basics are. He cites Kent Beck: everyone currently working grew up with a playbook, and now there isn't one.
His advice for juniors starts with resilience against pasting a ticket into an AI prompt and checking in the result: "you can be replaced with a very short off-the-shelf script." Use AI to widen your inputs and better yourself, not to immediately produce all your outputs. He invokes "Andrew's Axe," a metaphor from a think tank about a year ago: if you chop down a tree with an axe, you chopped down the tree, not the axe. He also quotes a 1979 IBM training manual: a computer can never be held accountable, so it should not make management decisions. AI labs won't indemnify you for AI mistakes, so you must stand behind every pull request with your name on it. That takes rigor and attention to detail, which he thinks is hard to train.
He relays what he hears from professors. The best students, B/B+ and above, are better than ever because they have an infinitely patient, always-available, well-read, "sometimes hallucinatory" tutor. The worst students are somewhat better because they can now produce slop instead of nothing. But the middle has collapsed: students who would rather party or play games slide into habitual copy-paste. Some professors at decent schools wonder whether students below a B understand how code works at all.
Banning AI is nonsense in his view, echoing old fears that high-level languages would prevent learning assembly. He co-published a piece in Communications of the ACM with academics, opening with a scenario in 2040: if schools treat all AI use as cheating and keep asking AI-native students to find palindromes or measure rainfall in a graph, they will learn nothing. The task is to assume AI use, define new learning objectives, and keep enough friction to force thinking, almost certainly at a higher level.
His simple fix is to stop grading only the outcome. Grade the process: the prompts used, the tests written, how the student verified the result. Across disciplines, he argues, we have used artifacts as proxies for struggle. "We don't need another essay on Catcher in the Rye... We don't need another linked list." Instead, ask for evidence that the student struggled and learned, and "turn in the whole package."
Titus's Ideal Curriculum
Pressed to be opinionated, while admitting "nobody knows, there is no playbook," Winters sketched his approach:
- An unaided first semester. Students learn without AI until they can read code and follow control flow.
- Well-scaffolded educational AI. AI deliberately designed not to hand over answers, paired with process-based assessment so that getting a full answer from ChatGPT isn't a threat.
- A "CS zero" on interfaces for majors and non-majors alike. How to describe inputs, outputs, and interface design: checkbox versus radio button, when to use a button, command-line flags versus positional parameters. Even if it is the only class someone takes, their vibe-coded tools will have a more coherent design language. He cites his condo building's laundry app, which uses a menu where it should use a button.
- Motivation before theory. Move into projects and teamwork, letting students hit problems that make them need to know how the web or a remote API call works. "You can't give people a solution to a problem they don't realize they have." Theory presented first is just disconnected trivia.
- Projects without single correct answers, since that's not how industry, or mostly research, works.
- Grading peer critique. Students review others' work and are graded on giving feedback without being a jerk, for the sake of psychological safety. AI could help with the workload, for example sentiment analysis to flag rudeness.
He worries this will happen in software engineering departments, not CS departments.
Advice for Breaking In
Winters says most of his own success came from not doing what everyone else was doing. Chasing the same hot topic as everyone else makes it nearly impossible to stand out. "To thine own self be true": find something you genuinely care about, especially if it's weirder than what others do, and put real time into it. That gives you a novel story for interviewers and makes you worth talking to when a startup founder is deciding whether to take a risk on you.
He closes with a worry and a prediction. With AI-assisted engineering, he thinks the emphasis on raw engineering skill is dropping, since you can get by with less depth. What replaces it is much deeper understanding of the problem domain. That might reduce mobility across the industry: identity shifts from "I'm a C++ engineer" or "I'm a back-end engineer" to "I'm an engineer in embedded audio," and that domain depth becomes the real advantage.
His book recommendation is Cat Hicks's The Psychology of Software Teams. He wishes a couple of its chapters could be downloaded into every leader's brain, calling its "brains in jars" analogy "haunting."
That was probably the largest refactoring task in human history. It was 200 or 250 million lines of C++ code, with 250 million lines of first-party, not counting external dependencies, and 10,000 engineers probably at the time making edits to that codebase in a month.
What is readability review?
When I left Google, I was the overarching owner for the readability program. That might be the largest educational program in tech because there were tens of thousands of people participating in that at any given moment.
Most of the CS curriculum hasn't substantially changed since 2000, maybe earlier. The average thing in the curriculum is exactly what was needed to build a database or an operating system in the '90s.
That doesn't sound good, does it?
Not for a field that prides itself on innovation. A lot of what I'm pushing for is we need to get out of the code. We need to up-level substantially. CS enrollment is going to plummet, but software engineering programs are going to kick up because, let's be honest, the stuff that's going to matter in an AI era and still get a job is
What if an engineer who helped define the contents of today's software engineering curriculum said that most of this should be burnt down? An actual software engineer says exactly this. In 2023, Titus Winters was the only active software engineer on the steering committee for the ACM and IEEE undergraduate CS curriculum. Before, he spent more than 10 years at Google, was the lead author of the book Software Engineering at Google, and ran the company's readability program.
In today's conversation, we discuss why Google made testing mandatory around 2006 after the flagship search product could no longer be released on a regular cadence due to persistent regressions. Ways to measure developer productivity and why the value of AI does not show up in the metrics that we measure today. Ways that Titus would change computer science education, including why he'd want students to be graded on their prompts and tests, not the code. And more.
If you want to know what Google learned about engineering at scale or how modern computer science education should look like in the age of AI, this episode is for you.
This episode is presented by turbopuffer, vector and full-text search built on object storage. It's fast, cheap, and extremely scalable.
Before we start, I'd like to talk about Linear, our season sponsor. Until recently, I was thinking of Linear as a tool that helps plan and organize the work, but it's much more than that. A lot of teams use Linear to ship changes from idea to merge in GitHub, all without leaving the tool.
I sat down with Dima, an engineer on Linear team, to see how devs inside of Linear use Linear to build all of Linear. So, here's a real issue. When you submit a review with unsent draft comments, Linear warns you about these comments, but it doesn't tell you where those drafts are. A customer complained and Dima decided to fix it.
Before the fix, Dima did not open his IDE. Instead, he opened up Linear and described the issue to the agent. The agent built a fix, created a PR, and attached Storybook screenshots for easy verification. You can see the fix with the draft comments linked on the screenshot.
In the PR, there's a Vercel preview. You can open the preview with one click from Linear to play around with the change, and we see that it worked.
And here's my favorite part, guided reviews, which is work in progress. Guided reviews is a new code review experience where the important stuff just pops out. The top of the review has a summary and diagram of the change. And the pull request is divided into chapters that walk you through what's changed. These blocks highlight the important changes in a way that's easier to understand and review.
Oh, and when Dima's finished with all of this, he just self-approved the code and it's merged into GitHub, all without leaving Linear. If you take one thing away from this, Linear is no longer just a place for planning tickets and organizing the work. It's where all the work happens, creating the issue, kicking off the agent, checking out the preview, doing the review, and merging to production. Linear is the product development system for teams and for AI agents, and I'm loving this evolution. Check it out and get a 6-month trial at linear.app/pragmatic.
Titus, welcome to the podcast.
Thank you so much for having me.
I'm so excited to have this one. It's awesome to do this in person again.
Yes. It's a joy. So, how did you get into tech? Growing up, did you know that you're going to do something with computers?
My earliest memories, I thought I was going to be a scientist, but as a three-year-old that means lab coats and test tubes and I don't know what else, right? My dad very insightfully knew that computers would be important. That was not his field at all. He was in carpentry and set design, things like that. So, there was always a computer in the house. It was usually a low-end thing of whatever we could scrape together and afford.
And then I remember I really got my start, I think I was around 11 or 12, and instead of buying candy for everyone for Valentine's one year, he went to the bookstore. And I don't know what my brother got, but I got a book on learn QBasic. And it was all downhill from there. I spent probably the next five, six years programming. Teaching myself to program was my hobby.
Occasionally I would wind up meeting other kids my age that were a little into it. We moved eventually to Las Vegas and I went to a magnet school that had a computer science program. That is actually a really well-regarded high school these days. I had no idea. And just self-taught initially, pre-internet, back when you had to really fight for it. But that set me on my path, that book.
And then after high school it was obvious you went to college for it.
Yeah. So I went to Harvey Mudd, which is a great school with a mix of tech and liberal arts education. The kind of place that'll teach you to build weapons and teach you not to use them, that sort of thing. And then I met my wife there and she was a little bit more junior than I was, so I didn't want to move away. She was still in school, so I did my master's at UC Riverside actually in computer science education.
Mhm.
All righty.
I had a very strange graduate experience. I spent four years at Riverside doing a PhD in CS Ed, applied AI, educational data mining, computer science education. So, some of the earliest conferences on CS Ed research, I published there.
And my advisor was the department chair. He never generally ran grad students. The research area was not his focus, but he was constantly the chair. He was this charming, social, political sort of guy that knew all the stories. And so, my graduate work was a lot of learn all of the stories of how we got here and teach all of the classes whenever there was a hole in the
Yeah, and we'll keep coming back to probably this education, computer science thing throughout this thing. We've talked before, so I know. But let's jump next. You now graduated from college. You were looking for a job.
Yep.
What did you find?
So, first I did a sort of consulting group. Again, two-body problem. My wife went to grad school in San Diego. I did not get a professorship after my PhD. So, I did this consulting stuff. I was at a small startup consulting group, about 30–50 people. Mostly, I worked indirectly for DirecTV doing the embedded system stuff on the set-top boxes.
Mhm.
And then, after four years of that, we were done with grad schools, we were done with all of that. I took the offer to do a Google interview. Got the offer. Decided not to go to the Bay Area. And they moved us to New York. So, we've been here ever since.
So, this is when Google was opening an office here in New York.
That office was already open and reasonably big. This was 2010. So, I think there were a few thousand people there.
And in 2010, so Google was founded in '98. They went public in 2004. I remember that they started to really accumulate a global brand as, "Wow, Google is the place to work at," around 2005 to 2010. So, right around when you were applying, it was actually one of the hardest places to get into. A bit like today, Anthropic, or last year OpenAI, or in the mid-2010s, Uber.
Yeah.
How did you find the interview experience and getting an offer? Because it must have been pretty competitive at that point.
It was competitive. I was a little surprised that they gave me the offer. I felt like I didn't do that great. I remember, for some stupid reason, I really didn't want to spend a bunch of precious vacation days. So, I flew up from San Diego the morning of instead of sensibly
And then they would have paid for the hotel and everything.
Yeah. I just didn't want to. I don't know. I'm an anxious person. I like my habits and routines and stuff. Getting up at 5:00 for a big interview is not a good plan, folks. Hot tip.
So, there were a couple things. I remember one interview was a relatively simple clock math sort of thing. It was technically camera angles on a Mars rover or something. And there was a point in there, it was like, "I'm tired and having interview brain and this is just basic math. So, here's a function that has the right inputs and outputs and if I wasn't stupid this morning, that would work." And I just did it all in terms of that.
And I think my interviewer actually really liked that approach of, "Yeah, this is not the important part. Let's just talk." And so, eventually, they gave me the offer and moved to New York.
You started up. And then you joined, you got the Noogler hat.
Uh-huh. Yep. Absolutely. It was
And so, what did you work on at Google when you joined?
So, briefly, I was on—and I don't think that Google had the best placement for new hires necessarily.
Onboarding process, as we say.
That was a little rough because, like I said, the things that I'm interested in are developer efficiency, problem solves. I have a background in networks, I have a background in API design, those sorts of things. They said, "Oh, you know networks, so let's put you on this team that monitors the amount of traffic and where that is crossing the edge of the Google network." Which is actually a databases problem with networks as the topic of all of the data that you're doing. I'm like, "I don't know anything about databases. That was never my thing." So, it wasn't a great fit. And then almost immediately after that, like two months in
I want to push a little bit back here already. It's like, "Come on." You're joining Google. You had college. You had four or five years of experience. When you join a company, especially a hot company, they said, "All right, do X." And you're like, "Ah, I'm not really." Would you not want to say, "Let me just go deep into this to see why it's important for the business"? I am pushing back on you here.
No, please do. Always. I did what I could to learn from that team, for sure. There was a little interpersonal tension in some of those things.
Okay, so it already started a bit like that. You had this nagging feeling.
Yeah, it was just like, "Yeah, this wasn't quite right." And then there was an email that went out to all the engineers in New York saying, "Hey, we're thinking about spinning up a new team that handles all of the old C++ nuts and bolts infrastructure code that was written by people whose employee IDs are two digits. They have more important things to do than to handle patch requests for all of those components. So, we would like people to maintain that." And me and, I don't know, nine or 10 other people raised their hands, and I was the only person that showed up to the second meeting. So, I just got onto that team.
Oh, I get it. So basically, I think the context is just important because I think folks hearing, "Oh, I'm joining this team. I didn't really like the first thing." I think the context that was missing is, first of all, Google was growing really quickly. New teams were starting up left and right. And also moving teams was fine, right? And you were more looking for, all right, you knew that this team, both for what they were doing and also sometimes there's not a great people fit, it would probably not be the one where you're going to be staying for many years. You did the best you can, and then when you saw an opportunity, you knew it was better for your strengths, you're like, let's go for it. So for you, C++ low-level stuff.
Yeah, back when I had been doing the consulting at DirecTV, there was stuff like build an API to do UPnP, universal plug and play stuff, local network RPC sorts of stuff.
Remote procedure call?
Yeah, right. And so it had been a lot of people, teams were manually assembling the XML structure for every request to implement a single client call or something like that. It was like, we could do better, folks. And I was probably a little too clever in the way that I assembled all of that API, but that was the thing that I liked.
This team, was it more of an initiative of, hey, Google has a bunch of C++ here and there, kind of grassroots, let's get together and initially in our free time, let's see if we can do something? Or was it more some manager type said, I'm actually thinking of assembling a team, a full-time team on this?
So this was, I think, the last year before Google got actually pretty formal about you're embedded to a particular team and headcount is limited. It's a little chaotic, it's a little loosey-goosey about how those things work. I could actually be wrong about that because I was still new enough that the internals of how headcount works might have been just not clear. But moved over pretty quick, and boy, that was exactly the place for me.
All right, then tell me about it. Why was it the place? What kind of work can we imagine? Because it sounds like, when you describe to me taking random C++ stuff that people wrote at Google, that sounds like the version of hell for me as an engineer, especially pre-AI, right? Understanding what another engineer at some point wrote who might have left, and a team is using it and it's working for them. It sounds like it's a lose-lose because if you take it over and you make a change and you break someone, they'll be mad at you. And if you do it and you do a great job, they'll probably not even think about it because it's always worked. It still works great.
Luckily, the fear and the safety net for making changes had been pretty well established a couple years before this played out. Specifically, and this goes to topics that we'll get to in a second, I'm sure. Specifically, Google already went through the phase of, yes, you have to write tests. Right?
In 2010 this was not an obvious thing in the industry.
Right. But I think the story is 2006, 2007, Search lost the ability to push a release on a regular cadence. It was down to, we were trying for monthly and they were maybe only getting once a quarter because
Oh, so this is why Google's obsession with tests and quality and all that.
Uh-huh.
So the core money-making product was handcuffed, not being able to push around 2006. Wow.
And circa 2006 almost anything that you did had some tendrils in the front
end or search or stuff like that. The whole architecture and separation and multi-app, multi-product world, that was still not right. So if you're making changes to Gmail, there were still chances that you were making changes to Search.
Wow.
And so everything was held up. So the release engineers, they banded together and they simply just said, "You will not merge this change if there are not tests. Period. We do not care what VP says anything. We do not care how important this feature is. You will not do that. We will roll that back."
Yep.
And as soon as you put a line in the sand, even the people that are like, "I'm very smart. How dare you ask me to write tests? This is beneath me."
And then so basically taking over or taking on maintenance or even changing a C++ library, you had the tests so you could have high confidence if you refactor these things, all the tests pass. Yeah, either it works or there was never enough test coverage.
Right, exactly.
And either way it's a win, because it's better if you expose the lack of test coverage. And okay, you have a crash today, but tomorrow or a year from now you'll have twice as much traffic anyway.
Right, exactly. And it's one of the things that I believe very deeply, is if you are held back from making changes to a thing out of fear, you have to go in and fix that. Fear is not an acceptable basis for engineering. And I think it's in the first two or three pages of the book. I've said now for 10 years, your codebase is sustainable if, for any valuable change that comes along, you are capable of making that change safely. You may choose not to make that change. That could be technical debt or different product priorities or whatever. But if you're incapable of reacting to change, then you're placing a high-risk bet that the change isn't critical or isn't important.
And I hate to break it to people, but the security world has gotten real, real bad in the last few years. There is nothing that is not going to change. The strategic advantage of being able to change is incredible.
Is it safe to say that if I join a company, a startup, a large company, whatever company, I join the team and I look at the codebase, and my first weeks, my first months, I'm just being afraid of pushing code changes because I worry this thing will break, I'll break production, that I'm working on a system that is just, let's say it's poorly engineered?
It could be the system is poorly engineered, or it could be there's some onboarding and mentoring that didn't happen there.
Right? So one or the other problem.
One or the other for sure. And this goes to some of the educational stuff. We, in undergrad education and the way that we portray what the job is like, there is a correct answer. There is right and wrong. It is not just the trade-offs. And you're graded anytime someone looks at what you have done.
Mhm.
Right? And I think especially for new grads, it's really hard. It's really easy to get into a fear of, "I don't know if I did this right. I don't know if this was the answer you were looking for," because they've been falsely given this perspective that there's a right answer instead of, of course, it's just a trade-off. It's at some point in the design space.
And I guess you're also saying that these new grads, they're also used to: you do something that is a meaningful effort, typically a class or an exam, and you get feedback. You get a scale, you get a grade, you get a thumbs up, a pass or a fail at the very least.
And I still remember when I went to my first workplace. I still remember this. This is really weird. I joined the company, I built this project, and then nothing. And then I do another thing, and then nothing. And then I have a one-on-one with the CEO of the company, and I'm thinking they're going to fire me because clearly I knew that, middle of the day, I was sometimes slacking off because I finished my stuff. In hindsight, I was pretty efficient, but I just finished my stuff and then I was just browsing the internet. I told them, "I'm ready." And I was pretty sure maybe I'll be reprimanded or something. And instead, they told me, "Oh, amazing job. You've onboarded so quickly. We're so happy with you." But the difference was that up until then I had years of getting direct feedback, and it didn't connect in my mind.
Yeah. The other part of that is the only time that a human is going to look at your code if you're an undergrad is to grade you, is to judge you. Whereas when you become a more experienced engineer, you're asking for feedback regularly, but it's just a check-in: "Are we going in the right direction? Does this make sense to you?" And the more of that feedback that you can get that isn't judgment, the more effective you will be. I hate that we've given all of the new grads and all the junior engineers exactly the wrong instinct.
So, going back to Google, what is Abseil? How do you pronounce it correctly?
Abseil.
Abseil. Yeah. So, Abseil was starting about 10 years ago. Shortly after the initial pivot to cloud, there was a motion to, "Hey, we have all of these amazing developer tools in addition to all this amazing production stuff. Maybe we should try to externalize the developer tools as well." And at the same time, there was also a little bit of Google shifting into Alphabet. So, for both the developer tooling aspects at one level of, how do we share developer tools amongst different companies within the Alphabet umbrella? And how do we share high-value internal interfaces and APIs and tech components? How do we share those when it's not one single company?
And also not one monorepo, right?
Increasingly, yeah. Exactly. And so there's a bunch of work that's kind of the normal, okay, what does it take to make these developer tools multi-tenant and slightly less completely tied into that ecosystem?
And just multi-tenant, for those who are not that familiar with it, what does multi-tenant mean inside of Google?
Well, so this would have been multi-tenant of other companies within the Alphabet umbrella, and then maybe extend that out to cloud customers.
Oh, got it. Yeah, so like a Waymo and Google could theoretically use it, something like that.
That became not the hottest topic, and they kind of dropped that idea after a year or two. But the low-level components, if you wanted to share these things that were written internally in the monorepo, what would it take to do that effectively? And Abseil was basically, here is the layer of common C++ idiom. This is the vernacular that everything at Google is written out of. Let's open-source that so that there's a common substrate for all Google tech.
At the same time, the question was, some of this vernacular is gross and tech-debty. Should we clean that up before we commit to it forever? And the answer was yes, to their great credit. So, the Abseil open-source project is the public projection of the stuff that got cleaned up. Most of the work was that. I would defend pretty strongly, that was probably the largest refactoring task in human history.
No way.
Yeah, we had 10 or 12 people working full-time with world-class tools that no one else understands yet, just cleaning up and making sure that idiomatic code externally would be compatible with idiomatic code internally.
And obviously when you're talking about cleaning up, you're talking about cleaning up utilities and components that are in the middle of some of the highest-use services in the world. Google Search.
Yep.
Gmail.
Yep.
Maps, you name it.
All of it.
If you would break something, you would probably see at some point an outage in one of those services.
It was at the time, I think it was 200 or 250 million lines of C++ code.
250 million lines of C++?
First-party, not counting external dependencies. And Google had that kind of, I think that was the public number at the time, 10,000 engineers probably at the time making edits to that codebase in a month.
And so our job was, we joked about, that's a New York reference, we joked about it as our job is the logical equivalent of Google has a new technology for the skyscraper foundations. We're replacing the foundation under the Empire State Building. We're not allowed to close the building or block traffic. And we discovered nuclear waste in the basement. It's okay, guys. We got this. It was a great team.
I just want to put the 250 million lines of code in perspective because today with AI, AI can generate, we have crazy numbers, it can generate tens of thousands of lines of code in an hour, etc. And you can get to millions if you want to. Honestly, just an agent can do it. However, pre-AI, Uber's mobile application was about a million lines of code on iOS and on Android, and that was each written by 200 engineers for years. Bun runtime at Anthropic just rewrote something like, I think it was 500 or 900,000 lines of code, and that was written by a small team.
I find it rare to find examples of a project that has more than, pre-AI, more than let's say 3 million lines of code, and that will typically be hundreds of engineers writing it for several years. The reason I was like, "250 million," that is about 100 times more than I've heard for a codebase. And we're talking about clearly this is written for production. You're not just goofing around and pushing some scaffolding here and there. And when you say first-party, it's easy to have a lot of code when you take on an open-source component, but no.
No. No, that was all Googler-authored. And so we have some insight into how things go.
Okay, so the largest refactoring at that point in human history, you're not exaggerating, likely.
I mean, I can't prove it, but I would be surprised.
So, you're in the middle of this, 10 to 12 people. You have this massive task that seems giant and almost... How do you go about it? How do you break it down? How do you plan for it? How do you speed it up? Because clearly if you just did it by hand, this would have not fully worked.
Right. So, one of the things that had enabled any of this, and that came out of having been on the low-level C++ infrastructure team to start with, was, okay, if we own the interfaces that are the nuts and bolts that everything at Google is written out of, way below the database level, just caches, containers, string manipulation routines, command-line flags, that sort of thing. If we own those things, maybe it's valuable to make them better. But how do you do that when there's 50,000 call sites or 100,000 edits that need to be made to change that API?
And so luckily I was friends, and org chart cousins eventually, with the compiler team, especially Clang, LLVM people. And so they started doing stuff in the AST transformation sort of thing.
AST?
Abstract syntax tree. So you take the parse tree. And it's Google, MapReduce. So if you do a MapReduce over the codebase where the mapper parses all of the translation units of your C++, all of the build targets, and produces, here is the parse tree, then you can match the parse tree for specific, it's not text match, it's, I guarantee you are calling this particular function in this particular context. And then you can apply some code to rewrite based on that. And so we had compiler-assisted, semi-automated refactoring to generate all of the text edits.
Turns out one of the things that was really hard at first was we could generate the edit, but we couldn't get the whitespace in the right places.
No way.
Yeah, seriously. So clang-format exists because I wished for it. This is the autoformatter for many C++ sorts of languages at this point. But it's like, I literally cannot accomplish this refactoring until someone figures out a way to hack together auto-whitespace for C++.
And so you do that, you generate one big edit. It is too large to fit in the codebase because for any given codebase there's kind of a natural background level of how much change is happening. And if you're trying to edit 10,000 files, 20,000 files, it is likely the case that you will never be able to test and resync before something has gone out of sync.
Yeah.
Right? Someone else edited something.
Yeah, exactly. Merge conflicts, parallel changes, that kind of stuff.
Yeah, exactly. You do not want, at a 10,000-plus-person company, to be like, "Everyone stop touching the repo for an hour. Let me do this thing." That's not going to scale. And so the tech is you figure out how to do your refactoring so that the old API and the new one can coexist in spelling.
Yeah.
And then you parallelize, just cut by directory structure. "All right, I'll send you this batch of updates, this batch of updates." So you wind up going like that. And then once everything is clean, then there's a race to make sure no one copy and pastes the old thing. And then you delete the old thing, the new thing is the default, you move on with life. You just do that a few hundred times and your codebase is in much better shape.
Yeah, wow. And then you're doing this all the while other engineers, they're just doing their thing, they're using these things they don't even know about. They don't want to know about you.
Exactly.
They might be aware of you, but they shouldn't have to change their, just as you said, slow down or...
Yep. But then we did get amazing feedback on some of this. At first people are like, "What are you doing? Why are you changing my code? I'm not okay with this." And very quickly they're like, "Oh, this is great. I go away, my code just gets better. And I don't need to worry about looking up, is this the right way to do a thing? Because if it was the wrong way to do a thing, you would have gotten rid of it already."
When you have someone that is going through and just janitoring the codebase like that, copy and paste becomes the correct answer for, how do I do this? Find someone else that did it once, you copy it, you move on with life, and you have evidence that it's probably a reasonable answer. Because if it was wrong, we would have fixed it already.
It's very interesting because today, obviously, if you did a big refactoring, the obvious tool that you would reach for is AI, as we've seen with so many companies having lots of success with, well, in the sense that AI is capable of... The challenge that you had is you had this massive codebase. Rewriting it by hand would have not been feasible, but at Google you came up with a very clever semi-automated approach, which was based on pretty hardcore language parsing and coming up with new ways where you could formally, maybe even formally, or at least prove that this thing will be correct, which is what most teams would not have the available capacity, even expertise, to do.
What everyone has the capability to do right now is, "Oh, AI, go and do this and do tests and all this." However, where I wanted to get to, rewriting with AI still feels so much more janky than what you did, which was so much more deterministic.
Yes.
And I wonder if this technique that you used at Google might at some point spread, or people will just put up with AI, it might or might not work.
I think if you're looking for it, it's already out there. In the Java space in particular, tools like OpenRewrite, it's a little configuration per run for it, and you say, "This is the pattern I'm looking for in the Java code. This is how the rewrite would look." And it goes through and blasts through all your Java source files and does that update.
Turns out AI tools are really, really good at generating OpenRewrite recipes, which is way cheaper on token consumption, and way more accurate, and much faster, and auditable. Right? It is a substantially better pattern. We've seen talks from, I want to say Netflix and Uber, I could be wrong, specifically on everyone seems to be coming to the same conclusion of what you need is a deterministic tool that declares the intent of the rewrite. You can have the AI write a little skill to train the AI how to create the config for those tools, and then you move on with life.
In the C++ space, actually, some good friends of mine have a startup right now. It's called BrontoSource that is specifically just about that sort of declarative rewrite pattern at a very high level of intent, very powerful for C++.
Yeah, I like the idea of using AI to help generate deterministic software tools, transformations.
Yeah. Kelsey Hightower gave a great talk recently on zero-token architecture. And his whole thesis, and I think everyone is coming to the same conclusion: don't use AI for constant inference. Use AI to generate the deterministic tool that will actually do the work, and then have AI fill in the gaps on the parts of the problem that you can't solve, or where to invoke the tool, things like that. But if your whole architecture is token consumption for all time, there's probably a cheaper and more effective way to do.
Yeah, I just recently heard a head of engineering tell me that they're using these AI code review tools, which are pretty good but very expensive and slow.
Right.
And this person said, hang on, what if we told the AI, okay, let's do a thorough code review every now and then, but when it finds something, let's have it generate a lint rule.
Yes, please.
Which is cheap and repeatable, and it will no longer take 30 minutes, the cost, and also it's just nondeterministic. It might or might not find it next time. That was frustrating. This person was like, look, when we know we have certain things. So now what they're doing is every now and then they still run some of these reviews, but whenever they find any issues, then they fire up inference AI, several agents, and say, all right, can you come up with an improvement in our deterministic linting and help it?
And the inspiration came actually from this person heard about how Uber—actually, I think it was heard me talk about when I joined Uber, I could barely commit my code the first week. It was so painful because the linter just kept rejecting it because Uber had this engineering culture that—
Yep.
Pre-AI, whenever you saw something break, you just put a lint rule. Architectural rules were all in lint rules. We had our architecture, and you could not inherit the wrong way because the linter would say, uh-uh, you cannot inherit from—a builder cannot inherit from a, I don't know what, a composer or whatever that was.
Yeah. No, I think it makes perfect sense. Let's try to learn our hard lessons only once, codify those rules somewhere deterministic, apply them equally and appropriately and fairly, and leave the interesting parts of code review. Make sure that the parts of code review that we're focusing on are only the things that humans need to be doing. Right? Which was true then. I don't think everyone invested in linting and static analysis quite to that extent. They should have. And then it's equally true now. Right? We can't review everything, but there are places that require this human attention, is well spent.
Titus just made the case for learning the hard lessons once, then writing them as rules, and saving the human attention for the things that really need such. That is pretty much the same way you want permissions for AI agents to work, which brings us to our season sponsor, WorkOS.
WorkOS provides the auth infrastructure that B2B and AI-native companies need to sell to enterprise customers with SSO, SCIM, RBA, and now authorization for AI agents. WorkOS Airlock is intent-based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it. Clear-cut rules like "no agent deletes cloud resources" are run as plain code. For the fuzzier calls, an AI just evaluates the policy at runtime. You write those policies in plain English, and Claude can pull the ones you already have out of Notion or Jira and import them for you.
Your agents, whether that's Claude or Devin, connect through a single access point, and your MCP servers are behind this access point. Airlock is a new layer on top of your existing permissions, which means that an agent can never get more access than the person who invoked the agent. When something risky comes up, Airlock can escalate to a human for approval, say over Slack. It's the same idea Titus described. Write the rules once and spend human attention only on the calls that need it. You can request early access at workos.com/airlock.
I also want to mention our presenting sponsor, turbopuffer, a ridiculously scalable, fast, and cheap hybrid search engine built on top of object storage. How often do you get to see a rewrite playing out in real time? With the turbopuffer team, we finally can. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale.
The new system, called turbopuffer V3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, V2. The turbopuffer team had been perf maxing query plans until they are as fast or faster than V2, doing all this in public all the way up to the V3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to being several times faster on things like cold vector search, order by attitude, or hybrid search.
The V3 system is still slower in areas like full-text search, but only one to two times slower versus 100 times slower when this perf maxing started. And the turbopuffer team is sharing exactly how they optimize things in their development logs with things like group count aggregation, compressed column encodings, batch scoring, and many more. A lot of these wins are basic foundational work right now, but the team will be showing some very cool systems work over the next few weeks.
But hang on, why is the team doing a full rewrite? It's because search is changing, and so is turbopuffer. Turbopuffer started as a vector database, but it is no longer a vector database. It has evolved into a high-performance database for search. The V3 discards the vector primary index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast and just generally a very fast database. The turbopuffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built.
Speaking of hard lessons and documenting them, can we talk about Software Engineering at Google, the book that you're quite well known for? How did the idea come? How did it come together? What was your motivation to start with?
So, this was a little bit at the tail end of, hey, maybe we'll ship the developer tools. Maybe we'll need to explain why Google's developer tools are the way they are, because otherwise it'll be a little bit like aliens came down and they're trying to give you something that doesn't make any sense in your world. Part of the reason that we had the hubris to tackle that sort of task was because we had been doing this refactoring at scale everywhere.
And when you run into everything that any engineer thought was a clever solution to their problem, it meant that my cadre of weirdos has run into so many, "Oh, yep, I can see how you thought that was a good idea. Don't do that," more than anyone else out there.
Well, you're talking some of the most battle-scarred people in the industry just because you're working on these large-scale systems for such a long period of time.
And it's not the production battle scars. It's the testing and local code design battle scars. Everyone else in the industry has all this production experience, and my job ends at merge. I know more about development and API design and test and keeping the codebase as an artifact working than most. But it does mean, I don't know, I'm kind of an outsider in that sense.
Well, interestingly enough, the book is very, very thorough on how it gets to production, but it does not cover the operational part, which I guess is the history of the book.
There is an SRE book.
Well, then this book would be twice as long. How did you pull that book together? Because you wrote some parts of it. Other folks inside of Google wrote different chapters of it.
So, it's very much in the spirit of the SRE books. Try to get the expert on, or an expert on, given topics to contribute something useful. And then the three names on the front of the book, myself, Hyrum Wright, and Tom Limoncelli. Hyrum and I were kind of the technical leadership for that. And Tom is plenty technical, but has been our tech writer friend for many, many years.
Mhm.
So, he handled a lot of the editorial, structural, organizational stuff. But Hyrum and I had been the tag team of Hyrum generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the codebase. And so everyone knew us as those two guys that clean everything up. So we had those scars, and so we led that. And then the trick was finding the experts to write something high quality. Some of those were great from the start. A lot of that took a bit of ghostwriting and stuff.
Iteration, yeah.
The whole task was actually almost two years. Certainly not full-time. We were at most probably putting in quarter to half-time in most of that time. But yeah, we wrangled together way too much text from way too many people, and I don't think I realized just how much we overshot until I saw a printed copy of that book. I was like, "Wow, that's huge."
Yeah, and even today, I do recommend the book because it gives a really good understanding of how Google at the time worked. Obviously, there's been evolving since, but how a company going from zero to building important production systems came up with a bunch of stuff that might not make as much sense. And you talk about things like monorepo, the stringent code review process, how there's a readability process at Google, and maybe we can talk about that. Google is the only company I know where, for code review, back in the day, they didn't just have code review, like an engineer who owns this part of the system and understands reviews it and says it's okay, but Google had a readability review. What is readability review?
So, for any change that's going in, someone that is involved, either the author or a reviewer, has to have two different permission bits. First is that ownership thing. So someone has to be in the list of people that is authorized to change these files. And the second is someone has to actually be certified in, we believe you understand how to write idiomatic code in this codebase in that language. And this is a step towards trying to make sure that everything is relatively consistent and that there's common patterns and you're not reinventing the wheel because of the strategic value of consistency.
And so the readability process is someone has to be certified for that. How do you get certified for that? And initially that came as employee number three sat down with every new employee and walked them through their first change. That stopped scaling, so they got a bunch of other people to try to proxy being Craig. And they would walk you through your first big substantive change.
And over time it got to the point where most change was not, I am writing a new component from scratch. It's an existing product, you're making little edits here and there, you're adding a feature or whatever. So finding something that was meaty enough to demonstrate in one go that you knew everything, it was kind of hard, and the whole thing felt a little bit more like a hazing ritual. And so, over the years, I think people in Java started it, it moved from that sort of one-shot to being explicitly kind of a mentoring program.
You do your normal work, and after your team has signed off on it, then we will round-robin assign someone from the pool of Java mentors to go through and see, is there anything substantively interesting worth commenting on here to up the quality of the code and make sure that it matches the rest of the norms? And you might notice that means if you're on a team that already has great standards, there's probably very little to comment on, right? And you'll probably graduate pretty quick.
Yeah.
And if you're not and you choose not to learn any of the lessons, you will probably graduate very slowly, if at all. And so the whole thing kind of grew into that. And we expanded that sort of mentorship approach into all of the other readability languages, and that's been the norm for a bunch of years. I haven't checked up on it for at least a couple years, but—
But it's very interesting because it feels to me like this turns code review into just education in the depth of a programming language. Was this by design, or was it just a byproduct?
I think it's absolutely by design. Certainly at the parts of it where I was pulling any of the strings, it was definitely by design. But yeah, when I left Google almost three years ago, I was kind of the overarching owner for the readability program. I would argue that might be the largest educational program in tech or maybe in the world, because there were tens of thousands of people participating in that at any given moment.
And then you just participated by joining Google, then you had no choice. It wasn't voluntary.
Right. And so certainly some people were super pissy about it.
But I guess there was also a goal where it wasn't just about, okay, so step one is you need to address the comments that readability expert puts on you. Step two, you probably say, I'd like to get fewer comments on them if you have any sense of pride, because it's kind of embarrassing. And then step three is, you know what? Okay, I now get zero comments. I actually want to become a readability expert myself. And then that's a different level. And then step four is now you are giving feedback. And I guess through that feedback maybe you meet people, maybe you start to form some, I don't know, mentorship, friendships, who knows.
Yeah. And you see other parts of the codebase, you see stuff that other people are doing, you learn about other tech that might be happening. It was a huge, subtle, below-the-waterline information-sharing program.
In the book, you write about the importance of a team, and I'll quote you. "In the realm of programming, lone craftspeople are extremely rare, and even when they do exist, they don't perform superhuman achievements in a vacuum. Their"
Always the result of a spark of inspiration, followed by a heroic team effort. A great team makes brilliant use of superstars, but the whole is always greater than the sum of the parts. But creating a superstar team is fiendishly difficult.
Yeah. I don't think I wrote that chapter.
It's in the book that you are a co-author of. So I wanted to reflect on this, especially with AI, because now I'm starting to hear a lot more people say, yeah, in the past we talked about the 10x engineer, up to 2022, 2023.
On that.
But I am now starting to hear people say, oh, there's now the 100x engineer, which is this engineer who is a great engineer, uses AI amazingly well, and we are also hearing stories of individuals doing heroic efforts by themselves and with a bunch of AI agents. So I just wanted to ask you how the concept of a team holds up as we have more AI tools, and the concept of a standout team.
Yeah. So I think, regardless of where you fit on the spectrum of AI believerism, I think the team is still critical. If you believe that the agents are the only answer and the only direction forward, we're still managing that team of agents. And I've heard an increasing number of voices talking about it as the problem right now is we need to make every engineer into an engineering manager for a team of agents. And we don't have the training or skill set or background to have done that. We're asking them to do a thing with no scaffolding or guidance.
The alternative is you think of the AI and the agents as your tools, your IDE, but it is still you in conjunction with the team of humans around you. It's still team behavior. I think there are certainly differences in the social dynamics and things like that between you and the agents, which I'm going to go out on a limb and say the agents don't have feelings.
They definitely do not.
Right?
Should not.
Let us hold that line for a bit. But I think understanding, clear communication, and being able to write things down and get it out of your head is still a substantial, important thing. I think it still unquestionably is teams at the root of it. Software engineering is a team sport, period.
Also, I guess just one pretty naive way to prove that this is probably still true is if we just look at some of the leading AI companies, the leading AI labs, the Anthropics, the OpenAIs. When I talk to people there, I see standout individuals who are working as a team. Claude Code, for example, they cleverly have a public face in front of both of them, Boris Cherny, who created it, and I think he did an amazing job. It makes it very relatable. This is a human, but there's a team behind Boris.
Yeah.
And that team, when I met them, you can tell that they gel. They work together. They trust each other. At some point Boris will be on holiday.
Right.
You will not know because things just happen.
Right.
You said the important keyword, right? Trust.
Right. A substantial part of this is the psychological safety aspect of it. The other thing, which I think is natural for the AI labs, but hasn't been nearly as dominant for the rest of the industry yet, is you're learning as you go. And if you look at all of the research, DORA research on stuff or any research on productivity or team efficacy or any of those things, at root it always boils down to culture, psychological safety, and the permission to learn.
It's the permission to admit that I don't know something. It is the incentive to learn from each other. It is the incentive to share your mistakes so that the team only has to do that once. Your lint rules or SRE postmortem culture, every effective structure and organization in tech is actually about information flow and learning.
And I think a really great perspective on that—I love Cat Hicks, the psychology researcher for tech. Her book, The Psychology of Software Teams, is beautiful. And she talks about these things a lot. Similarly, there's a Platform Engineering at Google book that some of my friends are putting out. There's early release chapters on O'Reilly. I might have written the chapter on education for that.
So very surprisingly slash not surprisingly.
Right. But education is the powerful root for, let's make this mistake a minimum number of times. Let's codify what we've done. Let's give people permission to admit they don't know and to grow, because it's not a one-off game. It's an iterated game. And you'll do better over time if your team is better next time.
I want to go back to psychological safety because back about 5–10 years ago, we would talk so much about it in engineering leadership circles. This was pretty well understood that a good team feels a safe team. If you, for example, make a mistake, you cause an outage as an engineer, literally, you push code and that code causes an outage. We should have a blameless postmortem where we don't say that, hey, Gergely pushed this code and he caused the outage, because I feel terrible. But let's talk about, for example, how could we have avoided this at a system level? Why did a test not fire? Why was it rolled out immediately when he was a new joiner on the team or she was a new joiner on the team? Maybe some onboarding could have helped, yada yada yada.
How do you define psychological safety and why is it important? Especially just playing devil's advocate, when we're talking about smart engineers who know stuff, is it all that important to us? Because we should know our thing, right? I'm kind of removing my emotional...
Yeah.
Yeah, but here, yeah.
We should know our thing, sure. So I would say, and it's been a little hard to root this stat, but from talking with people in DORA, the research, the one thing that I loved was we can explain 30% of the variance in whether or not your team will hit its goals based on how you score on psych culture questions, right? We could give you a 15-question true-false, or maybe Likert scale, but a 15-question assessment for the team and predict 30% of whether or not you will hit whatever your goals are. That's insanity. That is more powerful than any other signal.
Okay, so you think that we're all engineers and this is all just computers and tech and emotions have nothing to do with it. Find me anything that's even remotely that powerful a predictor of success. It's a people problem, period, right?
And going back to, it's not a single-shot game. It's an iterated game. I regularly cite Fred Brooks in The Mythical Man-Month essays. And the surgical team is my least favorite chapter in that book because it's structured as if we're only ever going to do one project. Get the best person to do this, have everyone else build the tools and do the documentation under them. It's very hierarchical, right? And it's very power-dynamics mad.
Yeah.
Right. But it might make sense if you're only ever going to do one task. Instead, we're playing, in a game theory sense, it's not prisoner's dilemma, it's iterated prisoner's dilemma. There's a different strategy. And the strategy is find the most junior person that can do it with supervision so that they learn and grow and you still get a decent outcome. And the higher-level people can solve something that they don't know how to do yet or that no one else can tackle.
This is the only way to structure it so that growth happens. But in order for that growth to happen, people have to be willing to admit that they don't know a thing. And that brings you straight back to psychological safety and learning culture, right? If it is a culture that highlights information sharing, learning, and growth, you will succeed.
And you mentioned that the AI labs already take this for granted and I'm going to...
Because they don't know what they're...
Well, I'm going to paraphrase it. The AI labs have a few things going. First of all, business is booming. They are growing. Revenue goes up. Customers go up. Funding goes up. Headcount goes up. Their teams are only ever expanding. They're in hypergrowth. We've seen companies like Netflix and Uber in hypergrowth; this is even faster. Your equity also goes up. So you don't need to worry about things like performance reviews because performance reviews are typically important where, how much bonus am I going to get, how much? But when you're at a place that goes like this and your equity goes up, your financial part is taken care of just by focusing on helping the company succeed regardless of your titles.
Second of all, because they're building new technology and they know no one knows, so they just tell, okay, I don't know, I'm learning from you. Here's what I learned. So it's just all natural. Whereas compared to a company that might not grow like this, they might still be using, let's say, AI, but now there's less of an incentive to say I don't know because there's a performance review coming up. You want to look like the smartest. You don't want people to say, well, this person says I don't know a lot. Your bonus will be lower. You might be put on a performance improvement plan because you're now at a company that every year lets go of 6% of people.
Right.
And your goal is suddenly going from, I just don't want to be there. I want to hang on to this job, but it's not hanging on. It's now...
Uh-huh.
And you're now wary of maybe your teammates because now there's internal competition. You want to hog that project.
Yep.
And so this is a huge difference then.
Yeah, it's huge. If you get into the mode of believing that the team outcomes are what matter for your own success, then it is much easier for you to realize, okay, I don't have to have this one shiny task. The team will be successful by virtue of someone doing this and it might be my turn next. When it breaks down into that individualism and I don't trust the guy across the way from me or shoulder to shoulder to me, even though we're on the same team, we're scraping by and fighting for attention and success. And that's not a recipe for innovation and growth. That is a recipe for stagnation and just bad.
One of the obvious things that I assume does not do good for psychological growth, or is terrible for it, is layoffs.
Mhm.
Can you tell me based on your observation? We've all been part of it, and in 2022, 2023 there has been a series of layoffs. Right now companies are doing it again. Back then it was to do with all these interest rates and overhiring during zero interest rates and growth kind of going down. Now some companies are actually openly saying that it's AI. Some are not as openly, but we find out that Meta, for example, wanted to reduce team sizes because they thought that AI will be there. From your observations at a company where a layoff happens, how does it change how software engineers show up, work, think? Speaking both about your experience and people you talk with.
It's stressful. I've had experiences of there being layoffs in the area and it'll be weeks or months of every one-on-one starts with, "So, how are you doing? How are you coping?" There's this huge amount of negative performance influence of, "Okay, we got to emotionally support each other first before we actually even get back to doing the work."
And I am very skeptical that in the executive accounting on all of this that they're taking into consideration just the sheer amount of lost productivity and lost time from, "Yeah, people are stressed. People are spending time brushing up their resume in case they're next on the chopping block. People are spending that one-on-one time in therapy instead of in planning." It is a huge hidden cost to all of that. And it's really demotivating.
It drives home, it is a reminder that, okay, this is just a business obligation. This is not a lifestyle choice. And if me putting in extra doesn't give me extra safety on this, then what's the point? There's a lot of cultural baggage that happens for that.
And it's one of the things that I actually really liked about Adobe was Adobe's had the same CEO for a long time. He just stepped down. He has a replacement now. But he was there for the Great Recession crash. And they did some layoffs at that era. And he saw how much it hurt morale and company productivity and all of the rest of that. And he learned, we thought it made numeric sense at the time, but it was not good for the soul of the company. And that's not the company that we want to be.
And so Adobe didn't have the layoffs in the last few years. There's group-by-group local things as things get reprioritized, but it's not the sweeping all that...
But we've seen that most of big tech has had it, especially in 2023.
Yeah. And speaking of Adobe, so you've moved to Adobe a few years ago. What are you focused on at Adobe?
So there I'm a very senior IC focused on developer experience. I happen to say this is a polite fiction. Developer experience is the business of productivity. We just know enough not to say that we can measure productivity. It is not given...
Unlike McKinsey, who me and Ken have bonded over.
Yeah. That's horrible.
We'll put a link in the show notes below.
Yeah. So it's what do we need for tools and culture and all of the things to improve the experience of being a developer and make you more productive and efficient, because most companies spend more on engineering comp than anything else.
So today you've been focused on this, thinking about it and working on it both at Adobe, but also to some extent at Google. What do you know today about developer productivity? How does this relate to AI, coding tool output, metrics, all of those things? What is it that we can measure? What is it worth measuring? And what are things that are hard to know?
Yeah. We've taken it as axiomatic that it is hard, if not impossible, to measure productivity because what are the units even? I had, a while back, a sort of clever idea. And I use that as a loaded term. I had this idea that if we don't know what the units are, economics has prior art for this, right? You have an index metric where you measure all of the things that might influence this and you come up with a way to effectively basket those together. And then you have a unitless but directionally useful and useful over time number that you can sort of pin to.
And my theory had been to measure all of the things that we think are positive or negative indicators of something going well. Releases, test coverage, number of PRs, everything, right? Just cast a wide net. My initial formulation on this was the wisdom of the crowds idea, right? No individual can estimate how many jelly beans are in the jar, but you get 100 people and you'll get real close. So we do a "Am I hot or not?" of which would you prefer? One release per week more or X tests, right? And you run a bunch of iterations of that game with your relatively experienced engineers.
Yeah.
To work out the linear trade-offs between all of those factors.
To figure out what matters more.
Right? To figure out what that is. You probably want to dedupe some things because they're just going to be the same stat in a different package. But find the ones that have some nonlinearity and then mush that all together. Now you have something that sort of gives you an aggregate, this is kind of our productivity number. And Adobe is doing something along these lines.
Slightly smarter than my initial formulation. But then you can watch that track up or down over time for a given population. What I've realized is that won't capture everything in this post-AI boom. And the reason that I came to that was a paper that I put out, the develop, deploy, operate paper recently, where one of the points in there is there's four forms of impact in tech.
There's product success, the stuff everyone talks about and everyone else is better at than I am. That's not my job.
Whether or not people buy a product or use a product, whatever.
And net promoter score, revenue, all of the stuff, right? There's compute efficiency, right? How much are we blowing on AWS?
Right. Or if you're shipping sensors and all that, I know that all too well.
Right. Or even the Android team has a trade-off for if you can reduce the size of an application download by a megabyte—
It makes a difference for many metrics, yes.
Right. And so all of those things, that is a form of impact. Then there's engineering productivity, or we phrased it as engineering capacity, because if we save you 5% productivity gains, capacity gains, you don't get to spend that somewhere else. You just have more capacity on doing more engineering.
Or having slack time for innovation or picking up when something happens, that kind of stuff.
But they're not dollar fungible. You don't get money in a different account. And then the fourth form of impact, and this ties back to the AI thing. Fourth form of impact is changing the game, right? Strategic innovation. Research makes impossible things possible.
Oh, yeah.
Security makes incredibly bad things impossible. Right? But if you change the nature of what we are doing, the cost structure, etc., then that's a different game.
And the problem that I think that we see with AI, even with a clever index metric approach to trying to capture the gestalt of productivity, if you're not looking for the value of a given change is substantially different, then there's no way you'll ever be able to capture that in the signal. Right? We don't have anything that says the value of a change of this size has changed.
So for instance, it is entirely possible at this point, and I hang out in security spaces and do a lot of low-level C++ library stuff, you can write a JPEG parser in a couple weeks.
Yep.
Right? With whatever additional features you need. That would be insane to say that sentence out loud three years ago.
Okay, so what you're saying is what none of the measurements really capture is the possibility, the impact, the effort it is to do a massive innovation, which right now with AI is happening. And I'll give you also one example, and tell me if this is what you mean with it. There's a friend who works as a CTO at a mid-size engineering company. They're a niche industry.
And this CTO, they have been looking at release frequency, all the usual metrics, talk with their engineering team, are they happy, are they not? And of course now they're adopting AI as well. They're seeing that people are more efficient. And just privately this person was telling me, "Look, I kind of know for a fact that we could probably do the same amount of work with half the amount of people, and then we could be cash flow positive. But of course there's a morale thing." He likes the people. So he's like, "I'm not sure I want the emotional baggage on me. But we can see some people are picking it up faster and some slower." And we kind of left it at that.
A few weeks later I'm talking with this person and says, "You wouldn't believe, but we're probably going to be cash flow positive." I'm like, "How? Did you go ahead with the cut?" He's like, "No. What happened is one of our engineers came to me and said, 'We license this component.'" Think of a library. It's a binary, but that was written by a vendor in the space that has a high-performance thing that they use in their software. It's a core part of it. And they have a revenue share where they pay, I don't know, 5, 10, 15% of their revenue for this component. It's so important. It's obviously licensed, closed source, but it's really important, and it would take them years of a five-person or ten-person team to write it, and it might or might not work. So it's not worth it.
The engineer came and said, "What if we just gave the specs?" Because they have massive amounts of tests around this thing, a test harness. They gave it to AI. AI wrote a component for them which is faster and better than what they have, no prior domain knowledge. And at first they thought, "Well, maybe we'll open source it," but then they told the supplier, and the supplier said—it's a huge business for them—they're paying them to not open source it and to give them the implementation, which they can re-license. And suddenly the game was completely changed.
If this person would have gone ahead with, let's say, the layoffs, this engineer might have just got a, and no one might have known that the game could be changed because of this.
AI makes crazy things sometimes possible.
Right. Exactly.
And so I think that this is the insight that tries to bridge the gap between, I have not seen a research-based number that says productivity post-AI is up more than 20 or 30%. But then everyone talks about these 10x sort of experiences. I super don't think it's actually 10x, but—
Yeah.
But how do you bridge that gap? And I think it's exactly this thing of a lot of the value of AI is not in doing the same thing faster. It's doing a harder, more valuable thing that you wouldn't have even thought about. And if your productivity measures are the logical equivalent of counting how many widgets are coming out of the factory, and then you start producing super gold-plated deluxe widgets, there's no way to measure that or track that over time because we just don't have the tracking of what's the value of this component or feature.
And that also matches with talking with senior engineers that are like, "Yes, we're up 70% on amount of code, we're up 30% on outages, number of features is the same." The thing that we can't know for sure is, are the value of those features up? Because how do you do that?
One challenge that AI is giving is entering the market for new graduates. There now seems to be this expectation. Companies are very eager to hire senior engineers who know how to use AI, and they have taste and judgment, and we have these things thrown around. It's not deliberate, but whenever I talk to the AI labs, they always tell me increasingly that they look for taste, judgment, agency, these kinds of things.
Let's go back to education, to computer science education. What are your views of how today's computer science education works, especially for how we know the industry is changing? And what are some of your ideas of what could possibly be changed?
I will riff a little on my bona fides there to add some weight to this. So, after the book, I got recruited into the ACM, IEEE, AAAI work on setting the curriculum requirements for an undergraduate CS degree. So this is the professional societies saying, "These are the things that we need to be doing." And for the steering committee for that work, I was on the steering committee, I was the only currently working engineer. Everyone else is academic.
Okay.
Yeah, in some sense, because it's what can we teach? And I managed mostly the software engineering requirements. But it does mean that I see exactly what the requirements are and have been living in that space for years.
And bluntly, and I gave a talk on this in a pre-AI keynote, most of the CS curriculum hasn't substantially changed since 2000, maybe earlier. Right? The average thing in the curriculum is exactly what was needed to build a database or an operating system in the '90s.
That doesn't sound good, does it?
Not for a field that prides itself on innovation, right? And there's all of these things where I have to genuinely ask, are we teaching this because it is relevant and important, or are we teaching this because it is tradition? And the history of—
Or we know how to teach it.
Right. Or we know how to teach it. Because the history of software engineering or CS departments, we're only at 50, 60, 70 years for the earliest CS departments. Right? This is a field that is way, way too young to be doing stuff out of tradition. Even pre-AI, I was like, I think a lot of this needs to be burned down because all of the curriculum requirements, all of the teaching has mostly been through accretion. We just throw more stuff in there and cut the amount of depth on the existing stuff by epsilon.
Yeah.
Right? And I don't think that works. Even pre-AI, I was saying the thing that I really want is you spend more time, practice and drill, and actually become a fluent programmer, algorithmic thinker, not you have a trivia understanding of a million different topics.
And now AI, right? Yeah. We guarantee you can do all of an undergraduate degree without ever thinking if you can just feed it into an AI, because it will do just fine on all of those programs and assignments. And so a lot of what I'm pushing for is we need to get out of the code. We need to up-level substantially.
You need to be able to read code and think as a systems thinker and algorithmically understand the structure and execution, but no one needs to implement a linked list. And we probably shouldn't even teach linked lists. It's way too slow, right? Memory is a slow thing. The more that we can refocus on design specification, precision in saying what you want to happen, and then testing, validation, verification, making sure that the AI actually did the stuff that you needed to happen. Right? If we can teach those things, then it's suddenly perfectly relevant for professionals.
And I think the big question is, will that be CS, or are we looking at CS enrollment is going to plummet, but software engineering programs are going to kick up? Because let's be honest, the stuff that's going to matter in an AI era and still get a job is teamwork, communication, design, specification, testing, validation, right?
Oh, that's a bunch of software engineering.
And also building production software, especially with AI. There's no—I was talking with Peter Mattis, co-founder, CockroachDB, Cockroach Labs, and he was telling me that during his college, on the side, he built the—
Mhm.
Yeah.
And he got a lot of opportunities afterwards, and then he built GTK+. And I was thinking, hang on, he did that on the side next to a CS degree. Would you not want to just throw aside computer science? I did computer science as well, but for a software engineering degree, would you not want to start on day one, especially with AI? People come in here and on one class you build one software, and the other you build software with a team. On the third one you extend the software. On the fourth one—because my experience being in a computer science thing, we did a bunch of theory. My favorite class was the one team-based class where we actually built software in the four years. There was only one.
Normal.
And then I took an extra class to actually learn .NET. And then when I got into the real world, it felt I was starting from scratch. I was thinking of all of these things which were not taught at the time, these days are, but the practice was missing. I had the theory, but I definitely did not have the reps.
Right. And I would argue that the LeetCode interview was because of this, because, okay, how do you find signal that someone has put in the reps? Right? It's the students that were playing in the programming contests, right? Those are the ones that had enough time on task.
That was the only reps that were kind of a little bit—
Common.
That were open to engineers who didn't have software engineering expertise. Sorry, yeah, people with affinity for programming who did not have software engineering expertise.
Exactly, right? And so it made perfect sense from that perspective for us to ask the sort of LeetCode questions in the programming contest styles, not because we needed solutions to yet another breadth-first search problem or reversing a linked list or any of that nonsense. But specifically because it was a proxy for you've been in an environment where you put in the reps. Whereas the amount of actual programming time on task for an average degree is pretty modest. And you go back to 10,000 hours to mastery in Malcolm Gladwell. I think that's in Gladwell. And where are you getting the time?
And if I think back of the boot camps that have spun up from the early 2010s, they started to get really strong 2014 to 2017. What they did is that three to six months, sometimes 12 months, but usually three to six months, they took people who had no prior computer science or programming experience. They started day one, let's build an application, which was typically a front end and a back end. They gave them the basics. It wasn't all that much, but that was three intensive months or six intensive months of building a working web application from the ground up.
Yeah.
And there was such a huge demand for software engineers that they got hired as a junior engineer level or almost at an apprentice level. But what they knew compared to computer science grads, they knew version control, they knew Git, they knew pull requests, they knew a database, they knew a front end, they knew Chrome Developer Tools. So they actually had a bunch of valuable skills in only three months that the people coming out from programming in four years might or might not have had. It was more of a toss.
Yeah, exactly. I've done the math. The number of hours in a CS degree is roughly equivalent to 15 months on average of full-time employment. And that includes all of the theory stuff, all of the things that are questionably relevant, and is similarly trying to take you from no experience to employable in either an industry or academic direction. And that is compared to a six-month boot camp, right? That's not a huge gap. Especially when one of those is laser-focused on these are all things that are just immediately relevant. And one of them is, here's some stuff that'll hopefully teach you how to think about these problems, maybe, or less charitably, this was useful when I was a student.
Also, can we just mention some of the most successful schools were actually outside of the US? A lot of the schools are, of course, the usual suspects are very highly regarded, from the MITs, the Stanfords and so on. But Waterloo in Canada, I was very surprised to see that a bunch of Silicon Valley companies have a high number of Waterloo grads. And I asked this on social media, what's up with Waterloo? They're up in Canada. Why is everyone hiring from there?
So it turns out that Waterloo has this program where, yeah, they teach you computer science or software engineering, I'm not sure which one. But the main part is that they have dedicated placements. By the time someone graduates, they will have had four placements, internships, at different companies. And because they have the relationships, this will often be the Shopifys, the Metas, this. So you will get an engineer who, three months times four, that's a year working inside of these companies.
Yeah. And that is the value.
Yeah, exactly.
It's not, you can probably
Throw away the three—not throw away, but that is more valuable itself, if you ask me. I don't want to downplay the importance of theory, by the way. But from a hiring manager point of view, oh yeah, I'll take that because that person will be up to speed. Wow, they work in different environments. They might even bring some ideas from day one to us.
Yeah. I'm incredibly jealous of—I don't think it's just Waterloo. A lot of the Canadian schools are—
Just one example, by the way.
Yeah, but a lot of the Canadian schools for CS have substantially stronger programs on internships and co-ops, in a way that a lot of those degrees run five years now. But I think it is a much stronger educational experience because there are some of those lessons—I will fully admit, there's stuff we did not put into the software engineering requirements because there's just no way to do that in the academic setting. There's some lessons you're going to have to learn the real way, in the field.
Even though you gave the example of doing a project, I want us to do more projects, but I'm really nervous about there being one project course in most degrees because you know what no team has ever looked like? A bunch of people with exactly the same level of background. It also gives the wrong taste about this is what teamwork is. I'll take it, it's better than not doing it, but there, yeah.
Well, we have what we have, and this system will hopefully, with people like yourself and others, start to also change or just adapt to the times. But AI is right now here, and one big question more experienced engineers are thinking about: we've learned to build software pre-AI. We know the fundamentals. Many of us went through computer science, so we actually have a very strong theoretical understanding, which we might or might not use some parts of.
But what about for new grads? Now that they'll enter the workforce of just today with just the capability that we have, what do you think the fundamentals are that they should focus on, and what are the things where they should just rely on AI? There's always the question of, should you just ask AI for this or should you learn the basics, right?
Yeah. I think the 'learn the basics' is an entirely loaded question because nobody agrees on what the basics are.
And that's fair.
Kent Beck said a couple months ago when we were last together something like, everyone that's currently working has grown up in an era where there was a playbook. We know, at least in theory, the right way to do this. And there's no playbook right now. So we all get a little comfortable of not knowing the answer, and we certainly don't know the answer on what is still the basics.
The things that I think that the juniors should be focusing on: one, the resilience to avoid just copying your ticket into AI prompts and then trying to check in that result. If that is all that you are doing, you can be replaced with a very short off-the-shelf script. That is not a defensible advantage for you being on staff. I think it's really important to use AI to better yourself and to widen your inputs, but not to immediately produce all of your outputs.
There's a metaphor that I really liked from about a year ago at a think tank. We called it Andrew's Axe. And it's, if I go chop down a tree with an axe, I do not say that it was the axe that chopped down the tree. It's me that chopped down the tree. The AI, as much as we occasionally anthropomorphize it, is still a tool.
And, 1979 IBM training manual: a computer will never be held accountable, therefore a computer should not be making management decisions. The AI is still not accountable. The AI labs are not going to indemnify us for the mistakes that these things produce. So you have to be able to stand up for whatever your name is on for those pull requests. And that requires a lot more rigor and not cutting those corners. That attention to detail, I think, is really hard to train.
Going back to the undergraduate version of this, the scary story that I've been hearing from a lot of profs is the best students, the B, B-plus and above students, they are better than they ever have been.
Mhm.
Because instead of relying on a TA that is available a couple hours a week and is one or two grades above them, they have an infinitely patient, always available, incredibly well-read, sometimes hallucinatory tutor right at the drop of a hat. They can always learn and explain and figure out—
And they actually have the autonomy and the drive to use this.
Right. And the metacognitive resources to avoid that, 'I'm just going to copy and paste.' The very worst students are somewhat better because they used to not be able to do anything, but now they can produce slop.
Yeah, they're just throwing slop, yeah.
Right. Which is a bit hard to detect, but I get what—
Right. But everyone in the middle has kind of collapsed. They don't have that resilience. They would rather go to the party or play video games than do the homework. And so they're like, 'Well, just this one time I'll just get in the habit of doing this thing.' And that slippery slope just gets worse.
And so there are a lot of profs from decent schools that are like, 'Yeah, I don't know, man. I'm not sure that anyone below—the vast middle, everyone below a B—might not understand how code works at all.' The system is just not set up to be tracking that sort of thing.
Or at least giving them feedback, for example, alerting, those kind of things, yeah.
Right. And so a lot of the discourse initially was, how do we prevent students from using these AI tools? And I think that that's nonsense because it's echoes of, we got high-level languages and there's compilers. 'But now you're not going to learn how to program in assembly,' right? The industry is using it anyway.
Right. They're going to be asked to use—they're going to be forced, pressured to use it harder in some places when they get a job.
Well, it will for sure be a red flag if a new grad says, 'I've not used these tools.' Come on. Have you not—
Yeah.
Do you not want to work in this industry? It's an open secret what we're doing. We're publishing it on blogs and everywhere.
Yeah, exactly. And so the line of, 'Well, we're going to never let you use these tools,' cannot possibly hold. I co-published a piece in Communications of the ACM with a bunch of academics this month. I think the opening line there is, imagine forward to 2040, and we hold the line treating all use of AI as cheating, but a generation of students, they've grown up with AI in the same way that the internet natives have grown up with the network.
If we keep continuing asking them these traditional questions about, find a palindrome in this string or measure the volume of rainfall in this graph, are they going to learn anything from a traditional CS degree? And I think the answer is obviously no. So if we know that we cannot stay here, and there's plenty of things that need fixing, let's burn it down and do something that matters and figure out how to use this appropriately.
And I think finding ways to assume that they're using AI, and what are the new learning objectives, and how do you make that still frictionful in the ways that cause them to think and learn about different things, almost certainly at a higher level. But that's actually the key to undergraduate education and curriculum design and assessment.
It sounds like the professors, the TAs, the people who are designing these courses, they also need to put in a bunch of work because no one has this figured out. You need to experiment. You need to first use these tools yourself.
Yep.
You need to come up with novel ways of creating an assignment which you need to use AI to solve, but AI itself is not enough. How do you get the feedback? You might need to put in more time. Let's say, sit down with someone afterwards and have them explain to you because, again, interviews are also changing.
And one of the interviews that seem to be working well is when you're pairing with someone, video or in person, you're using AI, but you're explaining what you're doing and how. Because now you cannot just fully fake it and go away, give it to a friend or just have Claude do it. You're now showing your cognitive ability. The interviewer can push you into uncomfortable—they'll get a gauge of how expert you are and push you further and further and so on.
Yeah, exactly. And at the undergraduate level, there is some move towards more oral exams and those sorts of things. But also, I think a really simple solution is stop grading just the outcome. Focus on the process. Even if you don't come up with wildly harder assignments for assessment, stop grading just the code that they turned in. Grade instead, what are the prompts that you used to get here? Also, what are the tests that you provided? How did you make sure that this thing is actually doing what you need it to do?
And when you start focusing on the process that got you there instead of just the outcome, I think not only CS, but most disciplines—we've been using, 'Go write a new essay on Catcher in the Rye.' We don't need another essay on Catcher in the Rye. 'Go write a new linked list.' We don't need another linked list. We've been using the artifact that's produced from that as a proxy for, you put in the time and the struggle and faced the friction that we assume that you now have learned something. It might be more appropriate to just focus on, do we have evidence that you struggled and learned? Turn in the whole package.
Curriculums are designed by committees, which is for a good reason, but if you had Titus's software engineering school where you could yourself put together the curriculum and study course for an undergrad course of, let's say, three to four years, how would it look right now? And we're talking about ideally having engineers who can hit the ground running at software and engineering companies, obviously using AI tools, building software, that kind of stuff.
Yeah. So I think the first course, the first semester, starts unaided by AI. We need to get you through at least the level that you can read code and have a vague idea of what's going on, can understand control flow and those sorts of things. I think there's some really interesting things on well-scaffolded and harnessed AI for educational purposes of, we're intentionally not just going to give you the answer. And in conjunction with, if you're assessing the whole package and process of creation, then you don't need to worry about them going to ChatGPT and just getting the whole answer. So some of that's covered. I'll be honest, I don't know. Nobody knows. There is no playbook, but—
I want to get opinionated. I want to get opinionated.
So one of the ideas that I've had is, let's make CS zero, non-majors and majors alike, intro to programs. Specifically, how do you describe the inputs and outputs and structure and design of the interfaces? So we could talk about what's the difference between a checkbox and a radio button. When do you use a button versus some other UI widget? For non-UI things, what command-line flags, positional parameters? How do you interface these components? Specifically so that if that's the only class that you ever took, at least the AI vibe-coded slop that you produce for your own use cases is going to have a slightly more coherent, sensible design language and be more fit for purpose.
I live in New York. The condo building I'm in, there's an app for the laundry room. And it drives me insane because they use menu UI instead of a button. Just the design language is completely wrong. Start with that. And then graduate kind of along the lines of what you were saying, of we want to move up into projects and teamwork and things like that.
Try to motivate them into, in themselves, wanting to solve a bigger problem, and then needing some background on, 'Oh, this is how the web works. This is how a web remote API RPC call works.' You can't give people a solution to a problem they don't realize they have. And until they're trying to solve these problems, those solutions that we present in theory are all just trivia. They're just isolated, academic, not connected to anything. Motivate that by getting them into a situation where they need that as the answer first. And then you can provide that background.
I also desperately want more projects for two reasons. I don't want the assignments to be, there is a correct answer, because that's not how the industry works. Mostly not how research works. And I want them to grade—part of your grade is critique of others. Look at their outcomes or their code or both, and be able to say, 'That was great. Did you think about doing a different thing here?' And then also grade them on, can you give feedback without being a jerk? Because psychological safety and teamwork, right?
And all of this sounds like a lot of work, except you can use AI to do some of that analysis. Let's do the sentiment analysis. Was that rude? Things like that. And I think we could have a much more practical, hands-on and much more effective educational structure. And I worry that that's going to be software engineering departments and not CS departments.
And as closing, for grads right now, college students sitting inside of CS programs, software engineering programs, who are seeing, okay, it's more competitive to get into the industry, tools are changing, what advice would you give in terms of tactics, strategies, activities to do to maximize your chances of getting into the industry and then doing well there?
This might be risky, but most of my success in life has come from not doing the thing that everyone else is doing. It is really incredibly hard to stand out if you are just listening to what is a hot topic and chasing the same shiny thing that literally everyone else is doing.
And so I think it is, to thine own self be true. Find something that you are actually interested in and put in the time doing something that you care about, especially if it's weirder than what everyone else around you is doing. Because then you will have a novel story to tell your interviewer. You will be worth talking to at a cocktail party, so that the startup founder is like, 'Yeah, I want to take a risk on this kid.' And the more time that you put into these things, the more expertise that you can bring to it, the better off you're going to be.
I have a worry that with AI-assisted engineering, our emphasis on raw engineering skill is dropping. You can get by with a lot less of that depth. But the only way to replace that is you need to understand the problem domain that you're operating in to a much deeper level. And I think that that might actually lead to less mobility in industry. It's not, 'I'm a C++ engineer' or 'I'm a back-end engineer.' It's, 'I'm an engineer in embedded audio.' You have to understand those problem domains to a much deeper level, and that's your actual advantage.
As a closing, what's a book that you would recommend and why, outside of your excellent book, of course?
I mentioned it earlier, but I'll stick with it. Psychology of Software Teams by Cat Hicks. Especially, there are a couple chapters in that that, if I could have that downloaded into the brains of every leader around, I would be so happy. The brains in jars analogy is—
Haunting. So, that one's a really good one.
Well, Titus, thank you so much for this conversation.
Thank you so much for having me. This has been a lot of fun.
I'm glad I got to finally sit down with Titus, and this conversation did not disappoint. To start with, I knew that Google's codebase was large, but I had no idea we're talking about 250 million lines of just C++ code. And this was all pre-AI, so we're not talking about overly verbose codebases generated by AI. As comparison, Uber's mobile apps had codebases of about 1 to 2 million lines of iOS and Android code back in 2016, and Uber crammed a lot of business logic into those apps.
How Titus and the team cleaned up these foundations without using AI was also pretty educational. They built deterministic code generation tools to parse the code, match patterns, then generate edits. This is an approach that still could work more reliably even today than just asking a non-deterministic AI to refactor a codebase, even though it is arguably more difficult to write advanced parsing logic on a codebase like Titus and team did.
I also appreciated Titus's view on where AI is helpful today. He mentioned this quote he heard from a senior engineer that said, "We are up 70% on the amount of code, but we're also up 30% on outages. The number of features is the same." Well, that doesn't sound all that good. Where's the value from AI then?
Titus says that we won't get all that much value from doing the same stuff we did before AI, but faster. The value is to be more ambitious and do harder things that we would never have attempted before AI.
Finally, I hear a lot of worry for new grads in tech about how they'll get a job or not, especially when AI agents are so commonly used, but CS programs have not incorporated them into their program. Titus is one of the few engineers who has been part of the steering committee for the CS curriculum and says that this program needs to either change substantially and be more practice-oriented or CS enrollment will plummet.
Titus also says that it might make sense to finally have a software engineering curriculum that is focused on students getting far more experience building software with these modern tools. I don't think any of us has an answer to what will work, but having more people from the industry get more involved in refreshing the CS software engineering curriculum sounds like a step in the right direction to me.
Check out the show notes for related The Pragmatic Engineer deep dives on Google's engineering culture, measuring developer productivity, and the impact of AI in software engineering.
If you enjoyed this podcast, I'd greatly appreciate it if you left a rating on it and perhaps a review. Thanks a lot and see you in the next one.
Article published · Updated
