Casey Muratori on Why Software Performance Matters and Why the Industry Keeps Ignoring It
The Pragmatic Engineer
Casey Muratori is a programmer and game developer, founder of Molly Rocket, and author of the Substack Computer, Enhance, which focuses almost entirely on software performance. His argument, repeated throughout this conversation, is that most software runs 10 to 100 times slower than it should. That gap, he says, usually comes less from missed micro-optimizations than from architectural decisions made by people who never learned what the hardware can actually do. The conversation covers why the industry has neglected performance and why he thinks that is starting to change. It also covers how optimization should be done, why every programmer should learn to read assembly, his critiques of "clean code" and test-driven development, how game development has changed, and why he writes all his code by hand.
From Digital Equipment Corporation to games on Windows
Muratori learned to program at seven, around 1982. His father was a programmer at Digital Equipment Corporation, the minicomputer maker behind machines like the PDP-11 and VAX. Parts of DEC were later absorbed by Intel and Compaq, and his father ended up at Intel without ever changing jobs. Because of that, the household always had computers at a time when few families did, and, more importantly in Muratori's telling, there was a programmer at home to teach him.
He got into games by chance, through an internship at Microsoft. At the time, Windows was not a gaming platform. It had little beyond Solitaire and Minesweeper. Muratori explains the technical reason. Games needed to fill pixels quickly with the CPU, draw to a back buffer, and flip it to the screen. Windows 3.x offered no fast way to do that. Programs had to hand Windows a bitmap that might not match the display format, and Windows then translated it. Games like Doom were not coming to Windows.
Chris Hecker, whom Muratori calls an unsung hero of Windows gaming, set out to change this with WinG, a library for fast blits to the screen. Hecker had no authority to build it. It was a skunkworks project inside Advanced Technology, an early precursor to Microsoft Research that was not supposed to ship core Windows libraries. Hecker's manager, Michael Edwards, provided cover. WinG shipped anyway, and Muratori describes it as the first step toward DirectX. He adds that the institutional push for DirectX came from three people, and one of them had been the tester on WinG, so the lineage was direct. Many programmers, such as Todd Laney, did essential core work on it.
Muratori arrived for his internship the week WinG "blew up" internally. Edwards, who was supposed to be his manager, had stormed out of the building and was not seen for about a week. Edwards later returned and was moved to another division. The upshot was that Muratori landed at "ground zero" of games on Windows. He met Hecker, who took him to Humongous Entertainment to meet Ron Gilbert, creator of the SCUMM engine and one of Muratori's childhood heroes. Gilbert gave him a Secret of Monkey Island mouse pad.
After that came an uneventful startup with Hecker, then Gas Powered Games (Dungeon Siege), then a long stretch at RAD Game Tools. There he built a character animation system that was widely used and, to his surprise, is still in some studios' pipelines through source licenses, even though he has not touched it since 2004. Since then he has worked independently through Molly Rocket: contract work, the Substack, and some game work, including the movement system for The Witness, which he did as a friend helping out on a big project. Molly Rocket also has an unannounced project. They are keeping it quiet because the Substack comes first and they cannot predict how much time they can give it.
Why the industry neglects performance and why that may be changing
About three years ago Muratori messaged the host to ask why the industry puts so little emphasis on performance when the evidence says it affects the bottom line. Looking back, he gives three explanations.
The first is the host's original answer, which Muratori accepts for part of the market: in enterprise software, the user is often not the buyer. Someone high up picks HR software based on cost, compliance terms, and legal liability. Whether each record takes 30 seconds to open never appears on that sheet. The people who suffer from slow software often have no say over it.
The second is monopoly and network effects. Muratori points to Bluesky and Threads trying to challenge X, and to the entrenched positions of Facebook, Instagram, and TikTok. Better responsiveness might be one part of a pitch against an incumbent, but without a plan for adoption and influence, it cannot sell a product into a monopoly space. Downloadable apps where users choose freely are a shrinking share of all software.
The third is more hopeful. Muratori thinks a decade of people, himself included, pointing at the problem has had an effect. He sees more discussion of performance, more published benchmarks, and new products that lead with performance against incumbents, naming File Pilot and the Blick video editor. The host adds Bun versus npm, where claims of being 10x or more faster got developers' attention, and Linear versus Jira, where Linear advertises a 300 millisecond budget per action.
Muratori says that 300 millisecond number shows how low the bar has fallen. Three hundred milliseconds is "an eternity in computing." Users often wait seconds for simple operations, even though network round trips to a physically distant data center can finish in under 10 milliseconds. When he says software is 10x to 100x slower than it should be, people don't believe it, but he argues there is plenty of proof.
Optimization starts from the theoretical maximum
The host brings up Simon Eskildsen's "napkin math" project, a list of baseline costs for operations such as transferring data between data centers or writing to an NVMe drive. According to the host, Eskildsen noticed that when Shopify teams compared database vendors with their own benchmarks, the results sometimes made no physical sense. A write that should take around 100 milliseconds at the theoretical limit was measuring 10 seconds, and it often turned out the benchmark was wrong.
Muratori says this is the core point of his Substack. The common idea of optimization is to run a profiler, find the big parts, change something, gather statistics, and keep the change if the numbers improve. He says that is "not how it is done" by any of the many excellent optimization people he has worked with. The right approach is to first ask what operations the system must perform and what the hardware can do at its theoretical peak. You then measure the gap between that peak and what you actually get, and your job is to shrink the gap until what remains can plausibly be explained. You often can't reach the theoretical number, which is why it is called theoretical.
Without that reference point, he argues, you are only finding a local minimum. He calls that "improvement," not optimization, because optimization means making something optimal. The theoretical baseline also helps you learn. Hardware keeps changing, and a big unexplained gap can point to something nobody knew about. He says this has happened while producing the Substack: they found a register-renaming behavior in recent Intel chips (he mentions a "RAT table") that nobody had documented and that now had to go into their performance model. Back-of-the-envelope estimates are how you discover such things.
Why you should learn to read assembly
Muratori's course teaches reading assembly language, and he explains why. Source code in Java, C, Haskell, OCaml, or Rust is only input to a compiler. It tells you nothing certain about what the CPU is asked to do. The assembly output tells you exactly. He stresses that the skill is reading, not writing. Writing assembly is rarely needed beyond occasional test cases where it is easier than coaxing a compiler into producing a specific sequence. Reading it, he says, is essential for optimization work.
It also opens up other knowledge. When a vendor presents a diagram of a new core, someone who reads assembly can see things like peak multiplication throughput right off the chart. Without that background the diagram looks like a meaningless flowchart.
The host notes that assembly is much simpler than high-level languages, and Muratori agrees strongly. To understand a modern website you need the JavaScript syntax and libraries, the DOM, CSS, and React. For assembly, he estimates you might need to learn 20 or 30 instructions, because compilers rarely emit most of the legacy x64 instruction set. And in performance work you usually look only at a small piece of code you already suspect. His summary: "If you can vertically center a div in HTML, then you can probably learn assembly language."
"Premature optimization" and the architecture you can't fix later
Playing devil's advocate, the host brings up "premature optimization is the root of all evil" and the usual practice of building first and optimizing only if needed. Muratori has a two-hour lecture on the history of that phrase from this year's Better Software Conference. Here he addresses the idea behind it, which he says is not entirely false.
Deferring optimization works when the slow code is local. If you wrote a naive loop, or a simple hash table where you could spend a week researching the fastest one, and you understand the problem well enough to know the surrounding architecture won't need to change, deferring is sound engineering. Maybe the naive version is always fast enough. If not, someone can target that one spot later.
It fails when you don't know whether your choices leave the code optimizable. His example is a codebase where every operation asks the server for something, waits, computes, and then asks for the next thing. Hundreds of thousands or millions of lines get written that way. When it turns out to be too slow, the performance experts who are called in can only say there is nothing they can do. The code is a serial dependency chain: each step waits for the one before it, so it can't be parallelized, multithreaded, widened, or amortized. Overall performance is generally bounded by the longest such chain. If the team had instead been told to request everything an operation might need at the start, and to chain requests only when truly unavoidable, the problem would not exist. Fixing it after the fact means rewriting nearly everything, if that is even possible.
His conclusion is that a codebase with a few hotspots that can be optimized later no longer happens by accident. It has to be designed that way up front. Everyone making architectural decisions must understand performance so that the people downstream inherit an architecture that can be optimized. Otherwise, "you're just rolling the dice."
As evidence, he points to the many company blog posts, from Facebook, Uber, and others, about rewriting entire systems for performance. If performance problems were always hotspots, full rewrites would never be needed. The host describes watching teams at Uber move from Python and Node.js to Go and Java. According to the host, OpenAI and Anthropic are now doing something similar, moving services first built in Python because their ML staff knew it toward Rust (or possibly Go) to get multithreading and more connections per machine. The host adds that public engineering blogs tend to describe these moves in flattering terms, since they help people get promoted and may be shaped by content teams. Muratori replies that the fact these rewrites keep happening is the signal. If the conventional wisdom were right, the only reasons to rewrite in another language would be preference or something like memory safety, not performance.
A learning path: much less than people expect
Asked how an engineer should get better at writing fast software, Muratori starts with good news. Heavy hotspot optimization, like hand-coding routines in assembly, is rarely needed today. Many libraries are already optimized, and modern CPUs are very good at running bad code quickly. The real task is avoiding the choices that make software 100x slower than it needs to be. If you avoid those, he says, you'll typically be within about 2x of optimal, "which is 50x better than the people who are 100x away."
So what you need is one solid round of experience: learn to read assembly, see how the CPU works, time some code, and experiment. He suggests "a month or two of nights." One of the first things his course shows is the assembly executed for a + b in Python. It is so long he has to skip most of it, while the C equivalent is a single add instruction.
The host asks whether a React or iOS developer would ever look at assembly in their daily work. Muratori says the point is to internalize the orders of magnitude. Once you know that an add in Python costs roughly a hundred times more instructions than in C, it makes sense why Python programs lean on libraries written in C, and why operations over large amounts of data can't be written in plain Python. With that knowledge you can decide whether a piece of code can afford a 100x slowdown, and if it can't, reach for a library or something like Cython and structure your code around it, so the 100x cost applies only to rare operations. Most programmers can make that call without being performance experts. The key is knowing the question exists. After that, a quick search, or asking an AI, will turn up the specifics.
What to understand about the CPU
Muratori clarifies that assembly is really a way to see what the CPU is doing, and the CPU is what matters. Modern cores such as Apple's M series, AMD's Zen, and Intel's Core line are not documented at the level of every internal mechanism, and they are too complex to study that way anyway. Treated as black boxes, though, they can be understood through a few categories.
The first is how data moves into and out of a core: load and store units, the cache hierarchy (L1, L2, L3, and now sometimes L0), and its granularity and policies. This matters because data layout and access patterns can change performance enormously, and those are architectural decisions that are hard to undo. The second is how instructions flow through the core, which covers concepts like branch misprediction and instruction cache misses. He says these are simple to understand conceptually even as predictors get more sophisticated. The third is execution scheduling: the raw throughput of floating-point multiplies, integer adds, divisions, and so on, and how assembly instructions become micro-ops distributed across execution units. With that background, he says, you can look at the diagram for a new core and roughly know its performance.
He doesn't think everyone needs to become an extreme optimizer. It is fun if you enjoy it, but not the important part. He argues this baseline should be standard for software engineers: "You go to school for four years to learn this. There's no reason you can't learn this in a few months."
The host brings up craftsmanship, and Muratori agrees that many programmers find high-level work unfulfilling and feel much more satisfied once they understand what is happening underneath, even if they keep working at the same level. He adds that it compounds. If library maintainers take performance seriously, everyone using those libraries gets faster, and APIs designed to be optimizable spread that benefit further: "The more people are doing performance, the less people need to do performance." The knowledge also stays current fairly easily, because CPU vendors present their changes and people running microbenchmarks publish the quirks they find.
How games used to get built
Muratori warns that his view of game development is dated. The biggest revenue earners today, including Fortnite, Roblox, GTA Online, and Minecraft, are live services that ship features incrementally, much like SaaS. He has friends at such companies but has not worked at one.
He can speak to the earlier era. Back then there were no licensable engines. Code reuse was informal, like borrowing a routine from a colleague at Atari. The first wider reuse came with id Software's Doom and Quake engines and Ken Silverman's Build engine, but games built on them tended to closely resemble the original game. A studio's existing codebase was a competitive asset. Blizzard could carry its Warcraft work into Warcraft II, while any competitor had to build pathfinding, level editors, and rendering from scratch.
Projects faced two big risks. One was engine risk: could the team build technology that did what the game needed, fast enough to develop on? Developers couldn't buy much faster hardware than consumers would have. Some bought SGI workstations for exactly that reason. There was no way to reduce this risk except grit. Some games failed on technology alone, and some studios survived on technical skill, like id with first-person engines and Bullfrog with the pseudo-3D engine behind Magic Carpet and Dungeon Keeper.
The other risk was whether the game would be any good. Without a working engine or level tools, it was very hard to know what you were making. Muratori cites Thief: The Dark Project from Looking Glass. From what he heard from the team, its core gameplay only came together near the end. As budgets grew into the millions, studios moved to building a vertical slice first: the whole studio builds one playable, hacky slice as fast as possible, proves it is engaging, and only then plans asset production. Stressing that he is not a game historian, he believes this was a major shift that made development much less "seat of the pants."
Licensable engines as the games industry's "AI transition"
The host suggests that widely available engines like Unreal, Unity, and Godot may hint at what AI will do to software in general. Muratori calls this a brilliant analysis and says he has told people the same thing: licensable engines "kind of was our AI transition already," and "the news is not probably that positive."
Early on, the effects were good. People who could never have assembled a technical team could now make games, which enabled new artistic work and some excellent titles. But the flood of releases followed. Muratori estimates Steam now sees tens of thousands, perhaps a hundred thousand, releases a year. It used to be that a fun game would be found through word of mouth or storefront exposure. Now a game will almost never be found organically. Small hits with no marketing still happen, but "the chances that you will be that game are like zero." Without a real marketing strategy, he considers it unwise to expect more than a few thousand sales. The host sums up that quality is now table stakes and distribution is the differentiator. Muratori agrees, and says he doesn't know whether it was a good trade.
New games also compete with old ones. The host mentions spending hours on the '90s game Death Rally. Muratori explains why this is getting worse. Old games used to look obviously dated, and in 1995 technical advances drove sales. A large share of today's market by revenue no longer cares much about graphics beyond the current level, so a 2017 game "just looks fine." Live-service games like Fortnite, Minecraft, League of Legends, and Dota take up many players' hours, and entertainment time is zero-sum, shared with Netflix and everything else.
Why GTA 6 is taking so long
Asked how GTA 6 can take more than a decade despite better tools, Muratori says to look at it as a business decision. GTA 6 is not only a new game for players. It replaces GTA 5, which he believes was by far the highest-grossing entertainment product ever, with the online portion earning billions, "kind of like Fortnite before Fortnite." Rockstar and Take-Two are replacing their most profitable product, which is still earning, and the worst outcome would be a successor that cannibalizes it and then earns less. He's sure many people on the team care about the single-player experience artistically, but he assumes a lot of planning has gone into the live component.
In his view, GTA 5's online success was probably a surprise to the company, and Red Dead Redemption 2's online mode didn't reach the same level. That makes GTA 6 the first release where Rockstar knows the audience is there. He compares it to relaunching Google Search: "if I was in charge of that project, I would be sweating bullets." He doesn't predict whether it will succeed.
Critique of "clean code"
The host brings up Muratori's video "Clean Code, Horrible Performance." In it, a polymorphism-based design in the style Robert Martin recommends ran about 1.5x to 15x slower than a plain switch or table version. Muratori says the reaction was more positive than he expected, though it was certainly controversial.
He separates two meanings of "clean code." If it just means code you consider well written, there is nothing to argue about. His target was the specific rules: keep functions under a certain length, avoid knowing types at runtime, always prefer polymorphism. He calls these "just bad programming practices" when combined, though some are fine alone. Many small functions are fine, for example, if the compiler can see them and safely inline and merge them.
He says the problem isn't the cost of a virtual function call itself, which depends on things like branch prediction and stack traffic. The real cost is that runtime dispatch stops the compiler from optimizing. When functions are known at compile time, a modern optimizing compiler can merge small functions, remove redundant work, and even widen code paths for SIMD, turning mediocre source into efficient machine code. When it must allow for any class being substituted at runtime, it can do none of this, and he says markers like final don't reliably fix that. He believes the video's example showed a milder slowdown than a large production codebase built this way would. His position is that code can be maintainable and readable without following those rules, while still leaving the compiler room to do its job.
Testing should inform design, not drive it
On test-driven development, Muratori calls his position pragmatic. Tests are worth writing when their cost to create and maintain is less than the time they save, for example by catching bugs that would be hard to find or expensive once shipped. At RAD, he kept a regression tester for the core routines he wrote so he could cover the cases customers would hit.
What he objects to is the "driven" part. Development shouldn't be test-driven by default. A team might sensibly decide to drive a particular project mainly through tests, but for another project that could be a costly mistake. The cost of tests includes what they do to the codebase. If changes mean rewriting tests, teams may avoid changes they should make. Some projects should have many tests and some very few, and he says knowing which is which is part of being a good engineer.
What good code and good engineers look like
For Muratori, good code does what the machine needs to do to solve the problem as directly as possible. It is split into well-chosen, well-named pieces that others can follow, especially your future self. It avoids redundancy; his example is writing a Euclidean distance function once rather than repeating the formula everywhere. And its pieces can be recombined by the compiler into efficient code. He has never understood the idea that well-architected code and fast code are in conflict. In his experience, properly architected code is usually the fast code. Readability only suffers when you push very close to the theoretical maximum on specific hardware, which he says is rare. Code only becomes both hard to modify and slow "once you think you need to have 27 factories and 8,000 microservices." He wants good code to leave the path to theoretical performance open for anyone who needs to go further later.
On good engineers, he uses a baseball analogy: are you asking about a pitcher or a designated hitter? Some great engineers are utility infielders who quickly learn a messy codebase and patch what needs fixing. Others spend eight months exhausting one problem, sometimes to the point of inventing new algorithms. His advice to anyone building a team is to think about roles, not a generic "great software engineer."
Pressed for traits they share, he gives two. He can't think of a great engineer who couldn't read assembly, so curiosity about how things work seems common, though how much they use it varies. Some work at that level constantly. Others just avoid architectural mistakes because they know, for instance, that certain work has to be batched. The second trait is not being dogmatic about ideas they haven't tested themselves. He calls much received programming wisdom "nonsense" that no one has actually tested. Its concrete downsides can sometimes be demonstrated while its upsides often can't. He values engineers who look at what works in practice and can be measured repeatably, rather than following a talk because "someone at Google says always call memset or never use if statements."
Why Muratori doesn't code with AI
Asked how Molly Rocket uses AI tools on its unannounced project, Muratori says, "We are not using them at all." He presents it as a philosophical choice, not a productivity judgment. The project exists because they want to program it: "If I just wanted an AI to program them, I'd just go get the Unreal Engine." He says he did not reject AI over time savings, copyright, or ethics, though he calls those fair grounds for evaluating it. It simply doesn't serve the project's goals.
More broadly, he says nobody can predict what AI will look like in ten years. But he expects a handcraft tradition to survive, as it has with other automated work. IKEA exists, and someone in an industrial district still welds iron-and-wood tables that some people want. People still knit hats they could easily buy, some even raising the sheep themselves. He doesn't claim this has more or less value, only that humans do it. Given the choice, he would be "the organic farming guy" and has "no interest in managing a division at IKEA." He wants to be among the people who keep programming by hand alive, and he says he has little to add about AI coding workflows.
AI's effects so far: too early to judge
On AI in the games industry, Muratori says it is too early to judge. Some people have been saying AI writes human-quality code for two years, but the people whose opinions he trusts didn't consider it very usable until much more recently. He suggests waiting at least another six months to a year for best practices to settle. He knows many people in games are using AI, but he hasn't seen visible results, like Fortnite shipping weekly without bugs. In his view, there are several possible explanations. Current tools still need humans to set them up and fit them into their work, so adoption takes time. The models may need to improve further. Or real gains may already exist but be too small to see from outside. A 10% improvement across the board would be impressive but almost invisible. If AI turns out to be truly transformative, he says, it should become obvious, "five people are now shipping Fortnite instead of 5,000."
The host describes hearing from many engineers who feel AI fatigue or burnout. They are prompting instead of coding, under pressure to produce more, and wondering why they are there. Muratori says he has heard of such cases but hasn't encountered them directly, likely because most people he talks to have a lot of say in how they work. The host connects this to something Armin Ronacher told them: people with autonomy tend to welcome AI, while people handed tickets tend to see it as a threat. Muratori finds this logical. If you have autonomy, you use AI for tasks you didn't want to do, so even when it fails, the downside is small. If you are told to use AI for the work you wanted to do, maybe right after layoffs, the psychological effect is completely different. He suggests a question: "Are you using an AI to do your job, or is an AI using you to do your job?" He notes that reports about Meta, from the host and others, suggest employees felt they were mainly generating training data to replace themselves. He agrees that people who have options might weigh autonomy when choosing a job.
Closing advice: read papers
Asked for book recommendations, Muratori instead recommends reading research papers. Before working in an unfamiliar area, he reads many papers, follows their references, and uses survey papers to find more. He learns historical context and, often, techniques he had never heard of. He doesn't name a specific paper. He suggests searching Google Scholar for a topic in your own domain, reading one paper, and following the references. Since he doesn't use AI, he can't say from experience, but he guesses AI tools might be good at suggesting papers based on your interests.
There's a misconception about the way you approach optimization. You run a profile, you identify the big parts, you make some changes, you measure statistics. I've worked with many extremely good optimization people and that is not how it is done.
Assembly language is not all that complicated, right?
All of the JavaScript libraries, the DOM, CSS, React, assembly language, maybe there's 20, 30 instructions you might have to learn total. If you can vertically center a div in HTML, then you can probably learn assembly language, I would say.
How are you using AI tools, if you're using them at all?
We are not using them at all. The reason that I want to program things in a game is because I want to program them. If I just wanted an AI to program them, I'd just go get the Unreal [laughter] Engine.
What are things that you think are non-negotiable for someone to be a great software engineer?
So, I find there's a lot of received programming wisdom that's just nonsense. Clearly, no one's ever tested it. And so in order for it to be received wisdom, you should
Why do most devs not care about writing performant software and should we? Today's guest Casey Muratori spent the last decade arguing that we should. He also says that most software out there runs tens to 100 times slower than it needs to. [music] Today we discuss why the focus on performance took a backseat across the industry and why Casey thinks the tide is finally turning. Why you'll want to learn reading assembly if you're serious about high performance code and why it's less scary than it sounds. The saying premature optimization is the root of all evil, why Casey says that the majority of people use it to avoid thinking about performance when they really should and many more. If you want to get better at writing faster software [music] and become a better engineer while doing so, then this episode is for you. And if you're one of the people who hit a like on this post by Ryan asking for this episode to be more about programming than about AI, this episode is also for you.
This episode is presented by Antithesis. If you work with agents, your job is no longer just writing code. It's also specifying and testing it. [music] Antithesis is the most effective method of verifying agentic code today.
This episode is brought to you by Sentry. You probably already know what Sentry is because you're a developer. If not, just ask a dev and they'll tell you. I use Sentry to monitor the back end of The Pragmatic Engineer for any and all events and errors. Of course, Sentry doesn't only do errors. They also have logs, replays, spans, profiles, metrics, and more because they're all connected to the same trace. One new capability Sentry has I'm really liking is its ability to fix errors. Let me show you.
Here's a list of errors on my admin back end. There's a recent error on O that I want to check out. Let's have Seer run an autofix for us. Seer is Sentry's AI debugging tool. First, it generates a root cause analysis. It's finding some problem with HTTP versus HTTPS URLs. Cool. Now that we know what's going wrong, Seer can create a plan on how to go about fixing it. I could go and edit this plan, but I'm happy with it. So, let's create the actual code fix. Here's a code fix that Seer generated. Assuming it looks good, and in my case, it does. Let's draft the pull request. And boom, the PR is created, ready to merge. What I love about Autofix is how Sentry went from showing a list of errors inside my application to offering me a fast way to fix it and close the loop while I stay in charge of this bug fix the whole time. Debugging got a whole lot faster and a whole lot easier. Check out Sentry at sentry.io/pragmatic and start detecting errors, diagnosing your root causes, and fixing issues and regressions today.
All right, Casey, welcome to the podcast. It's so nice to have you here.
Thank you so much. It's great to be here. Thank you for the invitation.
Now, I want to go back when we start to the beginning. How did you get into tech programming, computers?
Well, I guess computers, it's like very very early on. My dad was a programmer at Digital Equipment Corporation, which is a company that people will know if they studied computer history, but would not know if you just looked at the landscape today. They're completely gone, right? They got absorbed partly by Intel, partly by Compaq, I think. There was, you know, they kind of got broken up.
At that time, it was kind of a really big computer manufacturer. You know, computers like the PDP-11, that's a Digital Equipment Corporation computer. The VAX, like things that you may have heard of in computers.
Oh, these were these massive mainframes.
Yeah. Minicomputers as well. So like smaller also sometimes than mainframes, like the kind of next step down, right? And so in general that era, my dad was a programmer there. He would later end up at Intel because, you know, like I said, parts got acquired. He never actually left his job. He just ended up at Intel through kind of Digital's eventual demise.
But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd. You know, I learned to program when I was seven, which would have been in like, you know, 1982 or something like that. And so at that time, you know, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer. I don't remember the exact line dates of those computers, but most people didn't. And it was only until a little bit later that you would, and you probably wouldn't have had a programmer in your household to teach you, more importantly, right?
So I learned really early on and that's when I got into computers. How I got into games was, I ended up randomly interning at Microsoft and I met people there and like I went and, you know, sort of went off into games through them. That was how that happened, if that makes sense.
But wait, how is the Microsoft/game relationship? That's not kind of a given, right? Microsoft is not... They [laughter] have one game, right? It's Flight Simulator.
Yes, at that time it would have been very weird. The reason that happened was a guy called Chris Hecker who a lot of people don't really know his history because there was sort of two waves of bringing games to Microsoft Windows.
Because I mean now it's funny to think about now because people think of like what platform are you going to play games on if you have a PC? You know, Windows is like the default. Linux is now an insurgent to that. But Windows is the default. And so you think about how did that happen? Because that wasn't the case if you were back in the early days. Microsoft Windows, not a gaming platform. It's almost nothing on it. It's like Solitaire and Minesweeper and a few other sort of games.
How did it happen?
So what happened, the reason for this, this kind of gets into a technological reason. The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly.
And to understand why this is is like its own kind of topic, but in general, you can just imagine you have this operating system running, which is Microsoft Windows. It's controlling the graphics card. It's often running at a fairly high resolution compared to what a game might want to run at. You have to negotiate with it in order to display your bitmap in some way that won't destroy all the things that it's trying to display. So on, so forth.
Early versions of Windows, up through like Windows 3, Windows for Workgroups, let's say, if anyone remembers that name.
Was it after 3.1?
It was 3.51 I think is what it's called, or maybe just 3.1. Yeah, there's NT 3.51. No, so it's like, I think you're right. 3.1, I don't know, something like this. Yeah, Windows for Workgroups was, you know, around that time.
That version of Windows, which is in the 3 series, didn't really have a way to quickly use the CPU to fill pixels, which is what games need to do, right? There's no GPU acceleration really at this point. There's a little bit we could talk about, but it's not mainstream. It's not in consumer. So they need to be able to do this sort of thing. They need to do it in a double buffered way so they can draw to a back buffer and then show it to the screen. And that has to happen very quickly. And there just wasn't a way to do this in Windows. In Windows, you had to kind of go through this API where you would produce sort of a bitmap that wasn't necessarily in the right format for the display you were using and then it had to do a translation from that bitmap to the other one when it displayed it.
All this sort of stuff. So games on Windows, like things like Doom, they're not coming to Windows, right? That kind of future was not in the works for Windows. And that was kind of what some people in Microsoft wanted to change. And one of the people who brought this change about was a guy named Chris Hecker. And he was like, "Okay, we could actually just make a library that did the fast blits to the screen so that we could have a way that people could do these draws and get them on the screen quick enough to make gaming viable. Would it be 100% as fast as DOS?" Probably not. But could you actually run, you know, some of these new games? You know, Wolfenstein 3D I think would have been out at this time. Doom was kind of on the horizon and that sort of thing, right?
Yeah. And so he started this project that he was not supposed to do. He did not have the authority to do this, called WinG, which I believe at least stands for like Win graphics or something like that. Total skunkworks project. He had cover from his manager. I'm assuming I can say all this stuff now because it's ancient history. He had cover from his manager. The guy's name was Michael Edwards. And Michael Edwards basically just kind of ran cover for this, which is a thing that probably wasn't going to fly because they were in a division at that time which would have been Microsoft Research today, kind of. It was called Advanced Technology, was the early version of Microsoft Research. You were not supposed to be shipping core libraries for Windows, like it had nothing to do with that. Long story long, what ended up happening is that product did ship. WinG, I guess you wouldn't call it a product. It's an add-on for Windows. Did ship and it was the first step towards DirectX.
People forget that WinG was the first way you did this and then eventually we had DirectX and DIB sections and Win 95 and all that sort of stuff. When I went to Microsoft, amusingly the person I was supposed to be reporting to as an intern was Michael Edwards. When I showed up first day along with the other interns, a guy named Rudy, a guy named Rajie, we were all supposed to report to him, right? Because you get a couple interns in under one, you know, sort of manager. We show up and we're just taken aside, you know, after some kind of, you know, stupid HR orientation thing that was, you know, lame as it always is. And we get taken aside by somebody. I don't remember the lyrics. They're like, "Look, bit of a problem. The person you interviewed with and were supposed to like report to, they aren't here." Like I don't remember exactly how they put it. "So, you're going to go talk to this other guy and they'll find something for you to do."
Turns out the WinG thing had blown up the previous week and Mike Edwards had stormed out of the building and had not been seen since. That is what actually happened. [laughter] So, I arrive at Microsoft, the person I'm supposed to intern with, he just flamed out and left. No one knows when he'll be back. He did end up coming back like a week after I got there. But he was kind of moved over to a different division. Like, you know, there was a bunch of like triage work done there to figure out what's going on. So that was my experience, but suffice to say it meant that I was right in sort of like ground zero of games on Windows. So I met Chris Hecker. I got to talk to a bunch of people there. He actually took me out to see one of my childhood heroes, Ron Gilbert, who is the guy who did the SCUMM engine. Yeah.
He knew all these guys because working on WinG he had gone out to see a bunch of game developers and like work with them and this sort of thing. So, I got to go to Humongous Entertainment, which was Ron's new company. He gave me a Secret of Monkey Island mouse pad, I remember. So anyway, it was really cool. And that's how I ended up getting into the game industry, was through Chris Hecker. And he's kind of an unsung hero of getting games on Windows because, you know, he just wasn't out there making his story known, but, you know, I am now, I guess.
But it's so interesting to hear these stories. Obviously, now you can share it, I'm sure. You know, like for a while this would have been like only within the inner circle, but the fact that, you know, of course, DirectX was a huge success and it did, like as far as, you know, from my vantage point, huge reason why games are big on Windows, but now here's someone who just ignored, wasn't asked anything, just was doing something, got into conflict, fights, and just like pushed an idea.
It's more than you think, because there were three people who really were the core people who pushed DirectX, meaning institutionally pushed it. There are tons of programmers like Todd Laney who did, you know, really important core work. It never would have shipped without people like him. So, not on the program side, I'm talking about institutional side. It's Engstrom and Alex St. John. I don't remember which one is either, or Engstrom was the tester on WinG. So he came from that team. So the start of DirectX, one of the core members of DirectX was on the WinG team. So it's a direct lineage. It's not even like an unrelated push. So WinG really was the start of it. And then DirectX was kind of the actual full blossoming into an org with Microsoft's blessing at that point, you know, that actually became powerful internally.
And then after this Microsoft internship and getting exposed to all these folks, you actually went and built games tooling, you then started your own studio as well, right? So like how did that sequence?
I guess there's a couple of steps in there. I worked at a startup with Chris Hecker that didn't end up doing anything interesting. Then I went to a company called Gas Powered Games, which was actually a Microsoft, I guess they're not a Microsoft studio, but their publishing deal was with Microsoft. And they did a game called Dungeon Siege, which is kind of a, you know, I don't know, it's not a particularly well-known title.
From there I went to RAD Game Tools and that's where I stayed for quite some time. I did their character animation system. That was a very popular product that ended up getting used in lots and lots of games. It's still used to this day, much to my surprise. Because that's a very, very, I haven't worked on it since 2004, [laughter] but I guess other people had maintained it and some studios just kind of integrated into their, you could get source licenses. So I guess some studios just integrated it into their pipelines and have never removed it. And they just maybe keep updating it to keep it working the way that they want.
So anyway, that's what I did there. And then afterwards I've been independent since then. I just have a company called Molly Rocket where we do various stuff. I've done, you know, contract work for people through that. We now do like the Substack through that where we do educational materials. So I've kind of just done random stuff since then. Although I have done some work on games. I noticed your checklist, you were talking about The Witness. Obviously that one was an actual specific title that I worked on, but that was mostly just because it was a very big project and I was, you know, I'm friends with John, so I was just
trying to do some helpful programming on the side. I did some stuff on how the movement system worked. I thought there were some interesting problems that we could solve there for games. And so that was a really fun project to work on.
Yeah. And then today you're doing educational stuff on programming performance on your Substack. And what else are you busy with?
So we do actually have an unannounced project that we've been working on that uses up sort of the rest of the time that I have, if that makes sense, which is not always so much. And we're hoping to announce it sometime soon. But it's not quite out yet. Believe me, I will send you an email as soon as we have an actual announcement. But given the fact that it is kind of like a split time sort of thing for us, because we're pretty focused on making sure the Substack is good and all that sort of stuff, we're trying to keep it fairly tight-lipped until we actually know we're mostly done, because we don't know how much time we can always devote to it, if that makes sense.
Yeah. No, it's pretty typical for games-related things, right? Like tight-lipped until you have something, for good reasons.
Well, sometimes people play the other game. They go like, "Look, day one we're going to be very loud about this and try to build a community around the development of the game and all that sort of stuff." And that's great. So, you know, that's another route you can go. But if it's not your full-time thing, if you have, you know, other responsibilities, that doesn't seem great, right? Because you don't have any insight into how much time you will actually be able to devote to it, right?
Yeah. And on Substack, it's called Computer, Enhance, and you started it with performance-related topics, and that's how we started to talk, I think, about three years ago, when you already had a Substack and we had a direct message conversation. I remember you messaged me saying, "Hey, Gergely, why do you think in the industry people, software developers, just don't really focus on performance?" I think you specifically wrote... you were saying how there's little emphasis on performance, even though there seems to be overwhelming evidence that performance is critical to the bottom line of most software. And I wanted to ask you, we've had a good back and forth on this, and actually I think initially I told you, like, oh, here's why you don't need to care about performance when you're building, I don't know, distributed systems or the like. But since then, what have you learned? Why do most developers, or even most companies, teams, engineering teams, not care about performance all that much?
It's a really good question, and I do think your answer at the time, if I remember it correctly, is certainly an accurate one for some subset of industries. You said something along the lines of, look, a lot of these pieces of software that you're seeing, the user isn't the purchaser, right? You were like, this is some kind of thing where somebody very high up is going to look and say, we need software for managing HR. They're going to look at the cost of the software, they're going to look at the compliance terms of the software, the legal liability, whatever, right? And then they're just going to make a purchase decision on that sheet. They're not in there looking to see whether there's a 30-second pause every time you want to try and access somebody's record, right? And I think that's very true. Unfortunately, the situation for a lot of enterprise software probably is that way. So maybe an individual might well be upset about the performance of that software, and I certainly hear from people all the time who are upset about the software that they use. They might not be in any position to change it. I think that's one thing.
There's thing two, which is that in a lot of cases you simply have monopoly effects. People aren't right now realistically going to challenge the social networks that currently exist, for example. People have tried. It's very hard. You know, Bluesky and Threads have tried to assail X, and you've got Facebook and Instagram and TikTok, and they kind of just own those spaces, right? And it's very hard to push into those because of these network effects. And maybe performance could be part of a package where you try to take on one of those players, like, hey, look at how much more responsive our thing is than theirs. Might be a nice plus, but that's not going to be sufficient if you just show up with no plan for how you get adoption, no plan for how you get big influencers over there, all that sort of stuff. It can't sell a product on its own into a monopoly space, right? If you're just talking about apps that someone can choose to download, maybe you've got a shot there. But those are forming a smaller and smaller subset of what software is, right? And this bigger and bigger subset is these monopoly platforms you go on to sort of work with. So I'd say that's another thing.
Thing number three is I think now people sort of are caring about performance more. I think over the past decade, the people, including myself but many, many other people, who have been saying that this is a problem have actually had some effect. I don't think that it was a waste of time. I'm seeing a lot of new emphasis on performance, people talking about performance, people posting benchmarks on things. And so I actually think that the third thing is, well, actually it kind of does seem that pointing at this issue and saying this is something we should be doing better has not been completely a waste of time. I do see things as sort of starting to turn around a little bit. I also see people attacking major product categories now with performance-based pitches. Things like File Pilot or the Blick video editor, things like this that have been coming out lately, where it's like, oh, really performant software to try to take on incumbents in a space, and they've been getting traction. So I think that's also a really good sign.
Yeah. And I guess on this last category, a really good example in the developer community is Bun versus npm, where Bun just said, okay, we're like 10x or 20x or 50x faster, and devs were like, what, is this possible? So there was this outrageous claim. I wonder if you need to have these outrageous claims, because devs started to pay attention, because it was 10x faster in many categories. Linear versus Jira is also a good example, where Linear has this benchmark of 300 milliseconds for any action, and Jira, of course, we know is just slow because they have a bunch of complex... you can explain why, but it's slow. It was never built for that.
Yeah. And if you think about something like a 300 millisecond budget for an operation, 300 milliseconds is like an eternity in computing, right? And so if you're talking about, like, our pitch is that we're not more than 300 milliseconds, that just shows you how the bar was so far past where it probably should have been for something. And you see it everywhere. You go on to programs and you're waiting sometimes seconds for an operation. And I don't think people realize just what an eternity a second is in modern computing, especially when you're sitting on networking that has sub-10 millisecond ping times. Sometimes the actual packets had to travel physical distance to get to this data center, and that was being done far faster than this very simple operation that you were failing to do in a reasonable amount of time. We are massively underperforming, and people don't believe it when you say 10x, 100x, but it's actually true, and we've seen a lot of proof of it, as you point out.
I do wonder if one part of the lack of focus on performance is that a lot of developers don't know the baseline. I'm reminded of Simon Eskildsen, the founder of turbopuffer, who has this project called Napkin Math, where he did a list of mostly networking operations. How long does it take to transfer one byte between two AWS data centers? How much does a gigabyte take? How much does a terabyte take? How long does it take to write to an SSD, to an NVMe, and so on. And so he had these numbers, and he said that what he found is whenever inside Shopify they were deciding, do we choose vendor A or vendor B as a database, they would just run a benchmark that they would write themselves, and they would get, okay, storing this takes 2 seconds on this one, 10 seconds on that one, we will choose the 2-second one. And he looked at it and said, hey, this doesn't make sense. To store in a file system, here's a theoretical limit, which is, I don't know, 100 milliseconds. There's no way that's going to be 10 seconds. And it often turned out that the benchmark was just wrong. They were benchmarking the wrong thing and they were making decisions on it. So I wonder if there's a thing where many engineers, developers, are maybe just not aware of how truly devastatingly slow this thing is versus the resources you have.
That is the entire point of my Substack, right? So what you just said is exactly true, and it is the thing that I hammer home on the Substack through all the parts of the courses on there, which is that in general there's a misconception about the way that you approach optimization in computer science, or in software engineering, let's say. And that misconception is that what you do is you run a profile, you identify where the big parts of the profile are, you make some changes to those, and you measure statistics. The more statistics you can get, the better. And you look to see if those statistics improved. If they have, that was a good change, and you proceed as such. And this is completely not correct. That is not how anyone has ever... I've worked with many extremely good optimization people, and that is not how it is done. The correct way to do optimization is very much like what you just said. You first go, what are the operations that this system has to perform? What is the underlying hardware capable of doing at its theoretical peak? And then you measure the delta between that theoretical maximum and what you have achieved. And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained, and hopefully come up with explanations of why you aren't at theoretical, because oftentimes you can't hit theoretical. That's why we call it theoretical, right? And it's crucial that you do this, because otherwise, with that other method, where I'm just going to make something I think might be an optimization and look if my statistics improved, all you're doing there is finding a local minima. That's all you're doing. You've got this shape of your performance and you're finding some little spot and you're sitting in it. That's not optimization. That's improvement. But optimization means to make optimal, right? It means we're going to find what we actually should be able to get this machine to do. And so that's why I emphasize that approach, because it's the one that I've always seen great optimizers take. That is how they get good performance, by knowing what the maximum could be.
In addition to that, it also is what lets you become better at optimization. Because no matter who you are and no matter how much you already know, when you go to tackle an optimization problem, there may well be some things in the new way that the system is laid out that you don't know about: new things that people have not figured out about modern CPUs, new things that are different about the network backplane, new things that are different about the GPU drivers, who knows, right? And if you don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there. And you would be surprised at how many times we find anomalies like this, things in CPUs that no one knew about. I've literally had them in the course of making the Substack. I've been like, "What is this thing?" And I look into it and it's like, "Oh, there's this new renaming, this new RAT table thing that Intel chips seem to be able to do. We didn't know about that." And that's a new thing we have to model when we talk about how to do performance. And so that's the other crucial part of what you were calling napkin math. I also call it back of the envelope. That's the term I've heard used for it. Oftentimes they're kind of interchangeable, right? Knowing what the theoretical is is how you learn as well, how you learn about new hardware and new performance options.
Interesting. Plus, by doing this you're just learning. You're becoming a better professional. You understand more about the given hardware, or the inner workings of your computer, or software stack, or kernel, all the stuff that I guess goes way beyond the vanilla programming language. Because you can say, I'm an engineer, I'm a software engineer because I know how to use this programming language, but I'd argue you're probably an engineer if you can go down the stack and you have that ability and you have a good understanding of some of it at least, right? And you can learn the rest.
And I would also say that one of the other things that we do in the class is teach how to read assembly language. And people often ask, why assembly language? What would I ever need that for? There's a very good reason for it, and that is that everything else that you might use doesn't tell you anything about what the CPU is actually receiving. If I look at a Java program, if I look at a C program, if I look at Haskell, OCaml, whatever, right, Rust, all I'm seeing is input to a compiler. I have no idea what the CPU is actually going to be asked to do. If I look at the assembly language output from that compiler, I know exactly what the CPU is being asked to do. And it's not that hard to learn to read assembly language so that you can see very quickly: is the CPU being asked to do the things that I think it should be asked to do? And in that way, you hardly ever have to write it. It's very rare that you have to write assembly language to do anything, other than sometimes for test purposes it's easier to do that, so you don't have to try and convince a compiler to output something. So if you're just testing something, sometimes it helps to be able to write some assembly language. But if you're talking about the vast majority of tasks you might do in optimization: writing it, no; reading it, essential.
And it also unlocks this huge world of possibilities to you, because once you know assembly language, you can now do things like read those CPU diagrams. When they announce a new processor, they put up a little diagram. That diagram tells you stuff like the fastest this thing could do multiplication and stuff like that. If you know assembly language, you can read it right off the chart, right? If you don't know assembly language, you look at that chart and you're like, I have no idea what I'm looking at, right? It's just this weird flowchart that doesn't really tell me anything. And so one of the really great things about assembly is it unlocks all of this knowledge for you, because it's the actual input language to the machine, and it allows you to figure out how it's operating.
Plus, I guess we should add that
Assembly language is not all that complicated, right? Just by nature, it's a far simpler language. Okay, it's harder to read if you've never seen it, but in terms of the number of operations, it's so barebones, because, you know, that's what assembly is. Like, every single higher-level language will have way more keywords, structure, whatever you name it, right? Than assembly.
Massively more, and especially when you consider the subset that are actually used. If you look at the subset of constructs that you would need to understand to be able to understand, say, just a website from today: all of the JavaScript libraries, all of the JavaScript syntax, all of the DOM, you know, all of the behavior that's going to go on there, CSS...
React, CSS, right? All of that.
Assembly language, you know, maybe there's 20, 30 instructions you might have to learn total, because most things in legacy assembly like x64, most of them are hardly ever output by the compiler. So you only need to learn a very small subset. That's the ones that you're going to be seeing in 90% of the cases. It's so much simpler.
And also, when you're looking at performance, you're typically only looking at a very small part, right? You've kind of understood roughly what's going on, and you've seen the basic layouts of your program. You've identified what's supposed to be happening and you're just looking to see, like, wait, why is this part, which I don't think should be running this slowly, why is it running this slowly? It's just a very small piece you typically end up having to look at as well. So it's really much easier. If you can understand how to center a div, as they say, if you can vertically center a div in HTML, then you can probably learn assembly language, I would say.
Okay, you're super passionate about performance optimization. You also have really good educational materials, both free videos, your paid Substack, the free parts of it, etc. But let me just play devil's advocate. There's this saying that premature optimization is the root of all evil. And we typically use it, or I typically use it, so many times. We're like, oh, should we make this performant? Should we optimize the thing? And, like, nah, let's not do that. Let's first build it. Let's see if it's good enough for our customers, for ourselves, and if we need to, we can always optimize it. I mean, you know, it's not the hardest thing in the world. Okay, maybe not as good as how you mentioned, because maybe I don't read assembly, but that's kind of a thinking of building, you know, I guess SaaS software, building software at Big Tech. What is your reply to that? Because I feel really good that I made a really good argument here.
So I guess what I would say is the important part about that, and I guess I'll divorce it a little bit from the saying. I have an entire lecture on that saying, by the way. It's like two hours long, and I gave it at Better Software Conference this year, and I believe the VOD... We're linking that in the show notes below.
Okay. It'll be like a week or two, I think, till it's up. So it may be right at the same time as this. But so if you want to find out the history of that phrase, you can go look at that. But I wanted to talk about the idea behind it, because I don't want to dismiss it entirely, because it's not entirely false. And the idea is that, well, I'm just going to delay optimization work. I'm not going to think about that, and then I'm just going to make whatever I'm going to make, and then, you know, either myself or maybe I'll just hire some performance person to come in and clean up the mess later, right?
So here's the positive side of that first. The positive side of that first is, for some types of code, that will work. If you happen to have written some operation poorly where the optimized version of that operation just looks like someone taking a loop and changing the loop from your really, like, you know, naive version to a really well-optimized version.
The typical of, like, I wrote a bubble sort, we can later optimize that.
Who knows, right? Anything of that form. Okay, maybe we can just do that. So there are certain times where you do in your head want to be doing this, where you want to say, okay, I could go spend a week researching the fastest hash table implementation here, but part of software engineering is being smart enough to know it won't matter if I do that now or later. The architecture around this piece won't have to change, I'm quite certain, because I understand the problem well enough. So it's okay, I can defer that to later. Maybe it's never too slow with the naive one I put in there, and then we don't have to do any work. Maybe it's too slow later. That's okay. I just target this one hash table implementation and we'll get as fast as we need, right? If you're doing that, if you're applying that true engineering mentality to it, you don't have a problem.
The problem comes when you don't know if the choice that you're making produces that kind of optimizable hotspot. And I'll give you a very simple example that usually people have had experience with. A very simple example would be: we write our entire software thing, like, whatever, this massive thing that we're imagining doing where we're going to ignore optimization. We sit down and we write it, and we use a paradigm where we ask the server for something. We have, like, some API, you know, that we've built for asking servers for things. We ask the server and it returns to us what the server's response was. And that's kind of how we architect this thing. So everyone writes, you know, hundreds of thousands or millions of lines of code, and they all look like: ask the server something, do some calculations, ask the server for the next thing, do some calculations, right? Then at the end you find this is way too slow. But that's okay, you weren't worried about that, because you're like, at the end you call in some performance experts. They look at it and they go, there's nothing we can do for you. Sorry.
Why? Well, the reason is because you created a serial dependency chain. All of your code looks like: wait for a network request to come back, do something; wait for a network request to come back, do something; wait for... And that serial dependency chain can't really be shortened without just rewriting it. If instead you had made the paradigm and told your programmers, look, here's what you need to do: at the top of every operation, you need to figure out all the things you might want to ask the server for, you ask them for all of those things, right? And then you do all of your processing there. And you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask a server for. Now you're just in this situation because you didn't tell them to do that. You have to rewrite all your code. Everyone is now going out rewriting all the code, if they even can, if it's even possible to really do that in a way that's not slower than just rewriting the thing, right?
So what happened there? Well, again, we talk about this a lot on the Substack, but there's this idea of a serial dependency chain. It's when you stack things in order, right? And the performance of your software is generally determined by the longest serial dependency chain, because it's something that cannot be parallelized. If I have thing A, then B depends on that, then C depends on that, we cannot shorten that, because it has to go in order. Everything waits for it. We can't multi-thread it because it's dependent. We can't, you know, make it run wide. We can't, you know, amortize the network request, whatever. That kind of thing can be pervasive in the programming, and we can't cheaply remove it because it's not a hotspot. It's a way that you did things.
That's the part where that kind of thinking breaks down. If every software engineer knew to watch out for false serial dependency chains, things where they were creating series of dependent operations that could not be optimized away, or other sorts of architectural problems like that that cannot be easily fixed, then the world would look more like just wait and optimize the hotspot, right?
Yeah. So this is the architecture, the planning, right? Like if in that phase you're like, okay, as this thing grows, what will get in the way of performance, what will slow it down? Or you can ask all these questions, or different flavors of the questions, you know, or from the other side and so on.
Yeah. Another way to think of it is, because hotspots is the way that people talk about that, like, oh, it's going to be hotspot optimization, we just got a few spikes, someone will come in and clean up those spikes and we're done, right? The way to think about it is your codebase will not end up that way by accident in most cases anymore. You have to engineer upfront for a hotspot codebase that people can then optimize, right? And so that's the crucial takeaway: everybody on your team who is making architectural decisions, those people must know performance, and they must make decisions that will allow the other people downstream of them to use an architecture which can be optimized later. If you don't do that, you're just rolling the dice.
Casey just talked about how engineers making architectural decisions should know about performance. This is also true when choosing your dependencies, like which database to use. And this is where I want to mention our season sponsor, Turbopuffer. You already know how Turbopuffer is a vector and full-text search engine. But here's an interesting story from Linear on what happens when you stop thinking of Turbopuffer as a search engine and start using it as a primitive to reduce latency.
As context, Linear is a local-first app. So each client keeps a local database, and when that client goes back online, it needs to catch up with what happened, and do it fast. Their biggest workspaces generate around a million sync actions per day. Doing catch-ups by reading from Postgres was getting slow for large reads. So the tail latency got too large, and adding more replicas did not help either. Linear solved the problem cleverly. They started using Turbopuffer as a serving index for each client. This is because Turbopuffer itself is built on top of inverted indexes. So for every index value, it stores the documents that that value can be found in. The lookup cost for such an index is constant. So Linear took this structure and had each client's index point to the changes that they needed to sync. As a result, not only did they reduce latency, but they kept it constantly low no matter how long the change list synced to the client is. Linear published a blog post about this refactor titled "Rebuilding Linear's Delta Sync Read Path." Check it out.
I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system. If you're building systems where you store a lot of data or serve a lot of data, Turbopuffer can probably speed things up or save on your costs. Learn more at turbopuffer.com/pragmatic.
I'd also like to talk about a presenting sponsor, and this is: while Casey deliberately does not use AI coding agents for his work, most of us do. And when you work with coding agents, your job is no longer writing code. It's specifying and testing it. Antithesis is the most effective method for verifying agentic code today. Let me explain how it works. Antithesis runs your whole system in a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue.
To create such a tool, Antithesis needed to invent new kinds of debugging tools as well. For example, here's what's called a bug probability graph. The x-axis is virtual time and the y-axis is probability. As Antithesis runs hostile simulations, it plots time frames when the bug probability increases, which greatly helps with finding the root cause of the bugs. And Antithesis also has a log visualizer. Vertical lines going down represent events branching off from the same state, and the purple dots are where the bug happens. Antithesis is as good as it gets in being able to ship agent-written code. It's what teams at Jane Street, Fly.io, and the etcd community use to ship with full confidence. Head to antithesis.com/pragmatic to learn more. And with this, let's get back to Casey and how if you don't design an architecture that can be performance optimized later, you're just rolling the dice.
And we've seen so many projects. I have an entire video where I go through, like, look at all these blog posts of people who say, you know, it's Facebook, it's Uber, it's everybody. They've got blog posts of: we had to rewrite this whole thing because the performance was bad. If it was always hotspots that made your performance bad, you'd never have to rewrite the whole thing. So we know that that doesn't work anymore. Why? Because of the things I just said.
I was at Uber, where I was not making the decision, but the teams next to me were, and I kind of understood why they were making it. But typically, and right now it's happening with AI companies, oftentimes it's like: we chose this technology, which is Python, and it's single-threaded, and it made sense at the time on the web server, but now we're big. And this happened at Uber. It was Python and Node.js, and then they went to Go and Java on the back end. And now with AI companies it was Python. OpenAI and Anthropic are both going through this right now. They're both either public about it, or I've written with Anthropic, they share with me, but I put it out there. They used Python because data scientists or AI or machine learning engineers knew Python. They put it on a bunch of web servers. They had their API run on it. Initially, they just, you know, scaled horizontally, but now they're like, well, if we move over to Rust, right now they're choosing Rust or Go, but I think it's Rust, well, we can actually have multi-threading, and the same machine can actually handle more connections. So, cool.
I came across a lot of that because I think that's easy and safe to communicate, because it doesn't look bad on you. But you're right, a lot of times I don't think on engineering blog posts you'll get the real reason that these companies put out there. When it's kind of a very, you know, easy-to-own mistake, or not a mistake but just a decision which made sense, they'll tell you. But if it's something that was an oversight, you're not really going to get that on a public-facing engineering blog post, except for maybe some startups who are really there. But don't forget, a lot of those blog posts are going to help someone get promoted or get recognized, and they will always be way more positive, especially when there's a content writer team, which large companies do have. So it's not quite PR, but it's somewhere midway in between.
And I mean, yes. And also, I would just point out the fact that these things are happening, though, is all we really need to know for the signal, right? Because in general, this should not be happening. If the ideas about optimization were true, you'd never have to rewrite something in a different language unless you just preferred that language. The story would just be: we rewrote it in this language because we wanted to use this new language. Or for Rust, it might be just memory safety. We see those blog posts, right? It's like, why did we rewrite it in Rust? It wasn't performance. It was just we wanted the memory safety or something like that.
If I'm a software engineer, programmer, and I'd like to just get better at writing performant code, I'm interested, you know, maybe after this podcast or looking into some of the things that you did. What is a learning path you would follow outside of your Substack, where you cover a lot of these things? But what are areas that you think are kind of like, you need to understand these things to get better at writing performant code? I
think it's actually very simple and perhaps a little bit counterintuitive. So, I'll start with the very good news about learning to write performant software. The good news is that optimization of the kind that we sort of talked about, the like hotspot kind where it's like somebody's going to go in here, maybe they're going to even rewrite this routine in handcoded assembly or something crazy like this, right? That's very rarely necessary these days.
One of the reasons that you don't see hotspot optimization as a thing that really matters that much anymore, and one of the reasons I advise that architecture and not making bad decisions is much more important, is because a lot of libraries already have been optimized for you that you might use. CPUs are incredibly good at taking bad code and running it quickly and so on. So, typically when we're talking about the causes of negative performance that aren't squeezing every last little thing out of the hardware, but rather just making sure this thing isn't running like a hundred times slower than it should be, usually it's more just about having an awareness again of what the computer should be able to do and making sure you're making software architecture choices that allow it to do that. And if you do those things, you will generally be within, you know, 2x or something, which is 50x better than the people who are 100x [laughter] away, right? So the good news is in order to write software that's much better than a lot of the software you use today, you don't have to know that much.
So what do you have to know? What I argue and what we focus on in the Substack is I think you just have to go through the experience once of learning, reading the assembly language, seeing how the CPU works, seeing the difference, seeing why Python is slow, which we show on the Substack. So one of the first things I show is I walk you through the assembly language necessary to execute A plus B in Python, and it's so vast that, you know, I have to skip most of it. It's massive, right? It's like this huge. Whereas, you know, if you have the equivalent function in C, it's one instruction, add, right? So, you know, understanding basic things like that. If you go through, learn to read assembly language, learn to look at some code, learn to do some CPU timings, and you just have that experience, just spend, you know, a month or two of nights or whatever you want just understanding some performance stuff and going through a few examples where you play with it and you see the difference.
And just so I understand, you're saying do this not because, let's say you're doing iOS development or web development, React, like you will not look at the assembly that React does, but if you do this on a project you will be able to conceptualize what is likely happening, what the layers are, and you might be able to decide like, do I want this layer or do I want to use, let's say, WebGL? If you're a React engineer, you probably haven't touched it. But again, you can skip a bunch of those things and it comes to trade-offs with maintainability, yada yada, but now you will know kind of what you're saying by keeping this layer or not keeping it and so on. Do I get that right?
Essentially, yes. And like, you know, the simplest example is the Python example. Most people have never internalized the fact that it takes, you know, maybe on the order of a hundred more CPU instructions to do an add in Python than it does to do it in an equivalent language like C for the same piece of text, just A plus B compiled in two different languages, right? And so just understanding even just that is enough for you to kind of go like, "Oh, okay. A, now I kind of understand why if I'm using Python, I kind of have to use libraries to do things." And those libraries were written in C. Because it's like if I'm ever going to do any operations on a large number of things, I can't do it in this language because the amplification factor is so high on each operation that it just, you know, kills the performance immediately, whereas these other languages don't have that, right? And so understanding those orders of magnitude and what's actually going on, I think that allows the programmer to know, okay, if I think through what I'm doing right now, can I afford the super slowdown that I'm going to take?
And usually I don't think you have to be a performance expert to make that decision. You could usually know like, okay, is this a part of the code that can afford to be 100 times slower than it should be or not? Right? And, you know, most people can I think make that decision fairly logically. And if it's not, then now you know like, oh okay, if I'm in Python, then what I got to do is either I got to go find a library call that will do these sorts of things and structure around how that library works, or I should maybe get something like Cython or something where I can do compiled stuff inside my Python and make my code work around calls out to that kind of code. You know, you now have the tools you need upfront to make sure that when you write the program, you've put the parts that needed this and you've structured the code in such a way that you are only paying the 100x on things that you know are very infrequent or happen only, you know, once every so often, things like that, right? That's I think the biggest thing, is just the knowledge. And once you know, you can start to make much better decisions in any language, because it doesn't take you very long. You know, a simple search or asking an AI or whatever is the common practice that you're going to do. A simple bit of that, once you know what you're asking for, will get you this information back very quickly. Right? You just have to know that you should have been thinking about it.
Now, you mentioned it's good to understand how the CPU works. As a software engineer who is not a games developer, I'm not doing low-level stuff, what does that give me? Because for the most part, even in academia or in computer science, you know, there is some level of some basic CPU theory taught, but usually we just kind of stop at the code. Okay, maybe you look at the assembly, but you rarely go further than that. The folks who, you know, you've taught and they learn these things, what do you see them get out of this that they wouldn't otherwise?
So you're talking about specifically the knowing the CPU part?
Knowing about the CPU, knowing about the details about a CPU, because you mentioned that that's also part of it, right? It's not necessarily just stopping at assembly.
So the reason for that is more the other way around. The reason to learn the assembly language is so that you know what the CPU is doing. So it's the CPU part that's actually important, and the part that's important about it is that the CPU is basically, you can think of it as a little machine whose internal gearings we are not privy to, because for the CPUs that we care about, so, you know, an M-series CPU in a Mac, a Zen core CPU in a server or in a laptop, or an Intel, you know, Core series, those sorts of things, these CPUs are not documented at the level where you're going to be thinking about how each little individual part works. And to that end, it's unclear that you would have time to do so anyway, because these are massive, very complicated machines that we're talking about, right? But from a high level, from a more black-box perspective, they are machines that we can think of in relatively straightforward ways once we know kind of what their core instructions are that they tend to execute.
And they break down into a couple different categories. There's how does data move into and out of a core. And this is basically how like load/store units work, how the cache levels work, L1, L2, L3. We have L0 now sometimes, things like this. How does that work and why? What is the granularity of it? What is the policy? How does the CPU go about actually working with those things? Understanding that part of the machine is crucial because when you're working with a lot of data, the difference can be massive if you structure it in one way versus structuring it another way, right? Again, architectural decisions that have nothing to do with hotspots. They're how all the data is laid out and what the access pattern is, right? Things that are very hard to change sometimes. So, that's one part of the machine you want to understand.
Another part of the machine you want to understand is how the instructions flow through it. And, you know, a lot of people have heard about like branch misprediction or things like this, I-cache misses. There's words that you might hear, but you're not really sure what they mean. They're all actually pretty simple to conceptualize. Sometimes they're harder to pin down exactly how they work, because branch predictors are, you know, getting more and more complicated and so on, but you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't. And then finally, there's the execution unit scheduling part, which is about knowing what's the raw sort of throughput for any particular type of operation: floating-point multiplies, integer additions, division, whatever it is that you might want to know. Right? Once you learn a little bit of assembly language, you understand what it's reading, you understand how it turns those assembly language instructions into micro-operations, which it actually does, and how they get distributed through that machine. That flowchart that they put basically up when they're like, we've announced the new Zen core, that flowchart, you can look at it and go, I know the performance of this machine roughly, right? Not exactly, because like I said, there's all these little edge cases if you really want to be a crazy optimizer, which again I don't really advocate people do. I don't think it's important that they be like crazy hyper-optimizers. If you like to do it, great, it's a lot of fun, but it's not the important part. Just look at the CPU, like, okay, I see what the CPU should be getting in terms of what I could do with this size data load, that size data load. This is what I could probably get out of it if I was doing a bunch of like math ops on it, you know. And I think that's just something that should be kind of par for the course in software engineering. You go to school for four years to learn this. There's no reason you can't learn this in a few months. It's not that hard.
Yeah. Plus, I guess just from a craftsmanship perspective, like we should know our tools. We should know the machines that we're programming. Obviously, we know that our code, if you're doing web, it'll be running on all these different things, or if it's mobile, on all these different phones, but from a conceptual point of view, we should be able to know what's going on. So I feel there's a bit of pride as well. Like if nothing else, you would learn a bunch of stuff. Like I know some of it, but I'm now getting motivation to learn more about it.
I do think there's a craftsmanship angle. I think there's a large number of people who maybe don't feel fulfilled when they, I've certainly heard from lots of people who, when they write something and it's just kind of this amorphous high-level thing, they don't get as much satisfaction out of it. And then when they learn how they can look more deeply at what's going on, they feel much more satisfied. Even if they didn't change what level they were programming at, they feel much more satisfied that now they know what they're doing, right? And it's like, oh, I see, and I understand why this thing was happening this way and this thing was happening that way. That's very satisfying, right? So there's an aspect of that. I also want to emphasize another part, which is that it's a percentages game. If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster, all of a sudden all the people using the libraries, their code gets a lot faster, and so on and so forth. If the APIs start changing to make it easier to optimize the libraries because people now thought that through, right? Like it's infectious. The more people are doing performance, the less people need to do performance, [laughter] if that makes it kind of paradoxical, right?
Well, plus I do think that right now there's still an edge in just being performant. Again, you mentioned there are categories of software that are just winning by being much faster, and to do that you need to do this. And if you know how to do this, maybe you're going to spot opportunities as a software engineer. Maybe right now you're not as happy in your position, but maybe start something or do a side project that turns into a full-time thing, and so on and so forth. So I feel there's like, and worst case, you just learn net new knowledge, which will probably not be as outdated with AI, which we'll get into later. But this stuff, it just feels very interesting, right? Like kind of it moves your brain.
It does. And thankfully, it's also not that hard to update your knowledge, because again, you get these presentations that the CPU companies give and they're like, here's the changes we made. So you kind of are aware every time a new thing gets, you know, there's little tiny things that creep in that you don't know, but again, there's people out there who are running lots of microbenchmarks that you will find out about, and they often uncover these for you as well, right?
So I wanted to talk about games. You've built games for decades at this point. Can you give an overview for those of us who are not in the games industry: how is a game typically built, from the games that you know of, that you've observed or you worked on? Especially trying to compare, you know, like in typical SaaS or distributed systems, we're building a website or service, it's kind of, you plan this stuff, we'll do an estimate, we'll build it in a few months or a few weeks, we deploy it, and then we monitor it and then we keep tweaking it. And then, you know, fast forward 5 years later, it's now this gigantic thing with microservices, but it keeps evolving, right? We do a lot of this prototyping thing. For games, it's pretty obvious right from the get-go, as we're talking, there will be a launch. But when you join a game studio, what would you observe there in terms of what the process is like, and how it's different or how it feels weird compared to, I guess, I don't know, traditional SaaS software development, whatever that is?
So, I guess what I would say is, unfortunately, I'm probably the wrong one to ask because my knowledge is outdated at this point, because one of the things that has happened to games recently is I feel like they've moved closer. I don't want to necessarily say entirely in development practices, but at least in terms of the nature of the product, it has changed somewhat dramatically to be more like something like SaaS, where, you know, if you take some of the most popular things in terms of dollars, let's say, so I guess maybe popular might be hard to say specifically, but let's just say revenue generating. So if we were to measure the total games industry revenue and you look at what are the largest slices of that, you're seeing things like Fortnite, like Roblox, like Grand Theft Auto 5 Online, etc., etc., Minecraft. These things are starting to look a lot more like an always-on live service, kind of we-ship-incremental-features-to-our-customers kinds of things. And so I would actually say that, you know, I'm the wrong one to ask about what that actually looks like from the inside, because I haven't actually gone and worked at one of those companies. I have friends there, so I hear things, but I'm not the right one to give an accurate picture of it. But I just would
Point out from my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building.
But can we talk about it when you were building games, which was, you know, 10 plus years ago, when I understand these were the games where they were built, they were launched, you know, maybe they got a patch or two, and then the team moved on, they were disbanded. It was this time-boxed thing that was a lot of development, a big launch, and either it went big, huge hit, or, you know, a huge failure, right? And then the studio goes bankrupt. So how did that work? Because I feel that's a world where, okay, today a bunch of games don't have those constraints, but it has a bunch of constraints, and I'm interested in what worked in those constraints.
So in the early days you didn't have licensable engines. So up until sort of the point, like nowadays, like this is why I say it's a lot different now than it used to be. You know, nowadays you think of web development like I'm going to go grab like a thing like React and I'm going to make this thing or whatever. I'm going to go grab an off-the-shelf database thing, Postgres or Oracle or, I don't know, like what would be the thing of choice, right? But
By the way, just definitely Postgres and not Oracle for most people.
Okay, sorry. Sorry. [laughter] Postgres, I didn't want to slight anybody there, so I apologize. So, okay, definitely Postgres, sorry, Oracle.
Yeah.
So you're going to go use some kind of a variant of one of these off-the-shelf databases and so on. That's more, like I said, what people might be doing nowadays, too. Like they'll grab the Unreal Engine. They're not going to develop an engine on their own. They'll grab backend server stuff from people. It might even be some Postgres in there, right? Like, who knows? In the earlier days, none of this stuff existed for games. I actually worked, like I said, in middleware at the time. So, I was actually sort of one of the people who was working at the time on maybe changing that a little, like producing code that would get reused throughout games, which was actually fairly rare.
And so back in the day, every game built their own rendering engine, for example.
Correct. And so, yeah. And so really early on, right, if you rewind the clock far enough, then yeah, the degree to which people were reusing code for their thing, for like their rendering engine, it'd be like, 'cause I got some code from like Dave or whatever, who was, or we were both at Atari, and somebody wrote this good routine, so we used it, right? There was that kind of thing, but there wasn't like this sort of set engine. And the time when that sort of maybe you could say first started happening a little more widespread was with things like what id Software did, where they sort of started having like, oh, you know, someone's going to build something with the Doom engine or someone's going to build something with the Quake engine. There was also the Build engine at the time, made by Ken Silverman, and some things like that. So there were some early cases where a few people would make a game, but they were making a game very much like that. Like if we did the Doom engine, we're going to make a game very much like Doom. So it really was the case that for most games, people were rebuilding most of the things from scratch, at least for their studio.
And studios, often their existing code base was kind of part of the value of the studio, too. Like if you are, think Blizzard, and we just built Warcraft 1. Well, rolling all of that knowledge and code into Warcraft 2 is a huge advantage for us, because everybody else who wants to build a competitor to Warcraft 1 has to do all of that from scratch. They have to make the path. They have to make the level editing tools. They have to make the rendering. They have to make whatever. And so, you know, that was how things were traditionally done.
No wonder the games industry is so secretive compared to the rest of the software engineering industry. Like seriously,
Used to be.
Used to be, at least. Yeah. Maybe now it's changing.
And so there were two really big risks typically when you started a game project in those days. One was the engine risk: would we be able to make something that would be technically able to do what we need to do for this game? And that risk comes in a lot of flavors. One, will it happen at all? Two, will it happen fast enough for us to actually reliably build the game on it, right? One of the things, I mean, I don't know how detailed an answer you're looking for for this question, so stop me if I'm going down too many tangents, but one of the things you also have to remember is that at that time, and this is sort of still true today, but at that time it was very important, there was no way to really buy something all that much faster than what you had. There was not a huge strata of PCs that you could, you know, buy or anything like that. So the rendering engine, there wasn't like a way your level designers could like be playing on a faster thing than the consumer would have, really. You can only have the machine that you have now, and if this game comes out in a year, that's sort of roughly what the consumers might have, or a little bit. But, you know, so there are people who started doing things like buying SGI workstations because those were actually faster enough, right? And things like that is what, you know, you kind of had to do, and so on.
So anyway, there was a huge engine risk, and some games just failed because they couldn't produce a thing that could technologically do what they needed. You saw houses who survived on technological prowess. You had id Software that was kind of unrivaled at making those kind of first-person engines. You had Bullfrog, who had this engine, the pseudo-3D engine that they did for like the racing games, Magic Carpet, Dungeon Keeper, like they were all based on this, you know, one core tech and all these sorts of things. There's that engine risk that was huge. And how do you mitigate it? You didn't. You were just grit, right? Because there wasn't a way to buy one off the shelf. So, you were just kind of gritting your teeth.
The other big risk, and this one is still somewhat true today, but it's just much less because you can start, you could do prototyping early. The risk is, is the game any good? Like, what are we building? Is it interesting? Is it fun? And when you think about this problem of we can't even really run the game as it will be, 'cause we're just building this engine, and we don't even really have a way to test the game super well. And we can't really build much of a final level because we don't have level editing tools yet. Those are just coming online. Trying to guess what you are actually going to be shipping in terms of gameplay is incredibly hard. And there are games, famous games, I want to say like Thief: The Dark Project, a very famous game, Looking Glass game, it was formative in the stealth genre, launched a franchise, which was Thief, you know. I want to say everything I heard from people on that team was that like the final core gameplay only sort of came together like right at the end, right? And so the game just could have been a lot more, could have been polished a lot more, but just the timing of these things coming together was so hard. And so it really was an incredibly different thing, and nobody really had a way around it.
Eventually, there was sort of this push towards something. Well, it was basically early vertical slice prototyping, where as games started getting bigger and people were like, we can't keep doing this, especially if we're going to be putting millions of dollars on, like, this is not an option, right? They started to move toward this thing about, look, what we're going to do is we're going to focus the entire studio on building one vertical gameplay slice as fast as we can, as hacky as we can. Whatever we have to do, do that. Prove that that is engaging to play, and then start building out everything else, because we simply can't afford to not know what that thing is. And then we can start building like spreadsheets that'll schedule what are the assets we need, 'cause now we actually believe in the thing, we can see it running, and it fills in all those details, right? And that, I believe, you know, I'm not a game historian, so take what I'm saying with a huge grain of salt, that I believe was a pretty big paradigm shift for the industry when they started going, okay, we got to actually know. And it became much less seat of the pants after that, if you will.
Now I'm interested in your observation. I know you're not a game historian, but you were in the industry and you still remain connected to it. What happened when game engines became widespread? They became not only licensable, like we're talking about like Unreal Engine for larger studios, but ones like Unity or Godot, which now amateurs could also afford, or I mean amateurs in the sense that you're a college kid or you do some side project, you can already afford the license and you can build stuff. Because now that risk is gone, clearly, for the studio, so that risk is eliminated, and it also, I guess, now opened up so many more people who can now have a shot at creating a game, because you no longer have to have this massive amount to license this super expensive game engine. You no longer have that risk. The only risk is: is it fun? What have you observed happen in terms of both for the industry, for development pace, those kind of things? And the reason I'm asking is because I wonder if there's going to be a parallel with AI where, okay, you know, like you needed to have an engineer who was
Right. I feel games might give us a bit of a hint of what we might expect at the broader industry.
So that is actually, I would say, that's a brilliant analysis of the situation for not having lived through games and for noticing that. That's impressive. I'll say that first. And I totally agree with that. I've said to people in the past who have asked about sort of AI impact on games in that sense, and I've sort of said as much. I've said like the licensable engine thing kind of was our AI transition already, unfortunately, and I regret to inform you that the news is not probably that positive.
So there are some definite positive things that happen early on because, as you say, it opens up the ability to make games to people who could not have marshaled the technical, sort of the technical staff necessary to produce a competitive engine. And so giving them the ability to make games is a pretty important thing, and it allows a bunch of people to make sort of some artistic expressions that they just wouldn't have been able to do. Early on, that tends to be a net positive, because you just have some more games coming out. Maybe some of them aren't that good, but, you know, some of the existing games aren't that good. That's not that different. But then you get some really cool games coming from some sources that just simply wouldn't have been able to do it. Thumbs up.
Problem is, it rapidly kind of accelerates into this kind of a nasty scenario where you just have massive numbers of releases. And I think at this point we're at the point where, I want to say, Steam games are in the like tens of thousands or hundred thousand per year or something like that. It's so massive that there is no way that your game will be organically noticed anymore, pretty much, period. So essentially it's this really nasty problem where you just have the market flooded with products. And, you know, it used to be that if you made a quality game, if it was fun, people would find it, because there were so few games that someone would play the fun one and tell people about it and it would get purchased, right? Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game. You didn't need a huge marketing budget or anything like that.
Fast forward to today, where we have this sort of massive influx of games. Again, pre-AI, it's just because now the barrier to entry is very low. And you really need a strategy to make sure your game gets found. Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered and become a huge hit? Absolutely. It does still happen once in a while. The chances that you will be that game are like zero. So, you kind of now need a marketing strategy, a real marketing strategy, and going into the market for games without one and expecting to sell any significant number of copies above, you know, maybe a few thousand at best, is really unwise. If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this game.
So, if I'm getting this right, it sounds like the game itself being good is table stakes, but not enough on its own, right? That distribution, marketing, getting people to hear about the game is much more of the differentiator, because there's just too many good games out there, and now they're easier to create.
I think that's exactly right. And that's just the unfortunate reality of it now. Was that a good trade? I don't know. But that's what happened. And so that's where we are in the industry.
Now, there's this other thing that I heard about, which is how new games not only compete with other new games, but with old games as well, right? Like the other day, I spent a few hours playing Death Rally, which is a game from the '90s. And every year, there's more and more good games to play. They all take away from the time that the new games have.
Yes. And that problem will only get worse, because one of the things that the game industry could rely on in the past that is much harder to rely on now is that older games would look dated technologically in ways that consumers cared about. And we have now kind of also crossed the threshold where there is a segment of the market where people really do care about the latest, like ray-traced lighting and all these sorts of things, and, you know, more photorealistic rendering or whatever it is. But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much, in a sense that whatever we're doing today is good enough. So 10 years from now, if the games look much better for some reason, no one will really think of that as a huge differentiator in terms of sales. You go back to 1995, and technological advances were a huge differentiator in terms of sales. You come out with something that, you know, looks good, that takes advantage of the hardware of that day, and boy, did it look cooler and feel more responsive and all these other things as compared to earlier titles, right? And so that's also going to increase the degree to which the thing that you're talking about will happen. I can go play an older game because it doesn't feel obviously dated in an audiovisual way. I don't have to be an appreciator of retro gaming to go play something from 2017. It just looks fine, probably, right?
So, there's that. The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now. People are just logging on and playing Fortnite for several hours or something. That's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game, even. So you have these sort of incumbents: people playing Minecraft, spending their time playing Minecraft, spending their time playing League of Legends or Dota, and that's taking up a huge amount of their time. It's zero sum, right? They can only spend their hours in certain places, just like Netflix or anywhere else. They have to start thinking about, you know, they're competing with everyone else for entertainment hours.
Okay, so I need to ask you this. GTA 6, how is that in 2026 at a time when we
have better tools than we have ever before and we can build software and games faster than before? Like how do games take 10 plus years to develop? Is this some kind of outlier or has AAA game development taking many many years just not changed at all? What do you think is going on here?
So from a player's perspective, I can understand why someone would look at it and go, "Wow, Grand Theft Auto 6 has been in development a long time. How does that make sense?" Or, you know, something like this. From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game. That's not what it is from a product standpoint, right? What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5.
Grand Theft Auto 5 at the time was, if I'm not mistaken, by far the most revenue generating entertainment product in existence. The online part of that game was generating like billions of dollars. And like I said, not a game industry historian, so you know, take what I have to say with a huge grain of salt, but Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will. They were a huge huge live service revenue generating product.
So from Rockstar or Take-Two's perspective, right, Grand Theft Auto 6 is not just let's try to get out the next Grand Theft Auto as soon as we can cuz we'll make money selling that title. It's a we are going to replace the most profitable thing we have ever built, which is still generating a ton of money for us, with a new thing. And you can better believe that they want to make sure that they are going to do that right because the last thing you want to do is ship a new product that cannibalizes something from your old product and then is less revenue generating. Right?
So I'm sure that their planning around Grand Theft Auto 6 is not just about trying to produce a Grand Theft Auto that their fans will love and will buy as the original singleplayer gaming experience that it was. I'm sure they care very deeply about that. Just from a reputational and from an artistic standpoint, I'm sure there's a lot of people on those teams who care about that. But from a business standpoint, I am sure there's also been a tremendous amount of thought and work put into what does the live part look like? And that's a huge undertaking, you know, that I'm sure that they've been planning for quite some time as well.
So, it's a massive massive thing that they're doing here. How well will it succeed? I have no idea. But it is not just a new Grand Theft Auto is I guess the way that I would look at it.
Grand Theft Auto 5, I think, was somewhat of a surprise to them. I don't think they knew it was going to generate that kind of online revenue. I mean, maybe they had hopes, but I don't think they knew that it would be that kind of a massive money maker that it was. And so, this is the first product really where they know they will have the audience. For Red Dead Redemption, they kind of tried to do Red Dead Redemption 2. They did a similar thing where they tried to have the online thing. I don't think it hit nearly as big as Grand Theft Auto.
Grand Theft Auto 6 is the first time they're shipping a true update to what is their flagship. And so, you know, it's equivalent to like a relaunch of Google search or something like that is what they are doing here. And so, you know, I'm sure if I was in charge of that project, I would be sweating bullets. So, I'm sure that they are putting a lot of thought into it. And it's a very massive undertaking, I'm sure.
I'd like to switch gears to software craft. You made this video titled Clean Code, Horrible Performance, an essay/video showing how Uncle Bob Martin's polymorphism based refactoring pattern runs about 1.5 to 15 times slower than a plain table switch version. Can we talk about the responses to this piece?
Well, I guess I can put that in context. So, that is sort of from that course on the Substack. So, it kind of goes with a bunch of other videos that are part of like the Substack thing. I guess the first thing I'd say is I feel like the response to it was very positive. I was kind of surprised. There are plenty of people who didn't like it. Don't get me wrong. It's controversial to be sure, but I was surprised at just how many people were enthusiastic about it as well.
But what I would say is I don't really think it should be so controversial because there's one thing where people want to just use the term clean code to mean code that they like or think is written properly and that's not something you can argue against, right? Because that's just, you know, I probably have a version of what I think is clean code and obviously I don't think that's bad, right? Like it's my idea of what good code looks like. So if your idea of clean code is just whatever you happen to think are good programming practices, I might agree with those programming practices, I don't know.
So in this particular video, I was talking specifically about the things that were advocated that are like very specific things that are said like don't have functions over a particular length or these sorts of things, right? Things should not know the type at runtime or whatever, right? There's all these like kind of rules about it. Preferring polymorphism always, right? If you look at those things, they're kind of just bad programming practices. I don't really know how else to say them. They don't mesh well when you put them together. In isolation, some of them might be fine.
So, for example, if you really prefer lots of small functions, that's actually fine if the compiler can see all those functions and know that it can safely inline them and collapse them as necessary. This is a part a lot of people missed about the video I guess because it's a pretty short video so I didn't explain anything in detail, but a lot of redundant code happens when you have lots of these little tiny functions, and if they're all virtual functions in C++, let's say, the compiler can't know for sure which ones of them are being called and so on, even if you put things like final in them. People have a lot of weird beliefs about how the code works; you can just go do this testing. When you have lots of these little functions, if they're all like statically defined and aren't virtual calls, if they're just known calls or just member functions — when I say static, I kind of mean just known to the translation unit, not external — the compiler can put those together, collapse all of the redundant code and actually produce something reasonable that will run pretty fast out of that. It can also do things like widen the code path if it needs to vectorize to like run in SIMD and stuff.
The compiler has all these options to take what is fundamentally not particularly great code in terms of how you would want it to run at runtime, but it might be able to turn it into that because, you know, optimizing compilers are pretty heroic these days in the sorts of things they can do. If instead you use all of these, you know, things that were recommended, you completely block out the compiler from being able to do those things. Because if it can't tell what it's doing at runtime, if it has to leave open the possibility that you substituted in a different class here or something like that, then you end up in a situation where the compiler can't do any of that work.
And people mistakenly think that this is just because like virtual function calls cost too much or something like that. That's not what it is. It's not the cost of the virtual function call. We could talk about that as a separate thing, because you can analyze that cost as well. It doesn't have much to do with specifically whether it's virtual or not. It has to do with a lot of things like branch prediction and how much stuff is getting pushed on the stack and whatever else, right? But it's the cost of the compiler not being able to do any optimizations. That's the actual cost.
And that cost can be severe. I showed only I think a pretty mild degradation compared to what you would actually see in production if you really had a huge number of things doing this. And I think it landed pretty well. It, you know, it's a very widely viewed video and a lot of people seem to really like it. I thought it would be, you know, probably even more controversial than it was. So, I was pleased with that.
But yeah, I mean, all that stuff remains true today, I guess, is what I'd say. And I think it's good for people to hear because they need to hear opposing viewpoints. I think you can write code that is maintainable and easy to read that doesn't follow those principles in that way and that doesn't have those problems. I don't think you have to do those things. So I think it's worth exploring other options that are still maintainable, that are good code, but that allow the compiler to do the right thing.
What is your take on test-driven development? You know, when you write the test first, then you write the business logic. You've talked a little bit about this as well because it's a practice that used to be super popular, you know, especially when you're building services. Some of those things, especially in the 2000s, kind of got a little bit out of fashion and now it's unclear if it'll come back or not with agents.
I don't have that much of a spicy take on that one. My take is very pragmatic, which is that if you can identify tests that will save time in total, that's usually what I try to emphasize. In other words, if the amount of time it takes to create and maintain the tests will actually save us total development time because they will identify bugs that would be hard for us to find in production or would be very costly to get to if they got out, then great.
And I've used them before. Like I talked about working at RAD Game Tools, I had a regression tester that I ran on like the core libraries there that I had written — you know, they're not really called libraries, but the core routines — to make sure that, you know, anything that I could be testing for our customers I sort of was. And so I think there's good times for testing.
I would say the part that I don't like about test-driven development is the test-driven part. I don't think development should ever be driven by tests. I think tests are a thing that you should be aware of. You should know what your options are for testing and you should make intelligent engineering decisions about tests. Now could that decision be that for this particular project we are going to drive it primarily from the tests? Yes, that could be a decision that you make. But you shouldn't really think of development as something that is primarily test-driven like by default, cuz that might be a very bad decision for some other project where it just ends up costing you more to have done it that way.
So like with most things I would advocate for a pragmatic approach to testing. You should understand the cost of testing, the cost of developing, maintaining the test, and the cost to your codebase if it makes it harder to change your codebase because tests have to be rewritten and you therefore don't make changes you should make. All of that stuff should be in your brain and you should make an intelligent decision about what your testing strategy is. If that decision intelligently made turns out to be we are going to have a lot of testing on this project, that may well be a good decision. I don't think there's an absolute thing you can say about how many tests there should be. Some projects probably shouldn't have very much. Maybe some projects should have a lot. And I think knowing which of those you are doing is part of being a good software engineer is I guess what I would say.
You mentioned being a good software engineer. But before we get into what is a good software engineer, what does good code mean to you specifically?
So good code to me usually means that you have written something that is as straightforward to what the machine actually needs to do to solve the problem as it can be, and also hopefully that you have, I guess I'll say, properly identified ways of breaking it into easily digestible pieces and named those pieces in ways that are easy for someone to understand, especially yourself, because you are very likely to be someone who's going to have to modify it.
So that's the way I tend to code. I try to identify what do I actually need the computer to do. I try to write as simple as possible the thing that will do that and then I try to put that in terms that are, you know, I would say least redundant. So you know, I don't want to see the equation for Euclidean distance scattered throughout my code. I want to have a function that's like compute that distance and I want to use it, right. I want it to then be nicely broken into the pieces that it represents and I want those pieces to be reassemblable properly by the compiler in a way that will produce code that runs very efficiently. Right? And so that's what I'm usually trying to do when I'm trying to program.
And for me, I have never really understood the sort of mentality of there's a difference between code that is like well architected by some principles and code that runs quickly, because in my experience usually the code that is architected properly is also the code that runs quickly. And yes, there is a point where if we decide that something absolutely has to get as close to theoretical maximum as it possibly can, yes, we will start to make that code harder to read and modify because we are now like really over-specializing it for this piece of hardware or whatever. That's true. But that point is like, you know, way out on the curve. It's not the common case.
Most of the time, assuming you just want code that runs pretty darn well on most hardware, the simple readable version of the code is actually very fast. It's only once you think you need to have 27 factories and 8,000 microservices and all these things running that it starts to be this thing that's like good architecture, but also like hard to modify, hard to read, runs slowly, right? All these things. So I tend to think of good code as this nice nexus of: runs pretty darn well, easy to read, easy to maintain, isn't as close to theoretical maximum as it could be but it's close enough, and the paths towards theoretical maximum have not been foreclosed. We left the door open with the way that we wrote it so that if someone really needs to come along and boost its performance, it's set up to do that. Right.
And related to this, what is a good software engineer to you? Is it just someone who writes good code or does it go beyond that?
I would say it really depends on the environment a little bit because I think I've seen a lot of different kinds of good software engineers. And so I would liken it more to, you know, if you want a sports analogy, you'd imagine something more like a baseball team where it's like what's a good baseball player? Well, it's like are we talking about a pitcher or a designated hitter, right? And it changes quite dramatically.
So, there might be some things like, hey, if someone's pleasant to work with and, you know, doesn't goof off all the time and actually gets their work done, those are obviously things that we would say are true of any software engineer. You know, there are some general personality traits that might be positive. But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that, I would say I've seen a couple different kinds. I've seen people who are like the utility infielder. There are people who just like they can identify and go and try to fix a problem and succeed. Even if the codebase is kind of wacky and out there, they're good at getting the lay of the land very quickly, of identifying something that's going on, and they're not afraid to go in and like, okay, this is kind of this code base is kind of
ugly here. It's okay. I'm going to patch around. I'm going to do what I need to do and get things done. That's a great engineer to have around. I've also seen great engineers who are the exact opposite of that. They are just like, I take this one particular problem that we have and eight months later, I have ground out every last thing there is to know about this. And sometimes to the point of producing new algorithms that no one's even known before, right? That are like these, you know, breakthrough things, right? And that's a great software engineer to have on a project if you're going to be having that kind of thing.
And so I've seen a lot of different people that I would consider great software engineers and they aren't all the same person, right? So I think that it's kind of important if you're asking it from the standpoint of, hey, you need to put together a team to go build this project. What's a great software engineer? I would say the best advice you could give someone in that position is think about the roles. Think about what kinds of roles there are going to be here and don't think great software engineer. Think great that role, right? Who is going to be a great pitcher? Who's going to be a great first baseman? Who's going to be a great outfielder? Who's going to be a great this, that, the other thing? Great third base coach, whatever it is, right? And that's what you're trying to put together if you're trying to build a team, to me.
Yeah. So, it's just not one-size-fits-all. But I still want to push you a little bit. What are things that you think are non-negotiable for someone to be a great software engineer? You know, we talked about the things that we talked about, which is a recurring theme with you, just going deeper and deeper and understanding the next and next layer. You know, if you're doing web development, understand React. Once you understand React, understand what's going on in the DOM, go all the way to assembly. Once you've done that, understand how the CPU is doing operations and branch predictions and some of those things. To me that's a skill of curiosity, driving deeper, craft, whatever you call it, you know, there are different ways we could do it. But along these lines, what are those traits that you think no matter what kind of role we're talking about? If you think back on some of the different types of roles that you work with, do you see some overlap, that they all had something?
I would say that it's pretty unusual, I guess, that I can't think of someone I would think of as a great software engineer who didn't know how to read assembly or something. That is true. It might be that having that curiosity about how things work and knowing at some level what's going on is maybe something that's going to be very common to a great software engineer. But I would just underscore the point. The degree to which they are employing that knowledge may vary quite a bit. For some of them that may be their bread and butter and they're doing that all day. For others it's just really a thing where because they know how a computer works, they're not making those stupid architectural decisions that come back to bite us later. Right? And that's great, but they may not really be doing all that much actually at that kind of level or thinking at that level. They're just going like, "Yeah, I know we got to kind of push, okay, this stuff's going to have to be done in batch because I just kind of know that that's, you know, how the machine's going to have to handle it. So, you know, I'll make sure I write the code that way or whatever." Yes. So there's a little bit of that.
The other thing that I would say maybe is actually not being dogmatic about things that they haven't actually themselves proved out is probably a huge one. I find there's a lot of received programming wisdom that's just nonsense. Clearly no one's ever tested it, and if they did they would have found out that there's no actual basis for it. Doesn't necessarily mean it's false. It's just there's no actual tangible way you can demonstrate it. And sometimes you could demonstrate that there are actual concrete downsides to this received wisdom, right? And so in order for it to be received wisdom, you should have to be able to at least demonstrate concrete upsides, which oftentimes cannot be done. So I would say people who actually focus on what works in practice is a huge plus, and you could apply that anywhere, to anything, right?
Not just saying, "Oh, the flavor of the month is that we're writing everything with classes and virtual functions and hierarchies or whatever." It's like, did you actually determine that that results in less code or that the code actually is more maintainable? Did we do any testing to figure out if this is helping us rather than hurting us? And the answer oftentimes is no, or if it was done at all, it was extremely shoddily done and you would not take those results as conclusive in any way. And so it's like being more skeptical about coding practices and actually trying to focus on what is working in practice and what we can demonstrate and measure in some kind of a repeatable way is I think a really great thing for a software engineer to have as well. So people who don't tend to fall prey to that, just like, I watch some presentation and someone at Google says always call memset or never use if statements or whatever it is. If that's the level that you're thinking at, then I probably am not going to put you in that category of really good software engineer, because that's not how it works.
Plus it's not that hard to try these things out or set up or run an experiment. Now, the final topic I wanted to touch on, which I deliberately didn't get into until now, is AI and how it's changing your work. And I'd like to start with that. In the work that you're doing at Molly Rocket with this project that is as yet unreleased, how are you using AI tools, if you're using them at all?
We are not using them at all.
So, you're doing it just like before. You're writing your code. What made you decide to take this path?
Well, we're a little bit different obviously, for two reasons. One is that, like I said, we have sort of two projects here, and the Substack is our primary focus, and this other one is a thing that we're doing because we want to do it. And when you think about that perspective, it's like, well, why did you want to do it? Well, the reason that I want to program things in a game is because I want to program them.
If I just wanted an AI to program them, first of all, we'd probably just go use a licensed engine, right? I wouldn't even bother asking AI to do it. I'd just go get the Unreal Engine, right? Or something like that and so on. So, I think a little bit of that decision is probably not that relevant to your audience because it's more about what do you want to do? Why are you spending this time, right? It's a philosophical question, not a productivity question. So, it's not like I evaluated it and said, I don't think this will save us time, or I have questions about the copyrightability of it or the ethics of it or all the sorts of things that you could rightfully evaluate AI on. It wasn't necessarily that. It's more just like this does not further the goals of the project to use it. So, it kind of was a non-issue at that point. Right.
Stepping out a little bit more to a broader philosophical framework about AI, I guess what I would also say is I think that regardless of what you think will happen with AI in the future, because obviously we don't really have any way to predict what it will look like 10 years from now, it's anyone's guess really. I think there will probably also be at some point a notion of traditional handcrafting that will come into play, because we've seen this in most other times when you automate something. So, if you automate making furniture and you have IKEA or whatever, that doesn't mean that there isn't some weird guy down in the industrial district of your city making crazy wood tables with iron and welding and something. And that's just a thing that people are still doing, and some people want that table. I don't necessarily have an explanation for it and I'm not trying to argue that it has more or less value, but it's just something that happens, right?
And so if I imagine what I want, what I love about computers and what I want to do with computers, and you asked me, move that into some other context, which of these people would you be? My answer is always, I'd be the organic farming guy. I'd be the guy who's making the weird table in the industrial district. I have no interest in managing a division at IKEA. I literally couldn't care less about that, right? And so I think for me another reason why I'm not that interested in pursuing AI is because I would like to be part of whatever the set of people are who are going to keep this traditional craft alive, just because that's something humans do. Not because we're trying to say that that's the right business case, right? If that makes sense.
Yeah. And at this point there's a bit of a tradition, if you will, even if we assume that these machines will do as good or better than cumizit. For like what, 60 plus years, we've only exclusively handwritten software because that's how it got done, right? A lot of us, anyone who started coding before 2023 or the end of 2022, or probably honestly 2024 when these things have gotten decently good, you just wrote a lot of it by hand. Or tab complete still counts.
Yeah. And I guess I would say, again, it's just, you know, why do that, right, is the question. It's like, I don't know why humans do that. Humans do that because it's something humans do, right? Humans like to do things themselves sometimes. You know, people can buy a wool hat trivially, or they can buy whatever, and then someone's out there knitting a hat right now. It's just something humans do. They like to make things by hand sometimes, and at varying levels of handmadeness. You know, some people just buy the wool or buy the pre-made yarn. Some people raise the sheep or whatever and shear it, right? You can go arbitrarily far down. You could find somebody who's going all the way, right? Even further than probably I would ever go if I was in that thing. So, you know, you could imagine someone making their own hardware these days, right? I'm not doing that. And so that's kind of my take on it. So I'm kind of the last one to ask about, you know, AI coding or what you might want to do with it. I really have nothing of value to add.
Yeah. But I am interested in asking you through the lens of the games industry. We touched on game engines arriving, and now so many more people can make games. Not everyone, but it's a lot easier to enter. What are you observing? Most people outside of those who are still handcrafting code because they want to are using these AI coding agents, for two reasons. Either it just makes sense and they realize, well, this thing can now generate code as good as I did, which was a turning point in January. I had that turning point actually myself. Or some are actually just pushed with corporate mandates of, you need to use these tools, and eventually they kind of get on board, whether willingly or unwillingly. But so many folks are having AI write the code for them. You know, they're prompting it, but they're doing it. What do you observe of the effect it's having, from your vantage point? May that be on quality, craftsmanship, on just output, speed, etc. What are you seeing?
I think it's a little too early to assess, to be honest, because, kind of as you pointed out, obviously there's been people who, you know, we might derogatorily call AI shills, who have been saying that it was producing as good code as humans for, you know, two years now or something like that, right?
But in reality, the people whose opinion I would trust more, none of them thought it was really all that usable until much more recently. Right. And so they haven't had very many months to actually be figuring out how to use this thing, or to determine to what extent they can use it, what it's best at, what the workflow looks like that makes it produce the best results. It seems like at the moment we probably need to give it at least another six months, if not another year or something, to let everyone kind of shake out what are actually the best ways to use this thing. I know tons of people in the game industry are using it. So I know that they are doing various things with it. Whether those things are the same sorts of things they will eventually think are the way... like, the things that they're doing right now may be like, oh, that was kind of dumb, you shouldn't have used it that way, you should do this other thing with it and it's way more productive or something. So I feel like it's probably too early to assess. We haven't seen any real obvious, like, oh wow, you know, Fortnite ships once a week now and it's bug free. Nothing particularly interesting has happened in terms of output there, but again, it's been what, like 5 months or something. So it's way too early to see how it actually gets integrated into a reliable process. Right.
Yeah. And I know there are some companies who are now tying up, let's say, agents fixing bugs, but that's only a few months old. The oldest software that's widespread that is written close to 100% by agents is from the labs, OpenAI's Codex and Anthropic's Claude Code, but even there it's been since November, or some parts of it December, so maybe six months. And it's different, right? That is a product they're selling. So I'm not sure we'll know for sure, like, is it truly 100%, how much, you know, there's a marketing angle or not, but there's a self bias there. So I would put those aside in terms of trustworthiness, and you're right that the rest, we just don't really have the information. I'm sure there's so much experimentation, but to your point, it takes time to bake, right, to see the impact.
Most of these things are currently presented as tools, meaning a human has to operate them at least in some way, at least setting it up to do what it's going to do, and therefore you have to give it some time. You know, nobody currently is selling a product where it's just like, oh, just turn this thing on and it will just ship Fortnite by itself forever and you can just get rid of all your engineers. No one's actually selling that product yet, right? We could evaluate that product because we'd be like, did anyone do it? Did it start shipping Fortnite on its own, right? So if it's still something where humans have to kind of figure out how they want to slot it into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick in productivity that would be obvious to an external observer is because it's going to take a while for people to shake that out. Or maybe the AIs need to get a little bit better. Maybe we've got to go through some more update steps or, you know, whatever. I'm not sure. So all that's on the table.
Then there's another possibility, which is that it actually already has worked, but the productivity boost just isn't as big as would be obvious. If people got 10% more productive, that would still be pretty impressive, because it's hard to get a 10% across-the-board uplift. I've said this before on podcasts. I'm like, if you have a tool that can give everyone 10% off of that, that's great. Almost no one would know, right? It's like you can't, it's not really
externally observable that clearly if that's what you got, but it may have happened, right? So, it's really hard for all of those reasons. At some point, if the AIs are really fantastic and people figure out how to use them really well, it should be obvious. It should be like five people are now shipping Fortnite instead of 5,000 or whatever, right? But until that point, it's really hard to know because it's just like, especially if it was small, it'd be hard for us to see.
Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers. One impact it's having is there's this kind of AI fatigue/burnout from software developers who are like, look, I am good at coding. I've always been good at it. I enjoyed the work to various extents. But since this AI thing happened, since the end of the year, beginning of the year, I'm now prompting and now all my code is generated, whether that's corporate mandates or it's just faster. I'm starting to lose my drive, like, why am I here? Anyone could do this.
And I think there's a sense of, I'm using a lot less of what I'm capable of. There's all this pressure from above to be more productive with it, and I'm seeing more and more signs of what do we call it, burnout, AI fatigue, etc., loss of motivation. I don't remember a technology having this widespread impact, like everywhere. I'm hearing from folks at some of the leading, not AI companies per se, but big enough, like database providers who are now hugely into AI and they're powering a lot of the things, traditional companies, modern, everywhere. Have you observed some of this, and would you have any advice or any pointers to folks who are feeling like this right now?
I guess I would say observed, no. Heard about, yes, I guess is what I would say. I have talked to people who have been like, such and such has been having a really hard time with this. I've definitely heard that. Interacted directly with someone? Not currently, no. And part of that is probably largely because most of the people I talk to have a fair amount of latitude with what they do and how they do it. A lot of the people that I talk to on a daily basis are able to make their own decisions about what they want to do with AI and so on. And so I don't necessarily hear from as many people who are going to be in a position where some manager told them this is just what you have to do.
This is very interesting because one thing that keeps coming back, and Armin Ronacher was telling me the same thing on the podcast, is he's observed that autonomy at your work, how autonomous you are, how many decisions you can make on what you work on and how you do your work, the people who have a lot of that are typically like, oh great, I can use this tool for this. But the people who are told beforehand, you're given a ticket or the PM tells you this, they don't have much wiggle room. And now those folks are seeing it way more as a threat because of course, subconsciously or consciously, they're thinking, well, this thing could automate my job. Or it took away even that little effort I had to do. So I wonder if there's a connection here.
I mean, that sounds totally logical, right? If you're somebody with a high degree of autonomy, then when are you going to reach for an AI? Well, whenever there's something that you didn't want to do, right? So kind of by definition, I think at that point you're going to have a much more positive experience with it, because worst case it just doesn't work. In which case, I guess that's not great. You're going to be like, "Ah, this thing was kind of crappy." But assuming that it's able to accelerate some part of that, that was great. It's like, "Hey, I didn't want to do this thing already. I had this AI do it for me and now I have the thing." That's just a positive experience for them, right?
Whereas, yeah, if you're just told, like, you had this thing that you wanted to do, and you were told you can't just do it yourself, you have to do it with the AI, and by the way, we just had layoffs or whatever, a lot of that stuff obviously could totally change your mental reaction, because now it's not you deciding to use an AI because there's something you didn't want to do that you thought the AI could do for you. Now it's you just being told that you're supposed to be using this AI to automate whatever your job used to be. You can see pretty obviously why that would have different psychological effects on people, right?
So I guess it might be just an idea for folks in this situation that you might want to evaluate your current position, or if you're interviewing, your next position, based on how much autonomy you will have. Because the more autonomy you have, the more likely you're going to have control over how you're using this stuff, how much you can experiment, versus being given a mandate that, I don't know, we're expecting you to have this output increase or output change, whatever that is. I wonder if this will re-evaluate what is considered an attractive position, because for example, big tech was considered a great place to work because of high compensation, pretty clear expectations, easy to understand career advancement. But now they're the ones who are starting to measure your AI usage, which is giving a bit of a handcuff of what we're expecting you to do, or there might be mass layoffs, which again you have no control over, right? Or inside of Meta, reassignments of like, you will now do labeling for X months.
I mean, you could sort of think of, could we coin the phrase: are you using an AI to do your job, or is an AI using you to do your job, right? Because at some point it definitely felt like Meta, for example, from your reports on it, and I have seen the same thing said by other people, so it does not sound like a one-source kind of a thing. It sounds like this was kind of just accepted as fact that they were using you as AI training, right? You're just there to train the AI to do it so that we don't need you anymore, right? And so thinking about that from a perspective of choosing your job, it does make some sense if you have any latitude, right? But yeah.
As closing, what are one or two books that you would recommend that had an impact on you?
I'm gonna have a hot take here, if I might, because it's sort of a push I've been on recently. I don't think people should necessarily take a book recommendation from me. I want to recommend that people read a paper. I'm trying to get more people to just read papers, because I realized I read a ton of papers. I am constantly reading papers on things that I am interested in. If I'm going to go do some programming in an area that I haven't done before, I will read a ton of papers. I'll crawl the references on papers. I'll read a survey and go gather all those references and read those references and crawl them back. And I find that I learn a ton that way. And I feel like a lot of programmers just don't do that.
And so my recommendation would be, you don't have to read a specific paper. I'm not going to give you a specific paper to read. Just think about the domain you're programming in. Do a search on Google Scholar for some part of that that you're interested in. Try reading a paper, following the references, see what you think. I think it's a great thing to do. And I get a tremendous amount of not just enjoyment from the education of it, but also just more knowledge about what I'm doing pretty much every time I do this, even if it's just to learn a little bit more about the historical record of how things got discovered. But a lot of times I learn about whole new techniques I just was not aware of, because there's way too much out there for any one person to know.
And again, since I don't currently use AI in my workflow, I couldn't say. But my assumption would be that AIs would also be very good at helping you find some papers to read if you were interested, because chewing through a lot of the technical record is something that they do. And so maybe you could even ask your favorite AI to suggest a paper that you might like based on some things that you tell it. I don't know if they're good at that, but I'm guessing that's something they could do.
Casey, thanks a bunch for this conversation. This was great.
Thanks so much for having me. It's been a pleasure.
I've been wanting to talk about performance with Casey for such a long time, and I'm glad that we finally made it happen. I kind of wish the industry had more people as excited and interested in high-performance code as Casey is. If you made it to the end of this episode, you might just be one of them. I appreciate that Casey did not beat around the bush. If you care about performance, you want to be able to read assembly and spend some time reading it. Reading assembly is several times easier than writing it. If you can read assembly, you can see what's happening at the machine level. And it's a lot easier to understand, for example, why a programming language like Python is much slower than something like Rust or C when you see the assembly code for simple operations.
I was chuckling when Casey talked about these blog posts about how we rewrote our services in a new language and got a 10x performance improvement, and how those rewrites are usually not about the new language but fixing the architecture that caused the performance issues to start with.
And although we did not talk much about AI, I found it amusing for Casey to say that the games industry had its AI moment years ago when game engines became accessible to pretty much anyone wanting to build a game. Before, large teams were needed to build both a game engine and a game, and now teams of one or two can create full-blown games. After a brief spike of positive effects with lots of new good games released, games have flooded the market in such great number that it's now impossible for a new game to become a hit organically. So marketing and distribution become mandatory even for great games.
For more deep dives related to game development and performant software, check out the linked Pragmatic Engineer deep dives on these topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. And a big thank you if you also leave a rating on the show. Appreciate it and see you in the next one.
Article published