Sam Newman on Microservices as a Last Resort, Resilient Distributed Systems, and Staying Connected to Systems Built with AI
The Pragmatic Engineer
Sam Newman wrote one of the most widely read books on microservices, yet in this Pragmatic Engineer conversation he calls them "an architecture of last resort." The episode explains why, and then moves to his new book, Building Resilient Distributed Systems. In it he reduces the difficulty of distributed systems to three rules, makes the case that observability and business context come before any technical mitigation, and borrows four concepts from resilience engineering that go well beyond keeping servers up. The last part of the conversation covers AI. Newman is cautious there: he finds it useful at the tactical level, worries about the economics underneath it, and argues that modular architecture is how teams can experiment with AI without losing their understanding of what they are building.
From Fortran 77 to regular expressions
Newman grew up in the UK with an Acorn Electron and later a ZX Spectrum +3, and studied software engineering at De Montfort University. His course was a "sandwich" degree: two years of study, a paid year working in industry, then a final year. He says the industry year made things "click." He spent it at GEC Alsthom (later Alstom), which wrote software for the European Space Agency and for gas and steam turbines.
In his first week there he was given a Fortran 77 codebase. Machine time had been expensive and the team had worried about accidental infinite loops, so every one of the roughly 8,000 files capped its for loops at about 99,999 iterations and aborted past that. The models had since grown and the computers had become more powerful, so the limits had to go up. His boss estimated two weeks of fiddly manual edits, made harder by Fortran's punch-card-era line length limits. Then the boss slid the O'Reilly sed & awk book across the desk. Newman spent two days learning regular expressions and wrote an automated refactoring that respected the line lengths. He calls it "the closest I've ever come to feeling like a god."
There is a coda. Years later, near the end of his time at Thoughtworks, he told the story to some graduates, and one asked why he hadn't replaced the limit with a constant so it could be changed once in future. Newman says he was "learning once again." He tells the story to illustrate how the industry keeps humbling you. He also links it to something developers are struggling with now: as people move further from the code, they are trying to work out where that "fizzy dopamine hit" of programming comes from. Newman says it matters less to him because he has worked mostly in architecture and infrastructure for a long time. He notes that his son, also a programmer, has entered the field at "an interesting time."
Thoughtworks and the idea that "this stuff's all up for grabs"
Newman joined Thoughtworks when it had just under 400 people and left when it had about 4,000. It was primarily an Extreme Programming shop, influenced by Kent Beck's XP, with a London flavor from the Extreme Tuesday Club and the "London school" of test-driven development. For that school he points to Growing Object-Oriented Software, Guided by Tests by his friends Steve Freeman and Nat Pryce, which he calls "one of the worst titled books in existence." He chose Thoughtworks over a bank partly because the bank could tell him exactly what he'd be doing and the Thoughtworks recruiter couldn't. That uncertainty appealed to him.
Continuous integration and TDD were being talked about at the time, but the tooling mostly didn't exist. Newman started building build-pipeline tooling around 2004. Dave Farley was his first tech lead, and he interviewed Jez Humble as his own backfill on a project at AOL. He also built two replacements for CruiseControl and co-created DbDeploy, which he describes as the first database refactoring tool, because nothing like it existed yet. What he took from that period was the belief that "these things aren't set in stone." In 2007 he spent about a year and a half at Google as a Thoughtworks employee, alongside people from Industrial Logic and Pivotal, teaching Googlers to write automated tests and testable code. He remembers conversations with people at Yahoo and in large parts of Google who did not believe functional automated testing of websites was possible.
The host observed that consultancies are rarely seen as sources of innovation and that Thoughtworks seems to be an exception. Newman qualified this. He says the innovation happened under a different set of constraints. Few Thoughtworks alumni became strong startup founders, though many are good at scale-ups. As a consultancy doing mostly co-sourced delivery, its working conditions depended on the client. Most engagements began by convincing clients that testing and pairing were worthwhile, before TDD even came up.
That leads to an idea he credits to Martin Fowler: "programming is not typing." A client who objects that pairing halves output believes programming is typing, and the real conversation is about what programming actually is. The host noted that many CEOs now reason the same way about LLMs: code was historically half the work, and LLMs produce code. Newman answered with Peter Naur's view of a program as a theory that lives in the minds of developers working together, with code as a side effect of that collaboration. Product development extends the same idea. Through a theory-of-constraints lens, he adds, speeding up one stage exposes friction elsewhere, and coding was often not the bottleneck anyway. The real problem was how coding was done, for example with delayed feedback cycles. He cites Margaret-Anne Storey drawing on Naur and Kent Beck's point that software design is an exercise in human relationships. AI doesn't change that, he says. Programming is "the collective art of creating software and it's not just developers that do that."
After Thoughtworks, Newman did three startups. None worked out ("don't ask me for advice"). He left partly to avoid becoming "institutionalized": he had tried to change things that the company didn't want changed. He has been independent ever since, and jokes that the problem with working for yourself is the boss.
Where "microservices" came from
Newman says James Lewis coined the term, not Martin Fowler. Around 2011–2012, Lewis was talking to companies building service-oriented architectures where services could be replaced quickly. In the background was what Newman calls a "guerrilla SOA" movement: a reaction against WSDL and SOAP ("never has the word simple been less well used") and against centralized middleware, toward REST and "smart endpoints and dumb pipes," pushed by people such as Jim Webber and Ian Robinson. At a Thoughtworks architecture symposium in the Lake District, Lewis pitched the idea as "micro apps." Someone among the roughly ten people present, most of them hung over, said they were more like services. Lewis and Fowler then wrote the first article on Fowler's site.
Newman came to the idea from a different direction. He had spent nearly ten years on continuous delivery and was running Thoughtworks' continuous delivery practice for Europe, focused on "the final mile" to production. He kept running into an architectural obstacle: you can't change things frequently if what you want to change is tightly integrated with everything else. He gave a talk called "Designing for Rapid Release" in 2012. When Lewis put a name and patterns to the idea, Newman recognized it, and he had always wanted to write an O'Reilly "animal book." For him, microservices were an extension of continuous delivery, a style only practical because of CD and DevOps. The underlying ideas are older. He mentions Erlang and calls the whole effort "an exercise in smuggling through concepts from the 1970s": information hiding, coupling, cohesion.
When he wrote the first book, few companies spoke publicly about this style. He drew on Ben Christensen at Netflix and on REA (realestate.com.au) in Australia. Netflix originally called its approach "fine-grained SOA," and Newman understands the same was true at Uber and elsewhere. Then came rebadging: everyone started calling their SOA "microservices." Newman treats this as predictable semantic diffusion and has kept presenting microservices as "an opinionated style of SOA."
Two opinions, and why microservices are a last resort
He jokes that his definition wins because he wrote the books. It rests on two opinions. The first and most important is independent deployability: the unit of deployment and change is a single service boundary, not several services released together. The second, which he says has more exceptions, is that service boundaries should by default follow the business domain. Many SOA systems instead used technical decomposition into layers.
The host described Uber from 2016. Services were independently deployable with increasingly strongly typed interfaces, but boundaries were rarely domain-driven. Inexperienced developers spun up services, other teams came to depend on them, and years later audits of the hot path found obscure services that needed to be top-tier. The host relayed what former Uber CTO Thuan Pham said on an earlier episode: Uber never set out to have so many services. The monolith was slowing delivery, and the only enforceable way to stop it growing was a mandate that everything new had to be its own service. The proliferation was a side effect. Years later Uber has looked at merging services by domain, but the public story at the time was just "Uber has thousands of microservices."
Newman says this is what microservice adoption really looks like, and it is why he calls them an architecture of last resort. They push you toward aggressive distribution of state and everything else, so you need a good reason. The best reason, in his view, is that you want an organization where teams can act independently, and microservices are a "fantastically good fit" for that. But they are only a means to an end. The common mistake is to adopt microservices and expect "other magic stuff" to happen. Most enterprise clients he has worked with want exactly that autonomy: many developers, a lot of middle management, little work getting done.
He also draws a distinction between tech companies (he still calls them "dotcoms") and enterprises. Domains at tech companies are, in his view, very simple by comparison, and business processes evolve together with their digital implementation. Enterprises such as banks built their first digital products by encoding paper-based processes that are decades old. Eric Evans' Domain-Driven Design (2004) reflected that era's work of modeling existing non-digital processes. The host asked whether the same split will appear with AI, between AI-native startups and established companies retrofitting AI onto how they already work. Newman says it is already happening, and that enterprises rarely succeed at the first-generation attempts because "the corporate DNA will just crush that stuff from within."
Is the spec becoming the source of truth?
Newman frames the AI question as one of abstraction levels. We moved from machine code to assembly to general-purpose languages, and each time the mental model of the program was evolved at the most effective level. Specs might be the next step, perhaps with product people taking a bigger part in that conversation. His answer to whether we are ready is "no, not everywhere."
The host noted that at many tech companies, the code is where the truth about the system lives. Newman's first response, channeling Charity Majors, was that "production is truth." On what the system should do, he points to Birgitta Böckeler's article on Martin Fowler's site about spec-driven development, which separates spec-first from spec-anchored approaches. Some teams use tools like Kiro or Spec Kit to write a spec, let the LLM implement it, and throw the spec away. Others keep the spec and update it as the feature changes. Some now never look at the code. Newman doesn't know whether the end of that spectrum is the right destination. Much more needs answering, he says, before we have the kind of confidence in a spec that we have about what a line of Java does.
Collaboration and tacit knowledge are central here. The host noted that code review used to spread knowledge and reduce bus-factor risk, and that some companies are dropping it because of the volume of AI-generated code. Newman added that promiscuous pairing, with daily rotation, gave everyone shared context. He cites Chris Ford of Thoughtworks Spain, who says the biggest problem in AI-driven delivery is tacit knowledge: the feeling during review that something "doesn't really feel right" without quite knowing why. To work at the spec level and be hands-off, people would need to unpack that knowledge and get it into the spec.
He sees a related problem in how many people work today. Individuals go off in different directions with their AI assistants. He has seen people describe this as pairing, and objects that the assistant is "not a human." The shared mental model breaks down. AI was supposed to free developers from drudgery, but he says most are doing more work, switching context more, and losing the big picture. If teams collaborate on specs, the spec could become a source of truth, probably shared with people's heads, and teams might shrink. He cites a well-regarded firm that announced it was ending human code review so it could spend more time on design documents. He is comfortable with that because they are moving the thinking elsewhere, not skipping it. Teams that stop looking at the code without putting that effort somewhere else "deserve everything that's going to happen to you."
At a half-day O'Reilly online conference on software delivery that he hosted, four of roughly six speakers covered variants of spec-driven development. Newman reports that all of them said the same thing: you can't do it unless you can define what good looks like. One small team was feeding its OpenTelemetry data directly into its agent. He thinks this pushes developers to think beyond the narrow functional acceptance criteria common in user stories. "The expertise is still needed, it's just needed at a different place."
Why write Building Resilient Distributed Systems
Years ago, at O'Reilly's Velocity conference, someone pointed Newman to a resilience engineering paper by David Woods. He found it "utterly baffling" because it wasn't written for software people, and he says he isn't even a computer scientist. From around 2015–2016 he reread it every six months or so and read summaries by others until the ideas made sense. Meanwhile, many people were building microservice architectures even though, as he puts it, most of his book tells you not to. More people were being exposed to "the horror, pain, and suffering of a distributed system," and he wanted to give them a "rubber ring."
The book starts with tractable foundations. After the introduction, chapter two covers observability and OpenTelemetry, and chapter three covers timeouts. It then builds toward sociotechnical systems and the difference between safety management and resilience engineering. Newman includes real examples from large companies and from his own mistakes. A recurring theme is that things will go wrong. Resilience engineering is about making as many things go right as possible, not all of them, and he says it requires being honest about mistakes and sharing them.
That led to public incident reports. Newman separates marketing and PR from genuine post-incident review. A company might say little publicly and still run an excellent internal process, while some published reports "feel more marketing." He notes that regulation such as DORA in Europe now requires some disclosure, and argues there is a duty of care to inform users. He calls Cloudflare's write-ups some of the most "self-flagellating" around. The host added that Cloudflare publishes them quickly despite being publicly traded, which undercuts the claim that listed companies can't. Newman says much of how an organization is perceived comes down to how it handles mistakes.
The three rules of distributed systems
Newman says he could never remember the eight fallacies of distributed computing, so he reduced the root causes to three rules.
The first rule is that information cannot travel instantaneously from A to B. Developers control some of this, such as payload size and serialization technology, but not cabling, equipment, or distance. On the public internet, "BGP has a bad day" and packets take the wrong route. Something can look instantaneous without being so. (When quantum physicists raise entanglement, he replies that it only works for reads and isn't a networking protocol.)
The second rule is that sometimes the thing you want to talk to isn't there. You can reduce the odds, for example by running multiple instances behind a load balancer, but the load balancer can fail too. His examples: a SAN where one of two pairs of SSDs in its caching layer caught fire, producing strange latency graphs depending on which path requests took, and rabbits that moved into ducting between office buildings and ate a network cable. "I can't control for rabbits."
The third rule, which he credits a former colleague with pointing out he'd missed, is that resource pools are not infinite: CPU, memory, I/O, queues. In his experience most outages come down to a resource being saturated somewhere, often in a chain. Something goes down, a retry storm follows, connections are starved, and so on. His advice is to hold onto these three rules before worrying about Byzantine consensus, CAP, or PACELC, because most problems become easier once you do.
Observability as the precondition for everything else
Newman defines observability as a property of a system: how easily you can understand its behavior from external signals. A single program on a machine you can access gives rich signals almost for free. You can attach a debugger, take a core dump or a JVM thread dump, and CPU stuck at 100% is meaningful. In a distributed system those signals are harder to collect, can't be correlated without extra work, and individually say little about the system as a whole. That work is much easier to do from the start than to retrofit. He welcomes OpenTelemetry as a vendor-agnostic standard (adding that Honeycomb is the best and that Charity Majors pays him only in stickers). His practical point is that you can't even set a sensible timeout without knowing the normal distribution of response times and the success and failure rates.
The host asked about the gap between technical metrics and whether a user's button press actually works, for example whether an Uber ride request completes within about five seconds or hangs on a spinner. Newman's answer is to treat the raw telemetry as a stream of structured events and build higher-level views from them. Traces stitch spans together: a span is an activity, which can contain child spans such as a database call or a call to another service, linked by a shared trace ID and timing. Most organizations once had to build their own tracing, and projects like Zipkin were attempts to recreate what Google had. Above that sit service level indicators, which judge each event good or bad (an error or a slow response is bad), and service level objectives, which are commitments such as a target percentage of successful ride requests within a given time.
He connects this to how much resilience a system needs. Nobody says they don't want their system to be resilient, but resilience is a matter of degree, like security, and you need the equivalent of a threat model. Monzo, for instance, runs a constantly available stand-in bank with a stripped-down feature set (freezing cards, withdrawing cash, checking balances). It is a complete rewrite with no shared code, runs on a different cloud vendor, and by his account costs about a tenth of the full stack's cloud bill. That is right for Monzo, he says, and "there's no point you starting there." The host stressed that these decisions require understanding the costs: double cloud bills, extra latency from replicated writes, build time. Newman added that engineers have to communicate in terms stakeholders understand. Many expert product owners won't know what "percentile latency" means.
Targets also change over time. Newman says a resilient system "changes and evolves," and a system that doesn't is brittle. After the AWS us-east-1 outage late last year, some companies that had previously decided against going multi-region reconsidered, which he thinks is fine. He doesn't assume the engineers at Zoom, which was severely degraded, were "idiots." He thinks they probably weighed the cost of multi-region and judged it not worth it, and multi-cloud is a different conversation again. What matters is that the decision was made consciously and matches what users expect. That is why he contrasts SLAs with SLOs: the SLA is what you get sued for, and customers won't be happy if you only ever meet it.
Idempotency: keys versus fingerprints
Newman defines an idempotent operation as one that can be applied multiple times with the effect happening only once. A lift button is idempotent. A toggle light switch isn't. In computing, reads and typically deletes are idempotent. His main example is a payment. A service sends "pay Sam £100" and gets no response. Either the payment happened and the response was lost, or it never happened, because the request didn't arrive or the server crashed partway through. The host gave the familiar airline checkout warning not to refresh or resubmit. Because of rules one and two, the natural response is to retry, and without idempotency a retry can mean paying twice.
The preferred fix is an idempotency key: a unique ID, usually a UUID, generated client-side without an extra round trip and with a very low chance of collision. The server checks whether it has already processed that key and returns "processed" however many times it is asked. Newman notes that AWS, Stripe, and PayPal use this scheme. The drawback is that both client and server must support it as a required field, which makes retrofitting painful. His advice: anything that needs to be idempotent should use keys from the start.
The retrofit option is a fingerprint: the server hashes the request fields and treats a matching hash as a duplicate. This produces false positives where legitimate requests are rejected. (In the conversation he calls these false negatives, describing legitimate repeat requests being rejected.) He describes a Danish payment system where buying something for exactly 40 kroner twice in the same shop within about five minutes gets the second transaction rejected, while 39 or 41 would go through. The approach depends on identical legitimate requests being rare. It fails when they are normal, such as provisioning 20 identical EC2 instances in a loop. Keeping fingerprints only for a window (five minutes, ten, an hour) reduces the problem but can let duplicates through. The two approaches also work well together. Many public APIs fingerprint requests so that a retry reusing an idempotency key with changed parameters can be rejected as invalid. He calls the ideas simple, with nuances in implementation such as server-side storage.
Thundering herds and why mitigations depend on the cause
Newman uses "thundering herd" broadly, for any mismatch between requests and capacity, which is rule three again. His categories:
- Malicious: DDoS attacks, which come at short notice with limited options. His answer is to use Fastly or Cloudflare, and the book explains the details.
- Self-inflicted: for example, cache collapse. A restarted cache produces a wave of misses that overwhelms an origin that relied on the cache to reduce load. Mitigations include snapshotting the cache to disk, and some companies he knows turn services off until the cache refills.
- Retry storms: his example is Square's 2017 outage. Multipass, its multi-factor authentication system, read data from Redis used as a database. After a restart, clients retried failed Redis calls with no delay up to a hard-coded 500 times, hammering Redis repeatedly. Square fixed it with a saner retry policy and externalized configuration.
- Success: legitimate external load, as with early Twitter and HealthCare.gov. Newman says HealthCare.gov went live in 36 states at once, and in hindsight starting with one state might have helped. Constrained sign-ups, invite codes, and usage limits are the tools here, and Twitter did use some of them.
The point is that similar-looking failures need different fixes. Elastic scaling suits the "wave of success," where you might gain a thousand customers and keep a hundred, so you scale up and then back down. Against DDoS it turns the attack into a "distributed denial of cash" attack: you are still down, and now you are paying for nodes as well. "You can't just scale yourself out of trouble." Knowing how close you are to the red line requires production data, which brings him back to observability.
Fail open or fail closed is a business decision
Newman's chapter on formal ideas covers CAP and PACELC (and mentions harvest and yield). He argues that in practice these come down to business context. An e-commerce site that isn't sure an item is in stock will usually sell it anyway, because it can back-order or refund, and refusing loses a sale. Ticket sales for a Taylor Swift concert are different. If the link to the ticketing system is down, the vendor won't sell, because a ticket can't be back-ordered and the customer may already have booked flights to Portugal. The technical problem is the same, but the decision differs because the consequences differ.
The host offered an Uber example. When it was unclear whether a payment had gone through, Uber failed open early on to favor growth, then moved to failing closed as it pursued profitability. Newman added that this isn't one decision for the whole system. It is made feature by feature, and could even depend on whether a ride costs €10 or €1,000. His practical guidance: when you can't reach part of your system (rule two), try a technical answer like a careful retry. If that fails, before reaching for CAP theory, "go have a conversation with a product owner."
Four concepts from resilience engineering
The book's framework comes from David Woods, whose paper had first baffled Newman.
Robustness is absorbing known perturbations, for example Kubernetes replacing a failed pod without anyone noticing. Newman says most engineers focus only on this. Its nasty side is that robustness measures add complexity and so create new failure modes. Bring in Kubernetes to handle container failure, and now Kubernetes can fail. He says this is not solvable, only something to be aware of, and it is one reason companies decline multi-region or multi-cloud setups.
Rebound is how quickly you recover, covered by time to detection and time to recovery and by his disaster recovery chapter. It requires humility: admitting the robust design might be wrong. The host mentioned Netflix's Chaos Monkey and the need to repeat drills as systems change. Newman recommended Uptime Labs, a partly London-based company offering browser-based incident training.
Graceful extensibility is dealing with surprise, which is "now not about computers, it's about humans." You couldn't have predicted the rabbits. Command-and-control organizations with narrow job descriptions are structured around the known, so in his view they handle surprise badly. Drills help, especially when outsiders run them, as with Google SRE's "Wheel of Misfortune," where a physical wheel picks a scenario such as a state-sponsored DNS poisoning attack. The host recalled Meta's BGP/DNS outage, where engineers reportedly had to get physically into a building whose door security depended on the downed domain. Because Meta shared the details, other companies could check whether they had the same dependency.
Sustained adaptability is the continuous capacity to learn and adapt the system over the long term, including when nothing is going wrong. Newman recommends reading other companies' incident reports and points to The VOID's collection of hundreds of them, though he notes it hasn't been updated for a while. Both of the last two concepts depend on people and culture. Psychological safety is "absolutely required," because if people can't raise issues or challenge safely, surprises go unreported and lessons stay hidden. The book's final three chapters cover these human and social factors.
AI and resilience: useful at the micro level, worrying at the macro level
Asked whether AI sits outside these principles, Newman said it doesn't. At the micro level he sees useful applications, such as correlation analysis and post-incident support, and mentions incident.io's work. At the macro level, he says "none of it stacks up financially." He cites what he calls a conservative economists' analysis: OpenAI, Anthropic, and the hyperscalers would need about $2.6 trillion in revenue by 2030 to justify their capital spending, while the entire software market is about $1.4 trillion. He says he isn't smart enough to know how that resolves.
What he can do is hedge. That means flexibility on models, including open-weight models from different vendors. It means multi-vendor access: he says OpenAI and Anthropic's own services have poor availability, so people he knows reach the same models through Google, Amazon, or Microsoft as primary or fallback. And it means breaking single agents into workflows so that pieces can later be replaced with deterministic code that runs more efficiently. On nondeterminism, he had argued that the shift from deterministic to nondeterministic software was a new challenge. Charity Majors corrected him: distributed systems, with their people, rabbits, and fires, were never deterministic. He now sees AI as amplifying that rather than starting something new.
What we misunderstand about LLMs
Newman thinks the tech world is "quite naive about AI and fundamentally misunderstands what an LLM is." Why did the LLM delete your database? Because it has no concept of causality. A person thinking about pushing a glass off a table reasons about carpet versus hard floor. LLMs don't. World models, which he says arguably predate LLMs, could in principle reason that X leads to Y and Y is forbidden. That is why he thinks guardrails around LLMs are not the right long-term solution. Because LLMs seem so smart, people overestimate them. An open question for him is where to let the LLM act and where to write deterministic code. There is "a lot to be said" for having an agent call a script you know works.
The host observed that models increasingly default to writing code for tasks like comparisons. Newman said this works because it is tractable: generated code can be validated by running it. Other areas are much worse. He used AI very little for his book and not at all for the writing. He used NotebookLM for research, then clicked every link it gave him to check the claims. He was comfortable with that because it got through far more papers, including NIST papers, than he could have read himself.
Cognitive debt and cognitive surrender
Newman separates two ideas. Cognitive debt, as he describes it, is the erosion of critical thinking and reasoning that can come with heavy AI use. He mentions studies on chatbot use and an article by Margaret-Anne Storey that frames it as an analogue to technical debt. When everyone goes off in different directions with AI, the shared mental model of the system breaks down, and that becomes a debt.
Cognitive surrender is "basically LGTM" applied to whatever the AI says. Newman worked on Google tooling where "LGTM" was literally pattern-matched as approval, and says the phrase has come to mean "I didn't really look at it." Now AI writes documents nobody reads and AI reads documents no person wrote. He uses an oncology example to show how workflow design matters. An AI that flags "you might have missed this" on scans makes the specialist better. An AI that says "here are the scans I've reviewed, sign off on them" turns the expert into an approver, often after other specialists have been let go, with one person as the funnel. The original promise was to free people from drudgery for critical thinking. Instead, he says, studies show more context switching, less time to think, and longer hours. He doesn't blame the technology itself, but how we are using it.
The host quoted an earlier post of his: "I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space." Newman explained that platform teams exist because Kubernetes isn't developer-friendly, and he had hoped for a "complexity collapse," with newer technologies simplifying programming models. Instead everyone rushed to "AI all the things," adding new complexity on top of the old, and someone still has to maintain all of it.
Modules as the place where humans stay in the loop
Newman's response is modularity, whether the modules are deployed separately or not. Humans think carefully about the modules and the connections between them, and the AI can "go hog wild" inside boundaries that are easy to change. He sees Shopify as an example: it kept a monolith and stayed successful by structuring it as a modular, domain-driven architecture whose structure you can zoom out and understand.
Modularity also means teams don't have to make an all-or-nothing choice about "software factories" (sometimes called dark factories). One module could be run factory-style, with no one reading the code but with a specified interface and verification of what good looks like. Good candidates would be modules with strong test coverage, off the latency-critical path, or with good observability and well-defined SLOs, perhaps with extra SLOs added. More critical areas can stay collaborative with close human involvement until trust exists. Treating the whole system as one black box forces a single decision, which he thinks is unhealthy. He knows companies already all-in on factories and stresses that not everyone should be. His aim is to lower the barrier to trying these ideas safely. He notes that real-world dark factories are rare and mostly make components, not finished products.
His suggested starting point is specs: tools like Kiro, Tessl, or Spec Kit, with Böckeler's article as an on-ramp, because a factory needs a definition of good as input. Then think about verifying the output. He follows Kevlin Henney, who dislikes "non-functional requirements" (if something is non-functional, it doesn't work) and instead splits requirements into developer requirements (such as maintainability), operational requirements (latency, uptime), and business requirements (does the feature work). If you can't say what good looks like on all three, "this isn't for you," though you can still use specs. Newman admits this could be "model-driven development all over again," which he gets "very angry about," but hopes it might work this time.
Learning architecture
For engineers who want to build architectural judgment, Newman recommends Fundamentals of Software Architecture by Neal Ford and Mark Richards (noting they are friends and that he doesn't agree with everything in it) and Vlad Khononov's book on balancing coupling, which he calls fantastic. He mentions that Khononov has also built Claude skills based on its ideas, and that his DDD book is good. For Newman the core idea is still information hiding, which he says was the real subject he "smuggled" into the second edition of Building Microservices. He suggests reading David Parnas's original papers from 1971–72.
You can start modularizing within an existing codebase. The host argued there is little excuse not to experiment with AI-assisted refactoring. Newman agreed, with conditions: have a decent test suite first, and if you don't, start there. Statically typed languages already have strong IDE refactoring, and modern dynamic languages like Python and Ruby are much easier to refactor with LLMs than they used to be. The goal is for the code, and perhaps later the specs, to represent the mental model and not leave the architecture only in someone's head. He cites a pattern from George Fairbanks, an "architecturally evident coding style," and says programming languages generally have weak concepts of modules, so developers must do extra work to make architecture explicit. That is also why he likes spending time in production. When module boundaries are process boundaries, you can read the architecture off the network. You shouldn't have to, but "production is truth."
Closing
Newman points readers to the book in print and on the O'Reilly platform, to his videos on Dave Farley's Modern Software Engineering channel, and to samnewman.io for his talks, training, and consulting. Within the conversation itself, the questions he leaves open are whether specs can become a trustworthy source of truth, whether software factories will work or repeat model-driven development, and how the AI market's economics will resolve. His practical advice for all three is the same: define what good looks like, verify it in production, keep humans responsible for the boundaries, and hedge against what you can't control.
I describe microservices as being an architecture of last resort.
Really?
Yeah. Absolutely. What a lot of people get wrong, I think, is they adopt microservices and they think other magic stuff's going to happen.
You talk about the three rules of distributed systems. What are these three rules?
The first is that you can't send information instantaneously from point A to point B. Rule number two is sometimes the thing you want to talk to isn't there. And the third rule is that resource pools are not infinite. The vast majority of the time when you have a system outage, it's just down to that: some resource has been totally saturated somewhere.
What are the things that we know that we do not know about AI?
I think the tech world in general is quite naive about AI and fundamentally misunderstand.
Sam Newman is the author who wrote probably the most widely read book on microservices. And yet, as you heard, he said that microservices are an architecture of last resort. But why? Sam was in the room when the term microservices was coined. I find him to be one of the most thoughtful engineers on this topic, as well as on resilient systems, and his latest book, Building Resilient Distributed Systems, is out now.
Today we cover specs versus code as a source of truth and why Sam says that production is truth. Idempotency keys versus fingerprints and why you should avoid idempotency keys when possible. The four dimensions of resilience: robustness, rebound, graceful extensibility, and sustained adaptability. Cognitive debt and cognitive surrender, and Sam's advice on how to better use AI to avoid these, and many more. If you want to understand what it takes to build resilient systems from an expert who has been leading this topic for 10-plus years, this episode is for you.
This episode is presented by turbopuffer, the AI retrieval layer built on object storage. It's fast, cheap, and extremely scalable. Anthropic, Cognition, Notion, and many others are happy customers.
Before we start, let me mention Entire, our season sponsor, and a product and team I'm very bullish on. You probably already know that Git hosting has a problem: AI agents. Git is increasingly becoming a bottleneck for modern agent-heavy software development. Engineers like you or me are creating more code with agents. Those agents push more code, and most of us are running parallel agents, which push way more code. GitHub is clearly struggling to keep up and has frequent outages.
So, what's the solution? Entire was founded by GitHub's last CEO, Thomas Dohmke, who, by the way, was a guest on the podcast when he was still at GitHub. And Thomas rebuilt Git hosting for the agentic era from scratch. Entire was built to be very fast and to have your repos regionally close to you to reduce latency. And this allows for fleets of agents to push in parallel. And Entire's performance is next level. The repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market.
When GitHub is down, you can still keep working, and you don't even need to migrate away from GitHub. You just sign up to Entire, and the platform mirrors your repo. And here's a neat thing. Have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself, oftentimes. Entire captures all the prompt history with your agent right inside of the repo, easy to check back. And it's got a pretty innovative UI to show all of this.
If you're looking for Git hosting that works even when GitHub is down, head to entire.io/pragmatic, install the CLI and mirror your repo with a click. I've already done it, and I love it. And did I mention that it works with any agent and it's open source?
Sam, it's great to have you on the podcast.
Thank you so much for having me. Happy to be here. Really happy to be here. Big fan of the show, as they say. I'm not going to lie, I've become a bit of a millennial, and I think I watch a lot of shorts rather than watching the whole things now, and becoming my son.
Millennial is the new boomer, apparently, I've learned.
God, I was happy enough being Gen Y. Now I'm wherever I am now.
Let's go back to the beginning. You've been known for a lot of your books, your takes on resilient systems, microservices. Before that, you worked at Thoughtworks, but let's rewind to the very beginning. How did you get interested in tech and computers?
I'm quite fortunate living in the UK. We had, at places, access to computers quite early on. We had an Acorn Electron at home, which was one of the generations of the BBC computers that came after the BBC B. And then I got a Spectrum 128, which is the Spectrum that had the built-in tape drive.
This is way before the Commodores.
I think some of this was the B. I'm pretty sure that the Acorn Electron was pre-Commodore, but the Spectrum would have been around the same as well. The UK was a bit more of a Spectrum and BBC market, whereas the US, the Commodores and the Ataris and things were a bit bigger. So that got me a bit interested, and then I always stayed interested in it. Dad had a computer.
I did the very boring thing. I went to uni. I did a software engineering degree at a place called De Montfort University, which the running joke in my family is it's in the top five universities in the Leicester area. There are only two universities in Leicester. I loved it, actually. It was a great class. It's basically MIT syllabus. And then I just did the work after. I got a great—back then we did these things called sandwich courses.
Sandwich course.
Yeah. And so basically the idea was you would do two years of study, you do a year in industry that was paid, but you're effectively going in as a student, and then you did your final year.
Oh, nice.
Which is actually fantastic because I was drifting a bit, and when I did my year in industry, so many things clicked. It meant much more focus going in, and I got to do work for a company called—back then they were called GEC, now that was called Alstom—and we did software for the European Space Agency and gas and steam turbines. Yeah, it was really cool. I joined about the first time at that about, and then I went back after. And when I went back, it was just around the time of the first dot-com boom, and so everybody left. So I went from being the person that was doing nothing to the person who did everything.
But I remember my first week there, I was given a Fortran 77 codebase. So Fortran 77 was released in the year I was born, and every single for loop had a break condition because we were basically doing analysis on these Unix machines. The time on the machines was expensive, and we were worried about accidental infinite loops. I was trying to solve the halting problem, basically. So every for loop across these 8,000 files had a break condition of, I think it was, 99,999 loops. And if you went over that, the program basically broke and aborted. The models had got bigger. The computer got more powerful. Go through and change them all by hand back then.
By hand.
This is one of the nicest things that anyone's ever done to me in a professional career, because Fortran 77 is based on the old punch card limitations to an extent. So your lines can't go very long. So it's all a bit fiddly. You can't go and put a nine. And my boss said, 'I reckon it will take you about two weeks if you do it by hand, or'—and he slid a book across to me—'you can read this.' And he gave me a copy of the O'Reilly sed & awk book. And so I spent two days learning how to do regular expressions. And I wrote a refactoring using regular expressions, accounting for line lengths and everything else. And it's, I think, the closest I've ever come to feeling like a god. I did it, and I thought, that's the thrill. That's the absolute—
I'm not sure if it's a joke or a story, but the two builders who need to build a wall, and one of them starts to build the wall one by one by one, and the other one is drawing, making plans. It takes three days to build the wall, and the first two days this guy does nothing, and the builder building it feels really good. And then on day three, this builder builds a machine that builds, or puts together a machine that builds the wall for them, and then—
Yeah, but sometimes, of course, the wall falls down the moment you set the machine. The thing is, I love that story. That story was all true, and it's one of the earliest times I can remember learning something and having such a visceral reaction.
I remember this was just towards the end of my time at Thoughtworks. Some grads came in, and they asked to tell some stories about what had happened in my previous career. And I told the story. I said I added the nine, made the models bigger. And then one of the grads said, 'Why didn't you just replace it all with a constant so you could just change the number once anytime in the future?' And I was like—I thought, I'm learning once again. Yeah. So that's also why I love it. You get humbled all the time in this industry. You get to learn stuff. Yes, there's reinvention of the wheel going on, but we're inventing better wheels the whole time.
And I think it's just nice to remind us every now and then. I'm sure everyone who's built code and programs, at some point you have the realization that you do feel like a god, that you can do things with these things which are just so difficult to do by hand, right?
I think we forget about that when it's day-to-day, and we're just doing our stuff. Or even right now, we're using LLMs, which also do incredible magical stuff that would have been impossible. But it's cool, and we are in charge. We can do amazing stuff.
You get that little fizzy dopamine hit. And I think that's something we're struggling with a bit now, as people are getting further away from the code. They're trying to work out, how do I still get that kind of dopamine hit in the same way? It's not as much an issue for me because I'm later on. I've been doing this for almost 30 years now, and I've already been mostly working in the area of architecture and infrastructure for a while. So I feel a little bit more removed. But talking to my son, who's a programmer, he's got into it around an interesting time right now.
And then you eventually made your way to Thoughtworks, where you spent a bunch of time. What kinds of things did you do at Thoughtworks? And what are things that you learned there?
I did anything. I joined when there were probably just under 400 people, left with about 4,000, and I got to go all over the place. I joined at a good time because we're very much an XP shop, and I think that the core—
Extreme programming.
Yeah. I think the core of the business still primarily is, and I speak to people there as well.
By the way, XP, was that influenced by Kent Beck's XP?
Yeah. The extreme programming is very much Kent's extreme programming. There was a bit of a London flavor. So the London school of test-driven development and the Extreme Tuesday Club, XTC, around here.
What's the London flavor of XP and TDD?
If you want to know, you probably need to read Growing Object-Oriented Software, Guided by Tests, one of the worst-titled books in existence, written by a couple of friends of mine, Steve Freeman and Nat Pryce. There's a lot of people in the community around here that knew that. There's a rich meetup scene, and so I got introduced to some people through there and came in.
At the same time, I was offered a job by somebody else. It was a bank. The bank could tell me exactly what I was going to be doing, and Thoughtworks basically—the recruiter at the time, who's now one of my very good friends, one of my oldest friends now, said to me, 'I don't really know what you'll be doing.' And I'm like, 'That sounds like a job for me.' I went for that one.
And this is at a time when we talk about continuous integration, but the tooling wasn't there, and talked about TDD, and the tooling wasn't there. So it was around that sort of time that I started building the build pipeline stuff, in about 2004. And then I worked—Dave Farley was my first tech lead.
Worked alongside Dave Farley.
Yeah. He was my first tech lead at Thoughtworks. I was one of the people that interviewed Jez Humble to backfill for me on a project.
No way. So your—
Project at AOL, of all places. And we needed a backfill because I'm sort of half admin, half dev, really, is how I see myself. So we needed someone to take over my job, and so I interviewed Jez. And so I was around when a lot of interesting things were happening. So I was fortunate enough to be there while we were kicking those ideas around. And obviously people like Dave and Jez put those ideas together really, really well. You're surrounded by everyone talking and thinking about these ideas. I got the idea very early on that this stuff's all up for grabs. These things aren't set in stone, that we can push.
And when you're saying these things, you mentioned continuous delivery. You mentioned what other—
Test automation.
Yeah.
Build pipeline optimization, infrastructure automation. I went over to the US in 2007, actually, to work in a big—I actually worked at Google for a year and a half as a Thoughtworks employee.
Wow.
We were there doing a project. This has all been talked about publicly. It's a great write-up by Mike Bland all about our experiences. It was me, a few of my Thoughtworks colleagues, some people from Industrial Logic, and some people from Pivotal, the old Pivotal, pre-acquisition.
They were known for their really excellent engineering culture.
Back when it was just Robbie's company. And we were there to teach Googlers primarily how to write automated tests and how to write code that was testable. And so I got out there, and I was talking to people out in the Valley. We were going into these enterprise companies doing test automation using stuff like Selenium. And you go around talking to people—I remember having conversations with people at Yahoo and even big chunks of Google who did not think you could do functional automated testing of websites and stuff.
I guess because they just never did it, right?
Yeah. Absolutely. And for me, I hate it whenever I haven't pushed myself or found a better way of doing something. And I was lucky enough that early days in Thoughtworks, that was just a constant churn of ideas and reinvention. Goodness knows how many mocking frameworks and inversion of control things—I built two different replacements for CruiseControl myself, right?
It was at that time I created a thing called dbdeploy with my friends Nick Ashley and Graham Tackley, which was like the first database refactoring tool, and all these sorts of things, because they didn't exist. We needed them, and it was like, if you want to do it, do it. That was the mindset. And so I think there was always this idea, working at Thoughtworks was always like, what's next?
And I was quite fortunate that because Thoughtworks was growing so much, there were always things that needed to be done, and my default was normally just to say yes to everything. So I got to travel all over the world. It was fantastic. Met my wife through Thoughtworks. Most of my best friends have all come from Thoughtworks, still to this day. So although I've not been there for a long time, it was really formative, I think, in terms of that.
And it's really interesting because Thoughtworks has an alumni pipeline, people who used to work
There who are now really well known in the industry for their ideas, for thinking differently. Some of them educate people; others have created businesses to build software in better ways in certain dimensions.
And it's just interesting because usually I wouldn't think of—I'll just put it in a box—you have product companies like Google, like Meta. They build their own products, and you have different tiers of these companies. Some can afford the best people and they have an external engineering culture. Again, Google was known for this, Meta to some extent, and there's some companies maybe who are lower in the tier. And then you have the so-called consultancies, who don't build their own stuff or don't do too much of it. They come in and they help companies build, or you have a problem and you hire a group of experts. Thoughtworks technically is in this group, and usually you would not think that there's too much innovation going on in these places. But what Thoughtworks proves to me, yourself and other people, is actually when you are the best in what you do globally, you can have a lot of innovation happening there as well, right?
Yeah, and it was innovation with a different set of constraints. I would say there aren't that many people that have come from Thoughtworks that were really great startup product people. A lot of Thoughtworkers make excellent scale-up people once you're all together. Actually, very few real entrepreneurs came out of the Thoughtworks machine. I know a few of them. There aren't many of them. But I think there was always the opportunity to innovate even within a constrained enterprise environment.
So much of our working conditions as a consultancy who mostly did co-sourced delivery at that time—we were doing co-sourced delivery, a bit of enablement through delivery—your working conditions are often based on what the client does, unless you're there explicitly to change how the client builds software. So what you were, what you got. But that was self-perpetuating because we were already used to challenging and pushing and trying to do new things. That spirit carried on. And so I think we grew quite aggressively, but not that aggressively, rotating people through, and that spirit carried on. We had to start most engagements with convincing people why testing was a good—why pairing was a good idea, let alone TDD.
Yeah. And I imagine how the conversation went, a manager going, "Well, would you pair? Then I have eight engineers and now you're making it four," which sounds like a terrible thing to do, right?
I had good fortune because of the people, because of working at Thoughtworks. I got to know Martin Fowler quite well, and he supported me a lot when I wrote my first book. I still remember when he said, "Well, programming is not typing." And fundamentally, it made it click for me, having those conversations with clients. If the client thinks that programming is typing and that's why they don't want you to pair program, well, then you've got a fundamental problem that needs to be dealt with. And it's not a conversation about pairing. The conversation is, what do you think programming actually is?
Well, but let's bring this a little bit forward because right now we're having the same conversation. A lot of CEOs, even founders who have been building tech companies, they all look at what is the most time-consuming part of what we do in building software. Fifty percent of the time has historically been writing the code. So a lot of people think programming is code, and LLMs are really good at turning out code. So what is programming?
There's programming—I think it's Peter Naur's article, the guy who came up with Backus–Naur form. He's Naur in Backus–Naur. He wrote this paper years ago, which he won the Turing Award for, and he describes the program as being a theory that lives in the mind of the developers. That it's actually us together, working together collectively, who build a program. The fact that we happen to make that program work through code, the code is really a side effect of how we work together.
And I think product development is just that extended, right? What we're building is collectively we are building a system together, and the code is one of the mechanisms we use to deliver the system. It's not the only thing. So I think holistically that's what it's all about. It's just that we focus on the code because that's historically been where we spend the time. But this is also classic theory of constraints stuff, though, right? If you speed one bit up, you get constraints elsewhere and it exposes friction points elsewhere. I would also argue that a lot of the time coding was not the problem or not the bottleneck. It was how the coding was done, delayed feedback cycles and things.
But I do come back to that Peter piece again because Margaret-Anne Storey referenced it recently in her term "debt." But building software is a collective endeavor. It's people working together who build software. Kent put it in his 21st book that software design is an exercise in human relationships, right? AI doesn't change that. And I think all that's really happened over, say, the last 15 years, in my experience, is that we've got a bit better at understanding that we need better working relationships, like people like product owners, for example. But programming is the art of creating. It's the collective art of creating software, and it's not just developers that do that. Never has been.
And so after Thoughtworks you've gone independent, right?
I did three startups. None of them worked out for different reasons. So don't ask me for advice on how to make a successful startup. And then I went independent. Yeah, moved back. I was living in Australia at the time and moved back from Australia, and I've been independent ever since. So doing a mix of stuff really ever since. I'd written the first book about microservices in my spare time at Thoughtworks, and then that book came out. I'd been at Thoughtworks a long time, and I realized that there's always a danger of becoming institutionalized.
You get to a point with a company, it's like, I think there's things this company needs to change, and I've tried changing those things and they don't want me to change them. So either I'm bad at changing them, which is probably true, or they don't think it needs to be changed, which is also probably true. And it's like, I could hang around and get bitter. I'll just go now. I'll go now. So I went off and I went independent. It's been really interesting ever since because on one level the shackles are off. The problem with working for yourself, though, is that the boss is frequently an... So current Sam frequently is cursing past Sam for the commitments he's made.
That's one lesson, I guess.
Yeah. As I say, I'm almost 15. I'm still not learning the lessons of that, right? And you've made those mistakes for enough years now.
So I wanted to talk about microservices. You have written the book Building Microservices in 2015, and you started to write it in 2014. And as reference, Uber started to talk about how they have thousands of microservices in 2015. So you were writing this book right as the concept of microservices was starting to go mainstream across startups. Do you remember how and when microservices took off, and where was it even coined? Was it Martin Fowler?
No, it was James Lewis. So going back in the day, you've got to remember again, it's like something in the water. It's like we have a bushfire. There was a lot of buildup before the bushfire kicked off. So this would have been 2011 to 2012. James was talking to a bunch of different companies who were building what we'd have classed back then as service-oriented architectures. And earlier there'd been what we refer to as the guerrilla SOA movement, which is an aim to simplify service-oriented architecture, shift over to REST, work done by people like—
Yeah, the original service-oriented architecture had this XML format. What was the name?
Oh, WSDL and Simple Object Access Protocol. Never has the word "simple" been less well used.
SOAP.
You had SOAP.
SOAP, yeah.
Yeah. Start with simple and the most complex thing ever.
Exactly. So you had this whole heavyweight, and then people like Ian Weber and Ian Robinson and Jim Webber had been pushing centralized—had mantras like, "Make the endpoints smart and the pipes dumb," to keep things simple. Put all the smarts in. So this is all background.
We were also at this time doing a lot of work around—I was running the cont practice for Europe at the time, and so a lot of my engagements were optimizing the final mile. So you've written the code, and it's like, how do I get it here into production as quickly and effectively as possible? So basically taking the ideas of continuous delivery. I was around when Jez and Dave started working on those in 2006. I'd had the conversations with them. So that was all happening.
James was talking to companies that were creating service-oriented architectures where they could replace the services quite quickly. And so basically you'd look at it and say it's a finer-grained form of SOA. So that was all happening. We had this architectural symposium up in the Lake District, and James pitched this idea. He was calling them micro apps at the time in this room, and somebody else in the room said, "They're more like services." Don't remember who said it. There were about 10 of us in the room. Most of us were hungover. And then James and Ian—so James and Martin wrote the first post. So the first paper on Martin's website was written by—it was James's original discovery of this pattern that he called microservices.
At the same time I was doing a lot of stuff around infrastructure automation, and so I was coming at it from a different lens. I think James was coming at it more from an architectural lens, and I was coming into it from a continuous delivery lens. I'd spent at this point almost 10 years with that being my focus. And so I was starting to see the architectural impediment. So if I want to change things frequently, but the things I want to change are integrated with other things.
Yeah, like a monolith now. That's how we would describe it.
Exactly, right. I did a talk called Designing for Rapid Release in 2012, which is touching this. So basically James had put a name to this stuff and put some patterns on it, and I thought, hang on a minute, that stuff I was talking about over here, these things seem quite sensible. And I'd always wanted to write a book. Honestly, I wanted to write an animal book. I wanted to write an O'Reilly book because that sed & awk book I was given.
And so I just started writing. I'm quite lucky that I find writing enjoyable and quite easy. So I just sat down and wrote the book, and that's it. But for me it was an extension of continuous delivery, and it was a style of architecture that was only possible because of continuous delivery, because of DevOps. We'd had these ideas before. Erlang is a representation of these ideas, but making it real and practical and a tractable thing. But really fundamentally, the whole thing has been an exercise in smuggling through concepts from the 1970s, which is fine by me, right? If I get people to pay attention to information hiding and coupling and cohesion, that's all right.
But then, as you said, there was a spark before the bushfire, and then the bushfire did start. Suddenly, again, I think Uber was a very notable company starting to talk in public about having thousands of microservices on conferences like InfoQ, and it got people's attention. And then it seemed that overnight a lot of startups started to say, "Let's just do microservices."
There was a bit of rebadging going on. I was lucky when I wrote the first book that I met Ben Christensen fairly early on through the YOW! conference. Ben was looking after performance and reliability at Netflix. And I also knew people at REA, rea.com.au in Australia, and these were companies that were very much pushing the envelope in different ways. And so I had two very good real-world case studies to draw upon for my book.
But most other people weren't talking as publicly around that. And also before Netflix referred to their architecture as microservices, they just referred to it as fine-grained SOA. And to an extent my understanding is that's what happened at Uber and elsewhere. We didn't have a different word for it. Then, of course, what happened was that everyone just called their SOA architectures microservices. Semantic diffusion is a thing. It's to be expected. So I just kept fighting the good fight for, it's just independent deployability. And I've always positioned it as, it's an opinionated style of SOA. You may not agree with the opinions. That's fine, do something else then, but there are some opinions.
So then what is the difference between microservices and, what was it, service—
Service-oriented architecture.
Service-oriented architecture. What are the differences, in your very—
Now, to be clear, I have my definition. You might have your own definition, but I'm the one that wrote the books on it, so my definition wins. That's always been my argument. Write your own damn book. I've always presented it as there are two very strong opinions I think we hold with microservices. The first, and it's most important, is that your microservices need to be independently deployable. So your unit of deployment, unit of change, needs to be a single microservice boundary, a single service boundary.
And rather than deploying everything together, three services at the same time, et cetera, even many or less, right? It's a single service at a time. That's clear.
That's it. And the second one, which I think there can be exceptions to, is that your default way for defining your service boundaries is around your business domain. This is taking the concepts of—you can use domain, you don't have to—but a lot of SOA architectures were dominated to an extent by technical decomposition, the classical n-tier architectures, layered architectures, where you've picked your service decomposition based on the nature of... rather than thinking about the business functionality. That's a bit softer. Independent deployability is the main thing, though.
Okay. Well, because I'm now thinking back at Uber. I was at Uber at the time when your book was starting to become really popular. I was there from 2016, and we had independently deployable services. We didn't really think too much about the business or domain. They had obviously their interfaces. They had contracts that they offered, increasingly strongly typed, and we used Thrift to do it. But it was just more like, as you can imagine, there were just devs, oftentimes not very experienced devs, who were like, "All right, let's build a new service. Here's the interface, here's..." And then anyone can use it.
There were these things where a dev, or even a not very experienced dev, deployed a microservice and then others started to use it, and suddenly it became—years later there would be these exercises to figure out the hot path and label the services, and then this lone service was on the path of some really—it needed to be really accessible as a
Really high-tier service because a lot of others were using it. Also, those sorts of environments, when I was chatting to people at SoundCloud, similar about that. This is also about a corporate culture and what was acceptable in that kind of environment: just go and do stuff. It's absolutely fine.
I think that's why Uber, for example, went to—I just talked with Thuan Pham, the CTO at the time, about what was the real deal about microservices. He said Uber never really wanted that many services, but it was just more of a constraint because Uber had a monolith. The story, which he told on a podcast—we'll link it in the show notes below—was that Uber had this big monolith and it was really slow. It was slowing delivery down, and they needed to move fast because all the other competitors were out there.
They wanted to get rid of the monolith and put together smaller services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen. So the mandate was: everything new needs to be your own service. You take care of it. We don't care. And suddenly so many independent services were born, which was not the goal. Years after the fact, when Uber decomposed the monolith, they're now looking at maybe sensibly merging based on business domain, etc. But this story wasn't really told at the time. It was just, look, Uber has thousands of microservices, almost one or two per engineer, right? And this is what the world saw.
But this is the reality of microservice adoption. It's just a tool in the toolbox. I describe microservices as being an architecture of last resort.
Really?
Yeah. Absolutely, because it pushes you quite naturally to a more aggressive distribution of your state and everything else. So you have to have a good reason. One of the best reasons to use microservices is if what you actually want is an organization with a high degree of autonomy. If what you want is teams who are able to act independently from each other, the microservice architecture is a fantastically good fit. But then really it's a means to an end, right? That's often what a lot of people get wrong, I think, is they adopt microservices and they think this other magic stuff's going to happen. No, if you want organizational autonomy, that's what you're working towards, and this is just one of the mechanisms to get you there.
Most of the enterprise organizations, the vast majority of ones I've worked with in this space, that's what they're after. They've got huge numbers of developers, lots of middle management, they're not getting much work done. They're thinking, what do we do? Organizational autonomy becomes the solution to all problems. Again, I don't always get to have a say whether or not it's a good idea, and I think that's where microservices can be especially useful. There are other exceptions as well.
An organization like Uber—I know at Netflix as well, things like domain design weren't talked about at Netflix, but the teams were already fairly well scoped around areas of product already.
Yeah.
And so you had those natural high-level boundaries around the business domain. Those things flowed naturally, and so that for me was a logical stepping stone. You also have the difference, and this is something I realized quite early on when you go and do the work in the dot-coms versus the enterprise organizations.
When you say dot-coms, you mean the digital companies, tech companies?
Tech companies. I'm old enough that I still talk about dot-coms. But the tech companies, fundamentally, from a domain concept, are incredibly simple. The domains are very, very simple compared to enterprise organizations, by and large. Domains in there are much, much more complicated because they're based on business processes that have existed for 10, 20, 30, 40, 50 years.
Oh yeah. So what you mean, in a lot of tech companies, a product is pretty much: here's our product, here's how it makes money, and that's kind of it. And we have another product. Inside Uber, a simple thing is there's rides, which makes money like this, and then there's Uber Eats, which is a separate business. Over time, by the way, it does get more complex, but when you start it's really, really simple, and it's really separate, and it's clear where one ends versus where the other begins. Then you do some hacks when you need to launch Uber Eats really quickly, and you build it on the ride infrastructure, and then an Uber Eats order has two trips attached to it because—anyway. But as you say, early on it starts very simple.
But then also, in those environments, your business processes and the digital implementation of those business processes are evolving together. A lot of enterprise organizations didn't have that. Their first generation of digital products were codifications of paper-based processes.
Oh, so you're talking to these companies that existed before digital, let's say an Adidas or a shoe company.
Virtually all banks.
Virtually all banks. And now they're like, we have all these processes, and now we need to build software that mimics what we do, which of course is—
That's how you start off. That's what the whole Domain-Driven Design book—Eric's Domain-Driven Design came out in 2004, and it was a reflection of the fact the vast majority of software development was the process of us trying to model existing non-digital processes and how you do that.
I wonder if we will have a different but similar problem of the same thing now that we know AI tools are here. We know that, for example, they're great at generating code, and we have two types of companies. We have a digital company that has existed pre-AI. They're used to writing product requirement documents, deploying developers, product managers, designers, QA. They all have their roles, and it works like a really well-oiled machine. And now you have these so-called AI-native startups who have a few people. From the start, they're using agents for everything. They have a very different process. At some point, when this whole thing will mature and we will have best practices, we'll have the existing companies who, like the previous one, will try to retrofit this tooling to how they're working today.
It's already happening. And the thing is, enterprise companies will rarely be successful at doing those first-gen things because ultimately the corporate DNA will just crush that stuff from within, right? And it will take a while, but I think the whole industry is grappling with the idea of: are we now moving to a new level of abstraction? Software is all about abstractions, right? We went from doing machine code to doing assembly, went from assembly to general-purpose programming languages.
And you're coming back to the idea of the program being the mental model. We would evolve that mental model at the code point of view. That was the most effective place, us working at Java, C, whatever it is, rather than working at assembly level, right? And it's like, okay, well, is actually what we're talking about really that we're stepping that level up? We're still actually building a product together. We're still thinking about what the software is. We're still building that mental model, maybe now more collaboratively, involving more product people, for example, in that conversation when we're writing our specs. So the question for me is: are we ready to make that step? And the answer is no, not everywhere.
This is a really good point because if you think about it, one of the mantras inside of Facebook used to be, I think, something along the line that code is truth or something like that. But in general, in a lot of tech companies, if you want to know how things work, where the truth lives, is it in documents or is it in the coded business logic? It's in the code, in the business logic. And if it's well-maintained code, often well commented or structured, you can actually see the business logic and the rules for, I don't know, for example—
See, I feel like I've got to channel my friend Charity at this point and say the truth is production.
Everything else is the lies we tell ourselves. Production is truth.
But I think in terms of a definition of what the system should be doing, there is this ongoing debate about where is that definition? Is it the code or is it the spec? Birgitta Böckeler did a nice article on Martin Fowler's website recently about spec-driven development, and she talks about the idea of some companies being spec-first or spec-anchored. A lot of it comes down to where they see the source of truth being.
A lot of companies are using things like Kiro or Spec Kit to define a feature, and then the LLM executes that spec, and you can iterate on that process, right? A lot of people start off, they use the spec to write the code, and then they throw the spec away. Then you get other companies that keep the spec, and then when the feature changes they just update the spec. So they're seeing a bit the code and a bit the spec as a source of truth in terms of what the system should do. And then you're now starting to get companies where they don't even ever look at the code, and they just look at the spec.
Yeah.
And so we're kind of going on this spectrum. She had some great terms for these. And I think for me it's like, is that the right path? I don't know. I think there's a lot of questions about what we would need to do to get that. We have a degree of confidence about what happens when I execute a line of Java code, beyond the safety of the compiler and the runtime and all that stuff. There's a lot more questions to answer, I think, before the whole world is ready to say the spec is the source of truth of what our system is supposed to do. Production is always going to be the real truth though, just to be clear.
But I think it's a good question to just ask yourself, however you're building and however you're using agents, of where is the truth of your system right now? Is it more in the code? Do you have a spec? Is it scattered between the minds of people? Which goes back to the collaboration. Because again, a lot of the reality of a lot of startups is a few people hold these things in their head, right? Which is okay in the beginning. But of course, as you scale, one of the reasons we used to love code review, which is now going out of fashion, was we used to share information. When someone left the company or got ill, or the famous bus factor, it was okay because other people understood because they reviewed, they were taking part in that part of it. But now with AI code review, some companies are like, well, we're not going to do it because it's too much code.
That was always one of the superpowers of pairing, right? If you do promiscuous pairing, you rotate every day. When someone leaves, everyone's got the same shared context. That was always something people missed around one of the benefits of pairing.
I think there's a lot to be said around this. I was chatting to Chris Ford, who's head of technology for Thoughtworks in Spain, and he said the biggest problem we've got around AI-driven software delivery is tacit knowledge. There's a lot of knowledge in our heads that we've built up that we don't necessarily know that we have. When you use something like a code review, you've probably looked at that: that doesn't really feel right. I wouldn't do it like that. And sometimes it's subjectivity, but sometimes there's something about that that you can't quite pinpoint. Maybe you can unpack it, maybe you can't. But you have bunches of things like that in your head about how software delivery should be done, about how the business works and everything else. If we want to step up that level of abstraction to work at the spec level, we actually need, as human beings, to be able to unpack that tacit knowledge that we have and make sure it's in the spec if we really want to be hands-off.
I also think there's a fundamental problem that we've got in the current way a lot of people are doing software delivery. This comes back to the cognitive debt stuff, which is you've got a lot of people running off in different directions. I've seen a few people talking about pairing with their AI assistant. Not a human. Stop pretending it is. And so you're all going off. You're doing more in a different direction. That shared mental model of the program is breaking down, and there's gaps in between. AI was supposed to free us from drudgery, right? It isn't for most software developers because we're doing more work, we've got more context switching, we're losing that big picture.
So I think if we can work collectively on the specs, we can rebuild that model, then that absolutely could be the source of truth. I still think it's going to be part our brains and part the spec, right? But for that to work, there does have to be the collaboration piece. Whether or not that's our dev teams go from being 15 people all in to about five people or six people all in, don't know. But collaboration is really important.
Chainguard, proper company, right? They do proper stuff. They're an excellent firm. They recently announced they were getting rid of any human-based code reviews, and they said we're going to spend more time working on things like specs, working at design doc level. And actually I'm okay with that because firstly I trust that they know what they're doing, but also there's something about—I think a lot of companies are making a mistake of not doing the thinking, whereas Chainguard is saying we're going to spend that time doing more important thinking, right? And I think that's how we should view this. I'm okay not looking at the code as long as you're doing other things to make sure the system is operating correctly and that you're thinking critically about how the system needs to change and evolve. If you're just not looking at the code and you're not caring about those other things, well then you deserve everything that's going to happen to you.
Which also goes back a little bit to what Charity was talking about, on how you want to have systems that value that work, about having empathy, for example, for the work that QAs and SRE folks have had to own code in production that they did not write. They don't necessarily read or even understand, but they know how to operate. They know what good looks like in production. They know what it looks like when it goes poorly.
That's also a bigger thing that now more developers are having to struggle with because if you want to do spec-driven development, you have to know what good looks like. I run these half-day conferences for O'Reilly online, and we did one recently on software delivery. About six speakers, and four of them were talking specifically around different varieties of spec-driven development, effectively. All of them came back and said you can't do this unless you can define what good looks like, and if you can't, don't bother. Some of those people were much more bullish, like Addy is very bullish on all of this stuff. But one of the presenters, they've got a small software factory, only 15 or three people, but they're actively pulling in their observability data, their OTel stuff, right into their agent to understand how it's changing. And I think that's exposing more developers to understanding—I mean, this comes back to the acceptance
Criteria stuff people do on stories, they always narrow around functional requirement stuff. So now I think we have to be more grown up about those things. The discipline is absolutely important, and the expertise is still needed. It's just needed at a different place.
Sam made a really good point that you cannot hand your work over to agents unless it does these two things. One, define what good looks like. Two, prove that your system does what this good definition looks like. You already know who I need to mention when it comes to proving that your system works as expected: Antithesis, our season sponsor.
Agents are writing more of the code. So, how can we trust that our system works correctly without bugs? Code review is an increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fuzz testing. By running the simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it a lot easier to fix issues.
The first thing I thought when I heard about Antithesis and its automated bug discovery and fully deterministic testing was that it sounds like science fiction, but actually it's hardcore engineering when you look under the hood. Jane Street, Fly.io, and the CD community ship agent-written code with full confidence because they know it's been verified by Antithesis. To see more case studies and details, head to antithesis.com/pragmatic.
I'd also like to mention our presenting sponsor, turbopuffer. We know it's a database that's ridiculously fast and extremely cheap, and they are doing something amazing. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. If you follow turbopuffer, you know that their storage architecture was a massive part of their early success.
Redesigning a winning architecture is a big deal. It's one thing for query plans to pass unit tests, which is already happening. It's another to bring each query plan to performance par or better while maintaining correctness and reliability in production. And the team is sharing the messy reality of the rewrite and how the v3 version is currently a lot slower than the current production one. For example, we can see how cold text search is 11 times slower than prod, and hot full-text search is 126 times slower. But we can already see some wins, like hot and cold vector search is at par or better.
I really appreciate this work-in-progress view and the real-time logs and what it takes to do a successful storage architecture rewrite. The turbopuffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com/v3. I definitely will do. And with this, let's get back to Sam and what QA and SRE folks already know about running code that they didn't write.
I want to talk to you about your latest book that is just coming out, Building Resilient Distributed Systems.
Yeah.
What is the book about? And why did you feel this is a good time to write it?
Years and years and years ago, I went to Velocity, which was quite a well-known conference.
It's a conference. Is it still on?
No. It was the O'Reilly conference, and it was the conference where a lot of the early people went to talk about how they made their systems scalable and stuff. I got to go. It had been one of those career ambitions to go to. I got to go after I'd written the first book, and I got chatting to John there, who I still chat to every now and then. He now works in this resilience engineering space.
I don't know if it was him or someone else, but it pointed me at this paper by David Woods that was all about resilience engineering. And the paper was utterly baffling to me. It made absolutely no sense to me because it's not written for computer scientists to read, and I'm not even a computer scientist, so I was really in trouble.
But it was really fascinating seeing this whole other world about how people viewed resiliency, and then what we do in the software world. People like John were trying to take some of those ideas in a little bit, but I found it very difficult to take some of those ideas from safety-critical systems and that type of thinking and make it in any way, shape, or form palatable for normal people like me.
This was back, what, 2016, maybe 2015. And so I'd go back and try and reread that paper every six months or so, and I'd read bits around the edges and see other smarter people than me that summarized it better. And then I was thinking, okay, these ideas now are starting to make sense. And then also I realized that despite my best efforts, lots of people seem to be building microservice architectures, even though if you've read my books, most of my book is about telling you not to do it.
But you have to read the book, which is called microservices.
Let's be clear, as long as they buy the book, that's the most important thing, whether they read it or not. And so then I realized that there are more people exposed to the horror, pain, and suffering of a distributed system. And so it was like, okay, I've got this stuff happening. My son's a developer now, and I always thought that if somebody found themselves building a distributed system, working as part of building a distributed system, what was the equivalent of a rubber ring I could give them, say, stay afloat and don't drown?
And so the goal of the book is to start you off with really easy, foundational, simple engineering concepts about how you can start dealing with the problems of distributed systems, but to also take you on a journey into the whole concept of resilience engineering. And so it starts off with really easy ideas, and then steadily it gets more and more complicated. I introduce concepts like what a sociotechnical system is, the difference between safety management and resilience engineering. These are all big concepts, but the second chapter after the introduction is, what's observability? Let's talk about OTel. The third chapter is, what's a timeout? And these are all, I think, quite tractable things.
But they're building blocks, right? The primitives for these systems, and the vocabulary as well.
Yes.
Yeah, absolutely. And to go through and distill some of that knowledge down. I'm quite happy being a bit opinionated on these things, but I'm also lucky now because, partly because of my early work, I get to chat to more people. So I can actually talk to people who this is their full-time job and try and distill down some of that knowledge a bit.
And so I wanted that book. Partly it's about being evergreen, but it was also about saying there are lots of people building a distributed system. Let's put what type of systems you've got off to one side for a moment. There are lots of people building them, whether they should or not, and they're really important to us now. So how do I help people start and grow? And that's what I wanted to do.
I enjoy the process of writing anyway, but this is a book I wanted to be able to give to people. That's always been why I wrote a book. I want that book. Every now and then I've gone and sat down to write a book, and I found someone else has done it already. It's like, well, that exists now. So this, for me, was the book that I wanted to exist but didn't. That's why I wrote it.
Yeah. And I think it's a book I certainly could have used. I got dropped into a place like Uber, where we did have what I realized were distributed systems. Large systems handling a bunch of stuff, lots of services dependent on one another. Because even a simple thing of knowing, oh, you should pay attention to observability, you figure it out after a while.
Yeah.
But it takes a while.
Yeah. And I think that's my attempt to unpack some of that tacit knowledge to an extent. I've made so many mistakes, and I keep making mistakes. I think making mistakes is useful, but I think also creating opportunity where people can learn from other people's mistakes first is a good idea too. And so in the book I also try and pull in real-world examples, sometimes from big companies, sometimes from my mistakes, to make this stuff seem real.
And I think the other thing I really try and get across is that things go wrong. Resilience engineering is all about making sure as many things go right as possible. It's not about making sure everything goes right.
Yeah.
And the only way you can build resilient systems in general is also by being honest about mistakes that have been made and talking about them and sharing them, because that's how everyone gets better. And if you don't do that, you can't build a resilient system in your company. And so this again is my attempt to highlight all the awesome work that's gone before, the fantastic post-incident reports that some companies share.
When there's a system that we all use, or many of us use, that has an outage, it's annoying, especially when you're using it. But I always greatly appreciate when the companies, because they don't have to do this, share a public postmortem or incident recap or whatever that is. They go through what happened. For example, GitHub had an outage, and they shared a really interesting one on all the things that went wrong and in what order. And then you also have companies where something like this happens and they don't do anything. So, of course, you're not obliged to in any way, but I always have a bit of a lesson.
They are obliged to now, a lot of them. Regulations are now making them that they have.
Which I think is great.
Yeah, things like DORA and stuff in the UK—in Europe, rather. And I think we should also separate around that post-incident report stuff that there's marketing and PR versus, have you done a proper post-incident review? Now, I do think you've got a duty of care, really, to let users know as part of a proper post-incident review. But those companies that maybe say nothing, they might do a fantastic process internally. There's also the fact that some companies that put those things out, you read them and you're like, this does feel more marketing.
Yeah.
If you want a company that puts out some of the best self-flagellating post-incident review write-ups, go for Cloudflare's stuff. They absolutely go for themselves in those. I feel like, guys, go easy on yourselves, right? Everyone makes some mistakes, but I quite like it.
And they do it rapidly, within 24 hours, and they're a publicly traded company.
Yeah.
Which I think just goes back to the myth. Some of them are like, we're publicly traded, we don't want to do it because it would give away information for shareholders. Yet Cloudflare does it, and they're fine.
Exactly. And also it's—
And it builds trust.
It is. A lot of how an organization is perceived is down to how you deal with mistakes when they occur, right? Nothing ever goes perfectly for every company.
So in your book you talk about the three rules of distributed systems. What are these three rules?
I really tried to distill this down because I could never remember the eight, the rule of eight that came out of Sun, the fallacies of distributed computing. I always forget what those eight were. There's loads of complicated stuff, but I realized when you strip it all down, there's really only three things that really cause all the other problems. The first is that you can't send information instantaneously from point A to point B.
It's physics. Now it turns out when you get into it and start talking to people about quantum physics, they say, aha, and they go, there's a thing called quantum entanglement which does allow for theoretical instantaneous read of electron spin states across arbitrary distances. At which point I point out that, A, that only works for reads, as far as I'm aware, and B, not a networking protocol. But it takes time, right? I want to get from A to B, and I can't control all of that. I can control bits of that as a developer.
Yeah.
I could control how much data I send. I could control the payload size, technology I use to serialize and deserialize. I'm unlikely to be specifying the network cabling or the equipment we use or the distances between them. So, not really under my control. And the moment you go over the public internet, BGP has a bad day and your packet has gone down the wrong leg of the internet, right?
Yep.
So that's rule one: it takes time. And that's important to recognize because sometimes it looks instantaneous. It's not. Second thing: sometimes the thing you want to talk to isn't there. Sometimes that's under—and we do things to make it under our control, right? Like we would run, say, multiple copies of a service instance behind a load balancer. What happens? The load balancer fails. But you can't control for everything, right?
I've had services taken out by a fire in a data center. A SAN caught fire, and specifically one bit of a SAN. The SAN had a caching storage layer with SSDs. These two pairs of SSDs, one of the pairs of SSDs caught fire. The other pair of SSDs were working fine. And so we had these weird latency graphs coming out of the SAN, perfectly fine if your load went through the ones that weren't on fire. But the failovers in the other one were routing, to be fair. It was routing around the bits that were on fire.
I had a system where we had ducting between the office buildings, and rabbits moved into the ducts and ate a networking cable. I can't control for rabbits, right? And these things are real. Those are two things, right? Yes, you can control for that and you can reduce the chances, but you can't limit those chances, right? So rule number two is: sometimes the thing you want to talk to isn't there.
And the third rule, I'm very grateful to an old colleague of mine, Ben Cole, for saying I'm missing, is that resource pools are not infinite.
Meaning?
You can run out of computing resources: CPU, memory, I/O, whatever.
And basically your load balancer could be overloaded and then just not respond, even though it's there and we know the latency, but maybe a queue fills up.
And the vast majority of the time when you have a system outage, it's down to that. Some resource has been totally saturated somewhere. And that saturation might in turn have been caused by the fact that something is down and you're in a retry storm, which is in turn causing connection starvation, which is, of course, resource saturation, which is in turn yada yada yada.
But that's the killer. And again, I think we understand intellectually all of these things, but you can go to all the complicated concepts in distributed systems and say, why is a Byzantine consensus protocol a thing? Well, let's just go all the way down to this bit. What about CAP and PACELC? Let's put that on one side. Remember these three rules first, and then most problems end up becoming a little bit easier to deal with.
And in the book, after you introduce these rules, you start, as we mentioned, the second chapter is observability. To me, that suggests that this is a really, really important part for distributed systems. Why is observability so important?
The way to think about observability is it's a property of a system, right? So the more observable something is, the easier it is for me to
Understand how that thing is behaving based on the external signals. When you run a system which is a single program, you actually get some pretty high-quality signals coming out of that program. Therefore, they give you a high degree of observability.
I can get a lot of information out of that. I can attach a debugger to a process. I could generate a core dump and run that in. I could do a thread dump on the JVM and work out where my semaphores are blocked. If this is my whole system, knowing the CPU is stuck at 100% is actually really useful information because that's a problem, right? So this basic information that we pretty much get for free with a single program gives us a high degree of observability.
A program running on your machine that you have access to.
Exactly. But now when we go to a distributed system, you don't have access to the computer in the same way. You've got multiple computers, and so those basic signals are harder to get. You can't correlate them unless you do extra work. And those individual signals by themselves don't really give you a proper sense of how the system is behaving. So you have to do more work to make the system observable. And, as is annoying with these sorts of things, it's much easier to do that from the beginning than it is to retrofit it.
Luckily we've got things like OTel now, the OpenTelemetry protocol and specification, because that at least gives us an open framework that's vendor agnostic. Oh, Honeycomb are the best. Charity pays me in stickers. Just to be really clear, I always think I work independently. I have to be very clear: Charity pays me in stickers. But we get a vendor-agnostic open standard for this now. It makes it so much easier for me to have the conversations about this.
But if I don't get the signals coming out of the system, if I don't get that stream of structured events telling me what the hell's going on with my system, I'm flying blind. How do I know what an appropriate timeout is from a client point of view talking to a service? Well, I'm only going to know that if I have a sense of what its normal response time distribution is. So I'm going to need latencies, know the success or failure of those latencies. It's probably going to be a plot on a histogram, maybe, to take a look at it. If I don't have that information coming out, I'm not going to specify a timeout correctly.
And then we're talking about a low-level thing like latency and things that we can measure at the machine level. But, of course, is this observability when I'm talking about, does it work for the user or the customer? Does it do what they do when they click the button to submit? Does it submit? Do you look at that? Because observability is usually, we talk about these kinds of latencies, the statuses, the 500 responses. That's kind of bad. That means it's down. But there's always a layer that I always wondered about, and I found it hard even when we're working in teams to, okay, we have the easy observability. This is kind of textbook.
It's the metrics. It's the metrics.
But then there's the stuff of, does it work? Does it do what people care about? Again, in the case of an Uber, when people hit a button to request a ride, does it actually request within usually five seconds or something like that? Or is it stuck in an infinite spinner, which is bad? Even your logs can be as healthy as can be, but people are upset, right?
You treat the raw data of making the system observable as a stream of structured events, and then from that you build up the higher-level abstractions and the views, right?
So use them as primitives.
Use them as primitives. So a trace, right, where we get a sense of, for a given click of a button, what happened in the production system. And this is a big blind spot, right? Most organizations had to build their own tracing systems. So this is where things like the OpenTracing stuff came from, and things like Zipkin and things like that were built as an attempt to recreate—I mean, most people were just trying to recreate what Google had already built there. And this is an attempt to say, okay, can we at least look at a flow of calls? What happened during that flow of calls? Where did these things go? Well, that requires me to get a bunch of spans.
And a span meaning?
A span meaning an activity, basically something I did. And then a span can beget more spans. So I can say my first span might be a call has hit my server, right? So I've now got a span that starts. Within that span I might talk to my database and then do other operations, and I can put additional information on those spans. And then I call to another service, and then you have a trace ID that's shared across those spans, and you've got timing. So now you get effectively a view of what's happened across those calls, but that's all just based on individual events being stitched together.
And then you also start getting into a situation which is, is my system behaving correctly? And this is going away from—again, I think half of just saying what Charity would say—again, going away from the vanity dashboards. This is moving into the world we call the SLOs, right, the service level objectives.
And SLI.
And SLI. So an SLI, service level indicator, that's our way of saying, at this given point in time, for this thing, is this thing good or is this thing bad? And we use a collection of those. So one example of that might be a successful percentile response time, right? So that would be, is it a good one or a bad one? It could be a bad one because it's an error. It could be a bad one because it's slow. Good or bad. And I collect those together, and that helps me know if I'm meeting my SLO. So I might have an overarching SLO which is, I should be able to request a ride with 99 percentile successful ride requests within 300 minutes.
Yeah. And that would be your statement, and that would be a commitment. The SLO is supposed to be a commitment. That's something our team is working towards. You then get this raw data. You calculate your SLI on the individual things, and then you use that to work out what your SLO is looking like. If it's green, everything's a good time. If it's red, people are unhappy. I mean, you get a bit of red. There's way more nuance to it than that. I'm giving a very bad introduction to this stuff, I think. But without those building blocks—because I talk about it in the first chapter of the book—nobody says, 'I don't want my system to be resilient.'
Yeah, no one.
But at the same time, resiliency is not a binary property. It's not like off or on. It is a bit like observability. It's a degree. So then the question is, how much do you need? Because am I having a conversation with you about multicloud and doing complete rewrites of systems for critical stand-ins like Monzo do, for example?
Yeah, they have a standby bank when they go down. So it's sitting there, it's costing them money for the most part. But then just a few weeks ago, their main application went down for a short period of time, and they activated that bank, which I'm assuming cost them millions of pounds or dollars per year just to keep it there.
It's not even that. Slight sidebar, but firstly, their standby bank has a real stripped-down list of features. It's much more bare bones, but it's all the core ones, like you can freeze cards, you can take money out, you can check your balance. It's a complete rewrite, so there's no shared code because they're worried about a bug. Different cloud vendor. It's also, by design, significantly cheaper to run. So they've got thousands of microservices on the full stack. This is about a tenth the cost of the cloud costs.
They also run it constantly.
Yeah. So it's always there. But Monzo know they're Monzo, right? That's how that works. But there's no point you starting there. So actually, the reason I put the observability stuff in is, you need the raw data, but also you need to make decisions about—it comes back to what good looks like.
Then you need to know, okay, do I need to worry about multicloud vendor DR strategies because that's where I need to be going, or am I somewhere else? And so I think without that sense of your north star, how resilient do we need to be? Without that defined, you don't know how deep you need to get. This is more like trying to say, how secure does my system need to be? Have you done the threat model? If I know the threat model, I don't know.
And I guess to answer this question of how resilient do I need to be, it greatly helps, and maybe it's actually a requirement to make an informed decision, is how much will it cost me in terms of operating cost, in terms of time to build it, in terms of, for example, latency? Just a simple example is if you say, okay, the business is like, 'Oh, would you like multicloud?' And the business is like, 'Yeah, we'd love multicloud. If Amazon goes down, because we heard they had an outage last week or a few months ago, I actually want to be able to run on Google Cloud.' But then you'll be like, 'Okay, well, first of all, this will mean we're paying two cloud bills. And if we want full replication, it might increase our latency because we need to wait to write for both of them, which might mean customers might see things slower.'
Anyway, there's all of these trade-offs. And in my experience, oftentimes you might retrofit that, but if you can have a discussion up front, that's great. But to have the discussion, you go back to the basics. You need to know what the trade-offs are. You need to know what's happening in all these dimensions that you care about, like how long will it take to build? How complex will it be? How much money will it cost to operate?
But also, as technologists, we can't just say— we have also got to get good at communicating with our business stakeholders, with our end users.
Oh yeah.
What percentile latency do you want? Some developers know what I say when I say that. Not all, but some. How many experts in their field who are great product owners or subject matter experts but are not software developers would know what I meant by that? And so we also have to understand the business domain, the product domain, well enough to do the interpretation and help to bridge that gap.
And it is a dynamic thing as well. I mean, this is the whole thing about resiliency in general, right? A resilient system is one that changes. It changes and it evolves. If you don't change and evolve, you're a brittle system. You're not a resilient system. And so even those targets we set, that conversation we have to have about what is good enough does change. But think about something like, in the case of, say, the us-east-1 outage towards the end of last year. There were lots of AWS—
AWS's us-east-1 region went down. It was the oldest region, a big region. Lots of people were there first because that's often where new services got launched historically.
When that went down, it caused a lot of people to say, do we need to be multi-region? And some of those companies would have had the conversation before and decided, no, we don't. And then they'd seen the fallout, and they may have changed their minds. And that is fine.
This is what you mean by a resilient system evolves.
Yeah.
The real world.
But also, it is about what problems you're not going to try and solve. There were some big-name companies that got taken out when us-east-1 went down and had severe degradation in terms of what they could offer.
Yeah. Companies like Zoom, for example, they weren't taken out entirely, but they were severely affected. Do we think that people working at Zoom are idiots? No. Do we think they knew that if a cloud region went down, they'd be impacted? Yes. They probably looked at that and said, 'Well, for us to come out with a multi-region approach, it's going to have this impact. It's going to cost us this much money. It's not worth it.' Let alone talking multicloud, because multicloud versus multi-region is a whole different conversation. And that's okay. And I think as long as you've had the conversation, and as long as—this is the other secret bit of it, of course—as long as what you think you need to do is in tune with what your users think you need to be doing.
Yeah.
This is why I make the point in the book when talking about the difference between SLAs and SLOs. The SLA is what you're going to get sued for if you breach. If you only ever deliver to the SLA, your customers are rarely ever happy. You've got to do more than that, right? So there is the social contract there as well.
Yeah. Now, with building distributed systems, building resilient systems, it really helps to understand concepts that are not as straightforward. And I'm not going to go through all the things. You explain a lot of things in the book, but I do want to touch on a concept that I found a bit tricky to understand and then appreciate, and you really appreciate in distributed systems, which is idempotency.
Yeah. What is idempotency? So there's a mathematical concept behind idempotency that I don't understand. But basically, an idempotent operation is one that you can apply multiple times without it actually having an effect multiple times.
So it has an effect once, first time.
A really good example of an idempotent operation would be pressing a lift button. So you want to go down in the lift. You press the down button. It lights up. If I press that down button again, nothing's changed. The lift is still coming. Well, we hope the lift is still coming, right?
Yeah.
That's an idempotent operation. Some light switches are actually toggles. I toggle it, light goes on. Toggle it, light goes off, right? That's not an idempotent operation. In computing, we have some operations which are idempotent. Reads are. A delete is typically idempotent, right? But the example I think I use when I introduce this concept is a one-off payment.
I was about to say that I have a lot better idempotent example for you.
Yeah. Payment is a good one, right? So I'm going to pay you money. So I'm a service that can issue a payment, and I send you, 'Please pay Sam £100.' Yeah, I'm NLP-ing people back at home, right? Please say, 'Pay Sam £100,' and I don't get a confirmation back. I don't get back a response. The problem is, from my point of view, there are two possibilities. The first possibility is that you received that payment request and you've done the payment and it's been completed, and the result has got lost to me. For some reason, I didn't get the response. That's one possibility.
The other possibility is that you never got that. You never carried out the operation.
Maybe it didn't get to you. Maybe you crashed halfway.
I think a good example that I've seen in the past, and maybe it's over, but airlines. You're about to buy this airline ticket for $1,000, $2,000. You've checked all of your details, and it says pay or reserve. And then often some airlines would take you to an intermediate page which says, 'Please do not refresh this page or submit it again because you might be double charged.' And what you really want is, you pay, and you get the spinner, and it's not idempotent if you do it again and you get charged twice.
Yeah. But it is idempotent if you just refresh, you copy the same thing, you submit the exact same form.
Although that's interesting. So then the question becomes, right, I sent you that payment, book my airline ticket, whatever it is, and I haven't had that response back, but I really want to make sure that the airline ticket booking goes through. This is you with the browser or the people at home sending Sam £100, right? You really want to make sure Sam gets his £100. Now, when something doesn't work in the distributed system because of rule number two,
Rule number two means the thing sometimes might not be there. Rule number one, sometimes it can take ages.
And accepting this thing, you just try it again. And you retry because you really want it to happen. And in retrying, there's the danger that if the first operation had been processed successfully and you didn't know about it, or the first operation had been applied successfully and you do it again, you're now getting paid twice and two tickets. So that's the problem with idempotency.
You want to make it safe to retry important operations. So to do that, we need to make them idempotent. There's two ways to do it, really. One that's good and is the one you should do, but retrofitting is a pain, and one which is the one you just explained, which is useful but has problems. So the first one is idempotency keys.
So you have a unique ID. You normally use a UUID for this. And so the unique ID which you generate, when you generate two of them, they should be different. There's this tiny, tiny, tiny, tiny, tiny, tiny likelihood of a clash, which again happened, I think.
I know, but I think with UUID 7, I think it's like a 50 ncond chance. But it's something you can generate locally without any additional round trips. Your collision chance is extremely low, and there's ways to modify that. But you can work with that.
And this is—
Unless you're Amazon, probably.
Well, this is, I mean, the AWS APIs you see, Stripe, PayPal, they all use the same key scheme. So you send this UUID. So then if you send that to me, you say, "Pay Sam, and here's the idempotency key." I don't send you back a response for whatever reason. You retry it. You pass the idempotency key. If I had carried out the processing, I could look at it and say, "Oh, have I processed this? Yes, I have. Okay, great." But if I have processed it, even if I get a second or third or fourth, "Have you processed this process?" I just keep going, "Yeah, it's processed. It's processed. It's fine." That's a nice, simple thing to do.
The issue with making things idempotent with an idempotency key is it requires both the client and server to support it. So it has to be a required field that goes through, and it has to be sent to server support. So if you're trying to retrofit this in, it's a pain.
But if you're building it, use an idempotency key.
Use it. Exactly. If anything needs to be idempotent, do that from the beginning. The one you sort of mentioned was, I go back, I hit refresh, I fill in all the same fields again, and I submit the airline booking again. It should know it's a duplicate. That's another thing you can do. This is good if you want to retrofit idempotency in.
So if I've sent the same fields, so the same person, Sam, the same amount, £100, and I'm sending this through again, what you can do on the server side is generate a fingerprint. So I take these fields. I then generate a hash that represents those fields.
So if you do a second booking, it looks the same, and you're using that as a retrofitted idempotency key.
Effectively, yeah. There's a couple of downsides with that. You do get false negatives with that.
Yeah, the false negative meaning that legitimate second requests can get rejected.
Got it.
So, real-world example: I know this happens in Denmark. It might happen elsewhere. I was speaking to people in Denmark who work on a payment system. You go in, you buy something for 40 kroner, a particular payment system, and you give the card, scan the card, 40 kroner or whatever. If within the next sort of five minutes you buy something else for 40 kroner in the same shop, the till, the handset rejects it. If it was 41 kroner, it'd be fine. 39 kroner, it'd be fine. But that's because the chances of you buying something for the exact same money is so remote. They're doing some basic idempotency to stop double taps.
And so when you get situations where operations can legitimately look the same, but actually be different and carried out at the same time. So a really good example would be the AWS APIs. It is completely legitimate for me to, in a for loop, provision, say, 20 EC2 instances. Now there's better ways of doing it than sticking it in a for loop, but I could. So I could iterate through the for loop and spin up 20 identical EC2 instances. If I just relied on there being a fingerprint, they'd all look the same. So that's where this stuff breaks down.
And so if you've got situations where legitimately you have operations that look the same but are actually different, that could be done close to each other, you have an issue. If you've got a bit of space around them, what you can do is hold on to those fingerprints for a period of time: five minutes, ten minutes, an hour. So there's still some potential; you can still let duplicates slip through. So it's not ideal. Idempotency key is the way to go. Fingerprints when you have to retrofit.
If you use them both together, that's going to be really useful. So you can pick up situations where people change the parameters between the retry attempts, which leads systems into confusion. It's a bit more of a complex topic, but a lot of public-facing APIs will still generate a fingerprint. And so if you send, "Carry out this operation," and you give the same idempotency key you gave before, but you've changed the parameters, the server can reject it as being effectively an invalid request. So that's how those things work together.
The ideas are really simple, I think, but there's some nuances about implementing them. There's some interesting stuff that crops up in terms of how you store them on the server side and things, but the fundamental ideas are pretty straightforward, I think.
Now another interesting topic that comes up often these days, it feels more often, is thundering herds.
Yes.
It's when a system restarts and suddenly it faces this big load. Specifically, GitHub had an outage a few weeks ago where it was networking-related, and recovery took a lot longer because there were retries going on, and it created the thundering herd. What is a thundering herd?
Generically, I talk about it in any situation where you've got mismatched capacities. You've got too many requests for too few resources. Resource pool's not infinite. Coming back to rule number three.
Rule number three.
Yeah. So in the chapter I talk about a whole bunch of different ones from different types of scenarios. So you've got everything from the malicious thundering herd, like a denial-of-service attack. A denial-of-service attack, malicious attack, often short notice, and there's not that much you can do about them. The answer is you use Fastly or Cloudflare, but I go into details about how that happens.
You get things like internal, self-inflicted ones. So a good example would be something like a cache collapse. The system relies on the cache not only for speed of operation, but also for reducing load on other components. Cache gets restarted. You then have a lot of cache misses. You're hitting the origin, and the origin goes down.
Yeah, origin was not expecting as much load.
Yeah. So a common thing there would be, first you'd flush those cache snapshots to disk. Reduces the chance of you having high cache misses on a restart. But also a lot of companies I know will just turn their services off until the cache refills. And so that's another example of one.
Retry storm is another one. Back in 2017, Square had a big outage. They'd restarted Multipass, which was their multifactor authentication system. Multipass in turn pulled its data from Redis that they were using as a database, because you can do that, apparently. I don't think they're using—
As a cache, basically, right?
Well, no, I think they were using it as a database. There are some people that think Redis is a database. This is a different podcast. I'm thinking of starting a stupid technology podcast. We talk about using technology in stupid ways. That's a separate conversation.
Anyway, so they were pulling this data back from Redis, and they had a problem that the clients in Multipass were going, basically, if the Redis calls failed, they'd try again, but without any delay between the retries. And they had a hardcoded limit of 500 retries. So they just basically smashed Redis, and they kept restarting Redis, and then they'd restart Multipass. And again, this is a long time ago, and they fixed it. They fixed it by having a more sensible retry policy and externalizing those values. That's another fun thundering herd.
The last kind of sections of ones I talked about were ones caused by products being too successful.
Like Twitter and healthcare.gov.
Right. This is where you have an external, real, legitimate load that takes you out. And the thing is, you look at all of these as solutions, although what's happened is the same, the kind of solutions are a bit different. So when I've got a thundering herd that's coming from externally, that's legitimate, I've got different things I'll look at. I'd have a more constrained signup. I'll have fewer invite codes.
So with healthcare.gov, one of the issues—there was many others—was that they went live in like 36 states at once, and they were massively overloaded. So in hindsight, maybe going live in one state would have been a good idea first. For example, with Twitter, they were going down all the time. So they could have constrained signups or put tweet limits on you and things like that to constrain those things, and they did those things as well. And so although they look the same, the mitigations for each is a little bit different. So I quite like talking about that because they were all great stories I found, and I wanted to—
And of course, I think we're talking about solving it when you have the same resources, but then we get into things like scaling. Can you scale horizontally, add more resources? Because there's two sides of this, right? At what point can you do it, or are you going to face a bottleneck because you still have a central whatever?
But that's a great example of where sometimes mitigations work and sometimes they don't. So an example of something like your product is successful. If you're a startup or whatever and you suddenly got successful, that success comes like a wave. The wave of success hits you, and then a lot of it recedes. You'll get a thousand customers in, and maybe you'll keep a hundred, maybe. That's just how things work. So you need to be up during the peak, but then you probably can shrink down again to keep the cost low.
So if you've built your application to have some dynamic scalability in it, and if you're on the public cloud or a similar place where you can spin up and spin down, that's a viable mechanism. Okay, we can scale up to handle this, and that's fine. Would I use that as defense against a distributed denial-of-service attack? No, because then your distributed denial-of-service attack becomes a distributed denial-of-cash attack. You go from not only having no service, but also now no money, because you keep spinning up nodes on your Kubernetes cluster, and Amazon just keep taking the money. It's great.
And you still can't keep up. So that's a perfectly valid mechanism for some things, but you kind of have to know the context.
You can't just scale yourself out of trouble. I think we're going back to, a lot of times, as an engineer, you want to know about the business. You want to know what creates value, what creates revenue, what is important for the company versus what is not, and then where we want to maybe invest with our infrastructure, with our reliability. Or is this where it doesn't matter, and it's not worth doing?
It's all about trade-offs. And I think you don't know unless you get information out of your production systems. You don't know how close you are to the red line. So that's why observability is key. But it is all about trade-offs. I do a whole chapter on formal methods. So I talk about CAP and PACELC, and I say, look, these things are complicated, but actually, really, it all comes down to a conversation with understanding the business context.
So the example I give there is, if an e-commerce company, if I want to buy an item online, a physical item online, and if at that moment I want to purchase it, if you're not sure if you've got it or not, most e-commerce companies will say, "Yeah, sure, you can have it," because they know there's a good chance they do actually have it in stock. And if they don't have it in stock, they can get it in and backorder it. And the worst-case scenario, they can email and say sorry and refund you the money. So that's not great. It's not ideal.
Now, if they said, "We don't know if we have this item, therefore we're not going to sell it to you," they're missing out on a sale, and I could go buy that somewhere else, probably, and they might have had it. So e-commerce companies in that situation, because of understanding the relationship with the customer, the impact if they get it wrong, they're going to try selling the item.
Keep that problem exactly the same, but now change the business context. Now we're selling concert tickets. So you're getting tickets to go see Tay, right? You're going to go see Taylor Swift, and we're not sure if you've got the Taylor Swift ticket or not because our link to that system is down. But those Taylor Swift tickets are yours. You're taking your kids, taking your wife. It's all going to be awesome. We're going to Portugal. We're going to see Taylor Swift. It's going to be amazing. And then we go, "Sorry, we don't actually have those tickets." Now, the problem with that is, firstly, the devastation you're going to feel is a bit different. They can't backorder a ticket.
And so in that situation, the impact on the customer is so much greater.
Now booked the flight.
Well, you might have booked the flight. So what they do is they make a different decision. So in that situation, if they don't know if they've got the ticket, they don't sell the ticket.
I'm not sure if you cover it in the book, but this is the fail open versus fail closed, right? When you have an error or you're unsure, are you going to go through or not? And as you say, this can be a decision—the same problem, two companies can take a decision. A company years down the road could take a decision.
Uber had this classic example of when we weren't sure if the payment went through. Do we allow the ride to continue or not? And early in the company's years, we always failed open.
Yeah, just have the ride. We might have the money, and if not, we'll deal with it.
As the company started to go for profitability, we failed closed. We're not sure we have the money, if the charge went through, you're not going to get the ride. And again, same company, different plan, different focus. And if you're a developer, you needed to pay attention to what is important right now for the business. It was growth initially, and it's okay if we lose money. And then now it's unit economics, profitability.
But also that fail closed versus fail open discussion isn't one size fits all for the whole system.
Exactly.
It's on a feature-by-feature basis. And even then, you could say this taxi ride is €10 or it's €1,000. You can make different decisions. Now, if you approach this from a mathematical and from a computer science viewpoint, we just talked about CAP theory. That was a conversation about CAP theory, right? So in the chapter I talk about, here's CAP, here's PACELC, which is an extension of CAP. I explain what those models are because when you get deeper into systems, you'll
Hear about them, but fundamentally that's what this is about, right? When you have a partition, that's what the P in CAP really defines. I can't talk to one part of my system. Remember, it's rule number two, but I can't talk to another part of my system. What do I do? And all you need to know as a developer: I can't talk to that. What do I do?
Sometimes we can make technical decisions. I'll try again. I hope I don't bring my systems down with a retry storm, but I'll try again. If that doesn't work, what do I do? Well, actually often the answer is, yeah, sure, you can look at CAP and PACELC or whatever else, but go have a conversation with a product owner.
What should we do?
Yeah.
The resolution of that often ends up being a decision about: are we trading off consistency versus availability in PACELC, or harvest and yield if you're going into that version of Brewer's conjecture? But don't worry about that. That's in the book. That's why it's a later chapter. That's chapter like 12, I think, or 11. Don't start there. Start with having the conversation. Rule number two: I can't talk to this part of the system right now. What should we do? And that starts with having the chat. That starts with having the chat.
When we're talking about resilience, you talk about the dimensions to think about resilience. What are these dimensions?
I start in the first chapter by coming back to that paper by David Woods I mentioned that was impenetrable to me at the start, and he comes up with four concepts for resilience engineering. And I like the four concepts he comes up with. Helps us think broadly, right?
So it starts off with robustness. This is where we can basically, I think in his words, absorb known perturbations. So we model a system and something goes wrong, and we can just deal with it in a way that no one cares about, doesn't even know. So a very simple example would be: in Kubernetes, a pod fails and Kubernetes makes a replacement. Now the pod failing, that's a perturbation, and saying it's gone a bit wrong, but we've dealt with it. Great.
So that's the start. And the thing is, most engineers I speak to, they focus entirely on this. They don't look at the other three, focus entirely on this. One of the really nasty things about robustness is the things we often do to make our systems more robust in turn increase the complexity of our system, which in turn increases the surface area of new types of failure.
I bring Kubernetes in to deal with the failure of a container. Now I've got Kubernetes, and that becomes... Yeah. And that's not solvable, by the way. That's just something to be aware of.
Right. This is also why sometimes people don't go multi-region or multi-cloud, because it's increasing the surface area of other failure types.
So that's number one, robustness, right? Can always deal with those things. And for things to be robust, we have to know about them ahead of time. This is predictable. We deal with it. Then there's rebound, which is when something does happen and goes wrong, how quickly can we recover?
So this is typically the time to recovery.
Well, time to detection and time to recovery.
Yeah. Different ways of measuring that. And in the DR chapter, I talk about...
The disaster recovery.
Exactly. But that's like, how quickly can we get back up and running? So we've gone into a degraded functionality. How quickly can we get up and running? So that's thinking about the rebound. And again, for that you have to be very humble as a developer, because the robustness is: the system will never fail. And it's like, but what if we got it wrong? Okay, well, we have to admit things go outside.
And then I guess for recovery, on how you can recover, this is where, if you want to do it well, teams that are serious about this, they do simulations of the system going down. This is where Netflix's very famous Chaos Monkey comes in, where it goes and shuts off parts of the system to test this live. And there's drills that are being done on how our on-call process works, how our decision-making, et cetera, et cetera. This is actually a hard problem. And then you did it once, that's not enough, because you need to do it regularly because the system keeps changing.
I'd like to give a shout-out to Uptime Labs, who are based partly here in London. They do excellent incident training that you can do online in a web browser. It's fantastic. Really great.
We'll add them to show notes below.
They're nice people. So you've got that recovery process, like how do we get up and running? And I think techies are up with me, and then we go into the next two, which the third one is graceful extensibility.
And so graceful extensibility, again, it's a slightly weird name, but it's basically dealing with surprise. When we make our system robust, we're dealing with the known problems.
Yeah. We know what to expect.
What about things we don't expect? Right. If we don't expect it, this is now not about computers, it's about humans. This is why distributed systems are fundamentally a sociotechnical system. It's people and technology working together. We couldn't expect the rabbits. How do we deal with the rabbits? How do we deal with the fire in the data center? Turns out badly is the answer.
But if you're in a corporate environment which is very command and control, quite hierarchical, but you've got a very narrow job description, you're often constrained in what you're able to do. Those organizations tend to deal with surprise badly because the corporate structure is structured around the known. And so often structurally you'll find that difficult.
You mentioned drills earlier. Drills are a great way of helping with surprise. Coming in and having people often from outside saying, this has happened. How will you deal with it? Google had a thing. I don't know if they still do it. SRE pop around. They do a thing called the wheel of misfortune. They had a physical wheel. Today your system has suffered—they spin the wheel—a DNS poisoning attack by a state, insert name of rogue state here. Here's what's happening. How do you deal with it? That is all about people and experiences and trying new things and constantly pushing yourselves.
I guess learning also from others. I remember this very famous outage happened a few years ago at Meta where their DNS system, through BGP, it just all went down. WhatsApp, Meta, everything. To fix this, because they pushed out an update to their BGP table, which is inside DNS to their servers. Now to fix this, they needed to get access to the computer physically, for whatever reason, because they couldn't reach it remotely. So they needed to go into the building.
Now in going to the building, they had to go to a door that protected this system, which was running on their own domain, which was down. So they had to break physically in. But again, this is the thing where they shared the details, which was great, because now everyone else could think about: do we have any of our physical security, for example, tied to our own domain? Because if you have it, you want to change that, learning from what Meta have learned about.
And this learning takes us from graceful extensibility, which is dealing with surprise—that's sort of almost in the moment, are we ready to deal with the surprise?—into the fourth one, which is sustained adaptability, which is: how can we learn from what's happened? Not just the learning. Do we have the environment in which we can learn and share and grow, but then also adapt the system based on that learning?
And it's called sustained adaptability because that is a completely continual process. That's not just today. That's not just when you have an incident. That's when you don't have an incident. What are we doing to learn? And I think your point you raised there was excellent, which is I like reading other people's incident reports. There's a little bit of schadenfreude with companies I don't like, but there's also a little bit of, they've learned something. Can I learn from them and do something that resolves my system?
So when you don't have an incident, read other people's incident reports. There's a place called the VOID. It hasn't been updated for a bit. There's hundreds and hundreds of incident reports there. It's worth going through. So that idea of sustained adaptability is all about us adapting and changing over longer timescales.
For those last two things, sustained adaptability and graceful extensibility, this all comes down to people and culture. Having a proper environment, psychological safety is just absolutely required. If you can't raise issues, if you can't challenge in a safe way, you will deal with surprise badly because things won't get raised, and you won't learn because things will be hidden.
And so that all comes down to behavior stuff. And so the last three chapters of the book, where you focus on that, is looking at some of the people and the social aspects of that. So I've got like 14, 13 chapters—well, 12 chapters, sorry—about the other technical, and then the last three chapters are all around that side of things. So what can we do to make sure that we have that environment of psychological safety? Because without it, you can't actually be resilient.
It's pretty amazing because so far throughout this whole conversation, we talked about building resilient systems. We talked a lot about people, about how to think about resiliency. One thing that we didn't really talk about, just passively, is AI. Even though this book is coming out now, and I know it has a section on AI as well, but it's just interesting to reflect that even though AI is changing how we're building software, it seems like some of these ground truths of how to operate reliable software are a little bit just outside of this realm, right? Or is it?
No, it's not. Firstly, there's some AI stuff in there that can help. I detail quite a few examples around correlation analysis and support around post-incident. There's incident.io doing some amazing stuff in this area.
Saying we can use tools to do a lot of bits.
But really there's more fundamental things here, and the first thing is the macro and the micro. At the micro level, there's some useful stuff we can do here, and I talk about it in the book. The macro level is that none of it stacks up financially, right? So if you're taking a longer view about resilience, if I'm building my product based on a market sector that is fundamentally flawed, these companies don't have a viable path to sustainability.
When you say, which companies are we talking about?
OpenAI and Anthropic and the hyperscalers, right? Conservative analysis by the economists is showing that they need to be generating 2.6 trillion in revenue to justify their capex expenditure by 2030. The entire software market is 1.4 trillion. So macro level, I don't know how that gets sorted. I'm not smart enough. I also don't have any shares in those companies, but that's a separate conversation. I can't solve that.
What I can start thinking about is, okay, if that's the world I'm operating in, how do I deal with that? I can't deal with all that. What I can do, if I'm taking a view to resiliency, is being very careful about how I use AI. For example, if I'm taking a longer view, being more flexible about model choices, that's going to help me because I've got the opportunity to use open-weight models from different vendors.
Even just basic things like OpenAI and Anthropic are pretty terrible when it comes to availability of their core models from their services, which is why everyone I know also will use Google or Amazon or Microsoft as either their primary place they go to those models or their fallback, so they have better uptimes and things. So there's all that stuff we've got to deal with at that level.
There's also the conversation about nondeterminism, and Charity caught me up on this a bit. And I said, the thing is that we're going from deterministic software to nondeterminism. That's a challenge for us. We don't know how things work. And then she said to me, "Our distributed systems were never deterministic." And now they're really not deterministic. And of course she's right, because when we take our distributed system and we add the people in, beyond the rabbits, right, and the fires and the people, we always had a degree of nondeterminism. We're just amping that up a bit. So I think that's a continuation.
So for me, I think it's a confusing landscape. It's not good or bad. I think at the micro level, there's some really great things that we can do.
When we say macro level, that's the tactical things that we can do, right?
I think really what I mean by that is I'm taking the macroeconomic policy situation out of it. I'm saying, as developers, there's things that we can make use of, right? There's some good stuff in there that we can make use of, but I also think there's some hedging we need to do longer term if you're building AI into your product.
There's some simple hedging things. So being multi-vendor, being multi-model, for example, rather than having a single agent, breaking that down into more workflow models. Therefore, you can take bits of your workflow and realize, actually this could now be swapped out for a bit of deterministic code that I write that could run more efficiently.
I think these are all sensible things that you can do. We can't fix this. We can at least deal with our world, and I think there's some sensible things that we can do in that space.
What are the things that we know that we do not know about AI?
I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is. Let's be real. Why did the LLM delete my database? Well, because it has no concept of causality. That's what it comes down to. Has no concept of causality.
There's a glass here. If I push that glass onto the floor, you'll probably work out there's carpet down there. If I ask you what's going to happen, you'd say the glass is probably going to be fine. If I push it onto that floor, you're looking at it saying that's a hard floor. That might break. You're carrying out a bunch of calculations in your head. You've got an idea about the physical properties of the universe. You've got a sense: if I push the glass, this is going to happen. LLMs don't do any of that. They have no concept that if I do A, B happens.
Yeah. Well, there's a push for world models, but these LLMs are not world models.
They're not world models. And world models have been around for a long time. They arguably predate LLMs. They can do stuff like that. So they could make decisions, if you had the appropriate model and the appropriate inputs. They can at least get to the point of: if I do X, then Y happens, and I've been told not to do Y. This is why all the guardrails around LLMs are really not, long term, going to be the right solution. And so I think we expect LLMs to do more than they can actually do because they seem so smart.
And so I think that's the problem, and that leads us to overestimate their capabilities a bit too much, I think.
Or maybe over-rely on them without knowing, and we might have some incidents later.
Absolutely. And I think knowing where and when to use those things. And I think what we don't know right now is, where should I be having—and again, this is wider—where should I have the LLM do it, and where should I just write some deterministic code? There's a lot to be said for just having your agent call a script, and you know that script works, rather than having it implement the script in its magical brain for you.
Well, there's something to be said that it seems both leading labs, Anthropic and OpenAI, have realized that coding is so important. So oftentimes when you ask the LLM to do something, it will start to generate code because it writes a deterministic program, like
Because that part will be deterministic. It's very interesting: when you ask it, it will often just pump out Python or TypeScript or whatever code, especially when it has to do with involving compare this or that. And they have trained these models to default to coding. I'm not sure how this came, but it seems to be working to some extent.
It's also because it's very tractable, right? They are there to generate plausible code. They're there to generate plausible words, right? And in the context of code, they've generated some plausible code that we can now validate. Does it execute?
Yeah.
And so that's a really nice tractable aspect of it. And there are some places where it works really, really well, but there are other areas where it's terrible. When I wrote my book, I used very little AI at all.
Well, you didn't use it for the writing. You used it for...
I don't use it for writing at all. I use it for the research. But in the context of the research, I use NotebookLM basically. In that context, it's plausible, but then I just go and click all the links it gives me and I say, was that there or not? I am checking the work. It's done way more work than I could have done going through all those papers, and God knows there's only so many NIST papers you can read through, right? But in that context, I feel completely fine and able to do that.
In other settings, you're getting now what's this problem called cognitive surrender, which is where you're using AI to write documents that you're never going to read. That's not really around a problem with the LLM technology. That's the problem with us.
Okay. Can we talk about this cognitive surrender and cognitive debt?
Yeah. Cognitive debt's a term that's been around for a while, and that explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools. There's been a bunch of research into this. There's been some studies that's shown that people's ability to think critically is impacted the more they use chatbots and things. And I think we've seen other examples. Margaret-Anne Storey's got a great article all about this, and she's cited some of the research around it as well.
And that's explaining that. Her thing is, we talk about technical debt building up in a system. We have to be careful that our use of AI, it's that cognitive debt. We need to have a shared mental model of what this system is supposed to do, thinking about it narrowly through the lens of a program that we're building. And if our use of AI, as an example, we all go off in different directions, that shared view breaks down. That becomes a debt cognitively.
Cognitive surrender is a bit of a different thing, which is like, oh yeah, it's fine. It's basically LGTM, right? But for whatever the AI tells me.
Yeah, LGTM was typical. Looks good to me. Approving large pull requests where you just couldn't be bothered, or you tried to look at it and for whatever reason you didn't want to go through, or it came from someone who you trusted.
I worked on Google's tool team on old stuff, right? And so LGTM was literally something we pattern match on and takes that as an approval.
Yeah.
So if you saw LGTM, that person's approved, right? And that was where I think that may even have been where the acronym originally came from. I'm not sure, but...
I think it came from Google. Yeah.
Yeah. But it is this idea that it's become shorthand for, I didn't really look at it.
Yeah.
And I think that cognitive surrender thing is coming out in lots of ways. We're getting AI to write documents that no one's going to read, and we've got AI that's reading documents that weren't written by other people. And so the cognitive surrender piece is really that the humans—this comes back to the human being in the loop. There's very subtle differences in those workflows and to the kinds of differences it changes.
There's a big difference, Cory talks about, it's a big difference if you're, say, an oncologist or a cancer specialist. There's a big difference to you as an oncologist working and an LLM doing detection of some of the scans that you're looking at, going, oh, you should look over there, by the way, you might have missed that. That's actually quite useful, right? Versus the LLM-powered AI going, here's all the scans I've looked through. Here's the ones I need you to sign off, right?
The last one, the second one, takes the expert out of the loop and makes them an approver. And often in that context, they've got rid of a bunch of other oncologists. You've got one person into the funnel, which, that's going to lead into issues, right? The other model is we're making people better. I think that's to come back to that original idea of AI. The pitch was it was going to free us from drudgery.
Yep.
Just like the industrial revolution was going to free us from drudgery, but it would elevate us. We'd have less toil. We can be this. We can focus more on critical thinking and smarts. And so much of our current use of AI is not freeing us up from critical thinking. It's causing more context switching. It's causing less time for thinking. People are working longer hours. We've got studies that show this. Now, that's not necessarily AI's fault. It's just, how are we using it? We just need to address that a little bit.
Can we talk about complexity and AI maybe adding it? I'm going to quote you from a few months back. You wrote, "I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space."
Yeah, I can't remember the exact context in which we had that conversation, but the platform space was kind of interesting, right? Because that was an attempt to make things simpler for developers, because Kubernetes became popular and it's not developer-friendly. That's not what it's designed for, right?
But most developers never have to touch it.
No. Well, if you have a platform. If you don't have a platform...
Yeah. Yeah. There's a platform team problem, right? And that's why they get paid what they get paid.
And so I think there was this idea that I was hoping that there'll be a bit of a complexity collapse around that space. I had this hope for things like WSGI, for example, simplifying down those programming models a little bit. And I think there was starting to be a bit of a conversation starting to be had about some of that stuff going on. What then happens, of course, is AI's come out and everyone's like, I've got to AI all the things. And so rather than thinking about what we need and trying to strip that complexity back, we've now just got another piece that requires us to still have all the old complexity, but adding new complexity on top. And somebody's got to look after all that stuff.
Yeah, but I think it goes back to a little bit of, we seem to have more work with AI, and probably it doesn't help that it can add more complexity, especially when we're not being clear of what we want, when we're not being critical of the output, when we have cognitive debt or cognitive surrender, especially with cognitive surrender. So these things might be amplifying the fact that it's just more work, more complexity, right?
Yeah, and I think there's also that connection through to what the AI is doing. It comes full circle a little bit, which is the idea of potentially not looking at the code isn't to say that you don't care about what the system is that you're building. It's to say that you've got a better way of understanding what you're building. So one of the concerns is, if I'm using AI to create code, it's going to create awful loads of debt and a complicated system no one's going to understand. We can't let that happen. We still need to remain connected to the system that we're building, which is why I'm a big fan of modularizing systems. Whether or not you put those modules in different computers, I don't actually care.
Getting AI to work within the modules, having the humans think carefully about what those modules are and the connections between them. Let the AI go hog wild inside those boundaries that are easy to change. The gaps between them, that's where the human beings spend to think about the mental model of how things fit and slot together. That feels a very sane way for us to balance our time and our energies appropriately.
So do I understand correctly that you're saying that that's probably a sensible approach: have the architecture in your head. Know how your system works. Know how the boxes work. Know where there's AI inside it. Maybe it's generating code. Maybe it's running your incident response in isolated ways. And then spend your time talking with other people in the business, on your team, with your customers, because, as you said, software is people's ideas connected. And you have things in your head or you write it down, but now you're using AI as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions, because maybe you have more time to, as an engineer, understand more on the business, on what is working, what is important, those kinds of things.
Yeah. And also to make the structure and the model a bit more explicit as well. That's one thing a good modular structure does. This is why Shopify have been so successful keeping their thing as a monolith, right? It's because they spend a lot of time building their monolithic application as a modular domain-driven architecture, right? Because it makes the structure easier to understand. You can zoom out and see how those things fit together.
But also in the short term, while we're having these debates about, are we going full software factory or not, we don't have to make a one-size-fits-all approach. We can say, look, this is our architecture, be it a modular monolith or a microservice architecture, doesn't really matter. And over here, this area here is where we're trying a full software factory approach, where we don't look at the code anymore, but we specify the interface. We have verification around that. We know what good looks like for that module, right? We've got functional tests, tests that look at, say, some of the operational requirements as well.
These areas over here, we're still grappling. These are really key and really important to us. We're not ready to give up on those yet because we don't have the trust level yet. We don't want to take that risk just yet. And so over here, we're going to work with the AI more in a collaborative way while we work together, and we're still going to be intimately involved with the code because we're not ready to make that step here. But if we approach our whole system as one big black box, then we're forced into making one or either decision. I don't think that's healthy.
Do I understand correctly that you're saying that you think that the software factory, a dark factory, you giving it a spec and then the AI either building it or operating it, that you see that as viable? There are some companies I know that are already all in on this.
Not everyone should be doing it whatsoever. I think if that is our future, and I think there's a lot of ifs there, the question is about how you go about adopting and trying. So what I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way.
And low-risk environments and maybe simpler domains, those kinds of things, right?
But also a lot of dark factories don't create finished products. Beyond the fact that real-world dark factories, there aren't many of them, most of them create components. They don't create the finished product.
Yeah. So in the same way, if you think about a module being a piece of the overarching system, let's have a factory run these bits here. And actually looking at it, if we had to rewrite that from scratch, with a collaborative way with AI, it might take us a few weeks to have to rewrite that if we had to. So actually we're quite relaxed about leaving that to go as is, and it's been working fine, and it's a module around which we've got great test coverage already, for example, where it's not on the critical path from a latency point of view. Or it's a service where we've got really good observability data and really well-defined SLOs, and we've got good steady state. We can start running it as a factory now and seeing what's happening in production. We can have a look at the SLOs. We maybe add some special SLOs around it.
I'm hearing correctly that you're saying that, as a professional, as someone who wants to be up to date with the new techniques that we can use, if you have the opportunity to start to build a software factory, a somewhat autonomous place in a component, in something low-risk, it's probably worth trying it out to see how it works, to see what you learn, to see if it even works or where it breaks down.
But do it in a way where you're minimizing the blast radius a little bit. When microservices came out, the concept, you didn't start by moving your biggest thing. You started to build a small service that didn't really matter all that much, and you were okay to learn from it when it goes down, and it wasn't on the hot path for the business.
Yeah, I think the thing to start off with is getting your head around specs. So looking into spec-driven development, whether that's Kiro or Tessl or Spec Kit, whatever that is. They're all slightly different. Have a look at Birgitta's article on that as a good on-ramp to get ahead of it. Start off by playing around with that because that's the stepping stone. You have to define what good looks like as an input to your factory.
Yeah.
Start with the spec.
What good looks like for the input and also for the output.
Yes. And then it's how you verify. How are you going to go about verifying the output from the factory? Have a think about that. And not just in terms of the normal behavioral aspects of it, but I agree with Kevlin Henney on this. He says he hates non-functional requirements, because if something is non-functional, it means it doesn't work. So he breaks the world down into development requirements, operational requirements and business requirements. I think it's a nice way of doing that.
So along those three axes, how do I verify that the software being created by this module matches my developer requirements, so for, say, maintainability; my operational requirements, latency, uptime; and my business requirement, is it working? Features working. As long as you've got a way of validating your software around that. And if you don't know what good looks like for those three things, guess what? This isn't for you.
But you can still do specs, by the way. But you can't do a factory.
Yeah.
But you can do specs, and you're edging your way there. And I think it might not work. There may well be some blind alley. It could be model-driven development all over again, which we do not have time to get into because I get very angry about it. But it could be that. But I have a hope it might actually work this time, unlike MDD.
And then, as closing, I think a lot of what we talked about is it's important to hold the architecture in your head, to think about the architecture, to think how to modularize. Now, for engineers who want to get better at this kind of high-level thinking, strategic thinking, architectural thinking, outside of reading your book, which I think brings a lot of really good ideas on architecture as well and not just resilient systems, but outside of that, what would your recommendation be for a tech
Professional for an engineer who's like, "I'd like to get better at architecting systems, to know what good looks like."
I think there's a few books I can point people at. I think Fundamentals of Software Architecture by Neal Ford and Mark Richards is a good read. Just to be clear, they're friends of mine, so I'm not unbiased in recommending it. I would say around modularity, Vlad Khononov's book on balanced coupling is fantastic, and his book on DDD is also very good. But that balanced coupling book is really great. He's also developed some skills in Claude to take some of those ideas, so definitely take a look at those.
I think if you're looking at architecture, you think in terms of modules. For me, I don't agree with everything in Mark and Neal's book. There's lots of great stuff in there, but for me, actually, it all starts—I mentioned earlier that writing the second edition of Building Microservices was really all about smuggling in getting people thinking about a thing called information hiding. It's a very old idea from the 1970s. Still comes back there to me. Modules are still the whole thing for me. So I would read Vlad's book, and if you want to go read the original Parnas papers from 1971, 1972, they're still worth reading. But those would be good jumping-off points.
I would say you can still—I think the nice thing about thinking in terms of modules is even within an existing codebase, you can start to modularize it in some small ways. You can start to think about boundaries. You can think about clear separation, about using encapsulation as a way of information hiding.
And I guess it's fair to say, look, if you have an idea on how you can make your existing codebase a bit more modular, you have zero excuse not to do it. We don't trust agents fully, but this is a textbook example of if you can say, "Here's my goal, refactor this," if it would have taken by hand X amount of time, it will be faster. So you should be empowered to at least experiment with these, right?
Especially if you've got a decent test suite.
And if not, start with that.
Start with that. If you've got a statically typed language, you don't even need to. Just a good old-fashioned IDE does an amazing job, right? Dynamically typed languages now don't have the excuse, especially if it's a modern one that's well understood by the LLM. So if you're using Python or Ruby, it's so much better now than it used to be for refactoring codebases. Absolutely, you need to get the test suite in around that. But you can start modernizing, and that's about building. And that's also a little bit coming back again to taking—we don't want the architecture only in your head, right? We want the code to represent that mental model.
Yep.
Because we communicate through the code, and maybe in the future we'll be communicating through the specs. And when you make those things more explicit, then I think that helps everyone stay on the same page. George Fairbanks wrote a book. I've forgotten the name of the book now, but in it he has a pattern called an architecturally evident coding style, and that we should code in such a way that it clearly communicates the architecture of the system. And I think our programming languages are quite bad at having a good concept of modules in general. So I think we have to go a bit above and beyond ourselves, but we could do more to make the architecture of our systems that little bit more explicit.
This is also why I like spending time in production, because especially if your modular structure ends up being a process structure—in other words, it's process boundaries—I can go look at the networks to find out my architecture. I shouldn't have to, but at least I know production is truth. It's like football is life, production is truth.
So where can people get your book?
You can go and order it now. If you can go and buy it from a local bookshop, that's always a great idea. It's also going to be available for people on the O'Reilly platform in digital form. You can go and read it as part—if you're a subscriber, you can get it there as well. But lots of people love having the physical copy.
And we have these in the show notes below, so check them out to get the book directly. And also you've been doing some videos, right?
Yeah, I've been doing some stuff on Dave Farley's Modern Software Engineering channel. It's a good excuse to do some semi-ranty monologues to video and have some fun chats. So go for the Modern Software Engineering channel on YouTube, and you can find videos from me and Dave Farley, and Kevlin Henney's there. We've got stuff from Kent, Trisha Gee, Emily Bache. Dan North is there as well. I'm pretty sure I've forgotten somebody else. There's a couple of other people there as well, but we try and put videos out once a week, and it's a bit of a fun place to—
We're also linking that one.
Yeah.
And if people would want to work with you, where can they find you? How can they find you? What kind of stuff can they get help from you for?
I do internal talks with people, internal events. I do customized training courses on a whole variety of things around resilient systems and security and microservices. I also do a lot of consulting for clients, helping them around rearchitecting their systems and stuff like that. If you want to know more about that, go to my website, samnewman.io, and then you can contact me there. You can also find me on LinkedIn, but if you come straight to the website, use email. I'm still a bit old-fashioned like that. And I work with clients all over the world, so if I'm able to help, head over to samnewman.io. My contact details are there.
Wonderful. Sam, thanks again. This was great.
Thank you.
Thanks very much to Sam for this conversation. I really like Sam's three rules of distributed systems. One, it takes time. Two, sometimes the thing you want to talk to isn't there. Three, resource pools are not infinite. Sam said how most of the outages happen because of this last one: resource pools not being infinite, and services are getting saturated. But I was chuckling when he was talking about rabbits chewing through networking cables between office buildings. Then again, a few weeks ago, with thousands of flights stranded on the East Coast of the US, it was because workers accidentally cut a telephone cable in New Jersey used by the FAA.
Another food for thought from Sam is how failure should be handled differently depending on the type of business. An e-commerce shop might decide to show you the item when the inventory service is down. Worst case, after you order and pay, they just refund you because they're like, "Oh, we don't have the item." Basically, failing open is fine in this case. But on a concert ticketing website, you don't want to sell a ticket that might or might not be there. And so failing closed is the right call to make because, again, as we discussed in the episode, if you sell the ticket, people might book flights, they might get upset, you might lose them as customers.
Finally, I appreciated how level-headed, perhaps even cautious, Sam is about AI. He's got no doubts that AI is very useful at the micro level, basically at tactical, at things that we can already improve. But he is worried about the economics of the model providers. His suggestion is pretty pragmatic: hedge your vendors and your models. Be multi-vendor and multi-tool, and wherever you can, swap non-deterministic AI agent workflows to deterministic ones. Basically, have LLMs generate deterministic code, which I find very, very refreshing thinking.
Check the show notes below for the story about scaling Uber in the early days with Uber's first CTO, Thuan Pham, that we discussed, and it's an awesome story, and for more of The Pragmatic Engineer deep dives on resilient systems. If you enjoy this show, do subscribe on your favorite podcast player. And thank you very much if you recommend the podcast to your friends or your colleagues.
Article published · Updated