Kelsey Hightower on Kubernetes, Career Impact, and Retiring at the Top
The Pragmatic Engineer
Kelsey Hightower started in tech installing DSL modems door-to-door at 19. He later became a Distinguished Engineer at Google and one of the best-known voices in the Kubernetes community, and he retired about three years before this conversation. In this long interview on The Pragmatic Engineer podcast, he traces that path through self-directed learning, several job changes, the configuration-management and container era, and his years at Google.
One idea runs through his account. He says the point of technology is to solve human problems, and the point of a career is to make real impact rather than generate activity. He applies the same lens to generative AI. He calls himself a skeptic of its "naive promotion and adoption" rather than a hater of the tool.
Early lessons: McDonald's, a graphing calculator, and AutoCAD
Hightower's first paid job was at a McDonald's within walking distance of his house, which he took as soon as he could get a work permit at 14. He valued dealing with "real people that are in a hurry." He also noticed that many customers treated staff as just an intermediary between them and their food. By 15 he was an assistant manager. On nights and weekends, the other managers left him the keys, and he closed out the store by counting the money and faxing a large spreadsheet to corporate. He describes it as early training in taking responsibility, even over adults.
He came to computers almost by accident. After moving from California to Atlanta, he had missed several months of school and needed extra classes to graduate on time. As an athlete, he was wary of the "computer club." He ended up enjoying it, especially AutoCAD, and took second place at a state-level competition. He says he might have won if he had gotten his design to print. A classmate also showed him that a TI-86 graphing calculator could be programmed in TI-BASIC. Typing in a Snake game copied from a magazine was his first exposure to programming.
Choosing a certification over college
Georgia's HOPE program would have covered tuition at public universities for students with a B average. Hightower tried a nearby school and left within about two weeks because the pace felt too slow. It was 1999. People were lining up for new versions of Windows, high-speed internet was arriving, and the dominant story was about skills rather than degrees, with Bill Gates dropping out as the famous example. He didn't know any programmers or system administrators, though. Job postings asked for skills on Sun and IBM enterprise systems that he had no idea how to acquire.
While still delivering pizzas, he found a $35 A+ certification guide at a bookstore. Some support job postings asked for exactly that credential. He read the book cover to cover repeatedly and used the practice exams on the CD as a fast feedback loop. When he got a question wrong, he went back to the book. He says he finished the proctored exam in roughly 10 minutes of the allotted hour or more. It was, in his words, the first time he felt that putting in effort reliably earned the credential.
Asked why college never appealed, he says he never saw the endgame. The people he admired weren't taking that path, school had been easy for him, and the certification felt like "a path that I would control." He later came to see the difference between a bachelor's degree, which he compares to more of K–12, and graduate study, where you challenge or add to a field. At the time, though, he knew no one who had gone that far.
From DSL installer to small-business owner
With A+ and Network+ certifications, he joined contractors hired to install DSL for BellSouth. The unionized phone technicians, he explains, wouldn't touch customers' computers. The contractors ran cable, crimped Cat 5, and decided between a USB modem, which he says broke constantly, and installing a network card. On Windows 98, he estimates, about 20% of driver installs crashed the machine and turned a routine job into a troubleshooting session.
Small businesses kept asking how to get all eight of their computers online instead of just one. At first he didn't know. Then he bought a roughly $50 Linksys router, worked out how to share one connection, and started handing out his card for after-hours jobs. On his first independent job, the client wouldn't write a check to an individual. He invented a company name on the spot, "Digital Gateways," then found he needed a business license and a business bank account to cash it. He realized the side work paid better than the day job.
He eventually opened a small computer store outside Atlanta. A local shop vouched for him so he could get a distributor account, and he built custom machines from customers' parts lists. The store also served as the base for his service calls. Around 2000–2001, he added Pro Tools conversions for music studios moving off analog mixing boards. At the same time, he managed a comedian friend from high school. He took the job after watching the friend hold both a predominantly Black audience in Atlanta and a predominantly white audience an hour north with a completely reworked set. The comedy work led to road tours and IT work for an entertainment company.
He ran the business for about four years and says it was successful overall. He also argues that people "over glorify entrepreneurship." A parts-and-service business doesn't grow like a software company unless you open many locations. Employees get paid first, the owner last, and some months the owner draws on savings. When he wanted to settle down with a family, a salaried job started to look good.
Google's data centers and a metric he could control
Google had data centers near Atlanta. Hightower applied for a data center technician role, and the pay was striking to him: eight-hour days, a paycheck every two weeks, no invoices, no inventory. In the interview he struggled with Google's Linux questions but knew FreeBSD well. He says he got lucky because one interviewer had a FreeBSD daemon tattoo, and the conversation shifted to FreeBSD.
He describes the data center, around 2004–2005, as a huge warehouse, and says the figure people repeated, possibly exaggerated, was something like 200,000 servers in one place. Unlike the messy data centers many people know, everything was systematic. Machines arrived perfectly wrapped, PXE-booted, and burned in. Cabling followed strict rules for service loops and for keeping fiber and copper separate. A fleet that size could route around a few hundred broken machines. Technicians walked the floor with crash carts, diagnosed failures, and logged a prediction in the system, such as "replace the SATA controller" or "DIMMs one and two." Technicians were measured on how accurate those predictions were, judged by whether the machine came back for repair within something like 30 to 60 days.
He mentions that Tim Hockin, later known as a networking lead in the Kubernetes community, built a small device with about nine lights that plugged into the motherboard and indicated which DIMM slots were likely bad. Hightower learned to trust it. He says he kept his accuracy in the high 90s while repairing about three times as many machines as other technicians. When his score once nearly dropped below 90, he wrote shell scripts that diagnosed disks, RAID, and memory together and re-checked after reboot. After that he could move quickly without losing accuracy.
The host observes that this was stricter measurement than most software engineers experience. Hightower says it didn't feel oppressive because he could control the outcome, and the metric matched the actual work. He checked his numbers each morning and adjusted his approach. He calls it feedback that "was helpful for me, not just my manager." He also began changing jobs every three to six months to broaden his skills. He estimates each move brought roughly a 25% raise, so after several moves his salary had doubled.
Web hosting: activity versus impact
At Peer 1, a hosting company he describes as a Rackspace spin-out associated with ServerBeach (tagline: "latency kills"), he saw end-to-end automation for the first time. Many customers rented servers to host game servers. An order form triggered the whole process: PXE boot, PHP scripts that configured RAID and updated firmware while the machine ran from memory, VLAN assignment, OS installation, and delivery of an IP address and credentials in about an hour. Returned machines went back into the pool. Where there were no APIs, the team used curl scripts and command-line tools. For Windows 2000 servers, they screen-scraped Active Directory logins and mouse movements to apply patches. Customers paid about $99 a month, so there was no time to spend a week on a problem, and the lesson he took was "there is nothing you cannot do."
The inflection point there came in tech support. Every technician sat in a round-robin phone queue, and anything unresolved became a ticket. Tickets piled up across shifts, and customers waited days. Hightower stopped logging into the phone queue and cleared tickets with small scripts: vacuum the MySQL database, upgrade PHP, close. He told colleagues who got stuck on a call to open a ticket quickly and move on. When his manager, Mike, asked why he wasn't taking calls, Hightower explained the idea, and Mike changed the process so a couple of people stayed off the phones with a promise to keep the queue at zero.
He calls this the first time he understood "the difference between activities and impact." Answering many calls is activity. A team ticket queue at zero at every shift handoff is impact.
Financial services: doubling his salary and learning maturity
The next inflection point, he says, was about maturity and staying long enough to live with his decisions. He interviewed on his lunch break, in a shirt and tie for the first time in his life, at Total System Services (TSYS), a credit card processor. He assumed the questions would be beyond him. Instead he was correcting the interviewers' Linux flags. The offer took him from about $45,000 to $90,000, and he says he "almost drove off the road." His former manager at the hosting company, Joe Rodriguez, whom he credits with first making him a software developer, protested his departure until he heard the salary. Then he asked whether they were hiring.
At first the company felt slow to him, with runbooks, regulation, and change windows. He later understood why. He says a processor had around seven seconds to respond to Visa before a transaction defaulted to a decline, so a careful change was worth the wait. He learned to deal with "the no's," with executives, and with building trust among senior leaders.
His defining project there involved a move off the mainframe onto Java. The Apache front end used a Java connector that consumed so much memory per connection that the servers kept falling over, and failed change windows were costing penalties and chargebacks. In a dev environment, Hightower had replaced it with Nginx, ported all the Apache routes, redirects, and legacy config, and seen memory use drop to a fraction. The objection was that Nginx wasn't certified. His answer was that it came as a Red Hat RPM. Leadership gave him one chance in a midnight-to-6 a.m. change window. No one else at the company knew Nginx, and he says most colleagues thought it was a bad idea, so his reputation was on the line. After about two hours it was working. In his account, memory that had been maxing out a machine he rounds to 32 GB settled around 2 GB. The team verified it the way they always did: someone bought gas with a credit card, and they watched the transaction land in the Oracle database. He didn't sleep and watched top until the next morning's peak passed without alerts.
The CTO had promised him a steak dinner, but Hightower was vegetarian. He asked for pizza for the whole floor every day for a week instead, partly to turn "I succeeded" into "we succeeded." His conclusion from the episode was that "it's not about just having the right solution." You need consensus and buy-in, and you have to put your reputation behind the change. He stayed about three years.
Puppet, open source, and the elevator introduction
At TSYS he introduced Puppet (around version 0.24) to replace manual, ticket-driven server work. He puts it bluntly: some people have "20 years of one year experience." He hid Puppet behind Jira. Engineers picked RPMs and environments from dropdowns, and once a ticket was approved, a bot a friend had named after a video game character extracted the fields, ran the right Puppet manifests, and posted the output back to the ticket. Developers could order a full environment with database, IBM MQ, apps, and firewall rules and receive it almost immediately. Most of them never knew Puppet existed.
Contributing to open source wasn't allowed at work, so he contributed to Puppet on nights and weekends. His manager came back from a conference and told him he was "doing DevOps," a label Hightower half-resented. The manager had met James Turnbull of Puppet Labs, co-author of the Puppet book on Hightower's desk. When Turnbull visited, he greeted Hightower in the elevator with "Kelsey Hightower? Oh, we know you. We love your contributions," to the surprise of a manager who didn't know the team contributed anywhere. Turnbull said he had no recommendations for their setup. On the drive to the airport, Turnbull took off his tie and revealed his tattoos, and invited Hightower to speak at PuppetConf. That was his first real conference talk, and he put some comedy into it. Afterward, Puppet founder Luke Kanies offered him a job over coffee in Portland.
When Hightower resigned, his manager, with whom he says he had a sometimes difficult relationship but who "helped me mature," told him, "I'm surprised you stayed here this long." About seven years later, Hightower visited that office again. The company was running etcd and Kubernetes in production, and some of his old tooling was still in use. He took it as evidence of lasting impact, because the culture of bringing in new technology had survived without him.
Go, Docker, and the road to CoreOS
Hightower says he didn't see containers coming. At Puppet Labs in 2011–2012, he believed configuration management plus DevOps was the end state, with the only competition among Puppet, Chef, and Ansible. VMware and AWS were integrating these tools, and the team thought they were just getting started.
Go was the first crack. Puppet was hitting Ruby performance limits (the global interpreter lock), and the team had experimented with JRuby and Clojure and considered C or C++, but worried about losing Ruby contributors. Hightower prototyped Facter, Puppet's machine-fact agent, in Go. He compiled it on his Mac, copied it to a Linux box, gathered facts in parallel, and got results in a fraction of the time. The team declined because Go didn't support Solaris or AIX, which mattered for enterprise customers. He says he understood that decision. Terraform then challenged the node-centric worldview by focusing on cloud APIs. Mitchell Hashimoto had come from the same Ruby/DevOps world with Vagrant, and Terraform used Go.
When Docker appeared, he says most of Puppet dismissed it as a fad, "not disrespectful," but not seen as a challenger. He only understood it after leaving to become a VP of engineering. There he rewrote Java-heavy microservices in Go, decided they didn't need Puppet anymore, and open-sourced confd, which pulled values from etcd and generated just enough configuration. Docker's value, in his view, was repeatable packaging without the work of building RPMs or Debian packages for custom apps. It swallowed the Python virtualenv and Ruby gem environment problems into a single artifact. He thinks that's why developers embraced it.
CoreOS appealed to him because of an idea he summarizes as "Google's infrastructure for everyone else": a minimal OS that only ran Docker, written in Go, with a key-value store (etcd) to sync configuration across machines. He had tried to read the Borg paper and Mesos while at Puppet and couldn't justify the complexity. CoreOS fit his ops instinct to keep the OS small and isolate whatever changes.
He met the CoreOS team at the first GopherCon. Rob Pike opened with a keynote that went from Hello World down to the compiler. Hightower felt the organizers' introduction was too subdued for Pike, so he walked backstage and asked the two community organizers, Brian Ketelsen and Erik St. Martin, whether he could MC. They agreed. He told the audience that every speaker needed to be greeted loudly enough to get the conference kicked out. When his own turn came, with no one to introduce him, he slipped behind the curtain and walked back out to applause.
His talk, "Go for system administrators," was a live demo in Go's present tool, whose slides can run code. He started a PXE (iPXE) server written in Go from inside the slide deck, switched a VMware Fusion VM's network adapter, and booted machines while HTTP logs scrolled on the slide. He recalls Pike and the Go team watching closely from the front row, and members of the Ruby community arriving from a Ruby conference down the street after seeing the buzz on Twitter. The CoreOS team was also watching, and he was booting CoreOS. His lesson: "every presentation is an interview."
He also answers a common skepticism about stage speakers, the suspicion that someone else wrote their demos. At Strange Loop, an audience member challenged whether he knew if etcd was CP or AP under the CAP theorem. He demonstrated live on a three-node cluster: stop two nodes and writes fail, so availability is sacrificed for consistency; add a node back, reach quorum, and writes work again. He noted that the Raft paper doesn't dictate cluster membership or how an implementation behaves in each mode.
Kubernetes arrives, and CoreOS goes all in
At CoreOS, the team was building Fleet, which stored systemd unit files in etcd so the appropriate nodes would pull and run them. Then Google gave them about a day's notice, under embargo, that it would announce Kubernetes, with Red Hat, at DockerCon. CoreOS wasn't in the story. Hightower stayed up all night with the undocumented Go code, reverse-engineered a script that assumed Google Cloud, built binaries, applied patches, and got Kubernetes running on CoreOS on bare metal. There were no volumes or config maps yet. You could stand it up and submit a config that used Docker underneath. After Google's announcement hit the top of Hacker News, CoreOS published his guide, which also reached number one. People downloaded CoreOS to try Kubernetes. He replaced the planned topic of a keynote a week later with Kubernetes demos.
The rest of CoreOS wasn't convinced at first. Kubernetes competed with Fleet, Docker was dominant with Docker Swarm, and Google's earlier C++ container runtime, "Let Me Contain That For You," had drawn little interest. For two or three months he contributed on nights and weekends, and he credits founders Alex Polvi and Brandon Philips for not objecting. Eventually, with Hightower now CoreOS's product manager, the company gathered in its San Francisco basement office and went all in on Kubernetes, deprecating Fleet. Polvi coined the term "operators," and Hightower and a colleague worked on CNI, Kubernetes' networking layer. He says Kubernetes felt like "the thing I would build if I only knew how," a synthesis of 15 years in data centers, promise theory, and Puppet's strengths and limits. KubeCon itself grew from people in that GopherCon community. A small company called Kismatic proposed a GopherCon-style event and funded the first one. He notes that about 12 years later, under the CNCF, the Amsterdam edition drew roughly 13,000 people.
Why Kubernetes won
Asked what made Kubernetes break through, Hightower gives four reasons.
The first and biggest was Docker. Mesos/Mesosphere and HashiCorp's Nomad had their own runtimes, but Docker had already won broad consensus. Kubernetes could reuse existing Docker containers, so "now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers." He believes Docker Swarm's weakness was design. It stretched the Docker API, which worked well for one node, across many machines, and kept bolting on storage and networking it was never built for. Kubernetes, by contrast, adopted etcd and Docker and combined them with lessons from Google's Omega paper about what should follow Borg. Hightower paraphrases one of those lessons as scheduling that needn't be so complex if the scheduler gets more metadata about workloads. Kubernetes also filled gaps such as Docker's entrypoint. Instead of shell scripts imitating an init system to run, say, Nginx alongside an app, you could run separate containers together, which gave people clean building blocks.
Second, he says Kubernetes moved from "infrastructure as code" to "infrastructure as data." Instead of conditionals and loops in a DSL, you declare containers and resources in an object with a status field, hand it to an API, and let control loops act on it. Any language or tool could generate or modify that YAML, which he believes made onboarding feel easy: kubectl apply and off you go.
Third, and closely related, it gave infrastructure "a type system." Types reduce cognitive overhead, enable static analysis and validation, and make automation safer.
Fourth, he credits Brendan Burns with first-class extensibility. In Mesos, an extension was almost a separate system, such as Spark, Hadoop, or Marathon sitting on top. In Kubernetes, you describe a new object type, for example a firewall, and all the existing tooling works with it. Cert-manager is his example: declare the domain you want a Let's Encrypt certificate for and where it should live. A controller could be written in Python, or even as a bash loop that updates the status field. That, he says, is what let Cisco, Red Hat's OpenShift, and the rest of the ecosystem participate.
NASA or Google, and redefining DevRel as impact
By the end of his CoreOS years, with a Kubernetes book co-authored with Brendan Burns and Joe Beda, commit access, and constant keynotes, Hightower was already thinking about his exit. He observes that the tech industry has little tradition of retirement. Pioneers are only now leaving, he notes Rob Pike has just retired while Linus is still working, and unlike athletes, engineers get no signal from their bodies.
He nearly went to NASA's Jet Propulsion Laboratory. He toured the facility shortly after The Martian had filmed there, watched the new Mars rover being tested on a track, and saw scientists replicating Martian surface conditions. For the first time, he says, he saw people using technology for a larger human purpose rather than "just to make more apps." One interview question asked how he would deflect a meteor if it had to hit one state. He answered with radical transparency: a 24-hour live stream explaining every decision, the trajectory, the evacuation plan, and even estimated casualties, so there would be no room for conspiracy theories. He had signed the employment agreement to lead infrastructure for the Mars mission in Pasadena.
Google called. He initially resisted, since he already felt part of the Kubernetes team and feared becoming "a cog in the wheel." Google offered developer relations with the freedom to define the role. He believed that doing only the external version of DevRel (conferences, tutorials, evangelism) was activity, and that he would eventually be fired for it. So he focused on revenue and product. He asked sales to call him for any Kubernetes-heavy customer and flew to whiteboard for hours at companies like Disney and Walmart, including in countries where Google Cloud had no region yet. His financial-services background let him act as an executive sponsor who could go "from hello world to hello revenue." He read other teams' OKRs to find where he could help. He adds that no one required the revenue focus of him.
His signature program was "empathetic engineering." In his view, Google's hardest problem was "what to build, not how to build it." In the first session, he asked the Kubernetes team, including Borg veterans and original Kubernetes creators, to install Kubernetes without scripts. After an hour of confusion over Docker versions and distributions, he showed them his prepared approach, and the engineers themselves proposed improvements. One proposal was OS packages. Another recognized that people still needed to know ordering and config file locations, which led to kubeadm. Because he wanted people to understand how things work rather than just hide them, he also wrote "Kubernetes the Hard Way," which he used to help teach GitHub engineers to run Kubernetes on bare metal before GitHub's acquisition by Microsoft. He says the empathy sessions later became an official program with a dedicated team and product manager, integrated into onboarding.
He tried to know the two most impactful things in several product areas and nudge teams to discover them rather than dictate. He distinguishes "launches" from "landings": a launch is shipping and celebrating, a landing is customers actually paying. One example was pushing Go support into Cloud Functions, since Lambda already had it. The team sprinted to an alpha, and he launched it on stage at GopherCon with the PM.
Climbing the ladder deliberately
Hightower went from roughly L5 to Distinguished Engineer (L9) in about seven years, four promotions. Contrary to the host's guess, he says he was explicitly aiming at each next level: "if you show me the top of the mountain, I want to get there." In his organization, there was initially no L7 individual contributor role, so one had to be created. He explains that lower promotions are fairly formulaic and decided locally, while higher ones require people across the organization to agree on your impact, which keeps local favoritism in check. He mentions being told at one point to slow down and actually make an impact before seeking another promotion, and calls it good feedback.
One year he skipped the standard packet template and wrote in the first person about what he worked on, what he didn't, whom he brought along, and the results. By then he sat on promotion committees and knew how dry and safe most packets read. Some colleagues pushed back on the style, but he was promoted on it. His point is that Distinguished Engineers get there by different routes. For him it wasn't primarily code volume. It was influence on Google Cloud's culture and expanding beyond Kubernetes into serverless, Postgres interfaces for Spanner, and Go's cloud support.
The Microsoft offer
Hightower says he dislikes ultimatums. A competing offer establishes your market value, but using it as leverage can sour relationships. When Microsoft approached him, he had no plans to leave and didn't personally like Windows, Azure, or .NET, though he liked GitHub and VS Code. His wife encouraged him to set his ego aside and see what was out there.
The process was unlike anything he'd experienced. A first recruiter asked for a résumé, which he hadn't needed in about 15 years. That recruiter was swapped out with an apology, because this process wasn't about a résumé or quiz questions. Microsoft pointed to his GitHub history, his book, and his Wikipedia page, and arranged meetings with leaders, including Scott Guthrie, and anyone else he wanted to talk to. He was impressed by the vision, the GitHub acquisition, Microsoft's Kubernetes commitment, and hires like Brendan Burns. Afterward, Satya Nadella emailed him personally, with the offer attached as a PDF. He describes it as adding "a zero" to the equation, something that the kid who chose an A+ certification in 1999 didn't know existed. He countered, and Microsoft countered back higher.
He told his manager, Greg, whom he had reported to for six straight years, that he intended to take it and that it would be financially irresponsible not to. He explicitly declined to make it an ultimatum. Greg looked at the offer, smiled, and said, "I want you to know you're worth every penny of this." Hightower says Greg knew him better than anyone he'd worked with. He reasons that leaders managing thousands of people simply may not know an individual's pay. Within hours, Google matched with additional stock, and he stayed. He calls it the biggest raise of his career. He later became Distinguished Engineer at Google and tweeted "different company, same team."
Some months later, Nadella asked to meet him in San Jose. Hightower had just read Hit Refresh on the plane, and he broke the ice by teasing Nadella for posing by the window overlooking the mountains. Nadella said Microsoft executives had discussed which leaders "got away" and Hightower was on that list. According to Hightower, Nadella reflected that, with Thomas Kurian having just arrived at Google from Oracle, Microsoft had made him an offer "as if you were running away from something" when it "should have gave you something to run towards." Hightower says the conversation made him feel he belonged to the industry rather than to one company, and he felt comfortable retiring within the next year or so.
Money as "freedom tokens" and learning to slow down
He retired around 43. He attributes it to minimalism: his lifestyle didn't change as his income rose, and he cared nothing for jewelry or cars. Money became "freedom tokens." He calculated his target based on conservative interest income, "vanilla US Treasury bond" returns, rather than stock growth. Once that became visible, he made career decisions accordingly, such as choosing CoreOS over higher-paying offers and choosing Google over NASA partly for the pay-for-performance trajectory.
He admits he was "never great at work-life balance" and probably came close to burnout. After his daughter was born, he took an overnight NOC job so he and his wife could split childcare, then studied after she went to sleep. He compares it to sports: champions play more games than anyone else but tolerate the fatigue because the wins make it worth it.
Around 37, he started asking why he was doing any of this. He concluded he had been partly lying to himself about loving the job, and that as a Distinguished Engineer he was "a junior person" at living. The turning point was staying an extra day after a conference in Budapest, spending three hours at a thermal bath and an escape room with Liz Rice's team. Back home, those were what he talked about. He started small practices: reading lyrics while listening to music, attending school board meetings, teaching his daughter to drive, and deep-cleaning his own home. He says that routine reminds him of his good fortune and teaches him that slowness is a luxury of success. His keynotes changed too. He began asking how he wanted audiences to feel, sometimes excited, sometimes "a little bit embarrassed" by needless industry complexity. This is also why he pushes back on the industry's overemphasis on productivity: "you're a human. You're not a computer." Three years in, he calls himself "a junior retired person" who still advises, invests, and speaks, and he admits he is still holding on to that part of his ego.
Advising and investing
He argues that senior engineers are already advisors, because they tell stakeholders what they actually need rather than building exactly what's asked. Formal startup advisory, in his experience, starts out looking like a waste of time. Advisory shares are usually worthless, diluted, and tax-confusing. He decided never to work for free. His terms evolved to a quarter point to a full point of equity depending on stage and risk, one-year vesting with no cliff, a 10-year exercise window, and a monthly retainer, which he thinks of like dividends and gives as examples of $1,500 to $5,000. He believes advisory should last about a year and produce real impact rather than superficial advice.
His first meaningful exit was Pixie Labs, which did eBPF-based Kubernetes observability without agents. While it was still in stealth, he told the founders that few people would switch observability stacks for a slightly different technique. He pushed them to delay launch by a few months and reposition toward system administrators, using Pixie scripts as command-line-style tools that could span many clusters. He gave the opening keynote at their launch event and interviewed the founders. He says acquisition offers arrived the next day, and New Relic bought the company, accelerating his shares. Word spread among VCs, and other founders came to him, including Guillermo Rauch of Vercel during its move deeper into the stack, and Docker. His advice to aspiring advisors: be a deep domain expert, don't just say "at Google we did it this way," and don't be offended when a company outgrows you.
A people-first view of crypto and generative AI
Hightower says he is "a people person through and through," and that this makes him an outlier in an industry where cynicism and gaming the system feel normal. He applied the same lens to crypto. He didn't question the technology, but he argued with enthusiasts who wanted to debate transaction settlement and encryption while ignoring what currency changes mean for real people. He points to countries that have been through currency resets, where people with a little money can end up with none and retirees can't earn more.
On generative AI, he acknowledges a personal side. Programming has aesthetics and cultures, like Matz's vision for Ruby, Perl's subculture, and Go's, and to him writing code is decision-making. He's unimpressed by talking to a computer. He jokes about adopting a "zero token architecture": learning things and thinking for yourself. His deeper objection is that "we trained it." Everything engineers wrote and shared is inside these models, and he won't put the machine above a person. He pushes back on a "healthy subset" of people who question why humans are needed to build software. The job, he says, has always been solving human problems with whatever technology fits, and software isn't required for every human endeavor. He worries about newcomers who hear only that AI does everything.
In due diligence for the fund he works with, he looks at founders' code, AWS bills, architecture, and GitHub practices. He asks them not to say "AI" at all during the meeting. Forced to describe concrete problems, good founders explain the current industry approach, its drawbacks, and how they improve on it. He sometimes points out that a task like parsing PDFs may call for OCR or smaller, non-generative models rather than an LLM, which lowers cost. He also tells them to lead their websites and demos with value rather than prompt boxes. When startups want to "pivot to AI," he compares it to kids' soccer where everyone runs to the ball, and tells them to play their position.
His example is Mass Driver, a company he advises that offers visual infrastructure-as-code on top of Terraform. You draw a line from an app to a database and credentials flow under the hood. As customers began saying they'd just let Claude manage the cloud directly, he anticipated trouble. He has seen what humans do with the full AWS console and expects agents to wander similarly, spinning up Lambdas and load balancers. Instead of a naive pivot, Mass Driver repositioned its features: the context behind the visual view became a "context engine" that agents can query, scoped to the resources actually in use, and existing features became guardrails and skills. Humans, pipelines, and agents all use the same structure. Claude becomes "an alternative interface," and the skills file becomes an implementation detail. That, he says, is strategic adoption: "I'm not just like a Gen AI hater."
What AI should teach us about APIs and documentation
Giving AI "a little grace" on hallucinations, which he assumes will improve or be caught by humans in the loop, he highlights what it should do well and what it reveals. Humans turn repeated code into a function, then a library, then a framework feature, sometimes pushing it all the way into the OS or hardware, as with encryption offloading. Regenerating the same blocks with an AI every time "is still insane," and having it refactor the mess later is wasted energy.
He believes the MCP wave shows that many APIs were badly designed even for humans. Getting a cloud VM can take about seven imperative calls (VM, network, storage, VPC, credentials) rather than one intent-based request. MCP wraps these into intent. He says Kubernetes did exactly the same thing, where a Service or Ingress fans out into load balancers, certificates, and DNS. So he's baffled by treating it as new enough to need "a whole conference": "This is just API design." He hopes engineers stop fighting RPC-versus-REST battles and accept intent-based APIs, even versioned ones like create-VM v1 and v2. Natural-language prompts also expose how rigid many programming languages are at expressing intent.
On documentation, he's frustrated that many languages and libraries offer only reference listings and no complete working example, which pushes developers to Stack Overflow and blogs of uncertain freshness. AI tools close that loop, and he grants that it's an improvement because Stack Overflow answers were never perfect either. Watching people now write large markdown files to give agents context, he wishes the same effort had gone into human documentation years ago. He doesn't want AI to simply generate thin docs. Good documentation has personality and explains why something exists, which specs it follows, what good, bad, and high-performance usage looks like, and when not to use it. The host adds that Cat Cosgrove cited documentation as one reason Kubernetes won.
Advice for engineers worried about AI
For experienced engineers unsettled by agents writing code, Hightower's first step is reflection. Software developers have automated away other industries for decades. He cites diagrams showing the iPhone absorbing dozens of standalone devices and says some attribute newspapers' decline to the internet. So engineers shouldn't expect sympathy from other professions, and should ask whether their enthusiasm simply reflects being positioned to benefit, as at Anthropic or Nvidia. He says he doesn't want them to feel guilty, only to see the whole picture.
Second, ask what your job actually was. If you believed it was being the only person who could write code, and never learned networking, product, design, or talking to customers, you'll be afraid, because that one skill is being commoditized. Full-stack engineers who also do architecture and design may welcome the tools. Security engineers, he notes, never solved security even at the current pace. He recounts a security training whose best advice was "just go slow," illustrated by a hypothetical executive about to board a flight who receives a convincing, urgent text from the CEO and wires $10 million to an attacker. The executive was productive, but did the wrong thing.
He applies this to code. An insurance company could use AI to build payments or compete with DoorDash, but should it? Much of software engineering is decision-making: which database, whether to collect Social Security numbers at all. Writing code slows you down enough to notice that the loop is wrong or the data structure will make downstream reporting a nightmare. He isn't advocating a six-month waterfall, "maybe one more day before you go at it," and he fears people will skip that step because the model spits out code.
The host asks whether today's warnings are just the old guard repeating complaints once aimed at ReSharper and Stack Overflow. Hightower says both sides can be right. You don't need to code to make an impact, as no-code builders and Wix-based consultancies show. But if you want to be a software engineer in the full sense, where software is the interface between hardware and what people want, depth enables invention. He recalls someone describing process isolation without a VM using a technique before the kernel loads, possible only because that person knew the full boot sequence. His own thinking had stopped at virtualization, CPU isolation, and gVisor-style syscall interception. Knowing compilers or memory management is what lets someone imagine a new language. People whose work involves creation or reverse engineering, where "there is no spec," know the value of fundamentals. Jobs that can be done entirely with AI tools, he expects, will be commoditized.
He suggests dropping the fear-mongering and framing it differently: great artists know how to mix primary colors, so they don't need to buy 16 million tubes of paint. Learning the fundamentals is "just another skill that if you had it, you might just unlock some creativity," and he encourages people to learn it. If so much effort went into training the model, he says, you'd better be willing to train your own.
What do you think really made Kubernetes breakthrough?
The number one success criteria was Docker. Now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers. We were already off to a running start because you could just reuse the same Docker containers.
And I remember I get this email from Satya, the CEO of Microsoft, and I'm like, man, he wrote this nice email and I open a PDF and there's a zero get added to the equation. And so you're looking at this like, I didn't even know that they do that. We know that it happens, but the person that graduated from high school in 1999 that chose the A+ certification didn't know that was available. So, I was serious about going to Microsoft.
I'm not just like a GenAI hater. I just don't like the naive promotion and adoption of it. I think it should be way strategic. And since I think about GenAI as a tool versus the great human replacement, then I can use it in way more pragmatic ways. And when AI is involved, the one thing I just do before the thing kicks off in this meeting, do not say AI because
Kelsey Hightower is known as one of the most influential voices in the Kubernetes community, but you wouldn't guess from how his career started. At 19, he dropped out of college to be a DSL modem installer, became a self-taught developer, and still went on to become a Distinguished Engineer at Google. At age 43, he then retired at the very top of the industry. Today we cover Kelsey's unconventional path into tech and how he kept creating new opportunities for himself, often unknowingly. The inside story of the container wars, Puppet, Docker, Terraform, CoreOS and how Kubernetes eventually won. Going from a Google IC to executive level and how he rejected a Microsoft offer from Satya Nadella himself and still doubled his compensation. His grounded and pragmatic advice for software engineers worried about being commoditized by AI and so much more. If you're an engineer thinking about your long-term career trajectory, whether that's getting into a staff plus level, going independent, or even quietly planning to leave the industry, this episode is for you. This episode is longer than a normal episode, frankly, because I was so glued to my chair, mostly listening to Kelsey's stories and thinking.
This episode is presented by Antithesis. Verify your system's correctness without human review or traditional integration tests and avoid bugs or outages. Before we start, I'd like to mention our presenting sponsor, Antithesis, and maybe offer a little history lesson. Over the last two decades, software development has gone through a mindset shift from an imperative approach to a declarative one. Infrastructure is a perfect example. Think about how tools like Puppet and Ansible allow declaring how individual servers should be configured. Then came Terraform, the ability to declare the desired end state of your whole infrastructure, servers, networks, databases, and their relationships. And then with Kubernetes, we stop scripting container life cycles. Instead, we write manifests that say things like, I want the replicas of this application exposed behind a service with this much CPU and memory.
Once we didn't have to specify every little detail of our infrastructure anymore, deploying software became much faster. But then the bottleneck became how quickly we could test and verify the code to be deployed. Testing remained imperative. We had to write tests for every little detail. And now with LLMs, we're on the verge of a declarative shift in the way code is written as well. Just tell the model what you want and let it figure out the details, and it's going to make the verification bottleneck a million times worse.
Antithesis is a declarative testing tool that can keep up with your AI coding agents. You state the properties you want your software to have, and Antithesis figures out how to check them for you. Verify your code as fast as agents can write it and ship with Growlin confidence. Head to antithesis.com/pragmatic.
Kelsey, welcome to the podcast. It's so nice to see you in person.
Yeah, I'm actually happy to be here mainly because I kind of look at your stuff over the years. So, it's an honor to be here in Amsterdam as well.
How did you make your first dollar at a job?
Oh, my first dollar at a job, McDonald's, right? That counts. So, in high school, you get the job that's closest to you. So, it was in walking distance of my house. As soon as I turned legal age, 14, get a work permit. And I went there and it was one of those jobs where, you know, you go, you fill out the application the same day. You typically get your information or you're going to get hired the same day. When can you start? I'm like, right now. They go get a shirt for you in the back. What size do you wear? Men's large.
And the one thing I liked about that job is you're dealing with real people that are in a hurry. I guess one bad part about the job, you know, a lot of people don't respect people who have that job. So, they kind of look at you as just like this intermediary thing between them and what they want. But there's so many things that go into a restaurant like that. It's very efficient. You know, people have expectations. And I learned how to run the whole store. So, by the time I turned 15, I was an assistant manager. So, nights and weekends, you know, other managers would leave. They would give this 15-year-old the keys. And I knew how to do everything there, including close out the store, right? So, you have to count all the money. You have to fax this huge spreadsheet to corporate every night. And then my mom would pick me up. And so it was really good learning how to really be responsible even for adults at that age. So that's how I got my first dollar.
How did you get into tech? How did you get into programming?
In high school, since I moved from California to Atlanta. So right, you're going from one side of the country to another side of the country. And I missed maybe 3 to 6 months of school and in order to graduate on time, I had to take some extra classes. And so as someone who played sports, ran track, played football, played basketball, and it's like, you know, there's this computer programming, not, you know, computer club, Technology Student Association. There was a class component and then there was an after school component. And I was like, I don't know, man. This computer stuff, that's for the, you know, you know, I'm trying to be a cool kid. But the one thing I did, I really enjoyed it, right?
So I had a liking to AutoCAD. I even competed at the state level in AutoCAD. So we drove down and you compete. They give you a task and you sit in front of the computer. It was my first year doing it, but I really like the idea of like taking a specification, designing it, and I probably would have gotten first place if I would have got the product to work because you also have to print it out so that the judges can review your work. So I got second place even though I didn't.
This is the 3D modeling?
Yeah. AutoCAD, you know, just AutoCAD. Yeah. So part of the curriculum was, you know, you build bridges. We did this thing with chapter team where you know you have a coat of arms and you're kind of doing like a debate. You kind of learn all of these things related to business but CAD was one of the things I liked most. Also in that class one of the classmates taught me TI-BASIC.
Is that a version of BASIC?
Well, so TI-BASIC, so you know the graphing calculator, it's like a TI-86.
Oh yeah.
You can program it.
Oh.
Right. So in class they're like, hey, you know, it's not just a graphing calculator, you can actually program it. And I was like, what's that? And it's like, look, we can, you know, at that time you would create the snake game, right? So it's basically get a magazine, copy and paste the code and then you run it and now you're playing snake based on the code you wrote. And so you would toy around with this concept. So that was probably the first introduction to programming, was literally programming my TI-86 calculator.
And after high school did you go to college, or you considered college, right?
I considered college because in Georgia at the time and still today there's a thing called the HOPE program. So if you have a B average or above, you can go to any public school for free. And public schools in Georgia include Georgia Tech, Georgia State. These are pretty good universities.
They're really good.
And so I decided to go to one that was near me. The first two weeks I was like, this is too slow. This is not the pace that I want to move at. And also remember it's 1999 when I'm graduating. And so when you turn on the TV, people are standing in line for the next version of Windows. There's a lot of euphoria. AOL is starting to phase out and we're starting to touch on high-speed internet and I was like, yo, look at the pace this is moving, but also you're hearing the narratives. Bill Gates drops out of college. These people are not necessarily glorifying the degree anymore. It's all about the skill. Now, unfortunately for me, I didn't know anyone that was a programmer. I didn't know anyone that was like a system administrator because at that time all the systems, like Sun Microsystems or IBM mainframe, those are still the things that are in the enterprise. So when I looked at the job openings I'm seeing a bunch of skills that I don't even know how to acquire. And so instead of going to college, you know, I'm still doing fast food, delivering pizzas at this time at Pizza Hut.
And I remember going to a bookstore and they had the A+ certification guide and I looked and some of the job postings said, "Hey, you need to be A+ certified to take this support role or whatever it was." And I was like, "You know what? That doesn't require college. The book is only $35." And I remember buying the book and it is an official certification process. It looks like it was part of the job market. And so I remember buying that book and reading it cover to cover over and over again. And you're learning all the fundamentals, right? You're learning about, you know, how motherboards and how memory and all these things work and there's an OS component and then there's a little practice exam in the back. And so for someone like me, having that fast feedback loop of like you put the CD in, you take the exam and even though it was multiple choice, you kind of felt like if I got anything wrong, I would just go back to the book and make sure that I understood what was written there and then you go take the test again. And it had a little randomization to it so you couldn't just rely on absolute remembering everything. And I remember going to the facility to take the test and you know you're in that little room and they want to make sure you don't cheat. So there's a camera pointed at you and you're just going through it. And so the nice thing about those tests, there's no trick questions. Either you know it or you don't. And I think they maybe give you an hour, hour and a half. And I remember finishing that thing in like 10 minutes. And when I walk down, you know, you wait, the dialup goes and they calculate your score and say, "Hey, you passed." And then you walk out and you're like, A+ certified. And that was like the first time in my career that I felt like, oh, so if you put the effort in, you can gain the certificate.
And when I got that certificate, I remember there was like a job fair where, hey, anyone that has A+ and Network+ certification, you can be part of the contractors that were replacing people's dialup with DSL at the time. And so that's how I guess officially got into tech.
Is it fair to say that you saw that this could be the most efficient way to get into tech at the time?
I think I said I saw it as the only way.
Why was college never like telling you like, okay, that could be a way? Was it just you didn't see examples or
Yeah, I never saw the examples. I never saw the endgame. A lot of the stuff that they were teaching, the curriculum, it didn't make sense that you would pay all that money. You know, look, maybe it wasn't a good school. Maybe it was the wrong class that I took. There's so many factors that could have went into this. But when I looked at it, none of the people that at the time that I was looking up to, this is not the path that they seem to be taking. And so I had enough of school, right? If you're 18 at that time, you're like, look, that's enough of this. Because at the time, I kind of felt school was this, because it was so easy for me actually. You know, it's like it was easy to get straight A's. I didn't feel like there was a serious challenge. So it's like, hey, I want to go and do four more years of this? And I would later learn that, look, bachelor's is a lot of the similar that you go through, through K through 12, you kind of remember stuff, you listen to the lessons, but then master's, you challenge the material and of course if you make it to PhD, ideally you're adding something new to the field. And I never saw anyone that has made it that far. So I never put that in part of my calculus.
So, but just having that immediate feedback loop of like getting this A+ certification and feeling like, oh, I'm ready to participate in the actual economy, the ecosystem. So, to me, I was like, this seems like a better path. And it felt like a path that I would control.
Yeah. And then what was your first job that you could get with the certification? This was the CompTIA, right?
Yeah. So, at that time, BellSouth was, you know, the biggest telco probably in America. You know, they had been broken up by that time from the AT&T days. At that time the people who did phone lines, right, so those are the official BellSouth technicians, they drive the fancy trucks, they have all the equipment, and when they made the shift to high-speed internet, that means you had to actually touch the computer, and I think as a union they're like, look, we don't touch the computer, we don't even go into the house, we get to, you know, the demarc and we stop. And so they had contractors come in and the contractor's job was to come in, have to do a little bit of wiring. So, if you had to run some cable, you did that. Create Cat 5 cables, you did that.
Ooh, yeah. One of my first jobs was actually cabling. So, I still, I forgot the exact color code.
Orange, green, white, green, blue, white, blue, something like that.
It's burning in your head now.
And you know, so you did whatever it took. And the other thing you had to do was you have to open the computer. You have to make a decision, right? If they had a new enough computer, they can use a USB modem. Those were terrible. They always broke and you would always come out for a repair. But for a new install, if you really wanted to do a good job, you install a NIC, right? Cat 5 port on the back of their computer. And at that time, like we're talking Windows 98. And so usually, I don't know, 20% of the time as you're installing the drivers, the computer would crash. And now you have a whole another situation. You have to now troubleshoot getting this thing back online or back operational. But if everything went smoothly, they now had a network card and then you had an external modem that then you connected to, you know, the phone line and they had this high-speed internet connection and then you connected the network cable.
And I did that for about, let's say, a year and then I started doing the businesses. So you go into people's homes like you're going door to door and then when you go to a business you would hook up one computer but there's obviously eight computers there and only one of them has internet access. And I remember at the time, you know, the business owner, it could be like a small insurance company, and they would say like, "Hey, how do we get all the other ones online?" And the first time someone asked me that, I'm like, "I don't know, man. I don't." We put it on one computer.
Like pay grade, right?
Yeah. We make sure it works. But then I decided like, "Well, let me go learn."
And that's when I remember going to Office Depot, right? They sell computer equipment and things like this. And I went in the store and I asked him, I was like, "Oh, you can get one of these Linksys routers, right?" The infamous blue spaceship-looking Linksys router.
I still remember them.
And those things were probably like 50 bucks. And I remember just buying one and figuring out how to get multiple computers to use one connection. And so eventually I was like, look, I can't do it as part of the job because we have to do this and we have to leave, but here's my card. And then they will call you. And I remember one of the first installations that I did on my own, they wrote me a check and I was like, "Yep, you could just write it out to Kelsey Hightower." He was like, "No, we don't write checks to people. We write checks to companies." And I remember right there on the spot, I'm like, "Oh man, I need a company name." And I just made one up: Digital Gateways. And they wrote that on the check and I'm sitting there like, "So, how do you cash this?"
So, I went to the bank and they're like, "Sir, you have to get a business license. The business license, you could just do business as. You have to do this." So, I'm figuring out, now I'm 19 years old. Like, okay, I got to go get a business license. Have to figure all this stuff out. So, I do everything. I open a business account just so I can cash the check. But at that point, I'm like, "Oh, this pays more than this does."
And so, I got really good at doing those network installs. I really got good at troubleshooting, cuz sometimes someone gave them a USB modem, lightning comes, the USB modem is fried, and then you would swap them out for a network card. And eventually, I decided like, I can probably do my own business. And I decided to get some office space. So, I opened a small computer store right outside of Atlanta.
I would buy parts from the distributors. I was just like 19, 20 years old. And I wasn't buying enough to really qualify for an account. But luckily, one of the smaller computer stores I used to buy parts from gave a recommendation to the distributor. It's like, hey, this is our guy. He's just getting started. And they gave me an account. And I was able to buy motherboards and GPUs. And people would come over and like they bring their kid and they would have a parts list: I want a computer with all these things. And we would assemble machines and, you know, sell them, but also it was the headquarters for all the other service calls.
So I did that for like three, four years, you know, that ended up evolving into, at the same time, or now we're talking like 2000, 2001, a lot of the music studios were moving from analog gear and the large mixing boards to Pro Tools, right? The little rack mount unit. And they all needed Macs. They all need these conversions. So I added that to my abilities. And then I had a small setup in the store and artists and musicians would come in and say, "Hey, we want exactly that in our studio," and I would get the order and I added it to the list of things I could do.
I mean, at this point, you now have a small business. Sounds like it's going well. And suddenly you take a job, an employee job at Google, when I look back. How did that come up?
Up to now, it's almost like, this is like, you know, the story oftentimes will continue. You become an entrepreneur, you grow your business, you, you know, you just take it from there. Even during that time as that store owner, I managed a comedian and we went on the road. And
So you were helping out your, your managing
I had a buddy from high school. He was a comedian. Turns out he was actually really good at it. I went to go see him at a club. He's like, "Hey, I need a manager." I was like, "Are you even funny? Like, I know you from high school, but I don't remember you being funny enough that I would pay to see you tell jokes." And he was like, "I have a show tonight." And so I gave him a ride on the north side of town. And the interesting part, it was a predominantly black audience in Atlanta. Okay, makes sense. And he did these jokes and they laughed, and it was one of these comedy clubs where if you're not funny, you have like 3 seconds and they would just boo you off the stage and that's the end of it. And he held it. I was like, "Wow, you survived that. That's incredible. And you were pretty funny."
And then we drove about an hour north and the audience is predominantly white. And so on the drive there, I'm like, there is no way you can do those jokes in this room. I got to see how this is going to go. And at the time, they're only paying the comedians like $50. So, you don't make a lot in the early days. And he totally pivoted the set and he held that audience, too. And I was like, "All right, I can be your manager. I know business. I know kind of logistics. I know how to, you know, make a plan together."
And I did that for a number of years, and you know, he won some televised competitions. We went on the road with bands like Earth, Wind & Fire and some large comedians from Kings of Comedy and Queens of Comedy, and I actually picked up some IT work with the company behind it called Latham Entertainment. They had been doing movies at this time. And so now I'm like, you know, doing it for them, and I got the small business, I got the comedian. And so, look, I was able to save a lot of money, but man, luckily I was 20 years old, cuz I had all this energy, but I was working quite a bit.
Eventually, you settle down, you get a family, and you do the math, and it turns out people kind of over-glorify entrepreneurship. I think a lot of people believe there is tremendous upside, right? The type of entrepreneurship we talk about with software companies, the upside is crazy. But when you're doing like selling parts or a service business, unless you plan to open lots of stores and, you know, grow a larger employee base, it's not the same growth trajectory as software companies. And so I kind of did the math after four years of doing that. I said, look, I want to settle down. And if you've been an entrepreneur before, you know employees get paid first, the owner gets paid last. And there are months where you get paid less or you don't get paid at all and now you're kind of drawing from savings, because it's not their problem.
But I did well overall, like I did very successful. But I remember, it's like, you know what, I think I'm ready. And so I looked around and Google had data centers nearby, and I felt like I had a great combination of skills. I understood, you know, the rack-and-stack part of the world. I understood the physical part of the networking stack. I understood everything from Linux to Windows. I had an entrepreneur mindset. I didn't think there was nothing I could not do.
And I remember going to that interview and they were hiring like data center technicians. What it paid? And in my mind, I'm like, you only have to work for 8 hours. You don't have to issue any invoices and you get paid every 2 weeks no matter what. No inventory. This is crazy. No way they're doing this. And so I go and I remember doing that interview and I didn't know Linux that well. And luckily for me, I knew FreeBSD well. And as I'm answering the questions, I'm like, look, I am not an expert on Linux, not the way Google was asking these questions. And it's like three people on the other side of this table just rapid-firing. And I remember I was like, I know FreeBSD. I swear I got lucky. One of the interviewers, I think they had a FreeBSD tattoo on their leg, the little Beastie logo. And I saw this logo. I'm like, "Oh, I'm saved." And so we started going down the FreeBSD questions and I pass and I get this job in this data center.
And it was good because it's like, hey, I'm working with my colleagues, but it felt a bit slow because you only get one job. You come in, you do this thing. And I got really good at it, cuz to me, I kind of saw it as like a bit of a competition. Who is the best data center tech here? What do their metrics look like? How do I exceed their metrics? I want to learn how to do every particular thing in this data center, because previously as a business owner, the more skills you have, the more money you can make. And then I just started switching jobs every 3 to 6 months because I just wanted to explore everything to just amass my abilities. And doing the math, I think every jump was like a 25% pay raise. I mean, coming from a small base, it didn't feel big at the time, but after a few jumps, my salary doubled.
Puppet was a bit of an inflection point in your career.
You know what, and I would say the biggest inflection point in my career, there were two of them before I get into Puppet Labs.
Okay.
So you come from Google, you see this huge operation. There's hundreds of thousands of servers. The cables are perfect. They're immaculate.
Oh, by the way, can you help us imagine what a data center back then looked like at Google, and what did you do as part of the job?
Yeah. So in 2004, maybe 2005, that data center is like a warehouse. I mean, it's huge. So think about a place where, and I'm pretty sure they always exaggerated the numbers, so the exaggerated number to think about is like, think about 200,000 servers in one place. Everything is immaculate. So a lot of people have worked in data centers and it's a mess. Wires are all over the place. You know, you're ad hoc adding and removing servers. But Google was systematic. Those machines came off a truck. They were wrapped perfectly. When you wheeled them into their spot, you connected the network cables, they would PXE boot. They would burn in.
So part of the job was, you know, you walked around with a crash cart. So depending on what your abilities were, some people had pretty straightforward jobs. You had a crash cart, had all your tools on it, and you would walk around and you would find machines that were in need of repair. So if you have 200,000 machines, it's okay if like 300 of them are broken, right? The system can route around that. But if they were broken, they needed to be repaired. So you would go to rack, server 7, you will pull it out and you will look at it and, oh yeah, the SATA card is on fire. Like, it's literally burned. It's burning. It was burnt. And so it's like, I can diagnose this one with my eyes. It needs to replace the SATA card.
But the thing is, you would go into the system and you would say, this SATA controller needs to be replaced. So you would be making your prediction, and then you would replace it and it would go through its burn-in process. It would join back to the fleet, and then the way you were measured was how good were your predictions.
Oh,
Right. So you said it's a SATA controller, and to me, moving fast, you look as if, I think that SATA controller, I'm not going to waste any more time on this one because I'm trying to get my numbers up. So I just swap the SATA controller. You bring in all the cables, you slide it back, it goes through its process, and you move on to the next machine
before you get the feedback.
Yeah. And fun fact, there's a guy named Tim Hockin, he's very popular in the Kubernetes community. He's like the network lead. So back then, Tim Hockin is working at the other side of Google, like the bigger side of Google. And one of his first projects was, he built a little tool that you would put on the motherboard. It had about nine lights on it, and the lights would flash back and forth, and it would tell you what DIMM slots were potentially bad. The technicians that were fast, they didn't waste time running a program to do an extensive test of the memory. You learned how to use this little device and you would walk up to the motherboard.
So the thing I would do is, before I do anything, put this on the motherboard, get the reading, and I learned to trust it over time: DIMM one, DIMM two are bad. And the goal would be, all right, I have all this memory on my cart. You swap those two DIMM slots. Make sure that that's done first. Reboot the machine. And you might be like, I think that's the only thing wrong. And again, you would put in the system, I believe I only need to do DIMM one and two. And then the way you were measured is how long before that machine gets kicked back to repair. So if it doesn't get kicked back within, I don't know, 30, 60 days, you did a good job. If you didn't, your scores would be low.
So for the technicians that were just reckless, right, they wouldn't even try, and you're just swapping the wrong part. You're swapping the wrong hard drive. And so your stuff is always coming back for repair. You were not efficient. So I got to the point where I can maintain high 90s but also repair, let's call it, three times more machines than other people. So lots of machines, rate of return. And then I learned how to do the network switches. And then there's power audits where you're lifting up tiles from the floor and you're making sure that everything looks good. You got to be careful not to touch them because you could die. And so you learn everything about a data center, like the service loop. You know, you're running all this Cat 5 cable and it has to have a perfect service loop. Fiber runs on a different part of the rack, right? So you don't ever mix these things. So as a person still in my early 20s, I'm thinking this is how all data centers look. It's not the case.
But it's crazy because, just as I think back, you know, I was a manager before as well and of course a software engineer for a long time, but the way your performance was continuously measured and fed back to you, and you were evaluated based on it, it feels way more strict, should I say, than, you know, folks who work as software engineers, including at Google. I mean, the frequency, the expectations.
The reason why I didn't feel as bad as some of the metrics people are using now is because I felt like I can control the outcome. It didn't feel like it was just a metric that didn't do anything. If I felt like my score was taking a hit, I was like, you know what, I am being a bit sloppy in how I'm diagnosing these machines. And I remember one time where I almost had my score dip below 90. I started writing additional shell scripts, started combining different functions together. It's like, you know what? Can't be moving this fast. There's a way for me to diagnose multiple things on the machine at one time. And so, I would diagnose the SATA array, all the hard drives, while I'm doing the memory component. And then when it rebooted, I would just run the script one more time as I'm putting my cart back together to catch that one more thing. And once I started doing that, I can move as fast as I wanted and the scores are right.
So to me, when the scores actually match the things that you're doing, then it's healthy feedback. And again, no one really talked about it unless you needed to talk about it. And so I kind of leveraged it for a personal thing. I pulled it up in the morning. I kind of looked at my performance metrics and in many ways I calibrated my strategy based on this detailed feedback that I was getting. So, I think I appreciated that level, that granularity back then, because it felt like it was something that was helpful for me, not just my manager.
And you said you had two kind of big inflection points. One of them was
Google was a big one. Like, that was definitely one, of course, my entrepreneurship. And when I got to web hosting, I went to a company called Peer 1. They're a spin-off of Rackspace and they were all about fully automated self-hosting, right? So,
This was back in like what, 2005, '06, '07?
Yeah, this is like, yeah, 2005, 2006. Oh, they were already about, that was their thing. Their tagline was "latency kills." And so back then you would go online. A lot of the customer base was like people hosting their own game servers, right? So if you wanted to play a game, one thing you could do back then is host a game server, but you needed a game server that multiple people could hit. And so you would go along to ServerBeach. So this is a spin-out of Rackspace. So Rackspace is more like, you know, we'll buy a server and once
you get it and it's a lot of manual steps. Rackspace was more of a let's automate everything. So the machine would PXE, some PHP scripts would run. If you ordered a RAID setup, then we would configure the RAID while the machine was net booted and then we would put you on the VLAN that you belonged on. We would install the right operating system based on what you've ordered and we just took a form. We just went through all of these steps and then when we were done, and it took maybe about an hour. When we were done, you had an IP address, login credentials, and if you wanted email and plus, you know, website management, all these things, you got it. And when you were done, you gave it back and then we put it back into the pool ready for the next customer.
And so when I saw how we were doing that, I was like, yo, this is okay. You can automate things end to end. And the other thing that was important here, we were doing things like updating the firmware for RAID controllers because once you PXE a server, now you're in memory and you have access to all the hardware. You haven't committed to an operating system yet, but you have enough to do whatever you want. So if you want to configure the RAID controller, and back then there weren't like clean APIs. We were literally running curl scripts and command line utilities trying to get this machine into the right shape and then when we were done we would put it into the fleet. So at that early age I'm like, oh, there is nothing you cannot do.
When we had to automate the Windows 2000 servers back then we would just build tools that would literally screen scrape, log in to Active Directory and then we would screen scrape mouse movements so that way we can patch software on those Windows servers. So the concept of like I need a specific tool to do it, it's like no. Back then it was like you do whatever is necessary because these people are only paying like $99 a month. I don't have time for you to spend a whole week when they call. Random customer. You don't know their setup. You don't know their infrastructure. You have minutes to get them back online. So everybody learned to move quickly. No complaining. When you get that ticket it's on you to figure it out. Maybe you lean to your teammates for help. So I kind of learned how to move fast.
But the inflection point came from, I started that job in tech support. So people would call: MySQL isn't working, DNS isn't working. They can't even describe what isn't working sometimes. And so I realized that we were all in the phone queue. When the phone would ring, it would just round robin between everyone. Then if you couldn't solve it fast enough or you couldn't solve it at all, you created a ticket. And the ticket sat there and then ideally you get to it later, but the ticket queue was just getting long. And when a shift change happened, we just had all these tickets piling up. And of course, customers are now mad, 3 days, no response.
So one day I said, look, I'm just not going to log in the queue. I'm just going to resolve the tickets. And I'm back in that entrepreneurial mindset. I'm building little scripts to take a ticket. I see the issue. This is a MySQL issue. We need to vacuum the database. This is easy. Run that one. Ticket closed. Hey sir, everything is good to go. Please try again. Close. This is not even an issue. Close. Plus, oh, we got to upgrade PHP. Close. And then the ticket queue is zero. It's just empty all day.
And so someone's like, "Hey, Kelsey's not logged into the phone queue. He's not even on the phone. He's not taking calls." My manager pulls me in the office like, "Hey, Kelsey, why are you not on the phone queue?" I said, "Because we don't all need to be on the phone queue. We can just have one person making the ticket queue stay at zero." And then I would tell my colleagues, "Hey, look, if you can't figure it out fast, just open a ticket super quickly. I will take care of it." So some people specialize in Windows. They got a Linux call. They didn't know what to do. I said, "Don't worry about it. Just put it in the queue quickly. Move on to the next call. I got you." And so I became super efficient. So I explained this process to the manager and he thought about it. His name is Mike. And Mike was like, "Yeah, I like that. We're going to change it. We're going to have a couple people stay out of the queue, but the promise is that queue has to stay zero."
And it's the first time I learned the difference between activities and impact. Activity is you being an all-star answering a bunch of calls, but the ticket queue for the team is still high. You making this jump and saying, "Look, maybe I stay out of the queue and my promise to my colleagues is impact." The ticket queue is zero. So on Wednesday when we do the team turnover, we're handing off an empty queue. Now, of course, I didn't teach them that because I didn't think about it. And when we would come in, the queue would be high again. And it's like, yo, whoa, this is not, you guys keep handing off a bunch of burden to our team. And of course, we would clean it up, but then the management team was like, yo, everybody's going to do it. And then I was like, man, you can change the process.
So that was like the huge inflection point. I think the next one was more about being a mature person. Stop job hopping so much because again, after maybe a few promotions and some impact, I'm now off to the next thing, and I didn't have to necessarily live a long time with some of those decisions or, you know, stay long enough to really impact more of the culture.
So when I got to financial services, it was the first time I got restrictions put on me, right? Because in these companies, everything is about moving fast. Google, we got to move fast. We have to compete with the big guys. We're going to do it with cheap hardware, smart people, any means necessary. Web hosting, we're not charging a lot. Margins are thin. You got to move quick. Financial services like, no, we making money over here.
You joined the financial services company.
Yeah. So that was the first time my salary doubled. So I went on my lunch break for a job interview.
That must have felt awesome.
Oh my god. I don't remember. My manager was so upset. I went on a lunch break.
The previous manager, the one you
I met Peer 1. He was a good friend of mine named Joe Rodriguez. He's the first person that made me a software developer. So, I showed him some ideas I had on like modernizing our virtualization, optimizing our PXE boot process. And he was like, "Man, you have a lot of good ideas." So, in the interview, he's like, "Show me what you built." And I'm so excited. I'm still that entrepreneur thinking, and I got the job. So now I'm a software engineer working on this automation stack.
And I remember seeing jobs that I used to be afraid of 5 years ago. The same jobs that made me want to just go get an A+ certification and open a computer store instead of even trying. And so I looked at those job descriptions again and it was like you need Linux. So like I got that. Even at that job I got Red Hat certified, right? So I was like, I got all the qualifications now.
And I remember the job. I didn't know how much it paid before I went, but you had to wear a shirt and tie. It was the first time in my whole life I had to wear a shirt and tie.
Oh, so you had to go and buy one.
Yeah. And I remember my home teacher, she hemmed my pants for me, right? I was like, "Hey, I got a job interview. I know. Does this look right?" I'm driving up on my lunch break and I get there and, you know, this is enterprise. I'm like, "Wow, this is the big league." I didn't even think Google was big league. I thought financial services big league. Obviously, if you have to wear a tie to do your job, you must be doing something so serious that you need a shirt and tie to do it. And so, I get to the interview and I'm sweating like, "What? They're going to ask me stuff that I've probably never heard of or seen before."
And they started asking Linux questions. I'm like, "No, you got to use this flag. That's not the right flag." No, look it up. That is not the right flag. You can't do that with grep. Nope. You got to pipe it this way. The PS table doesn't show you that. Not on that version of Linux. Not with that curl. And I'm like, it's too easy. And part of me is like, either I'm really good or something else. And so I'm thinking like, yo, that was, maybe there's another round.
And so I'm driving home and I'm loosening up the tie and I'm calling my wife like, "Hey, I think I did a really great job." Like, you know, I think, and this has been a pattern, you know, I get good at something and I find a better job. I got good at something, I find a better job. But now it's like this is like a career.
And so I'm talking to my wife and I was like, "Hey, I got to call you back." And it's the recruiter calling. Hey, you have a second? It's like 100%. They want to make you an offer. I'm like, you know, make you an offer is different. Like it's usually this is how much the job pays.
That's how it was until then. How much were you making?
45K.
So you were making 45K at that point.
$90,000.
Yeah.
I almost drove off the road. Like double. This is insane. I'm thinking 90 and you're doing the, you know, retirement calculations. We can buy a house. You're thinking of all of these things that you could do. And so I'm on the way back to the job and I remember, and they were like, then when can you start? It's like Thursday. I'm telling them we can start Monday. Like two week notice. I'm like I don't know man. Like it's devil
for this.
I have to. So, I remember I called Joe, and Joe's based in San Antonio where the headquarters of Rackspace was. I was like, "Hey, Joe, man. Hey, man. I got to quit." He was like, "What, man? You just, you know what I'm saying? You're doing so well on the team. How you going to quit on me? I made a bet on you." You know? I said, "Hey, Joe, calm down, bro. They pay 90K." He's like, "What? Are they hiring?" [laughter]
And so, I got to that company, Total Systems, TSYS. And it did feel a bit slow. They had runbooks, change windows, everything was regulated. This is a financial institution. We're processing credit card payments. We're doing work for the government. We're doing all of these things. And I remember the team was just, everybody just moving at this pace.
But isn't it crazy when you get in there? I had something similar when I moved into London and my salary more than doubled going from Edinburgh, where you have all these expectations because you're being paid so much more. The tie. Same thing for me for the tie, and then it's a bit disappointing, and I think did you not feel a little bit like am I missing something, like this should be, you know, higher, this should be, you know, the interview was okay, maybe not as hard, but it was something
I mean, I did, because I was naive, because I didn't understand the consequences of a mistake. And so yeah, I was like, oh, these people are just moving slow, and I looked at what they were doing and everything was like a risk. If you made a mistake, you remember you only get seven milliseconds, 7 seconds to give Visa a response. If you don't, then it's a default decline. Now everybody's losing money. And so the cost of getting a good change, it was just worth waiting.
So I didn't know that in the beginning. So I'm just like, oh, everybody's moving a little too slow. I was seeing how they were doing deployments. I'm watching how they're provisioning servers. I'm like, you know, you can automate this whole thing because I've done it multiple times. And then I learned to be a little bit patient. All right, hold on. I learned how to deal with the nos. I learned how to deal with the executives. I learned how to talk not just to engineers but with the senior leaders and get their trust. And I won't talk about everything that I did there, but I was there for about three years. So the job hopping stops.
And I remember there was a task where we were using Apache and some Java plugin to talk to our JBoss instances. And so they were using a lot of memory per connection on the load balancer. And so we got a new, we're moving off the mainframe. We're moving into this new, you know, Java world and the servers just kept falling over. We couldn't handle all the transactions and I'm just watching the team go into these change windows and then we fail. We would fail. We would fail. And the CTO at the time was like, you know, this is costing us money. We're getting chargebacks. We're having to pay out penalties. And I was like, I had a dev environment where I was using Nginx. I got rid of all this Java stuff. Like it's just HTTP. You don't need a Java connector thing. I'm reading the spec. You don't need it. And my memory usage is a fraction
by getting rid of it.
Yeah, you don't need it. And Nginx had a better threading model, all these things. And I was like, I got a perfect config. I've ported the Apache config to this one. I even handle all of our redirects, all of our routes, all the legacy crust. This thing will work. It's like, I don't know, Kel, this is not certified stuff. It's like, no, no, no. I got it from Red Hat. RPM install Nginx. It's in Red Hat. And they're like, okay, it's in Red Hat. Okay, sure. Gave me one chance.
Yeah.
And about after a week, they gave me a shot and they let me be in the change window. And
What is the change window?
So a change window typically in a financial institution is like we can start changes at midnight and we have to be finished by 6:00 a.m. and you have to notify every customer. We're going to change something in the environment. Things may get a little weird. We would like your permission. And if it's something that can be detrimental to your business, now's your time to speak up. So once we got permission, we had a window. That's the only window you got. And if you can't finish it on time, you got to shut it down and get us back to where we were.
Yeah. Roll back.
Roll back. Roll forward, or if you could, roll back. And I remember I put Nginx in place. So at this point, it's all you. No one at this company has Nginx experience. Also, most people don't want to do this. They're like, "Hey, we think this is a bad idea." And so now it's your reputation on the line. There is no hiding behind the team. It's almost like everyone's sitting back like, you got it. Now luckily the leadership was like, we are supporting you and we're putting our careers on the line as well.
I mean, they probably would have been safe, but after about 2 hours I got everything working and we just watched the memory pressure just drop. So if we were at 90% utilization, I don't know, let's call it 32 gigs, and we're just blowing stack every time, everything's crashing, this thing is hovering at like two gigs of RAM. And everyone's like, yo, are we getting peak load? Is this legit? And we're all just sitting there like, not sure. And the way we used to test the platform is someone would drive to the gas station, get a credit card, buy some gas, and we wanted to see the transaction land in the Oracle database.
Yeah, you could track it.
So we can track the whole thing. We're like, yo, it works. And so the thing is, you can't really be comfortable until about 10:00 the next day.
So now it's 3:00 a.m. We all go home. I don't go to sleep. I'm like, "Yo, we've changed a big part of the infrastructure in production. Let's see." So 10:00 goes by and we're looking around and there's no Nagios alerts going off. I'm like, man, we might be good. You know me, I got top going. I'm looking like memory usage is holding steady.
Yeah, top is, for those who don't know, it shows the CPU, PS tree in a loop, right? So just like if you ran the ps aux command, you see all the processes running
processes.
And so you just run top and then it's basically that every second being refreshed, or whatever timer you give it.
But one of the bets I made was if I get this to work, you have to buy pizza for the whole floor for a week. Every day we get a new order.
And because he promised, like, Kelsey, if you
get this to work, I'll buy you a steak dinner. And at the time, I still am, I was vegetarian. I don't eat steak. But here's what I would like instead: you got to buy pizza for the whole floor. Now, I don't know how strategic I was thinking, but I figured that I could score some points with the whole team from turning it into an 'I succeeded' to a 'we succeeded'
and then people eating together was one good way of doing it. And I remember after it worked, he was like, "What's the order?" And I just ordered pizza from one place and we just put it in the middle. Everyone was like, "What's the pizza for?" Right? Cuz usually you only get this for special events, like for the thing we did. And the next day some more pizza and some more pizza.
And so when I got that accomplishment, I was like, "Okay, this is what maturity looks like. It's not about just having the right solution. You literally have to get consensus, buy-in, make sure, and then your reputation's on the line." And I got that change in. It just kind of changed the way I thought about what success looked like. And so that was the inflection point. And then of course I brought Puppet into that organization, repeated a similar process of automating everything with this tool.
Kelsey just talked about what CI/CD looked like in the 2010s and how difficult deploying to prod was, which leads us nicely to our episode sponsor, Buildkite. This type of manual deploy era was the time Buildkite was created. Engineering orgs were trying to figure out how to ship more frequently, more than once a week, and those running into walls first turned to Buildkite. Shopify started running on Buildkite in 2015. Airbnb, Canva, Uber, and OpenAI have all been using Buildkite for 7 to 12 years. Throughout the 2010s, the hardest CI problems in the industry for global software leaders were getting solved on Buildkite pipelines. That's exactly what makes Buildkite the CI platform most in tune with what's happening right now. GenAI throughput is pushing everyone's build queue to its limits. Coding agents are pushing commits at 5, 10, 50 times the volume that your defaults were built for. The same architecture that absorbed Shopify's traffic at scale five to 10 years ago now absorbs a shared push from Anthropic, Cursor, Meta, Mistral, Cohere, vLLM, xAI, Lambda and more, who are all Buildkite customers. Where other CIs are cracking under the weight, Buildkite is running about 1.4 billion job minutes a week. Agents run on your infrastructure or on Buildkite's. Every artifact and log is captured, and when something fails, you see the why immediately. If you're hitting ceilings on your existing CI or downtime is becoming more problematic for your delivery, head to buildkite.com/pragmatic: 30-day all access trial, no credit card, and an actual human engineer on standby. His name is Ola, and he is very helpful.
I'd also like to mention our season sponsor, Sentry. AI agents don't have a feedback loop from production observability logs, but should they have it? I mean, we probably don't want to build a system where after each production error, AI agents automatically push out a fix into production without supervision. That would be a recipe for disaster.
But what if we did something more modest? When a production error fires, an agent investigates this with context from Sentry. Sentry already has all the context in the error, after all. Sentry MCP is one way to plug in Sentry to agents that support the Model Context Protocol. Claude Code, Cursor, Codex, VS Code, Copilot, they all do.
After you hook up the MCP server, you can do some very useful things. For example, you could do this when an already resolved Sentry issue resurfaces. You can kick off a Cursor agent to investigate the regression, read the relevant code, and open a PR with a suggested fix. There's a little work involved to get all of this going. You need to connect Sentry to your code repository, add Sentry MCP to Cursor, define the instructions for Cursor's agent to investigate, configure the trigger that launches the automation, and test that it all works. But once you have it up and running, you can get regressions fixed faster whilst you're reviewing every and all fixes. This feels like a sensible and helpful use of AI and MCP and Sentry to me. Check out Sentry at sentry.io/pragmatic and start monitoring and fixing regressions today.
And with this, let's get back to Kelsey and when he introduced Puppet into his organization and automated DevOps processes.
And I remember we had someone from Puppet come by
And just for, again, those who don't know Puppet, Puppet is this
configuration management tool. So Puppet is, we went from lots of shell scripts doing random things and pseudo-automation, but most of the time you just did a manual thing. You had a ticket come in, someone wanted a new SSH key on a server, someone wanted something installed, you did it manually, copied and pasted the output in the ticket and you closed it, and you did that every day over and over again. And that's where the saying that I say sometimes: some people have 20 years of one year experience. Some people in IT have been doing that 20 years in a row. They never made the leap to automation or learning new tools or skills.
And so I saw, man, I can bring in a tool, and I brought the tool called Puppet at the time, 0.24.8, and this is right before DevOps becomes a word. But it's like, all right, I'm starting to learn how to write code, like for the Java production stuff. I'm writing Puppet manifests using the Puppet DSL. Sometimes you have to write a little bit of Ruby to build a new resource type, and I'm contributing, and also I get introduced to open source, like, hey, there's something there that doesn't work, I'm going to contribute it upstream. So I'm doing all this behind the scenes, getting ready for act two.
And I remember my manager went to a conference and he came back and like, "Hey, Kelsey, I finally know what you're doing." I was like, "What's that?" He was like, "You're doing DevOps." And part of me was like upset. How do you get to name what I'm doing?
And so everyone's talking about DevOps and he's like, I met a guy named James Turnbull. And James is an Australian dude. He worked at Puppet Labs and he wrote the first Puppet book that I bought and read. And so I'm like, James Turnbull? That name sounds familiar. Oh, I'm looking at my desk, James Turnbull. So he's one of the co-authors of this book. And it turns out James wrote a lot of tech books back then. And he's coming to the office. And my manager wanted him to check out how we were using Puppet. I was like, "Okay, so this is a world-class expert."
And by that time, I had Puppet hidden behind Jira tickets. And so you could just open a Jira ticket. I had RPMs for everything. You picked the RPM you wanted from a drop-down. You picked your environment from a drop-down. And magic happened and you got results. Most people didn't even know Puppet existed. Most people never touched it. They just knew they can get anything on demand.
You could go to Jira and it does it. So for example, you could provision a new machine or
Whole environment. I want a database. I want an IBM MQ Series message server. I want these three apps and I need this firewall setup. No problem. Open the right Jira tickets. Get them approved. When they're approved, we had this little, I called it Mr. Receti. One of my buddies gave me a name from some video game. Once it got approved, Mr. Receti would own the ticket and we would just extract the custom fields and we would just call the right Puppet manifest and we would take the output and update the ticket. And so PMs, developers, they just got what they want damn near instantaneously. So we built all the systems they didn't know. Nights and weekends I'm contributing to Puppet because it wasn't allowed at work at the time. This is 2007, 2008.
They probably didn't even really understand what open source was anyway.
So they was like, hey, you got to do it on your own time. So I would just do it nights and weekends. So James shows up and I remember we all get in the elevator. We're going to the third floor. My manager's introducing James to me, to our team, like, "Hey, this is this person. This is this person. This is Kelsey." And James turns to me, he's like, "Kelsey Hightower? Oh, we know you. We love your contributions." And my manager is just like, "What contributions? We're not doing contributions."
And James is shaking my hand and he's just talking to me like we're friends, because the thing is, I have been working with the Puppet Labs team through open source, contributing to Puppet.
And so we get up there and my manager is like, hey, let's show him our setup. I was like, all right, James, we have a lot of Puppet manifests over here. I'm using external node classifiers. I'm setting config from data. I read Mark Burgess's promise theory and we got it all dialed in. But from an interface perspective, I just didn't want everyone to have to learn Puppet. So we hooked it up to Jira. I'm showing him this entire workflow and he's just sitting there like, "Yeah, I have no recommendations at all. Like, this is amazing."
And here's the thing that was kind of the game changer. So he came all the way to Atlanta. We're 45 minutes from the airport. He spent time with our team and it's the end of the day. And then he's on the way out and he's about to call a cab to go to the airport. I'm like, "No, you're in my hometown. I cannot let you go to the airport. I will give you a ride to the airport." So I get in my car and he's in the front seat. My colleague's in the back seat. He takes off his tie and rolls up his sleeves. I can see all the tattoos. I'm like, "Oh, so you're like a legit tech." Like, "Yeah, I don't do all of this. I just did this for you all. This is your dress code." I was like, "Oh, James is cool." He was like, "Man, yeah, you guys are doing great work." And now I'm talking to the real James Turnbull. And he invited me to give a talk at PuppetConf, and I went to go give that talk. It was my first real conference talk, and I talked about some low-level kind of Puppet integration stuff and I had a little comedy in there. It was like the first time I felt like I could just be myself on stage.
Before I came home, Luke Kanies, the founder of Puppet, we sat down in a little coffee shop in Portland, Oregon, and he was like, would you like to work here? I was like, yes, of course, 100%. And of course there was a nice salary increase and I could work remote. I didn't have to leave Atlanta immediately.
And I remember coming back, and my manager really appreciated all the progress I was making over the years. I kind of made a name for myself. And I remember he gave me a nice little raise just out of the blue. And he slid this envelope to me and I opened it. It was a nice number in there. I was like, "Oh, that's great." I was like, "But here's the thing. I'm resigning. I'm going to go work for Puppet." And then he said something that was dope, because we had a kind of difficult relationship depending on how it went. But overall, he helped me mature. That's one thing I will always say. He helped me really be mature. And he was like, "I'm surprised you stayed here this long," right? As like this ultimate compliment that you stayed here beyond just getting better or making an impact. You literally changed this culture.
And the full circle moment happened like 7 years later when Kubernetes came out. I remember coming back to that office and they had already been running CD in production, Kubernetes in production, and even some of the old tooling I had was still running. And to me that was a really important feedback that sometimes if you make an impact, lasting impact is way different, that even when I wasn't there, they still had the culture to say there's new technology and we know how to bring it into the stack.
So at this point containers were starting to start, right? How did you first come across containers, and very early on you realized this is going to be big and important?
I didn't.
Oh, you didn't?
No, because I'm at Puppet Labs. You know, before that I was contributing to core Python stuff like virtualenv, pip, on the team that was trying to integrate some of the Python. Python had a package management problem even back then. And so I'm all in on Python, and then when I'm learning Puppet, I'm all in on Ruby at that time. I'm thinking DevOps plus configuration management is the be-all and end-all for everything. And really at that time, we're talking 2011, 2012, in my mind the competition was only between Puppet, Chef and Ansible. That was it.
And so I'm all in and I'm working at Puppet Labs. And so in some ways I felt like we were dictating the future. We were going to companies and getting them to understand configuration management. We were talking about the benefits of speeding everything up and doing all these compliance. We were moving people from SharePoint to executable code to implement these things. So we thought we were just getting started. There was no world where we didn't think that we needed 10 more years of that, right? We even got money from VMware to integrate Puppet into that. Chef was being integrated into AWS at that time. So this was like the thing that we were finally getting there. We just got to teach system administrators to become engineers via DevOps and start embracing these tools. And then the first thing that I saw come out was Golang.
And I remember sitting at my desk and I was like, you know, at that time we were hitting performance issues with Ruby, right? The global interpreter lock. You can't do more than one thing at a time, not easily.
And so we moved to things like JRuby. We even did Clojure for some part of the Puppet stack. And I remember we were thinking about, should we start doing stuff in C++? Should we start doing stuff in C? But the problem is contributors. What would we do with all the Ruby contributors, right? We were relying on things like Ruby gems and it was going to be a hard sell.
And I remember downloading Go for the first time, and the thing that sold me on it, I remember one of the prototypes that I built was Facter. So Facter is one of the agents in Puppet that gives you facts about a machine. So this is Red Hat, this kernel version, this is what the users look like, and it would give that to Puppet so that your configuration code had context on what to do. And that was written in Ruby, and sometimes it would run slow because you're reading all these files serially. And so I remember I was like, "All right, let's try this Golang out." And I remember I wrote the code on my Mac, compiled it, SCP'd it to a Linux box. I was like, "Oh, this is crazy." And I remember running all the facts in parallel, and it came back in a fraction of the time. I was like, "Yo, we got to use Golang for stuff."
And I remember the team was like, "No, it doesn't work for Solaris or AIX," because Puppet was now moving into the enterprise. I was like, that's the criteria? No way. But I got it, right? I've gone through this maturity thing before.
And then Terraform comes out. But before Docker, Terraform is starting to challenge things a little bit, right? And it's starting to use things like Go, at least that was on my radar. So Terraform is like, who cares about the node? It's all about the APIs. And all of us come from the server world and everything's about an agent on a node, versus the cloud starting to take over at this point, right?
So Terraform was built with the cloud in mind, right? You could configure your cloud environment, infrastructure as code.
Yeah. And we were trying to do this from a node. We were trying to teach Puppet how to talk through, by having this indirection where you have to go through a node to configure a server in the cloud. It was so weird. But then Terraform, Mitchell Hashimoto, right? He was a big part of the DevOps movement early too. Then Vagrant was written in Ruby. So he was part of that same ecosystem. And then he splits
off and there's this going. I'm like, look at that. They're using Go for this.
And I remember when Docker came out, Puppet at that time was the one pushing innovation. We're asking people to think a little different, get out of their comfort zone. And then Docker comes out and most of the office was dismissing this as like, nope, this is a fad. They don't even have config management in this thing. They don't really understand enterprise. This is just kind of not like a toy. They were not being disrespectful, but no one saw this as a challenger to what we were doing, and I looked at it and I didn't get it immediately until I left.
When I left Puppet, I built this tool called confd. I went to go be a VP of engineering and I got to write Go code. So, we started rightfully so. We looked at all the Java heavy usage and I earned the trust of the team and we started rewriting some of the microservices in Go, shrinking our cloud footprint, and we sunsetted it and got it all into production, the Go code. I was like, we don't really need Puppet anymore. And I open sourced this project called confd and it would pull variables from etcd and generate just enough config, just the parts of Puppet that I thought made sense.
And then Docker was out. And then I was like, wow, we can probably stop moving Python files around. We can probably package them up. Not in our case.
And the idea with Docker was right, that it would define a virtual machine or a container, right? So to me the big value of Docker at the time was, previously I had definitely did the work to make RPMs for every app, even the custom apps. RPMs are the Red Hat Package Manager.
Yeah.
So if you're on a Red Hat system you can do yum install nginx, yum install Postgres, but most people even today don't package their third party apps, like the apps that a development team would write. You're like, no, we just put CI/CD, we'll copy them over there, maybe put them in a tarball, but we usually never went to making official Debian packages or RPM packages.
But Puppet meant you didn't have to go through all of that work and you can still end up with a package, something that was repeatable. So all the stuff we used to do with Python and virtualenv, all the things you used to do with Ruby gems and, you know, the virtual environments we have for that, we got rid of it. Just squish it all into a Docker container and you got rid of a lot of dev tooling. So this is why I think it resonated so much with developers, because we cleaned up the mess of working on multiple projects. And so I was attracted to that. confd was compatible with that.
And then CoreOS was the thing that I was like, you know what, I think I know what I want to do, but I didn't understand distributed systems, not to the degree what CoreOS was doing. So when I moved to the... what was the idea behind CoreOS was like Google's infrastructure for everyone else. And so at that time there was a tool called Mesos. The Borg paper had already been published. I tried to read it when I was at Puppet Labs. I don't understand any of this stuff, it's hard to install, I couldn't justify it. But we would see kind of the rumblings of the Twitter folks talking about the distributed system, and all this maybe big data stuff is going around, but I was like, I just don't understand it. It seems incompatible with the stuff that I think about, even as someone who's worked at Google that had a system like this, right? They had Google cluster file system at this time. It didn't seem as complicated as this Mesos thing. So I was like, I dismissed it.
But what CoreOS did was build on top of Docker. CoreOS is like, what if we had an operating system that only had Docker on it? Everything is written in Go. We can have a little key value store where you can put your config and we would just synchronize it to all the machines. And that was so compatible with the confd way. And so I'm looking at this CoreOS thing and I'm like, yo, this looks a little bit more like the future, because that's what we used to do. You know, at this time I was a software developer at Puppet. I was a software developer, VP of engineering at the other company. And I was like, you know what, ops can learn a lot from the Docker way of thinking about the world, because as a system administrator, we always try to make the OS small, remove things you don't need so it can be secure and repeatable. Docker was like, for the things that need to change, just put it here and isolate it. And CoreOS is like, "What if you had an operating system designed only to do that?"
And I met the CoreOS team at GopherCon, a Go language conference. And they saw me present. And what I learned from the Puppet days, every presentation is an interview. I don't think a lot of people think about it that way, cuz people are looking at you on stage and they get to see what you're about.
And so at GopherCon, I remember also Rob Pike and the original creators of Golang. Oh, this is like the first GopherCon, right? And so the original creators, Russ Cox, all these people I look up to, Brad Fitzpatrick, and they're all in the audience. And I remember I had a talk. Two things. I'm sitting in the audience waiting for my turn. I'm number four on the list or something like this. And the two people who started GopherCon, they're just from the community, Brian Ketelsen and Erik St. Martin. And they're new to this conference scene. Like you could tell, right? You know, they were the best MCs in the world. But they were like, "Hey, welcome to GopherCon." And at the point in time, Ruby is still dominating. And down the street, there was the Ruby conference going on. And we were just in this other building.
And I remember the first talk, they introduced Rob Pike and he gave this amazing keynote around Hello World. He went all the way down to the compiler. He went all the way up and he described how the language took shape. It felt simple on the surface, but it went super deep. And it's like Rob Pike had also come full circle from his AT&T days, through Unicode, through all the stuff that he's ever done. And now we get to see one of his best works, Golang. And my talk was titled Golang for system administrators. I wanted them to see that there's a better way.
The way they introduced Rob Pike, though, I was like, "Oh man, this is Rob Pike. You got to have more energy than that for Rob Pike." So, I'm sitting in my chair and I don't know them, but I'm sitting with a buddy of mine from that company where we rewrote everything in Golang, Billy Click. We're sitting next to him. Say, "Hey, Bill, I'm going to go ask them, can I be the MC?" He's like, "What?" Okay. So, I'm whispering. And I walk to the backstage. It's like, "Hey, Erik, Brian, can I try my hand at being the MC?" And they're like, "Sure." Right? And they knew at least who I was because they accepted the talk.
And so I don't know if it was after Rob, but I came out next. So like, "Hey, I'm Kelsey Hightower. You may not know me, but I'm going to be your MC for the rest of the day." And I don't know what happened, because it was my first time doing that. I'm cracking jokes. I'm having fun. And I said, "Hey, from here on out, when anyone comes to the stage, it needs to be loud in here to the point where we can get kicked out." And so I was, "All right." Hey, so our next speaker, and I think it might have been Derek Collison, you know, he's like a Google... He was talking about some Go stuff, and I remember it got really loud and then they come on stage. So if you were a speaker ever before, and it's like you walk up to basically a standing ovation, you're energized, everybody is excited. I don't care what you do, the energy level is high. And so he comes out, he's having a good time, the audience is having a good time.
And then eventually it was my turn. And I realized, like, who introduces me? So I go onto stage and I was like, I'm next. So we're going to try it like this. I'm going to go back and I'm going to come out and you guys gonna make a lot of noise for my talk. And then I remember just like sliding slowly behind the curtain. So everyone's now laughing. And then the music comes on and I walk out surprised like, hey. And everyone's clapping.
And I do this talk, and at that time it's live demo or nothing. And at the time they had this thing called Go Present. So Rob Pike's team, they wanted to make sure they could use Go for everything, even generating a slide deck presentation.
It was all in Go.
It was all in Go. So you had this nice HTML representation of your slides. You just open up your browser.
Hopefully it won't crash.
Hopefully it doesn't crash. And you can run executable code in your examples. So any code snippet you put there, it was formatted nicely. And you had the little run button so people can see the output of the Go code.
And so I was like, hm, what would a system administrator value you can get from Go? How can I prove it to them? And so I wrote an iPXE server in Go. And I remember in VMware Fusion that runs on your laptop. So if you want virtual machines on your laptop, you can use Parallels Desktop or VMware Fusion. The thing about VMware Fusion on your Mac, you can create virtual machines, but you can also swap out the network card. And I did one where you can have a network card that talks to iPXE. So, I wanted to show them how it would boot up multiple machines from a Go PXE server running on my laptop.
So, part of that demo would start the... I had this kind of snippet of my PXE server and I hit the run button. So, now my PXE server is running in the background. I was like, "So, what can you do with it?" And I remember looking in the front row and Rob Pike and team are just intimately looking at this thing like, "Wow, he just started a PXE server from his laptop from Go Present, the slide deck tool." And so this thing is running and I'm like, "All right, let's bring up VMware. I got to make sure I switch the adapter to the one that supports iPXE. Got to do some firmware thing." And then I remember I booted the VM. You can see the logs in the Go Present of like HTTP handing off the image, giving the IP, and the virtual machine is booting up, and you can just see the amazement on the audience's face like, did you just do that?
And then I booted another thing in the thing and I'm going through why I think that this is a game changer for the craft that we have. We finally have a tool where we write high performant things with the simplicity of imperative programming and we can go beyond just scripts. We can actually build systems. The audience was dialed in and it got loud, like after, you know, things were working, people were clapping. I look in the back and there's a whole bunch of people now standing, and I recognize some of the names because they're from the Ruby community. These are people that are writing the Ruby books. And then, you know, after I'm done, there's a break and I walk back there. They're like, "Hey, we saw on Twitter that you guys are just like going crazy over here. It's out of control, it's electrifying. We left the Ruby event to come here."
And so I felt like, man, it had arrived. But guess who else was there? The CoreOS team. The CoreOS team was watching me because I was PXE booting CoreOS. This is what I mean, every job's an interview. The team was like, all right, we can see that, yeah, you should join the CoreOS team. So that's how I ended up at CoreOS. This is how I really felt like the container movement had legs, and this is what, maybe a year or two before Kubernetes comes out.
And then Kubernetes came out and you were starting to contribute to that as well.
Yes. So my CoreOS... and I think the thing that's very important, a lot of software engineers sometimes, they look at the people on stage and they ask questions like, does that person know what they're doing? Is this person just like an evangelist? Did someone write this demo for them? Did they give it to them and they just run the code?
Yeah, we think that, don't we?
Well, I mean, I can see why, because sometimes it's hard to value skills you don't have. So, a lot of software engineers are terrible. If you put a soft some, they can't talk. They can't simplify concepts. Maybe they can write code really well, but this is a set of skills that they may or may not have. And so, when you see someone like that, you're questioning them.
And I remember giving a talk one time at Strange Loop about etcd and CoreOS and someone was like, do you understand, is etcd like a CA system or AP system, like the CAP theorem? And the question was kind of loaded, like we don't think you even know what that means. I was like, etcd is going to always favor consistency. He was like, that's not correct, the Raft paper, blah blah blah. I said, listen, you said etcd. Let me show you. This is the Raft log. This is my three nodes. I'm going to turn off two. And you notice I can't write any keys. So availability has been sacrificed. Consistency is being preserved. I'm going to start another node. There's going to be a handshake. There's going to be quorum. I will be able to write keys. Now it works. That's the CAP theorem in reality. So it doesn't matter what the Raft paper says. You're talking about a Raft log on a single implementation. Raft doesn't talk about cluster membership, leader election, how it's implemented, and what you should do in the different modes. That choice is yours. And this is how etcd is implemented. And I remember he was like, "Oh, this guy actually knows what he's talking about."
Being at CoreOS, we were working on our own fleet management system called Fleet. We're using systemd and we're trying to synchronize configs through etcd, and remember, in a CoreOS cluster all the nodes communicate via etcd. So imagine using systemd. For those that have never used systemd, you put a unit file, like I want this process to start with these flags, you know, bind to this port, and you put it in a directory and then systemd will start it. And if you had a thousand nodes, of course you could SCP that file to all 1,000 nodes and then there you have it. So we decided, instead of you copying all the files to all the nodes, just put the unit file in etcd, and the node that should run those things would then pull from it and just run those unit files, and we called it Fleet. So we had our own vision of giving people a distributed OS or distributed system.
About a year goes by, Kubernetes comes out and everyone was like, what's that? And we got like a day notice. So the Google team reached out like, hey, tomorrow we're announcing this thing. Here's the GitHub repository. You guys are under embargo. Don't talk about it. And so I'm thinking we're not part of this story. Like our names are not in here, just says Google and Red Hat, and they're going to announce it at DockerCon. There's no CoreOS in here. So, what can we do?
And I remember I stayed up all night. I got access to the GitHub repository. I'm reading all the Go code trying to figure out what all these binaries mean, cuz there's no docs.
Yeah.
And so I got everything working on CoreOS. So when they did the official Google announcement, of course there's a famous Docker keynote where they unveiled it, to the surprise of even Docker to some degree. And I remember they post, they're number one on Hacker News, and then we post, hey, what they just said, but here's how you run it on CoreOS.
And some examples and commands, and I had to do a few patches to get it to actually work, and I had to build some binaries to get things glued together.
You did that overnight.
Overnight. I didn't go to sleep. I'm just like, hey guys, I finally figured out how to compile everything. I think the kubelet does this. You have to put this there. They had like a kube-up SSH script, but I had to reverse it because I'm not using Google Cloud. We got CoreOS. We're on bare metal. So, I have to PXE boot some things. But I think if you put all these things in the right place... there's this etcd thing that's ours. We know that. So, we know how to use that. But I think the API server connects to that. There's no volumes. There's no config maps. So, all you can do is get this thing stood up and then you can submit a config and then it will just basically use Docker in the background. Okay, I can document that.
So, we get everything to work and I write this nice little guide. Someone in our team, they publish it to the official CoreOS website. So then we launch that on Hacker News and then we go to number one and everyone is like, Google just announced this thing. We don't know what it is. And then Kelsey launches this thing. We now know what it is. We know how to install it. We know how to run it. So now people are downloading CoreOS just so they can play with Kubernetes.
And I had a keynote probably in a week. I don't know what I submitted to the conference, 'cause usually you submit like months ahead of time. I was like, "Hey, I know this talk was supposed to be about this, but it's not today. It's going to be about Kubernetes." And people like, "What's that?" It's like, "Yeah, it just got announced. I'm going to show you." And I started giving people live demos of Kubernetes and how to make it work, you know, using CoreOS and all these things. But the team still wasn't sold because we had our own road map.
Yeah. And also at the time...
And you had your own fleet management software.
Yeah. We had our own. Kubernetes was now competing with it, with Fleet specifically.
Yeah. And also we didn't know we can trust it, right? Remember, Docker is the king. Docker's number one. There's Docker Swarm, right? And we're with Fleet. And right now, we're like, Google also launched years prior a thing called Let Me Contain That For You. It was a container runtime written in C++ to compete with Docker. And no one cared.
Yeah.
And so we didn't know, like, maybe no one's going to care about this either. It's only when I started going home and starting doing small contributions and starting to read everything and getting a feel for it, I was like, "No, there's something here." And so for maybe two or three months, and luckily the founders of CoreOS were really nice, Alex Polvi and Brandon, they were just like, they didn't get mad or anything. I was contributing nights and weekends, kind of like I was at that other company. All of my keynotes were more like Kubernetes plus CoreOS.
And I remember at some point it was inevitable. We all got in a room. I became the product manager of CoreOS at the time, and Alex is like, I think you got the vision here. And we got everybody in the basement in San Francisco, in the office, we were like, hey guys, all in on Kubernetes. Fleet, deprecated. All these things that we're building, deprecated. We're going to go all in. And I'm glad we did, because Alex Polvi came up with the name operators, which is like a core idea in the Kubernetes community. We put all this effort in there, me and another guy working on a thing called CNI, which is the networking layer for Kubernetes.
And what Kubernetes really meant for me was that previous 15 years of experience as a practitioner in the data center, learning promise theory, learning Puppet, where it works well, where it doesn't, and then understanding that Puppet wasn't the only way, and then going through all these loops. And so when I ended at Kubernetes, it felt like this would be the thing I would build if I only knew how. And that's the way I explained it to the rest of the world.
And at that GopherCon, maybe the next one, there were people from the now early Kubernetes community. It's a small company called Kismatic. They were kind of a CoreOS competitor to some degree, but they came up with the idea, like, we should have a conference just like GopherCon. We called it KubeCon. Joseph got the logo going, and the Kismatic team kind of put up the money for the first event, and we really welcomed the entire community. And now it's been 12 years later, and the CNCF has done a fabulous job of keeping it going. Now there's like, what, 13,000 people here in Amsterdam keeping that thing going.
So I asked this from Kat Cosgrove as well, who was on the podcast. What do you think really made Kubernetes break through and then just become the de facto way of orchestrating nodes and just winning? Again, there was, at CoreOS you were building Fleet, Docker had Swarm. Like, as you said, in the beginning it didn't seem like this will be this big.
I think the number one success criteria was Docker. So remember, there was Mesos and Mesosphere, and they had their own runtime. HashiCorp had come out with Nomad, and they had their own runtime. But the biggest runtime that had already got global consensus was Docker. So by that time there were so many Docker containers and Docker workflows and Docker Swarm.
Maybe the Achilles heel to Docker Swarm was its design. They tried to take the Docker API, which worked really well for one node, and expand it across multiple systems, and it was not the right API to scale to another type of thing that we needed. And so they kept trying. They tried to add storage. They tried to add networking. But the Docker API was never meant for that. And so the Kubernetes team was smart. Instead of trying to say Google's better than everyone on everything, they did a couple of things correct. Let's just use etcd. Let's just use Docker. So you take those two things and you take the experience of the people who wrote the Omega paper, which is kind of thinking about what would come after Borg, and at least the things like Mesos.
So Omega was a system where, what Google would have built after Borg, but they never built it, right?
Yeah. So they had elements of it. So like the Omlet, you know, this agent that would be more declarative. A lot of hints from the Kubernetes world that will come later.
And then just to be clear, Borg was and is still Google's way of managing their, back then, hundreds of thousands, now probably millions or tens of millions of servers, and they were best in the world with this, or they still are, right? So they learn...
I think Borg was one of these things where you integrate the hardware, the software, the package management, the configuration management, MapReduce, right? Borg is this thing that just expands and grows. In some ways I guess it's extensible, but all that insight and knowledge. But then they get so much experience with that. If you were to do it again, what would you do? And you read the Omega paper and it's like, here's what we learned from scheduling. It doesn't need to be that complex, especially for certain workloads. You don't need this high-performance, overengineered thing. There's a simpler way to do scheduling, especially if you can give the scheduler a bit more metadata about the workload. They're also big game changers. Now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers.
And so I think that's the number one success criteria, that we were already off to a running start because you could just reuse the same Docker containers. You didn't have to rebuild a new image thing. So given that, what they tried to do in the early beginning was just fill in the gaps. And in many ways, yes, it's a new system, but it fills in the gaps. The one gap that they filled in was Docker had an entry point. So if you needed a Ruby app that needed NGINX and your process, you used to have to write a little shell script, the entry point script, that would do all this magic, almost imitating an init system. Kubernetes is like, no, no, no, you don't need to do that. You can just make separate containers, and then Kubernetes would run them as a process tree. And so for many people, it's like, finally now we can have a clear way of thinking about application architecture.
Like blocks.
Like blocks now, instead of like you have to open the entry point to see what we're going to do, versus full life cycle management, independent. So it solves that number one problem.
The other big one that I think that they solved: number one, we went from infrastructure as code to infrastructure as data. And infrastructure as code is like, if this, do that, bring in this module, for loops, all this stuff. And Kubernetes is like, no, no, no, you have to specify exactly the containers you want, how much memory they need, and then we have the status field to tell you if they were running or not. And you would take this data object that you could write by hand, give it to an API, and then the control loops would operate on this state. So that means it didn't matter if you had Ruby, Python, or anything. You can just take your IDE, write some YAML, give it to another tool, manipulate the YAML, and then pass it down to the API servers. You can build any combination that you want without having to be a compiler first. That to me was a fundamental game changer that I don't know if a lot of people understood, why it felt very easy to onboard to Kubernetes. kubectl apply object, off you go.
And the last thing, I think credit to Brendan Burns: the ability to extend Kubernetes in a first-class way. OpenStack didn't have it. Mesos didn't really have it. In Mesos, you have a scheduler and you built the other part of the scheduler. So you can have Spark, Hadoop, Marathon, but you had all these other tools sitting on top of a thing. So an extension in Mesos was heavy, almost like a whole other system. The thing that makes Kubernetes powerful: there's a data model. We gave infrastructure a type system. So instead of imperative shell scripts, you finally had types. So if anyone's ever come from Python to a typed language, types do a lot for you in terms of cognitive overhead. Like, you really know what goes into this function. If you put the wrong thing into this function, it doesn't even work. Like, you can't pass a string where an integer needs to be. Kubernetes brings the same semantics to infrastructure. And finally, now it's much safer to automate things because you're gluing together things that actually have structure and types.
I was about to say, the reason we love types and every language goes towards them is safety, and it eliminates a whole class of errors.
Yeah, you can do things like static analysis. You can have other tools compile different things and ensure that they have the exact same thing, and you have this validator that tells you that's not the right object, that's not the right field. And so once you have all of those things, you can build a really nice deployment system. So kubectl, deploy these containers, no problem. But what about everything else? So instead of trying to evolve Kubernetes to do everything, Brendan Burns, I remember sitting next to him, he's like, "Kelsey, let me show you this thing. You can extend Kubernetes just by giving it a description of what you would like your object to be." So if you were thinking about this from the REST world: hey, I need a user, here's the CRUD operations, and just give it to the thing and it does everything else for you.
And so when that came out, it's like, so if I wanted to manage, let's say, a firewall? Like, yeah, you can describe a firewall and you give that to Kubernetes, and all the tooling works. You can now say kubectl apply a firewall. And so now we got tools like cert-manager, where if you want a certificate from Let's Encrypt, you can just say what domain you want, where it should live, give it to it, and now you had a first-class extension. You didn't have to uninstall it, you didn't have to make some magical binary, and it really didn't matter what language you're on, because once you put the data model in place, you got the machinery. And if you care, if you like Python, you can have Python running a loop, grab the data, and then make it so. If Bash was your thing, you can literally pull the config using a Bash script and make it so, and just update the status field. That opened up the entire ecosystem. So Cisco could come in, do what they wanted to do. Red Hat can come in with OpenShift and do what they wanted to do. To me, that was the game changer that brought the rest of the community in.
And then you joined Google, not very surprisingly at this point, I guess, from the outside of course. But how did that go? You've been contributing to Kubernetes as well. The team was there. Did you join the Kubernetes team?
No. So by that time I'm at CoreOS, Kubernetes has definitely taken off. I'm giving lots of keynotes now. Everyone wants to know my opinion. I'm making all these prototypes. I'm kind of moving things forward. There's a Kubernetes book now, right? I'm a co-author with Brendan Burns and Joe Beda at the time. I'm like, you know what? I'm thinking about my exit.
So, in our careers, we do a lot of work to get into this field. All the certifications, the boot camps, the studying, some of us college. And then once we get in, we're thinking about career progression: dev, senior engineer, principal engineer, distinguished engineer. And we spend almost our entire lives on that trajectory. And our field is so young that some of our pioneers are finally, like, no longer here for the first time. We're not used to that, and we're not really used to people retiring. Like, Rob Pike just retired. The concept of an exit for an individual contributor or a leader in the tech space is, we didn't have a lot of those. Linus is still at it. He's not retired, right? So we don't spend a lot of time thinking about the exit plan in our field. If you're a professional athlete, your body will tell you when it's time to go. And so when at CoreOS we reached this peak, I felt like I've done everything I've ever wanted to do.
In the tech industry.
In the tech industry. From 1999, to being unsure of myself, to seeing myself on the side of museum buildings, full landscape view, because people are coming to see what I think about where technology is actually going. And so it comes full circle. You get a bit of taste of the fame. You can look at GitHub and you meet people: we use your libraries, we used your command line tools. I started my career, like, some people were not even born when I graduated high school, and they started their careers from those books. And so I felt at the time that I had come full circle, and I was starting to think about the exit part of that journey.
I remember spending time with Jet Propulsion Lab, JPL, part of NASA. And I remember being there, and I was so excited because the movie The Martian had just wrapped up filming there, and they gave me a tour of the facility, like the Mars rover, the new one, before they launched it to Mars. They were QAing it. It was just going in a circle around the track. It had a little laser on there so it can split rocks, and they were showing me how they improved the wheels over time. I went to another lab and there were a bunch of scientists working in there, and it looked like a fish tank. And they described it that way, like, yeah, we determined that if you want to replicate parts of Mars' surface, and I might be getting this wrong, the rocks that you find in like a fish tank, we can replicate some of this stuff, and maybe slightly different than that. As I'm watching these people work, and he showed me how spacecraft has evolved over the last 20 or 30 years, and I'm like, wow, you all seem to have an actual purpose. For the first time I've seen people using technology not to just make more apps, not to add numbers in a database, but to actually have humanity do something. And so they were not all about Kubernetes and Docker and Python and Go. They were like, we're just trying to get a person to Mars and back again.
I remember in part of the interview process, their interview questions are: if you had to deflect a meteor, how would you do it? But it has to hit one state. So now you're in the leadership position. What would you do? And you're just explaining the answer, like, you know, I would kind of bring a bunch of experts, and then, you know, you got to think about the ability to evacuate people, and you have to explain yourself. And the core part of my answer was you would have to explain yourself, almost a 24-hour live stream. Here's the trajectory. Giving everyone the countdown. Explain every decision you're making. I chose transparency. I chose truth. I chose, like, look, we have to deflect it. There's no way to make it zero. So we've chosen this state, and this is the evacuation plan. And we estimate that this number of people won't make it. We're just being honest with you. No need for conspiracy theories. It's live. And I was like, wow, what an interview. A 20-year career at that point, never
had an interview that made me feel that way. And so I actually was going to go to NASA after CoreOS. I even signed the employment agreement. I was going to move to Pasadena, California and work at NASA on the Mars mission and lead up the infrastructure and the infrastructure teams. And of course Google called. I was like, hey, come to Google. And at that point, I was like, for what? You have hundreds of thousands of employees. I admire Google. I've been there before, but not in that capacity. I was going to go to the headquarters.
Yeah. Not to the
Not to the data center but to the headquarters. And I've always admired people like Brian Grant, Eric Tune, Dawn Chen, all of these wonderful people that I got to work with through the Kubernetes community. In many ways, I felt like I was already working on the team because by that time I had commit access to Kubernetes. So, I kind of felt like all the things I wanted to achieve in that regard was there. And so, I was like, why would I come work there? I'm just going to be a cog in the wheel. I'm going to go there. I'm just going to disappear. You're going to just make me work on Google stuff all day long. What's the value in that? I've seen the peak of this. And they were like, we won't do that. I was given the opportunity to do DevRel and it's the first time I ever did it. But DevRel represented freedom. No tickets, no write code measured against SWE benchmarks. I was like, I don't want that. I want to be able to make impact. And so the team was smart. They were like, look, we got this area called DevRel and we'll let you define what you do.
So you got to write your job description pretty much.
Yeah. But as an entrepreneur, I know how this goes.
How does it go?
If you come in and you really do DevRel stuff, in my mind, you're going to get fired because if you limit yourself to the external perception of DevRel, you go to conferences and you become more of an evangelist, you do tutorials and guides. For me, those are activities. I'm a person of impact. And so, the first thing I did when I got to Google was like, where's the customers? How do we make money on cloud? So, I need to figure out how to talk to the customers. So, hey, where's the sales rep? If you ever need anyone with Kubernetes expertise, call me. I will fly to Disney. I will go to Walmart on site and I will whiteboard for 6 hours because that's the revenue component. So globally went to Australia, Canada, doesn't matter. We don't even have a region there yet. I'm going to bootstrap it. So now I'm growing my impact on the revenue side.
And the reason you said if you would have gone as DevRel, you would have been fired because you would have not been generating any revenue. I just felt like I was going to be fired.
But this is just like thinking as an entrepreneur, as like you want to make money.
I want to make sure that I'm impacting the business. And for most businesses, revenue is the criteria. And no one ever made me do that, by the way. It wasn't like Demetri. It's like no, no. To me, I understood the value of revenue. So I would go out and I was able to do it in an authentic way. I'm just talking about the same things I was talking about before. And then product impact. This is cloud. Why limit yourself to Kubernetes? There's serverless, there's databases, there's metrics, there's so many things here. So now I'm like, I need to learn everything and I want to employ all of my skills. So it turns out my time in financial services means I can be an exec sponsor. I knew how to go from hello world to hello revenue. So if I got into an exec briefing, I didn't waste everyone's time showing them the latest feature of Kubernetes. Doesn't make sense. They want to know how these tools come together to lead to actual app impact and outcomes. And so I matured there and I also got smart. You got to go to other teams and you read the OKRs, right? So another team might say, "Hey, Kelsey, we're really trying to get more adoption on our metric stack." And I remember the first thing that I started implementing at Google was a thing called empathetic engineering. So you have a lot of smart Googlers. These people are brilliant. I mean extremely brilliant, to the point where the hardest problem Google had, in my opinion, was what to build. Not how to build it, what to build.
They could do that.
And in some cases you end up with like five messaging systems. But the thing is, what to build seemed to be the most pressing problem. And so the one thing that I tried to do was like, how do you convince other engineers, you know, their manager, how do you get them to trust you? And I started these empathetic engineering sessions where the first one was like, get the Kubernetes team in one room. All of them, engineering off-site. I want you all to install Kubernetes but you can't use any scripts. And remember, some of them are distinguished engineers and principal engineers. Some of them worked on Borg. Some of them are just the original creators of Kubernetes itself.
And it was so fun to watch them struggle because it's like, do we install Docker first? What version of Docker? Can we put this on Ubuntu or does it need to be Red Hat? And so an hour goes by, teams of four are like, "Nah, man. This doesn't work." I was like, "Great, you all can stop. I'm going to show you how I would do it." And of course, I know how to do this because I get to prepare, right? So, not a knock against them. I'm just like, "All right, Debian, tune the kernel this way. Put Docker on there. Put etcd, put the API server, put all these things. There you go. That's how you do it." And I was like, "Yeah, I mean, of course, you had to prepare, of course." And so, the question then was, from an engineering perspective, how will we make this better? And then people are like, well, if we had OS packages this could have been apt-get install and we could have just used local machinery. Like, that's a good idea. Someone was like, we will make that happen. Another person was like, even if you had those packages though, you still need to know what order and where the config files go. So kubeadm was born, which was a command line tool that gave you a procedural thing. But the other thing that I remember from my career was I don't want just a tool that abstracts everything from me. I want to know how it works. So I wrote the guide Kubernetes the Hard Way. And that guide is what I used to help teach people at GitHub in the early days, pre-Microsoft, how to run Kubernetes on bare metal and walk them through that guide.
And so it was that empathetic engineering that helped me make a huge impact on cloud because I can go to every team, every org. And instead of guessing what their road map should be, given someone who had spent time in the field, given someone that had this enterprise background, hands-on experience across lots of tooling, I knew where people were coming from. I would say, based on where Google sits in the landscape and in the competitive landscape, given what our abilities are and what our customers need, I think this is it. But I never said it that way. I would get everyone in the room.
You would have them discover it.
Have them discover it. So you
And nudge them.
So you knew what you think the key, the most important problem areas were, for example, and then you orchestrated a session to help people.
I try to always know the two things in multiple product areas that would make the most impact, aka revenue, that would get adoption. And then launches versus landings. You would put all these things in motion. They would land at different times. So launches is we ship the thing, people have a big celebration. I know that doesn't matter as much as the landing. People are actually paying for it. So when I got good at that cycle, the promotions were a little predictable because I would be able to make impact, right? Sometimes you build prototypes and sometimes you would contribute to things like Cloud Functions. But the overall goal though was to use every skill you had. So working with PMs, we got to add Go support to Cloud Functions. We're the team that made Go. Amazon has support in Lambda for Go. We need to do this. And also, I have a keynote at the next GopherCon and I'm talking about serverless. And I have two choices. I have to use Lambda or we can use Go functions. And so, we sprint. We get it done. We get all the Googlers' readability. We get it checked in. It's in alpha. And I tell the PM, "Hey, you want to go to GopherCon? We'll launch it on stage." We launch it on stage. We tell people how we designed our worker, how the internals work, and then they can just come sign up. That was the trifecta. You do the work, you have the vision, you execute, you launch, and you land.
Land.
And the landings compiled over seven years is how you go from, like, I think maybe L5 to distinguished engineer.
L9. That's like four promotions.
Yeah. Four promotions, seven years. But it's interesting because a lot of times when you ask someone how they got promoted, let's say, yeah, four times in seven years at a place like Google, I would expect, just being naive, that they will tell you like, okay, this is how I planned, or like, this is what you need for each level. But it sounds like it's very different. You just had landings that created impact. And you were focusing, do I read it correctly, that you were not focusing on your promotions or the next level? You just wanted to, you know, do the best work that you could.
Oh, no, no. I was focused on your next levels because that was the goal.
Because think about it, my whole career, I've always tried to acquire the skills and make the impact so I can move to the next level. And so at Google, the levels were expressed as promotions. And there was a point in time in the org that I was in, there was no level seven for an individual contributor. So now we have to make a level seven. And then once you get to a level seven, now you kind of pave the way for the other people that want to come up through that icy path. And then level eight isn't the same as level seven. Level eight isn't the same, or level 9 isn't the same as level eight. And yes, there is a formula to some of the promotions early. So from 3 to four, four to five is a little formulaic. You have a ladder. There are things that are expected of you. And the decision making on those type of promotions are localized, meaning your manager, maybe a director. But then outside of that though, when you start to go higher, it's now expanded, where there's now other teams that have to understand your impact in a way that can't be biased by a local team. So if you think you've made a lot of impact and then people across the org do not agree, that allows you to really throttle what impact means, right? Because if it's just your team, I really like this person, now everyone's a distinguished engineer. And so I think Google did a really good job of saying, look, it was okay to be like L5 or six for your entire career. Yeah, that's their terminal level. I think it used to be L5, now it's L4 actually. Okay. But they moved it back 'cause I think L5 has gotten a bit tricky and they don't want to fire really good L4s.
Exactly. So, and I think for a lot of people, you can follow the formula and get to where you need to be. But I was like, at the time, I think there's like a couple hundred distinguished engineers. So, as an entrepreneur, if you show me the top of the mountain, I want to get there.
So then you were targeting, you're like, all right, how can I get to the next one? To the next one. And impact was the name of the game, right?
Yeah. And I guess the only thing I would probably add to the moment or to the situation was I wanted to do it authentically. And so there was one part of the time, I remember I didn't get one of the promotions, but I was getting promoted pretty fast. So of course people are like, "Dude, you need to actually make an impact before you get promoted again. Stop it." And that was good feedback. But I remember I took a chance. So there's a pattern to doing a promotion packet, like structure. There's examples and you go through all of this stuff and you get feedback. I remember one year I was like, I'm not doing that. This year I'm just going to talk in like the first person. "Hey, I'm Kelsey. I work on these things, not these things. These things are important to Google. So, here's exactly what I did, but more importantly, here's the people I brought along. Here's the teams I've impacted and here's the results of this. So, I did this project and here are the results of that." And I'm talking in this way like I'm ignoring the process. I don't really care about the template.
And as someone who at that point in time was on some of the promotion committees, where we're looking at the promotion packets and making a decision as a team, I decided to write my packet for those people, so that when they got it... 'Cause look, if you're having to read a lot of these, it's hard. You're like, "Oh man, they're all so dry. Everyone's being very safe. Everyone's only telling me what I want to hear. I don't even know the person after reading this whole packet." I said, "I'm not going to do that. I want them to see me as if I was in the room advocating for myself." And I remember getting feedback on that package like, "Kelsey, this is like, come on, bro. This is, you know, things." I was taking it seriously. I wasn't making a mockery of it. I just wanted to make sure that they understood what I was doing, and I was aware that I wasn't just trying to play the game. I was trying to approach this process authentically. And I got promoted off of that packet. And look, it could just be sometimes the work speaks for itself, but a lot of times people can't see the work if it's not presented correctly. And so that kind of slingshot... I mean, and the reason why I tell that story is that every distinguished engineer doesn't get there the same way. Everyone thinks like, oh, how much code did you write? What complex thing did you build? And for me, I think it was the impact on Google Cloud's culture. The empathetic engineering thing became an official thing that they used to onboard other engineers. It became things that had a whole team behind it. They used to go give them, and a product manager that would evolve the program and integrate into HR the philosophy, around other engineers saying, "Kelsey, we just did an empathy session. I want to show you the results of it because we're about to ship it." And then engineers started really thinking about the customer. I mean, Amazon was always known for that to some degree, but to help Google get on the same page... I mean, I'm pretty sure other people had an impact, but there was a direct line of impact from those type of programs, and also me diversifying, moving away from Kubernetes into the serverless realm, moving to the world where you're helping out the Postgres or the Spanner team add a Postgres interface, the Go team getting a little bit better with cloud. Just making other people successful around you is one of those things that helps you become distinguished. It's the impact, the ability to influence.
Before you got to the distinguished level, you shared a story about Microsoft and an offer. Can you tell us about that?
Yeah. So, I'm one of those people. I don't like ultimatums. It's hard. An ultimatum would be: you're working at a company for a couple years. You're doing a really good job. And I'm just going to make up numbers here just to protect everybody, but just say you're making $100,000.
And for a lot of times, you could be happy with 100,000. And so a company says, "Hey, we're hiring." And you go there, you do an interview, and they say, "Look, we're paying $120,000." At that moment, you are actually worth $120,000, 'cause you now have evidence from the market. The $100,000 you currently make doesn't look so good anymore. You can't unsee a job offer for 120.
And so now you have to make a decision. You can commit to that, leave your job and go make 120 and start over. Or you can do the ultimatum thing. If you don't
pay me 120, then I'm leaving. And that puts everybody in a weird predicament because sometimes it doesn't have to be that adversarial. Sometimes it's just this is evidence that, hey, look, I want to advocate for myself. I know we're not in promotion cycle, but I believe I'm worth 120 and I would like to have that conversation. And someone would say, hey, well, you need to go prove it. It's like, well, I have an offer if you want to see it, but I would rather be here. And that can turn into a you went to go look for another job. It's like, oh my god. So I'm stuck.
They can have weird dynamics, right? Especially with your manager or your management chain.
Exactly. So I was always wary of that, but I also knew that in business, that's just how it goes. And so I'm really at the peak now. I'm thinking like early retirement. I'm probably can get out of here at what, 55, 60 if I continue on this pace.
Yep. And then Microsoft is like, "Hey, come through." And look, I never had an executive recruitment process. So by this time, I'm probably considered an executive at Google. Once you hit like L7, you're kind of considered like an executive.
What is an executive?
So an executive just means that you probably have an admin to help you with some of the tasks you're doing. You're probably going to be asked to be an executive sponsor of things and programs. So if a team wants to have a program for something and there's going to be a budget for that, you may have to help oversee that program. So they need executive sponsorship. Another engineering team knows that they're going to need budget for something and they need someone that has a little bit of political capital, a little bit of weight in the organization to help endorse them. So that can be executive sponsorship or you might be assigned to one of the largest customers the company has. But that executive set of duties means that you're going to be making impact above and beyond yourself typically to support other parts of the organization.
So I grew into that at Google. Microsoft was like, we want you to start there. And here's the thing, I didn't think about my role that way there. I'm still the old Kelsey from CoreOS days, right? I'm just doing well here. And I wasn't going to interview at Microsoft. I'm like, for what? I'm at Google. I mean, I had a lot of freedom at Google. I was making impact at Google. I had a good reputation. And so, I'm like, "No, my trajectory is fine. I'm not doing that."
And to be honest, I don't like Windows. I didn't like Azure. I don't like .NET. I like GitHub. VS Code is nice, but my whole career, the majority of it, has been rooted in authenticity. I've been working on the things I actually like and care about. This would be one of the first times, maybe outside of some of the enterprise roles, that I'm going to go work on a set of technology that I wouldn't use on purpose.
And so I went there and I met the Microsoft team. And the weird thing though is I had a recruiter and they swapped the recruiter out and it's like, "Hey, our mistake." I'm like, "What do you mean mistake? This person was super nice and they asked for a resume and I didn't have a resume cuz it's been seven years
cuz you had to create one."
Well, it's been actually 15 years at that point since I ever had to show someone a resume. So, I'm like, "Resume? Oh, man. I haven't made one of those in a long time. I'm looking online for a template. Like, what's the style of résumés these days?" And so, I'm like trying to figure out a resume. I'm like, "See, this is why I don't waste time interviewing. I don't got time for this anymore." And they swap the recruiter out and it's like, "Hey, sorry about all that. We don't need no resume. Sorry, that's not this kind of process. What we want to do is make sure that you meet the right people who represent Microsoft." I was like, "Okay, that's different."
So, I get on site. I'm like, "Okay, what kind of quiz questions are we going to be prepared for?" You're not doing quiz, bro. You had commit access to Kubernetes. You wrote the book on Kubernetes. You're leading these things. You helped start K. Come on. Like, you have a Wikipedia page. We don't need to try to figure out whether you made the impact. We can just look at GitHub. There's evidence there.
And so, I'm meeting all these leaders. They're bringing all these people who are behind the scenes. I remember meeting Scott Guthrie for the first time. And I didn't really know who he was because I just didn't know the lineage and the history of Microsoft. And you know, of course, I'm looking up who these people are and I was like, "Wow, they're courting me. Oh, this is what that feels like." I mean, Google did it a little bit too, but not like this. I'm looking at, these are the executive directors of
This is like reverse interview. They're not interviewing you. They're trying to sell themselves.
You know what? And that's the thing that I appreciated a lot because they were like, this is your career. You built a fantastic career over there. We can't ask you to throw that away without understanding what you would be doing here. And so it was like, if you want to understand the business, if there's someone you haven't gotten a chance to talk to, just give us a name. And I'm talking to them and the one thing that I really respected about Microsoft was I think by that time they had also acquired GitHub. And so they had a big vision for themselves, a lot of diversity. And I was like, "Okay, there's a lot of opportunity here." They're also all in on Kubernetes. They had just acquired people like Deis and Brendan Burns is there. So I was like, "All right, Kelsey, you can come and make an impact there. There's room to grow." I'm like, "All right, I might do it." All right. So I'm glad I did the interview.
And I get home and it felt like that time I doubled my salary I told you on the way home, and I remember I get this email from Satya, the CEO of Microsoft. And I'm like, man, he wrote this nice email, Kelsey, that I heard you had a good experience with the team. Remember, I did the interview at the Microsoft headquarters, right? So, hey, heard really good things from the team. Just wanted to let you know, you're going to be respected here. We're going to support you as a team. I'm like, damn, support as a team coming from the CEO. And the offer was like a PDF. It's an attachment. So, I read this thing and so number one, what an honor. This is the CEO of Microsoft. So many more important things to be doing than to be emailing me about a role.
And I open a PDF and, not very often in your career, there's a zero that gets added to the equation. And so you're looking at this like, I didn't even know that they do that. We know that it happens, but the person that graduated from high school in 1999 that chose the A+ certification didn't know that was available even while I was at Google having all this success. And Google paid me pretty well, too. But I know you can add another zero still. And so I'm like, whoa, this is crazy. And I'm like, wow. So I showed my wife and she was the one that said, "You should just go interview. Like put your ego to the side and let's go see what's out there." So, shout out to my wife.
And so, I get the PDF and I'm like, okay, this number is perfect. Honestly, I don't know what to say, but let's just find out, like, is this really the only number? So, I remember giving a counter like, you know what, I think it should be this. And the funny thing is Microsoft countered back higher. So, we're not playing around. I'm like, oh, whoa, now I understand that I don't understand this part of the game.
Yep.
And so I have this offer and I knew that I wasn't going to go interview there if I wasn't serious about taking it. So I was serious about going to Microsoft. Been almost 6 years at that time at Google and I went to my manager. I had the same manager, which is legendary at Google. I had the same manager for 6 years straight even with all the reorgs, same manager the whole time. He was a director. So even when I got leveled up, they allowed at least someone one level above still to report to a director. And I had such a good relationship with them and I told them what happened. I said I wasn't looking and they asked me to come in. So I went in and I'm going to take it. Number one, it would be financially irresponsible to not do this. So that will be the driving force, and also I get to stretch myself in another way and see if I can make an impact again. He's like, "Okay." But it was no ultimatum. I was like, "I'm leaving." And he's like, "I'm just curious what Google would say." I said, "No, not great. I don't want to do ultimatum."
Yeah. You don't believe in it?
I mean, I do believe in it, but I didn't want to do it.
You don't want to do it? Yes.
I didn't want to do it
cuz you understood the dynamics.
The dynamics, especially, in many ways, Google had been really, really good to me in every facet. So, it wasn't ultimatum time. It was more like I've earned it. And he said that too. I gave him the PDF and he looked at it. He just started smiling and like, "Oh, wow. Whoa." And then he said something that was really dope. He says, "I want you to know you're worth every penny of this." And I was, "Oh, the whole
That's
a game changer because he knows me like no other person I've ever worked with. No one knows me as well as Greg does. He knows my strengths. He knows my weaknesses. He knows my ambitions and my motivations. He knows all of them. For him to say that, it was important."
And so I'm thinking if Google wanted me to be at that level, they would have done it already. But also, pragmatically speaking, no one really knows your situation. Like all these people have thousands of people they manage. They're not targeting you or being mean to you. At least in my case. And so maybe they thought I was making that kind of money already. Maybe they thought that I was already a distinguished engineer. How would everyone in the org know? They got other things to worry about. So he presents the thing to them and I think within hours they're like, this is no problem. Like, hey, don't worry about that number. Here's a bunch of stock and all these things. And I'm looking like, whoa, whoa. So now I have the money and I'm at the company I want to be.
And you didn't do an ultimatum.
I didn't do the ultimatum. So I felt good about the relationship. I didn't feel like there was going to be some retaliation. I had no fear about that. And I continued to be successful, eventually got promoted to distinguished engineer at Google, told the Microsoft team no. But the one thing that people saw from this, I didn't talk about this at that time, but I did do a tweet and the tweet was "different company, same team." A lot of people were like, what does that even mean, different company, same team? But people were retweeting it and they liked it, and people thought, oh, it's because of the community stuff and how the different people in the community work together. And that was part of it, but a big part of it was that moment that Microsoft got me the biggest raise in my career at Google.
And about 3 to 6 months later I was in San Jose and Satya was there and his admin is like, hey, Satya would like to just meet with you. I'm like, the CEO of Microsoft got time to do anything? So I'm in San Jose and then I go to this hotel and the admin meets me downstairs like, hey, Satya is kind of ready to see you now. And I'm going up and now he knows I'm coming, right? Like, let's just say the meeting is at 1:00. He knows I'm coming for like days. And then you go into the hotel room and like the doors are open overlooking a mountain range and Satya is sitting there overlooking the mountain range like a Vanity Fair photo shoot.
And I remember before, on the plane ride there, I'm reading his book Hit Refresh, and I remember the opening chapter, he talks about starting at Microsoft as a developer advocate and now being CEO of Microsoft, advocating for the soul of Microsoft because he had been there so long. He saw the birth of cloud and all these things. And the book was actually a pretty good read about his trajectory in the industry and his time at Microsoft. So, now I'm done with the book and I'm about to meet him. So, I have all this context in my head. So, when I walk through, I felt like I had this rapport with him for some reason because I read the book. And then he's looking out over the mountain range and I walk through the door. I was like, "Satya, why are you sitting here like this? You knew I was coming, so you're just posing like we're doing a photo shoot." And he just started laughing. And so the tension, at least for me, was way down.
And we had a discussion. I won't repeat everything here, but he said something dope. He said, "We were sitting around at the table and we asked ourselves, what executive did we want that got away?" That means that you still were in the minds of at least some people. He's like, "You were on that list." And we had a discussion and he was very transparent and he said, we gave you a good offer. We think we gave you a good offer. And at that time Thomas Kurian had just come from Oracle. My personal opinion, I liked his leadership at Google, but I can understand why some people were afraid of the Oracle DNA being brought to Google. And I think maybe some people in the industry felt like, oh, this is a moment to go and maybe poach a few people that didn't want to make that transition. And he said something like, we gave you an offer as if you were running away from something and we should have given you something to run towards. And I was like, damn, that's poetic. [laughter]
And so we wrapped up that meeting and I really felt like, man, I actually belong to the industry. It wasn't like I'm a Googler or Kubernetes person. I really felt after that moment that I was an industry person. So I was very comfortable at that point retiring within the next year or so.
You talk with Satya Nadella and about a year later you retired. How did that thinking come up? Because you were mentioning that you kind of had your retirement. You were thinking about your kind of endgame or exit game.
So before joining Google, I kind of felt it was going to be possible, right? Because now I'm making money. I'm saving. I really practice a whole life of minimalism. You know, I live way below my means. So even when the money changed, the lifestyle didn't. I was very conservative in terms of what I was spending. I didn't care about jewelry. I didn't care about buying cars to impress other people. That was gone. I felt free from that kind of thing in society. So the money started to become like freedom tokens. This money means I can get out of the game.
And so everyone, at least the people I know or what I attempt to do, you set this number and then when you blow past the number and if you're still a little young, you're like, "Well, maybe I can change the number." But the thing I was careful not to do was to change the lifestyle, because I watched people around me change the lifestyle, so you can make a lot of money and still be broke. And I was like, I want to avoid that. Because at that point I used to ask questions, why am I doing this? And for me personally, I felt like I was lying to myself. I love this job. And it's like, no, you learned to love it. And being someone who was good at it, you tried to figure out, working around the people you like helps you love it a little more. Working on the things that you're curious in helps you love it a little more. But you can't deny the pressures of just enterprise, stock price, you know, this person wants it now. There's a debate on whether we should do this or do that. There's personalities involved. There's parts of it you didn't love.
And then the other one I thought about was time. Everyone thinks they're going to probably live to 90 or 100, or we don't even think about it at all.
I think a lot of us just don't think about it,
right? And maybe it's not healthy to think about it, but at some point, I think around maybe 37ish, I'm thinking like,
What's the point of doing all of this work? Why are we doing this? And I used to ask that question in my job. Why am I doing this? Why am I writing Python? Well, it's because you're a software developer. It's like, that's not the answer. That's the easy, obvious answer. So I learned to just zoom out. And when I started zooming out on my career, it's like, what are you doing this for? Because once I started having a better answer when my daughter was born, I'm doing this for her. I'm doing this for my family. I want to make sure that we're all safe and protected. And so I started just changing my attitude.
So for example, when my daughter was born, I remember just taking a job where I can work overnight. I'm just going to work in a NOC. I don't really care about this job as much in terms of career progression as long as I can be home with her. So we don't have to do daycare. I'll take my shift, my wife takes her shift, but this was the priority. So then I started to structure my work life around this.
Now, I was never great at work-life balance. I won't lie. My daughter goes to sleep, I'm back on that computer trying to learn new skills. I probably approached a lot of burnout in my career.
I'm glad it's not just you.
No. Yeah, I ain't going to lie. I'm saying like, oh, and I did not have that part figured out. And also the thing about burnout, what I think is interesting is if you play professional soccer and you put in a lot of effort and you lose every year, you're going to feel burned out. I'm doing a lot of work and we never win. But the teams who actually win play more games than everyone else because they have to play the playoff semi-finals. They got to World Cup. And if you win it multiple times in a row, you play way more games than everyone else. But for some reason, the champions aren't tired.
Yeah. They're not burnt out.
Because, I mean, they probably are, but they ignore the feeling because they know what's on the other side. And so my career had a lot of winnings. So as I'm pushing the limits, you're getting the win. You're pushing the limits. So some of the burnout that is psychological, you just dial it back because, like, wow, this was worth it.
And so what the money became for me is that feedback loop of saying, you know what, let me store these things away, because here's what the math means. And I always calculated the math on interest payments, not stock increases. At some point, you get away from that. Like, I want vanilla US Treasury bond. What does that pay? So how much money do you need to live off a fraction of this money? And I started just making the calculations. And then you start negotiating salary around these particular things, and you start asking yourself, am I making an impact that is worthy and deserving of these things? So then, of course, you get into things like investments, startup stuff, blah, blah, blah. But then I'm starting to say, oh, I'm getting close. I can see it now.
Which looked impossible early.
It looked impossible. So like halfway in my career, it looked impossible. Maybe my 401k would do something, and how much does Social Security pay, and if I do everything just right. And then it was like, oh, I don't need Social Security anymore. I don't care what the stock market is doing, but I have to stay the course and be disciplined, and I have to structure my career in the way.
So once I got that light at the end of the tunnel, I started making decisions based on that. I'm going to bet to go to CoreOS instead of something that pays way more money. Ah, NASA looked great, but man, this changes the trajectory over here at Google. Plus, I'm going to have to step way up to be able to walk on that particular stage. At least that's what I told myself. But also, Google was the type of place that could pay for performance at that level. So I was like, wow, this is a good opportunity.
And so now that I spent all these years thinking about why do you work? I never had a good answer, but I never accepted the lie. And so I was like, I'm working to be me. And you become a distinguished engineer, but you realize you're a junior person. You didn't put as much work on learning how to live and the relationships and the things that you do when the computer gets turned off. You didn't put any effort into that. Not a lot. Not as much effort as you put into the work.
What parts are you talking about? Is this the kind of the friendships? Is this the outside of work? Is this the community? What?
Just developing myself. So most of my friends I've known for 20 years. We talk on the phone all the time. I see them. I fly, I see them. That stuff is important. I've been married for 20 years, going to celebrate my 20th year anniversary in a couple of months, and I felt like the core parts of my life that I wanted to be healthy and stable, I think I did those things.
But there was things where, you know, I remember going to Budapest for the first time and stayed an extra day. So it wasn't leave as soon as the conference is over. It was stay one more day. And I remember hanging out with people like Liz Rice and her team, and they were like, we're going to a bath. The, what's a bath? It's like, oh, it's like a big swimming pool. And we went to one of these huge parks where there's like, you walk in a circle and then you go in the building and there's all these plunges, you bath. Yep.
And I was like, this is not the kind of thing I do. The truth was it was the kind of thing I never did. Didn't even know. So I said, sure, I'll go. So I went there. We rented some shorts 'cause I didn't bring any swim things. It wasn't on the agenda. We were there for like three-plus hours. And I was like, man, what an experience. And so when I got back home, I didn't talk about the conference. I didn't talk about the keynote. I talked about the escape room we went to and the 3 hours we spent at the bath house.
The escape rooms are also good. Yeah.
And so I was like, what the hell was I doing? I was moving too fast through this thing. And so then I started dialing back a little bit. Hey, I got to dial back a little bit. And then little things like, as a minimalist, I always tried to live intentional. So it wasn't like the first time. I just realized there were other areas where I wasn't being as intentional. I remember when I was listening to music, I was like, hey, my wife can sing and she knows the lyrics. So sometimes I'm singing a song, she's like, "That's not how that song goes." I'm like, "I was what I remember." And now when I listen to music, I actually pull up the lyrics and I listen while reading the lyrics so I can really understand what the song is about. And it's little nuance things like that where I was like, hey, life doesn't need to be so fast.
And this is why you see me sometimes online, like, the fact that people are overindexing on productivity doesn't necessarily sit well with me, because it's like, if you just do productivity, you're going to miss everything. You're going to miss the experience. You're going to miss the parts that are hard. You're going to miss the collaboration with your team. If you just go through too fast, you're going to move right past it, because you're a human. You're not a computer. You're an actual human, and you don't work only for productivity. Maybe that's what your job believes you are. And there's that saying, you're not your HR title. And so for me, part of that was like dialing back to like, oh, you're a human. Act like one.
So I started to invest in like my relationships, talking to people, being patient, go to the school board meeting, do some of the off-site things, spend more time with your child and her games, teach her how to drive instead of going only to the driving school. Cleaning became a big part of my everyday routine.
And look, if you have enough money, you can hire someone to clean your house for you. No problem. And there's actually nothing wrong with that. But boy, I enjoyed the parts where it reminded me of all the success you've made. There are some people who don't have a stove. There's some people who don't have a refrigerator. And so when you would clean them thoroughly, like take everything out, look at all the expired food, sort everything back out, clean it back to the condition when you bought it, put it all back together. And what I noticed was, for the people who had, in my opinion, success, they could afford to do that. It is a luxury to be able to afford to go slow, because when you're really, really busy, you need an admin because you can't afford to book your own tickets. When you're really busy, you cannot afford to clean your own house. And some people would say, I make way more money having someone else clean my house than I can go make money doing everything. It's like, I promise you, money isn't everything. It is not.
And so for me, I said, wow, now I have time. 'Cause some people only have money. Now I had both. And I decided that I could actually slow down. And the thing that maybe some people noticed, even my outward projection changed. I was way more methodical. And so the work changed, the keynotes changed, because I started to incorporate the philosophy. I brought the people into the keynote. And I started asking questions like, I know what I want to show them, but then how do I want them to feel? Because sometimes I wanted people to feel excited. Sometimes I wanted people to feel a little bit embarrassed by the state of our industry and the complexity of what we added for no reason. And I noticed I was just starting to be like, the full Kelsey was starting to be on display.
So then I was like, all right, it's time to retire. What am I going to walk out to? And luckily, I was practicing just enough of like who Kelsey is to start doubling down on that as a retiree. And I didn't necessarily do a good job. I'm only 3 years into this. So I'm a junior retired person. And I make time for lots of things, but I still want to hold on to all of this knowledge, all these parts of our craft. So I'm still doing advisory, I'm doing investing, I'm still doing public speaking. I don't speak as much about low-level technology things. I do try to put a little bit of philosophy in there, but I know that I'm still holding on to that part of my ego.
Can you tell me a little bit about the advisory and the investing, both how you got started, what advising means? I think a lot of us software engineers are curious about this, and some of us will have opportunity. And also the investing part, the good, the bad, and the ugly. You can now speak freely.
Yeah. So look, I think if you're a software engineer, as you progress in your career, you will be an adviser. Because if someone wants to build something, a junior engineer does exactly what they ask. If I get this ticket, I want to do a good job. I want to get it done on time, exactly as you ask. Bugs and all. And then as you get more experience, you know that the person asking may not know what they're asking for. So you're going to be an adviser. Hey, if we did that, it's going to add a lot of complexity and you're not going to get what you want. What I think you want is this. Let me show you.
And as an advisor, you're not necessarily the person's boss. You're just trying to give them something that they don't have. So you're advising. Sometimes you become an engineer and you get really close to the executive team, but they check with you before they make any big decisions. So you're an advisor. When you start to advise at a very high level, then you also share in the outcomes, right? A lot of software engineers that work at larger tech companies, they have equity. So if you just don't focus on heads down, do your job, and you get into more of those advisory roles, then you start to realize that maybe you start to have a little bit of effect on the stock price, right? So my actions, if they turn out well, then we get there.
So, okay, so how does advisory work in the startup world? When I first started doing it, it feels like an exact waste of time. Number one, VCs have large amounts of money, so they can invest in a thousand companies and ideally one of them will return the entire fund. But when you and I do, like, angel investing or advisory, typically when I started advising, you would take advisory shares. 99.99% of the time they're worth absolutely nothing. Number one, you're going to get diluted to hell and back.
Oh yeah.
You don't even know how the taxes work on this thing. You may exercise them in the wrong way and you may end up paying money.
Yep.
And getting nothing in the end. So then you start having this allergic reaction to like advisory shares, especially if you get the ones that are way low on the totem pole. So then I'm like, you know what? I'm never going to work for free. I used to say this to myself. You can't be working for free. Can't be working for free. And the people who respect you, they'll never let you work for free. And so when someone says, "Hey, Kelsey, you want to advise?" I say, "Stop this." When I first started, it was like, I need some equity. So advisory shares, but I understood something different now. I said, look, I might need a quarter point, half a point, or a whole point depending on where you are with your funding rounds. How much risk am I taking giving you this time, and how much impact will I make? Let's say I ask for a quarter point of a company.
Cool. Quarter point being 0.25%.
Yep. Of equity. And I would say, look, you know, I used to think you can be advising for four years. That makes no sense, actually, in my opinion, from experience. Advisory, I think, is really good for like a year at a time, because advisory should have impact. It shouldn't just be like this: hey, let's talk about what we're doing, and you just give superficial advice. Like, no, it should really make an impact. So if you're going to make an impact, then you've earned the equity. And so what I started doing, instead of a four-year vesting, I would say, look, I need one year, no cliff, 10-year exercise window. I'm never losing again, not on early—
Not on early exercise.
So that worked really well. Also, you look at Carta and it's all stacking up. But I realized that I wasn't necessarily avoiding the working for free problem, because if they don't have a good exit, you still get nothing. So I started adding the retainer component. And the retainer component, I used to think about them as dividends, right? So you may give a dollar amount, it could be 1,500, it could be 3,000, could be 5,000. And you get that every month for that one year. And so in your mind, it's like, all right, I get $60,000 plus equity. And now it's like, all right, you should probably call me, because it's just a very expensive person sitting there.
I remember the first time I had a decent exit. I advised a company, Pixie Labs. They were doing observability with a small twist on Kubernetes. They were doing this observability, and what they were doing is leveraging eBPF. So that way you didn't really have to add any agents or instruments. So the way eBPF works is you almost get to the kernel level. So if two applications are talking to each other at the lower level, I can actually see that networking traffic. I can see what port it's binding to, and I can also see that it's a Go program, and I can even walk the tree like a debugger would. And so they were doing observability this way. And they had, you know, almost like, hey, we're going to compete with Datadog or something like that.
And as an adviser, I looked at it and said, look, you could try that, but most people are really not interested in changing observability stacks just for a slightly different way of doing things, even if it is eBPF. They're like, what do you think it should be? So they're in stealth mode. Their investors are like, yo, it's time to come out of stealth. We got to start, you know, getting some revenue from this thing. I said, hey, we need a few more months and we need to take another approach. So I'm sitting with the team and I'm like, you know what you have? You have like an agentless thing. And they also have this thing called Pixie Scripts that allows you to kind of make your own dashboards or aggregate your own metrics. And I started like, oh, if
you're a system administrator, you can wrap them like command line tools and you can create a thousand clusters with these Pixie scripts. And so this idea that we should pivot just a little bit, change the messaging just a little bit before we come out of stealth. And they gave a couple presentations with this agentless observability. And then we did a keynote, or we did like a Pixie day before coming out of stealth. And we showed system administrators this new vision. It went well. I did this opening keynote. I showed the vision. I interviewed the founders: why did this need to exist? Why not just Datadog? The next day they had offers.
I think one of them was from maybe VMware, but the other one was from New Relic, and they got acquired by New Relic. And I was like, "Wow, what does this mean?" And I remember the lawyer came and said, "Hey, we need to accelerate all your shares and this is the money we owe." I was like, "Whoa, this can work." But also I felt like I actually made an impact. We did pivot. And then the way it works in advisory with VCs, they like returns and word gets around. Kelsey's advisory can have impact. So other founders are coming, VCs are recommending some companies that need help.
So over time, some founders reach out, like Guillermo from Vercel reached out, like, "Hey, Kelsey." This is when they were making their pivot from just being purely serverless and front end to thinking about going a little deeper in the stack. So I spent a couple years with the Vercel team. Docker, it can go on and on, and that helps you build out a portfolio. And so I was like, you know what, stocks are cool. Remember, I started to divest from that once I had my retirement plan in place. I was like, but I do like the concept of the entrepreneurial mindset. And so helping these companies get to the next level, even for my little short amount of time, the little small impact, and then being able to share in the outcomes became a major part of my advisory work.
So my advice to anyone that wants to do advisory work: it really helps to be a domain expert, deep.
Yeah.
So if a team is about to build out their engineering team and you've been an engineering manager or a team lead, don't just go say, "Oh, when we were at Google, we did it this way." That's not what a startup needs. What a startup needs is you to say, "Listen, as you build out your team, you have to think about growth trajectory. You're going to have to think about vesting schedules. You have to think about personality types, impactful work, junior versus senior spread, when to bring in engineering leadership, when not to." That's the type of advisory that can help accelerate a startup from one stage to another stage. So then that might be your type of advisory role. So when you meet a founder, say, "Hey, listen. Here's my domain expertise. There's an impact I can have, if you think that's going to be good for you at this stage." And don't get offended if your advisory is no longer necessary, because maybe they're going to move on to something much different than what you're good at. Let them go get a new set of advisors, because you've done your part.
Yeah, I think this is the part of low ego, right?
Exactly.
One thing I really appreciated about you, especially, this was very visible after you retired, is that you took intention to understand technologies. You had some time with crypto where you went deep and tried to understand it and asked really good questions. The community response was a little bit weird. I think hostile, but I'm not a fan of that specific community. But the other thing that struck me was with GenAI as well. Of course it's everywhere now. It's impacting everyone. Everyone is using it, trying to figure out how to best use it. How have you gone about understanding GenAI, especially with your approach of, all right, one step at a time?
I'm very much a people person, all the way through and through. I'm very in tune with myself. Again, when I'm cleaning, I'm reflecting. And so this whole game to me feels like it's not about the ones and zeros. I know everyone wants to make it that way. We judge too much of society based on this. If you're a billionaire, you're automatically getting respect. If you have no money, people walk past you without a second look. And it's unfortunate that it's that way. But I really do think about it. And the weird thing is, when you say you think about people, people find that very odd. If you're mean, if you're a narcissist on Twitter, that's normal. That's expected. If you want to get over on people, if you want to game the system, that's normal. Almost expected. But when you say, I want to just be kind to other people, that feels weird. People are like, what the hell is this? No, no, no. We're all just gaming this thing.
My philosophy around technology really is this people-first situation. So when crypto came out, everyone was like, "Oh, we got these tokens, blah, blah, blah. Kelsey, it's open source, you know, blockchain." I said, "I don't doubt any of these things. I've been a part of open source movements. I've been a part of things that maybe have threatened other people's jobs. I get it. But this crypto thing, I can't help but think about the financial system in a way that impacts real people." Real people are forced to work, right?
So we go back thousands of years. You can go out into the forest, get something to eat, and that's it. Now you have to get a job because the forest is off limits. So now we force people into the cycle. This is reality. Now you're coming and saying you want to change the currency, right? Not all of them, but some of them did.
So I'm not as concerned about how blockchain, crypto, that's not as interesting. You're saying you want to change the currency. There have been countries that have gone through currency resets. This is not a nice experience, because if you had a little bit of money, a currency reset can mean you now have no money. If you had no money, it may feel like it's impossible to ever get any more money. If you're retired, what do you do during a currency reset? You don't have any way of making new money. You're retired now. So these are very, very real things that I thought that group of technologists were ignoring because money go up. And so when I would have those conversations with them, they wanted to debate the low levels of crypto and how transactions are settled and things about encryption, the blockchain. I was like, "Yo, that stuff is fine. We can go back and forth, but at some point we have to talk about how it impacts actual people."
When GenAI comes out, I'm now super into the philosophy of everything. So there's one part of this that's a little personal. Of course, we spent all of our careers learning all these skills, training our own models, right? We learn to program.
Yeah, we learn to program. And it's hard.
Well, not just hard, but there's aesthetics to this.
It's not just blindly typing code.
There's an art form to it, right? This is why we have Ruby. And when you read about Ruby and Matz's vision for having something that can be almost romantic to write. Perl has its own subculture. Go has its own culture. So we're not just writing code. And my entire career, I always thought about writing code as decision-making. So before we do anything, we all figure out what needs to happen, and then we have to convince the computer to do it. And every keyword, every if statement, every function call is a decision we're making. And of course, the syntax kind of gets in the way from time to time. So Stack Overflow we go.
So GenAI comes out. And early stages, it's kind of like people just talking to the machine. And I've never been impressed by talking to a computer. I'd rather talk to real people. So I don't really care too much about that part. Yes, it mimics human capabilities. For people that want to talk to a computer, knock yourselves out. But then we get into the code generation piece. Now we're back to where I was with the crypto stuff. I've used the compiler. It generates a lot of code for me. I've been doing that for a very long time. I haven't written any machine code.
Now, I post things like, "Hey, I'm adopting the zero token architecture." People are like, "What's zero token architecture?" It's like, instead of burning tokens, you learn things and you think for yourself and just complete tasks. And they're like, whoa, why would you want to do that? It's like, because we taught the machines. I don't know why people skip this step. "Hey, Kelsey, there's going to be this artificial intelligence, going to do all the things." It's like, but we trained it. So all those times I'm writing code, the books I've published, the comments back and forth on helping people solve problems, it's all in there. Maybe it's arguable that they have their own worldview based on that, and maybe it's slightly different, but I can never put the machine over a person under any circumstance.
And I think there's a subset, I don't want to say everyone in the space is doing this, but there is a healthy subset of people who really believe: what is the purpose of a person? Why do we need them to write code? Why do we need them to build software? It's like, maybe you don't understand what the job has always been. We are trying to solve human problems, and we use whatever technology is required. In some cases, the technology happens to be software, and software ain't required for every human endeavor. And I think a lot of people are just in this bubble where they believe software is the only way to solve any problem. And then they think GenAI solves all human problems.
Yeah.
And this is where I start to push back on that narrative. It's like, you learned to love this job and you forgot what the rest of the world is doing. And so I feel like some people are now trapped. And that person, again, I keep going back to that person walking into the industry for the first time. Luckily for me, they were looking for people that had skills, and there was a pathway. So many pathways for us. Now the new generation that's coming out, they're unsure of themselves. "Hey, I'm watching the news. You guys keep celebrating people just using GenAI to do everything. What am I going to do?" And I just can't accept the answer being, you're just going to come in here and use GenAI to do everything, and all of you are now just the same.
Okay. So this is the one that you cannot accept. But of course, through advising, through meeting people, through talking to people, what are some of the promising parts that you might see, or even parallels to previous technology revolutions? May that be Kubernetes, or may that be, as you know, we were moving to a lot more powerful computers, like when you were coming out.
So the way I think about it: for example, if I'm doing due diligence for the fund that I do due diligence for, that means before you make that decision to write a check and explain to our LPs why we took this position, we need to do a little due diligence. And the way I do due diligence, I want to meet the founder, I would like them to walk me through the particular product. And I go one step deeper. Let's look at the code. Let's look at your Amazon bill. Let's look at the architecture. Let's look at GitHub. How do you manage issues? How do you all work together? I want to get a sense for the team, the product, and its trajectory.
And when AI is involved, the one thing I just do before the thing kicks off in this meeting: do not say AI. Because what we don't want to do is use a big umbrella to describe what you're doing. Let's get concrete details. These are computers. These are computer programs. Yes, just like when I saw a regular expression for the first time, it's a different way of thinking about software than imperative things, if, else, then. So I get that, but now you have to show me what you're actually doing. So when we do that, when I put that handicap in place, now they're forced to show me the problem they're solving. They don't just say, "Hey, AI for healthcare." Nope. Show me exactly what you're doing.
And so with that handicap in place, the really good founders, really good technologists, what they do is they say, "Hey, here's our problem, and here's how the industry currently solves the problem, and here's the drawbacks from that." And since they can't say AI, they can't say agentic, they just have to show me how they make the problem better. Now, as a technologist, I know that if I gave you a random PDF, there is no easy way for you to procedurally program parsing PDFs. You just can never anticipate everything you will ever see. So it makes sense to use OCR, or if you don't know what OCR is, you might say, I will go use Claude to do it. Whatever.
There's probably an easier way to do this than using a large language model.
There are smaller models, smaller techniques. So my advice to them would be, you know, you don't need to use an LLM for this. There are smaller models or smaller AI techniques that are not GenAI. So they're like, oh, good feedback, we can probably lower cost here. But then when they take these technologies and they do something novel with it, I'm like, you know what, that is a good product for the people doing this work. And I'll say, you see that you did that without saying AI. So on your website, why are you burying all the value of this platform by putting AI in big letters before we get to the value? And so maybe they do change the website to actually talk about the value. When they do the demos, they start with the value versus handwavy text boxes where they put in a prompt and it does something magic. Physicians don't want to do that. They want to see some of the other contextual things and leverage AI to bridge the gap between what they're currently doing and now what's possible.
So my advice to people that I'm advising, you know, all the startups: "I got to make an AI pivot." I said, "If you all pivot to AI, then you all will have a problem." It's like kids learning to play soccer. They all run to the ball. No strategy. Spread out, figure out where you add value, and play your position. So when I'm advising a startup: what is your position in this big landscape? You can't all run to the AI ball. You got to stand back and figure out what value you're going to add.
So one of the companies I advise, they're called Massdriver. They have a visual kind of infrastructure as code. You take Terraform, you give it some metadata, and then you can interact with it visually. And part of that visual interaction allows you to do things like, this app needs this database. And just drawing that line
allows the credentials to flow to the other app under the hood, and now you have a config.
Awesome.
So now Claude is the big thing. Everyone's like, "No, I don't need that. I don't need Terraform. I'm just going to use Claude to manage the cloud." Now, someone with experience, I'm like, "This is about to be real fun, because I've seen what humans do when you just give them the AWS console. Watch what Claude's going to do when you give it the AWS console." And so, knowing this, it's like, okay, here's how we can add value to this.
Number one, we can take the things that you have in this visual box and split it up into value props. Number one, in order to show things visually, you have to have context. So let's call it the context engine. And then that context engine can be queried by Claude. So instead of pointing at AWS, it's now pointed at only the resources you use. And for people to understand why this is important, if you take some of these agents, they just start investigating the console like, ooh, what's Lambda?
Nah, don't need that. But Lambda now is running. What's this load balancer? Oh, don't need that. But now that could be running, and you don't even know the mess that it made.
Yeah, they don't even know the mess that it made.
Yeah, it doesn't even know the messes it made. So now we say guardrails, or give it context. So I said, "Oh, okay. So instead of doing a full-on pivot in a naive way, let's reposition here." So now some of the features we have become guardrails. Some of the things, the way Massdriver does things with IaC or
Infrastructure as code can become skills. So now if you bring Claude to the scenario instead of starting from scratch, we can just allow the agent to interact with this platform the same way humans do. And then what we end up with is if a human wants to interact visually, it works. If you just want to have some automation in your pipeline, then just call the APIs and you still can interact with the context and the deployment engines. But if you really just want to use Claude Code, then now Claude can interact with the same guardrails and structure that the rest of your system does. And then Claude becomes a little clear. We're not asking it to be magic. We're asking it to be an alternative interface for getting something done. Skills.md becomes implementation detail. And then when we do a webinar where we present this, just watch the light bulbs go off for the people watching and for the team.
And that to me is the type of advisory where you can look at GenAI. I'm not just a GenAI hater. I just don't like the naive promotion and adoption of it. I think it should be way more strategic. And since I think about GenAI as a tool versus the great human replacement, then I can use it in way more pragmatic ways.
Thinking about it as a tool, what capabilities do you think it gives us software engineers specifically, and also maybe what are some areas where it can maybe give some overconfidence?
So, one thing I've tried to do is to be a little bit more positive in my thinking because it would be very easy for me to go down the rabbit hole of identifying all of its flaws, right? Like, oh, look, it makes mistakes or it hallucinates sometimes or it overly confidently gives me a config that then blows up production. I'm going to assume those will get slightly better over time or human in the loop will catch those things. So for the sake of this discussion, I'm going to give it a little grace.
The things that I think it should do well that I think don't get used often enough: there's this concept where people really think it makes sense to do inference every single time. For example, if a human writes a piece of code, we'll write it once, right? Let's say I wanted to authenticate to an endpoint. So I will call the code. I'll go look at the documentation and go to Stack Overflow, figure out the example. I'll call it once. If I have to do this twice, it's probably going to become a function that I call multiple times in the codebase. If I do this like five to 10 times, this may become a library that I import in multiple apps. So, that tends to be the human flow because there's no need to infer or write from scratch this particular thing. And we've done that. That's where open source comes into play. And when we're really doing a good job, these things get just baked into the framework and they're just there.
Sometimes even the OS.
In the OS, and then sometimes things like encryption find their way all the way to the hardware, right? We do offloading. So this has been the loop that software engineers go through for a long time. GenAI to me should be no different. So if you find yourself generating the same blocks of code over and over, even though it feels fast or convenient, it is still insane.
At some point you should say, well, why is Claude generating the same block, copying things everywhere? Because we know where this leads.
Yeah, we've seen that. Of course we do.
Yeah. And then people get excited, well, Claude Code can refactor all of it. But that's a waste of energy. There's no reason to do it just because it can. So, I hope what developers are really realizing, what I realized myself from looking at this, the number one thing I realized is that most of our APIs were designed incorrectly even for humans. Right? Now a lot of our APIs, think about infrastructure. You have to call like seven APIs to get a VM in the cloud. Create a VM, a network, a storage device connected to a VPC and then attach credentials. That's not intent-based. I want a VM. So the first thing you kind of see from this movement is things like MCP where we wrap these imperative calls into an intent-based thing that says create a VM and then that reflects out to the other ones. But here's the thing. I saw this before that. I saw this with Kubernetes. Kubernetes does the exact same thing. When you say give me an ingress or a service in Kubernetes, that reflects out to seven calls too to create load balancers, SSL certificates, DNS setup.
It's just this high-level interface, right.
Exactly. So I've seen this before. So I'm looking at this like, guys, the fundamentals of this isn't like everything is going to change because of MCP. What the hell? A whole conference, guys? This is just API design. So in this particular case, I hope developers realized that we shouldn't have fought so hard, RPC versus REST. There was this big tug-of-war around composable APIs.
2000s.
Yeah. And a lot of people went down this rabbit hole of like create VM. Oh, it's not flexible enough. It's like, so what? Create VM v1, create VM v2, who cares? Because these are intent-based APIs. So I think these new tools are reminding us of this. When I look at the way people are prompting and how they write their prompts, a lot of our programming languages were too rigid to express what you really wanted to happen. We were so afraid. Maybe Ruby did a better job than some other languages where they try to give you things like until. So that way it just flowed better as you were writing the code. Other languages are a bit more rigid. It's like, what does FN mean? So you have to go look it up every single time. So writing code becomes very laborious. So when people write a natural language prompt, it kind of tells us a lot about the intentions of querying a database. So I'm hoping developers learn better API design from that front.
The other part was, I still get frustrated. I want to learn new technology. I go to the website. There's a little bit of documentation, but for some reason developers still write documentation as hints, as clues, right? I'm trying to learn a new programming language. I won't throw any of them under the bus. And I go there and I'm just going to go look at the standard library. First thing I like to do sometimes is just, I'm going to parse a JSON file, get a feel for the language, and there's a JSON library. Sweet. We're off to a good start. Click. And you look at the documentation, you just see function calls. I'm like, can I just see a working example? What do I import? What do I put in this thing? What comes out? And then maybe what do the errors look like? I just want to see a full example. They're like, "No, you just get reference." So then what do we do next? Then we search the internet and you might land on Stack Overflow, someone's blog that gives you a full example. So now I got to copy and paste it and see what this thing does and hopefully it's up to date with the actual documentation. This has been the loop we've been going through for so many years. So when I use things like Claude or various tools, they close that loop. You get the example right here.
And let's not pretend that Stack Overflow examples were perfect. They were not. So to me, having something where the tool would then try to build it to tell me if this is correct syntax, or thinking to make sure that it's giving me a good suggestion, even I can say that is an improvement. But I hope we learn that maybe we should not just give hints in the official documentation. I'm watching people write these huge markdown files to give the agent context. How about you write documentation to give me context so I can have fully working examples.
And now with agents, we could actually generate documentation a lot more intentionally. As devs, most of us just don't like, I'm not sure if there's anyone who likes writing documentation. We don't like doing tests. We don't like writing tests in general. These tools can help with that.
But this is where I think we tend to make a mistake. I remember when I was learning Java for the first time and the Java developers were like, we don't write docs, the code is self-documenting. I was like, "No, it's not. It's just documenting hints. I still have no context." Because to me, again, software development is a human endeavor assisted by tools. And so there's a style to documentation. There's a personality to documentation. It's like writing a movie. You're trying to educate a person. I don't want just hints. Say, "Hey, this thing exists for these reasons. We conform to this particular specification. There are multiple ways to write this code. Here's the most popular way. Here's what bad code looks like. Here's what high performance code looks like. Here's when to use this library. Here's when not to use this library." I want that kind of in-depth as I'm training my own model. And what I'm seeing now, which I think is a good thing, is people are writing a lot more documentation to be consumed by the agent to give it context. I was like, man, I wish we had the same motivation just a decade ago because I think a lot of us would have been way more productive if we didn't have to do a wild goose chase every time.
Well, interestingly enough, Kat Cosgrove told me that one of the reasons she thinks Kubernetes won was documentation. They take it extremely seriously. Few other projects, if any, do it at that level. So, just proving a point.
One thing that a lot of experienced software engineers are worried about right now is all the AI. We're all using AI agents. They do generate code really quickly, which is something, writing code used to take a lot of effort. It took a long time to be good at it. Now with code reviews, there's now more tools coming in and some people are worried like, okay, what is happening to my profession? The craft of writing code seems to be something that we can offload, and more and more people are offloading, including prominent people. What will this do to software engineering, and what advice would you give to people who are experienced software engineers? They're a little bit worried because it's a big shift. They still want to, you know, be whatever a great engineer will look like in the future, but what steps might they be able to take? And especially your take, because I think you're pretty grounded in just looking at this from a vantage point that some of us are not.
I think as a software developer, the first step you have to do is have a bit of reflection. For the last 20, 30 years, you have been automating a lot of industries away yourself. All those programs. I remember seeing, maybe it was a diagram of every device that has been replaced by the iPhone. The radio, the calculator, the compass, all of these tools people used to buy individually. The top 30 of those electronics from the last 40 years are all in your iPhone. All of them. That means some electronics makers have gone out of business. They're gone. You did that. Not in a malicious way, but you were part of that. And so the software developer has been glorified for a very long time. The internet, some people would say, caused the downfall of magazines and newspapers because of the convenience of having a software approach.
And so you have been part of the change to other industries and other people yourself. What did you think about that? Did you even think about it at all? So let's not be surprised if you find no sympathy from all the other professions that you've helped force change upon. So I think that's step one. You have to go do that reflection because if you don't do that reflection, you won't know how to behave now. You're going to be complaining and people are going to look at you crazy because where was this empathy before? You might be very excited about this and not realize you're only excited because you're in position to benefit from this. So if you work at an Anthropic, of course this is the future because it's in your hands. If you're at Nvidia, of course this is the future because you will be selling the picks and shovels. And so you got to ask yourself, why am I excited? Now what I don't want you to do is necessarily feel guilty about it, but I need you to see the big picture. It's going to help frame everything else.
The second thing I want you to do is ask yourself, what was my job? Remember, there was a point in my career where I was lying to myself. I thought my job was to be the last best Linux administrator ever. You as a software developer, you may have thought your job was to be the only person in the organization that can write code. And since no one else could do it, you were safe, right? And so you didn't learn any other skills: networking, product management, design, talking to customers. Nope. All you had to do was write code and you were safe. And you probably made more money than everybody and you were fine with that. Now you got caught. The only thing you were good at has now been commoditized. And again, you did this to others. So let's say the vision you have for yourself is only in this very narrow realm. You're going to be very afraid of this trajectory because all you know is software developers write code. That's it. Some software developers still don't write tests, still don't know how to deploy anything. And so they are really afraid because they can't see any other way that this plays out.
Now if you're a full stack engineer, you're probably like, man, there's so much more than just writing code. You have to do architecture. You have to do design. You have to do so many other things that I love Claude because now I can focus on those things and I can use these tools instead. So I can see why that person would have that perspective. Now, I understand why that full stack person has a perspective of watching the same people commentate that the code generation piece replaces everything else. They're going to be like, "No, you don't know what this job is." It's way more than just writing code. Writing code is the last step.
If you're a security engineer, you're probably like, "We never figured out security for the pace of the current enterprise."
The one before.
Yeah. Everyone thinks they're moving slow. I remember I took a security training thing, and most of them aren't that good because they can't go super deep. They just tell you, "Hey, here's how to avoid phishing. Here's how to not leak information, adhere to various laws and things like that." And then they said one thing this time that I learned that was pretty good. They're like, "What's the key to protecting yourself?" And they say, "You know what? Just go slow." A lot of attacks are: I'm about to board a flight, and let's say you're an admin or you're a VP and the CEO texts you right now: "We need to wire the money to Oracle to pay for the license. They're going to cut it off right now. This needs to be done immediately." You look at your phone, it's definitely from the phone number of the CEO. You have a good relationship. Everything looks right and it's moving fast. So, you're like, "Man, I'm on a 10-hour flight. I need to do this now." Turns out the attacker knows you're about to board this flight. They've seen all your previous text messages. They know how your manager talks to you. They know that you've moved fast in the past. And so they now are primed to get you to do the exact same behavior again. And you could be the VP of security. So you should know better. And sometimes that naive confidence will make you feel like, I'm obviously not being phished. I've done this many times. This is definitely the CEO. Who would know how we actually operate? And who would know that I can actually do that? So what do you do? You make the transfer. And just like that, 10 million has been wired to the wrong place because you moved fast. It wasn't because you were not smart. It wasn't because you were not productive, because in this case, you were productive, but you did the wrong thing.
So when I think about code, there is value in having a healthy pace. Let's say you're an insurance company. You sell insurance. Hey, make, model. How old are you? Have you had any
accidents? Okay, here is your insurance for the year. Simple, very simple thing. If you're an insurance company, that's all you are. You're kind of close to being done. Now, you could say with GenAI, we should get into payments. We should compete with DoorDash, right? We have all these tools. Let's go and
We could build it.
Yeah, we can build it. So, but the thing is, should you just build it because you can? And the answer is typically no. So we usually optimize ourselves as humans around the pace needed for the task, and when we don't need to do that work anymore, we move on to something else.
So now I think what we're going to end up with is people not realizing a lot of this stuff we were doing in software engineering was decision-making: what database to use, what schema, should we really collect someone's social security number or should we avoid it? Not "yeah, I can write code to parse a social security number really fast." Like, no, no, should you even do it?
And so when you write code, it almost makes you slow down again, because there's been times where I thought I had a good design. That's that phrase, writing is thinking. So is writing code. So as you're writing the code, you're like, "Hey, wait a minute. This loop is ridiculous, right? Not only is it going to make the computer warm, this is not the right thing to do. There's a better data structure than the algorithm that I'm using."
So then you stop and say, "Hey, the data structure is wrong. We need to change the way we print receipts on the cash register. Sure, I can write this code, but this is the wrong data structure. While I can generate the code, doing reports are going to be a nightmare. Summarizing this data downstream is going to be a nightmare. Stop everything. Now that I've thought about it, we need to change the architecture from the top down."
So decision-making sometimes does benefit from slow. And when I'm saying slow here, we're not talking waterfall six months.
Yeah. No,
we're just talking about maybe one more day before you go at it. And I think some of us are going to miss that part because Claude spit it out, ship it.
Yeah. And that's also one thing that you always have: the more experienced generation be worried about the young generation. I remember when I joined the industry, ReSharper had come out. ReSharper. And the experienced old guard was like, "Nah, you're not a real developer if you use ReSharper, cuz you know you're not going to learn the library, and you need that, and, you know, like, that's what makes you a real developer." And then I remember when I was now five-plus years of experience and Stack Overflow started to become big, and I was like, nah, you don't want to go to Stack Overflow because you're not going to, you know, learn the real thing.
So, but now what the current old guard is saying, which is, you know, I guess I'm part of it, is like, well, if you use AI, you're going to miss learning the basics, and when you have learned the basics, it's so much easier to use AI. And I wonder if we're just repeating the same mistake as the previous ones did, which is the new generation usually figures out the tools, they understand how to do it. Or are we rightly concerned that some people who are coming into this AI native, they're now learning to code, they can jump through so many layers that they will just not, you know, see what's under, understand? Or are we just making assumptions that might not be true?
Here's where I think it can be right on both sides. Do you need to learn how to code to make an impact in this industry? The answer is no. You do not have to. There are some people who use these no-code platforms where they drag and drop and they produce a really good app. There are some people who have built a consultancy business by just using Wix, right? They go there, their website actually looks pretty good, and so they got really far with that. Now, for what they're trying to do and accomplish in life, they'll probably be fine.
But let's just say you are a software engineer. And the idea behind software engineer is not limited to just producing apps. Software is the interface between hardware and things people want to do. So there's a whole bunch of things you need to learn. So if you want to be that type of software engineer, you got to learn hardware, too. If you don't understand hardware, you can never work at that level. And look, if that's not your job, then so be it. But you will never have that creativity.
I remember seeing someone was like, "Hey, you can do isolation without a VM." I was like, "How would you do that?" He's like, "Oh, because when you boot the kernel, there's a thing you can do before the kernel loads to isolate it in a way that you can lock down processes." The only reason why this person knows is they know the full boot sequence from firmware to switching to the kernel and the tricks you can do in between. Now, for me, that doesn't work at that depth. I'm thinking there's only virtualization, CPU isolation, things like gVisor where you intercept system calls. But never did I think about the boot sequence.
And so yes, you can get very far, but as someone like, we applaud every version of Opus that's released, or ChatGPT, but there are versions of yourself that get deeper from these new trainings. So no, you don't have to. But if you ever want to get better at anything, and sometimes that depth, that nuance, is the thing that leads to an invention, right? If you know how a compiler works, if you know how memory management works, that might give you enough information to say, "Oh, I can make a new programming language." If the only thing you know is the surface, you can't even imagine how you can create another programming language that is a better fit for the task at hand, because you've never gone that deep.
I'm not saying everyone wants to do that. So I think it is fair to say, all I want to do is come in, get a job, and if that job can be done by using AI tools, I think the side effect of that is then that job will be commoditized. It has to; that's just the way it's going to go. But I've always seen myself, for my entire career, I want to learn more. I want to go deeper. I want to go so deep that I can create.
And I think a lot of people who are doing this, the reason why we're having this reaction, some of us, part of our careers have been the creation part. There is no spec for this. There's no protocol for this, but we're going to make it work. A lot of people that are doing, like, the reverse engineering, the hacking, there's no framework for what I'm about to do. I just know how memory works, and I don't care what your security tool does. I will make it do what I want it to do. You need to go way below the surface.
And so for a lot of us that are saying this, we know the value of the fundamentals that lead to the other stuff. And so if you tell a next generation, oh, you don't need to learn these things, it's like, that may be right in the short term, but we know for a fact your career will be limited, and that may not be a problem, and you have to decide. But make no mistake, if we put this much effort in training the model so that it can spit things out, you better make sure that you are willing to train your own model.
So my advice to people would be, and maybe we should talk about it differently. Maybe we shouldn't have so much fear-mongering around it. Maybe we shouldn't put it as a versus this. We should just say great artists tend to know how to mix colors, and it is to your benefit to understand the primary colors so you can mix them to get the other colors. And it's a superpower, right? So you don't have to go buy, you know, imagine an artist trying to go buy 16 million colors and put them on the desk because they don't know how to mix colors. If you teach a person how to mix colors, you can get any color you want. And I think that's the way we have to approach it. It's just another skill that if you had it, you might just unlock some creativity. So I encourage you to learn it.
Kelsey, thank you very much. This was just an amazing conversation.
Awesome. Thanks for having me.
I will admit I was glued to my chair for the whole of the conversation. Apologies that it took this long, but I hope you agree that this specific one was worth it. Kelsey's path was just so unlikely. He's someone who was raised by a single mother, dropped out of college in favor of installing DSL lines door-to-door at 19, and still ended up as a Distinguished Engineer at Google Cloud. Only a few hundred people hold the title at the company. And he retired at the top three years ago when he decided that he no longer needs to work for others. What an inspiration.
One thing I took notes on was when Kelsey said how every job is an interview. When Kelsey was giving the GopherCon talk and PXE booting CoreOS from his slide deck, he had no idea that the CoreOS team was sitting in the audience, and that's how he ended up at CoreOS. When he was contributing to Puppet at nights and weekends, he didn't know that James Turnbull would walk into his office one day and recognize his name. He just kept showing up and doing the work in public. It's a lesson worth remembering. Do the best work you can at work. It might unknowingly be your job interview for the next step in your career.
I also found the Microsoft offer fascinating. Kelsey did not use the offer from Microsoft as an ultimatum at Google, even though he could have. He just told his manager the truth, and then Google matched the offer. Obviously, at this high of a level, there's no universal compensation negotiating advice that always works. But being a straight shooter with high integrity is something that is good to keep in mind.
I was also inspired by Kelsey's focus on minimalism. He treats money as freedom tokens and made sure that his lifestyle never inflated with his salary, so that early retirement was always a real option, not just fantasy.
Finally, Kelsey's AI take: his point is grounded and pragmatic. AI does not change what software engineering is actually for. The job was never just to write code. The job was and is to solve human problems. The engineers who understand this are going to be fine.
Do check out the show notes below for related The Pragmatic Engineer deep dives on Kubernetes and other related topics. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show.
Article published