How Uber's Pre-AI Developer Infrastructure Became the Foundation for Its AI Agents
The Pragmatic Engineer
What internal tools do engineers at a company with thousands of engineers and thousands of services actually use to build software with AI agents? In this episode of The Pragmatic Engineer, the host, a former Uber engineer, talks with Uday Kiran Medisetty, one of Uber's longest-tenured engineers and a Distinguished Engineer who now leads much of the company's agentic AI work. Uday walks through roughly two dozen internal systems, many of them shown on screen. The thread running through the conversation is that the coding harness is the easy part. In Uday's account, what made agents useful at Uber was a decade of investment in developer platforms built long before generative AI: cloud dev environments, standardized builds, API registries, distributed tracing, and test-routing infrastructure.
From Two Services to Thousands
Uday joined Uber in May 2015 on the Real-Time API (RT API) team. At that point, they recall, Uber ran essentially two services. "Dispatch," written in Node.js, handled all rider and driver traffic using a simple four-second ping protocol that returned the full state of the world. "API," written in Python, took over once a trip was completed and handled post-trip processing such as payments and receipts. RT API was created to abstract internal services away from the mobile apps and to replace the four-second ping with REST APIs plus a server-to-client "push protocol." Because the app's state changes every few seconds as things happen in the physical world, Uber needed an efficient way to update riders and drivers. RT API was one of the highest-QPS services at the time. It was later rewritten in Go and, according to Uday, is still one of Uber's highest-QPS services.
The microservices boom came out of hypergrowth. Uber was constantly entering new countries and adding products, including Uber Eats, and teams needed to build functionality quickly. Uber invested heavily in service infrastructure for metrics, tracing, monitoring, observability, and deployment. Uday says little off-the-shelf software could handle that scale and rate of change, so much of it was built in-house. The company went from two services to the order of thousands. Around 2018–2019 Uber standardized on Go and Java as its primary languages, modernized its stack, and thought harder about architectural layers and platformization. On one fulfillment platform Uday worked on, the team actually consolidated services. Uday says Uber today runs in a similar ballpark of service count even though the business is about ten times larger. A single global tech stack serves every country, and shared building blocks serve all mobility and delivery verticals.
On headcount, Uday was roughly the 600th engineer, and the whole marketplace team was perhaps 50–60 engineers. Engineering grew to a few thousand. From around 2019–2020, Uber expanded to 12 global tech sites across almost every continent where it operates. Uday says this lets teams build products for hyperlocal markets, some of which are later rolled out globally. For their first ten years Uday worked on business platforms: fulfillment, Uber for Business, earner onboarding, and marketplace pricing. For the past year they have focused on agentic AI for engineers and the rest of the company, along with some user-facing AI work.
The Pre-AI Tooling Layer
Uday lists several systems Uber had to invent because nothing on the market fit its scale:
- uOwn, which identifies who owns any asset. During an outage, for example, it tells you which team owns a service and who is on call.
- Centralized design docs and incident management. Product and engineering design docs live in one place, and so does incident tooling. Uday describes Uber as "really obsessed" with how it handles incidents and learns from them.
- M3DB, the metrics system. Uday estimates that around 2015–2016 Uber was already emitting about 500 million metrics per second with high cardinality. M3DB was later open-sourced, and companies have formed around it.
- Michelangelo, the end-to-end ML platform covering feature creation, model building, monitoring, and serving for hundreds of models. Uday notes that parts of its deep learning infrastructure were open-sourced a couple of months before the recording.
- Cadence, a durable workflow engine for processes that span days, such as driver onboarding or CRM campaigns. Before it existed, engineers kept building their own resiliency and state management. Cadence was open-sourced, and the host notes that Temporal drew heavily on its ideas.
- Morpheus, the configuration, feature-flag, and experimentation platform. Uday calls Uber a very data-driven company where "everything goes through a feature flag, everything goes through an XP." Because Uber runs two- and three-sided marketplaces, simple A/B tests are often not enough, so the platform supports switchback experiments and automatically generates analyses such as p-values and metric choices. That in turn required investment in trustworthy metrics and data foundations.
The host remembers wondering why Uber needed its own feature-flag system. The answer turned out to be that no vendor treated cities as first-class citizens. Uday confirms that cities are deeply baked into the infrastructure. Some critical platform services shard by city. A city is pinned to a specific region and can be failed over to another region at any time. For some services, "logical deployment groups" route groups of cities to groups of containers. Uber uses this for services that need that level of isolation and reliability, not for all of them.
Uday's summary of the pattern: in the early years Uber had to invent a lot. As the industry matured, Uber now buys where it can, builds where it sees a unique need, and tries to open-source what might interest the rest of the industry.
DevPods: Cloud Dev Environments That Agents Inherited
The host remembers DevPods, built around 2017, as ahead of the industry. Uday explains the backstory. In the early microservices era every service had its own repository, and teams used two or three build systems and two or three test systems. Library upgrades across services were very hard, even for a central team. Around 2017–2018 Uber moved to one monorepo per language (Go, Java, iOS, Android, and so on), while still deploying services independently. Once the monorepos grew large, building them on a laptop and getting through local setup became painful.
DevPods are cloud developer boxes, one flavor per monorepo, and an engineer can have any number of them. They come with build caches, run on machines beefier than laptops, have the right tools preinstalled, and let Uber set SLAs on build and test times. In a live demo, Uday shows that a new DevPod spins up within seconds. Uber runs them on Kubernetes clusters across multiple clouds, keeps them warm, and clones them from prebuilt images that already include repositories and build caches. Bazel is the build system across all repositories, so one command builds any module.
Uday says this foundation paid off in the AI era because agents need the same kind of workspace. Uber created a "mega DevPod" containing all repositories, so agents can read context and open PRs across repos. Standardized building and testing meant Uber could put AGENTS.md and CLAUDE.md files at the top level, and most teams got value immediately without per-module setup. Claude Code, Codex, OpenCode, and other AI tools come preinstalled, so even non-engineers can create a DevPod and start experimenting.
Model Gateway and AI Guard: One Entry Point for Every Model
Uday describes the guiding principle for Uber's AI foundations. Models and harnesses change quickly, so Uber invested in building blocks that are model- and harness-agnostic. The first of these was the model gateway, started soon after ChatGPT launched. Uday first said 2013–2014 and corrected it to late 2023 and early 2024. It is built on Michelangelo, which now has a GenAI set of components, some of which are not yet in the open-source version.
A team creates a GenAI project in Michelangelo, picks models, and gets a project UUID. The client passes that ID and a caller token. The gateway exposes one OpenAI-compatible or Anthropic-compatible API, and all models from all vendors, open-weight or closed, sit behind it. Uday stresses that every model request at Uber goes through this gateway.
The first middleware added was AI Guard, intended to make sure no sensitive content is sent back to users. Because there is a single entry point, the security team can iterate on its own models. Uday mentions about five fine-tuned models covering content, safety policy, and communications policy, which run on every request and response under strict latency guardrails. Centralizing traffic also gave Uber per-user and per-caller attribution, spend tiers for projects and users, audit logging, session traces, and tracking of billing and cache-hit rates. Uday sees it as a natural place to add model routing in the future.
AIFX: Making Harnesses Interchangeable
The next question was how to let engineers switch harnesses, for example from Claude Code to Codex, without losing their settings, skills, tools, and plugins. Uber wraps all harnesses in a tool called AIFX ("AI foundations"). It is preinstalled on DevPods and can be installed locally. It has a concept of "personas" that ship with default plugins and skills, so engineers get useful defaults without installing anything. Settings are centrally managed and synced across a user's local machine and all their DevPods. A status line shows tips and tracks spend within the current session, within a harness, and across harnesses.
How Uber handles token spend
Asked whether a new senior engineer needs to worry about cost, Uday describes a tiered system. Everyone starts at a default spend tier. When they approach the limit, they can request the next tier with a simple manager approval, which prompts a conversation about ROI and responsible use. Uday says Uber does not want to restrict engineers who are spending on the right projects, because the goal is to maximize the gain, so it keeps the paperwork minimal. Interactive harnesses and background coding agents have separate tiers. A plugin called cost check analyzes an engineer's sessions and gives personalized tips. Uday says this has helped with real-time coaching on context health and model sizing and encourages useful conversations between engineers and managers.
MCP Gateway: Every Internal Service as a Tool
Around early 2025 Uber enabled Cursor, and MCP was emerging as a spec. Uber realized that exposing internal tools to AI could extend it well beyond coding. In April 2025 it started building an MCP gateway, described as "one gateway for all of the tools."
The key enabler was a pre-existing registry of IDLs for all services, defined in Thrift or Protobuf. Older services use Thrift, newer ones use Protobuf, and teams usually switch to Protobuf during rewrites. From that registry, Uber auto-generates a default MCP server for every service, with tools derived from existing APIs, including request and response specs and descriptions. Uber later used the API descriptions to generate LLM-optimized descriptions of APIs and fields. Uday says some auto-generated tools were 3,000–4,000 tokens, so this optimization mattered. Enabling a service takes one config and security-policy change, with no code changes. Teams can also build custom MCP servers with an internal framework and proxy them through the gateway.
Vendor tools such as Google Workspace and Slack are exposed through the same gateway, so internal and external tools install the same way through AIFX. In recent months Uber also projected every MCP server as a CLI, letting harnesses choose between MCP and CLI usage. Uday credits the speed of the build to the fact that the API gateway team, already expert in protocol transformation, also built the MCP gateway. Once Google Workspace and Slack were included, Uday says, people could automate almost anything. Slack, Google, Jira, Linear, and any internal service became "a CLI call away."
The Skills Monorepo
Early in the year of the recording, Uber created a skills monorepo. It contains core skills built by central platform teams and team-specific skills built by individual teams, packaged as plugins and skills that work across all harnesses. A single set of quality gates runs across them. Installing a plugin through AIFX updates the configuration of every harness, whether the engineer is currently using it or not. Because all skills use tools from the MCP gateway, there is one standard way for skills to access any tool.
The host asks why this is more complex than a Git repo of skill folders. Uday's answer: with current models, a v0 skill is easy to write, but making it good across all use cases is hard, so central management matters. He gives an example. When CLI projections of MCP servers became available, a couple of engineers used AI to update every skill in the repo to the CLI pattern. Uday says this is more token-efficient: if two plugins each declare three MCP servers, all six sit in context permanently. The marketplace is auto-installed for everyone, which helps discoverability and lets Uber build eval and discoverability pipelines around it. Some teams still keep their own repositories, but most skills go through this one.
Work still in progress includes a tool that gives engineers tips on improving their skills and a pipeline that collects harness traces and generates improvement patches for skills. Uday reports about 30,000 skill executions per day in recent weeks.
Context Graph and Cortana
Uday calls the context graph their favorite project. It grew out of Cortana, an AI assistant an engineer built in December and available on the web and in Slack, which can answer questions about any codebase by searching across repositories. People used it for research, query help, finding data sources, ownership questions, and root-cause analysis. Looking at the traces, the team saw that models spent many turns just getting grounded, because information was spread across siloed systems: task management, on-call dashboards, uOwn, and others.
The context graph connects all of these into one map for agents. Uday's live visualization showed more than 100 node types and more than 130 edge types, with data about three days old. A person node links to the assets they manage, design docs they wrote, features they work on, on-call events, config changes, incidents, Jira events, and PRs. Every node has relationships to other nodes. Cortana now uses the graph as default grounding. It first identifies the domain and services a question is about, then goes deeper into documents. Uday says this cuts the number of turns, and that for many categories of questions quality goes up while cost and latency go down. They describe this as still early.
Uday's example is the question "how many mobility trips around the world are paid in cash?" Answering it requires knowing which tables and columns exist, whether others have written similar queries in the last 30 days, and which team they came from. The graph helps the agent reach "blessed" queries quickly. Uday observes that writing Cypher queries by hand is hard for humans, but agents navigate the graph the way they navigate a file system with bash. Scale is not a problem: the graph platform team that runs Uber's production graph, which has billions of nodes, helped build it, and the context graph is much smaller.
Cortana is available in Slack, on the web, and in the CLI. Uday says it breaks down information silos across organizations. Someone in operations in any country can ask how something is built, who works on it, or what its history is. It can also produce a "360-degree" health report for a service by examining dependencies and storage systems and running queries. Existing user permissions carry over, so users can only see data they already have access to.
Planning: PRDs, ERDs, and Physical-World Edge Cases
Moving to the feature lifecycle, Uday says some phases are further along than others. The idea of a combined plan that agents and engineers iterate on together is still at the stage of "a lot of cool ideas." What exists today is a single place for product design docs, engineering design docs, launch reviews, experiment reviews, and production-readiness reviews, with agentic skills for working on them. Each project links to Jira or Linear, and all of it feeds into the context graph. The result is a source of truth for specs, approval status, and ownership that both humans and agents can trace from spec to execution.
Uday argues that planning matters more at Uber because of product and country heterogeneity. Corner cases outnumber the happy path, and operating in the physical world multiplies them: what happens if this goes wrong, how a support case gets handled, and so on. A good plan up front makes it easier for engineers or agents to decompose and execute the work.
Minion: The Background Coding Agent
Minion is Uber's background coding agent. The current version is Minion v2. Uday says around 15% of all production PRs go through it. It runs on the mega DevPod, so it can work across repositories and produce one or several PRs from a plan. It can be used from the web or Slack, with an optional plan mode, or given a single prompt for simple tasks.
In Slack, an engineer can align with a PM and EM in a thread, then write "@minion follow this thread and create a PR," and Minion uses the thread as context. Uday describes it as especially useful in support channels. Someone reports a bug, the on-call engineer asks Minion what it sees, Minion returns a root cause, and if the fix is simple it is made on the spot instead of going into the backlog. Minion started with mostly maintenance tasks. As more self-validation tools are added for web, mobile, and backend APIs, Uber aims to expand the kinds and share of tasks it handles.
Shortening the PR Cycle: Self-Healing CI and Code Inbox
Uday says PR volume has gone up year over year. Uber tracks P50 and P90 cycle times for the inner loop (getting a change to green and creating the remote PR) and for the outer loop (human review and CI). Reviews have become a big bottleneck, as at other companies.
Uber took several steps in response:
- Shift-left validation. Many simple CI jobs, including linters, coverage rules, build rules, and template rules, moved into the inner loop behind a single command,
aifx validate, so agents can run them before creating a PR. - Visual and environment validation. Agents can spin up emulators, web instances, and staging deployments to test changes, and skills teach them how.
- Self-healing CI. It can be applied to all PRs or per PR, with automations such as auto-request-review, auto-merge, and auto-fix.
- Code Inbox. A view showing PRs waiting on an engineer and PRs they are waiting on. For each PR it shows what the AI code review checked, sanity-test screenshots and videos, and build checks. Uber is adding a risk score and blast radius.
The growth in CI jobs, which Uday says doubled or more than doubled year over year, forced Uber to do an internal code red on its CI system to prepare for more PRs.
Will human code review go away?
Uday says Uber prefers to err on the safe side because incidents are very costly. Every PR still requires a human engineer's review. If confidence grows in risk scoring, which can be back-tested against years of data, and in AI code review, Uber might enable a "green lane" for auto-landing some classes of low-risk PRs. Uday stresses that Uber wants to be "really really sure" first. The current P0 is getting reviewers the right signals so they can review faster. Auto-landing is a P1.
Slate: End-to-End Testing That Agents Can Drive
Before agents, engineers tested multi-service features using Slate. An engineer deploys their branch to a personal Slate instance and pins a test account UUID to it. Requests from that account hit the branch for the services under test and normal staging for everything else. This works across two or three services at once.
Once Slate existed, Uber only had to teach agents to deploy to it and give them the relevant APIs. In the demo, an agent uses the Slate tool to create an environment, deploys a test service, runs an agent browser against both main and the branch, and produces a before/after report showing no regressions. That report can be attached to the PR. The same works for iOS, Android, web, and backend APIs.
The host stresses how hard this was to build. Uday explains that it relies on Jaeger, the distributed tracing system Uber built around 2017–2018 and later open-sourced. Every call has a parent span, and every hop adds a child span that can carry "baggage." Slate adds baggage naming the services under test and the pinned account, and the service mesh reads it to route requests to the Slate environment. Uday says it took years to retrofit every existing service and to make sure new ones were built this way. In their view, without systems like Slate it would have been very hard for agents to validate their own work.
Capri: Capturing and Replaying Production Traffic
Uday calls Capri (capture-replay) very Uber-specific and hard to get right, noting there were several earlier attempts. This is the version they believe can scale to all teams. The problem it addresses: the rider app has hundreds of screens across product lines, cities, and other dimensions. Writing tests for every combination would mean maintaining thousands of tests.
Because language frameworks are standardized, a single middleware can capture real production traffic. It records a service's inbound and outbound requests and responses, including calls to downstream APIs. Platform teams write simple rules that map request fields to dimensions they care about. From the captured traffic, Capri derives the full set of live dimension combinations a platform actually supports, called MatchGuard. MatchGuard runs pre-deploy in CI/CD and shows the before/after delta for every PR in a deploy. Uday's example is catching a change that breaks only cash trips in Brazil under a particular configuration, something hard for engineers or agents to test manually.
In the demo, Uday browses captured sessions tagged with metadata such as marketplace type and line of business "eats," inspects outbound calls, and shows how a bug affecting only San Francisco Uber Eats trips could be fixed and then verified by replaying that specific request against a Slate environment and diffing the result. The host recalls Deputy and Cerberus as earlier tools. Uday explains that Deputy routed production requests to a test instance and Cerberus routed a local instance back to staging servers, and that Slate now makes both largely unnecessary. Capri is being exposed to agents as well. Uday describes the intended loop: the agent fetches sessions of a given type, composes a matching request on its Slate instance, checks whether the fix worked, and iterates until it does.
Edge Gateway, Shepherd, and Fox
The Edge Gateway, the successor to RT API and written in Go, is now the single entry point for external APIs, including gRPC, webhooks, and GraphQL, with shared monitoring and alerting. It has its own capture-replay built on Capri. Using Jaeger data, Uber knows each service's dependencies and has automatically tagged them as required or optional. Uday says this gives both engineers and agents richer context when diagnosing alerts, for example by pointing to the likely unhealthy node in a service graph.
Shepherd handles large-scale changes in the monorepos. It started with OpenRewrite policies for deterministic changes such as signature updates. Shepherd knows which modules belong to which teams and creates a PR per module. With AI, it gained agentic campaigns: for example, fixing context propagation across all services so tracing works better, using prompts and skills that an agent runs on every module. Each migration is a "campaign" with a YAML policy, per-module PRs, and status tracking for the campaign owner, as in the JUnit 4 to JUnit 5 migration.
A few months before the recording, Uber added Fox, aimed at continuous improvement rather than one-time migrations, or "self-improving code bases" in Uday's words. Service owners enroll their service in skills such as dead-code cleanup, performance analysis, or noisy-log fixes. Each week, around Sunday when CI load is low, Fox generates a PR per enrolled skill. It is opt-in for now, and some skills may move to opt-out as they improve. Uber reviews maintenance-category PRs to decide which new skills to create or improve so more of that work can move into this managed loop. The aim is to cut toil so engineers can spend more time on feature work.
Wisdom and Debug Assist: From Bug Report to PR
All employees dogfood Uber's apps. Wisdom is the system for bug reports filed from employee beta builds. AI routes each report to the right team, and the report includes logs and telemetry: screens visited, analytics, network requests, and feature flags. Uday says reproducing bugs used to be a huge problem. Now Wisdom gives engineers reproduction steps along with expected and observed behavior.
On top of this, a managed agent called Debug Assist examines the telemetry and the codebase and proposes a root cause along with an impact and evidence timeline. If the engineer agrees, they hand it to Minion, which creates a PR and attaches it to the Jira task alongside the RCA. If someone later corrects the root cause or opens a different PR, that becomes data for improving both Debug Assist and Minion. Engineers can also make a test device replay the network responses served to the employee who filed the bug, so they see the same issue and can confirm a fix.
The Changing Job and How Uber Measures Productivity
Asked how the role is changing, Uday frames the work as repeatedly finding and removing the next bottleneck, using both AI and conventional systems engineering, with the goal of maximizing the number of features shipped. They mention "capability overhang": current models could do much more than organizations use them for, because orchestration, harnesses, and tooling lag behind.
On productivity, Uday says Uber previously measured standard things such as code throughput, plus periodic developer experience surveys (NPS and sentiment), incidents per diff, and code coverage. The internal conversation is now shifting from diff velocity to feature velocity: "you can create 100 diffs but if they're not all feature diffs," that does not necessarily add user value. Uber tags PRs by complexity and category (feature, bug fix, maintenance, testing, documentation) and tracks the mix year over year. Uday reports that the feature share is rising and the maintenance share is falling, which they take as a sign of going in the right direction. Because links between product docs, engineering docs, and tasks were often missing, Uber is using AI to cluster and tag the gaps so it can compute P50 feature cycle time by complexity, per half, over the past two to three years. The aim is to find where the bottleneck sits, whether in the PRD phase, the launch phase, or the rollout phase.
Personally, Uday says that in their whole career their way of working has never changed this much. Even Distinguished Engineers at Uber are hands-on, and Uday says they are producing far more PRs than a year ago and exploring more ideas. They see the core skills shifting toward context engineering (deciding what context an agent needs and setting up the environment to provide it), giving the right specs and constraints, multitasking across many parallel sessions, and quickly verifying output so regressions don't ship.
The host suggests a new primitive: instead of asking how to build and verify software, engineers ask how to set up an agent that can build and verify it. Uday agrees and gives an example from customer support. Converting a human runbook into a policy an AI support agent can follow used to involve many manual steps: finding the right APIs and attributes for dynamic elements, configuring systems, running manual evals before launch, and running continuous evals after. The team taught agents to produce a solid v0 policy, automated configuration and testing, and used past context to generate synthetic evals. Uday says the process went from weeks to days per policy, with guardrails and human verification steps still in place.
Asked about a favorite tool, Uday mentions recently experimenting with personal agents to automate manual tasks in their personal life, following the same pattern: give the agent the context, provide inputs, and keep a human-in-the-loop verification step. They close by saying Uber is still early, with much left to build.
The Host's Takeaways
In a closing segment, the host, who left Uber six years earlier and found much of this tooling new, adds a few observations. Uber was known for about 5,000 services with about 2,000 engineers a decade ago. It has since consolidated toward well-sized rather than micro services, though the count is still in the ballpark of 4,500–5,000. The host singles out Capri as a big improvement over Deputy and Cerberus, which made capture hard. On code review, the host notes that every PR at Uber still merges only after a human decides it is ready, while AI-driven risk signals help engineers decide which PRs deserve deep review and which can get a quick "LGTM." The host's broader conclusion is that Uber's decade of developer-productivity investment, such as DevPods from 2017 being reused for Minion and existing validation systems becoming agent feedback loops, is paying off. The host suggests that, at least in Uber's case, the companies getting the most from AI are those that already had strong developer platform teams before AI.
What does development look like today? What kind of tools do you use?
We expose one OpenAI-compatible API or Anthropic-compatible API, and all of the models across all the vendors, open weight or closed weight, they're all behind this gateway.
What is AI Guard?
We wanted to make sure we are not sending any sensitive content back to the user. Around April of 2025, we started working on MCP gateway. Similar to model gateway, this is one gateway for all of the tools.
We have 100-plus node types, 130-plus edge types, and this is live data as of, say, three days ago.
We call it Slate. Essentially, you can create your own instance that deploys the branch that you're working on. For some of our most complex platforms, we call it capture replay. Essentially, you can capture real production traffic with just one middleware.
How is your mental model changing of software engineering compared to pre-AI?
In my total tenure in engineering, there's never been this much change to how I work. For example,
What kind of internal tools do software engineers at Uber use to build software? Today, we get an exclusive insider and visuals-heavy look from one of Uber's longest-tenured engineers, Uday Kiran Medisetty. Today, Uday leads much of the work on agentic AI at Uber.
In this conversation, we get into internal developer systems Uber has built and used pre-AI, like uOwn, M3DB, Michelangelo, Cadence, Morpheus, and others. AI foundational systems the company built, like its MCP gateway, AI Guard, and AIFX. Recent systems used by nearly all engineers, like Minion, Code Inbox, Slate, Shepherd, Wisdom, and others. If you want to understand what AI-native development looks like inside of a modern tech company with thousands of engineers like Uber, and why it's much easier to retrofit AI on top of battle-tested developer infra, this episode is for you.
This episode is presented by turbopuffer, vector and full-text search built on object storage. It's fast, cheap, and extremely scalable.
Today's episode is about the close to two dozen internal systems that Uber has built to help engineers ship with AI agents. One of the takeaways will be that production-ready code is far more complicated than just using an AI coding harness. The harness is the easy part. What's hard is all the things around them.
This is where I need to mention our season sponsor, WorkOS. One piece that is always a hard problem with agents is permissions. Do you want an agent to have standing access to everything it might touch? Or do you just approve every single prompt until you just stop reading the prompts? WorkOS built Airlock for this. Airlock is an authorization layer that sits between your agents and the tools they call, and it judges every single request against the agent's intent and your rules. You write the rules in plain English. Agents never see your credentials, and every verdict is logged. It works with coding agents like Claude Code and Codex. WorkOS Airlock is how agents do real work without over-permissioning. Check it out at workos.com/airlock.
Uday, welcome to the podcast.
Thank you so much for having me. I've been following your work. It's amazing to see the kind of impact you've had and all of the great content you publish for all of the engineers. I'm super honored to be here and talk to you today about what we're doing.
Well, we worked together well before The Pragmatic Engineer was born, back at Uber. Not together, but I knew of your work as well. When did you join Uber exactly?
I joined Uber in May 2015. In fact, the two engineers who referred me, they are still here. I joined in our real-time API team. It was during the period where we were trying to break our monoliths into microservices, and I was working on our API gateway for all of our mobile apps, and I worked on changing our mobile protocol.
Since then, I worked on many different business platforms, from our fulfillment, Uber for Business, on our earner onboarding, marketplace pricing, and the last one year has been all agentic AI, both for our users, customer support, end users, and so on. It's been a roller coaster, last one year, working on this.
And for those people who didn't work there with us, can we recall a little bit what real-time API, or RT API as we used to call it inside, what it was and why it was a big deal and why it had so much load?
When I joined, we had just two services at Uber. It was called Dispatch, written in Node.js, and API, written in Python. Dispatch used to handle all of the rider and driver traffic at that point in time. And the protocol was pretty simple at that point. It was a four-second ping protocol, we used to call, that would give you the entire state of the world. And when the trip is completed, we would hand off to API. It would handle all of the post-trip processing, payments, receipts, and everything.
RT API was born at that point in time to abstract all of the internal services and mobile, and also changing the protocol from simple four-second protocol to back, where we have both REST APIs and we also have server-to-client communication. We used to call it push protocol. Because so much of our app is very dynamic, everything keeps changing every few seconds, we needed a way to efficiently update the state of the app across both riders and drivers as and when we get new updates, as and when things change in the physical world.
It used to be, at that point, one of the highest-QPS services. And over the years, because all of our stack was just Node.js and Python, that was the natural choice at that point. But in the subsequent years, we have converted that into Go, and it's still one of our highest-QPS services at Uber.
And I think for people listening, they're going to be surprised when they hear, "Oh, there was only two services in 2015, or around that time, at Uber," because of course Uber was known for the microservices boom. And Thuan was on this podcast. He talked a little bit about this, but would you mind just recalling how those microservices came to be and, from your perspective, what it meant to suddenly have so many services?
I think there's so many blogs and so many people talk about that. When I joined, as I said, we had just two services. And you also think, going back in those days, the kind of hypergrowth phase where we were in. We were constantly expanding to new countries. We were constantly expanding our product portfolio across mobility, and later we started Uber Eats, and that also was expanding.
We needed a mechanism for teams to really build all kinds of functionality very quickly to match up to the scale at which the business was growing. So we have invested a lot in all kinds of service infrastructure, making it easy for teams to build services, metrics, tracing, monitoring, observability, deployment, the whole bunch of stuff around that. And there's not much off-the-shelf software for these things at that point, at the scale and rate of change that we had. And there's a lot of stuff that we had to build in those days internally, and that created a boom cycle. We went from two services to order of thousands of services.
Then around 2018, 2019 was when we standardized on our languages. We picked Go and Java as our primary languages. We modernized a lot of our tech stack. We spent a lot of time in thinking about the layers of our architecture. How do we platformize? And over the years, in fact, when I worked on one of our fulfillment platforms, we actually reduced some of the services to a few services.
Even now, we are in the similar ballpark of number of services, even though our business is ten times larger, we are doing many, many more products. And this whole bunch of platformization and years of investment that has gone into it, we're still running one single global tech stack across all the countries we operate in. There's many, many platforms that serve all of the mobility verticals and all of the delivery verticals using one shared building block. And I think that allows us to build more mobility verticals, more delivery verticals, because we have platformized a lot over the years.
Would you mind telling us, when you joined, how big was Uber's engineering team, and how did it grow over time? Because I don't think people from the outside appreciate just how big that growth was.
When I joined, I was around the first 600 engineers. And in fact, the whole of the marketplace team at that point used to be maybe 50, 60 engineers. And over the next few years, as we've scaled our business, as we scaled our traffic, we grew to some few thousands of engineers. And around 2019, 2020, we started also expanding across different sites.
Now we have 12 global tech sites across the world, in almost every continent we operate in, in fact. And that helps us build products for those hyperlocal markets. And some of those products are then rolled out globally across all the markets where we could roll out that kind of functionality. So I think that gives us a lot of ability to tap into talent globally and also build nuanced local products. And that also helps us build one global tech stack.
And speaking of building products, tech stacks, you still work on the platform team, right?
Yeah. At least until my first ten years, I worked on different business platform teams. And in the last one year, I'm spending more of my time on a developer platform, on agentic AI for engineers, agentic AI for the rest of the company. And in the process, I've also worked on some user-facing AI initiatives as well.
Can you take us back to 2015 on how Uber's internal tooling changed? Because again, I was there, and I was really surprised to see just how much custom tooling we had. We talked about RT API, but there were things like Morpheus, Edge Gateway, and other things. Would you mind talking through, back in 2015, 2016, if an engineer joined Uber, what kind of tools did they use and how did those evolve? I'm talking, obviously, the main ones that most engineers, backend or mobile engineers, touched at Uber.
Yeah, absolutely. I think in those days, given the way in which we were scaling, there's not many off-the-shelf products that we could tap into. We talked about RT API and Edge Gateway. Some of the other stuff that we had to build was a system called uOwn that will enable anyone in the company to identify who owns a specific asset.
It was services, right? And you could look during on-call, if there's an outage, you could figure out which team owns it, who's on-call for that kind of stuff, right?
Yeah, absolutely. And as the engineering size grew, we also had to build end to end, our engineering design docs, product design docs, getting all of those into one single place. We had to work on getting all of our incident management into one single place. We are really obsessed about how we handle our incidents, the learnings from that. So we built a lot of really good tooling around that.
I think some of the more complex ones was our metrics. Even back in 2015, 2016, we used to probably have around 500 million metrics emitted per second with the cardinality. And there's no off-the-shelf metric system that we could use, and that gave rise to M3DB, which was open-sourced later. And there's even companies formed around that. And that enabled us to handle the kind of metrics volume that we had.
We also had to build uOwn end-to-end machine learning platform because we used to have hundreds of ML models. And if you take the whole life cycle, from creating the features, building the models, monitoring, serving...
This is Michelangelo, right?
Michelangelo. And we recently, just a couple of months ago, open-sourced our deep learning and part of the Michelangelo infrastructure. So I think there's a lot of really good, solid foundational building blocks there.
The other interesting one is also Cadence. If you think about some of the workflows, they span across multiple days, whether it's driver onboarding, whether it could be some CRM campaign. You could see a lot of engineers doing their own resiliency stuff, their own state management, and that gave rise to Cadence, which is a durable workflow engine. And later, even that got open-sourced.
I think not many people will know Cadence, but a lot of people will know Temporal, right? Which was, as I said, either inspired by Cadence or a lot of the ideas came from there. I'm not even sure it was the open-source version of Cadence, but it had a lot of, it's kind of the external version of whatever Cadence is for Uber, or used to be.
Yeah. And I think those folks, they saw there's a real need for something like this, and they were able to go and build, scale it even further beyond Uber. But I think the pattern that I'm trying to say is there's a whole bunch of things that we had to invent in those days for the scale that we had. And over the years, as the industry also matured, now we have a few products that we build. Some where we think there's some unique need, we buy. And if we build anything, if we think it's interesting for the rest of the industry, then we try to open-source wherever possible.
Yeah, I think two more tools that I think are worth mentioning, because I just remember them. One is Morpheus, or it's basically LaunchDarkly for Uber, but it was way more integrated into everywhere, right? You could go in there and run experiments. You could see the results immediately. It was almost like feature flagging and experimentation, but integrated into mobile feature flags. And I remember inside Uber, all of these integrations, it was just integrated into a bunch of tools, all of them, right?
Yeah. And since we were also expanding to so many countries, you need a really good configuration platform, whether it's for configuring our backend services or also having a feature flag system and also experimentation. And Uber is a very data-driven company. Everything goes through a feature flag. Everything goes through an exp—
Experiment.
And since we are a two-sided marketplace and three-sided marketplace, some of the experiments are also really complex. It's not just simple A/B. We also have switchbacks, and we have to build really deep experimentation that's baked into all of our tools. And it can automatically generate analysis, what are the p-values, what metrics to use. And that also led to working a lot on our metrics foundations, data foundations, so that whatever analysis that we generate from these metrics are trustworthy, and we can make business decisions based on that.
Yeah. I remember I was always wondering, why do we have uOwn feature flag system? But then I was thinking, show me a feature flagging solution from a vendor that supports cities. In fact, cities are first-class citizens, because inside Uber, I think cities were so important. And a lot of our tools had cities as, for example, for cardinality, every city had its own metrics. And then you start to realize, very few things treat... Uber had a bunch of interesting constraints, right, that no one else had at this scale.
Yeah, absolutely. I think the city one is an interesting one. It's so deeply baked into our infrastructure. For example, some of our critical platform services, we also do sharding by city, because then we can also observe across different tiers of cities. Because we need to make sure, even now, across all of the regions where we serve our traffic, we shard a city to a specific region. We can also at
At any point in time, fail over that city across regions. And we also have logical deployment groups where a group of cities are routed to a group of containers. This is for a few services. We don't do it for all the services. We do it for some of the services where we need that level of sharding and that level of reliability.
One other system I wanted to ask you about that I remember using, and it was being developed in 2017, I believe, which is DevPods. It felt really ahead of the industry at the time. How did that come along, and what did it turn into?
Yeah. So I think the backstory for DevPods is because we already talked about the growth of microservices. Initially, we had micro repos where every service was its own repository, and we didn't have a standard build system at that point. Each service had their own. We had at least two, three build systems, two, three testing systems, and it was very hard even for a central team, when we wanted to do library upgrades, to go across all the services and do upgrades.
So around 2017, 2018, we then changed our strategy to getting to monorepos. Even though they're microservices, all of the services were deployed from the monorepos. We had one monorepo per language. So we had a monorepo for Go, one for Java, one for mobile iOS, one for Android, and so on. Then once the scale of that repository was pretty big, it was pretty hard to run the build and get all of those systems set up on your local laptop. Also, then you have to go through all kinds of setup issues.
So we took on this thing of DevPods, which is cloud developer boxes, and you can have any number of them. We had one DevPod per monorepo. So if I'm working primarily in the Java monorepo, I create the Java version of the flavor of the DevPod, and it has all the build system cache. I can have faster builds. It's working on cloud machines that are beefier than even the laptops. So we could have really good SLAs on your build time, on your test time. And we don't have to worry about the setup issues because we can install the right tools in place.
And now the interesting part is, as we got to the AI era, even the agents need similar workspaces and cloud boxes, and we are able to kind of latch on to that. We also created now a mega DevPod with all of the repositories because then I can have my agents create PRs across the repositories. It has all the tools it has to work in any repository, to look at context across any repository. This also helped us to create the AGENTS.md, CLAUDE.md at the top repo, at the top folder, because we have standardized build, standardized testing. Once we set up the top-level files, most teams will get really good value out of the bat. They don't have to really go into per-module setup.
And on the DevPod, all of the AI tools are installed by default. So you can get a new DevPod. You don't have to go through even installing Claude Code, Codex, OpenCode. They're all there in the DevPod. So this made it really easy even for non-engineers in the company to create a DevPod and start playing with Claude Code, start playing with Codex out of the bat.
Yeah. So I guess having cloud development environments really early helped a lot.
Yeah. I think all of those years of platformization bets also helped a lot as we expanded on our agentic journey, because you could use all the things that we built making life easier for engineers, and it makes it easy for our agents as well.
Yeah. And I wanted to talk a little bit about before we got into what's on today, because I think a lot of things are connected, right? So there were so many things being built for platforms, for tools, often custom for Uber. But having talked enough about that, what does it look like today? What does development look like today? What kind of tools do you use? And maybe you could give us a little bit of a primer of the foundations that you put in place specifically for AI, and what you built and why.
First thing is, especially in the AI world, one thing we realized very early on was, because the landscape is changing pretty quickly, harnesses are changing pretty quickly, models are changing pretty quickly, we wanted to invest in the right set of building blocks that are harness- and model-agnostic, so that no matter any of these things change, we can easily port across different models, different harnesses. So a few things that we invested in: we started working on our model gateway from the initial introduction of ChatGPT, late 2013, early 2014. So we had a model gateway back then which would expose OpenAI-compatible API or Anthropic-compatible.
So sorry, this is 2023, 2024, right?
Yeah. Sorry. 2023, early 2024, when ChatGPT was first launched and GPT models were first launched. And we quickly noticed we wanted to make it easy for any teams in the company who want to leverage and build agents with these models. And so we had a model gateway that would abstract all of the complexity behind, where we had all of our security and policy guards in place. We had all of the observability in place. So you could switch across different models or different vendors for these models, and you could just have the ID for the organization or the project, and then you can use any of the models exposed.
So this was, you just exposed it into a model selector, and it piped into the model gateway, and it routed to the right, may that be ChatGPT at the time or Opus at the time, those models.
Yeah. And it was built on Michelangelo itself. So we created Michelangelo GenAI.
Oh, Michelangelo. But that's the ML platform, right?
Yeah. And then now there's a GenAI set of foundations on the Michelangelo platform. So then you go to create a GenAI project or AI project on Michelangelo. Then you can set up the project. You can select the model that you want. Then you'll get a unique project UID that you can use for your agents. So we're able to use the same Michelangelo infrastructure then to start building on AI-specific evals and stuff on the same foundations. Looking back, I think that was a pretty early investment that gave a lot of ROI later.
What is model gateway? I can show a quick primer into what we had built. This was late 2023 when the first set of LLMs were launched, and we quickly realized that we had a whole bunch of internal teams wanting to use it for different use cases, and we wanted to make it really easy for them. This was built on top of the Michelangelo platform, and one more component inside that they can set up there.
And Michelangelo, just to recap, that's the ML platform that was built from 2015, 2016. So it was built a long time before LLMs.
Exactly. And now we have a GenAI or agentic AI set of components also inside Michelangelo. And we're also building a lot more right now. Those are not in the open-source version yet, but we're actively building a lot of stuff for the agentic AI world. But model gateway was the first such one. In the initial version, we didn't have all of these middleware and components, and we had built some of these over the last one, two years.
But what the experience for any team in the company is, they create a client, they give their project ID, which they initialized in Michelangelo, and then the caller token, and that's all. And we expose one OpenAI-compatible API or Anthropic-compatible API. All of the models across all the vendors, open-weight or closed-weight, they're all behind this gateway. And the first thing we added there was what we call AI Guard.
What is AI Guard?
Essentially, we wanted to make sure we are not sending any sensitive content back to the user. So this was one entry point where we could add this middleware that we can enforce consistently across all the projects. And the security team had the ability to iterate on all of the AI Guard-specific models, whether it's on content, whether it's on safety policy, whether it's on a whole bunch of comms policy. There were five different sets of fine-tuned models that they built there, which run on every request and every response with strict latency guardrails. This helped a lot to make sure that all of our projects have the same standard baselines.
And then because every project is tracked here, we could have per-user, per-caller attribution. As we progressed later this year, we were able to add spend tiers for projects, spend tiers for users. And since all the requests go through one place, we could also add audit logging, session traces, and a whole bunch of infrastructure around this. Every model request at Uber goes through this. So there's no model request that doesn't go through this. That gives us one single entry point across all the models.
And I'm assuming you can just track billing as well, costs and all of those things.
Exactly. You can track billing, cache hit rate, and all kinds of things. And now, as we progress, there's a lot of discussion of model routing and all of that. I think this could be a good entry point that we can leverage because all of our models go through this.
I think on the same journey of being able to make it easy for engineers to switch across models, the next thing was, how do we make it easy for engineers to switch across harnesses? If I'm using Claude Code, then if I switch to Codex, then I need to carry forward all of my settings, all the skills, all the tools, all the plugins. So we internally wrap all of the harnesses with a wrapper called AIFX, AI Foundations, and this is preinstalled on DevPods, or you can install it on your local machine.
And we have a concept of personas. So then that persona comes with a whole bunch of default plugins and skills, so that you don't even have to install. Any engineer, they get a whole bunch of default skills that they can use out of the bat. Since we wrap everything, all of the settings are managed. So even if you're using any of the harnesses on your local or on DevPod, or any of your DevPods, the settings are synced. So then centrally, we can manage one set of settings that we can propagate across all the users, and any user-specific settings are synced across all of their environments.
And again, this makes it really easy because now we have one entry point for switching the harnesses, switching the tools. And then we also made it so there's a status line where we can give tips, where we can have tracking of your spend across harnesses, within that harness, in that session. So all of those additional things that we could add without any explicit setup from the engineers.
Nice. And then speaking of tracking spend, how do you approach that right now? As an engineer at Uber, if I just join as a senior engineer somewhere, do I need to be aware of spend, of cost? What's the thinking there?
See, we want to make sure engineers have access to any of the tools that are available, and we have this notion of spend tiers. So by default, you get a certain tier, and then we have many levels of tiers. Once you're close to that tier, then you can raise a request to go to the next tier, and you just need a simple manager approval. So then you can discuss with your manager the ROI for that project, how you're spending, and it is just a conversation around responsible use of this, right?
But we don't want to restrict engineers. If this is going to the right project use, they should be able to spend whatever they need, because ultimately we want to maximize the gain from this, but you also want to make sure there's responsible use. So we try to do the minimal amount of paperwork needed to cross the tiers. And they will see across any of our tools how much they're spending within that harness and across all the harnesses. And we also have different tiers for different managed agents. So for all of the interactive harnesses, we have some spend tier. For some of our background coding agents, we have a separate independent tier. So we make sure they can easily manage across all of these.
Yeah. But I guess it just makes sense. Even if I do my side projects, I will be aware of my own spend on whatever limit I have. And do I sense it correctly that the thinking is just, don't use tokens on stupid stuff that makes no sense, or accidentally you might just be burning and you don't know? So just give that feedback: these things are not free.
Exactly. I think just having that feedback. And we also have a plugin called Cost Check that will run across all of your sessions and give you personalized tips based on your usage to say, hey, this is the kind of thing, this much you're spending on this project and this session. These are the tips that you can use to further improve your cost profile. And I think all of this creates a good conversation between managers and engineers, but engineers will also see good feedback into how they are efficiently spending across. And that helped us a lot with all of the real-time coaching, with context health, model sizing, so that engineers are also aware of working in this new age. We also don't want inefficient spend.
And what other foundations did you build related to AI? MCP gateway?
Oh yeah. I think the other backstory was around early 2025. As Cursor kind of picked up, we enabled Cursor, and MCP was a spec that was created around that time. And we quickly realized that we can really increase the power of AI by exposing all of our internal tools in an easy way. So then we could potentially leverage this beyond just coding. We could leverage it for all kinds of software engineering tasks.
So around April of 2025, we started working on MCP gateway. Similar to model gateway, this is one gateway for all of the tools. So what we did was, because we already had a good IDL registry across all of our services, whether they're using Thrift or proto, since we have the IDL specs, we autogenerate the MCP specs for any of these services.
And just to add, I guess for those who don't know, Uber services either emit Thrift or Protocol Buffers as strong contracts, right?
Yeah. And initially, we were just Thrift, and later we added support for Protobuf as well. But all of the new services in the last few years, they're all Protobuf. The ones that were created a few years back, they're all in Thrift. And as they go through rewrites, usually teams switch to proto. But whether they're in Thrift or proto, since we have all of the IDLs in one single registry, we could use that registry to generate a default MCP spec for them.
So anyone in the company can go to this tool, MCP gateway. They can see all of the MCP servers that we autogenerated, and if you go to any of that, you could see all of the tools that were generated from that existing API. So these might be the APIs from those services that are projected as MCP tools, and we autogenerate the request pack, response pack, the descriptions. Later we
Also, based on the API descriptions, we generate LLM-optimized descriptions for the APIs and fields. We saw some tools with 3,000, 4,000 tokens, so then we can optimize it for model consumption. It is just one single config and security policy change, and then they can enable it. They don't have to do any code changes. If they want any custom MCP, we have a framework for creating a custom MCP and proxy their custom MCP also through MCP gateway.
The other interesting thing we did was even all of our vendor tools, whether it's Google, Slack, all of them are also exposed through MCP gateway. This reduced a lot of burden on configuration management. Since we already spoke about AIFX, I could install any of the tools in the same manner, whether it's internal MCP tools or external MCP tools. In the last few months, we also projected all of these MCPs as CLI. So you could execute any MCP as a CLI. If some agentic harness wants to use it as CLI, they can use it as CLI. If they want to use traditional MCP, they can use MCP.
And do I understand that this all comes because the APIs to start with were already either generated or written? A lot of them were generated. You already had a platform for the APIs. So adding MCP was somewhat trivial. Of course it's work, but you know what I mean. And now generating CLI is also somewhat trivial because you had a foundation before AI, right?
Yeah. And then, that's what I'm starting to see with Uber, that you built a bunch of stuff before 2022 and you could just kind of add AI infra pretty easily on it.
Yeah. And the same team that works on our API gateway also worked on MCP gateway. They're already in the business of building API gateways, so they know how to do any kind of protocol transformations, any kind of API gateways. We were able to pretty quickly build this. Especially once we got our Google Workspace and all of our Slack tools also into this, it created all kinds of amazing agent workflows. People could automate anything at that point, from Slack to Google, Jira, Linear. Any vendor tool or any internal service is a CLI call away.
And then the other set of foundations, as we started thinking about the space, was, I think all of these go in this camp of vendor-neutral, harness-neutral building blocks that we think you would have to do irrespective of what happens, because agents need some way to access your tools. Then the other thing was around skills. We jumped onto this earlier this year. Since we already have this notion of monorepos, we created a skill monorepo, in some sense, that has all of the core skills that central platform teams build and all of the team-specific skills individual teams can build.
And are these skills just the skills folders, the plugins that you can use for Claude, Codex, etc.?
Exactly. Plugins and skills that you can use across all the harnesses. Then we had one single quality gate that can run across all of them, and we made it portable across all of the harnesses. So you install with the same AIFX tooling that I mentioned, and if you do plugin add that skill, it will update the configurations across all of the harnesses, whether you are at that point using that harness or not.
So that even if you switch across harnesses, you have these plugins auto-installed. You don't have to worry about reinstalling them and reconfiguring them and so on. And since all of the skills are using the tools from the MCP gateway, we also have one standard mechanism for all these skills to access any of the tools.
Now we are working on some tools where engineers can go select their skill. It will tell them all kinds of tips to improve their skills. The next thing in flight that we're working on is how do we then collect the traces from the harnesses and generate improvement patches for the skills so that we can continuously improve the skills based on learnings? There's a lot of discussion around skill quality, skill evals, and how do we improve that continuously. When you first build a skill, they might not be perfect, and you need some mechanism to continuously improve that. I think both the MCP and skills gave us a lot of benefits. If I look at the last few weeks, there's probably around 30,000 skill executions per day.
This is kind of interesting for me to see, because if you told someone, "We're building a skills repo," what most people would think inside a startup—I'm not talking about Uber—is you just do a Git repo and you have a bunch of skills there, just a bunch of skill folders, and that's kind of it. But it's pretty clear that it's more complex, right? Is it more complex because of how many engineers you need to support? Is it because you're also running evals, because you keep cleansing these skills or tracking if they're used or not used? Those kinds of things.
Yeah, absolutely. I think now, with any of these latest models, it's very easy to create a v0 version of the skill. It's very hard to make it really good across all kinds of use cases, and that's why you want to really make sure we have one place we can update all of these skills. In fact, when we exposed the CLI projection for all of our MCPs, just a couple of engineers using AI updated all of the skills in this repository to use the CLI pattern because it's more token-efficient. If every skill says MCP A, B, C, MCP D, E, F, then if I install these two plugins, I'm going to have six MCPs across these two always in my context, and there's so much context load. Then we replaced all the skills with the CLI pattern, so that had a lot of impact on our token efficiency as well.
And the other thing is, because we auto-install this skill marketplace for everyone, it reduces one more step of configuration. Discoverability is easy. The setup and configuration and management is easy, and we could build a whole bunch of eval pipelines around that, discoverability pipelines around that. We still have teams that have their own repositories, but the majority of the skills go through this. My favorite is context graph.
What is context graph?
The backstory is, back in December, one of the engineers built an AI assistant that you can go to. We call it Cortana. You can go to web or you can go to Slack and ask any question from any codebase. It would search across all of our repositories and answer questions. Then we saw people asking all kinds of questions. It could be research, it could be how to run this query, what is the data source for this, who owns this, can you do RCA for this—a whole bunch of software engineering tasks. When we were looking at some of these traces, it was clearly obvious that we could optimize the number of turns it takes for the models to get grounding because we have so many systems.
Historically, all of these are siloed systems, right? An engineer has to go into the task management platform. An engineer has to go to our on-call dashboard, has to go to uOwn dashboard that we talked about. Then we're like, okay, how do we plug all of that into one mental map for the agents? Just in the last few months, they built what they call Uber Context Graph, and this is a cool visualization. We have 100-plus node types, 130-plus edge types, and this is live data as of, say, three days ago.
If you click on any person node, you know what assets they managed, what design docs they wrote, what features they're working on, what on-call events they worked on, what are the configuration changes, what incidents they managed, what Jira events. If you take any node, it has relationships to all other nodes.
And what this allows us is, now the same assistant I talked about has the context graph as a default grounding. If you ask any question, it will first use this to navigate, okay, what is this person asking about? Once you get grounded into, oh, this is about this domain, this set of services, then it can go deep and read other documents, read other things. It reduces your agent traces and the amount of turns it needs to give you a good answer. We already see, in a whole bunch of categories of questions, the quality goes up, the cost goes down, the latency goes down. We're still early in harnessing the full potential of this, and we are super excited about all of the things that we can build on top of this going forward.
And then the nodes, are they people or are they tools?
This is the schema visualization. Every person in the company will have a node. There's a node for me, a node for every employee, and then it will have relationships to all of the instances. Every PR that I generate, there will be a node for that, and there's a link saying, okay, Uday created this PR, Uday manages this asset, I work on this product feature, I contributed to this design doc.
So then the nodes are both people, but they're also assets.
Yeah, exactly. One simple example is, let's say if you ask any question like, how many mobility trips around the world are with cash payment type? You first need to understand what tables exist, what columns exist. Are there other people who wrote these queries? Are they from science team? Are they from mobility science? Now all of this data is interconnected. You can go and check, are there other queries that were done in the last 30 days around this, and what kind of tables exist? Go and explore those columns. Because all of this is interlinked, you can quickly get to the blessed queries that we have and answer these kinds of even complex questions pretty quickly. Otherwise, you would have to go through many, many docs, many, many tool calls. This makes it really efficient. It's much harder for us to manually write all of these Cypher queries, but agents are really good at navigating the graph, like how they're really good at navigating the file system with Bash.
This seems very futuristic to me, but it's pretty cool, and I assume it's a ton of data to store just because Uber has 10,000-plus people working there easily.
Yeah, the good part is, similar to how our API edge gateway team worked on MCP gateway effort, our graph platform team that manages our production graph with billions of nodes also helped build this graph. Compared to our production graph, this is much smaller. It's not a scale challenge for them anymore.
And I already talked about the AI assistant Cortana, which plugs all of the context graph, tools, skills, and it's omnipresent in Slack, web, CLI. So anyone in the company, even someone in ops in some country, can ask any question about how is this built, where is this located, who works on this, what was the history behind this? It is able to take all of this data, all of our Google documents, and give them a really good answer. It breaks down the information silos across orgs. They're able to self-serve, ask anything, execute any skill anyone has built.
And I guess it must have access to the data sources, at least for the right people. So it can go into things like logs and monitoring, those kinds of things.
Yeah, it has logs, monitoring. You can ask, okay, what is the health of this service? It can go look at all of its dependencies, what kind of storage systems it uses, run all the queries, and give you a 360-degree health report for that asset. We carry forward all of the existing user permissions. So I'm only able to access the stuff that I had access to. Otherwise, that data is not accessible.
Yeah. Because inside Uber, I remember it used to be called break glass. When you added a permission for a system you didn't have access to, you go through an approval flow which goes to either a manager or director or whatnot. But I know Uber was really big, at my time, they were really big on this already.
Yeah, absolutely. I think those were some of the foundations. We have some more in the workings. We think about some even more futuristic use cases, but some of these building blocks really helped us in our agentic journey and helping take it from using it just for coding to all kinds of engineering tasks.
Uday just mentioned how Uber wrapped up AI foundations work, but already more foundations work is to come. Our presenting sponsor, turbopuffer, knows this all too well. They are rebuilding their storage architecture from first principles. They're doing this to make more types of storage faster, cheaper, and more reliable at scale. The new system, called turbopuffer V3, is not yet in prod, but it already passes CI. Still, it's not nearly as performant as the current production version, V2. And so the turbopuffer team is now perf-maxing query plans until they are as fast or faster than V2. They're doing this in public, documenting how they increased performance all the way up to the V3 production launch. For example, we can see how the team increased full-text search performance 9x batching reads. A lot of this is basic foundation work right now, but they will be sharing some very cool systems work over the next few weeks.
But hang on, why is the team doing a full rewrite? It's because search is changing, and so is turbopuffer. Turbopuffer started as a vector database, but it is no longer a vector DB. It has evolved into a high-performance database for search. V3 discards the vector primary index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast search and just generally a very fast database. The turbopuffer team is keeping a worklog of their journey, and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built.
I also want to mention our season sponsor, Linear, the place to coordinate work between humans as well as agents. One of the biggest changes for us software engineers is the sheer amount of code that we need to review. Uber has built custom tools to deal with this, but so has Linear. It's called Linear Diffs, and it lets engineers review PRs right inside of Linear. One thing I really like about Linear Diffs is how a diff sits next to an issue inside a project. And so it's super simple to see all of the context in one place and know why the change has happened without needing to jump through different tools.
For large PRs, guided reviews walk you through the core change with an explanation of what has changed. Guided reviews also group the glue code and the supporting edits separately. These small but important changes make a large PR readable in one sitting. And Linear also supports stacked PRs. If you use stacked PRs, you'll know that they're a great way to break up complex work. And if you've not yet used them, you'll probably love them when you try them out. How we do code reviews keeps changing as AI agents take on more of the work. These neat changes coming from Linear help us engineers understand code reviews more, and expect Linear to evolve Diffs to keep up with how we work more.
With AI agents. To learn more and to get six months off of Linear, head to linear.app/pragmatic. With this, let's get back to Uber and how building a feature looks like today in terms of tools used by devs at the company.
PRD and ERD, I guess for those who don't know, product requirements documents, and ERD was engineering requirements documents, right? It's like the product spec and engineering plan, for those not familiar inside Uber.
Yeah. I think we went through some of the AI foundations, and now if you think about the life cycle of a feature, then how can we truly uplevel each of those life cycles with the right set of AI foundations and also rethinking those tools for the AI era? Some we are farther ahead, some we are early in the discussion. I think even the notion of a combined plan that agents and engineers can iteratively work together, there's a lot of cool ideas.
We also have all kinds of agentic skills for working with any of your design documents. As I was saying before, whether it's a product design doc, engineering design doc, launch reviews, XP reviews, production readiness reviews, all of those things are in one place. I'm filtering some really old projects so that we can look through them, but you'll know exactly who is working on them, track the status of that. All of this data also goes into the graph as well, so that we know the full history of what are the source of truth design docs, approval status, who works on them. Each of these also have the Jira link or the Linear link, so then we can trace from this project how was the execution translated. This gives a source of truth system for both humans and agents to know the specs for every project.
Yeah. I guess planning is still really important, especially for something that is not just a trivial one-prompt or one-shot project.
Yeah, exactly. Because with the kind of product heterogeneity and country heterogeneity, we need to really think about all kinds of corner cases more than the happy path. You have to think about all kinds of corner cases and how they should exist. Since we also operate in the physical world, the number of corner cases are even more, right? You have to really think about, what happens if this goes on? What happens if this goes on? How do we handle a support case for this? How do we handle—
Yeah. I guess, needless to add, but I'll just add it because it might not be as obvious, but Uber operates in so many countries and cities. Even between two cities, the reality might be different, or might be different expectations, or people are used to different things, and so on.
Yeah, absolutely. That's why it really becomes important to think about the right plan up front. This will also make it easy for both the engineers or agents to take that plan and decompose and execute on that. Once we have the plan, then the next thing is, how do you convert that into, decompose that and execute?
Now we're going into what we call Minion internally. This is our background coding agent. Around 15% of all of the production PRs go through Minion right now. Essentially, there's a whole bunch of PRs that some engineers want to do interactively, but we also want to make it really easy, once you have a task that I have a really good spec or I want to do minimal steering, I can hand off that to a Minion. Again, we did multiple iterations. This is the latest iteration. We call it Minion v2. You can use it from web, you can use it from Slack. You can do a plan mode and then let it execute behind the scenes, or you can give the prompt once and let it execute behind the scenes if it's a simple prompt.
This is using the mega DevPod, so then it has access to all of the repositories, all of the codebase that we have. It can work across repositories and generate one PR or multiple PRs across repositories based on the plan and spec that you have. When we initially started, mostly maintenance kind of tasks used to be offloaded to Minion. As we progress on this journey, as we are adding more and more tools into the Minion environment where Minion can self-validate its work in web or mobile or backend APIs, our goal is we can increase the kind of tasks people can do with Minion over time, increase the percentage of tasks over time. I think we can also see how it's—
It's integrated in Slack. You can just mention, do you mention @Minion, or what do you say?
You can do @Minion, and then it will show you the status. You don't need to go to the web. You can steer it on Slack itself. Maybe you have a Slack thread with the EM, with the PM. You can iterate on the design if it's a simple thing. We are just iterating and get alignment there, and then say, 'Hey, @Minion, follow this thread, and can you create a PR?' Then it will take the Slack thread context and create a PR for you.
It's been really useful even in our support channels. I would see someone reports a bug, the on-call asks Minion, 'Hey, can you see what is the issue?' Then Minion will come back with, 'Oh, this is the root cause.' 'Oh, this looks simple enough. Let's just fix it right there.' Then we would fix it. There's no loop of adding it to backlog, doing it in the next thing. You were able to fix it right there. We see more and more cases of that as Minion gets powerful as well.
I think we went through the build step. We also before talked about DevPod.
It's improved since, hasn't it?
Yeah. This is showing all of my DevPods. Since I'm working on context graph, you see that. If you want to create a new DevPod, you'll see which flavor you want. For Minion, there is a DevPod flavor that has all of the repositories and is optimized for Minion usage. But for interactive, it will create a DevPod, and within a matter of seconds you will have that running, and you can—
That's impressive, seconds. I'm sure you do some clever stuff like having things warm or prebuilt so you can copy those kind of things.
Yeah. All of this, because this is developer stuff, we have separate multiple clouds that we are using for this. It's a Kubernetes cluster, and everything is warmed up. So then when you ask it, we are able to just clone from the prebuilt image. It has build caches, everything, repositories, everything already cloned. It's ready for you to build.
Since we use Bazel across all of our repositories for build system, I was able to go into a repository, run Bazel for that. We went through this in the order of seconds, where any engineer, if I want to make a code change here, within order of seconds I create a DevPod. It's one single way to build any module, and then anyone knows this command. Now agents also know the same way, right? In the CLAUDE.md, AGENTS.md, they know how to build, test any module in any repository, and they're able to use the exact same commands.
Yeah. I think that covers what kind of stuff you have to do in code generation. You can do interactive harness, you can do through Minion. The next thing is, okay, you have the PR. What do you do after that? Maybe we go through self-healing CI first. One thing that we realized is because our PR volume has gone up year over year.
Oh yeah. It must have exploded.
Yeah. Our P50 cycle time, P90 cycle time, both for inner loop, where you have the PR and you want to get it to a green state and create the remote PR, or once you have your PR created, getting the human review, getting all the CI things to pass, I think review has become a big bottleneck, same as any other company. We see year over year the delta. We're really focused on breaking down every aspect of this, and how can we go really deep in optimizing it for the AI era?
The first thing that we did was we took a whole bunch of simple CI jobs that used to run once you create the PR, and we moved it to inner loop. We kept all of this behind one single command, like aifx validate. So then when an agent creates a PR, it can run the linter, the coverage rules, the build rules, the template rules, all of these things, so that the probability of this PR being in a good condition goes up. Then we hooked on the ability to do visual validation where I can spin up an emulator and test it. I can spin up a web and test it. I can spin up a staging environment where I can deploy this change, run the APIs and test it. Now Minion or even my interactive harnesses, these agents have the skills for this. So then they can go through this verification loop.
Let's say you create the PR and still something fails. Then we have a self-healing CI you can apply to all the PRs or per PR. This is an Uber-specific line that we add to all GitHub PRs that you can link to the issue that you're working on, and you can see which checks are failing. You can have a whole bunch of automations: auto-request review, auto-merge, auto-fix. Essentially everything that we can do to reduce the P50, P90 cycle times.
Once you do all of this, then we have this thing called Code Inbox. This is showing a mock version of that, where any engineer can go, they can see all the PRs that are waiting for them, all the PRs that they created that they're waiting for reviews. If an engineer sends this for review, I go to this page. I can see all the things that we did so far. It makes it really easy for me to get, 'Oh, the AI code review passed. These are the things it checked. This is the sanity test screenshot.' So I know.
You can just quickly take a look. You have a video as well.
Yeah, I can see the build checks and all of that. We're also adding risk score, blast radius of this PR, and so on. All of this will give engineers really good signals to figure out, okay, is it good to land, and which parts of the PR I should focus on. We are aggressively working on this whole bunch of ideas that we have, but we want to make sure we are ready for the influx of PRs. We in fact had an internal code red for our CI system because year over year the number of CI jobs also doubled or more than doubled. So we have to also beef up a whole bunch of foundations in this space so that we can be ready for the influx of new PRs.
Yeah. I guess you're also seeing that there's no way engineers can review all PRs with the same rigor as we did in the past. So I guess you're building tools to signal which ones to pay more attention to, which ones might be low risk in so many ways, maybe in impact, maybe in blast radius. What is your take on what will happen with code review over time? Do you think there will always be human code review, or for some things it might go away?
Yeah, absolutely. I think one is, for us, we want to err always on the safer side because the cost of an incident for Uber is really high. Even with all of this tooling, right now we still require an engineer to review every PR that goes in. The more confidence we get in our risk scoring, and we can also backtest it with years of data, right? We have all of this data. The more confidence we have in our AI code reviews, then maybe for some class of low-risk PRs we could enable, we call it green lane, auto-landing. But we want to be really, really sure about all of the dimensions that we want before we go there.
But right now the main priority is, how do we make it really easy for engineers to get all the signals they need to review faster? If we do a good job there, then I'm sure we can increase the percentage of auto-land PRs over time. But that's P1. P0 is first getting all the signals in the right place.
Nice. Yeah, but Code Inbox is nice. I like it.
This is a thing that we should touch upon, maybe how I can do sanity testing.
Sanity testing, just for those who don't know the internal lingo, that's the manual testing, right?
Yeah, manual testing. Let's say I'm working on some new feature. It touches two or three services. How do I know if this feature is working end to end? The typical workflow for engineers, pre-agentic world, is we call it Slate. Essentially, you can create your own instance that deploys the branch that you're working on, and you can route your test account UID to the instance that you created.
So when you log in, you are executing that branch.
Exactly. Execute that branch, and the rest of the upstream and downstream goes to the other staging setups. That way you can test end to end without worrying about other parts of the infrastructure. This will carry forward across the service graph. So you can say from your mobile app, this is the test account I want to test. It will go through other services, and when it touches the service under test, then it will go to your Slate instance. If you're testing two or three, then it will go to the service under test across all two or three. So you can do even multi-service tests with this. Now that we had this, we had to just teach agents how to deploy Slate. We also give it all of the past APIs that you can use any of these services.
Basically, this was a thing where I, as an engineer, want to sanity test, meaning I want to manually test out my changes. I had a way to do this in a pretty cool way, and now we can just give it to agents and they can actually test their own changes end to end in a safe way.
Yeah. Here we are showing a web version of that, and you can see, in fact, this is the CLI pattern we went through. It's using Slate MCP, create new Slate environment, and it's saying, 'This is the environment name.' Once it creates the environment, it's saying, 'Deploy service to this environment,' and giving the service name. This is a test service. Now it has its own environment. This is the Slate UI. If you see here, you can pin test account UIDs to that particular instance, and you'll see all the services that are in that particular environment. You can see the Git SHA. Now the agent internally can use all of these skills and tools.
If you see, what it did was it did A/B testing of that web thing. It ran agent browser in that Slate environment with what is on main and what is on my branch, and it created a report for me to review the full set of changes that happened. After this, we can go to a report that it generates. See, it generated a report that shows before and after, and it's showing me there's no regressions. This is very easy for engineers to validate, and we can attach some of the screenshots and videos to the PR. The same thing we have for iOS, Android, web, and if you want to do backend APIs, then you can also test backend APIs.
I just want to add, one of the things that, again, you might not appreciate if you're outside of Uber, but Slate is so important because Uber literally still today has thousands of microservices, or maybe not micro, but to run a change on just one service when you want to do end to end, that's
Just difficult, right? This is why Slate took years to build to start with.
Yeah, and you also have to think about the routing of the pinned accounts, right?
And the pinned accounts: this is the test account that I want you to route to my branch that is just deployed in a staging environment. That's not trivial at all. In fact, how this works is one of the things that we built back in maybe 2017, '18. It's called Jaeger, which is a distributed tracing infrastructure. It's open source now.
All of our services have Jaeger spans. Essentially, wherever the call originates, there's a parent span, and at every hop there's a child span. You can add any baggage on that trace, which can be propagated across the call stack. So when you pin from the Slate web, we are essentially adding, saying, hey, these are the services under test, and this is the account that I want to route. As it is going through our service mesh, the service mesh knows this baggage, and then it can say, oh, I want to route this request not to this environment but to the Slate environment.
Because we had tracing built out, and because it's propagated across the stack, now you could actually test, and we can reliably route these kinds of test requests to the right environments so that it makes it really easy to test.
This thing took years to make sure that all existing services were retrofitted with this and all new services were built with this in mind. So I'm starting to sense a trend where Uber built stuff over years, pre-AI, just infrastructure that is easy to test for humans or easy to automate. And now with AI, it seems like, oh, you just built this the last year, but actually you built on top of a foundation that was built for years, right?
Yeah, I'm sure. I think if we did not have a system like Slate, it would have been really hard for agents to self-validate. Then you cannot automate this thing. As you said, this kind of stuff really helps. I'll also talk about one more pretty interesting thing which is so Uber-specific.
What is Capri?
Yeah. And this is very, very hard to get right. We had even many versions built like this in the past, but I think this is probably our best version, and we can really scale it to all the teams. Essentially, the idea is context, because even if you look at our core rider app, right, we have hundreds of screens.
And the rider app is what most people think of as the Uber app, right? The app.
Yeah. We have different businesses, different product lines, different cities, a whole bunch of dimensions. If I want to test any new thing, how do I know every combination of these has no regression? And if you were to create all permutations across this, you would have to maintain thousands and thousands of tests across the stack, and that alone is, again, a really hard problem in itself.
So for some of our most complex platforms, we call it capture replay. Essentially, you can capture real production traffic with just one middleware, because now we have standardized language frameworks across all the services. It's very easy to hook this into, and it will capture not just the inbound and outbound at that service layer, but also any of the downstream APIs that service calls. Those requests and those responses are also captured. So in some sense you have the full request-response and all of your outbound request-responses captured.
Now, with all of these captured requests, they go through some pipeline, and then you can build simple rules that say, oh, if you see this kind of field in the request, this is the value for that dimension. So then you can capture all kinds of dimensions that that platform team cares about. And once you have all the requests and all the dimensions, you can, in some sense, generate the full set of live dimensions and live combinations that that platform is currently supporting, which we call MatchGuard.
That I can run pre-deploy in our CI/CD that shows, for all of the PRs that are going in this deploy, let me validate against the full combinatorial set of combinations this platform supports and show that delta before and after. So then you get really high confidence: am I breaking maybe cash trips only in Brazil with this configuration, which would have been really hard for engineers to manually test, or agents also to manually test?
So step one is you capture what you want to capture, and then you have the system that will rerun it, check that there's regressions.
Yeah. Then I'm going to one of the domains that has this enabled. So you can see all the sessions that it captures. I can go to any of the sessions. Then you can see all of the metadata and tags. Essentially, it's saying, oh, this request is tagged as marketplace type Eats. It's tagged as line of business Eats. It is in this kind of terminal state. It captured all the requests. You can also see all of the outbounds that it used. If we click on that, it will also show you the request and response for the outbounds.
And if you see at the top, there is Slate workflow UID or search environment name. So then let's say I'm working on a bug that's only affecting, for some reason, San Francisco Uber Eats trips. I fix that bug, then I can create a Slate environment for that bug, for my code changes. I can go here and then say, replay just this request on my Slate environment and show me the delta here.
Show me the difference, and you're expecting a fix or some change.
Yeah, I'm expecting the fix. Because I know the before request, after request, and then I can do it for a specific dimension. And then we also generate what we mentioned, MatchGuard, which is the full set of dimensions possible for that. So then you can create, in some sense, an automated testing suite for that team, and also a more easy way to manually test one specific combination that you are focused on.
When I was there, we didn't have this, but we had something called Deputy and Cerberus, but those were, I guess, very early versions.
Yeah. In fact, what Deputy and Cerberus allowed was the ability to route any—what Deputy allowed was the ability to route any request from production to your testing instance, and Cerberus allowed from your local testing instance back to the staging servers. But now that we have Slate, we don't necessarily have to use Deputy and Cerberus, because you can deploy a Slate environment, and we don't need to use any of them.
I just want to stress, because I think it's hard to see from the outside, but it's really hard to build this because, again, you have the thousands of services. All of them have to support these things, from routing, for example, to capturing production to replaying, right? This one's a lot of work, I assume.
Yeah. And now I think, similar to how we exposed Slate to agents, we can expose all of the Capri functionality to agents. So agents can check, oh, I'm trying to fix this. Let me get all of the sessions of that type and then see what a request looks like. Can I compose a request of that type on my Slate instance and see, did it fix? Did it not fix? Let me go in a loop until I fix this.
Yeah. And now the agent can, even though it cannot see, it can see the delta, and it can tell that nothing has changed, for example. So it can get feedback loops.
Exactly. I think that helps a lot. Now the agent can also do capture replay on Slate with any of these. We already talked about the Edge Gateway.
Edge Gateway, that's just kind of Uber's internal API, right? It used to be RTAPI, real-time API, but now it's called Edge. It's a Go version. It was rewritten in Go, if I remember right.
Yeah, it's in Go, and now it supports a whole bunch of endpoint types. So there's external API, whether it's gRPC, webhooks, GraphQL. Again, all APIs go through one single entry point where we can build a whole bunch of tooling around that, on monitoring, on alerting, and all the reliability features.
Edge also has its own capture replay, similar to Capri. It's using Capri behind the scenes. If you want to look at reliability of anything, we look at the entire service graph for that service and tell you what is the potential unhealthy node that we need to look at. With Jaeger tracing, we know exactly the service dependencies. We've also automatically tagged which is a required dependency, which is an optional dependency. So that metadata is also there. Even when we're looking at alerts and diagnosing alerts, all of this rich context really helps both engineers to debug faster and also agents to debug faster.
Yeah, we already talked about experiments. The last thing maybe we'll cover is what we call Shepherd and Fox. After we moved to monorepo—we moved to monorepo because upgrading libraries was very hard, right?
Yeah, on all of the different repos. Yeah.
Exactly. Then you move to monorepo, but now someone has to go and upgrade this library usage across all of the modules in that repository. So how do we automate that? We first took a stab at using OpenRewrite policies, saying, okay, if it's a simple signature change, you can write that in an OpenRewrite policy, and then Shepherd knows what are the unique modules owned by different teams, and then it automatically creates a PR for each of those modules.
When AI came into the picture, then we could add an AI-based Shepherd where, okay, if I want to fix context propagation across all the services so that tracing works better, I can create a set of prompts and skills that the agent can run across every module and fix all the issues in every service faster. So this is our large-scale changes, whether you want deterministic changes or whether you want agentic changes across the modules.
We were able to do a lot of migrations over the years through this, and we call each one a campaign. If you see JUnit 4 to JUnit 5, there will be a YAML file with the policy, and you'll see a whole bunch of PRs created for each module. A campaign owner can track the status of that and go through that. So it makes it really easy to manage these migrations.
A few months ago, we took the same platform and added one more feature. We also want to make it easy. We want to continuously improve the codebase, not just one-time migrations. Let's say I want to—
So, code maintenance.
Yeah, code maintenance. So we call it Fox. Essentially, we want to create self-improving codebases, since we already have a whole bunch of agent skills. Essentially, any service at Uber, they can go here, enroll their service into different skills. They can say, I want—
Dead code cleanup.
I want dead code cleanup or performance analysis or noisy logs. There's a whole bunch of these skills that you can onboard to.
Oh, smart.
And what Fox does is every Monday, or essentially every Sunday, when we don't have that much CI load, it will generate a new PR for this service using dead code cleanup. It says, oh, can I clean up any code? Can I improve the performance of this? Can I fix any noisy logs? So it's automated codebase improvement, showing you how you can—
I like it because I heard people do this thing where they would set an AI loop, let's say in Claude Code or Codex, saying, look for ways to improve my codebase, and they would execute it every night or once a week. But it sounds like this is just a lot more optimized. And I love that you can enroll your codebase, because everyone knows a codebase is owned by an engineer or team at Uber, right?
Yeah, exactly. And you can see, because we're rapidly scaling, we're still in the opt-in phase. Once we improve some of the skills, we'll move some of them to the opt-out phase. But anytime you opt in a skill for a service, then every week, in some sense, you are getting a PR created. It's like a managed agentic loop. You are continuously improving this.
In some sense, the more toil we reduce, the more time engineers can spend on the true feature work. So constantly thinking of, okay, how do I eliminate more toil from engineers and automate all of these things? We constantly look at all kinds of PRs, maintenance-category PRs. Do we need to create a new skill? Do we need to improve that skill so that that class of PRs can be onboarded to this automated managed loop?
This is nice. I like it. And then finally, I heard about Wisdom. What is Wisdom?
Oh, yeah. We use Wisdom because all of our employees dogfood our products. So if you're using our rider app or earner app, and if you see any bug, then you can report that bug from the app. We now have all kinds of AI stuff where it will automatically route it to the right team based on the stuff. And when you report a bug, because it's from the employee beta apps, we can capture a whole bunch of logs and telemetry around that. That makes it really easy for the team owning that feature to fix it.
So in this case, it's an internal feature. Essentially, if you want to test any feature in a different country, you can mock the location on the device. So if someone reported that issue, the first thing is, okay, now how do you reproduce that? Before, it used to be a huge problem to reproduce. Now we know the set of screens the user has gone through, because we have that in the logs. And then we can give the engineers: these are the reproduction steps, and this is the expected behavior, and this is the observed behavior, so that we can boil down to the issue quickly.
And all of the telemetry that we collected from the app, you can see all the logs in one place, whether it's analytics, network requests, all kinds of feature flags. So that also gives a lot of context. And now that we have all of this context, we built a managed agent called Debug Assist that looks at all of this, looks at the codebase, and it will try to pinpoint what is the root cause and what is the impact, evidence, timeline around that.
Now, once I think, okay, this is the right root cause, I can hand this off to Minion that we talked about. Hey, coding agent, this root cause looks good. Now why don't you create a PR for this? Minion will take all of that context that we have. It creates a PR. It will attach that PR to the Jira task. The Jira task also has the root cause analysis, so anyone who's working, they know the root cause analysis.
If someone updated the root cause in the comments or created a new PR, again, all of that is data that we can then figure out, okay, how do we improve Debug Assist? How do we improve Minion? We didn't get the right RCA, or we didn't get the right PR from the RCA. Both we'll be able to continuously improve.
And then, do I understand correctly that you can capture and reproduce bugs so much easier, right?
Yeah. And we also added the ability to, because for the employee-reported bugs we have all the logs, I can take my test device, and I can ask it to simulate what that employee has experienced. So once I give the bug UID, we can essentially serve the same network responses that we served to the person who filed, so that I can experience the same issue in my test device app too. So that way I know, okay, once you fix it, then I can see, okay, this actually got fixed, and then it—
Makes it really easy to reproduce. I love the recording of the thing because that's so important, right? It's hard enough to do even on the web. You need to build custom tools for yourself if you want to do a website. But that's awesome that you did it.
Yeah, I think this helps a lot in reproducibility and getting to the right root cause.
So you built all of these systems, and especially last year, it seems, so many brand-new systems that are just doing things like they're maintaining the code quality, agents are validating themselves. How do you think the job of software engineers inside of Uber is changing, and what are you learning from all of these new tools? It seems like you're learning yourself.
In some sense, if you're in the agentic AI world, you're constantly trying to find the next bottleneck and trying to remove that, and figuring out how you can, both using AI and using actual systems engineering, reduce that bottleneck. Because ultimately our goal is to maximize the number of features that we can ship across all of our products. And that will require us to look at the entire life cycle: how do we rewire, prepare that for the new world, and what kind of tools do we need to build? I think this is going to be an ongoing journey.
There's a lot of discussion of this concept called capability overhang, where I feel like even right now there's so many things that we could leverage the latest models for, but we don't have the sufficient orchestrating capabilities or the harnesses or the tools around that to leverage those models in the right places, in the right way. I feel like the more we work on this, the more we can unlock, the more features we can ship, the more productivity that we can unlock. We are trying to measure a whole bunch of this and constantly think about what is the next set of foundations that we need to build to make it easy for engineers to work and to make it easy for agents to work.
You mentioned productivity. How do you think about engineering productivity? Because when I was at Uber, this was again up to 2020, us managers did look at things. It wasn't the main thing, but we did look at things like diffs per engineer, just to show if someone didn't contribute for two weeks, it was like, hmm, what are you doing here? Or if someone did a lot of code reviews, that's actually what we wanted to notice. But it sounds like a lot of those things, the code output, the code artifacts, are less relevant potentially. What is a productive engineer in your mind right now inside of Uber or inside of your team?
Yeah. So I think before the AI world, we were measuring the standard stuff like code throughput and stuff like that, right? Those are the easy ones. And then we also have, every few months, we do the developer experience surveys. We look at NPS, sentiment across all kinds of dimensions. We look at our code quality dimensions. What's our incident per diff? What's our code coverage and all of these metrics, right?
I think now, once you got to AI, I feel like we can leverage the new set of capabilities to get more nuance into this discussion. Because now we are internally talking about not necessarily just diff velocity, but feature velocity. How much of all of this additional throughput is going towards unlocking new features, and how can we even change our organizational processes so that we can reduce the P50 feature time? Because you can create 100 diffs, but if they're not all feature diffs, then maybe it's not necessarily increasing the value that we can provide to the users, right?
So, for example, one of the things we did is tagging all of our PRs to different categories, on complexity level, and is it a feature PR, bug fix, maintenance, testing, documentation, on different categories. And we could see year over year how is that mix changing. Luckily, at least right now, our percentage of feature PRs is going up. Our percentage of maintenance PRs is going down. So that gives us a good indication that, okay, we're in the right direction. We see the code volume go up and the percentage of PRs for features going up.
Then we are investing a lot because before, even though we had the uPlan tool, sometimes you might not have the right tracing from this product design doc to this engineering design doc to this particular task. So it becomes a bit harder to measure what is the cycle time end to end reliably across all the features. So now we're using AI to automatically cluster these things and tag any missing things so that we can reliably get what is the P50 feature velocity for different complexity of features for every half over the last two years, three years. And then we can go into, okay, is the bottleneck in the PRD phase? Is the bottleneck in the launch phase? Is the bottleneck in the rollout phase? What is going on there, and what kind of things do we need to invest in to accelerate the number of features that we can ship per half?
And how is your mental model changing of software engineering compared to pre-AI and these tools? Is it still the same job? Have things changed?
At least in my total tenure in engineering, there's never been this much change to how I work. For example, we say all the engineers at Uber are hands-on. Even distinguished engineers at Uber are hands-on, and they create PRs. I used to think I was a hands-on engineer, but my hands-on definition year over year has completely changed. Even personally, I'm doing way more PRs than what I was doing last year. It's more easy to do all of these things today, right? And I'm able to explore more ideas than what I could possibly do in the past.
But the job of engineering is, in some sense, how do you build the right context for the agents to produce reliable output? In some sense, you need to get really good at context engineering. Before, I was operating with some context to do some task. So I'm always thinking about, okay, if I want to offload some of this work to an agent, what kind of context would it need, and how do I set up the environment with that right context so that it can produce that work? Then you have to become really good at how you give the right specs, right constraints for the agents, apart from building the right context.
Then multitasking has become really a thing. Before, it would take so much time to create a PR, so at least you're laser-focused on that. But now you're constantly switching between so many things, so you need to become really good at multitasking across all the sessions, figuring how do you scope the work, how do you track the work, how do you verify the work really quickly? Because if you're not able to verify the work and pick any course from the agent, then that's hard. Then you're shipping some regressions into production, right? So context engineering, ability to multitask, ability to review and verify the outputs, these things have become much more important now because you're able to parallelize a lot more than before.
One thing I'm picking up is maybe a new primitive is we used to build software thinking, how do I build this software and how do I verify it? Whereas now the question that you said is often, how do I set up an agent to be able to build this and to be able to verify it? So it's kind of almost a meta—
Exactly.
Sounds like there's more of that, right?
Yeah. I'll give you an interesting example. I was working with some of the engineers that work on our customer support team. We are going through a journey of having AI agents handle some of our customer support policies. That process would go from, oh, you have a runbook that human agents would follow. So then from that runbook, you are going through a whole bunch of steps to create a policy that an AI agent can follow. Then you have to think about, okay, there are some dynamic elements here. Then which APIs do I need to call to get that data? Which attributes do I need to use? Then I need to configure my system, change a bunch of configurations. Then I need to do manual evals before launching. Then I need to do continuous evals post-launch.
And then in the last few months, we are able to take down all of the steps. Okay, how do I now take all of the context and all of the principles that I want to use for the policy conversion and teach agents to be able to do that so that I have a really good v0 policy that an agent can create? Then you automate all of the configuration setup, manual testing, finding the right APIs, finding the right properties so that we can hook the right dynamic elements into this. Again, that used to take weeks. Now it's on the order of days for any policy.
Then instead of doing manual evals, how can you use all of the past context to create synthetic evals on top of that so that you can accelerate, you can deploy this and then verify this quickly? Essentially what we're discussing, right, which is now, instead of doing that work, how can you build the right tools and harnesses on top of this so that you can automate the workflow? Obviously, you need the right guardrails and human verification steps in between, but you're designing a system that can do these things instead of relying on so much manual work across all of these steps.
Yeah. And it sounds like you're learning a lot because these are all new concepts which you're now using. You're learning about it. You're doing limitations new. I see what you mean by more learning, more PRs. And as closing, what is a tool that you really like using? This can be a digital tool, this can be a non-digital tool, something you find helpful.
Yeah, the last few days I've been hooked to some of the personal agents like Instinct, Mio, Grocks of the world. I'm playing around with all of them. It's helping automate a whole bunch of manual work that I do in my personal life. And I can, in some sense, just follow the same process. Give it all of the context it needs to do that task for me. Give me all of the inputs I need. Then I can have a human-in-the-loop verification step and do that task. But it's helping a lot to automate all of the manual personal work.
Awesome. Well, Uday, thank you so much for walking through all of the internals that are being used at Uber. I found this super interesting, especially I haven't seen this for quite a while.
Yeah, thank you so much for giving us the opportunity. I feel like we're still pretty early in this. There's so much to work on, and there's so much to build. And we're super excited to be on this journey to build the right stuff and to see how software engineering changes and how, at the end of the day, to unlock the value for the company, value for our business, by working on the right set of things.
I hope you enjoyed this look into Uber's tooling as much as I did. It's now been six years since I left Uber, so a lot of the tooling that Uday showed me was also new to me. Uber was known for having about 5,000 services with about 2,000 engineers ten years ago. But since then, the company started to consolidate services into well-sized services. And today, it has fewer microservices than they did a decade ago. Today, it's in the ballpark of 4,500 to 5,000, even though these services are not as micro, and they're much more well-sized instead.
One piece of software that I really liked is Capri, the event capture and replay software. Back in my time at Uber, between 2016 and 2020, we had two tools called Deputy and Cerberus that could do some level of production data replay, but it wasn't all that easy to use, and it was just really hard to capture replays to start with. With Capri, it's far more trivial to capture sessions and replay them. And this works really well to reproduce bugs and also to test that a fix worked for a certain bug fix. Building a system like Capri is especially complex when you have closer to 4,500 or 5,000 services running like Uber has.
Another thing that was interesting is how we're seeing where code review is headed. It's slowly dying everywhere. But looking at the tooling Uber built around code review with Code Inbox, every PR still gets merged after a human engineer decides that it's ready to do so. But Uber is building a lot of automated checks. Many of them are with AI to give more signal about the riskiness of the change and the quality that AI agents make of it. And so in the end, it's a lot easier for engineers to decide which PRs they should spend time looking at in depth versus ones that are safe to approve with a simple LGTM, aka you might decide you're not going to read the code.
Finally, I had the sense that Uber investing ten years in developer productivity is really paying off with their AI integrations. For example, they built cloud development environments called DevPods back in 2017. Uber got to reuse almost all of its infra in building out their cloud agents called Minions. The same is true for existing systems to validate software, to safely roll them out. And now agents can use these as tools or as feedback loops. So it feels like companies that are getting some of the most benefit from AI are the ones that have had developer platform teams pre-AI as well, at least in the case of Uber.
Do check out the show notes for more of The Pragmatic Engineer deep dives on Uber's engineering culture. Uber is probably one of the companies I have covered in the most detail so far. If you've enjoyed this podcast, please do give it a rating and tell your colleagues and friends about it.
Article published · Updated