Why AI Products Fail: Non-Determinism, Agency vs. Control, and Continuous Calibration
Lenny's Podcast
Aishwarya Naresh Reganti ("Ash") and Kiriti Badam have worked on or supported more than 50 AI product deployments at companies including Amazon, Databricks, OpenAI and Google, as well as startups and large enterprises. Kiriti works on Codex at OpenAI and previously spent about a decade building AI and ML infrastructure at Google and Kumo. Ash was an early AI researcher at Alexa and Microsoft, has published more than 35 research papers, and now runs a consulting practice and startup. Together they teach a course on building AI products.
In this conversation with Lenny, their argument is that AI products fail when teams build them like traditional software. They say two properties change the development lifecycle: non-determinism, and a trade-off between the agency given to an AI system and the control the humans keep. From those two ideas they derive a practice they call "continuous calibration, continuous development": start with narrow autonomy, log what humans do, watch for behavior you didn't expect, and expand what the system is allowed to do only after it has earned trust.
What changed between 2024 and 2025
Ash began with what she sees from the ground in her consulting work. In her view, 2025 differed from 2024 mainly because skepticism dropped. In 2024, many leaders suspected AI might be "yet another crypto wave." Many of the use cases she saw were essentially chat over a company's own data, marketed as an AI product. In 2025, she sees many more companies rethinking user experiences and workflows. They accept that building a successful AI product requires deconstructing and reconstructing existing processes.
The bad news, as she sees it, is that execution is "still all over the place." She notes the field is about three years old and has no playbooks or textbooks. The AI lifecycle before and after deployment differs from the traditional software lifecycle. As a result, old contracts and handoffs between PMs, engineers and data people have broken down. Previously each role optimized its own feedback loop. Now, she says, these people probably need to sit in the same room, look at agent traces together, and decide how the product should behave. Companies are still learning to work in that tighter way.
The first difference: non-determinism on both input and output
Ash stressed that AI systems and conventional software share many similarities. But she named two differences that she says fundamentally change how products are built. The first, which she thinks most people ignore, is non-determinism.
She used Booking.com as the contrast. In traditional software, a user who wants two nights in San Francisco has an intention. The product is a well-mapped decision engine that converts that intention into an action through buttons, options and forms. In AI products, that layer is replaced by a fluid, mostly natural-language interface. Users can express the same intention in countless ways, so the builder no longer knows how users will behave. On the output side, the builder works with an LLM, which she describes as a non-deterministic, probabilistic API that is sensitive to prompt phrasing and essentially a black box. So builders are dealing with an input, a process and an output, none of which they understand well. They are trying to anticipate behavior and build for it. With agentic systems, she says, this gets even harder.
She also offered the upside of the same property. People are more comfortable talking than clicking through buttons, so AI products lower the bar to use. But there are many ways to communicate. Most of the surrounding systems are deterministic, and teams want deterministic outcomes from a non-deterministic technology. "That's where it gets a little messy," she said.
The second difference: the agency–control trade-off
The second difference is what they call the agency–control trade-off. Ash said she is surprised how few people discuss it, given how obsessed many teams are with building autonomous agents. Every time decision-making capability is handed to an agentic system, she argued, the builder gives up some control. More agency means less control. So the builder must make sure the system has earned that ability, meaning it has become reliable enough and has built up trust over time.
Kiriti framed the practical consequence with an analogy. If the goal is a very long, hard hike, you don't attempt the full route on day one. You train on smaller parts and build up. Similarly, he argued, teams should not start with agents wired to every tool and all of the company's context and expect them to work. Teams should deliberately start where impact is minimal and human control is high. That way they learn what current capabilities are, what problem the AI actually solves and to what extent, and what context and tools to add next. He sees a good side to this. Nobody needs to feel intimidated by impressive-looking agents elsewhere, because everyone starts from minimal structures and evolves them. He says this pattern repeats "over and over" in their work.
Climbing the ladder: examples of graduated autonomy
Kiriti's main example was customer support, which he said OpenAI itself faced. Launches such as the image model or GPT-5 caused spikes in support volume and brought new kinds of questions. The wrong move, he said, is to dump every help center article into an agent. A staged approach starts with AI suggesting answers to human support agents. The humans signal which suggestions are good or bad in particular cases. That feedback exposes the system's blind spots so they can be fixed. Only then does the system graduate to answering customers directly. After that, new capabilities can be added, such as issuing refunds or filing feature requests with engineering. Starting with all of this on day one, he said, makes the complexity "incredibly hard to control."
Ash described the higher-level idea as "behavior calibration." It is effectively impossible to predict up front how the system will behave, so the goal is to avoid damaging the user experience while gradually reducing the amount of control humans need to exercise. There is no single right way to constrain autonomy. Her example was insurance pre-authorization, which she called a very promising use case because clinicians spend a lot of time pre-authorizing blood tests, MRIs and similar procedures. Blood tests and MRIs are relatively low-hanging fruit, since approval is easier once the patient's information is known. Invasive surgery is higher risk and should not be approved autonomously. Teams can decide which cases go through a human-in-the-loop layer and which the AI can handle. Throughout, they log what the humans do, to build a flywheel that improves the system without eroding trust.
Lenny read two further progressions from their earlier guest post. For a coding assistant, the stages run from inline completions and boilerplate, to larger blocks such as tests or refactors for human review, to applying changes and opening PRs autonomously. For a marketing assistant, they run from drafting emails or social copy, to building and running multi-step campaigns, to launching, A/B testing and auto-optimizing campaigns across channels.
Why starting small forces a "problem-first" mindset
Lenny suggested that teams get into trouble when they jump straight to the V3 stage. Kiriti agreed. When an agent is doing things wrong "in like 100 different ways," teams can't realistically tabulate and fix all of them, even with good evaluation practices. Starting in the wrong place makes correction very hard.
He pointed to a less obvious benefit of starting with high control and low agency: it forces teams to think about what problem they are actually solving. They use the term "problem first." Kiriti said the idea seemed obvious to him, yet it resonates strongly with people. Amid rapid AI progress, he said, an easy slippery slope is to keep thinking about the complexities of the solution and forget the problem. Starting at a small scale of autonomy makes teams break the problem into levels of autonomy they can build toward later.
Ash cited a recent paper from researchers around Matei Zaharia at UC Berkeley and Databricks. According to her reading, about 74–75% of the enterprises surveyed named reliability as their biggest problem. That was why they weren't comfortable deploying customer-facing products. The authors also connect this to why so many current AI products focus on productivity, a lower-autonomy category, rather than end-to-end agents that replace workflows. She said this matches what her startup sees.
A brief detour on prompt injection
Lenny mentioned an earlier episode on prompt injection and jailbreaking, which that guest had described as possibly an unsolvable problem with guardrails that are easy to get around. Ash predicted it will become a huge problem once systems go mainstream. For now, she said, teams are too busy building to worry about security. Because the underlying API is non-deterministic, there are many instructions that can be injected into a prompt.
Kiriti took a more optimistic line. He agreed it is a real problem. But he argued that adoption is still at a very early stage, even though 2025 was a busy year for agents, and that penetration is far below what companies could benefit from. With the right human-in-the-loop checkpoints, he believes many of these risks can be avoided while companies streamline their processes. He said companies need to try adopting AI rather than focusing only on what could go wrong. Among the companies he has talked to at OpenAI, he said, none has concluded that AI cannot help them at all. There has always been some set of things it could optimize.
The success triangle: leaders, culture, technical practice
Asked what distinguishes companies that build AI products successfully, Ash described a "success triangle" with three sides: leadership, culture and technical progress. "Every technology problem is a people problem first," she said.
On leadership, her point is that leaders have built intuitions over 10 or 15 years and are respected for them. With AI, those intuitions must be relearned, and that requires vulnerability. Her example was Gajen Kandiah, now CEO of Rackspace, with whom she used to work. Gajen blocked out 4 to 6 a.m. every day for "catching up with AI," with no meetings, to take in podcasts and other information. Gajen also held weekend vibe-coding sessions. According to Ash, Gajen relied on a short list of two or three trusted sources. Afterwards Gajen would bring questions to a group of AI experts, which she was part of, and said this learning trickled down into company decisions. Leaders need to be hands-on, she argued, not to implement things but to rebuild intuition. They should accept that their intuitions may be wrong and that they may be "the dumbest person in the room." She believes this has to come top-down. Engineers can't win buy-in from a leader who distrusts the technology or has misaligned expectations. She has heard from many builders that their leaders either underestimate what AI can do, or vibe-code something and assume it will be easy to take to production.
Lenny connected this to an earlier conversation with Dan Shipper, who had said the number one predictor of a company's AI success is the CEO using ChatGPT, Claude or similar tools many times a day.
On culture, Ash said many enterprises where AI isn't the core business adopt it because competitors are doing so. That often creates a culture of FOMO and "you will be replaced." This is harmful because subject matter experts are essential. Teams need them to understand how the AI behaves and how it should behave. Yet she has spoken with companies whose experts won't engage because they believe their jobs are being replaced. Leaders should instead build a culture of empowerment, where people add AI to their own workflows to "10x" what they do, rather than guarding their jobs. She added that AI opens up more opportunities than before.
On the technical side, she says successful teams are obsessed with understanding their workflows. They decide which parts suit AI and which need a human in the loop. Automating part of a workflow is rarely just a matter of "use an agent." Usually some part is done by a machine learning model and some by deterministic code, and teams need to pick the right tool for each part rather than fixate on the technology. These teams also understand that working with a non-deterministic API changes the lifecycle. They iterate quickly in ways that don't damage the customer experience but still produce enough data to estimate behavior. "It's not about being the first company to have an agent among your competitors," she said. "It's about have you built the right flywheels in place so that you can improve over time."
Skepticism about "one-click agents"
From that followed a strong claim. If someone promises a one-click agent that will show significant gains within two or three days, Ash would be skeptical. The reason, she said, is not model capability but messy enterprise data and infrastructure. Taxonomies are messy. Tech debt accumulates, with functions like "get customer data v1" and "get customer data v2" both existing and both being called. An agent needs time to understand these systems. She called one-click agent pitches "pure marketing." She would prefer a vendor that builds a pipeline which learns over time. In her estimate, replacing a critical workflow or getting significant ROI "easily takes four to six months of work," even with the best data and infrastructure layer.
Evals versus production monitoring: a false dichotomy
On the evals debate, Kiriti called it a false dichotomy to believe that either evals or production monitoring alone will solve everything. He described evals as a team's product knowledge encoded into datasets: what matters, and what the agent must never do. Production monitoring means deploying and tracking key metrics that show how customers actually use the product. He noted that production monitoring has existed for a long time. With AI agents it needs finer granularity and must include implicit signals. His example was ChatGPT: users may not give a thumbs-down, but if they regenerate an answer, that is a clear sign the first answer missed expectations.
He described how the two fit together. Nobody ships without some testing, whether vibes or a set of ten questions that must never go wrong. Call that an evaluation dataset. After deployment, a high-traffic product can't have every trace reviewed by hand. Explicit and implicit signals tell the team which traces to look at. Examining those traces reveals failure patterns. If a pattern is something the team truly cares about, it becomes a new evaluation dataset. His example was an agent offering refunds even though it had been explicitly configured not to. After fixing tools or prompts and shipping a second version, there is still no guarantee that this was the only problem, so monitoring remains necessary. The view that only one approach is needed is, in his words, "completely dismissible."
Ash added that "evals" had to carry an enormous amount of meaning in the second half of 2025. Data labeling companies say their experts write evals. Commentators say PMs should write evals, calling them "the new PRDs." Others say evals are the whole feedback loop. She argued these refer to different parts of the process. When lawyers or doctors at a labeling company "write evals," they are doing something closer to error analysis or annotation, not building LLM judges. When people say a PM should write evals, that doesn't mean a production-grade LLM judge. She invoked Martin Fowler's term "semantic diffusion": a term gets redefined by everyone until it loses its original meaning. She believes practitioners would all agree on the underlying need, an actionable feedback loop for AI products. How to build it depends on the application. In complex use cases, she said, LLM judges become very hard to build. New patterns keep emerging that a judge, for example one testing for verbosity, doesn't catch. Teams can end up with too many evals. At that point it may make more sense to look at user signals, fix issues, check for regressions and move on. Her advice was to avoid being obsessed with prescriptions, because they will change.
She also told of a client who said they "do evals." When she asked for their dataset, they explained that they checked LMArena and Artificial Analysis to choose a model. Those are model benchmarks, she told them, not evals.
How the Codex team approaches evals
Lenny raised the head of Claude Code's remark that the team doesn't do evals and relies on vibes, and asked how Codex handles it. Kiriti described a balanced approach. He said coding agents differ from agents in other domains because they are built for engineers and for customizability. They are not meant to solve five or six fixed workflows. They are used across many integrations and tools, which makes it very hard to build eval datasets covering every interaction. Still, the team keeps evaluations to make sure changes don't damage core functionality.
At the same time, Kiriti said, the team pays close attention to how customers use the product. His example was a recently launched code review product, which he said has gained a lot of traction and has caught many bugs both inside OpenAI and for external customers. If the team changes the model behind code review or trains it with a different RL mechanism, they want to A/B test whether it finds the right mistakes and how users react. Users annoyed by incorrect reviews sometimes switch the product off entirely, and that is a signal worth watching. He said such scenarios are very hard to anticipate in advance. The team also watches social media closely for reported problems. When a new model arrives, team members each focus on different areas and run a shared list of hard problems against it, which he described as something like custom evals for each engineer. He doesn't believe anyone who claims to have a concrete set of evals they could bet their life on without needing anything else.
The CC/CD framework: continuous calibration, continuous development
The centerpiece of their work is the framework they call continuous calibration, continuous development (CC/CD), named as a nod to CI/CD. Ash explained the backstory. Many companies feel competitive pressure to build fully autonomous agents, and she worked with a few customers on end-to-end agents. Because nobody knew in advance how users would interact or how the AI would respond, debugging a workflow of four or five steps with many decisions became overwhelming. In one customer support project, the example used in their guest post, the team had to shut down the product because of how many hotfixes they were making. There was no way to keep up with the problems that kept emerging. She also cited the Air Canada case, where, as she put it, an agent hallucinated a refund policy that wasn't in the company's playbook, and the airline had to honor it for legal reasons. The framework aims to preserve customer trust, avoid dangerous decisions, and still build a flywheel for improvement.
The development side of the loop starts with scoping capability and curating data: a set of expected inputs and expected outputs. Ash considers this worthwhile before building any AI product, because it often reveals that the team isn't aligned on how the product should behave. This is where PMs and subject matter experts contribute a lot. The dataset isn't comprehensive, but it's a starting point. The team then sets up the application and designs evaluation metrics. She deliberately uses "evaluation metrics" rather than "evals": evaluation is the process, and metrics are the dimensions measured during it. Then the team deploys and runs those metrics.
The calibration side deals with behavior nobody expected. Users almost always behave in ways the original dataset didn't anticipate. Metrics offer some insight, but new error patterns often appear that the metrics don't cover. The team analyzes behavior, spots error patterns, applies fixes, and designs new metrics for emerging patterns. Not every error needs a new metric, though. A one-off tool-calling failure caused by a poorly defined tool can simply be fixed.
Across these iterations, the framework calls for low-agency, high-control versions first, with autonomy increased over time. Ash walked through the customer support example in three versions.
Version 1: routing. The agent classifies tickets and routes them to the right department. It may sound trivial, but Ash said routing in enterprises can be very complex because of messy hierarchical taxonomies. In one case she worked on, "shoes," "women's shoes" and "men's shoes" all sat at the same level. Further down, another section under shoes said "for women" and "for men" and was never merged. A human agent might notice that a node was last updated in 2019 and treat it as dead. She said she isn't claiming models can't handle this. Her point is that enterprises have undocumented rules, and agents need that context rather than simply being handed the problem. Routing is a high-control stage, since humans can correct misroutes. It also surfaces data issues that must be fixed so the data layer is good enough.
Version 2: copilot. Once routing works and data issues are fixed, the agent drafts responses based on the support team's standard operating procedures, and human agents edit them. The system logs how much of each draft was used and what was omitted. Ash called this getting "error analysis for free," which feeds back into the flywheel.
Version 3: end-to-end resolution. When drafts are mostly used as-is with few edits, the agent drafts a resolution and can also close the ticket.
Their table for each version lists what is done, what is learned and what feeds back into the loop. Routing, for example, produces better-quality routing data and shows how prompts should be structured, which she calls figuring out the structure for context engineering.
Ash added two caveats. First, CC/CD doesn't solve problems once and for all. A team may reach V3 and still meet a data distribution nobody imagined. It is one way to lower risk by learning how users behave before reaching full autonomy. Second, to people who ask why this is needed when evals exist, she answered that evaluation metrics catch only the errors a team already knows about. Emerging patterns show up only in production, and the framework provides a lower-risk way to find them without facing a flood of errors at once. Autonomy can be constrained in other ways too. The example constrains the number of actions, but teams could constrain by topic, since some domains are too risky for full autonomy while others are fine. PMs, engineers and subject matter experts should decide this together.
Knowing when to move to the next stage
A common question is how to tell when a stage is calibrated enough. Ash said there is no rule book, but the principle is "minimizing surprise." If a team calibrates every day or two and stops seeing new data distribution patterns, and user behavior is consistent, the information gained is low and it is probably time to move on. She acknowledged that at that point "it's all about the vibes."
Events can also throw calibration off. Her example was GPT-4o being retired, including in the API. Companies must switch to GPT-5, which she said has very different properties, so they need to recalibrate. User behavior also changes over time. People talk to ChatGPT differently now than two years ago as capabilities have grown, and people who see a system solve one task want to try it on others.
Her clearest example was a system built for a bank's underwriters. Loan applications can run 30 or 40 pages, and the system helped underwriters find relevant policies and bank information to approve loans. For three or four months everyone was impressed, and underwriters reported time savings. Then, she said, they became so enthusiastic that they began asking questions nobody had anticipated. They would drop in an entire application and ask what previous underwriters did "for a case that looks like this." To users this felt like a natural extension. For the builders it required very different engineering. The system had to interpret what "a case like this" means, whether a certain income range or a geography, then retrieve and analyze historical documents rather than simply look up a policy. Something that seems natural to an end user can be very hard to build, she concluded, and when behavior evolves like this, it's time to recalibrate.
Overhyped and underhyped
Kiriti said he wouldn't call anything overhyped, but he thinks multi-agent systems are misunderstood. The common idea is to split a complex problem into agents by function, connect them, and expect "agent utopia." He agreed there are successful multi-agent systems, and named the supervisor agent with sub-agents as a successful pattern. What he doubts is dividing responsibilities by function and expecting agents to coordinate peer-to-peer through something like a "gossip protocol." He doesn't believe current building practices and model capabilities are there yet. Humans can orchestrate multiple agents, or one larger agent can orchestrate. In peer-to-peer setups, especially in customer support, it becomes very hard to control which agent is responding to the customer, and guardrails have to be placed everywhere.
As underhyped, he picked coding agents. That may sound odd given the online chatter, but he said engineers at ordinary companies, especially outside the Bay Area, have adopted them far less than their potential impact would suggest. He expects 2025 and 2026 to create a lot of value by optimizing these processes.
Ash joked that she would name evals as overrated, then clarified that she means they are misunderstood, not unimportant. Her real pick for overrated was constantly jumping between tools and chasing the newest one. "Building is really cheap today," she said, while design is more expensive. Deeply understanding the problem and the design is underrated, and building for its own sake is overrated.
What 2026 might look like
Kiriti expects progress in background or proactive agents. In his view, AI mostly fails to create value today because it lacks context, since it isn't plugged into where work actually happens. As agents gain context, they can understand what metrics a person optimizes for and what they're trying to do, and then prompt the user back. He cited ChatGPT Pulse, which gives a daily update of things a user may care about. He imagines a coding agent that tells you at the start of the day that it has fixed five of your Linear tickets and the patches are ready for review. He sees this as a strong direction for products in 2026.
Ash is betting on multimodal experiences. She argued that humans are multimodal and language is "probably one of our last forms of evolution." In conversation people constantly adjust based on signals like nodding or boredom, which she described as "a chain of thought behind your chain of thought." Richer multimodal systems could come closer to human conversational richness. She also pointed to mundane work: many handwritten documents and messy PDFs can't be parsed even by the best models today, and better multimodal understanding would unlock a lot of data.
Skills that matter: judgment, agency, persistence
Asked which skills builders should develop, Ash said implementation will become "ridiculously cheap" in the coming years, so builders should focus on design, judgment and taste. Early career years have traditionally focused on execution mechanics. AI can now help people ramp up quickly, and after that, she believes, everyone's job comes down to taste, judgment and what is uniquely theirs. That doesn't require seniority. Her example was a recent hire who showed up to a meeting with a self-built, vibe-coded app to replace an expensive task tracker the team had paid for over years, and onboarded everyone to it. She acknowledged that vibe-coded apps have high maintenance costs and may need replacing as the company grows. But she was struck by the agency and ownership involved. People used to a certain way of working attach a cost to building, and younger people attach a much lower one. She suggested this same eagerness to try new tools also explains why AI products have retention problems. She also predicted "the end of the busy work era": people can no longer sit in a corner doing work that doesn't move the needle. Lenny added an anecdote from Jason Lemkin, who replaced most of a sales team with agents. One salesperson quit after an agent began automatically updating Salesforce from calls, apparently because it would have exposed that the salesperson wasn't doing much.
Kiriti emphasized persistence. With information at everyone's fingertips, anyone can learn quickly, but the real advantage comes from going through the pain of learning, implementing and discovering what works. "Pain is the new moat," he said. Asked to expand, he argued that successful companies in new areas succeed not because they were first to market or had a flashy feature. They went through the pain of identifying their non-negotiables and matching them against the features and model capabilities that could address them. There is no textbook or well-trodden path for this. The knowledge built through repeated trial and error, whether in the form of evals or something else, becomes the company's moat.
Ash's closing advice was to be obsessed with customers and the problem, and to treat AI as a tool. She estimated that 80% of so-called AI engineers and AI PMs spend their time understanding workflows, customer behavior and data, not building the fanciest models. Software engineers new to AI are often surprised by the advice "look at your data," she said, but it has always been true, and she sees it as a major differentiator.
Lightning round
Ash recommended When Breath Becomes Air by Paul Kalanithi, a neurosurgeon's memoir written after a lung cancer diagnosis in their early thirties. She highlighted Kalanithi's reply to Socrates' "the unexamined life is not worth living": "was the unlived life worth examining?" She took it as a reminder for people constantly reinventing themselves in the AI era to pause and live. Kiriti recommended the Three-Body Problem series, especially for its theme of how much abstract science matters to human progress and how devastating it is when that progress is halted, even if the effect isn't noticeable day to day. Lenny recommended A Fire Upon the Deep by Vernor Vinge.
For shows and games, Ash is rewatching Silicon Valley, which she finds eerily similar to the current AI wave. Kiriti is playing Expedition 33. Ash's favorite recent product is Wispr Flow. She values that it transcribes with context, recognizing variables in Codex and turning a spoken instruction like adding three exclamation marks into the actual punctuation. Kiriti named Raycast and the caffeinate command, which keeps a Mac awake so a Codex task can run locally for four or five hours.
Ash's life motto came from her father: "they told it couldn't be done, but the fool didn't know it, so he did it anyway." There will always be data suggesting you'll fail, she said, so sometimes be foolish and go ahead. Kiriti, a self-described overthinker, cited Steve Jobs' line that you can only connect the dots looking backwards, as encouragement to keep moving and experimenting. Asked what they admire in each other, Ash praised Kiriti's calm, grounded nature and ability to anticipate problems, then revealed that the two have been married for four years. Kiriti praised Ash's ability to teach and simplify complex topics, combined with persistence, which he says empowers people in a fast-moving field.
We worked on a guest post together, had this really key insight that building AI products is very different from building non-AI products.
Most people tend to ignore the non-determinism. You don't know how the user might behave with your product and you also don't know how the LLM might respond to that. The second difference is the agency control trade-off. Every time you hand over decision-making capabilities to agentic systems, you're kind of relinquishing some amount of control on your end.
This significantly changes the way you should be building product. So we recommend building step by step. When you start small, it forces you to think about what is the problem that I'm going to solve. In all these advancements of the AI, one easy slippery slope is to keep thinking about complexities of the solution and forget the problem that you're trying to solve.
It's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time.
What kind of ways of working do you see in companies that build AI products successfully? I used to work with the CEO of now Rackspace. He would have this block every day in the morning which would say catching up with AI 4 to 6:00 a.m. Leaders have to get back to being hands-on. You must be comfortable with the fact that your intuitions might not be right and you probably are the dumbest person in the room and you want to learn from everyone.
What do you think the next year of AI is going to look like?
Persistence is extremely valuable. Successful companies right now building in any new area, they are going through the pain of learning this, implementing this and understanding what works and what doesn't work. Pain is the new moat.
Today my guests are Aishwarya Reganti and Kiriti Badam. Kiriti works on Codex at OpenAI and has spent the last decade building AI and ML infrastructure at Google and at Kumo. Aish was an early AI researcher at Alexa and Microsoft and has published over 35 research papers.
Together, they've led and supported over 50 AI product deployments across companies like Amazon, Databricks, OpenAI, Google, and both startups and large enterprises. Together, they also teach the number one rated AI course on Maven, where they teach product leaders all of the key lessons they've learned about building successful AI products. The goal of this episode is to save you and your team a lot of pain and suffering and wasted time trying to build your AI product. Whether you are already struggling to make your product work or want to avoid that struggle, this episode is for you. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It helps tremendously. And if you become an annual subscriber of my newsletter, you get a year free of a ton of incredible products, including a year free of Lovable, Replit, Bolt, Gamma, n8n, Linear, Devin, PostHog, Superhuman, Descript, Wispr Flow, Perplexity, Warp, Granola, Magic Patterns, Raycast, ChatPRD, Mobbin, and Stripe Atlas. Head on over to lennysnewsletter.com and click product pass. With that, I bring you Aishwarya Reganti and Kiriti Badam after a short word from our sponsors.
This episode is brought to you by Merge. Product leaders hate building integrations. They're messy. They're slow to build. They're a huge drain on your road map, and they're definitely not why you got into product in the first place. Lucky for you, Merge is obsessed with integrations. With a single API, B2B SaaS companies embed Merge into their product and ship 220 plus customer-facing integrations in weeks, not quarters. Think of Merge like Plaid, but for everything B2B SaaS. Companies like Mistral AI, Ramp, and others use Merge to connect their customers' accounting, HR, ticketing, CRM, and file storage systems to power everything from automatic onboarding to AI-ready data pipelines. Even better, Merge now supports the secure deployment of connectors to AI agents with a new product so that you can safely power AI workflows with real customer data. If your product needs customer data from dozens of systems, Merge is the fastest, safest way to get it. Book and attend a meeting at merge.dev/lenny and they'll send you a $50 Amazon gift card. That's merge.dev/lenny.
This episode is brought to you by Strella, the customer research platform built for the AI era. Here's the truth about user research. It's never been more important or more painful. Teams want to understand why customers do what they do. But recruiting users, running interviews, and analyzing insights takes weeks. By the time the results are in, the moment to act has passed. Strella changes that. It's the first platform that uses AI to run and analyze in-depth interviews automatically, bringing fast and continuous user research to every team. Strella's AI moderator asks real follow-up questions, probing deeper when answers are vague, and surfaces patterns across hundreds of conversations, all in a few hours, not weeks. Product, design and research teams at companies like Amazon and Duolingo are already using Strella for Figma prototype testing, concept validation, and customer journey research, getting insights overnight instead of waiting for the next sprint. If your team wants to understand customers at the speed you ship products, try Strella. Run your next study at strella.io/lenny. That's S-T-R-E-L-L-A.io/lenny.
Aish and Kiriti, thank you so much for being here and welcome to the podcast.
Thank you. Thank you for having us. Super excited for this.
Let me set the stage for the conversation that we're going to have today. So, you two have built a bunch of AI products yourself. You've gone deep with a lot of companies who have built AI products, have struggled to build AI products, build AI agents. You also teach a course on building AI products successfully, and you're kind of like on this mission to just reduce pain and suffering and failure that you constantly see people go through when they're building AI products. So to set a little foundation for the conversation we're going to have, what are you seeing on the ground within companies trying to build AI products? What's going well? What's not going well?
I think 2025 has been significantly different than 2024. One, the skepticism has significantly reduced. There were tons of leaders last year who probably thought this would be yet another crypto wave and kind of skeptical to get started, and a lot of the use cases that I saw last year were more of chat on your data, right? And that was, you know, calling themselves an AI product. And this year, a ton of companies are really rethinking their user experiences and their workflows and all of that and really understanding that you need to deconstruct and reconstruct your processes in order to build successful AI products, right? And that's the good stuff. The bad stuff is the execution is still all over the place. Think of it, right? This is a three-year-old field. There are no playbooks. There are no textbooks. So you really need to figure out as you go, and the AI life cycle, both pre-deployment and post-deployment, is very different as compared to a traditional software life cycle.
And so a lot of old contracts and handoffs between traditional roles like say PMs and engineers and data folks has now been broken. And people are really getting adapted to this new way of working together and kind of owning the same feedback loop in a way, because previously I feel like PMs and engineers and all of these folks had their own feedback loops to optimize, and now you need to be probably sitting in the same room. You're probably looking at agent traces together and deciding how your product should behave. So it's a tighter form of collaboration. So companies are still kind of figuring that out. That's kind of what I see in my consulting practice this year.
So, let me follow that thread. We worked on a guest post together that came out a few months ago. And the thing that stood out to me most, that stuck with me most after working on that post, is you had this really key insight that building AI products is very different from building non-AI products. And the thing that you're big on getting across is there's two very big differences. Talk about those two differences.
Yes. And again I want to make sure that we drive home the right point. There are tons of similarities of building AI systems and software systems as well. But then there are some things that kind of fundamentally change the way you build software systems versus AI systems, right? And one of them that most people tend to ignore is the non-determinism. You're pretty much working with a non-deterministic API as compared to traditional software.
What does that mean and why does that have to affect us? In traditional software you pretty much have a very well-mapped decision engine or workflow. Think of something like Booking.com, right? You have an intention that you want to make a booking in San Francisco for two nights, etc. The product has kind of been built so that your intention can be converted into a particular action, and you kind of are clicking through a bunch of buttons, options, forms and all of that, and you finally achieve your intention. But now that layer in AI products has completely been replaced by a very fluid interface which is mostly natural language, which means the user can literally come up with a ton of ways of saying or communicating their intentions, right? And that kind of changes a lot of things because now you don't know how your user is going to behave. That's on the input side. And the output is also that you're working with a non-deterministic, probabilistic API, which is your LLM. And LLMs are pretty sensitive to prompt phrasings and they're pretty much black boxes. So you don't even know how the output surface will look like, right? So you don't know how the user might behave with your product and you also don't know how the LLM might respond to that. So you're now working with an input, output, and a process, and you don't understand all the three very well. You're trying to kind of anticipate behavior and build for it.
And with agentic systems, this kind of gets even harder. And that's where we talk about the second difference, which is the agency control trade-off, right? What we mean by that, and I'm kind of shocked so many people don't talk about this. They're extremely obsessed with building autonomous systems, agents that can do work for you. But every time you hand over decision-making capabilities or autonomy to agentic systems, you're kind of relinquishing some amount of control on your end, right? And when you do that, you want to make sure that your agent has gained your trust or it is reliable enough that you can allow it to make decisions. And that's where we talk about this agency control trade-off, which is if you give your AI agent or your AI system, whatever it is, more agency, which is the ability to make decisions, you're also losing some control, and you want to make sure that the agent or the AI system has earned that ability or has built up trust over time.
So just to summarize what you're sharing here, essentially people have been building software products for a long time. We're now in a world where the software you're building is, one, non-deterministic, can just do things differently. Like, you know, as you said, you go to Booking.com, you find a hotel, it's going to be the same experience every time. You'll see different hotels, but it's a predictable experience. With AI you can't predict that it's going to be the exact same thing, the thing that you plan it to be every time. And then the other is there's this trade-off between agency and control: how much will the AI do for you versus how much should the person still be in charge. And what I'm hearing is the big point here is this significantly changes the way you should be building product, and we're going to talk about the impact on how the product development life cycle should change as a result. Is there anything else you want to add there before we get into that?
Yeah, it's definitely like one of the key points that this kind of distinction needs to exist in your mind when you're starting to build. For example, think about if your objective is to hike Half Dome in Yosemite, right? You don't start hiking it every day, but you start training yourself in minor parts and then you slowly improve and then you go to the end goal, right? I feel like that's extremely similar to how you want to build AI products, in the sense that you don't start with agents with all the tools and all the context that you have in the company on day one and expect it to work, or you don't even tinker at that level. You need to be deliberately starting in places where there is minimal impact and more human control, so that you have a good grip of what are the current capabilities and what can I do with them, and then slowly lean into more agency and lesser control. So this gives you that confidence that, okay, I know that this is the particular problem that I'm facing and the AI can solve this extent of it, and then let me next think through what context I need to bring in, what kind of tools I need to add to this to improve the experience, right?
So I feel like it's a good and a bad thing, in the sense that it's good that you don't have to see the complexity of the outside world of all of these fancy AI agents and feel like I cannot do that. Everyone is starting from very minimalistic structures and then evolving. And the second part is, the bad thing is that as you are trying to build these one-click agents into your company, you don't have to be overwhelmed with this complexity; you can slowly graduate. So that's extremely important and we see this as a repeating pattern over and over.
Okay. All right. So, let's actually follow that, right? Because that's a really important component of how you recommend people build AI stuff. AI products, AI agents, all the AI things. So give us an example of what you're talking about here, this idea of starting slow with agency and control and then moving kind of up this rung.
Yeah. For example, a very important or very prevalent application of AI agents is customer support, right? Imagine you are a company who has a lot of customer support tickets. And why even imagine? OpenAI faced the exact same thing when we were launching products, and there was a huge spike of support volume as we launched successful products like image gen or GPT-5 and things like that. The kind of questions you get is different, the kind of problems that the customers bring to you is different. So it's not about just dumping all the list of help center articles that you have into the AI agent. You kind of understand what are the things that you can build.
And so initially the first step of it would be something like, you have your human support agents, but you will be suggesting, okay, this is what the AI thinks is the right thing to do. And then you get that feedback loop from the humans that, okay, this is actually a good suggestion for me in this particular case and this is a bad suggestion, and then you can go back and understand, okay, this is what the drawbacks are or this is where the blind spots are, and then how do I fix that? And once you get that, you can increase the autonomy to say that, okay, I don't need to suggest to the human, I'll actually show the answer directly to the customer. And then we can actually add more complexity in terms of, okay, I was only answering questions based on help center articles, but now let me add new functionality, like I can actually issue refunds to the customers, I can actually raise feature requests with the engineering team, and all of these things. So if you start with all of this on day one, it's incredibly
hard to control the complexity. So we recommend, you know, building step by step and then increasing it.
Awesome. And you have a visual actually that we'll share of what this looks like. But just to kind of mirror back what you're describing, this idea of start with high control, low agency. The example you gave is the support agent is just kind of giving suggestions, is not able to do anything, the user is in charge. And then as that becomes useful and you are confident it's doing the right sort of work, you give it a little more agency and you kind of pull back on the control the user has. And then if that's starting to go well, then you give it more agency and the user needs less control to control it.
Awesome.
I think the higher level idea here is with AI systems, it's all about behavior calibration. It's incredibly impossible to predict up front how your system behaves. Now what do you do about it? You make sure that you don't ruin your customer experience or your end user experience. You keep that as is but then remove the amount of control that the human has, and there is no single right way of doing it. You can decide how to constrain that autonomy, right?
A different example of how you could constrain autonomy is pre-authorization use cases. Insurance pre-authorization is a very ripe use case for AI because clinicians spend a lot of time pre-authorizing things like blood tests, MRIs and things like that, right? And there are some cases which are more of low-hanging fruits, for instance, MRIs and blood tests, because as soon as you know patients' information, it's easier to approve that and AI could do that, versus something like an invasive surgery, etc. is more high-risk. You don't want to be doing that autonomously. So you can kind of determine which of these use cases should go through that human-in-the-loop layer versus which of the use cases AI can conveniently handle. And then all through this process, you're also logging what the human is doing, right? Because you want to build a flywheel that you could use in order to improve your system. So you're essentially not ruining the user experience, not eroding trust, at the same time logging what humans would otherwise do so that you can continuously improve your system.
So let me give you a few more examples of this kind of progression that you recommend. And the reason I'm spending so much time here is this is a really key part of your recommendation to help people build more successful AI products, this idea of start slow with high control and low agency and then build up over time once you've built confidence that it's doing the right sort of work. So a few more examples that you shared in your post that I'll just read. So say you're building a coding assistant. V1 would be just suggest inline completion and boilerplate snippets. V2 would be generate larger blocks like tests or refactors for humans to review. And then V3 is just apply the changes and open PRs autonomously.
And then another example is a marketing assistant. So V1 would be draft emails or social copy, just like here's what I would do. V2 is build a multi-step campaign and run the campaign and then launch, and V3 is just launch it, A/B test it, auto-optimize campaigns across channels.
Awesome.
Yeah.
And again, just to summarize where we're at, just to give people the advice we've shared so far. One is just important to understand AI products are different. They're non-deterministic. And he pointed out, and I forgot to actually mirror back this point, both on the input and the output. The user experience is non-deterministic, like people will see different things, different outputs, different chat conversations, different maybe UI if it's designing the UI for you, and also the output obviously is going to be non-deterministic. So that's a problem and a challenge, and then...
I mean, if you think of it, it's also the most beautiful part of AI, which is, I mean, we're all much more comfortable talking than following a bunch of buttons and all of that, right? So the bar to using AI products is much lower because you can be as natural as you would be with humans. But that's also the problem, which is there are tons of ways we communicate. And you want to make sure that that intent is rightly communicated and the right actions are taken, because most of your systems are deterministic and you want to achieve a deterministic outcome but with non-deterministic technology, and that's where it gets a little messy.
Awesome. Okay. I love the optimistic version of why this is good. Okay. And then the other piece is this idea of this trade-off of autonomy versus control when you're designing a thing. And what I imagine you're seeing is people try to jump to the ideal, like the V3, immediately, and that's when they get into trouble. It's probably a lot harder to build that and it just doesn't work, and then they're just like, okay, this is a failure. What are we even doing?
Exactly. I feel there's a bunch of things that you actually have to get confidence in before you get to V3, and it's easy to get overwhelmed that, oh, my AI agent is doing these things wrong in like 100 different ways, and you're not going to actually tabulate all of them and fix it, right? Even though you've learned how do you deal with the evaluation practices and stuff like that, if you're starting on the wrong spot, you are actually going to have a hard time correcting things from there.
And when you start small and when you start with building a very minimalistic version with high human control and low agency, it also forces you to think about what is the problem that I'm going to solve. We use this term called problem first, and to me it was obvious in the sense that yeah, I do need to think about the problem, but it's incredible how well it resonates with the people that, in all these advancements of AI that we are seeing, one easy slippery slope is to just keep thinking about complexities of the solution and forget the problem that you're trying to solve. So when you're trying to start at a smaller scale of autonomy, you start to really think about what is the problem that I'm trying to solve and how do I break it down into levels of autonomy that I can build later. So that is incredibly useful, and we keep repeating this pattern over and over with everyone we talk to.
And there's so many other benefits to limiting autonomy, because there's just danger also of the thing doing too much for you and just messing up your, I don't know, your database, sending out all these emails you never expected. There's so many reasons this is a good idea.
Yep. I recently read this paper from a bunch of folks at UC Berkeley, basically Matei Zaharia, Stoica and the folks at Databricks, and it said about 74 or 75% of the enterprises that they had spoken to, their biggest problem was reliability, and that's also why they weren't comfortable deploying products to their end users or building customer-facing products, because they just weren't sure or they just weren't comfortable doing that and exposing their users to a bunch of these risks, right? And that's also why they think a lot of AI products today have to do with productivity, because it's much lower autonomy versus end-to-end agents that would replace workflows. And yeah, I love their work otherwise as well, but I think that's very in line with what at least we're seeing at my startup as well.
Okay, very interesting. There's an episode that'll come out before this conversation where we go deep into another problem that this avoids, which is around prompt injection and jailbreaking and just how big of a risk that is for AI products, where it's essentially an unsolved and unsolvable problem potentially. I'm not going to go down that track, but it's a pretty scary conversation we had, and it'll be out before this conversation.
I think that will be a huge problem once systems go mainstream. We're still so busy building AI products that we're not worried about security, but it will be such a huge problem, especially with this non-deterministic API again, right? So you're kind of stuck because there are tons of instructions that you could inject within your prompt, and then yeah, it's going to be bad.
Okay, let's actually spend a little time here because it's actually really interesting to me and no one's talking about this stuff, which is, like, the conversation we had is just it's pretty easy to trick AI to do stuff it shouldn't do, and there's all these guardrail systems people put in place, but turns out these guardrails aren't actually very good and you can always get around them. And to your point, as agents become more autonomous, and robots, it gets pretty scary that you could get AI to do things it shouldn't do.
I think this is definitely a problem. But I feel in the current spectrum of customers adopting AI, the extent to which companies can actually get advantage of AI or improve their processes or streamline the existing processes that they have, I feel it's still in the very early stage. 2025 has been an extremely busy year for AI agents and customers trying to adopt AI, but I feel the penetration is still not as much as you would actually get advantage out of it. So with the right sort of human-in-the-loop points in here, I feel we can actually avoid a bunch of these things and focus more towards streamlining the processes. And I am more on the optimist side in the sense that you need to try and adopt this before only highlighting the negative aspects of what could go wrong. So I feel strongly that companies have to adopt this. No company we talked to at OpenAI has ever had the case that, oh, AI cannot help me in this case. It has always been that, oh, there is this set of things that it can optimize for me, and then let me see how I can adopt it.
Sweet. I always like the optimistic perspective. I'm excited for you to listen to this and see what you think, because it's really interesting, and to your point, there's a lot of things to focus on. It's one of many things to worry about and think about. Okay, let's get back on track here. So we've shared a bunch of pro tips and important pieces of advice. Let me ask, what other patterns and kind of ways of working do you see in companies that do this well and teams that build AI products successfully? And then just what are the most common pitfalls people fall into? So we could just maybe start with what are other ways that companies do this well, build AI products successfully?
I almost think of it as like a success triangle with three dimensions. It's never always technical. Every technology problem is a people problem first. And with companies that we have worked with, it's these three dimensions, right? Great leaders, good culture and technical progress. With leaders itself, we work with a lot of companies for their AI transformation, training, strategy and stuff like that. And I feel like in a lot of companies, the leaders have built intuitions over 10 or 15 years and they are kind of highly regarded for those intuitions, but now with AI in the picture, those intuitions will have to be relearned, and leaders have to be vulnerable to do that, right?
I used to work with the now CEO of Rackspace, Gajen. So he would have this block every day in the morning which would say "catching up with AI, 4 to 6 a.m.," and he would not have any meetings or anything like that, and that was just his time to pick up on the latest AI podcasts or information and all of that, and he would have weekend vibe coding sessions and stuff like that. So I think leaders have to get back to being hands-on, and that's not because they have to be implementing these things, but more of rebuilding their intuitions, because you must be comfortable with the fact that your intuitions might not be right, and you probably are the dumbest person in the room and you want to learn from everyone. And I've seen that being a very distinguishing factor of companies that build products which are successful, because you're kind of bringing in that top-down approach. It's almost always impossible for it to be bottom-up. You can't have a bunch of engineers go and get buy-in from the leader if they just don't trust in the technology or if they have misaligned expectations about the technology, right? I've heard from so many folks who are building that our leaders just don't understand the extent to which AI can solve a particular problem, or they just vibe code something and assume it's easy to take it to production. And you really need to understand the range of what AI can solve today so that you can guide decisions within the company.
The second one is the culture itself, right? And again, I work with enterprises where AI is not their main thing, and they need to bring in AI into their processes just because a competitor is doing it and just because it does make sense, because there are use cases that are very ripe. Then along the way, I feel a lot of companies have this culture of FOMO and "you will be replaced" and those kind of things, and people get really afraid. Subject matter experts are such a huge part of building AI products that work, because you really need to consult them to understand how your AI is behaving or what the ideal behavior should be. But then I have spoken to a bunch of companies where the subject matter experts just don't want to talk to you because they think their job is being replaced. So, I mean, again, this comes from the leader itself. You want to build a culture of empowerment, of augmenting AI into your own workflows so that you can 10x what you're doing, instead of saying that you'll probably be replaced if you don't adopt AI and stuff like that. So that kind of an empowering culture always helps. You want to make your entire organization be in it together and make AI work for you instead of trying to guard their own jobs, etc. And with AI, it's also true that it opens up a lot more opportunities than before. So you could have your employees doing a lot more things than before and 10x their productivity.
And the third one is the technical part, which we talk about, right? I think folks that are successful are incredibly obsessed about understanding their workflows very well and augmenting parts that could be ripe for AI versus the ones that might need human in the loop somewhere, etc. Whenever you're trying to automate some part of a workflow, it's never the case that you could use an AI agent and that will kind of solve your problems, right? It's always, you probably have a machine learning model that's going to do some part of the job, you have deterministic code doing some part of the job. So you really need to be obsessed with understanding that workflow so you can choose the right tool for the problem instead of being obsessed with the technology itself.
And another pattern I see is also folks really understand this idea of working with a non-deterministic API, which is your LLM. And what that means is they also understand the development life cycle looks very different, and they iterate pretty quickly, which is: can I build something, iterate quickly in a way that it doesn't ruin my customer experience, at the same time gives me enough amount of data so that I can estimate behavior, right? So they build that flywheel very quickly. As of today, it's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time.
When someone comes up to me and says, "We have this one-click agent. It's going to be deployed in your system and then in two or three days it'll start showing you significant gains," I would almost be skeptical because it's just
not possible. And that's not because the models aren't there, but because enterprise data and infrastructure is very messy and you need a bit to even, the agent needs a bit to understand how these systems work. There are very messy taxonomies everywhere. People tend to do things like get customer data v1, get customer data v2, and these kind of things, and all those functions exist and they are being called, and basically there's a lot of tech debt that you need to deal with.
So most of the times, if you're obsessed with the problem itself and you understand your workflows very well, you will know how to improve your agents over time instead of just slapping an agent and assuming that it'll work from day one. I probably will go as far to say that if someone's selling you one-click agents, it's pure marketing. You don't want to buy into that. I would rather go with a company that says we're going to build this pipeline for you and that will learn over time and kind of build a flywheel to improve, than something that's going to work out of the box. To replace any critical workflow or to build something that can give you significant ROI easily takes four to six months of work, even if you have the best data layer and infrastructure layer.
Amazing. There's a lot there that resonates so deeply with other conversations I've been having on this podcast. One is just, for a company to be successful at seeing a lot of impact from AI, the founder CEO has to be deep into it. I had Dan Shipper on the podcast, and they work with a bunch of companies helping them adopt AI, and he said that's the number one predictor of success: the CEO chatting with ChatGPT, Claude, whatever, many times a day. I love this example you gave of the Rackspace CEO, like, catch up on AI news in the morning every day. I was imagining he'd be chatting with the chatbot versus reading news.
With the kind of information you have as of today, you could just, I mean, you want to choose the right channels as well, because everybody has an opinion. So whose opinion do you want to bank on? I feel like having that good quality set of people that you're listening to really makes sense. So he just has a list of two or three sources that he always looks at, and then he comes back with a bunch of questions and bounces it around with a bunch of AI experts to see what they think about it. And I was part of that group, so I kind of know
I love that.
about the questions that he comes up with. So that's cool.
It's pretty cool. I was like, why are you doing so much? And then he says it trickles down into a bunch of decisions that we take.
Okay, let me talk about another topic that's been a hot topic on this podcast. It was a hot topic on Twitter for a while: evals. A lot of people are obsessed with evals, think they're the solution to a lot of problems in AI. A lot of people think they're overrated, that, well, you don't need evals, you can just feel the vibes and you'll be all right. What's your take on evals? How far does that take people in solving a lot of the problems that you talk about, in terms of what is going on in the community?
I feel there's this false dichotomy of, like, either evals are going to solve everything, or online monitoring or production monitoring is going to solve everything. And I find no reason to trust one of the extremes, in the sense that I will entirely bank my application on this or that to solve the thing, right?
So if you take a step back, think of what evals are: basically your trusted product thinking, or your knowledge about the product, that is going into this set of data sets that you're going to build, in the sense that this is what matters to me, this is the kind of problems that my agent should not do, and let me build a list of data sets so that I'm going to do well on those.
And in terms of production monitoring, what you're doing there is you're deploying your application and then you have some sort of key metrics that actually communicate back to you how customers are using your product. You could be deploying any agent, and if the customer is giving a thumbs up for your interaction, you better want to know that. So that is what production monitoring is going to do, right? And this production monitoring has existed for products for a long time, just that now with AI agents you need to be monitoring with a lot more granularity.
It's not just the customer always giving you explicit feedback, but there is much implicit feedback that you can get. For example, in ChatGPT, if you are liking the answer, you can actually give a thumbs up, or if you don't like the answer, sometimes customers don't give you a thumbs down but actually regenerate the answer. So that is a clear indication that the initial answer that you generated is not meeting the customer's expectation, right? So these are the kind of implicit signals you always need to think about, and that spectrum has been increasing in terms of production monitoring.
Now let's come back to the initial topic of, okay, is it evals or is it production monitoring? What does it matter? So I feel, again, we go back to this problem-first approach of what is it that you're trying to build. You're trying to build a reliable application for your customers that's not going to do a bad thing, that's always going to do the right thing, or if it is doing a wrong thing, you're basically alerted very quickly, right?
So I break this down into two parts. One is, nobody goes into deploying an application without actually, you know, testing. This testing could be vibes, or this testing could be, okay, I have these 10 questions that it should not go wrong on no matter what changes I make, and let me build this, and let's call this an evaluation data set. Now let's say you built this, you deployed this, and then you figured, okay, now I need to understand whether it's doing the right thing or not.
So if you're a high-throughput or a high-transaction customer, you cannot practically sit and evaluate all the traces, right? You need some indication to understand what are the things that I should look at. And this is where production monitoring comes into the picture: you cannot predict the ways in which your agent could be doing wrong, but all of these other implicit signals and explicit signals are going to communicate back to you what are the traces that you need to look at. That is where production monitoring helps.
And once you get these kinds of traces, you need to examine what are the failure patterns that you're seeing in these different types of interactions, and is there something that I really care about that should not happen. And if those kinds of failure modes are happening, then I need to think about building an evaluation data set for it.
And okay, let's say I built an evaluation data set for my agent trying to offer refunds where explicitly I have configured it not to. So I built this evaluation data set, and then I made my changes in tools or prompts or whatever, and then I deployed the second version of the product, right? Now, there is no guarantee that this is the only problem that you're going to see. You still need production monitoring to actually catch different kinds of problems that you might encounter. So I feel evals are important, production monitoring is important, but this notion of only one of them is going to solve things for you, that is completely dismissible in my opinion.
All right, a very reasonable answer. And the point here isn't just as simple as do both. It's more that there are different things to catch, and one approach won't catch all the things you need to be paying attention to.
Exactly.
Awesome.
I want to take two steps back and kind of talk about how much weight the term evals has had to take in the second half of 2025, because you go meet a data labeling company and they tell you our experts are writing evals. And then you have all of these folks saying that PMs should be writing evals, they're the new PRDs. And then you have folks saying that evals are pretty much everything, which is the feedback loop you're supposed to be building to improve your products.
Now step back as a beginner and kind of think, what are evals? Why is everyone saying evals? And these are actually different parts of the process, and nobody's wrong in the sense that yes, these are evals. But when a data labeling company is telling you that their experts are writing evals, they're actually referring to error analysis, or experts just leaving notes on what should be right. Lawyers and doctors write evals; that doesn't mean they're building LLM judges or they're building this entire feedback loop. And when you say that a PM should be writing evals, it doesn't mean they have to write an LLM judge that's good enough for production.
I think there are also very prescriptive ways of doing this, and plus one to KD, which is you cannot predict up front if you need to be building an LLM judge versus you need to be using implicit signals from production monitoring, etc.
I think Martin Fowler at some point had this term called semantic diffusion, back in the 2000s, which kind of means that someone comes up with a term, everybody starts butchering it with their own definitions, and then you kind of lose the actual definition of it. That is kind of what is happening to evals as of today. Everybody kind of sees a different side to it, I guess. But if you make a bunch of practitioners sit together and ask them, is it important to build an actionable feedback loop for AI products, I think all of them will agree.
Now, how you do that really depends on your application itself. When you go to complex use cases, it's incredibly hard to build LLM judges because you see a lot of emerging patterns. If you built a judge that would test for verbosity or something like that, it turns out that you're seeing newer patterns that your LLM judge is not able to catch, and then you just end up building too many evals. And at that point it just makes sense to look at your user signals, fix them, check if you've regressed, and move on, instead of actually building these judges. So it all depends. I think one statement that every ML practitioner will tell you is it really depends on the context. Don't be obsessed with prescriptions. They're going to change.
That's such an important point, this idea that evals just mean many things to different people now. It's just a term for so many things. And it's complicated to just talk about evals when you see it as the stuff data labeling companies are giving you, and things like that. And there's also benchmarks. People call benchmarks evals a little bit.
I recently spoke to a client who told me, we do evals. And I was like, okay, can you show me your data set? And they said, "No, we just checked LMArena and Artificial Analysis. These are independent benchmarks, and we know that this model is the right one for our use case." And I'm like, "You're not doing evals. That's not evals. Those are model benchmarks."
That makes sense. Like, the word could be used in that context. I get why people think that, but yeah, now it's just confusing it even more.
Yep.
Just one more line of questioning here that's on my mind. The reason this became kind of a big debate is Claude Code, the head of Claude Code, Boris, was like, "Nah, we don't do evals on Claude Code. It's all vibes." What can you share, Kiriti, on Codex and the Codex team, of how you approach evals?
So at Codex we have this balanced approach of, you know, you need to have evals, and you need to definitely listen to your customers. I think Alex has been on your podcast recently, and he's been talking about how we're extremely focused on building the right product, right? And a big part of it is basically listening to your customers.
And coding agents are extremely unique compared to agents for other domains, in the sense that these are actually built for customizability, and these are built for engineers. So a coding agent is not a product which is going to solve these top five workflows or top six workflows or whatever, right? It's meant to be customizable in many different ways. And the implication of that is that your product is going to be used in different integrations and different kinds of tools and different kinds of things. So it gets really hard to build an evaluation data set for all kinds of interactions that your customers are going to use your product for, right?
But that said, you also need to understand that, okay, if I'm going to make a change, it's at least not going to damage something that is really core to the product. So we have evaluations for doing that. At the same time, we take extreme care in understanding how the customers are using it.
For example, we built this code review product recently, and it has been gaining an extreme amount of traction, and I feel like many, many bugs in OpenAI as well as even for external customers are getting caught with this. And now let's say I'm making a model change to the code review, or a different kind of RL mechanism that I trained it with, and now if I'm going to deploy it, I definitely do want to A/B test and identify whether it's actually finding the right mistakes, and how users are reacting to it. And sometimes, if users do get annoyed by your incorrect code reviews, they go to the extent of just switching off the product, right? So those are the signals that you want to look at and make sure that your new changes are doing the right thing. And it's extremely hard for us to think of these kinds of scenarios beforehand and develop evaluation data sets for them.
So I feel like there's a bit of both. There's a lot of vibes, and there's a lot of customer feedback, and we are super active on social media to understand if anybody's having certain types of problems and quickly fix that. So I feel it's a, how do I put this, it's like a domain of things that you do here.
That makes so much sense. Okay, what I'm hearing: Codex is pro evals, but it's not enough. Yes, but also just watch customer behavior and feedback, and also there's some vibes, just like, is this feeling good? As I'm using it, is it generating great code that I'm excited about? That I think is great.
I don't think, if anybody's coming and saying that I have this concrete set of evals that I can bet my life on and then I don't need to think about anything else, it's not going to work. And with every new model that we're going to launch, we get together as a team and test different things. Each person is concentrating on something else, and we have this list of hard problems that we have, and we throw that at the model and see how well it's progressing. So it's like custom evals for each engineer, you would say, and just understanding what the product is doing in this new model.
If you're a founder, the hardest part of starting a company isn't having the idea. It's scaling the business without getting buried in back office work. That's where Brex comes in. Brex is the intelligent finance platform for founders. With Brex, you get high-limit corporate cards, easy banking, high-yield treasury, plus a team of AI agents that handle manual finance tasks for you. They'll do all the stuff that you don't want to do, like file your expenses, scour transactions for waste, and run reports, all according to your rules. With Brex AI agents, you can move faster while staying in full control. One in three startups in the United States already runs on Brex. You can, too, at brex.com.
We've been talking for almost an hour already, and we haven't even covered your extremely powerful software development workflow for building AI products that you two developed, that you teach in your
Course that basically combines all the stuff we've been talking about into a step-by-step approach to building AI products. You call it the continuous calibration, continuous development framework. Let's pull up a visual to show people what the heck we're talking about and then just walk us through what this is, how this works, how teams can shift the way they build their AI products to this approach to help them avoid a lot of pain and suffering.
Before we go about explaining the life cycle, a quick story on why Kiriti and I came up with this is because there are tons of companies that we keep talking to that have the pressure from their competitors because they're all building agents. We should be building agents that are entirely autonomous. And I did end up working with a few customers where we built these end-to-end agents. And turns out that because you start off at a place where you don't know how the user might interact with your system and what kind of responses or actions the AI might come up with, it's really hard to fix problems when you have this really huge workflow which is taking four or five steps, making tons of decisions. You just end up debugging so much and then kind of hot fixing to the point where at a time we were building for a customer support use case, which is the example that we give in the newsletter as well, and we had to shut down the product because we were doing so many hot fixes and there was no way we could count all the emerging problems that were coming up, right?
And there's also quite some news online recently. I think Air Canada had this thing where one of their agents predicted or hallucinated a policy for a refund which was not part of their original playbook, and they had to go by it because legal stuff. And there have been a ton of really scary incidents, and that's where the idea comes from, right? How can you build so that you don't lose customer trust and your agent or AI system doesn't end up making decisions that are super dangerous to the company itself, and at the same time build a flywheel so that you can improve your product as you go, right? And that's why we came up with this idea of continuous calibration, continuous development.
The idea is pretty simple, which is we have this right side of the loop which is continuous development, where you scope capability and curate data, essentially get a data set of what your expected inputs are and what your expected outputs should be looking at. This is a very good exercise before you start building any AI product, because many times you figure out that a lot of the folks within the team are just not aligned on how the product should behave, and that's where your PMs can really give in a lot more information, and your subject matter experts as well. So you have this data set that you know your AI product should be doing really well on. It's not comprehensive, but it lets you get started. And then you set up the application and then design the right kind of evaluation metrics. And I intentionally use the term evaluation metrics, although we say evals, because I just want to be very specific on what it is, because evaluation is a process, and evaluation metrics are dimensions that you want to focus on during the process, right? And then you go about deploying and run your evaluation metrics.
And the second part is the continuous calibration, which is the part where you understand what behavior you hadn't expected in the beginning, right? Because when you start the development process, you have this data set that you're optimizing for, but more often than not, you realize that that data set is not comprehensive enough, because users start behaving with your systems in ways that you did not predict. And that's where you want to do the calibration piece, right? I've deployed my system. Now I see that there are patterns that I did not really expect, and your evaluation metrics should give you some insight into those patterns. But sometimes you figure out that those metrics were also not enough and you probably have new error patterns that you've not thought about, and that's where you analyze your behavior and spot error patterns. You apply fixes for issues that you see, but you also design newer evaluation metrics to figure out that there are emerging patterns.
And that doesn't mean you should always design evaluation metrics. There are some errors that you can just fix and not really come back to because they're very spot errors. For instance, there's a tool calling error just because your tool wasn't defined well and stuff like that. You can just fix it and move on, right? And this is pretty much how an AI product life cycle would look like.
But what we specifically also mention is while you're going through these iterations, try to think of lower agency iterations in the beginning and higher control iterations. What that means is constrain the number of decisions your AI systems can make and make sure that there are humans in the loop, and then increase that over time, because you're kind of building a flywheel of behavior and you're understanding what kind of use cases are coming in or how your users are using the system, right? And one example I think we give in the newsletter itself is the customer support. This is a nice image that kind of shows how you can think of agency and control as two dimensions, and each of your versions keeps on increasing the agency, or the ability of your AI system to make decisions, and lowering the control as you go.
And one example that we give is that of the customer support agent, where you can break it down into three versions. The first version is just routing, which is: is your agent able to classify and route a particular ticket to the right department? And sometimes when you read this you probably think, is it so hard to just do routing? Why can't an agent easily do that? And when you go to enterprises, routing itself can be a super complex problem. Any retail company, any popular retail company that you can think of, has hierarchical taxonomies. Most of the times the taxonomies are incredibly messy. I have worked in use cases where you probably have a taxonomy that says some kind of hierarchy, and then that says shoes and then women's shoes and men's shoes all at the same layer, where ideally you should be having shoes and then women's shoes and men's shoes should be subclasses, right? And then you're like, okay fine, I could just merge that, and you go further and you see that there's also another section under shoes that says for women and for men, and it's just not aggregated, it's not fixed for some reason. So if an agent kind of sees this kind of a taxonomy, what is it supposed to do? Where is it supposed to route? And a lot of the times we are not aware of these problems until you actually go about building something and understanding it, right?
So when real human agents see these kind of problems, they know what to check next. Maybe they realize that the node that says for women and for men that's under shoes was last updated in 2019, which means that it's just a dead node that's lying there and not being used. So they kind of know that, okay, we're supposed to be looking at a different node and stuff like that. And I'm not saying agents cannot understand this or models are not capable enough to understand this, but there are really weird rules within enterprises that are not documented anywhere, and you want to make sure that the agents have all of that context instead of just throwing the problem at them, right?
Yeah. Coming back to the versions we had, routing was one where you have really high control, because even if your agent routes to the wrong department, humans can take control and undo those actions. And along the way you also figure out that you probably are dealing with a ton of data issues that you need to fix, and make sure that your data layer is good enough for the agent to function. V2 is what we said, a copilot, which is now that you've figured out routing works fine after a few iterations and you fixed all of your data issues, you could go to the next step, which is: can my agent provide suggestions based on some standard operating procedures that we have for the customer support agent, right? And it could just generate a draft that the human can make changes to. And when you do this, you're also logging human behavior, which means how much of this draft was used by the customer support agent or what was omitted. So you're actually getting error analysis for free when you do this, because you're literally logging everything that the user is doing that you could then build back into your flywheel. And then we say post that, once you figured out that those drafts look good and most of the times maybe humans are not making too many changes, they're using these drafts as is, that's when you want to go to your end-to-end resolution assistant that could draft a resolution that could sort the ticket as well, right? And those are the stages of agency, where you start with low agency and then you go up high, right?
We also have this really nice table that we put together, which is what do you do at each version, and what you learn that can enable you to go to the next step, and what information do you get that you can feed into the loop, right? When you're just doing your routing, you have better quality routing data. You also know what kind of prompts you need to be building to improve the routing system. Essentially, you're figuring out your structure for context engineering and building that flywheel that you want, right?
And while I go through this, I want to also be very clear on two things. One is when you build with CC/CD in mind, it doesn't mean that you fix the problem all for once. It's possible that you've probably gone through V3 and you see a new distribution of data that you never previously imagined. But this is just one way to lower your risk, which is you get enough information about how users behave with your system before going to a point of complete autonomy. And the second thing is you're also kind of building this implicit logging system. A lot of people come and tell us that, oh wait, there are evals, right? Why do you need something like this? The issue with just building a bunch of evaluation metrics and then having them in production is evaluation metrics catch only the errors that you're already aware of. But there can be a lot of emerging patterns that you understand only after you put things in production, right? So for those emerging patterns, you're kind of creating a low-risk kind of a framework so that you could understand user behavior and not really be in a position where there are tons of errors and you're trying to fix all of them at once.
And this is not the only way to do it. There are tons of different ways. You want to decide how you constrain your autonomy. It could be based on the number of actions that the agent is taking, which is what we do in this example. It could be based on topic. There are just some domains where it's pretty high risk to make a system completely autonomous for certain decisions, but for some other topics it's okay to make them completely autonomous, and it depends on the complexity of the problem. And that's where you really want your product managers, your engineers and subject matter experts to align on how to build the system and continuously improve it. The idea is just behavior calibration and not losing user trust as you do that behavior calibration, I guess.
We'll link folks to this actual post if they want to go really deep. You basically go through all of these steps, step by step, with a bunch of examples. And the idea here is, as you said, that everything about what you're describing here is about making it continuous and iterative and kind of moving along this progression of higher autonomy, less control. And this idea of even calling it continuous calibration, continuous development is communicating it's this kind of iterative process. And just to be clear, this naming is kind of an ode to CI/CD, continuous integration, continuous deployment. And the idea here is that this is the version of that for AI, where instead of just integrating into unit tests and deploying constantly, it's running evals, looking at results, iterating on the metrics you're watching, figuring out where it's breaking, and iterating on that.
Awesome. Okay, so again, we'll point people to this post if they want to go deeper. That was a great overview. Is there anything else, before I go into a different topic, around this framework specifically that you think is important for people to know?
I think one of the most common questions we get is how do I know if I need to go to the next stage, or if this is calibrated enough, right? There's not really a rule book you can follow, but it's all about minimizing surprise, which means, let's say you're calibrating every one or two days and you figure out that you're not seeing new data distribution patterns, your users have been pretty consistent with how they're behaving with the system, then the amount of information you gain is kind of very low, and that's when you know you can actually go to the next stage, right? And it's all about the vibes at that point. Like, do you know you're ready? You're not receiving any new information.
But also it really helps to understand that sometimes there are events that could completely mess up the calibration of your system. An example is GPT-4o doesn't exist anymore, or it's going to be deprecated in APIs as well. So most companies that were using 4o should switch to GPT-5, and GPT-5 has very different properties. So that's where your calibration's off again. You want to go back and do this process again. Sometimes users start behaving with systems differently over time, or user behavior evolves. Even with consumer products, right, you don't talk to ChatGPT the same way you were talking, say, two years ago, just because the capabilities have increased so much. And also, people just get excited when these systems can solve one task; they want to try it out on other tasks as well.
We built this system for underwriters at some point, right? Underwriting is a painful task. There are agreements that are like, you know, loan applications that are like 30 or 40 pages. And the idea for this bank was to build a system that could help underwriters pick policies and information about the bank so that they could approve loans, right? And for a good three or four months, everybody was pretty impressed with the system. We had underwriters actually report gains in terms of how much time they were spending, etc. And post three months, we realized that they were so excited with the product that they started asking very deep questions that we never anticipated. They would just throw the entire application document at the system and go like, for a case that looks like this, what did previous underwriters do? And for a user that just seems like a natural extension of what they were doing, but the building behind it should significantly change. Now you need to understand what does 'for a case like this' mean in the context of the loan itself. Is it referring to people of a particular income range, or is it referring to people in a particular geo, and stuff like that? And then you need to pick up historical documents, analyze those documents and then tell them, okay, this is what it looks like, versus just saying that there's a policy X, Y and Z and you want to look up that policy. So something that might seem very natural to an end user might be very hard to build as a product builder, and you see that user behavior also evolves over time, and that's when you know that you want to go back and recalibrate.
What do you think is overhyped in the AI space right now, and even more
importantly, what do you think is underhyped?
I am, as I said, super optimistic in different things that are going on in AI. So I wouldn't say overhyped, but I feel kind of misunderstood is the concept of multi-agents. People have this notion of, I have this incredibly complex problem. Now I'm going to break it down into, hey, you are this agent, take care of this. You're this agent, take care of this. And now if I somehow connect all of these agents, they think they're the agent utopia. And it's never the case. There are incredibly successful multi-agent systems that are built, right? Like there's no doubt about that. But I feel a lot of it comes in terms of how are you limiting the ways in which the system can go off track. For example, if you're building a supervisor agent and there are sub-agents that actually do the work for the supervisor agent, that is a very successful pattern. But coming with this notion of I'm going to divide the responsibilities based on functionality and somehow expect all of that to work together in some sort of gossip protocol, that is extremely misunderstood, that you could do that. I don't think current ways of building and current model capabilities are right there in terms of building those kind of applications. I feel that is kind of misunderstood than overrated.
Underrated: I feel it's hard to probably believe, but I still feel coding agents are underrated, in the sense that you can go on Twitter and you can go on Reddit and you see a lot of chatter about coding agents, but talking to an engineer in any random company, especially outside of the Bay Area, you can see the amount of impact these coding agents can create, and the penetration is very low. So I feel like 2025 and 2026 is going to be an incredible year for optimizing all of these processes, and I feel that is going to be creating a lot of value with AI.
That's really interesting. On that first point, so the idea there is you'll probably be more successful building and using an agent that is able to do its own sub-agent splitting of work versus a bunch of, say, Codex agents where you do this task, you do that task.
You can have agents to do these things and you as a human can orchestrate it, or you can have one larger agent that is going to orchestrate all of these things. But letting the agents communicate in terms of a peer-to-peer kind of protocol, and especially doing this in, say, a customer support kind of use case, is incredibly hard to control what kind of agent is replying to your customer, because you need to shift your guardrails everywhere and things like that.
Yeah. Okay. Great picks. Okay, Ash, what do you got?
Can I say evals? Will I be cancelled?
In which category? Which bucket do they go?
Overrated.
Overrated. Okay, go for it. We won't let you get cancelled.
Just kidding. I think evals are misunderstood. They are important, folks. I'm not saying they're not important. But I think just this "I'm going to keep jumping across tools and going to pick up and learn a new tool" is overrated. I still am old school and feel like you really need to be obsessed with the business problem you're trying to solve. AI is only a tool. Try to think of it that way. Of course, you need to be learning about the latest and greatest, but don't be so obsessed with just building so quickly. Building is really cheap today. Design is more expensive. Really thinking about your product, what you're going to build, is it going to really solve a pain point, is what is way more valuable today, and it will only become more true in the near future, right? So really obsessing about your problem and design is underrated, and just rote building is overrated, I guess.
Awesome. Okay. Similar sort of question from a product point of view. What do you think the next year of AI is going to look like? Give us a vision of where you think things are going to go by, say, the end of 2026.
Yeah, I feel there's a lot of promise in terms of these background agents or proactive agents, which are going to basically understand your workflow even more. If you think of where AI is failing to create value today, it's mainly about not understanding the context. And the reason that it's not understanding the context is it's not plugged into the right places where actual work is happening, right? And as you do more of this, you can give the agent more context, and then it starts to see the world around you and understand what are the set of metrics that you're optimizing for or what are the kind of activities that you're trying to do. It is a very easy extension from there to actually gain more out of it and then let the agent prompt you back. We already do this in terms of ChatGPT Pulse, which kind of gives you this daily update of things you might care about, and it's very nice to actually have that jog your brain in terms of, oh, this is something that I haven't thought about, maybe this is good. And now when you extend this to more complex tasks, like a coding agent which says, okay, I have fixed five of your Linear tickets and here are the patches, just review them at the start of your day. So I feel that is going to be extremely useful, and I see that as a strong direction in which products are going to build in 2026.
That is so cool. So essentially agents kind of anticipating what you want to do and getting ahead of you, and here's, I've solved these problems for you, or I think this is going to crash your site, maybe you should fix this thing right here, or I see the spike here and let's refactor our database. Amazing. What a world. Okay, Ash, what do you got?
I am all in for multimodal experiences in 2026. I think we have done quite some progress in 2025, not just in terms of generation but also understanding. Until now, I think LLMs have been our most commonly used models, but as humans we are multimodal creatures. I would say language is probably one of our last forms of evolution. As the three of us are talking, I think we're constantly getting so many signals. I'm like, oh, Lenny is nodding his head, so probably I would go in this direction, or Lenny's bored, so let me stop talking. So there's a chain of thought behind your chain of thought, and you're constantly altering it with language. That dimension of expression is not explored as well. So if we could build better multimodal experiences, that would get us closer to humanlike conversation richness. And yeah, I think also, just given the kind of models, there's a bunch of boring tasks as well which are ripe for AI if multimodal understanding gets better. There are so many handwritten documents and really messy PDFs that cannot be parsed even by the best of the models as of today, and if it's possible, there'll be so much data that we can tap into.
Awesome. I just saw Demis from DeepMind AI, Google, whatever they call the whole org, talking about this, where he thinks that's going to be a big part of where they're going, combining the image model work, the LLM, and also their world model stuff, Genie, I think is what it's called.
So that's going to be a wild, wild time. Okay. Last question. If someone wants to just get better at building AI products, what's just maybe one skill or maybe two skills that you think they should lean into and develop?
I think we did cover a bunch of best practices for AI products, which is start small, try to get your iteration going well and build a flywheel and all of that. But again, if you kind of look at it at a 10,000 ft level, for anybody building today, like I was saying, implementation is going to be ridiculously cheap in the next few years. So really nail down your design, your judgment, your taste and all of that. And in general, if you're building a career as well, I feel for the past few years, your formative years, say the first two, three years of building your career, is always focused on execution mechanics and all of that, and now we have AI that could help you ramp pretty quickly. And post that, I mean after a few years, I think everybody's job becomes about your taste, your judgment, and kind of, you know, what is uniquely you. I think nail down on that part and try to figure out how you can bring in that kind of a perspective. And it doesn't have to mean that you should be significantly older, have years of experience.
We recently hired someone, and we use this very popular app for tracking our tasks, right? And we've been using it for years and we pay a high subscription fee for it. And this guy just came with his own vibe-coded app to the meeting. He onboarded us to all of it and he's like, okay, let's start using this. And I think that kind of agency and that kind of ownership to really rethink experiences is what will set people apart. And I'm not being blind to the fact that vibe-coded apps have high maintenance costs, and maybe as we scale as a company we have to replace it or we have to think of better approaches, but given that we're a small-size company now. And just, I was really shocked because I never thought of it.
If you've been used to working in a certain way, you associate a cost with building, and I feel like folks who grew up in this age have a much lower cost associated in their mind. They just don't mind building something and going ahead with it. They're also very enthusiastic to try out new tools. That's also probably why AI products have this retention problem, because everybody's so excited about trying out these new tools and all of that. But essentially, having the agency and ownership. And I think it's also going to be the end of the busy work era, right? You can't be sitting in a corner doing something that doesn't move the needle for a company. You really need to be thinking about, you know, end-to-end workflows, how you can bring in more impact. I think all of that will be super important.
That reminds me, I just had Jason Lemkin on the podcast. He's very smart on sales, go-to-market, runs SaaStr, and he replaced his whole sales team with agents. He had 10 salespeople, now he has 1.2 and 20 agents. And one of the agents was just tracking everyone's updates to Salesforce and kind of updating it automatically for them based on their calls. And one of the salespeople is like, "Okay, I quit." And it turns out he wasn't really doing anything.
He was just sitting around, and he's like, "Okay, this will catch me. I got to get out of here."
Yes.
So to your point about you can't, it'll be harder to sit around and twiddle your thumbs, I think is really right.
Yeah. I think to add on to that, I feel like persistence is also something that is extremely valuable, especially given that for anybody who wants to build something, the information is at your fingertips even more than the past decade, right? You can learn anything overnight and become that sort of Iron Man kind of approach. So I feel like having that persistence and going through the pain of learning this, implementing this, and understanding what works and what doesn't work, and as you are going through this pain of developing multiple approaches and then solving the problem, I feel that is going to be the real moat as an individual. I like to call it "pain is the new moat." But I feel that is exactly super useful to actually have, especially when you're building these AI products.
Say more about this. I love this concept. Pain is the new moat. Is there more there?
Yeah, I feel as a company, I mean, successful companies right now building in any new area, they are successful not because they're first to the market or they have this fancy feature that more customers are liking. They went through the pain of understanding what are the set of non-negotiable things and trade them off exactly with what are the features or what are the model capabilities that I can use to solve that problem. This is not a straightforward process, right? There's no textbook to do this, or there's no straightforward way or a known trodden path to be here. So a lot of this pain I was talking about is just going through this iteration of, okay, let's try this, and if this doesn't work, let's try this. And that kind of knowledge that you built across the organization or across your own lived experiences, I feel that pain is what translates into the moat of the company, right? This could be a product or eval or something that you built, and I feel that is going to be the game changer.
That is awesome. It's like turning coal into diamond. Diamond. Yes. Okay. I feel like we've done a great job helping people avoid some of the biggest issues people consistently run into building AI products. We've covered so many of the pitfalls and the ways to actually do it correctly. Before we get to our very exciting lightning round, is there anything else that you wanted to share? Anything else you want to leave listeners with?
Be obsessed with your customers. Be obsessed with the problem. AI is just a tool, and try to make sure that you're really understanding your workflows. 80% of so-called AI engineers, AI PMs spend their time actually understanding their workflows very well. They're not building the fanciest and the, you know, most cool models or workflows around it. They're actually in the weeds understanding their customers' behavior and data. And whenever a software engineer who's never done AI before hears the term "look at your data," I think it's a huge revelation to them, but it's always been the case. You need to go there. Look at your data, understand your users, and that's going to be a huge differentiator.
It's a great way to close it. AI isn't the answer. It's a tool to solve the problem. With that, we have reached our very exciting lightning round. I've got five questions for both of you. Are you ready? Yay. Yes.
All right. So you can both answer them. You can pick one which you want to answer. Either way, up to you. What are two or three books you find yourself recommending most to other people?
For me, it's this book called When Breath Becomes Air, Lenny. It was written by Paul Kalanithi. I think he was an Indian-origin neurosurgeon who was diagnosed with lung cancer at 31 or 32, and the whole book is his memoir, written after he was diagnosed, and it's really beautiful, especially because I read it during COVID, and all we ever wanted to do during COVID is stay alive. There are a bunch of really nice quotes within the book as well, but I remember one of them: he was kind of arguing against a very popular quote by Socrates, which is "the unexamined life is not worth living" or something like that, which means you really need to be thinking about your choices. You need to, you know, understand your values, your mission and all of that. And Paul says, "If the unexamined life is not worth living, was the unlived life worth examining?" Which means, are you spending so much time just understanding your mission and purpose that you've forgotten to live? And I think everybody who's staying in the AI era and building and continuously going through this phase of reinventing themselves needs to take a pause and live for a bit. I guess they need to stop evaling life too much. What really
I was going to say, that's where my mind went. Generate some evals for your life. Oh my god, we've gone too far.
Yep. Yeah. That's my favorite book.
I like more of science fiction books. So I really like this Three-Body Problem series. It's a three-book series. It has elements of grander than science fiction, life outside Earth and how it impacts the human decision-making process, and it also has elements of geopolitics and how much important
or like valuable abstract science is to human progress, and then when that gets stopped, it's not noticeable in everyday life, but it can cause like devastating effects. So I feel like AI helping in these areas, for example, is going to be extremely crucial, and that book is like a nice example of what could happen otherwise.
Completely agree. Absolutely love it. Might be my favorite sci-fi book, or series even, and it's three. I have to read them all three, by the way. I find that it only got really good about one and a half books in. So if anyone's tried it and is like, what the heck is going on here, just keep reading and get to the middle of the second one, and then it gets mind-blowing.
Yes. If you love sci-fi and you're in AI, you got to read this book called A Fire Upon the Deep by Vernor Vinge. Check it out. It's incredible. I saw Noah Smith on his newsletter recommend this book, and there's like sequels to it, but this is the one. It's so incredible, and it actually turns out it's about AGI and superintelligence and all these things, and it's just so epic, and no one's heard of it.
Thank you.
There you go. I'm giving you one back. Okay, next question. What's a favorite recent movie or TV show that you've really enjoyed?
I started re-watching Silicon Valley, and I think it's so true. It's so timeless. Everything is repeating all over again. Anybody who's watched it a few years ago should start re-watching it, and you'll see that it's eerily similar to everything that's happening right now with the AI wave.
That's a good idea to rewatch it. I love that their whole business was like an algorithm to compress, like a compression algorithm. It's like maybe a precursor to LLMs in some small way. Very good. All right, Kiriti, what you got?
I'm going to digress and say not a movie or a TV show, but there's this game I picked up recently called Expedition 33. It has nothing to do with AI, but it's an incredibly, incredibly well-made game in terms of the gameplay, or like the movie and the story and the music. It's been amazing.
I love that you have time to play games. That's a great sign. I love that. At OpenAI, I'm just imagining there's nothing else going on except just coding and
Yeah, it has been incredibly hard to find time for that.
That's good. That's a good sign. I'm happy to hear this. Okay. What's a favorite product that you've recently discovered that you really love?
For me, it's Wispr Flow. I've been using it quite a bit, and I didn't know I needed it so much. The best part is it's a conceptual transcription tool, which means if you go to Codex and start using Wispr Flow, it starts identifying variables and all of that, and it's so seamless in terms of transcription to instruction. You could say something like, "I'm so excited today, add three exclamation marks," and it seamlessly switches. It adds those three exclamation marks instead of writing "add three exclamation marks." And I think it's pretty cool. If you're not using it, you should try it.
I'll do a plug. Get Wispr Flow for free for an entire year, for a year for free, by becoming an annual subscriber of my newsletter.
And that's how I got access to it, Lenny.
There we go. I think I pitched this deal. I think people don't truly understand how incredible this is. They're like, "No way. This is real." It's real. And 18 other products. Lennysproductpass.com. Check it out. Moving on. Kiriti.
Awesome. I actually am a stickler for productivity. I keep experimenting with new CLI tools and things which can make me faster. So I feel like Raycast has been amazing. I've discovered all these new shortcuts that you can use to open different things, type in shortcut commands, and things like that. And caffeinate is another thing that I've recently discovered from my teammates. It helps you prevent your Mac from sleeping. So you can run this really long Codex task for like four or five hours locally, let it build the thing, and then you can wake up and be like, okay, this is good. I like this.
That's hilarious. That combo, Codex and caffeinate. You guys need to build that yourself, an OpenAI version of that, or the Codex agent should just keep your Mac from sleeping. That's so funny. By the way, Raycast is also part of Lenny's Product Pass. One year free of Raycast.
Lenny didn't tell us these, folks. These are actually our favorites.
These are just two of 19 products. No caffeinate though. I don't know if that's even paid. Okay, let's keep going. Do you have a favorite life motto that you find yourself coming back to in work or in life?
For me, I think this is what my dad told me when I was a kid, and it's always stuck, which is: they told him it couldn't be done, but the fool didn't know it, so he did it anyway. I think be foolish enough to believe that you can do anything if you put your heart to it. Especially now, because you have so much data at your hand that could be pointing towards the fact that you probably will be unsuccessful, with how many podcasts made it to more than a thousand subscribers, or how many companies hit more than 1 million. There's always data to show you that you won't be successful, but sometimes just be foolish and go ahead with it.
That's great. Yeah, for me, I am more of an overthinker, so I really like this quote from Steve Jobs that you can only connect the dots looking backwards. A lot of the times there are numerous choices and you don't really know the optimal one to pick, but life works in ways that you can actually look back and be like, "Oh, these are actually beautiful in terms of how I transitioned." So I feel like that is extremely useful to keep moving forward, keep experimenting.
Final question. Whenever I have two guests on the podcast at once, I like to ask this question. What's something that you admire about the other person?
I think with Kiriti, he's pretty calm and very grounded, and he's always been my sounding board. I can throw a ton of ideas at him, and he's able to anticipate the kind of issues that we might run into. And he's extremely kind and lets his work speak instead of doing a lot of talking, I guess. But if I had to pick one, I think he's the most incredible husband.
Reveal little people know.
Yeah. We've been married for four years, and it's been the most beautiful four years of my life.
Oh wow. Okay. How do you follow that?
Yeah, it's super hard to follow that. I would say I am extremely privileged in terms of working with really smart people in great companies in Silicon Valley. And I feel the unique thing that stands out with Aishwarya across any other smart folks I've worked with is that she has this really amazing knack for teaching and explaining something in a very understandable and easy-to-comprehend way. And that, combined with persistence, is super useful, especially in this fast-moving AI world that we are in, in the sense that there are so many new things coming up, it feels overwhelming. But when I hear her talk about, this is how you make sense of this entire thing, this is where it plugs in, I feel like, oh, that is so simple, I can also do that. So she empowers a lot of people by simplifying things and explaining things in the most understandable way. So I feel that is an incredible quality.
Amazing. How sweet. I've got to do this all the time. I need more. Yes to that. That was great. Okay. Final questions. Where can folks find stuff that you're working on, find you online? Share your course link, and then just how can listeners be useful to you?
I write a lot on LinkedIn. So if you want to listen to pragmatists who've been in the weeds working on AI products and what they're seeing, you can follow my work. We also have a GitHub repository with about 20K stars, and that repository is all about good resources for learning AI. It's completely free. And if you liked what we spoke about today, we also run a super popular course, we'll leave a link to it, on building enterprise AI products. And the course is a lot about unlearning mindsets and following a problem-first approach instead of a tool-first or a hype-first approach. So you can check that out as well. And if you don't want to do the course, we write a lot. We give out a lot of free resources. We have free sessions. So make sure you follow our work.
Yeah, I would also add that you can also find me on LinkedIn. I don't write a lot, I guess, but I'm super excited to just talk about any complex product that you're building. And if you have thoughts on how you can use coding agents to make your life better, or what the problems are that you're seeing, my DMs are always open and we can have a great discussion.
Awesome. Well, Kiriti and Aish, thank you so much for being here.
Thank you so much.
Thank you, Lenny. This was so much fun.
So much fun. Bye everyone.
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.
Article published · Updated