Robby Stein on the PM's Core Skill: Making Great Decisions

Open on YouTube ↗
Overview

When AI can help build almost anything, what still separates a product people love from one nobody uses? Speaking at the Lenny and Friends Summit, Robby Stein, who worked on Instagram Stories, Reels, feed ranking, and direct messaging and now works on AI-era Google Search, argued that the product manager's defining skill has shifted. It is no longer mainly about getting things done. It is about making great decisions. Stein laid out a three-part process drawn from nearly 20 years of building consumer products: understand people deeply, diagnose root causes, and refine the craft. For each part, Stein also described how AI is starting to change the work.

15 min read
2:33

From Project Management to Judgment

Stein opened with context. After almost two decades of building products, including startups and large products like Instagram and Google Search that now reach billions of people, Stein said they have also "built some things that zero people use today." Across the successes and failures, Stein sees a repeatable pattern in what made products work, and that pattern is the subject of the talk.

Stein said the talk would focus on the craft of product rather than on how AI is transforming products. Stein's view is that things that once took teams of people years to build can now be done by almost anyone with a handful of agents or AI assistants. That changes the PM role. In the past, Stein said, much of a PM's value came from organizing, creating energy and momentum, and acting almost like a project manager. That is still true in some cases. But in a world where almost anything can get built, Stein said it is clearer than ever that the PM's real value lies in judgment and taste.

People often describe PM work as doing many things reasonably well. Stein called that somewhat true, but argued that above everything else, the craft comes down to one thing PMs should be the absolute best at: decision-making. PMs should be students of what makes a great decision. Stein called this humbling, because many decisions turn out to be wrong. The three "chapters" that follow are, in Stein's words, an unwritten book about the process Stein and their teams have used, presented in this format for the first time.

3:58

Chapter One: Understanding People Deeply

The first chapter starts with people and their fundamental needs. Stein cited Clayton Christensen's Competing Against Luck and the jobs-to-be-done framework: we don't use products, we "hire" them to do things for us, and builders need to know what those jobs are. The best way to draw out those insights, according to Stein, is through a story. Stein's Instagram team once held an offsite with the people who created the framework and went through the process themselves.

Stein walked the audience through the exercise. Think of your last major purchase, then put yourself back in that moment. When interviewing someone, you ask about the details of their life that day: where they were, what they were reading, who they were with, what they were wearing, whether their kids were there, whether they were in a store or online. The aim is to put yourself fully in that person's position.

Stein's own example was buying a bed. Stein and their wife were trying out beds, and three or four felt great. Then Stein's wife asked Stein to jump on one and "do a barrel roll." After Stein flopped onto it, she said they were buying that bed. What she needed most was a bed that would not wake her up when her partner moved. Stein admitted this was "a little offensive" but real. A PM for beds, thinking about a classic roadmap, might focus on cooling, eco-friendly materials, or price. Stein acknowledged those do matter. But what drove the decision that day was motion transference. Without talking to the customer, Stein said, the team would never have learned that. That is what discovering the job to be done means.

Jobs to Be Done at Google: Visual Inspiration and Trusted Local Information

Stein gave two digital examples from Google. Early in Google's AI experiences, the team realized people were coming for inspirational journeys, such as figuring out what to buy. But two years ago, Stein said, chatbots were text-only, and you cannot really describe a couch to someone in text. Stein said the team was among the first to invest in state-of-the-art multimodal understanding, retrieval, and knowledge systems. The goal was for someone to ask about a couch, get a strong visual response, and continue a multi-turn conversation: asking for emerald green, asking what "fuzzy" means, focusing on a particular throw pillow, then getting a grid of throw pillows. Stein described these as very hard problems at the time. They are common now, but in Stein's account they came from recognizing a core need.

The second example came from early exploration of AI Mode, which lets people have a conversation with Google Search. Early versions gave rich information in response to something like a neighborhood food question, but something was missing: the things people expect from Google. That meant accuracy and authority, seeing places on a map, star ratings, closing times, cuisine type, and price level. Once the team brought that together, a user could ask about food and get a rich experience with trustworthy entity information. Stein called that "the magical product" and said it remains one of the most popular parts of the product based on thumbs-up ratings and other metrics the team tracks. Again, Stein tied it to a human insight about why people used the experience in the first place.

8:19

How AI Might Scale User Understanding

Stein was candid that their thinking about AI's role in this first chapter is still developing. One idea Stein is excited to try more is feeding transcripts of user interviews to a model and having it extract jobs to be done, so the process can scale. Another is an agent that interviews people using the same methodology Stein described, asking many more users why they use a product and what their underlying problems were. Stein presented both as ideas under consideration, not finished systems.

8:42

Chapter Two: Diagnosing Root Causes

The second chapter begins after you have an initial vision and build something. That first version, Stein said, is usually not great. Almost everything Stein has worked on started out that way. The process that follows has four parts: get a clear, analytical view of the specific problems in the product, rank them by priority, understand them deeply, and fix them. Then ask again, and keep looping. Stein compared this to a training process, something like an epoch of model training, where each round of changes moves you closer to product-market fit until the product works.

9:46

Instagram Stories and the Audience Problem

Stein's first example was Instagram Stories. Stein said Stories was one of the few things that worked fairly well out of the box, though there were many internal versions that placed stories in the feed that didn't work well and caused plenty of headaches. Even after launch, it was clear that not everyone was adopting it or felt comfortable sharing. The product's premise was sharing your life throughout the day, so the question became: why aren't you doing that?

Stein described this as the inverse of the jobs-to-be-done question. Instead of asking why people did something, you ask why they aren't doing the thing you want them to do. People answered, and Stein described the answers as a cloud of "leaf nodes." Grouped by theme and sorted, the top theme was roughly an audience problem: my ex is on it, my teacher is on it, my aunt, my sister. People didn't want to post something goofy in front of all of them. The team then ran a large quantitative survey with thousands of people, using the qualitative themes as input, to quantify how much each factor weighed.

The conclusion, according to Stein, was that unless the audience burden could be overcome, no creative tool or clever product idea would work. The team spent two years cycling through ideas until they arrived at Close Friends. Along the way there were many strange versions. One was called "favorites." Another had a back door on your profile that showed a private version of it. Another let you post to your feed under a close-friends label. Stein called these "super confusing": no one understood them, and it made little sense to put a raw photo in the middle of a polished feed. The only version that worked was inside Stories, so the team cut everything else and relaunched it as a Stories experience. That version worked.

11:38

Reels: A Failed Launch in Brazil

The second Instagram example was Reels. The team launched an early version in Brazil. Their assumption, which Stein said made sense at the time and which the team shared, was that people posting goofy videos and dances would not want them on their profile forever, so the format was ephemeral. The launch was, in Stein's words, a "huge failure."

It turned out that if you put a lot of time into an impressive dance, you don't want it to disappear after a day. Stein called that obvious in hindsight. The bigger insight was that the people posting, even young people, wanted to be entertainers. They wanted to build businesses from their content, have it last, and have it go viral. The team made a large update to its change list: Reels became its own format that lives permanently and can be seen widely, including by "your whole school." The earlier assumption had been that people wouldn't want that. Stein said Reels has done quite well since.

12:35

Using AI to Mine Feedback at Scale

Stein said AI now makes this diagnostic process much easier with internal tools. The team uses Google Antigravity alongside a system in which users who have opted in, and are comfortable sharing as they use Google products, are asked what they thought and where a response missed the mark. That creates a large library of feedback that a model can analyze.

Stein's example was shopping for a backpack. One user said it was great that Google knew about each backpack and its sizing, since Google has canonical product information, but it should have asked about their son and his height, because otherwise the backpack might be too heavy for the child. When the team stack-ranked the feedback, this kind of comment came up often. The insight was that the product needed to be collaborative and hold a conversation, because users don't always know everything they need to know to reach their goal. Stein said this was critical to getting users to rate responses positively, and that investing in it has been one of the biggest wins the team has seen in engagement, usage, and helpfulness.

Chapter Three: Craft, Part One — Removing Pain

The third chapter applies when a product is ready for prime time: you are launching to all your customers and sending the blast email. This is about fit and finish, which Stein said is often overlooked. Stein believes people can sense whether a product's creators cared about it. Two things drive that impression: whether the product works perfectly without user pain, and how it makes people feel, such as joy or excitement.

On pain, Stein said the old approach was for the team to be obsessive power users of its own product. AI now makes it possible to scale this. Stein showed a tool they built internally with Antigravity. It takes a large set of potential queries and uses the product as an agent would: it goes to Google, types a question, enters the AI experience, asks a follow-up, records and screenshots the result, and evaluates it against a comprehensive rubric.

Stein said the model proved very capable. For a math or homework question, it could check whether math and LaTeX rendered correctly or whether stray characters appeared. For a question about bioluminescence, it could check whether the response included a visual tray, as it should for a visual topic, or was only text. This surfaces the best responses and makes breakages easy to spot. Stein said the team has since invested in agents that detect when things are broken or off-spec and increasingly fix them on their own. In Stein's view, where teams once relied on a QA team or people who obsessively used every part of the product, many of those days are now over, because the process can be scaled more elegantly.

Chapter Three: Craft, Part Two — Sparking Delight

The second part of craft is delight. Stein's example was the redesigned Google search box, which Google reinvented for the first time in about 20 years and announced at I/O. The question was what an AI-based search box should be. Stein said it should grow with the user and accept uploads of files, videos, and photos for context, and the team had to consider how daunting that could feel. The goal was for Google to still feel like Google while signaling an upgrade, with AI inside helping you. The design uses a color gradient to represent the AI power behind the product.

Stein pointed to details from the design team. The blinking cursor cycles through Google's colors, and Stein said they have been amazed by how often people notice it and post about it on X. When you tap the search box, it jumps out slightly, as if excited, then loads a ring, and there is haptic feedback. Stein described this as one small example of how motion, color, feel, touch, and intentionality can communicate the humanity behind a product and give it a soul. Stein thinks many people overlook this part of the creative process.

17:49

The Playbook, and Where to Spend Your Time

Stein closed by summarizing the playbook for PMs in the AI era. Much of it is not new, but Stein said AI puts it in a new light. Understand people deeply and get at their core needs and problems. Be rigorous and analytical about what needs fixing for the product to work for people and meet its objective. Build it so it is simple, works well, and makes people feel good using it. Stein said the craft of PM, and the skill of making decisions that lead to great products, has never been more important.

Stein also acknowledged this is just one approach. The audience had probably heard ten others that day that were just as valid, and each person should decide what makes sense for them. Stein's final ask was that the next time people think about their products, they spend less time on features out of the blue, technology to throw at the problem, or new distribution tactics, and more time thinking about the people on the other side of the screen.