Running a Product Team Like a Research Lab: Dan Shipper on Building on a Moving Frontier
Lenny's Podcast
Every co-founder and CEO Dan Shipper opened his talk at the Lenny and Friends Summit with what he called "the unreasonableness of business as usual during technology revolutions." His question was how to keep building a product when frontier AI capabilities change every few months. His answer was to add a small research lab to the product organization. The lab explores what new models make possible, and the product team keeps improving what already works. He then walked through why to set up a lab, how to run one, and how to move its successful experiments into the main product, using examples from Every and from AI companies.
"The World Just Changed Again"
Shipper framed the problem with the models released the week before: Fable 5.1 and Astra 6. He showed several things built with them. One was a 3D reproduction of the Battle of Waterloo, which he described as historically accurate. He made it with a single prompt, and the model worked on it for about four hours. A colleague gave a model a scientific paper about how agents interact and are simulated together. After a couple of prompts, he had around a thousand agents running on his computer with a visualization. Shipper said Astra is now good enough to go into Premiere and do real editing on Every's videos. He added that Astra made the animations in his slide deck and much of the deck itself, and that he never touched the animate tab.
He then asked what a product leader should do. Keeping your head down and focusing might work for a while. Asking customers is of limited use, because most of them aren't familiar with these tools and expect the company to tell them what to do. Turning a B2B SaaS app into a 3D multiplayer strategy game is possible but hard. He said everyone in the room faces this problem, himself included.
His first rule was a joke with a serious point: never make a major life decision within 30 days of a meditation retreat, a psychedelic experience, or your first encounter with a frontier model.
Exploration and Exploitation Pull in Opposite Directions
Shipper's diagnosis is that product teams expect everyone to do two things at once: execute the roadmap and stay at the frontier. He sees these as opposed ways of working. Exploring the frontier is divergent. You try new models, build demos, do many different things, and throw most of them away. Exploiting is convergent. You focus, say no, and execute a roadmap you've already planned. Asking the same people to do both means they are always pulled in two directions.
His proposal is to run the product org like a research lab, or at least to add lab elements to it.
Why Make a Lab: Harnessing Early Adopters Without Distraction
In most organizations today, Shipper said, everyone does some exploration and some execution. He is sure every organization also has a few people who explore much more than others, and he said they are important to identify. They are the early adopters. Shipper counts himself among them: people who already run Astra and Fable on weekends for personal projects or experiments they might bring to work. Many of the things they get excited about won't be useful. But because they are "living in the future already," they have a good sense of what the product might become.
The downside, in his words, is that early adopters "can be a massive distraction." So the key question for a product leader is how to use the early adopters on the team, and possibly among customers, without distracting everyone else.
The solution Every found, and one he thinks many strong companies are converging on, is a separate labs team. Some people are responsible for improving and scaling what already works. Others are responsible for exploring what comes next.
He expected the objection that this is expensive. His response was that AI makes a labs team of one possible. One person could be tasked with exploring each new model when it comes out and reporting back. Because each person can now have Astra and Fable do so much work, he argued, the investment required is small.
Different Expectations for Lab and Product
Shipper distinguished the two teams mainly by their expectations. A labs team explores capabilities, especially when new models drop, and runs many experiments in parallel. It should expect to throw away about 90% of what it makes. A product team improves and scales the product, delivers for existing customers, and keeps the product coherent and understandable instead of "throwing a bunch of garbage at them." It should expect to adopt about 10% of what the lab tries.
He acknowledged that labs teams are not new. His claim is that they are starting to work in a way they never have before because of this technology. His main example was Anthropic Labs. Shipper said Claude Code came out of that small experimental group, and he called it one of the biggest and most successful productivity products of all time. He said MCPs, skills, and Claude design also came from that small group, with Anthropic investing more in the experiments that worked. He added that there were probably a thousand other experiments nobody outside ever saw.
How to Run a Lab: Two-Slice Teams of Pirates and Architects
Shipper's first recommendation is very small teams. The old standard was Jeff Bezos's two-pizza team of eight to ten people. For the AI age he proposes a "two-slice team" of one or two people at most. His reasoning is that one or two people can get very far very fast now, and anything larger brings coordination overhead and competing visions.
He described the pairing that works for Every, and that he thinks is starting to work elsewhere, as pirates and architects. The pirate is a "slop cannon" obsessed with finding value. They do a lot of messy work, throw most of it away, and eventually find something interesting. Shipper calls himself a pirate. The architect looks at a messy system, such as something the pirate vibe coded, and shapes it into something valuable, beautiful, and extensible. He considers the combination very powerful in AI.
Dogfooding and Tight Feedback Loops
His second practice is dogfooding. The most important thing for a labs team, he said, is making the loop between building something and knowing whether it's good as tight as possible. The tightest loop is building for yourself. If that isn't possible, he suggested recruiting a few early-adopter customers to test quickly. Ideally the experiments should be used for real work, so the team can tell whether something is useful or merely new. He stressed that this distinction matters a lot.
Parallel, Competing Experiments
Shipper noted that this practice probably differs from how a product team should operate. A lab should run many experiments in parallel, even when they aim at the same goal with competing approaches. It may look messy and incoherent. His reasoning is that when capabilities shift, the frontier becomes very uncertain, and having different people attack the same problem from different angles helps map it and decide what is valuable.
Making Even the Discarded 90% Pay Off
The last practice is making experiments net positive in ROI even though most get thrown away. Shipper described three ways:
- External content. Every has had good results publishing what it tried, what worked, and what didn't. He said people enjoy this, and it brings readers and customers to Every's writing and products.
- An early adopter program. Many customers want to be closer to the company, and working with the lab gives them that.
- Sharing learnings with the product team. At minimum, the lab should tell the product team what is now possible, so the product team can account for new capabilities without exploring the frontier itself.
The Research Pipeline
Next Shipper covered how to move work from the lab into the product. He described a pipeline running left to right. Ideas start as lab-only experiments. Most fail, but a few begin to work. Those get tested in real work. At Every, the first test is whether other people on the team adopt the tool and like it. He noted the right setup depends on what kind of customers a company serves. If internal adoption happens, the idea is often ready for early customers and for the product team to consider where it fits in the roadmap. Even then not everything survives, but eventually some things are ready to scale and release.
Case Study: "Kate Bench"
His concrete example involved Kate, Every's editor-in-chief, whom Shipper said he has been trying to automate "very lovingly" for three years. Kate has strong taste in copy editing. With Every now at about 30 people, she can't do every copy edit herself. Hiring someone with her level of taste and training them would be hard. Shipper wanted AI to extend her impact without her spending more late nights on copy edits.
He showed a message Kate woke up to: "Not important, but I downloaded every single one of your copy edits over the last 3 years and had Fable try to do a copy edit on the latest piece." He said he had been running experiments like this for a couple of years, and only recently did the capabilities get good enough for Kate to actually use it. Internally they call this "Kate bench."
The next step was internal use. Every has an internal agent, which Shipper described as a single agent for the entire company. Kate now asks it to do "a Kate pass" on a draft. The agent files suggested changes the way she would, based on her historical edits, and is meant to improve over time. Shipper built it in his spare time. He said it "just sort of works." Kate is adopting it and it is spreading to others, which he called a very good sign.
Bringing In the Architect
At the internal-use stage, Shipper brings in an architect. In this case that was Yanek, who works with him and takes messy things that kind of work and makes them good. Yanek built a dashboard that shows, for each document, how many suggestions were accepted and how much work Kate still has to do after the agent's pass. It does this by tracking the edits she makes after the agent finishes. Shipper reported that Kate did 12% less work on these kinds of edits than the month before. He described Yanek's job as turning the prototype into a real system that can keep improving and compounding.
The project is now at the point where Every is considering putting it in front of early customers. It is also asking whether the same process works beyond copy editing. Shipper thinks it does but presented that as an open question. He summarized the trajectory as moving from a lab-only experiment to something they think they can give customers.
Keeping the Pipeline Moving
Shipper gave two practices for pushing ideas through the pipeline.
The first is reviewing the pipeline regularly. He keeps a tracker in Notion, and at every weekly all-hands everyone discusses what's in it. He called this crucial. People on the product team who aren't experimenting still learn what the lab finds interesting and what is moving forward. They can think about the product implications if an idea proves out, without scrambling to run their own experiments. In his view this keeps the lab and product orgs in sync.
The second is defining clear decision criteria for advancing an idea and making each advancement an event. Every's criteria:
- Are people using it, and are they coming back? Internal use serves as a proxy for value.
- Is it 10x better than what exists? Shipper called this an interesting filter. New things feel exciting, but the real question is whether it's still better a month later, which you can only learn by watching usage over time.
- Is it affordable? Something may work now, but can it be served to customers at scale?
He said clear criteria keep the process rigorous.
Merging the Winners: Codex
The end goal, Shipper said, is to merge the winners into the main product. He noted that labs and deliberate self-disruption are old ideas, but argued they are only now working well because AI lets teams move so fast.
His example was OpenAI's Codex. As he told it, a small team built Codex outside the main app. Several teams inside OpenAI were working on the future of coding at the same time and tried different form factors, such as an IDE or a CLI. Codex launched its desktop app in February 2026, which Shipper said he thinks is fantastic. He said it grew so fast that it was merged into ChatGPT and became its foundation. In his telling, the small Codex team took over what he described as an 800-million-daily-active-user app because their way of experimenting worked so well.
The Signal That It's Working
Shipper closed with what he sees as the promise of the approach: building the next version of your product while scaling the current one, "without going crazy." His test for whether it's working is how the team reacts to new model releases. If the structure is right, he said, you will welcome those moments and be excited by them instead of dreading them.
Okay, so I'm going to talk about how to build products on a moving frontier or as I like to call it the unreasonableness of business as usual during technology revolutions and what to do about it.
So, the world just changed. The world just changed again. We had Fable 5.1 launched last week. We had Astra 6 launched last week and these are [ __ ] sick models. Like look at this.
This I made this like Battle of Waterloo 3D like historical reproduction that's historically accurate with like a single prompt and it just turned for like 4 hours and then I got that. Like that's crazy. That is crazy. Someone else on my team made this sim agent simulation where he fed it a scientific paper about how agents interact and how they get simulated together and then he had like a thousand agents all running on his computer and like visualized by this after like just a couple prompts and putting a paper in.
It's also now starting to do our video editing. We do a lot of videos and like Astra is actually good enough to go into Premiere and like do a bunch of editing. And also fun fact, all of the animations in this deck and a lot of this deck was made by Astra. Like I did not touch a lot, I didn't touch the animate tab at all and we'll get into that in a sec. So, this is crazy.
What should you do? Do you keep your head down and focus if you're a product leader and you have a product team? That might work for a little while. Do you ask your customers what they want? Like maybe, but most of your customers are not familiar with this stuff. They're actually looking to you to tell them what to do. Do you turn your B2B SaaS app into a 3D multiplayer strategy game? Maybe. Maybe, but it's hard. This is hard. This is a big problem that I think everyone in this room faces. I face it.
So, the first thing to know, rule number one, is never make any major life decisions within 30 days of a meditation retreat, a psychedelic experience, or your first encounter with a frontier model. All right, we're settled. We're settled. That's rule number one.
So, let's get into the problem, though. So, what's the core problem? Why is this so hard? It feels like if you're running a product team, it feels like it's everyone's job to execute the road map at a high level and to stay at the frontier at the same time. And that's really, really hard because these are very opposed ways of working. If you want to explore the frontier, you're exploring. That's divergent. You're doing lots of different things. You're probably going to throw a lot of this stuff out. You're trying all these new models. You're doing demos. Exploiting is very different. You're converging. You are focusing. You're saying no to things. You're trying to execute against a road map that you've planned out. And doing both of those, you're pulling in the opposite direction. So, what should we do?
Well, the thing I want to talk about in this talk is my contention that you should run your product org like a research lab or you should add some research lab elements to your product org. And we're going to go through why you should make a lab, how to run a lab, and then how to bring what works from the lab back into your main product. We're going to use examples from stuff we do at Every and from some of the best companies in AI and how they do it.
So, let's start. Why should you make a lab? The problem is, as you said before, right now everyone in your product org is doing both exploration and execution. The yellow circles are the execution. Exploration is purple. You can see everyone in your org is doing both. And what's really interesting, though, is there are some people in your org, I guarantee it. There are some people in your org who are doing way more exploring than others. And they're really important to identify.
Because if you identify these people, they are your early adopters. We are your early adopters. And by the way, that animation and all this stuff all built by Astra, round of applause for Astra, incredible, incredible, I did not do any of this.
So, we are your early adopters. We love new things. We're already running Astra and Fable on the weekends to do our own personal projects or to explore things that we might bring to work. And because of that, there's a lot of things we might be excited about that aren't useful, but because of that we're going to have a really good sense of what your product might become because we're like living in the future already. The problem is that we can be a massive [ __ ] distraction.
So, the key question to ask yourself if you're running a product org is how do I harness the early adopters on my team and maybe even some of my customers who are early adopters without distracting everybody else.
And the solution that we found at Every and that I think a lot of really great companies are trying to do is make a labs team. On a labs team what you've started to do is separate concerns. Some people are in charge of improving and scaling what already works. They're on the product team. And some people are in charge of exploring what comes next, exploring the frontier, doing all these demos, trying these new models, all that kind of stuff. That allows you to get the best of both worlds and I'll explain how that works and why.
And if you're looking at this and you're thinking, "Oh, that's really expensive and I don't have the resources and all that kind of stuff." What is amazing about AI is it allows you to have a labs team of one. You can just have one person on your team whose job it is, maybe even like right when a new model comes out, to go explore all that stuff and come back and tell you what they learned. It actually does not require a lot of resources because all the people on your team now have so many superpowers because they can just go off and have Astra and Fable do a bunch of stuff that it doesn't require a huge investment to do this.
So again, to sort of pull apart the difference between a labs team and a product team, on a labs team your job is you're going to explore capabilities, especially when new models drop, to see what's now possible. You're going to run a ton of experiments in parallel, and we'll get into that. But crucially, the expectation for a labs team is very different. On a labs team, you are going to expect to dispose of like 90% of what you make. You try it and then you throw it away. Product team's very, very different. Your job on a product team is you're going to improve and scale the product. You want to deliver for existing customers. You want to make sure that they feel like they understand the product, it's coherent. You're not just throwing a bunch of garbage at them. So, for a product team, you want to expect to adopt about 10% of what the labs team makes or tries. And this is how they sort of start to work together.
And there's a lot of really good examples of this starting to work in AI in a way that, like, we've had labs teams for a long time, but it's starting to work in a way that it has never, ever worked before because of this technology. So, a really good example is Anthropic Labs. I'll be talking with some people at Anthropic a little bit later about this, but Anthropic Labs, Claude Code, like one of the biggest, most successful productivity products of all time, came out of Anthropic Labs, a separate little group in charge of doing lots of experiments. MCPs, skills, Claude Design, all of these things came from a small group of people inside of Anthropic experimenting with things, and then Anthropic investing more and more into the winners and the ones that worked. And there's like a thousand experiments that you've never seen.
Let's say you're convinced now that this sort of structure gets you the best of both worlds. It helps you to unleash your early adopters without distracting the people on the product team who need to actually deliver on the road map. Question is, like, how do you run a lab well?
So, the first thing that I've started to come to is you really want to use very, very, very small teams. For a long time the standard for team size was the two-pizza team. So, that's like eight to 10 people. That's what Jeff Bezos came to. I think in the AI age, I call it a two-slice team. Like you want one or two people max. You can get so far, so fast with one or two people that anything more, there's a lot of coordination overhead, and there's differing visions, and it just doesn't work. One or two people is really, really great.
And I think the composition that works for us, and I think it's starting to work for other people, is I call them pirates and architects. So, one person on the team is the pirate. They are slop cannons who are absolutely obsessed with finding value. That's me. Like I'm just out there just doing lots and lots of stuff, and I'm throwing it away, and it's all messy, but I'm going to find something really interesting. And then architects are people who are like looking at a messy system, like maybe something that I built that's completely vibe coded, and trying to help shape it into something that's valuable and beautiful and extensible. And pairing those types of people together, I think is very, very powerful in AI.
The next best practice that I think is really important is dogfooding. The biggest thing that matters on a labs team is making the feedback loop as tight as possible between making something and knowing if it's good. And the tightest feedback loop is making something for yourself. If you can do that, I would do it. If you can't, then it's really important to get a couple of early customers or early adopter type customers to help you test with a tight feedback loop so that you can iterate really, really fast. And the stuff you're building, all the experiments, ideally should be used for actual work that you actually have, so you can tell is this useful or is it just new? It's a big thing you need to differentiate.
Another best practice, which I think is quite different from how you should probably run a product team, is you should be building many experiments in parallel, even if those experiments are trying to do the same thing. Try competing approaches to the problem. It might look like a mess and sort of like a lack of coherence, but everybody's going to have a bit of a different take on how to solve a particular problem, and when capabilities move, the frontier becomes very unknown. And so, having people try the same thing from different perspectives helps you map that frontier and decide what's valuable and what's not.
And then the last thing that I think is really important for doing labs is to figure out ways to make your experiments net positive in terms of ROI. So, we're going to throw out 90% of what we make. How do we make even the 90% that doesn't make it into the product eventually, how do we make that actually useful for us? So, one thing that we've done a lot at Every that has worked really well is we turn them into external content. People love to see here's all the things that we tried, here's what worked, here's what didn't, and then that brings customers to read our stuff, use our products, all that kind of stuff. If that's possible, I highly recommend it. Another thing to do is use these experiments to feed an early adopter program. There are probably a lot of customers that you have that want to be closer, want to be in the fold, who are early adopters, and this can be a value add for them to get to work with you more closely.
And then the last thing is, like, at base, the labs team should be sharing what they learn about capabilities and what's now possible with the product team, so the product team can take that into account as they build things without getting distracted by having to go, you know, explore the frontier themselves.
So, let's say you've done that, you've decided to make a lab, you're starting to figure out all the little best practices. Now, how do you get the stuff that you're making from the lab into your product? The thing you need to do is make a research pipeline where ideas start on the left, and they start out as lab only, and there are lots of little experiments, and then you move them progressively from the left to the right, and at some point you have a handoff with your product team where they start to get incorporated into the product. So, to give you a sense of what this looks like, you've got the labs team, they're running lots and lots of experiments. And most of them are not good, but a few of them might work, and those become things that you start to test in real work. One of the things that we do all the time internally is we just let other people on the team adopt it and see if they like it. There are different configurations that work depending on what kind of customers you serve, but the first step for us is do other people on the team start to use it? If that's the case, it's often then ready to put in front of early customers and ready for the product teams to really take a look at it and figure out where does this fit in our road map and how might we use this? And even there not everything makes it, but at some point the thing that you do will be ready to scale and release to customers. And we'll go through some examples of how that works.
That's the research pipeline, and I'll give you a little bit of a concrete example. This is Kate, our editor-in-chief, who I've been trying to automate for the last 3 years, very lovingly. And Kate is fantastic at a lot of things, but one of the things she's really good at is she has a very good taste for copy edits. But as the team has grown, we're about 30 people now, she can't do all of the copy edits herself. And we could hire someone, but it's really hard to get someone who has the level of taste that she has and train them and all that kind of stuff. So, what I've been trying to do is figure out how do we make AI help her with this so she can expand her impact in the org without having to spend additional time doing copy edits really late at night.
And so, Kate often wakes up to messages like this from me saying, "Not important, but I downloaded every single one of your copy edits over the last 3 years and had Fable try to do a copy edit on the latest piece." And I've literally been doing stuff like this for a couple years, and it just started to work where the capabilities are good enough that it's actually ready for Kate to actually use. We call it Kate bench. So, we're starting to move from, okay, we've done a bunch of experiments, now there's one that's starting to work. Now, let's push it out into the org and see if we can get some internal use. So, we have an Every agent. It's a single agent for your entire company. It helps your company get AI built and use all of AI workflows. agents@every.to. But, we use it internally and what happens now is Kate is starting to be like, "Okay, @Every do a Kate pass of this draft." And it'll go in and it will literally file suggested changes like she would based on her historical edits and then improve over time. And this is something that I built just in my spare time that just sort of works, but now we're starting to see that there's some value for the organization cuz she's starting to adopt it and it's starting to spread to other people. And that's a really good sign.
So, when we get to this internal use stage, this is the time where I would then bring in an architect. And I have one in particular who I work with on my team. His name is Yanek. He's fantastic. And Yanek goes and takes this messy thing that kind of works and makes it really great. So, now we have a whole dashboard of, okay, for each document, how many suggestions were accepted
and then how much work is remaining for Kate to do after we go in and do it. So, it checks what are the edits she does after the agent goes and does the edits, and you can see we're improving. She did 12% less work on these types of edits than she did the month before. And his job is to make it like a real system that we can improve and compound in an ongoing way.
Now, it's starting to work well enough that we're starting to think about, okay, how do we put this in front of early customers? Does this sort of process work for things beyond copy editing? I think it does, but you know, we've sort of taken it from it's only an experiment in the lab to something that we're ready to be like, "Okay, I think we can give this to customers."
So, a few best practices for how to continually push things through your lab pipeline. One thing that we do that I think is very important is we review the pipeline regularly. I have a tracker in Notion. Every week on our all-hands, everyone talks about what's going on in our pipeline. And I think that's crucially important because what you want to be able to do is you want the rest of the product team who's not playing around with everything to know here's what we think is interesting, here's what's starting to move up the pipeline, so that you can start to think about, okay, if this does actually prove out, what are the implications for the product, without having to then go scramble to do your own experiments and all that kind of stuff. It helps keep the lab and the product orgs in sync.
Another really important thing is defining clear decision criteria for moving an idea through the pipeline and really making it an event when you do it. So, some of the decision criteria, like our big one is, are people using it and are they coming back? Because what you're trying to do is use internal use as a proxy for value. The second big question: is it 10x better than what currently exists? And this is actually a really interesting filter because what is new feels very exciting, but the question is, when it's a month later, is it actually any better? It's hard to know unless you use time and usage over time as a filter.
And then also, is it affordable? Like maybe this works now, but can we actually serve it at scale for our customers? Another really big question. And having that clear criteria allows you to make sure that you're being rigorous about how you do this. But the ultimate goal is that you merge the winners into the main product.
And this full cycle of labs to merging winners is something that is starting to happen really, really well. We've had labs for a long time, and the idea of doing a self-disruption, of building something that then disrupts your own product, has been around for a while, but it's actually starting to work now in AI because you can move so fast. I think a really good example is Codex.
So, Codex was built by a small team working outside of the main app. There were a bunch of different teams inside of OpenAI who were working on the future of coding at the same time as Codex was. They did a bunch of different iterations on different form factors for what the future of coding would look like. Is it inside an IDE? Is it a CLI? All that kind of stuff. But they launched their desktop app, which I think is fantastic. They launched it in February 2026. And it grew so fast that they eventually just merged it into ChatGPT, and it became the foundation of ChatGPT. The small little Codex team then just took over this 800 million daily active user app because they were so successful with this way of experimenting.
So, I think the promise of doing something like this is that you can actually build the next version of your product while scaling the one that you already have. And you can do it without going crazy. The way that you know this is working is you will welcome moments when new models drop and be excited about them instead of dreading them.
And that is my talk. I'm Dan Shipper. I'm the co-founder and CEO of Every. You should check out Every. We're the only subscription you need to stay at the edge of AI. Thank you very much.
[applause]
Article published · Updated