Inside Amazon S3: Consistency, Correctness, and Durability at Hundreds of Exabytes
The Pragmatic Engineer
Amazon S3 looks simple from the outside: you put data in and get it back out. In this episode of The Pragmatic Engineer, host Gergely talks with Mai-Lan Tomsen Bukovec, VP of Data and Analytics at AWS, who has worked on S3 since 2013. The conversation asks how a system this large stays reliable and keeps evolving. Topics include the move from eventual to strong consistency, the use of formal methods to prove correctness, how failure is designed for, and why S3 now stores tables and vectors as well as objects. Tomsen Bukovec's recurring position is that S3's scale is only manageable through constraints, a design that assumes failure, and a strong commitment to simplicity.
The scale of S3
Tomsen Bukovec gave the scale in numbers. S3 holds over 500 trillion objects and hundreds of exabytes of data. It serves hundreds of millions of transactions per second worldwide and processes over a quadrillion requests a year. The physical system underneath consists of tens of millions of hard drives across millions of servers, in 120 availability zones across 38 regions. The system is layered: disks sit in servers, servers in racks, and racks in buildings. The illustration the team likes to use is that if all of S3's drives were stacked on top of each other, the stack would reach the International Space Station and nearly come back.
The host said they had to look up the term "exabyte," which is a thousand petabytes, because a company with a few petabytes is already considered large. Tomsen Bukovec replied that some individual customers store exabytes in what they call a data lake. They mentioned that the Sony Group CEO had recently described Sony's data as a "data ocean" rather than a lake, and said that at exabyte scale the ocean is fundamentally S3. Their point was that most customers never think about this scale. They assume the drives are always there and experience S3 as something that "just works" for any kind of data.
Origins: unstructured storage and eventual consistency
The host mentioned a story they had read about a distinguished engineer in a Seattle pub who was frustrated that Amazon teams kept rebuilding the same infrastructure. Tomsen Bukovec did not confirm or deny the story. Instead they described the problem S3 was built for. Development began in 2005, and S3 launched in 2006 as the first AWS service. Amazon's engineers were building things like e-commerce sites and had large amounts of unstructured data such as PDFs, images, and backups. They wanted to store it at a price low enough that they would not have to think about storage growth.
The original design was built around eventual consistency. S3 would not acknowledge a put until it actually had the data. However, a subsequent list operation might not show the new object yet. For e-commerce this was acceptable: if an image did not appear immediately after upload, a person would simply refresh the page.
Tomsen Bukovec noted that Apache Hadoop also started as a community in 2006. "Frontier data customers" such as Netflix and Pinterest combined Hadoop with S3's properties, which Tomsen Bukovec summarized as unlimited storage, good performance, and a good price point. They extended unstructured storage to tabular data and built the first data lakes around 2013–2015. Tomsen Bukovec described these companies as born in the cloud. From about 2015 to 2020, enterprises adopted the same pattern. Around 2020, Tomsen Bukovec began seeing "a ton of exabytes" of Parquet files as customers applied S3's characteristics to tables.
Parquet, Iceberg, and S3 Tables
Around 2019–2020, Apache Iceberg began to rise. Iceberg gives table semantics to underlying Parquet data, and many of S3's largest data lakes adopted it across industries. Tomsen Bukovec explained why customers care so much about it. They want a "decentralized analytics architecture," in which different business lines or teams choose their own analytics engines as long as those engines are Iceberg-compliant. With Iceberg as the common format for tabular data, chief data officers and CTOs get future-proofing: they can replace analytics engines or adopt new analytics and AI tools while Iceberg on S3 remains at the bottom of the stack.
AWS responded by launching S3 Tables in December 2024. Tomsen Bukovec said the team added over 15 features to it in the following year. S3 Vectors entered preview in July and became generally available the week before the recording. Tomsen Bukovec described this history as a story that customers had written for data, with the S3 team following along.
The developer model: put, get, and new primitives
Tomsen Bukovec said the goal since 2006 has been a very simple developer experience. When engineers discuss what to build next, they return to the question of how to keep S3 simple to use. At its core, S3 is about the put and the get, and doing those well at scale.
The host added the other basic operations (delete, list, copy) and the core concepts of buckets, objects, and keys. Tomsen Bukovec explained that new capabilities have been layered on top based on what developers are trying to do. Conditional operations are one example. S3 previously added put-if-absent and put-if-match, and more recently added copy-if-absent and delete-if-match. These let applications make writes depend on their own logic.
"Object" is also no longer the only native unit. The two newest primitives are Iceberg tables (S3 Tables) and vectors. Under an S3 table is a set of Parquet files that S3 manages for the customer. Vectors are different. A vector is basically a long string of numbers, and Tomsen Bukovec called it a new data structure for S3 that sits in S3 alongside objects.
Pricing: removing the cost of keeping data
The host recalled that S3 launched at 15 cents per gigabyte per month. The host described this as a third to a fifth of the going rate at the time, which they put at around 50 to 75 cents. They had read that 12,000 developers signed up on the first day. The host also noted that AWS kept cutting prices, to about 2 to 2.3 cents today, and asked why the team did this when customers would probably have paid more.
Tomsen Bukovec answered by pointing to S3's mission, which they described as providing the best storage service on the planet. They cited IDC's estimate that data grows about 27% per year. The host remarked that this sounded low, and Tomsen Bukovec agreed that it is an average: many S3 customers grow two or three times faster, driven by sensors, applications, AI, and ever higher-resolution phone cameras. To keep all that data, customers need to grow storage economically. Tomsen Bukovec said S3 customers don't have conversations about which data to delete because they are running out of space.
According to Tomsen Bukovec, the approach is about total cost of ownership, not only headline prices. AWS sometimes lowers the price of storage and sometimes the price of capabilities. For example, it significantly reduced the cost of compaction for S3 Tables within a year of launch. It also offers tiering and archiving. Intelligent-Tiering, which the host said launched in 2018, watches access patterns. If data goes untouched for a month, it applies an automatic discount of up to 40%, so customers don't have to manage tiers themselves. Tomsen Bukovec tied this to how data is used: pre-training and fine-tuning models, analytics, and future uses customers have not yet imagined.
Glacier and designing to constraints across the whole stack
The host asked about Amazon Glacier, which they said launched in 2012 at one cent per gigabyte per month when the going rate was around 15 cents, in exchange for retrieval that could take hours. How was that trade-off possible?
Tomsen Bukovec described engineering as working within constraints, and said the constraints on availability and cost push the team to be creative. Because S3 is built "all the way down to the metal," including drives and hardware, the team can find efficiencies at every layer. Engineers set a target such as the cost of a byte and pursue it throughout the process. That process includes the data centers themselves and how technicians operate S3 physically, in the same way the team optimizes the software layers. In Tomsen Bukovec's account, this whole-stack view down to the buildings, together with close attention to the cost and lifetime of every byte, is what makes products like Glacier possible.
Why eventual consistency favored availability
The host returned to the original eventual consistency model and asked what it had bought. Tomsen Bukovec said the main optimization was availability, not durability. Consistency here means that a get reflects the most recent put to the same object.
To explain, they described S3's indexing subsystem, which holds all object metadata such as names, tags, and creation times. Every get, put, list, head, or delete goes through the index. More requests hit the index than the storage layer, because head and list requests are served entirely from metadata. At the center of the index is its own storage system, which must be sized to meet S3's availability and durability promises.
Index data is stored across replicas using a quorum-based algorithm, which Tomsen Bukovec called very forgiving of failures. Replicas run on servers in separate availability zones so that data is not correlated on a single fault domain. The failure of one disk, server, rack, or zone affects only a subset of data and never all or a majority of the data for a single object. S3 also caches heavily at the front end. In the eventually consistent design, a read could be routed at random to a cache entry that did not yet reflect the latest write. Quorum guaranteed that reads and writes overlapped at the index storage layer, but the cache did not provide that guarantee, because it was optimized for availability.
Strong consistency: the replicated journal and cache coherency
S3 later became strongly consistent. Tomsen Bukovec said this took time to build and called it the "secret sauce" of S3, which AWS rarely discusses. The requirement was to deliver strong consistency without compromising availability. The team would no longer trade one against the other, and that constraint required a new data structure.
That structure is a replicated journal: a distributed data structure that chains nodes together. A write flows through the storage nodes sequentially, with each node forwarding to the next. When a storage node is written, it learns the sequence number of the value along with the value itself. On a later read, for example through the cache, the sequence number can be retrieved and checked. Tomsen Bukovec called the replicated journal "the heart" of S3's strongly consistent, highly available design.
The host asked about failures, since a sequential chain seems more fragile than an eventually consistent system. Tomsen Bukovec said the second part of the design is a new cache coherency protocol. It keeps the property that multiple servers can receive requests and some are allowed to fail. Tomsen Bukovec calls this a "failure allowance." The replicated journal and the coherency protocol together produce strong consistency.
This came with a real cost in hardware. Tomsen Bukovec recalled a debate in the room with S3 engineers about whether to pass that cost to customers, and said they explicitly decided not to. Strong consistency would be free, would apply to every request, and would not be limited to a particular bucket type. The reasoning was that it should become part of the building block and customers should not have to think about its cost. The host said they found this remarkable, because strong consistency normally adds latency or cost, and they described the latency as unchanged. That latency claim was the host's; Tomsen Bukovec did not discuss latency figures.
Proving correctness with automated reasoning
Tomsen Bukovec stressed that it is one thing to say S3 is strongly consistent on every request and another to know it. S3 runs every kind of workload, and part of its value is that scale decorrelates those workloads. How can the team be sure the model holds everywhere?
Their answer was automated reasoning, which they described as what you would get if computer science and math "got married and had kids." The host asked whether this meant formal methods, and Tomsen Bukovec confirmed it. The team built a proof of the consistency model and runs it on code check-ins to the index subsystem, covering both the caching and storage sublayers. Anyone who changes consistency-related code paths is checked against the proof, so a regression in the consistency model would be caught.
The host asked what such a proof looks like in practice. Tomsen Bukovec answered at a high level. For consistency, the proof covers all the combinations of cases to show the model is correct. S3 also uses formal methods for cross-region replication, to prove that data replicated from one region arrived in another, and to prove the correctness of APIs. They named correctness as a design principle as important as durability, availability, and cost. The goal is to verify correctness on every check-in and every request, not just once. In Tomsen Bukovec's words, "at a certain scale math has to save you," because no one can test every edge case at S3 scale. They offered research papers on the subject, which are linked in the episode notes. The host remarked that formal methods are still uncommon even at infrastructure startups.
Durability: auditors and constant failure
The host raised S3's durability promise, which they described as eleven nines. They noted that even four nines of availability is considered hard in backend systems. With 500 trillion objects, they asked, how do you verify durability on real data rather than relying only on a proof that assumes hardware failure rates?
Tomsen Bukovec said durability is handled mostly in the storage layer and depends on both software and the physical placement of data across disks, servers, racks, availability zones, and regions. Availability zones are physically separate locations, sometimes far apart, and some regions have more than three, giving additional fault domains. The most important element, in their view, is the auditors. More than 200 microservices sit behind a single S3 regional endpoint, each doing one or two things well, loosely coupled through well-defined interfaces. They include health checks, repair systems, and auditor systems. The auditors inspect every byte across the fleet, and when they find something that needs repair, repair systems take over. A significant share of these microservices is dedicated to durability.
The host asked whether someone at S3 can say at any moment what durability was over the past week, month, or year. Tomsen Bukovec answered yes. They also said servers were failing during the conversation itself, because servers always fail. S3 is built on that assumption. Systems constantly assess where a failure affected a node, which bytes were involved, and what repair to start. This all happens separately from the gets and puts customers see; Tomsen Bukovec called it the "whole universe under the hood" of managing bytes at scale. The host contrasted this with their own side project, where a full disk was an unusual event that happened once in three years.
Correlated failure, crash consistency, and failure allowances
Tomsen Bukovec said the key is to think about correlated failure: "if you're thinking about availability at any scale, it's the correlated failure that'll get you." Quorum can tolerate one node failing. If all the nodes are in the same availability zone or rack and fail together, availability suffers and the failure allowance is gone. S3 therefore designs around how workloads are exposed to different levels of failure. When an object is uploaded, it is replicated many times. That replication supports durability, and Tomsen Bukovec emphasized that it supports availability too: if a rack, server, or entire availability zone fails, a copy is still available elsewhere.
They also described crash consistency: a system should always return to a consistent state after a fail-stop failure. If engineers reason about the set of states a system can reach when failures occur, and always assume failures will occur, they can design the microservices to preserve consistency and availability together. Tomsen Bukovec said correlated failures, crash consistency, and failure allowances in caches are the everyday work of S3 engineers.
The host asked how failure allowances differ from the error budgets many companies use loosely. Tomsen Bukovec said a failure allowance is necessary, because assuming no failure leads to "a very bad day for your customer." Using the cache as an example, the allowance is managed through sizing so that customers never notice it. Many microservices exist only to track metrics, and cache sizing is based on those metrics and the size of the underlying system. Tomsen Bukovec argued that because S3's layers are so large and already manage correlated failures and failure allowances, every application built on S3 benefits from them.
Culture: respecting the past while being technically fearless
The host quoted distinguished engineer Andy Warfield. Warfield said he once believed large-scale software was basically code, but learned on S3 that code is inseparable from organizational memory, operational practices, and the scale of the system. The host asked how engineers manage such an intimidating system.
Tomsen Bukovec pointed to culture and commitment. S3 engineers range from people just out of school to people who have been on the team for 15 years. Two Amazon engineering tenets pull against each other. "Respect what came before" says that anything that has worked for many years deserves respect, which argues for conservatism. "Be technically fearless" pushes for invention. Tomsen Bukovec said the tension is part of what makes the work enjoyable. New capabilities must preserve S3's existing properties: it has to keep working, with the same durability and availability. At the same time, work such as conditionals, native Iceberg support, and vectors extends the foundation for future applications. In Tomsen Bukovec's view, the team embodies both tenets every day. In the outro, the host said this pair of tenets was one of their favorite parts of the conversation. They argued that a system this critical could easily become purely conservative and fall behind.
From Hive to Iceberg, and SQL as the interface
The host asked whether S3 is "done," since it can already store any blob. Tomsen Bukovec returned to the rise of Parquet around 2020. In their account, Hive gave Hadoop file-system-style access to S3's unstructured storage. Iceberg replaced Hive by giving tabular access to Parquet data, including compaction and table maintenance. Tomsen Bukovec said they believe the world's tabular data will live in S3 in the future.
As an example, they cited Supabase's announcement the week before. According to Tomsen Bukovec, Supabase's Postgres database would do secondary writes directly into S3 Tables, and its Postgres vector extension would integrate with S3 Vectors. Tomsen Bukovec argued that SQL is the lingua franca of data and that LLMs have been trained on decades of SQL (the host added Python). Many AWS customers already know the S3 API, but S3 Tables let anyone who knows SQL work with data in S3 without knowing S3 or cloud development, whether that user is a human or an AI agent. Tomsen Bukovec predicted this would grow quickly in the coming years.
S3 Vectors: why vectors belong in storage
The host asked what it takes to build a new data primitive like vectors. Tomsen Bukovec compared the situation to tabular data. People once put tables into databases mainly to query them, even when they did not really need a database. Open formats like Parquet later let that data live in S3. Tomsen Bukovec described S3 Vectors as doing the same for vectors, which today often live in dedicated vector databases.
They then described what they called one of the great ironies of data: "you have to know your data to know your data." You need to know the schema, the types, and where the data is. As data lakes become data oceans, this gets harder. Embedding models understand the data for you, and their output is a vector. Tomsen Bukovec said a company's knowledge is not organized in rows and columns. It lives in PDFs, phones, customer-care audio recordings that capture how customers feel, whiteboards, and documents spread across dozens of systems. Understanding what data you have across those formats is a real problem that AI models can help with. They said these models have improved greatly in the last 18 to 24 months. What customers lacked was a place to store billions of vectors, and S3 Vectors was built for that. Tomsen Bukovec emphasized that it is not a database: it has S3's cost structure and scale, applied to vector storage.
How S3 Vectors works
The host asked whether vectors are built on existing object storage. Tomsen Bukovec said no. S3 Tables build on objects, because Parquet files are objects, but vectors required a new data structure and data type.
The core difficulty is nearest-neighbor search in high-dimensional space. A naive approach compares a query against every vector, which is very expensive. S3 does not keep vectors in memory; they are stored across S3's fleet, yet queries still need low latency. At launch, Tomsen Bukovec said, warm queries were returning in about 100 milliseconds or less. They called this "not database fast, but pretty fast."
The approach is to precompute what Tomsen Bukovec called "vector neighborhoods": clusters of similar vectors, such as vectors for one type of dog. These are computed offline and asynchronously so they do not affect query performance. When a new vector is inserted, it is added to one or more neighborhoods. At query time, S3 first performs a much smaller search to find the nearest neighborhoods. Only those vectors are loaded from S3 into fast memory, where the nearest-neighbor algorithm runs. Tomsen Bukovec gave the following limits: up to 2 billion vectors per index and up to 20 trillion vectors per vector bucket, with warm queries at 100 milliseconds or less.
Tomsen Bukovec connected this to an S3 service tenet: "scale is to your advantage." A design cannot get worse as it grows; it has to get better. They cited S3 itself as the example: the bigger it gets, the more decorrelated the workloads running on it become. For vectors, the team asked how to make 100 milliseconds "just the start" and how to ensure that S3's characteristics improve as more vectors are stored.
The 50 TB object limit and how the roadmap is set
The host asked why the largest object size is 50 terabytes. Tomsen Bukovec pointed out that the limit is ten times the original 5 TB. When customers ask what could possibly be that large, the answer is high-resolution video. Size limits reflect optimizing the underlying systems for particular patterns. Raising the limit tenfold, like the tenfold increase in batch operations scale announced the week before, meant optimizing for the new typical distribution of work. Tomsen Bukovec said S3 has few limits and will keep adjusting them as workload distributions change. Larger objects are appearing because of better cameras and phones, and the team wanted customers to be able to grow without restriction.
On the roadmap, Tomsen Bukovec said S3 has launched over a thousand capabilities since 2020. About 90% of the roadmap comes from explicit customer requests, such as larger objects for media customers or batch operations improvements. Some capabilities are invented by watching what customers do with their data, and vectors fall into that category. The team asked how to make data usable in an industry-standard way, like Iceberg for tabular data, and how to make it usable now that embeddings can provide semantic understanding, if only storing billions of vectors were affordable. S3's goal is to remove two constraints: the cost of data and the difficulty of working with it. When both are addressed, Tomsen Bukovec said, the team has what they call a "product shape." They described S3 as a living, breathing organism whose shape changes while staying consistent with the traits people expect. The host compared it to a plant, and Tomsen Bukovec mentioned being a former Peace Corps forestry volunteer who often uses metaphors from nature.
Simplicity as a discipline
The host asked what makes engineering at this scale so different from engineering at a startup. Tomsen Bukovec's answer was simplification. S3 is a very complex system, so each microservice must do one or two things well; otherwise a distributed system becomes unmaintainable over time. Simplicity applies in two places. In the user model, it means a simple API, and now SQL through S3 Tables and semantic understanding through embeddings, instead of annotating a whole metadata layer by hand. Under the hood, Tomsen Bukovec said, S3 engineering meetings constantly return to implementing each capability as simply as possible.
Who works on S3
S3 hires engineers at all career stages. The common trait Tomsen Bukovec emphasized was ownership: a personal commitment to preserving each customer's bytes and keeping them useful, so customers can focus on their applications rather than storage. Tomsen Bukovec argued that every modern business is a data business, and that the data teams feel responsible for that data.
For mid-career engineers who want to work on deep infrastructure, Tomsen Bukovec recommended "relentless curiosity." Working on a system that keeps redefining storage means you are not coloring within the lines. You draw today's lines knowing you may have to erase and redraw them. They said they tell their own three children, who are at university or graduate school, the same thing: step back and read the latest research. They pointed to the papers they would share, such as bringing formal methods into storage systems or thinking about failure in new ways, as examples of that creativity. The host added that startups such as Turbopuffer are now building on S3 as a base layer. Tomsen Bukovec said it was exciting to see so many kinds of infrastructure built on S3.
Closing: multimodal embeddings
Asked for a paper or book recommendation, Tomsen Bukovec pointed to research on multimodal embedding models. They argued that the world we experience is multimodal, so our understanding of data should be too. In their view, the next generation of data lakes will be built on metadata and semantic understanding, which will be created through vectors and searched across multiple modalities. They expect vectors to become very large, especially at the price point AWS introduced for S3 Vectors, and said "we're just getting started" with understanding our data. Their book recommendation was outside computer science: a book on ecology and supporting native bees and insects. The title was not given in the conversation.
AWS S3 is the world's largest cloud storage service, but just how big is it and how is it engineered to be as reliable as it is at such a massive scale? Mai-Lan is the VP of data and analytics at AWS and has been running S3 for 13 years. Today we discuss the sheer scale of S3 in the data stored and the number of servers it runs on. How seemingly overnight AWS went from an eventually consistent data store to a strongly consistent one and the massive engineering and complexity behind this move. What is correlated failure, crash consistency, and failure allowances, and why engineers on S3 live and breathe these concepts, the importance of formal methods to ensure correctness at S3 scale, and many more. A lot of these topics are ones that AWS engineering rarely talks about in public. I hope you enjoy these rare details shared. If you're interested in how one of the largest systems in the world is built and keeps evolving, this episode is for you.
This episode is presented by Statsig, the unified platform for flags, analytics, experiments, and more. Check out the show notes to learn more about them and our other season sponsors.
So, Mai-Lan, welcome to the podcast.
Thanks for having me.
To kick things off, can you tell me the scale of S3 today?
Well, if you want to take a step back and just think about S3, it is a place where you put an incredible amount of data. And so, right now, S3 holds over 500 trillion objects. We have hundreds of exabytes of data. And we serve hundreds of millions of transactions per second worldwide. And if you want another fun stat, we process over a quadrillion requests every single year. And what's under the hood of all that is also pretty amazing scale. If you think about, you know, what's underneath the hood of S3, fundamentally we're disks and servers which sit in racks and those sit in buildings. And if you try to think about all of the scale of what is under the hood, we manage tens of millions of hard drives across millions of servers. And that is in 120 availability zones across 38 regions, which is pretty amazing if you think about it.
So deep down it all starts with hard drives sitting inside servers, sitting inside racks, and then you have a bunch of these racks and then rows of them, buildings of them, right? And that's what you said. So there's tens of millions of hard drives deep down in the bottom of this.
That's right. In fact, if you think about the scale of this, if you imagine stacking all of our drives one on top of another, it would go all the way to the International Space Station and just about back. And so like that, I mean, it's kind of a fun visual to have for us who work on the service, but you know, kind of fundamentally, it's really hard to get your brain around the scale of S3. And so a lot of our customers, they assume the scale is there. They assume that you know all of the drives are always there and they just focus on what S3 is to them which is it just works. It just works for any type of data and all of your data.
Yeah. I mean even for me for the scale, when you talk about exabytes I actually had to look up exabytes because I know of petabytes which is already massive. If a company has like one or two or three petabytes of data it's tons. And an exabyte is, yes, a thousand petabytes is an exabyte, and you told me that you're thinking in that level. It's just hard to fathom.
Yeah, I mean we have individual customers that have exabytes of data. Individual customers who have exabytes of data in what they call a data lake. Although last week I heard a great term. We had the Sony Group CEO talk about what Sony is doing with data and they refer to it as a data ocean, not a data lake but a data ocean. And so like if you have exabytes of data in your data lake it is in fact a data ocean and that ocean is kind of fundamentally S3.
Can you tell me how S3 started? I did some research and there was a story about a distinguished engineer sitting in a pub in Seattle. Who knows if it was true or not, but I read that this was a story that he was a bit frustrated with engineers at Amazon building a lot of infrastructure again and again.
Yeah. If you think back, you know, S3 development really started in 2005 and we launched as the first AWS service in 2006. And if you think about the technical problems of 2006, you know, a lot of customers were building things like e-commerce websites, right, like Amazon.com. And so the engineers at Amazon knew that they had a lot of data that at the time was very unstructured data. It was PDFs, it was images, it was backups, and they wanted a place where they could store that at an economic price point that let them not think about the growth of storage. And so they built S3 and they really built it for a certain type of storage. And so the original design of S3 in 2006 was really anchored around eventual consistency. And the idea of eventual consistency is that when you put data in storage for S3, you know, we're not going to give you an ack back on your put unless we actually have your data. So, we have your data, but the eventual consistency part is that if you were to list your data, it might not show up because it's being eventually consistent. It's there, but it might not show up on a list. And so, we built that consistency model at the time because, you know, we were really optimizing for things like durability and availability. And it worked like a champ for, you know, e-commerce sites and things like that because, you know, when a human was interacting with an e-commerce site and an image happened to not show up exactly at the moment where you put the data into storage, it was okay because a human would just refresh.
And so when we launched in 2006, here's a fun fact for you. 2006 is actually when Apache Hadoop first began as a community as well. And so we had a set of what I think of as frontier data customers like Netflix and Pinterest who took a look at things like Hadoop and they put it together with the economics and the attributes of S3, which is, you know, unlimited storage with pretty good performance at a great price point. And they decided to build what we first began to call data lakes at the time. They decided to extend the idea of unstructured storage and include things like tabular data. And so the first wave of frontier data customers were adopting quote unquote data lakes in about 2013 to 2015. Those were the frontier data customers born in the cloud. And around 2015 to I would say 2020, we started to see all the enterprises take that same data pattern of how can I use S3, the home of all the unstructured data, you know, on the planet, and extend it to tabular data. And that's when, about five years ago, 2020, I started to see a ton of exabytes of, you know, basically Parquet files. And you know, I have worked on S3 for a minute. I started working on S3 in, I guess it was 2013. I'd been at AWS since 2010, so kind of a while. And the rise of Parquet was really interesting because what people did is they said, "Oh, okay. I like the traits and the attributes of S3 and I want to apply it to a table. And so I am going to run my own Parquet data in S3." And then, you know, around I would say 2019, 2020 we started to see basically the rise of Iceberg. And Iceberg at the time, you know, is incredibly popular and it gives the table attributes to the underlying Parquet data, and customers started to do it in, you know, many of my largest data lakes across different industries and different customers. And so one of the things that we did in 2024 is we introduced S3 Tables.
Just for those who don't know what Iceberg is. So, it's an open-source data format for like massive analytic workflows. Right.
That's right. If I ask our customers of these data oceans why they care so much about Iceberg, it's because they want to be able to have what a lot of customers are calling this decentralized analytics architecture where, you know, they can have lines of businesses or different teams within their company that pick what type of analytics to use as long as it's Iceberg compliant. And so if Iceberg is the common metaphor for tabular data, then you have choice, you have flexibility and choice for what type of analytics engines you use in a decentralized analytics architecture. And so I think that's one of the reasons why Iceberg has just taken off, is that it makes it easy to use data at scale, but it also gives a business owner, you know, the chief data officers or the CTOs of the world, it gives them future-proofing for analytics. They can replace their analytics, they can change it out. They can adopt new types of analytics and AI because you have this Iceberg at the bottom turtle of S3. We launched S3 Tables in December 2024. This year we've had over 15 new features that we've added to S3 Tables. And then this year of course we launched the preview of S3 Vectors in July and then last week we were generally available. And so, you know, the story of S3, it's like a story that our customers have written for data, but it's been super fun to work on all these different evolving attributes.
As an engineer, what is the kind of basic architecture and the basic terminology I should know about when I'm starting to work with S3?
When we first launched in 2006, the whole goal for S3 is to provide a very simple developer experience and we've really tried to stick with that. In fact, when the engineers and, you know, when we're sitting around and we're talking about what do we build next, we always go back to that idea of how do you make things really simple to use S3. And so fundamentally, S3, we have a lot of different capabilities now, but it's really about the put and the get. The put of the storage in and the get of the storage out, and where we can do that really well at scale, that is kind of the heart of S3. Now we have a ton of extra capabilities that we've launched over time but, you know, fundamentally when customers think about using S3 they think about the put and the get.
Yeah. So like put data, get data, and I guess some of the other operations, it's a bit like HTTP, right? There's also delete, list, copy, a few kind of other, I guess, primitives.
There is, and you know, if I think about where we have gone over time, we've added capabilities on top of that just based on what developers are trying to do. Okay, let's just take put. We recently added a set of conditionals to the put capability, and like last year we did put if absent or put if match. This year we did a copy if absent or a put if match and we did delete if match. And the core thing for us with conditionals is that we can give developers the capabilities of doing things like the put, but to do it based on the behaviors of their application.
Outside of the get and put, the basic operations, I guess the base terminology that you should just know about is the buckets, objects and keys, right? That's how we think about our data.
Yeah. And now it's not just objects. If you think about the two latest primitives or building blocks we've introduced as native to S3, one of them is the Iceberg table with our S3 Tables and the other one is vectors. And, you know, under the hood of an S3 table is a set of Parquet files that we're managing on your behalf. But that's not the case for vectors. A vector is just basically a long string of numbers. And that is a new data structure for us and it's sitting in S3 just like your objects.
Mai-Lan was talking about the building blocks of S3 like the put, get, tables and vectors. Speaking of primitives for building applications leads nicely to our season sponsor, WorkOS. WorkOS is a set of primitives to make your application enterprise ready. Primitives like single sign-on authentication, directory sync, MCP authentication and many others. One feature does not make an app enterprise ready. Rather, it's the combination of primitives altogether that solves enterprise needs. When your product grows in scale, you can always reach for new building blocks for infrastructure from places like AWS or similar. Similarly, when you need to go up market and sell to larger enterprises, WorkOS provides the application-level building blocks that you need for this. WorkOS has seen the edge cases, the enterprise complexity, and solves this for you so you can focus on your core product.
One example of such a building block is adding authentication to your MCP server. This is a typical screen when you're about to authenticate with an MCP server. If you would have to build it from scratch, it gets pretty complex to set up the OAuth flows behind the scenes. But with WorkOS, it's a few simple steps. Add the AuthKit component to your project, configure it via the UI, then you just direct clients of your MCP server to authorize via AuthKit, verify the response you get via some code, and that's pretty much it. This is the power of well-built primitives. To learn more, head to workos.com. And with this, let's get back to S3 and how it all started.
So I'd like to still go back to the beginning of S3. When it was launched, it was pretty shocking for the broader community because S3 launched with a pricing of 15 cents per gigabyte per month, which was about a third to fifth cheaper than anything else. The going rate at the time was something like 50 cents or 75 cents. And on the first day, I read that like 12,000 developers signed up immediately. A lot of companies immediately or very quickly moved over, and then the surprising thing was that S3 kept cutting prices. It was unheard of before. You were there in the 2010s when some large price cuts happened. Can you tell me what was the thinking inside the S3 team on this unusual pricing? It seemed customers would have been willing to pay more, and also the cutting of prices continuously, even today? I think today it's something like 2 cents or 2.3 cents, something like that, for the same storage as it was 15 cents on launch.
Yeah. You know, I think part of this goes back to what the goal is for S3. Okay. And so the mission of S3 is to provide the best storage service on the planet. Okay. And our goal too is that if you think about the growth of data, IDC says that data is growing at a rate of 27% year-over-year. But I have to tell you, we have so many customers that are growing so much faster than that.
Yeah, I was about to say it sounds pretty low.
I know, like that. But that's an average across everything. We have a lot of customers that grow twice or three times that rate. But if you think about that, okay, you think about all the data that's being generated from sensors, from applications, from, you know, AI, from all these different...
From just taking photos. I mean, every day, right?
Photos. That's right. Like, you know, and if you think about your phone too, think about the resolution and how the resolution of the cameras on their phone have grown. You just have this, like, kind of what Sony talked about with the data ocean. Okay. And in order to have all that data and to grow it, you have to be able to grow it economically. You have to be able to grow it at a price point where you don't really think, okay, what data am I going to delete now because I'm running out of space. You don't have that conversation with S3 customers because of two things. One is, you know, we do lower the price of either storage or the capabilities of what we're doing. Like for example, we lowered the cost of compaction for S3 Tables pretty dramatically within a year after launching S3 Tables. It's not just that, it's like the overall total cost of ownership of your storage. We give you the ability to tier and to archive, right, storage. We give you the ability to do something called Intelligent-Tiering, which is if you don't touch your data for a month, we'll give you an automatic discount on that data because we're watching your storage, and if you don't touch it for much, we'll give you up to 40% discount on that storage. And it's like dynamic discounting so you don't even have to think about it. And
So our whole goal is that you can grow the data that you need to grow because we know that's being used to pre-train models. We know it's being used to fine-tune and do any type of post-training of AI. We know you're using it for analytics. We know you're using it for all these different things either now and in the future. And so our goal is so that you can keep your data and you can use it in a way that advances whatever the thing is that you're doing, whether it's life sciences or you're an enterprise, you know, in manufacturing, right? Whatever you need, the data should be there and you should be able to grow it and keep it and use it any way you want.
Yeah, I did want to ask you about this part. So there's Intelligent-Tiering, which was launched in 2018, so like 12 years after S3 was launched. One thing that really got my attention: Amazon Glacier, which was launched in 2012. So a long time ago, and you can store data that you don't need immediate access to. You're okay waiting for some time to get access to it, I think maybe even hours. When it launched it was only one cent per gigabyte per month, which was, again, back then the going rate for storage was like 15 cents, so almost 10 times cheaper. How do you do that? Like, what is the architecture and thinking behind how you're able to have this trade-off of, like, look, if you don't need your data quickly we can do it a lot cheaper? How could I imagine the kind of trade-offs that you and the engineering team were thinking of making?
Well, you know, I mean, as you know, you're an engineer yourself, and as you know, a lot of engineering is about constraints, right? And that is the fun part about working on S3 is that when you think about constraints, you think about constraints that we have for availability, you think about constraints that we have around, you know, the cost of storage, we start to get really, really creative. Okay? And in S3, because we build all the way down to the metal of the drives and the capabilities that we have in our hardware, we're able to drive, you know, efficiencies at every single part of our stack. Okay?
And so our engineers, when they get together and they talk about the constraints, they talk about the design goals, we'll do something like we'll set a target for, you know, the cost of a byte and we'll drive for that, and we'll drive for it at every single part of the process. And the part of the process that we are also including is, you know, it includes a data center. How do our data center technicians be able to operate the service of S3 from a hardware and a data center perspective, like the physical buildings, just like we do the same thing for the software and the layers of S3 itself? And when you have that ability to run across the whole stack all the way down to the physical buildings, and we're thinking so deeply about the cost and the lifetime of every byte, you're able to do things like Glacier.
You mentioned something really interesting, that when S3 started it was eventually consistent, which means that, you know, data eventually arrives, it might not be there and you might be behind, and there's a lot of things that you can do with this and it gives you some constraints. But you mentioned that the reason that the team launched this is because durability and availability was more important, and I assume of course cost as well. But during those initial phases while S3 was eventually consistent, what kind of benefits does it give to have eventual consistency? Is it a cost constraint? Is it just easier to do high availability systems from an engineering perspective?
Well, I mean, from an engineering perspective, the main optimization was availability. It was not necessarily durability, but it was availability. Okay. So if you take a step back and look at the original design of S3, we were really focused very hard on availability.
So let's take a step back. Okay. So when you talk about consistency, it's the property where the object retrieval, the object get, reflects the most recent put to that same object. Okay? And so if you think about, you know, what parts of the system of S3 that really hits, a lot of it just kind of starts with our indexing subsystem. So if you think about the indexing subsystem in S3, that holds all of your object metadata. And so that's like its name, its tags, its creation time. And our index is accessed on every single get or put or list or head or delete, any API call like that. And so every single data plane request where you go back into our storage system to go get an object goes through our index. And if you think about it, more requests go through our index than our storage system because, for example, it's serving things like head requests and list requests that don't actually end up going back into our storage system at all. Those are, you know, metadata or index requests.
So, you know, if you think about our indexing system, we have a storage system in there. Okay? And that is a really central concept, a storage system in the middle of our indexing system.
So you need a storage system for your index in the index system, right?
That's right. And so we have to configure and size the system to deliver on, you know, our design promise for both availability and durability. Okay. And so the data in our index system is basically stored across a set of replicas, and it uses, you know, basically a quorum-based algorithm. Okay. And a quorum-based algorithm tends to be very forgiving to failures.
And so if you think about how we implemented quorum in our index system, we start first from servers that are running in these separate availability zones. And the reason we do that is that it lets us avoid correlation on a single fault domain. Okay. And since the failure of, like, a single disk, a server, a rack, a zone, it only affects a subset of data, it never affects all of the data for a single object or even a majority of the data for a single object, which we have sharded across, you know, a wide spread of servers. So this core of availability for us is this idea that we spread everything.
And so when a read comes in, it's coming into the S3 front end, and we just heavily cache objects across our systems. When a read comes in, it could route at random and you could create a situation where you're creating an inconsistent read. And so when we have quorum at the index storage layer, we can see reads and writes overlap, but in the cache, they don't, because we're optimizing for availability.
So just so I understand the first part, the eventual consistency: correct me if I'm wrong, you can just, you know, write to all these distributed nodes and you ask one of them, and if it doesn't have it, no problem, because it will be eventually consistent. You now have high availability because you don't need to worry about all of them being in the same state, correct? And that's phase one of AWS, and it gives you availability. And now you're explaining how you're able to, behind the scenes, turn this into strongly consistent. Strong consistency means that it's guaranteed to have the whole system's state, which is hard to do because you could have distributed failures, etc.
And this replicated journal, you know, it took us a while to build, I won't lie. We don't talk about this stuff very much, okay, because this is kind of the secret sauce of S3. But, you know, again, our engineers who are in the room, they were thinking about how do you deliver on both the strong consistency without compromising availability. So I go back to constraints. Okay.
So in that case we were not trading off consistency and availability anymore. And so the engineers had to come up with a new data structure. Basically we do this in S3. Vectors basically is a new data structure that we came up with as well. But, you know, if you think about what we had to invent for strong consistency at S3 scale without relaxing the constraint of availability, it's that we had to build this replicated journal. Okay.
And the replicated journal is basically a distributed data structure where we're chaining nodes together so that when a write is coming into the system, it's flowing through the nodes sequentially. Okay. And so a read or write in a strongly consistent system for S3, it flows through these storage nodes in the journal sequentially. And so every node is forwarding to the next node. And when the storage nodes get written to, they learn the sequence number of the value along with the value itself. And therefore on a subsequent read, like through our cache, the sequence number can be retrieved and stored. And so now you have this strongly consistent and highly available capability in S3. And the heart of that is actually this replicated journal.
Okay. But what's the catch on one end? Because there's always something with trade-offs. You always have something. So on one end you obviously have more complicated business logic. And then I guess the second obvious question is, what about failures? Because in the case of eventual consistency, you don't worry too much about one failure. Clearly in this case, what if a node in the sequence fails, either at the first time or later? How does the system monitor this and recover? Because I guess that's going to be the tricky part, right?
There's another piece to this puzzle that we implemented, which is, you know, basically a cache coherency protocol. And the idea is that this is where we built what we think of as a failure allowance, where in this mode, we needed to retain the property that multiple servers can receive requests and some are allowed to fail. And so it's kind of this combination of this replicated journal as a new data structure, plus we implemented this new cache coherency protocol that gave us a failure allowance, and those two things working in concert gave us this strong consistency. I will say too, this does come at some actual cost.
I was about to say, nothing is free in engineering, right?
There's hardware cost in this because, you can imagine, we've done some more engineering behind the scenes. But I remember sitting in the room with our engineers on S3 and we did a debate on this. We debated it. We said, you know, there's actual costs to the underlying hardware for this, and do we pass it along to customers or not? And we made that explicit decision not to.
Really?
Yeah. We said that when we launch this, we should launch strong consistency, we should make it free of charge to customers, and it should just work for any request that comes into S3. We shouldn't sort of say it's only available on this bucket type or what have you. This should be true for every request made to S3. And part of that mindset for S3 is, how can we provide these types of capabilities, and how can we make it something that becomes a building block, like part of the building block of S3, and you shouldn't have to think about the cost of it.
This was the very surprising thing of this launch, by the way, that suddenly AWS said, like, okay, everything is strongly consistent, it does not cost you more. Latency-wise, your latencies shouldn't have changed significantly. I mean, I'm sure when you roll out initially you do your measurements, etc., but that was the promise, and that was why I couldn't really believe it when I reread history, because it typically doesn't happen. Typically strong consistency does add latency, or it increases cost if it doesn't add latency. There's always these trade-offs. And I mean, it sounds like you either swallowed the cost or cost caught up, but it's very unusual. So,
If I think about that, one of the things that was also very important for us, and we haven't really talked about this as much, but we think about it a lot on the S3 team, is correctness. Okay? So it's one thing to say that you're strongly consistent on every request. It's another thing to know it. And so when we built this strong consistency, you know, I talked about our new caching protocol, I talked about this replicated journal as a new data structure. You know, that took a little bit of time to do and to get right. But at S3 scale, we could not say that we were strongly consistent unless we actually knew we were strongly consistent. Okay. And so what does that mean? How do you do that at S3 scale when everybody is using it for every last workload? In fact, one of the reasons why people use it is because our scale is such that we're decorrelating workloads and you can run absolutely anything on S3. But how do you know?
Mai-Lan just talked about how strong consistency made it so much easier to trust S3. Trust is something that is just as important when writing code, especially when with AI we write more code than before. And this is a good time to talk about our season sponsor Sonar. What is the impact that AI is having on developers? Let's look at some data. A new report from Sonar, the State of Developer Survey report, found that 82% of developers believe they can code faster with AI. But here's what's interesting. In this same survey, 96% of developers said they do not highly trust the accuracy of AI code. This checks out for me as well. While I write code faster with AI agents, I don't exactly trust the code it produces. This really becomes a problem at the code review stage, where all this AI-generated code must be rigorously verified for security, reliability, and maintainability.
SonarQube is precisely built to solve this code verification issue. Sonar has been a leader in the automated code analysis business for over 17 years, analyzing 750 billion lines of code daily. That's over 8 million lines of code per second. I actually first came across Sonar 13 years ago in 2013 when I was working at Microsoft, and a bunch of teams already used SonarQube to improve the quality of their code. I've been a fan since. Sonar provides an essential and independent verification layer. It's an automated guardrail that analyzes all code, whether it's developer or AI generated, ensuring it meets your quality and security standards before it ever reaches production. To get started for free, head to sonarsource.com/pragmatic. And with this, let's get back to the importance of strong consistency at AWS.
How do you know that you're strongly consistent? And that is why we used automated reasoning.
What is automated reasoning, for those of us who are not as familiar with this, which will be most people outside of very few domains like S3?
Yeah, I mean, S3 uses automated reasoning all over the place. Okay. And automated reasoning is a specialized form of computer science. Okay. And Gergely, if you kind of think about it, if computer science and math got married and had kids, right, it would be automated reasoning. It's
Is it formal methods or based on formal methods?
That's exactly.
Oh, yeah. I mean, I studied computer science. So, yeah, that's fun. So it's actually proper formal methods that you're using.
That is right. And we use formal methods in many different places in S3. But one of the first places that we adopted it was for us to feel good that we actually had delivered strong consistency across every request. So what we did is we proofed it, right? We basically built a proof for it, and then we incorporated our proof on check-ins into this index area that I talked about, right, where you have your caching and then you have your storage sub-layers of the index capabilities. And so when anybody is working on our index subsystem now and they're checking in code into the code paths that are being used for consistency, we are proofing through formal methods that we haven't regressed our consistency model.
And can you just give us a rough idea, because the formal methods that I have studied, they were pretty abstract
the things like designing languages, how to have like the different operators, and of course there are maths involved as well. But what are they, like primitives like servers, network, etc., and models being built, data flows? Like how can I imagine a simple proof of something inside S3, roughly, at a really high level?
Yeah. I mean, if you go back to the fundamental notion of a proof, you are proving something to be correct. Okay. And so the places that we use these proofs, we use them in consistency, where we built a proof across all the different combinatorics to make sure that the consistency model is correct. We use it in cross-region replication to prove that a replication of data from one region to another arrived, and we use it in different places within S3 to prove the correctness of API. In all of these cases, you know, we talk about durability, we talk about availability, we talk about cost, but just as strong of a principle, a design principle for us across S3, is correctness. It's a correctness of, you know, a thing, an API request, you know, an operation, as it were.
And the key thing for us too is that you don't want to just proof it once. You want to proof it on every single check-in, and you want to proof it on every single request, so you can verify, you can validate and verify that you are doing in fact what you say you do. And I think for us, you know, at a certain scale, math has to save you, right? Because at a certain scale you can't do all the combinatorics of every single edge case, but math can save you and help you on this at S3 scale. And so we use formal methods in many different places of S3. We have some research papers too. I can send you some links to some research papers where you talk about
Yeah, please do, and we will put it in the show notes below so anyone can check it out, because I think it's really interesting. I feel formal methods are not really a thing in a lot of startups, and even infrastructure startups, yet, but it sounds very reassuring to me to actually have an ongoing proof of that.
And speaking of which, I want to ask about one thing that is related to this: durability. Amazon S3 has very, very high durability promises. I think it's 11 nines, which I had to do a double check on, because in backend systems, whenever you say three nines, when you say four nines of availability, we're not talking durability, availability, four nines is already hard to achieve, and beyond that it just gets very expensive. And I have never heard of 11 nines of durability. Now, this is durability and not availability.
One question that I got when I shared this stat publicly, one thing people were asking, and I was also thinking: how can you prove that, not just in a formal way? But you're now storing, as you said, 500 trillion objects, which is now large enough that just by this durability promise you might be losing some of them. Do you actually validate it on the actual data as well, outside of the proof? Because I assume in the proof you will have assumptions on hardware failure rate, which might or might not be true. So my question is that at Amazon S3 level, when you are able to look at 'are we living up to, for example, our durability promise', how do you go about that, and what are your findings?
Yeah. So we just spent a lot of time talking about our index subsystem, because that is the subsystem that is related to consistency. But when you think about durability, I mean, you think about it at different levels of the S3 stack, but we really think about it in the storage layer. And so if you think about it in the storage layer, you have this design, this promise of, you know, the design here, and underneath that is a combination of things. It's software, but it's also the physical layout of where our data is across everything that we have in S3.
And you know, one of the things that I talked about is that we have disks and servers, which sit in racks, which sit in buildings, and we have tens of millions of these hard drives. We have millions of servers, and we have 120 availability zones across 38 regions.
Yeah. And one availability zone is, like, two availability zones are two physically separate locations, just physically separate, and sometimes they're a ways away from each other. And in some of our regions we have more than three availability zones, which gives us a different domain, a fault domain. If I were to think about durability, I think the most important thing for us is our auditors.
So if you think about a distributed system, we talked about the put and the get. We have many, many, many microservices that are all doing one or two things very well in the background. Okay? And so we have many different varieties of health checks, but we also have repair systems, and we have auditor systems. And our auditor systems go and they inspect every single byte across our whole fleet. And if there are signs that there is repair needed, you know, another repair system will come in place.
And these are all, you know, in the world of distributed systems, these are all microservices working together, loosely correlated, but communicating through well-known interfaces. And so that collection of systems, which are over 200 microservices now, all sit behind one S3 regional endpoint. And a fair number of those subsystems, those microservices, are all dedicated to the notion of durability.
So they will go and check and log and report back. So do I understand correctly that in any given time frame at S3, someone or some people or some systems can actually answer the question of what is our durability the past week, month, year, and so on?
Yes.
Okay. Great. So you can actually verify your durability promise, check if the math is mathing.
Yes. And you know, part of our design is that at any given moment in this conversation that you and I have had just today, we're having servers fail, because servers fail. And so what we are building, and what we've built in S3, is an assumption that servers fail. And so a lot of our systems are always, you know, first of all, they're checking to see where any failure might hit an individual node. How does it affect a certain byte? What repair needs to automatically kick in place? And so this system is constantly moving behind the scenes, if you will, and that is a completely separate thing from the get and the put. The get and the put is what the customer sees. There's this whole universe under the hood of how do we manage the business of bytes at scale.
I'm just thinking, because for a lot of us engineers who are building moderately sized systems, I'll say, compared to S3, they can already be big, but a failure is a big deal, like, you know, a machine going down. Again, I have a small side project, and my storage filled up and I started to give errors, and this is a big deal because it rarely happens to me. This is the first time it happened in 3 years.
Yeah.
But I understand in your business, or when you work at S3 scale, this is just every day. And the question is not when, it's just how often, how do you deal with it? I guess it's a different world.
It is a different world. And the trick is to really think about correlated failure. Okay. So if you're thinking about availability at any scale, it's the correlated failure that'll get you. And
And what is a correlated failure?
Okay. So that's super interesting. So if you think about what I talked about with, you know, eventual consistency, we talked about quorum. Okay? And with quorum it's okay for one node to fail, but if all of the nodes go south, for example, and they're in the same availability zone or on the same rack, then you're really going to be messing with your availability of the underlying storage, okay? You've just lost your failure allowance that I talked about with the cache, because they all fail together. And so a correlated failure is an incredibly important thing to think about when you're thinking about availability.
And so when we're designing around correlated failures, the thing that we have to think about is, like, do we expose, or how are those workloads exposed to different levels of failure. So when you upload an object to S3 with a put, we replicate that object. Okay? We don't just store one copy of it. We store it many times. And that replication is important. It's important for durability. But what's interesting about it, it's also important for availability, because if any of those correlated failure domains fail, like if a whole AZ fails, there's still a copy somewhere else, and the data is still available somewhere, even though an availability zone has failed, or a rack has failed, or a server has failed, or so forth.
Okay. And so that idea of how do you manage and design around correlated failures with our physical infrastructure is super important for S3, for both availability and durability. We also do things like, we think about something called crash consistency. I mean, Gergely, you can tell I can go on and on about this, so you just have to stop me.
No, but this is the interesting stuff.
All right. So the whole idea of crash consistency is that a system, any system that you build, should always return to a consistent state after a fail-stop failure. And if you can do things like reason about the set of states that a system can reach in the presence of failure, and you just always assume the presence of failure, then you also assume the presence of consistency and availability. Then you just design all of these different microservices to all work together in an underlying capability like S3. But that's what our engineers do. They think about crash consistency. They think about correlated failures. You know, they think about failure allowances and caches, right? And it's all that deep distributed systems work that our engineers come in every day to work on.
Can we talk about how you think about failure allowances? Because again, there is a concept of error budgets outside, in other companies as well. I feel it's a bit loosely handled, whereas I feel this is kind of your bread and butter. So what is a failure allowance, how do you measure it, and what do you do if you overstep it or overspend it?
Yeah, I mean, I think that the idea of a failure allowance is, you want to have it, like, you have to have it. If you assume, you know, that you'll never have a failure, you'll actually have a very bad day for your customer. And so we account for failure allowances. But the most important thing is, let's just talk about the failure allowance in our cache. So how do we manage that? Well, we manage it in such a way that you'll never experience it, because we size it, right? And if you're sizing the cache, and you're making sure that the underlying capabilities and the hardware are always there, and we have, like I talked about, those distributed subsystems, those microservices that are all interoperating under the hood, we have a ton of them that do nothing but just track metrics, right? And, like, you know, the sizing of our cache is all related to the metrics and the
The size of our underlying system.
All the metrics. Yeah.
Yeah. That's right. And so one of the really big benefits of running on S3 is, because our system is so huge, you have these massive layers, right? And the massive layers are all managing things like correlated failures and failure allowances. And because they are so huge at the scale of S3, any application that's sitting on top of S3 gets the benefit of it.
Let's take a break a minute from S3 to talk about a one-of-a-kind event I'm organizing for the first time: the Pragmatic Summit, in partnership with Statsig. Have you ever wanted to meet standout guests from the Pragmatic Engineer podcast, plus folks from cutting-edge tech companies, and learn about what works and what doesn't in building software in this new age of AI? Come join me 11 February in San Francisco for a very special one-day event. The Pragmatic Summit features industry legends and past podcast guests like Laura Tacho, Kent Beck, Simon Willison, Chip Huyen, Martin Fowler, and many others.
We'll also have insider stories on how engineering teams like Cursor, Linear, OpenAI, Ramp, and others built cutting-edge products. We'll also have roundtables and a carefully created audience where everyone is interested to meet and chat with. Something I'm hoping will make this event extra special. Seats are limited, and you can apply to attend at pragmaticsummit.com. Talks will be recorded and shared, and paid subscribers will get early access afterwards as well, as a thank you for your additional support. I hope to meet many of you there, and I am so excited about this event. And now let's jump back to S3 and the massive scale of the service.
To get a sense of what the reality is like working as an engineer, an engineering leader, inside an organization like this, I read a quote from a distinguished engineer, Andy Warfield, who said, I'm just quoting what he said: 'Early in my career, I had this sort of naive view that what it meant to build large-scale commercial software was basically just code. The thing I realized very quickly working on S3 was that the code was inseparable from the organizational memory and the operational practices and, you know, the scale of the system.' Since you've now been more than a decade in S3, how do you think of this beast, this really complex system, hundreds of microservices, data that is hard to fathom, you know, unless you think of the hard drives stacking all the way to the space station? And how do your engineers kind of wrangle this? Because it does feel a bit intimidating, I'm not going to lie.
Well, I think so much of this just comes back to the culture and the commitment on the team. And you know, I've worked on S3 for a very long time now, and I have such deep respect for the engineering community on S3. And you know, honestly, I mean, this is true for all of the services in our data and analytics stack, but we have engineers in S3, and they come in every single day with this deep commitment to the durability and availability and the consistency of your bytes. And so the type of conversations that we have are so interesting, because we have people, and really, you know, these are people who are early out of school, there are people who've been working on S3, we have engineers who've been working on S3 for 15 years, and everything in between.
The creativity and the invention of S3, like, you have this tension, which is, on one side, you have to be very conservative with S3, right? And on the other hand, I mean, we have this principal engineering tenet called Respect What Came Before, and that's an Amazon engineering tenet, which is, if it has worked for many, many years, you have to respect that. But then there's also this tenet, and these two tenets are a little bit in tension with each other, which is kind of what makes it so fun. This Amazon engineering tenet is called Be Technically Fearless.
And I believe that the S3 engineers are just amazing at this, at respecting what came before, because if we build new capabilities in S3, we have to maintain the properties, the traits of S3, which is it just works, and you get that durability, availability, etc. But at the same time, we have to be technically fearless, because our ability to go into the world of conditionals, our ability to go into the world of, you know, native support for Iceberg or for vectors, means that we are extending this foundation of storage in a way that helps customers build whatever application they need, now and in the future. And so that combination of the two things, that is sort of, when I think about our S3 engineering team, I think they come in every day and they embody that.
Now, going back to the evolution of S3 from unstructured to structured data. You were mentioning how Hadoop, the data warehouse, was a big use
case where customers started to use it on top of S3, and then at S3 you noticed what a lot of customers or some of your biggest customers were doing, and you kind of built it yourself with more structured data, and then S3 Tables came along, and then Vectors. Would you mind sharing a little bit more on how you evolve S3? Because this was another question: when I asked people about what they'd like to know about S3, one of the questions was, is it done? Is it finished, or is it still evolving? Because there is this notion that S3 can store anything already, right? Any object, any blob. What new thing is there? And yet we have a lot of new things.
Yeah. And if you kind of go back in time a little bit and you think about, you know, the rise of Parquet. Okay. So the rise of Parquet data in S3 started about 2020, and we started to see more and more people store their tabular data in S3. And if you think about what Iceberg provided, it provided a replacement for Hive. Okay, so if you think about Hive and Hadoop, Hive was basically giving your file system access into S3 unstructured storage. Iceberg is giving that tabular access, including the compaction and all the table maintenance that goes along with it, into your Parquet data.
And I actually think that the world's data for tabular data is going to live in the future in S3. And if you just think about the launch that, for example, Supabase did last week, Supabase announced that their Postgres database is just going to do secondary writes directly into an S3 table, just like their Postgres extension for vector is going to integrate directly with S3 Vectors. And so if the world of database, if the world is data as a source, if you will, goes directly into an S3 table, what does that mean for the world's data?
Okay, so SQL, as we know, is a lingua franca of data, and the world's LLMs have all been trained on decades of SQL, and therefore
And Python, SQL and Python.
Python and the stuff that's already out there. And so if you think about this, we have many, many AWS customers who know the S3 API pretty darn well by this point. It's a pretty simple API, but now you have the ability to interact with data in S3 through SQL. And what that means is that you don't have to be somebody who's building cloud applications or know S3. You just need to know SQL.
And this is with S3 Tables, right?
With S3 Tables. And so you can just write SQL into an S3 table, and whether you're an AI agent or a human, right? You're introducing the lingua franca of data as a native property of S3 with S3 Tables, and I think you're just going to see that take off in the upcoming years.
And your latest launch is S3 Vectors. Can you share a little bit what it takes to build a new data primitive like vectors, just behind the scenes: how long it takes, how the team comes together, and maybe what are some engineering challenges of launching something like this? And again, we're talking about vectors, right? So you use embeddings whenever you have LLMs; you create an embedding, it's a vector, you want to store that somewhere, you will need to do search on it. There's specialized vector databases, there's specialized vector additions, etc. So I'm assuming this is the functionality that S3 Vectors supports very nicely.
Yeah. And, you know, today a lot of customers use vector databases, just like back in the day a lot of people put their tabular data in just databases. Okay. And they just used the structure of the database in order to take advantage of being able to query their data. But they didn't really need to use a database. They just put it in a database. And then S3 came along, and then we introduced this way, with the help of open formats like Apache Parquet, of being able to store that structured data in S3. That's kind of what we're doing with vectors right now.
Okay. And if you think about vectors, vectors are basically a bespoke data type. A vector at the end of the day is a very, very long list of numbers. And vectors have been around for a long time and they've been in vector databases for a while, but they really kind of took off in people's data worlds in the last couple of years with the rise of, as you said, the embedding models.
Okay. And so if you take a step back and you think about one of the great ironies of data, it is that you have to know your data to know your data, right? You have to know what your schema is. You have to know what the data types are. You have to know where it is. And as these data lakes become data oceans, you have this situation where it gets harder and harder to know what's in your data, right? And the beautiful thing about embeddings is that embedding models will understand your data so that you don't have to understand your data. And the format that these embedding models put this semantic understanding of your data in is in fact a vector.
And so when we talk to customers, they're so excited about how these embedding models are getting better and better, they want to apply more and more basically semantic understanding to their underlying data, whether it's unstructured or structured, that they have in storage, and so they kind of want to store billions of vectors.
But just to say, when you say they want to understand, correct me if I'm wrong, but hypothetically you have a bunch of text data or maybe some image data, and you're saying that a lot of people, customers, teams, would like to write queries to say, hey, can you find an image that looks like a puppy, or can you find an article that contains this or this. And embeddings, as we know, are great for that, but then you need to actually create the embedding, build the system, etc. Right?
Yeah. And exactly what you're saying. I mean, if you think about what vectors can do, if you think about all the data that a given company has, your knowledge across your business or your knowledge across your life isn't organized into rows and columns like a database. It's in PDFs. It's in your phone, right? It's in audio customer care recordings, which capture the sentiment of how a customer actually feels about their interaction with you. It's whiteboards. By the end of this day, this whiteboard is totally filled up with ideas. And it's in documents across dozens of systems.
And so it's not that you don't have data. You have tons of data. But understanding what data you have across all of those different formats is a real problem. And it's one that AI models can help you with. And so the capabilities of those AI models have gotten so much better in the last 18 to 24 months. But we needed a place to put billions of vectors, billions of, you know, the semantic understanding of relationships, and that's what we built S3 for. The state-of-the-art embedding models combined with the ability to have vectors across S3 is a really important part, and it's not a database. I mean, it's the cost structure and scale of just S3, but it's for vector storage.
And then do I understand, did you need to build new primitives to store this, like going down to the metal, figuring out exactly where we do this, or did you build it on top of your existing primitives as well, like blob storage, etc.?
It's actually a new primitive. And so we had talked about S3 Tables. S3 Tables is building on objects, because those individual Parquet files at the end of the day are an object. Vector is totally different. So with vector we built a new data structure, a new data type. And it turns out that when you're building vectors, searching for the closest vector in a very high-dimensional space, which is basically vector space
Yes.
it's often really hard to find the nearest neighbor. And so basically in a database you have to essentially compare every vector in a database, and that's often super expensive.
And so what we do in S3 is, because we aren't storing all of our vectors in memory, we're storing it on our fleet of S3, a very large fleet, we still need to provide super low latency. And in our launch last week, we were getting about 100 milliseconds or less for a warm query to our vector space, which is actually pretty fast. It's not database fast, but it's pretty fast.
And the way that we do that is we precompute a bunch of, think of them as vector neighborhoods. Okay? And so it's basically a cluster, a bunch of vectors that are clustered together in similarity, like a type of dog, as an example. These vector neighborhoods, if you will, are computed ahead of time offline. They're computed ahead of time asynchronously so that when you're doing your query, it's not going to impact your query performance. And then every time a new vector is inserted into S3, the vector gets added to one or more of these vector neighborhoods based on where it's located.
And so when you are executing a query on S3 Vectors, there's a much smaller search that's done to find the nearest neighborhoods. And it's just the vectors in the vector neighborhoods that are loaded from S3 into fast memory. That's where we apply the nearest neighbor algorithm, and it can result in really good sub-100 millisecond query times.
And so if you think about the scale, S3 will give you up to two billion vectors per index. You think about the scale of an S3 vector bucket, which is up to 20 trillion vectors. And you think about that combined with 100 milliseconds or less for warm query performance. That just opens up what you can do with creating a semantic understanding of your data and how you can query it.
It sounds very interesting and also challenging, because you have to build this for scale from day one. I guess that's one of the benefits and curses of working at S3: everything that you launch, you need to prepare for what would be extreme data elsewhere, but here it's just Monday.
We have S3 service tenets as well. And one of the tenets, and one phrase that I use all the time and our engineers do too, is "scale is to your advantage." So if you are an engineer and you think about that, and you think about one of your tenets for anything you build being that scale must be to your advantage, it just changes how you design. It means that you can't actually build something where the bigger you get, the worse your performance gets or the worse some attribute gets. It has to be constructed so that the bigger you get, the better your performance gets. The bigger S3 gets, the more decorrelated the workloads are that run in S3. That is a great example of scale is to your advantage.
And so when we built vectors, just like we built everything in S3, we asked ourselves: how can we build this such that scale is to our advantage? How can we build this such that 100 milliseconds or less is just the start of the performance that we're going after? And how can we make sure that the more vectors we have in storage, the better the traits of S3 for vector?
I have a different question about the limitations of S3. I read that the largest object you can store in S3 is 50 terabytes. Why is there a limit on the largest object? I mean, I think we can imagine this will be through multiple hard drives and so on, but why did you decide to have a limit? I'm just interested more in the thought process of how the team comes up with, okay, this will be the limit and this is why.
I mean, first of all, that limit of 50 terabytes is 10 times greater than what we launched with. We launched with five terabytes and now we're 50 terabytes. And sometimes we sit and tell customers that and they go, what am I going to store that's going to be 50 terabytes? And we're like, high-resolution video, right?
Known customer.
Right. And so if you think about this sort of thing, if you think about size limits, generally speaking, we do try to optimize for certain patterns. And when you raise the size of an object by 10 times like we did, we're just optimizing for the performance and scale of the underlying systems. It's like we increased the scale of our batch operations by 10 times last week, too. And the idea behind that is that the underlying systems were just optimizing for distributions of work that are the new norm for how people are doing things.
And we'll just keep on changing. We don't have too many limits, to be honest, but we'll just keep on looking at what customers are doing across a distribution of workloads and seeing if there's something that needs to be changed. The big thing for us, again, we did have a lot of conversations with customers, and they're like, "Really? I don't have that many individual objects that are that big." But with the increase of cameras and phones and things like that, we are seeing more and larger size objects, and we just wanted them to be able to grow unfettered in S3.
And so how does S3 evolve, and how has the roadmap changed? Because so far, what I picked up is everything that you told me is saying, well, our customers were doing this or that. And obviously here you live and breathe data, so you see the patterns, you see stats, you see the objects, you also talk with them. Is it only you talking with customers, seeing what's happening, what they're struggling with, what they're using more of, and then deciding to improve that, may that be the limits, may that be figuring out we need a new data type because they're now building their own data types on top of it? Or is there also some kind of, all right, here's a vision, here's a roadmap of what we'll do?
It's a great question, and in fact one of the things that we talk about all the time is the coherency of S3, right? And so there are certain things that people always expect from S3. It's the traits of S3. It's the durability and availability attributes that we talked about. And so a fair amount of engineering goes on under the hood for that. Okay. And it's a set of capabilities that we may or may not have talked about today.
In fact, if I think back to 2020, I think we've launched over a thousand new capabilities since 2020 in S3. And some of them are what we think of as the 90% of the roadmap, which is what people ask for explicitly. Okay. And so, for example, some of our media customers want the bigger object size, and so we delivered that. We have other customers that do a lot with batch operations. But then we have some things that we invent, because we look at what customers are doing with the data and we ask ourselves how we can build that. Vector kind of falls into that category.
For vector, when we looked at S3 and how S3 is evolving, we told ourselves, look, we can continue to make S3 the best repository for data on the planet. And we will. We will. We have engineers that come in every day working to make that so. But there's this other element of how do you make sure that the data that you have is in fact usable, and how do you make sure that it's usable in a way that's industry standard, like that Iceberg layer on top of our tabular data. But it's usable because AI models have now gotten so good at embeddings that you can have AI give you a semantic understanding of your data, if only you had the cost point of putting billions of vectors into storage, so you could actually understand and use your data in a different way.
And so for us, a lot of it is kind of taking a step back and looking not just at what customers ask us for, but we want to remove the constraint of the cost of data, which is what we do in S3. And we want to remove the constraint of working with your data, which is what we do in S3 too. And when we can do both of those things, if we can make it possible that your data grows as your business needs it and you can tap into all the capabilities that you're getting with AI and how the world is changing for data, then we have a shape. We call it a product shape. Then we have a product shape.
Product shape.
What's a product shape? It's sort of like an emerging… when I think about S3, I think of it as almost like this living, breathing organism where the shape of the product is evolving, but it's evolving with coherency around what you expect for the traits of S3. But it's evolving in a way that lets you steer into how you want to use data, and how you want to use data not just now but in the future. And we will continue to evolve the product shape of S3 based on what you want to do with data. And so in a lot of ways, we're sort of transcending the boundaries of what object storage was or what a database traditionally was, because now we have tabular formats, we have conditionals, and we're evolving into this new shape, and it is ultimately uniquely S3.
It kind of sounds like you have all these microservices. It's kind of evolving almost like a plant or a living organism, no?
Yes, I am in fact a former Peace Corps volunteer from forestry, and so a lot of times I will go back to the natural world for my metaphors. And yeah, I mean, S3 is this living, breathing repository of data that lets people do things with data that they never thought possible.
It's just interesting because I think as engineers we don't often think to relate the systems that we build with a living organization, when in fact, I mean, obviously there's code, but as you said, there's people, there's servers, there's failures that now happen at a cadence. You can probably predict how many hard drives are failing today, in fact, at your scale already. Do you think it's because of the scale? When things become large enough, they start to have these characteristics? Because what I find fascinating talking to you is the way engineering works inside of S3 feels very different to how it works inside a smaller organization, your kind of startup, which does terabytes of data or maybe even a few petabytes, but that's kind of it. And you've seen some of these organizations. What changes at this large scale? What do you think makes it feel pretty different, the world that you and the teams work in?
It does. So in order for us to sustain the traits of S3 and to evolve it over time, we have to constantly go back to simplification. We have a very complex system with all of our different microservices, but I kind of go back to: those microservices have to do one or two things really well, and we have to stay true to that. Otherwise the complexification of a distributed system, it's unmaintainable over time.
And for S3, there's this concept of, okay, there's a simple in S3, and the simple in S3 is a couple of things. One, it's a simplicity of the user model, where not only do you have a simple API, but now you have the simplicity of using SQL with S3, or you have the simplicity of being able to leverage these AI embedding models, which makes semantic understanding of your data so much easier than having to annotate a whole metadata layer. And so that concept of simplicity is in the user model of S3, but under the hood, if you sit in any of our engineering meetings, you will hear our engineers talk about how do we make sure that we implement this capability with the greatest simplicity that we possibly can.
Speaking of which, what type of engineers do you typically hire to work at S3, in terms of what kind of traits, potentially past experience, do you look for?
Well, we hire all kinds of engineers. We have a lot of engineers on S3 who are early career. They're straight out of school, or they're at undergrad or graduate school. And like I said, we have a ton of engineers who have been on S3 for a long time, and everything in between.
I think there's a really strong element in our teams that work on data around ownership. People feel this personal sense of commitment. I feel it. I feel it every day I come in, where I feel a personal sense of commitment to your byte, to the preservation of your byte, to the usefulness of your byte, to the ability for you to think about what your application does next and not the types of storage that you need or how you grow it. And that deep sense of ownership and that deep sense of commitment is a very, very common thread across our data teams, because we know that at the end of the day every modern business is a data business, and everything that people are trying to do with traditional systems, AI, whatever, is based on your data as shaping the core of your application experience. And so that data is our responsibility, and we feel it very deeply.
And what would your advice be to, let's say, a mid-career software engineer, someone who has a few years of experience working at different places, who, after listening to this, gets really enthusiastic and decides one day they'd love to work on a deep, strong infrastructure team like S3? For more experienced folks, what are experiences or activities that you might look for that might help you consider these folks more?
There's a strong value in relentless curiosity. And I talked a little bit about coloring within the lines, and how when you work on S3 or a large-scale distributed system which continues to reinvent what storage means, you're not really coloring within the lines. You're taking a step back and you're saying, I will draw what the lines are today, and I will know that I might have to rub those out and draw new lines in the future for wherever things go.
And so, I have three kids who are in university. I have two kids in university and one in grad school. And that is one thing that I think is really important: to always take a step back, take a look at the latest research. And some of the papers that I'll share with you are around how we either took formal methods and brought them into storage systems, or we thought about failure in a different way. That creativity, that relentless curiosity and that creativity with engineering, I don't think you can go wrong with that. I think the next generation of software, no matter if it's built in S3 or elsewhere, is all driven by the creativity of the engineering mind, and it is in all of us. We just have to unlock it and unleash it, and we'll build amazing things like S3.
And I also love that with S3, not only has S3 created something that did not exist, and I think it just was unimaginable because it didn't exist, but now I'm hearing of startups that are building on top of S3. I think Turbopuffer is a good example. They're building innovation because now they have a base layer. And I feel there's different levels of innovation. You decide where you want to innovate: at the very lowest level, one level higher, and so on. And you just use the right primitives. In your case, this is just doing hardware and storage better than anyone. In the other layers, it will be using the right primitives better than anyone.
Yeah, it's very exciting for us to see so many different types of infrastructure built on S3 now.
And as closing, what is a book or a paper that you would recommend reading, that you enjoyed, and why?
I read a lot of different papers. I am fascinated by how quickly the evolution of embedding models is coming along now, and in particular, a field of science that I'm quite interested in is the multimodal embedding model. Because, as you know, the world that we experience is multimodal, and therefore the understanding that we have of data should be multimodal as well. And so there's this whole field of science that's emerging quite rapidly around multimodal embedding models. That is something that I encourage people who are working in the field of data to look at, because I think that is the next generation of data.
If you think about the next world of data lakes, I think it's actually going to be on metadata. It's going to be on the semantic understanding of our data, and understanding how that is created through vectors and how it's being searched across multiple modalities is an important area of both research and advancement. And so that's what I would encourage people to look at in the world of data. I think vector is going to be quite big, particularly at the price point that we've introduced for S3 storage for vectors. And I'm excited about it. I think we're just getting started with data and an understanding of our data, and I can't wait to see what comes next.
Amazing. And do you have any book recommendations?
I will give you a book recommendation, just in case your readers are interested. It won't be in the field of computer science. It will be about the evolution of the ecology around us and supporting the bees, the native bees and insects around us. So, a tiny bit farther afield, but I'll give you a book recommendation, and if your readers are interested, they can take a look at how to support the bees of the planet.
Well, Mai-Lan, thank you very much. This was fascinating and very interesting, to get a peek into this massive world of scale of data, and respecting the byte and treating it and making sure that it's durable.
It was great talking to you, and thank you both to yourself, I know you're a fan of S3, and to all of your listeners who use S3. We quite literally wouldn't be able to do what we do without the feedback and the encouragement from everybody who uses S3 today. So thank you for that.
Just wow. I always suspected there's a lot of complexity behind a system like S3, but I just did not realize the scale of it. Whenever I worked on systems with even hundreds of virtual machines, failure of one machine was a rare event and not something that we really counted on. During my conversation with Mai-Lan, she casually mentioned that several machines have failed during our conversation, which is something that the S3 team knows and prepares for and treats like an everyday event.
I personally really liked how AWS has two conflicting tenets heavily used on the S3 team: respect what came before, and technically fearless. For such a massive system, it would be easy to say let's move conservatively because of how many companies depend on us. But if they did so, S3 would fall behind.
Finally, I'm still in awe that AWS put strong consistency in place, rolled it out to all customers, and did not increase pricing, nor did they increase latency, at S3 scale. This is an absolutely next-level engineering achievement. In fact, it was probably one of the lesser-known engineering feats of the decade.
I hope you found the episode as fascinating as I did. If you'd like to learn more about Amazon and AWS, check out the exclusive deep dive I did with AWS's incident management team on how they handle outages in the show notes below. In the Pragmatic Engineer, I also did other deep dives about Amazon and AWS. They are also linked in the show notes. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show.
Article published