Inside the RubyGems Compact Index: How bundle install Finds Gems, and What's Changing
Ruby on Rails
Jenny Shen, a senior developer at Shopify based in Ottawa and a maintainer and operator of rubygems.org, gave the last talk before the closing keynote at Rails World. The subject was the RubyGems compact index, the system Bundler queries whenever it needs to resolve dependencies. Few people in the audience raised their hands when Shen asked who had heard of it. Shen's case for discussing it at a Rails conference was that installing gems is a core part of running a Rails application. The team found that the compact index sat at the center of recent work on performance, reliability, and security for that process.
The talk had three parts: how the index works internally and how Bundler talks to it, two recent changes (dependency cooldowns and content-addressable gems), and ideas the team is considering next.
A Magic Trick as a Metaphor
Shen opened with a small magic trick. Michelle, the host, was asked which major Rails version they had first developed on. The answer was "four." Michelle then pulled a card with a four on it out of Shen's pocket. Shen explained that the trick relied on a "Quiver compact index," a wallet that holds up to 16 cards and lets you pull out a specific one quickly. The commercial version had sold out online, so Shen built a homemade "V2."
Shen also mentioned a compact index algorithm from finance, which reduces hundreds or thousands of stocks to a handful while preserving diversification. The common thread is compressing and organizing a lot of information so it can be retrieved quickly or used simply. The RubyGems compact index does the same job for Bundler: it compresses information about more than 200,000 gems on rubygems.org so Bundler can resolve dependencies.
The Three Endpoints
Shen said the compact index runs on three endpoints and serves more than 50 million requests per day. The endpoints are info, versions, and names. Bundler uses only the first two.
Bundler has to re-resolve and consult the index in three situations: there is no Gemfile.lock, the Gemfile has changed, or some dependencies are not available locally. Its first step is to make sure it has an up-to-date list of gems, which comes from the versions endpoint. Each line in that file lists a gem name, the versions available, and an "info checksum" that Bundler uses later.
The index is also stored on your machine in Bundler's cache directory, and Bundler tries to keep the local copy in sync with the server. To update the local versions file, Bundler sends a request with an ETag. The server returns only the new bytes, meaning the newly added or updated entries. In Shen's example, one gem's line gets updated locally this way.
Next, Bundler needs more detail about each gem to decide whether it can be installed. That detail includes transitive dependencies and their version constraints, Ruby and RubyGems requirements, and the gem checksum. The checksum matters if you store checksums in your lockfile and Bundler needs to compare against them. The info endpoint provides all of this, with one row per version listing the dependencies and the requirements, including the checksum. Info files are cached locally too. To decide whether a cached info file is stale, Bundler hashes its contents and compares the result with the info checksum from the versions file. If they match, Bundler skips the refetch. If they differ, it downloads the new info file.
What Happens When a Gem Is Pushed or Yanked
On the server side, pushing a gem to RubyGems triggers background jobs that update the versions file and the gem's info file. Both are written to an AWS bucket. Shen explained the bucket's role: the CDN checks each incoming request, and compact index requests are redirected to the bucket and served from it. If the file doesn't exist or something is wrong with the bucket, the CDN falls back to the application server.
A push appends a new line to the versions file and updates the info file. A yank also adds a line to the versions file, one that subtracts the version, and removes that version from the info file. Many pushes and yanks would make the versions file grow large, so a monthly rake task consolidates each gem's redundant lines into a single line.
Dependency Cooldowns
The first new feature Shen covered is security-related: dependency cooldowns. If you set a cooldown of, say, seven days, Bundler installs a new gem version only after it has been public for seven days. A Rails release published today would not be installable under that setting until a week had passed.
Shen's motivation was a recent rise in package takeover attacks, where an attacker gains control of a maintainer's account and pushes a malicious version. Shen attributed part of that rise to AI making account takeovers easier. The incident that "inspired it all," in Shen's words, happened earlier this year in the npm ecosystem, when Axios was compromised: someone took over the maintainer's account and pushed a malicious release. A cooldown gives scanners time to detect malicious versions, and gives maintainers time to yank them, before they reach most users. Shen noted that PyPI and npm had quickly added cooldowns to their tooling and said RubyGems should follow.
Adding Timestamps to the Index
To enforce a cooldown, Bundler has to know when each version was published, and the info file had no timestamps. The compact index spec allows additional entries in each row's requirements section, so adding a created_at value there was simple in principle.
The difficult part was operational. All the info files live in the AWS bucket, so supporting the new field meant regenerating about 200,000 files, one per gem. The team compared approaches and chose a blue-green migration. Shen described the steps:
- The CDN kept pointing at the current index.
- Every new push and yank updated both the current index and a new copy under a
v2subdirectory. - The info files for all ~200,000 gems were backfilled into the new directory.
- When the backfill finished, the CDN was switched to serve the new directory.
The deciding factor, Shen said, was reversibility. If something went wrong, the team could point the CDN back at the existing version.
The compact index now includes creation timestamps, and HSBT added the cooldown feature to Bundler. You can pass a cooldown flag to install commands, set it in your configuration, or declare it in your Gemfile. Shen pointed the audience to a blog post with more details.
The Performance Problem: Native Extensions
Shen's second topic was performance. Shen held up uv, the Python package manager, as an example of how fast package installation can be and said RubyGems should keep pace. The team found that more than 70% of bundle install time goes to compiling native extensions.
Native extensions are C or Rust code that some gems include for performance, and that code must be compiled before use. During bundle install, Bundler runs the compilation script specified in the gemspec.
The existing remedy is precompiled gems. Shen used Nokogiri as the example: it publishes separate builds for each platform, and each platform build contains binaries for several Ruby ABI versions. Shen explained that the ABI (application binary interface) changes with each minor Ruby version because internal data structures change, so a binary compiled for one Ruby version won't work on another. Each version needs its own compiled binary.
More precompiled gems mean faster installs, but Shen described several drawbacks of the current format:
- Precompiled gems can be large, because one gem file bundles binaries for multiple ABIs, while most users run only one Ruby version and need only one of them.
- Maintainers have to write code in their gem to load the correct binary at runtime.
- When a new Ruby version comes out, an existing gem release can't be updated to add a binary for the new ABI.
Publishing a separate gem for each ABI would produce smaller gems, faster downloads, and less configuration for maintainers. It would also let maintainers add support for new Ruby ABIs after a release.
Content-Addressable Gems
The team added exactly that capability under the name content-addressable gems. Shen credited the original idea to Aaron Patterson. A content-addressable gem is a precompiled gem scoped to a single Ruby ABI.
The naming had to change. Today a precompiled gem's name carries a platform suffix and the gem contains multiple ABIs. A content-addressable gem instead carries a SHA suffix and contains a single ABI. Shen said the rename was required because the current scheme can't publish several ABI-specific builds of the same version without the names colliding.
In the compact index, the slot that used to hold the platform now holds a suffix equal to the first eight characters of the checksum. The platform moves into the requirements section, so Bundler still gets that information, and the Ruby requirement is narrowed to a single Ruby ABI.
One caveat came with the rename: older Bundler clients interpret the new suffix as a platform and will crash if they try to install such a gem. The team therefore made content-addressable gems visible only to new clients by adding a required RubyGems version to their requirements, which excludes older clients.
The change has landed in rubygems.org and in the RubyGems and Bundler clients. Initial support ships in RubyGems 4.1 beta, which Shen said had been released a week or two before the talk.
What It Means for Maintainers and Users
Maintainers of gems with native extensions can push content-addressable gems using the Ruby ABI option, specifying the required Ruby version and required RubyGems version. The team is updating build tooling to support this. A pull request is open against rake-compiler, and Shen hoped that within a few weeks people who use rake-compiler to precompile their gems could try publishing content-addressable gems with it.
Shen also asked maintainers whose gems have native extensions but no precompiled builds to start precompiling. Shen recommended trying a cibuildgem created by Edouard, which generates a GitHub workflow that precompiles the gem for each platform and pushes the builds to rubygems.org.
For users, Bundler prefers content-addressable gems over the legacy multi-ABI versions when both exist. They appear in a new section of Gemfile.lock.
Measured Impact
To estimate the impact, Shen pushed single-ABI versions of every precompiled gem in a new Rails application's bundle. Shen reported that this cut the bundle's total download size by about 20% and called that result "okay." Shen said the larger payoff lies elsewhere: more than a dozen gems in that bundle have native extensions but aren't precompiled, so the bigger savings will come as those gems adopt precompilation.
Future Ideas
Shen closed with ideas that have not been built.
The first is security warnings. Shen would like Bundler to tell you when you're using a vulnerable version of a gem, possibly by adding advisory information to the compact index.
The second is more requirements on precompiled gems to guarantee compatibility, starting with the Ruby engine. Shen described a current RubyGems bug: on an alternative implementation such as TruffleRuby, binaries built for CRuby can get installed. Recording the engine in the index would prevent that and would make it possible to publish binaries for other engines.
Closing
Shen asked attendees to turn on cooldowns and to test the RubyGems beta releases, because finding bugs now gets them fixed before the stable release. Shen pointed to the release page for details on the new features, said blog posts on them are coming, including one from Shen on content-addressable gems, and thanked the people who worked on the project.
Awesome.
Thanks, Michelle. Hey, everyone. Thanks for coming, especially since this is the last talk before the ending keynote. I've been having a great time at Rails World so far. Even though the weather is so hot, it's amazing to hang out with cool folks like y'all.
Today I'll be talking about the RubyGems compact index. To get us started, who has heard of the compact index before? Raise your hands. Looks like there's a couple of you. For those who don't know, and to get us warmed up a bit, I have a simple magic trick for this. I'll get Michelle to get back up here. Let's have a round of applause for Michelle.
Amazing. Simple question for you.
Okay.
What is the first major Rails version you started developing on?
Four.
Okay.
Yeah.
Well, that number has traveled a meter and is in my pocket right here. Would you mind opening?
Okay.
And seeing what it is.
It's a four.
Whoa. Amazing. Okay, I'll get that back. Thanks, Michelle.
A magician never spoils their secrets, but I'm not really a magician, so I'll share the secret to this trick. It's the compact index. More specifically, it's the Quiver compact index. It is a wallet that stores up to 16 cards and allows you to quickly take them out. Unfortunately, it was all sold out online, so I made my own right here. I think it's definitely an upgrade. It's V2. So, yeah, that is the V2 version.
But the compact index is actually also an algorithm. It's related to finance. It compresses hundreds to thousands of stocks within an index, and it compresses it down to only a handful while maintaining diversification.
This isn't the compact index I'm supposed to be talking about, though. It's the RubyGems compact index, but they all share a similar theme. They compress and organize large amounts of information for quick retrieval or simplicity. The Quiver compact index compresses a deck of cards into an organized wallet, while the financial algorithm compresses hundreds of stocks into a tiny fraction while maintaining diversification. And the RubyGems compact index is not that different. It compresses info from all 200,000 gems on RubyGems.org for Bundler to resolve dependencies.
RubyGems is important for the Ruby and Rails ecosystem. You might be thinking, why are we talking about RubyGems at Rails World? It's supposed to be a Rails conference. Well, gems and gem installation are an important part of running a Rails application. It's important to keep it up to date by improving things like performance, reliability, and security. When we made these improvements, we found that the compact index is a core part of making these changes.
Today, we'll be talking a little bit about the internals of the compact index and how it interacts with Bundler, recent evolutions to the compact index and the new features in Bundler, and some future additions that we can make. Hopefully, out of this talk, you can use some of these new features to level up your dependency installation game.
But before we get started, let me introduce myself. I'm Jenny. I am not a magician, but I can be considered a software magician. I can make code appear out of nowhere. The secret to that is AI. And I can sometimes make bugs magically appear as well. I work at Shopify as a senior developer, and I'm based in Ottawa, Canada. I'm also a RubyGems.org maintainer and operator. We always welcome contributions to the project, so if you find some of the stuff interesting, come find me afterwards.
Okay, let's talk a bit about the RubyGems compact index. As I mentioned before, it compresses information from more than 200,000 gems on RubyGems.org for Bundler to use. It runs with three endpoints and serves over 50 million requests per day. These three endpoints are the info endpoint, the versions endpoint, and the names endpoint. Bundler only uses the former two.
When you bundle install, if there's no Gemfile.lock, there's changes to the Gemfile, or if there's dependencies not found locally, Bundler will need to re-resolve and interact with the index. First, it'll make sure there's an up-to-date list of gems, and this is the versions endpoint. As you can see here, this is what it looks like. This is the list of all gems available to download. You can see the name of the gem, the versions available, and something called the info checksum that we'll get back to later.
The index is also stored locally on your machine, and Bundler tries to keep them in sync. It's stored in the bundle cache directory on your machine. Bundler will try and update the versions file locally if it needs to be updated. It will request with an ETag, and the index will return any bytes that are new, or new versions updated here. So the gem called d gets updated locally.
Then Bundler needs more info about each gem in order to resolve or determine if it can be installed. Some of this includes the transitive dependencies and their requirements, Ruby or RubyGems requirements, and the gem checksum. If you have gem checksums stored in your lockfile, it needs to compare. This is exactly what the info route does.
This is an example. It gives the version for each row, the dependencies, and also the requirements, like the checksum. It's also stored locally, and to determine if it needs to be updated, Bundler will calculate a hash of the contents of the file. If it matches the info checksum we mentioned earlier, then it doesn't need a refresh. But if the versions checksum is different, it'll request the new info file.
Something else that I want to talk about is what happens when a new version gets pushed. When a gem gets pushed, RubyGems will fire a background job to update the versions file. It'll do the same for the gem info file, and it gets updated to an AWS bucket. You may be asking, why are we updating to an AWS bucket? Well, our CDN checks if a request comes in, and if it's a compact index request, it'll redirect to the AWS bucket and serve it through there. It'll fall back to the server if the file doesn't exist or there's something wrong with the bucket.
When a gem gets pushed, the new gem gets appended to a new line in the versions file, and it gets updated in the info file. When a gem gets yanked, it'll subtract that version and remove it from the info file. You may be thinking, "Oh, if there's a lot of pushes and yanks, wouldn't the versions file get really large?" Well, we have a monthly Rake task that consolidates all of the redundant lines of a gem and rolls it up to one line here.
So, yeah, that's a little bit about the index and how Bundler interacts with it. Now we can talk about some of the exciting new changes we have to the index. The first one is related to security, more specifically dependency cooldowns.
If you specify a cooldown of something like seven days, then new versions of the gems only get installed seven days after publish. If a new Rails version comes out today, you will need to wait. After seven days, it will be able to be installed.
You may be asking, why do we need this feature? Well, over the recent while, there has been an increase of gem takeover attacks, more so because of AI being able to take over maintainers' accounts and push malicious versions. The one that inspired it all is earlier this year in the npm ecosystem, Axios was compromised. Someone was able to take over the maintainer's account and push a malicious version. Cooldowns allow time to detect malicious gems before installation. Scanners will be able to detect them and then yank them. PyPI and npm quickly added cooldowns to their tooling, and RubyGems should follow suit.
How do we do that, you may ask? Bundler would need to know publish timestamps of the gems. But as you can see here, there's no timestamps or create info in the info file. Luckily enough, in the compact index spec, you can add more requirements in the requirements section. So we can easily just add the created_at information here.
What makes this difficult, though, is, as I said before, all of the files are stored in an AWS bucket. So we had to regenerate 200,000 files, for each gem, for each info file, in order for this to work. We investigated different approaches, but we landed on something called the blue-green migration. Essentially, what we did is that we have the CDN pointing to the current index, and then for new pushes and yanks, we update the current index and the new one under a V2 subdirectory. Then we backfill 200,000 gems' info files into the new directory.
Then, once that was completed, we were able to direct the CDN to serve the new directory. We chose this approach because if something did go wrong, we could easily revert back and go to the pre-existing version.
Now there is created_at info in the compact index, and hsbt added the cooldown feature in Bundler. You can add the cooldown flag to your gem install methods. You can set it in your configuration and in your Gemfile. You can also read more in this blog post right here.
Cool. That's cooldowns. Next, I want to talk about performance. uv is super fast, and in order for RubyGems to stay up to date, we want it to be fast as well. We found that more than 70% of bundle install time is compiling native extensions, which is a lot. You may ask, what are native extensions? Well, some gems have C or Rust code to make the gem more performant, and they need to compile before use. When you bundle install, Bundler will run a compilation script specified in the gemspec right here.
There is a pre-existing solution to this, though, which is having the gem precompiled for you. An example of this is Nokogiri. You can see there's multiple versions compiled by platform, and within each platform, there are many ABI versions for each gem. An ABI is an application binary interface. What that means is that with each minor Ruby version, there are data structures and internals that change, so binaries aren't compatible between each Ruby version. You need to have one compiled for each one.
The more precompiled gems, the faster, but there's one problem, or one thing that is suboptimal: a precompiled gem can be big. As I mentioned before, a precompiled gem supports multiple ABIs, but most of the time, you'll only need to use one because you'll probably use one Ruby version. Also, gem maintainers need to add code in their gem to load the right binary as well. Being able to publish gems for one ABI allows for smaller gems, faster downloads, and less configuration for maintainers.
Another perk is that for existing gems, when a new Ruby version comes out, you can't update the gem with new Ruby ABI versions. It would be great to be able to push new Ruby ABI support afterwards. And we added that exact ability.
This is coined as content-addressable gems. This concept was originally proposed by Aaron Patterson, and it basically is a precompiled gem scoped by Ruby ABI. As before, the gem is platform-suffixed and is multi-ABI, and now it's SHA-suffixed with a single ABI. The reason why we had to make this naming change is that currently we cannot specify multiple ABIs per version because that will cause a name collision.
What does that mean for the compact index? It means that the suffix, what was the platform, is now a suffix that matches the first eight digits of the checksum. We move the platform to a requirement so we can get that information still. We also scope the Ruby requirement to a single Ruby ABI.
There's one caveat. By doing this rename, old clients of Bundler think this is a platform. If older clients do install this, it'll kind of crash out. This will only be available for new clients, and we guarded against this by adding a required RubyGems requirement here that only scopes to newer clients.
We added this change to RubyGems.org as well as RubyGems and Bundler. The initial support is part of RubyGems 4.1 beta, which was released a week or two ago.
For gem maintainers of native extensions, you can push content-addressable gems with the Ruby ABI option, and they'll specify the required Ruby version and required RubyGems version. We're currently updating tools in the toolchain to support pushing content-addressable gems. We have a PR up to rake-compiler, so hopefully in the next couple of weeks, folks that are using rake-compiler to precompile their gems can try pushing content-addressable gems.
If you own a gem that has native extensions, but they aren't precompiled, you should precompile them. You can try out the cibuildgem that Edouard created. Essentially, it's a gem that helps you create a GitHub workflow that will help you precompile per platform and push it up to RubyGems.org.
For people that install content-addressable gems, they will be preferred over the legacy multi-ABI versions. You'll see them in your Gemfile.lock in a new section.
You may be thinking, okay, what's the impact of all of this? We can push single ABIs, but what's the impact? Well, I pushed single-ABI versions of all the gems that are precompiled for a new Rails version, and it reduces the download size of the entire bundle by 20%. It's okay, but the real savings here is that there are more than a dozen gems that have native extensions but aren't precompiled. So we're unlocking a lot of future savings.
Cool. Those are some of the new features that have been shipped over the past year. Let's talk about some future ideas. The first one is related to security, like cooldowns. These are security warnings. It would be great for Bundler to help you identify when you're using a vulnerable version of a gem. We can add that into the compact index by possibly adding some advisory information.
Another feature that we can add is more requirements to precompiled gems to ensure correct compatibility. Right now, we lack Ruby engine information. There's a bug currently in RubyGems where if you're using another Ruby implementation like TruffleRuby, CRuby binaries could be installed. We can add the engine information so that we can compile more binaries for different engines.
Okay, great. To conclude, we talked about the compact index internals and how it interacts with Bundler, some recent evolutions to the index and the new features in Bundler, as well as future additions to the index. Hopefully you can use some of the new features like cooldowns and try out the beta versions of RubyGems. That will be very helpful to figure out bugs and get them solved before the stable release gets released.
You can take a look at more about all the new features that are coming in the release page, and look out for blog posts on these new features. I'll be posting a blog post about content-addressable gems soon, so keep an eye out for that. I would like to thank all these people that helped out with all this work. It's been great to work with y'all. And, yeah, thanks for listening.
Article published · Updated