Hotcell: Sandboxing Active Storage After an AI-Discovered Rails Vulnerability
Ruby on Rails
Mike Dalessio, the newest member of the Rails Core team, had planned to give a Rails World talk about a Markdown API for Action Text. He described it as "conference driven development": get the talk accepted, then write the code before the conference. Instead he spent most of the summer reacting to security issues in Rails. That work felt more important and more urgent, so the talk became about security.
His framing is simple. Humans have written a lot of software, and some of it has bugs, unexplored corner cases, or designs that are very trusting. AI has become very good at finding those problems, and Dalessio says security is "really chaotic right now." He calls himself a worrier, and he set up the talk to scare the audience first and then give them the tools to act. His message to anyone who thinks he worries too much: "you're not worried enough."
The CVE that started the summer
The summer began with a vulnerability Dalessio cites as CVE-2026-66066. When it came time to publish the advisory, he deliberately wrote it in the most opaque language he could and left out all the details. He was worried the flaw was severe enough that attackers would go after systems before teams had time to upgrade Rails, so he wanted to buy them that time.
The advisory had four sentences. It affected essentially every Rails version ever released and said that Active Storage did not disable libvips's untrusted loaders. He went through it sentence by sentence and used it to explain how Active Storage works.
Active Storage is the Rails subsystem for storing images and other blobs and for transforming them. The typical case: a user uploads a file, Active Storage saves it, and later the app wants a thumbnail for an activity feed, so Active Storage produces a smaller variant. The CVE concerns the transformation half. In the default Rails configuration, a vips transformer class calls into libvips.
Over a hundred libraries behind libvips
Libvips does not handle every format itself. It inspects the incoming file, decides which library should handle it, and routes it there. Some of those libraries are familiar, such as libpng and libjpeg. Others are less reliable; Dalessio pointed to ImageMagick and its history of security problems. Others are libraries most Rails developers have never heard of. His example was libmatio, which reads MATLAB files: "For all of your MATLAB Rails apps." Libvips knows about it, even if you don't.
When Dalessio installs libvips on the Ruby slim Docker image, the dependency list is more than 100 libraries. He told the audience there is no way all of those are secure and that vulnerabilities exist in some of them right now. Libvips does distinguish trusted from untrusted loaders, and ImageMagick and libmatio are marked untrusted. That is why the advisory mentioned untrusted libraries.
That points to the shape of the attack: upload an image crafted to trick one of those libraries. Who is vulnerable? Anyone using libvips who accepts image uploads from untrusted users. Since libvips is the Rails default and, as Dalessio put it, every user should be treated as untrusted, that means essentially every Rails app.
What an attacker could get
The exploit let an attacker read arbitrary files, including environment variables and application secrets. Image processing runs in the same container as the application, alongside the source code, config files, credentials, and environment. If secrets are injected through environment variables, they are visible to libmatio in this scenario. The trick was that the thumbnail returned to the attacker would contain those secrets.
The last sentence of the advisory described the real impact. With secrets in hand, an attacker could achieve remote code execution and move laterally: use a database secret to tamper with the database, use Redis credentials to tamper with the cache, and possibly reach corporate networks, depending on how secrets are set up. The issue scored 9.5 on CVSS, "about as bad as it gets."
Dalessio said the worst part was what happened after publication. Working only from the deliberately vague advisory, people reverse-engineered the attack in under an hour and published their results online. "I bought you 60 minutes of time to upgrade."
Seven small flaws, chained by agents
Dalessio stressed that there was no single big vulnerability. There were seven small issues that, he said, AI agents chained together from very little information:
- Direct uploads were enabled by default. Even apps that didn't use Active Storage exposed the route to attackers. That default has since been changed.
- Direct uploads never examine the bytes. The file goes from the browser straight to the blob store without passing through the Rails app, so Rails has to trust the content type the client reports. Lying about the content type was the way in. This is a performance trade-off, since downloading and re-analyzing every file would be costly.
- Signed variation keys. These make the attack easier to pull off but aren't a security problem in themselves, and Dalessio said Rails probably won't change them.
- Not calling libvips's "block untrusted" function. This was the actual fix in Rails. Dalessio noted that libvips shipped the secure behavior as an opt-in feature buried in a changelog.
- Libvips was fooled by a spoofed MATLAB header. This was the actual flaw. It was fixed, but it was small enough that no CVE was published.
- Libmatio said this was working as designed. Its maintainers' position was that untrusted files shouldn't be handed to libmatio in the first place.
- Known escalations from the secret key to remote command execution. The Rails security team knows several ways to turn a leaked secret key into remote command execution. Fixing them requires breaking changes, so, as Dalessio admitted, the team has been "dragging our feet." He called this the really bad link in the chain.
To drive it home, he showed a video of the exploit script pulling every secret out of a development app in real time. Once you know what to do, he said, it is very fast.
Why fixing one CVE isn't enough
Dalessio added two more data points. So far this year, 378 vulnerabilities have been published in libvips and its dependencies, with more reported every day. He also showed data from 37signals' HackerOne bug bounty program, where valid reports are "going parabolic."
That is the deeper problem. He could fix this one CVE, but the next zero-day in some image-processing library would put everyone through the same thing again. He frames the fix in terms of risk. The probability of another vulnerability in these libraries is very high and not likely to drop soon, so the only lever left is impact: how much damage can be done when the next one is found. The question for the second half of the talk was how to run Active Storage so that it doesn't matter if a vulnerable library processes an untrusted file. His answer, like most people's would be, was a sandbox.
Requirements for the sandbox
Dalessio presented the design as "five easy steps," starting with requirements. That list also became the prompt he gave his coding agent as they built the system together:
- Drop-in replacement. Moving to the sandbox should be a configuration change, not a code change.
- Least privilege. A very limited user, a filesystem that doesn't allow setuid, all Docker capabilities dropped, and no Rails privileges.
- Hard limits on CPU time, memory, file size, number of files, and number of processes, to make an exploit as hard as possible to turn into damage.
- No access to the app's filesystem or environment, so there are no secrets to exfiltrate.
- No network interface at all, so a compromised process can't call out or move laterally.
- Disposable workers. A process handles an image and is then killed, limiting the lifetime of any remote command execution.
- A familiar deployment lifecycle, easy to deploy with Kamal or Kubernetes.
- Extensibility. Active Storage is one thing built on top, but developers should be able to build their own operations, for example a secure place to unzip files.
The name comes from a real-world analogue. Dalessio showed a photo from Oak Ridge National Laboratory of a "hot cell," a large lead-shielded box for handling radioactive material. The window is filled with mineral oil so radiation can't escape, work is done with remote manipulators, and everything going in or out is tightly controlled. That is what he wanted for Active Storage, and naming it was step two.
Plugging into Rails
Rails already had the hooks he needed. Three config parameters each control one part of Active Storage:
- Analyzers extract metadata, such as how many pages a PDF has.
- Variant processors do transformations like rotating, cropping, and resizing.
- Previewers generate thumbnails for non-image files such as PDFs or videos.
Because existing classes with existing APIs sit behind those settings, Hotcell can provide matching twin classes. Switching should be a config change, and switching back should be just as easy.
Hotcell is five gems: hotcell-core for shared pieces; a Hotcell client and server that define how work is done, how error codes are returned, and how the two sides communicate; and Active Storage client and server gems that subclass them to make the drop-in replacement possible.
Unix sockets and passing file descriptors
The open question was communication between client and server without a network interface. The answer is Unix sockets: a file on disk that processes can read and write, a common way to do interprocess communication, and not routable on a network.
The key detail, which Dalessio called "like magic," is that the sendmsg system call can pass open file descriptors between processes. When an image is uploaded, the Hotcell client in the Rails app opens the file and hands the descriptor to the Hotcell container. The container can't see the host filesystem, but it can use that one file because it was given the handle. All inputs and outputs work this way.
Step five, actually doing the work, is a supervisor/worker design like Active Job or Puma. Dalessio said it was straightforward and didn't dwell on it. The finished system, he said, is running in production at 37signals now. Having scared the audience, he told them they should run it too.
Rolling it out
After installing the gem, hotcell install creates three files in a hotcell subdirectory:
- A Gemfile.
- A Hotcell config file. It is plain Ruby running in a Ruby process, not a Rails process. It sets global limits, which he suggested tuning once things are running.
- A third file used to configure the cell itself.
For deployment, the only thing shared between the app and the cell is a directory containing two Unix sockets. That directory is configured in the app and in the accessory configuration. The accessory configuration also sets volumes to match the application, Docker container limits, and capabilities, including network: none, which Dalessio pointed to as proof that the system needs no network.
On the Rails side, you replace whichever analyzer, variant processor, and previewer classes you use with their Hotcell twins, set the two socket properties, and you're done. In the cell's configuration you require classes explicitly, because it isn't a Rails app and Zeitwerk autoloading doesn't work there. This is also where per-operation limits go: the cell might have a 10-second deadline overall, but a specific operation could be capped at 2 seconds. For a vanilla Active Storage and Action Text app, he said, that's all it takes: configure Active Storage, configure Hotcell, deploy.
Observability, cost, and flexibility
The observability model follows from the trust model. The worker inside the container can't be trusted, because it might be compromised. The supervisor and the Rails app can be. So the Rails app logs Hotcell calls the way it logs database hits, the app periodically pulls metrics such as queue depth from the supervisor, and the supervisor writes JSON logs to stdout, including which children were killed and why. 37signals uses this for an internal Grafana dashboard, and alerting can be set up however a team prefers.
The overhead is about 8 milliseconds per call. Dalessio said that was lower than he expected, though maybe still higher than people would like. He thinks it can come down and believes the current cost is related to garbage collection. His view is that it's cheap insurance against getting "zeroed" the next time ImageMagick has a problem.
Other options include running multiple cells on different sockets with different queue depths, timeouts, and deadlines; operations with multiple input and output files; and bringing your own container in a corporate environment, with a conformance test that checks whether the container is configured correctly.
Writing custom operations
Custom operations borrow heavily from Active Job. An operation class has a name (his example was "extract text"), default limits that can be overridden, and a perform method that takes whatever arguments it needs. In the Rails app, a thin shim calls perform_in_hotcell, analogous to perform_later for jobs. Hotcell serializes the arguments, runs the operation in the cell, and returns the resulting file.
The one extra step is opening the input and output files before the call, because the cell needs live file descriptors. Dalessio expects that to be hidden inside whatever class wraps the operation.
Lessons from production
Rolling Hotcell out in HEY and Fizzy, both relatively modern apps using vanilla Active Storage, was "absolutely no problem." Basecamp was harder. Active Storage was extracted from Basecamp, which never went back and cleaned up, so Dalessio had to write many custom operations. Basecamp also uses attachments heavily; its users, he said, love sending each other memes.
There were a couple of outages along the way. He didn't go into detail, only that he learned from them and folded the lessons back into the library. He recommends budgeting time for tuning.
He considers Hotcell ready to use now but not yet 1.0. He is waiting for a Rails release, wants to add more observability features and some syntactic sugar, and may add Linux Landlock. He asked anyone with Landlock experience to talk to him. As he described it, Landlock is a kernel feature that lets a process ratchet down its own permissions permanently. For example, a process could give up the ability to fork and exec, and even an attacker who takes over the process couldn't get it back. He called this a "suspenders" layer on top of the existing protections.
From fear to a call to action
In closing, Dalessio said it's easy, and forgivable, to blame AI for this situation, but he wanted a more nuanced view. He sees this as a chaotic period that is trending in a good direction. The alternative to working through it is that "we all quit our jobs, turn off our computers, and become sheep farmers." He wants the Star Trek future, not the Terminator one that David had mentioned that morning.
He reframed the scary HackerOne chart as the system working as designed: people are finding vulnerabilities and reporting them responsibly. It's frustrating to deal with, but the number of vulnerabilities is finite, the current rate can't last, and eventually there will be a "last vulnerability party." He also corrected his earlier statistic: the 378 libvips-related vulnerabilities were not just found, they were fixed. Maintainers are fixing problems and shipping updates, and he takes that as reason for confidence.
He then gave three things the audience can do:
- Ship agent skills with vulnerability reports. In response to the CVE, he published a repo extracted from the work he did to check 37signals' own apps. You can point an agent at it and at your application and ask whether you're vulnerable ("Probably") and whether you were exploited. The agent goes through your Active Storage database records to answer. If you need to rotate your secret key, you can point it at the Rails guides for help. He sees this as how the community gets through the current period.
- Attack your own work. While building Hotcell, he ran an adversarial review on almost every commit. Jeremy helped with red teaming by setting agents loose to break in, and they found gaps. That doesn't make a library perfect, but it means fewer problems, and doing it across our own software gets everyone through the chaos faster.
- Improve or replace existing systems. In his view, the only way through is for software to get better or be replaced, and Hotcell is his example of the latter.
He ended by asking people not to be too scared, but to be proactive and constructive, so that the future is "Star Trek, not Terminator."
Hey everyone. So the original intention for this talk was to talk about a Markdown API for Action Text, and it was aspirational in nature. It was like conference-driven development: I'll get the talk accepted, and then I'll write the code in the meantime.
And something strange happened over the year, which was I spent most of my summer heads down working in reactive mode on security issues in Rails. And I worked on that instead of the Markdown API because it felt more important and more urgent. And then when it came time to prepare the talk, I was like, well, maybe I should give a talk about the thing that's important and urgent. And so instead of a Markdown API talk, you're going to hear a security talk. Thank you in advance for your patience.
In the beginning, the earth cooled. Dinosaurs roamed the earth. Humans started building things and started writing software.
A lot of the software written by humans was not great quality. Some of it had mistakes. Some of it had bugs. Corner cases weren't totally explored. Maybe the design of the software was very trusting in nature.
You are here. AI has created all kinds of chaos because AI is really good at finding these problems, and security in particular is really chaotic right now. And I'm a worrier. A lot of what I do is security, and so I worry a lot, and I worry about the future. I don't know what's coming. This morning David talked about maybe it's going to be Terminator, maybe it'll be a more optimistic sci-fi movie.
But at this moment right now, I'm very worried, and you should be too. And so I'm going to give you a talk that will get you scared before I equip you to go and be proactive and fix things.
So you may say to me, Mike, you worry too much. But I'm saying to you, you're not worried enough.
My summer started with CVE-2026-66066. Don't read this, by the way. Focus on me right now. Okay, I'll explain it all.
When it came time to write this CVE and publish it, I intentionally chose the most opaque, obtuse language I could, and we did not disclose any of the details because I wanted to buy you time to upgrade Rails. I was really worried that the nature of this is so bad that there would be attackers immediately going after your systems before you had a chance to upgrade. So I wrote it to be very obtuse.
Four sentences. Let's take them one by one. In versions prior to blah blah, every Rails version ever, Active Storage does not disable libvips. So there are some underlined terms here. You're free to read now.
I just want to maybe define those terms. It gives me an opportunity to go into exposition mode and explain a little bit about Active Storage too. If you're not familiar with Active Storage, it's basically Rails's subsystem for storing images, blobs, whatever you want, and for transforming them. So the typical case is I'm using Lexxy. I upload a file, Active Storage saves it, and then I want to make a thumbnail for my activity feed, a little version of it, and Active Storage transforms it and does that. And so this CVE is about the second half of this. It's about transforming.
And specifically, Rails calls VipsTransformer, is the class that calls out to libvips on the bottom. Okay, this is the default configuration for Rails, and libvips does something really interesting, which is it outsources the work, right? The image gets handed to libvips. Libvips looks at it, tries to figure out who should handle it, and routes it to the right library.
And some of these libraries you know and love: libpng, libjpeg, old reliable. ImageMagick, maybe a little less reliable. It's got a history of security problems. But there are also libraries that you don't even know about. Libmatio. Does anybody know what that library is? Go ahead, shout it out.
MATLAB. MATLAB. For all of your MATLAB Rails apps.
So libvips knows about libmatio. You don't. And in fact, libvips—don't read this slide either—but when I install libvips on the Ruby slim image, this is the dependency list that it installs. It's over a hundred libraries. And I'm here to tell you that there's no way that those are all secure. There are bugs, there are vulnerabilities in some of those libraries right now.
And libvips has a concept of trusted library versus untrusted, which is why that sentence I showed you earlier mentioned untrusted libraries. So ImageMagick and libmatio are marked as untrusted. But we're starting to get a sense for the shape of the vulnerability now, right? It's like, oh, an image is uploaded that's carefully crafted to try to trick those libraries into doing something. Cool. What are we tricking the library into doing?
Well, first let's talk about who's vulnerable: when you're configured to use libvips and you accept image uploads from untrusted users. So this is basically, again, every Rails app uses libvips by default, and every user is an untrusted user. Remember that. Don't trust anybody.
So you can exploit the library behavior to read files, read arbitrary files, including environment variables and application secrets. How does that work? Remember, this is all running in a container along with your source code. Maybe that's sensitive. You have config files, maybe your credentials. You've got your environment variables too. So if you're injecting secrets into your container through environment variables, those are all going to be available to libmatio in this case.
And the real trick is, what's the exploit? Because you can trick this system into then the thumbnail image that comes back out is going to contain your secrets. And that's pretty scary. I think that's scary.
The last sentence is the real impact. Once you have secrets, you can do remote code execution on a server, and you can do what's called lateral movement. And that is, you have a database secret, you can go mess with the database. You have the Redis login, you can go mess with the cache, and you can maybe even get into corporate networks. I don't know how you guys all have your secrets set up, but this is really bad. This is about as bad as it gets. This is really bad news. This was a 9.5 on the CVSS score. About as bad as it gets, but I haven't gotten to the best bit yet.
Once we published the CVE based on this obtuse, opaque information, people reverse engineered this hack in under an hour and published their results on the web. I bought you 60 minutes of time to upgrade. Super scary. Not done yet, though.
The other interesting thing about this is that there was actually no big vulnerability here. There were seven small vulnerabilities that got chained together by the agents. I'll run through them real quick so you won't get bored.
First of all, Active Storage direct uploads are enabled by default. They're not anymore. That's been fixed. But it meant that even if you weren't using Active Storage, that route was still available to attackers.
Direct uploads never examine the bytes, right? Direct upload means it goes from the user's browser to the blob store, and the Rails app never sees it. And so it's forced to trust the content type reported. That's the vector in, is by lying about the content type. And that's just a performance thing, right? It would cost you performance if you had to download the file and reanalyze it.
Signed variation keys. This one is a little opaque. It makes the attack easier to pull off, and it's not even a security problem. So we're probably never going to change that in Rails.
This is the actual fix. We decided to call that block_untrusted method. And it's notable that this came out in libvips as a feature that was buried in a changelog, that you had to opt into the secure behavior. Weird.
Libvips. This is the actual flaw. Libvips was fooled by a spoofed MATLAB header. And this was fixed, and no CVE was published. It was so small it wasn't even worth a CVE.
Libmatio pointed the finger back at libvips and said, this is functioning as designed. This is not a bug. This is not a security problem. You shouldn't be handing untrusted files to me. Cool.
And the last thing was, once you get the secret key, there's a couple of known ways that you can get root, like remote command execution based on the secret key. You can escalate that. We know about these on the Rails security team, but they're all breaking changes to fix, and so we've been dragging our feet. But this is the really bad one, which is once you get the secret, you can then leverage that.
So all seven of these were chained together by AIs based on very little information. This should really scare you. They're getting really good at this.
If you're not worried yet, here's a video of the exploit script running in real time, pulling out all of my secrets from a development app. Pretty scary. Once you know what to do, it happens really fast.
It should also scare you that these libvips dependencies, so far this year, there have been 378 vulnerabilities published in libvips or its dependencies. So there's a lot of them, and there's more being reported every day. This is 37signals' HackerOne account data. This is when you have a bounty program. This is what the valid reports are looking like now. This is going parabolic at this point. Super scary.
So there's a whole category of problems here. I can fix this one CVE, but what's going to happen when there's a zero-day vulnerability found in the next image processing library? We're going to have to go through all of this again. And so this is when I started thinking about how to fix it. You're probably all screaming inside, "Fix it, Mike. You're on the security team. Fix it. Fix it."
When I think about fixing it, I think about risk, and how do we decrease risk? And here it's the risk of having a vulnerability in one of those image libraries. And I just covered this territory where we know the probability of finding another vulnerability in these libraries is super high and probably not going down anytime soon. So we're forced to think about the impact of blast radius. How do we minimize the amount of damage that can be done if one of these vulnerabilities is found tomorrow?
And that's what part two of this talk is about. How do we actually reduce the blast radius? Specifically, the question I want to answer is how can we run Active Storage so that it doesn't matter if there's a vulnerable library that's running this untrusted code on an untrusted file?
And the solution that you probably all would reach for, just like me, is some kind of a sandbox. We want to make sure this runs somewhere.
And so I started thinking about, what are the five easy steps? Five easy steps to secure Active Storage. Step one, let's define the requirements. And this will actually become the prompt that I feed into my agent as we build this system together. So the grab bag of requirements I had is I want it to be a drop-in replacement for Rails. You shouldn't have to make any code changes. It should just be a configuration change.
I want to run everything with the least privilege, right? I want it to be a very limited user in a file system that doesn't allow SUID. I want to drop all the capabilities in the Docker container, least privilege. I don't want it running with the Rails privileges, right?
Hard limits on the container. I want to limit CPU time. I want to limit memory. I want to limit file size. I want to limit number of files, number of processes. Let's make it as hard as possible to do damage if there is an exploit.
I don't want to give any access to the app file system or the app environment, so there are no secrets to exfiltrate. I don't even want there to be a network interface, because if something happens in that box, I don't even want them to be able to make a network call back out. I don't want them to be able to move laterally within your network.
I want it to be disposable. We're going to start up a process to handle an image, and I want to kill that process when it's done so that if there is a remote command execution, the lifetime of that is limited. I want the deployment life cycle to be very familiar. I want it to be easy to deploy with Kamal or Kubernetes.
And I want you to be able to build your own things. I built Active Storage on top of this system, but you should be able to build whatever you want. If you want a secure place to unzip files, you should be able to build that.
So this is the requirements list, and I started thinking about real-life analogies to this. And this is a photo from Oak Ridge National Labs, which deals a lot with radiation, radioactive materials. And they have a device that they call a hot cell, and basically what it is, is it's a big lead box. And even the window is like mineral oil so the radiation can't get out, and you have to do everything with these little manipulators, right? And anything that comes in or out of that box is tightly controlled. And this is exactly what I want, but for Active Storage. So this is where the name comes from, and that, by the way, is also item number two on the five easy steps to fixing Active Storage.
And this is the system as we can see so far. There's a container, and there's something going on, but let's get more specific. How do we jam this into Rails? Now, thankfully Rails has a config for this already. This is amazing. So there are three different config parameters, each of which controls one aspect of Active Storage.
There's analyzing, which gets back metadata like how many pages are in this PDF. There is a variant processor, which is transformations. It's for rotating or cropping or resizing. And previewers are for generating a thumbnail for non-image files like PDFs or movies.
And so really all I want to do here is I want your config change to be like that. I want to be able to go back and forth, right? That should be all you need to do here. So we know that now the shape of this is there's an existing class with an existing API. I can do that.
And so Hot Cell, the project, is actually five different gems. There's Hot Cell core, which is the common bits, and then there's client and server on the Hot Cell level. And this defines how does work get done, and how do error codes get returned, and some of the communications. And then there's Active Storage client and server gems, which subclass, and that's what allows us to do a drop-in replacement for Rails.
And so now this is what the system looks like in my head, where we got a Hot Cell client and a Hot Cell server, and I have to communicate somehow. And I've kind of handwaved this so far because I said I don't want a network interface on this. So how do we actually get inputs and outputs working? And the answer is Unix sockets.
Unix sockets are great. It's a little file on disk that processes can read and write to, but it's not routable on the network. But it's a very common way to do interprocess communication. And the really interesting bit is sendmsg, the system call, supports passing open file descriptors. And this is really cool. This is like magic.
So the way this works is on the left-hand side, the image upload goes in, the Hot Cell client class opens the file on disk, and then hands the file descriptor over to the Hot Cell container. So the file system is not accessible inside the container, but that file is because I've given it the handle. And this is what we use for all the inputs and outputs. It's pretty good.
But we still don't know how the work is going to get done. And that is step five: how to actually do the work. Unsurprising. It's a supervisor-worker style thing, just like Active Job or Puma or anything else that you can think of. This is pretty straightforward and probably not worth spending too much time on.
And this is the finished system. And this is the code that's actually running right now in production at 37signals. And I'm here, now that I've scared you, to say to you, you should be running Hot Cell in your apps. And I'm going to tell you how, rolling it out.
Once you've installed the gem, you can
Just run hotcell install. It'll create three files in one empty directory. And that's pretty much it, inside of a hotcell subdirectory.
So the Gemfile is exactly what you think it's going to be. Hotcell config is a file where you can actually configure Hot Cell. It's just Ruby code. It's just Ruby code. It's not a Rails process, but it is a Ruby process. And you can set the limits. Some of the things that are available are listed here. I suggest you tune those once you get going. But this is where you do that initial configuration. Those are just global limits. But then we can actually get more refined along the way as well.
When you deploy your app, you've got to set up the socket directory. So the only thing that's shared is a directory with two sockets in it, two Unix sockets in it. You just have to configure that. And you do the same thing in your accessory configuration. I'll break this one down. It's a lot of code. You set up the volumes to match what you have in your application. You set up your Docker container limits. And you set up any of your capabilities that you want to have too, including network none, which is the proof that none of this actually needs the network to work.
The drop-in replacement for Rails is, you could be using any number of these Rails classes on the left, and all you have to do is change your config to use the corresponding twin on the right that's provided by Hot Cell. So this is what your typical application configuration would probably look like. Just set the three configs, set the two properties you have for the Unix sockets, and you're good to go.
When you configure the Hot Cell itself, you just have to load the classes because, remember, this isn't a Rails app. So Zeitwerk does not work here. So you require the classes you need and you configure them. So here's where you can do per-operation configuration. So maybe the cell itself has a deadline of 10 seconds, but this operation, you want it to be like 2 seconds. You can do that kind of a configuration here.
And that is all it takes to port a kind of a vanilla Action Text, Active Storage application. Like, that's it. You configure Active Storage, you configure the Hot Cell, you deploy it. That's it.
It also comes with great observability. Even though we can't trust the worker in the container, we don't know if that's been hacked, we can trust the supervisor process, and we can obviously trust your Rails app. And so we log events in the Rails app just like it's a database hit. You'll get good metrics around that. We periodically pull metrics from the supervisor, things like queue depth that we can't get from the requests. And then the supervisor itself will do a little JSON log to standard out. So you can actually see if children are killed, why were they killed, that kind of information. And you can light up a Grafana screen. This is what we use internally for our dashboard. And I won't go through it all, but you can set up alerting just however you want to do it in your environment.
The interesting bit is this only cost us 8 milliseconds per call, which is lower than I expected it would be. Still maybe higher than you'd like. We could probably bring that number down. Right now it's related to, I think, garbage collection. But that's pretty cheap to know that you won't be zeroed the next time ImageMagick has a problem.
It's also very flexible. If you want to have multiple queues, you can just stand up multiple cells that are all listening on different sockets and have different queue depth and different timeouts and different deadlines. You can have multiple input and output files if you have a custom operation you want to write. You can bring your own container if you work in a corporate environment. There's a conformance test that will tell you whether your container is configured properly to work well. And again, all the per-operation limits are totally configurable.
If you want to write your own operation, so you have your own thing you want to jam into Hot Cell, this is what it looks like. It's probably familiar to some of you. I drew heavily from Active Job. There's a perform method, and the perform method can take whatever arguments you want. So here we see we're giving it a name, operation Extract Text. We set some default limits that can be overridden, and then the perform method.
And to call that from the Rails app, we simply have a thin shim called Extract Text, and we call perform_in_hotcell. The same way for a job you would say perform_later, here you can say perform_in_hotcell, and it will serialize the arguments and then bring whatever the file is back. And you'll notice that we have to do a file open on the input and the output because we need to hand a file descriptor, a live file descriptor, to the call. But that's really the only hoop you have to jump through, and generally speaking that's going to be hidden by whatever class you write. So that's kind of how custom operations work.
And when I rolled this out in HEY and in Fizzy, which are both relatively modern apps that use vanilla Active Storage, this was absolutely no problem. Basecamp was harder. For those of you in the know, Basecamp predates Active Storage. Active Storage was extracted from Basecamp, and we never went back and cleaned it up. So there were a lot of custom operations I had to write, and there's also really heavy usage of attachments. Like, for some reason our user base loves to send each other memes. Don't ask me.
But we had a couple of outages along the way that taught me a lot. And I won't spend too much time on these. Just know that there were outages, and I learned lessons, and I rolled that back into the library. Tuning is something that you should probably allocate time for.
But really this app is ready to go right now. It's not a 1.0 yet because I'm waiting for Rails to ship. I also want to provide a little more observability features, some syntactic sugar, so to speak, and I may add Linux Landlock. If anyone is interested in talking to me about that, I would love to know more. Landlock is a kernel extension that allows you to ratchet down permissions within a process so that once you've ratcheted it down, there's no way for the process to regain those. So you could say, I don't want this process to be able to fork, exec, and no matter what happens, if an attack takes over that process, it won't be able to regain any of those privileges. That's like a nice built-in suspenders kind of a thing I'd like to add.
Okay, so I scared you. Now I've given you the tools to go and roll out secure Active Storage in your apps.
Now, I just want to reflect for a moment. I started with, "Oh, sorry. It's really easy to do this. Is this QA's fault?" And blame it. It's really easy, and you would be forgiven for doing that, but I want to have a little bit more nuanced view. Like, again, Earth cooled, yada yada yada. We're in this time of chaos right now.
And I think that I'm seeing things trend in a good direction. So I want to be optimistic, but I also want to give you a call to action. There are things that we need to do to get through this chaos and come out the other side. The alternative is, I don't know, we all quit our jobs, turn off our computers, and become sheep farmers or something, right? I want to get through this chaotic period. I want you guys to help. And I want the future to be that optimistic future. I don't want Terminator. I want Star Trek.
So here are the things that I'm seeing. I showed you this chart before to scare you, but now I'm going to reassure you with it. This is the system working as designed. People are finding vulnerabilities and they're reporting them. That's great. That's exactly what we want. We know the vulnerabilities are out there. We want them to be reported responsibly, and that's happening. And this is frustrating to deal with, but it's the system working well. And there's a finite number of vulnerabilities, right? Right. And at some point we're going to have the last bug party. We'll have the last vulnerability party. This is unsustainable. It can't go on like this forever. The number has to come back down at some point.
When I said 378 vulnerabilities found, what I meant to say was 378 vulnerabilities were fixed. They were fixed. All of these libraries are trending in the right direction. Responsible maintainers fixing the problems, rolling out software updates, system working as designed. This is great, building confidence.
What can you all do? Here's the thing: I think vulnerability reports should all come with their own agent skills. And so this is actually what I did in response to the CVE, is I published a repo that was extracted from my own work at 37signals to examine our apps and see if we had been exploited. And so you can just point your agent at this repo and at your application, and you can ask it, "Am I vulnerable?" Probably. And you can ask it, "Was I exploited?" And it will actually go through all the Active Storage database records and tell you if you were exploited and if you need to rotate your secret key. Then you can point it at the Rails guides and say, "Well, then help me rotate my key." So this is the future, and I think that's going to help us all get through this period of chaos.
Attack your own work. When I was building Hot Cell, I did this religiously, which was adversarial review on almost every commit. And Jeremy helped me out with some red teaming where we actually set up agents and told them to break in, and they found some gaps. This is great. So doing this doesn't mean that it's going to be a perfect library, but it means that it's going to have fewer problems with it. So if we do this on our own software, we'll get through this period of chaos faster.
And last of all, improve or replace existing systems. Yay, Hot Cell, baby. The only way through is that software has to get better or it has to be replaced. That's the only way through this period of chaos. And so I hope that you all will take some of this advice to heart. Don't be too scared. Be proactive and constructive. And the future will be Star Trek, not Terminator. Live long and prosper. And thank you for listening to my talk.
Article published · Updated