Inside Solid Cable: How a Polling, Database-Backed Action Cable Adapter Catches Up to Redis

Hey there, I'm Nick Pezza. I wrote and maintain Solid Cable, and today we're going to look at how it works and how it compares to the other Action Cable adapters in Rails.

Prior to Rails 8, Action Cable relied on one of three adapters: Redis, Postgres, and Async. They all solved the same basic problem. When your application broadcasts a message, that adapter is responsible for getting that message to the Rails processes that need to know about it. From there, those processes can send the message down to the connected browsers over their WebSocket connections. The important difference between these adapters is really about where those messages are able to travel.

The Async adapter is the simplest and has no dependencies, but it only works within a single Rails process, which works great in development. But production Rails apps usually run across multiple processes, often spread across multiple machines. Think of it like talking inside of a single room. Everyone in that room can hear you, but that message can't leave the room.

Redis and Postgres solve this by giving your Rails process a shared place to communicate through. One Rails process publishes a message, Redis or Postgres distributes it, and every process that needs that message can receive it. But they come with their own trade-offs.

If you choose Redis, deploying your Rails app now meant learning how to manage yet another data store alongside your primary database.

If your app already used Postgres, the Postgres adapter mostly just worked. The trade-off is that its pub/sub mechanism limits payloads to roughly 8 kilobytes. That's usually fine, but if you're using Turbo to broadcast larger chunks of HTML, you could run into that limit. And if your app uses SQLite or MySQL, standing up Postgres just for Action Cable didn't make much sense. So Redis tends to become the default choice.

At work, we were using Redis for both Action Cable and caching. I unexpectedly found myself responsible for managing all of our infrastructure. So I started looking for ways to simplify things and make my life a lot easier.

I moved our caching over to Solid Cache, which just left Action Cable as the last thing keeping Redis around. It was right around this time that I saw this issue that David had filed in Rails about extracting an Active Record-based adapter out of Campfire. So I figured I just had to wait for Campfire to land, drop the new adapter in, and finally remove Redis from our stack. But then a little while later, I saw this comment come through where Campfire had ended up using Redis for Action Cable after all. So I figured, how hard can it be to implement an adapter that only uses Active Record?

Redis and Postgres both keep a persistent connection open to each Rails process. So when a new message arrives, they can send that to the connected Rails process immediately.

To build an adapter on top of Active Record that's database agnostic, we can't depend on Redis's pub/sub or Postgres's LISTEN/NOTIFY mechanisms. So just like Solid Queue, we can lean on a little bit of low tech: polling. We can set up an infinite loop, ask the database if it has any new messages, and then sleep for a few milliseconds before checking again.

Before looking at the code for the adapter, at the core of Solid Cable is our messages model, which has a channel and a payload along with an index on channel, since that's what we'll be querying by as it polls for new messages.

The adapter is really made up of two classes. One inherits from Action Cable's base adapter class, which sets up some shared state and defines the methods every adapter needs to implement. The other is the listener, which in Solid Cable's case handles polling. It inherits from Action Cable's SubscriberMap, which keeps track of who's subscribed to each channel and handles delivering broadcasts out to them.

Our adapter class implements four main methods: broadcast, subscribe, unsubscribe, and listener. Let's start with broadcast. It just inserts a row into the messages table. Using insert lets us skip Active Record callbacks, validations, and the usual transaction wrapping, none of which we need here. So we can make the write as lightweight as possible.

Next up is subscribe. It calls listener and then calls add_subscriber on listener with the channel and the subscriber. So let's look at what that listener is actually doing. The listener method is responsible for instantiating the second half of the adapter, the listener object. But the listener itself doesn't define the add_subscriber method. That comes from its SubscriberMap superclass.

Inside SubscriberMap, initialization sets up a hash that tracks subscribers by channel. When we add a subscriber, we append it to the list for that channel. And all that happens inside of a mutex to avoid those pesky race conditions. If this is the first subscriber for the channel, we call the add_channel method, which is implemented by our listener class.

add_channel just adds the channel name to a list of watched channels that we'll use to gather messages to be sent out during polling. This is where Redis would call its subscribe method or Postgres would call its LISTEN command inside of their adapters.

That's subscribe. Let's look at unsubscribe. As you'd expect, unsubscribe does the inverse of subscribe. It calls remove_subscriber on the listener object. And just like add_subscriber, that method actually lives on SubscriberMap.

We remove that subscriber from the channel's list, and if there are no subscribers left, we call remove_channel, which is implemented by our listener class. It just removes the channel name from our list of watched channels. This is where Redis would call unsubscribe and Postgres would call UNLISTEN inside their adapters.

And with that, we've covered the main methods the adapter class defines. Pretty simple stuff. Now let's go look at the listener class initialization.

At the end of initialization, the listener starts a new thread and calls listen inside it. Let's go look at what that listen method is actually doing.

Listen starts a loop that keeps running until the listener gets shut down. On each pass through that loop, we query for any messages for the channel with an ID greater than the last one we've seen and then broadcast each of those messages out to their subscribers.

After we broadcast each message, we update last ID to that message's ID so the next poll knows where to continue from. And when the listener first starts up, we lazily initialize last ID to the most recent message already in the table so we don't replay old broadcasts. And finally, we sleep for a configured amount of time before starting the loop again. By default, it's 100 milliseconds.

And that's it. In version one, the adapter and the listener together were only about 112 lines of code, plus the message model and a little configuration parsing. That's the version I ended up deploying at work, and it let me remove Redis from our infrastructure without a hitch.

And then a few months after the release of version one, to my surprise, I got this email from DHH about adding Solid Cable to Rails as the default adapter. But one of the blockers to doing so was adding autotrimming. Once a message has already been broadcast, we don't need to keep it around forever. Autotrimming periodically deletes older messages so the table stays small and the polling queries stay fast.

Now when we broadcast out a message, we check if the config has autotrimming enabled, and if so, we run the trim job. Rather than trimming on a fixed schedule, Solid Cable ties cleanup to write activity. With the default settings, each write has a small chance of triggering a trim of up to the batch size of old messages. The probability is tuned so that over time, trimming has roughly twice the capacity of incoming writes. That gives the table consistent downward pressure as traffic increases, while a configurable retention period determines which messages are old enough to delete.

The other change the folks at 37signals suggested was adding a hashed version of the channel name to each message. We use SHA-256 to turn the name into a fixed-size integer, store that alongside the payload, and index that column. That gives us a much smaller index and cheaper lookups. So when the listener checks for new messages, it's querying by the channel hash rather than the channel name.

One of the more notable bugs that was found after release was around how we determine which messages should be broadcast. Originally, Solid Cable tracked a single last ID for the whole listener. The problem is that if you subscribe to a new channel while a listener is already running, there may be older messages for that channel still sitting in the table that this new subscriber should never receive.

So we started tracking a separate last-seen ID for each subscribed channel. When a channel is added, we record the newest ID at that point, which gives that subscription its own starting line. Because the per-channel last-seen IDs can be added, updated, and removed from one thread while the listener is polling on another, we move that channel state into a concurrent map and use atomic updates to avoid races.

We still keep the global last ID to efficiently scan forward through the table, and we eagerly initialize it now when the listener starts, so we don't miss any messages between initialization and the first poll. Then before broadcasting a message, we also check the last-seen ID for that specific channel. So the global ID keeps polling efficient and makes startup safe, while the per-channel ID makes sure subscribers only receive messages that were broadcast after that channel was added.

All right, let's look at some numbers and see how these adapters compare. All these benchmarks are run using Solid Cable version 4.0 and Rails main from a July 31st commit on a box with two CPUs and 8 gigs of RAM. Redis, MySQL, and Postgres were all running on that same machine to avoid any network hops, and Solid Cable was configured with a 100-millisecond polling interval. I also disabled autotrimming to isolate the adapter performance.

I use Grafana k6 to generate the load and capture the metrics, and all the benchmarking app and load testing scripts are available in the bench directory of the Solid Cable repo.

The first benchmark up is constant load. It keeps a fixed number of users active at all times. Each user connects, sends a ping, waits for the pong message, and repeats that 20 times. Once they finish, they disconnect, and a new user immediately takes their place. That cycle runs continuously for 5 minutes. So we're simulating a steady stream of users connecting, exchanging messages, and leaving.

And these are the throughputs for each adapter at different user counts. We're measuring how many messages per second make the full trip through the system and back. At 250 users, all the adapters are pretty close, around 280 messages per second.

Once we really start saturating the system, Redis and the native Postgres adapter keep scaling. Solid Cable on SQLite holds up pretty well, while Postgres and especially MySQL start to level off around the 500-user mark.

The other thing we care about in this benchmark is latency: how long it takes a ping to go out and the corresponding pong to come back. As latency increases, clients spend more time waiting, which eventually limits how much throughput the adapter can sustain.

Up through about 500 users, things are still pretty healthy. Around 750 and a thousand users, though, latency starts climbing across the board. Solid Cable degrades more aggressively under that load, especially on MySQL, while Redis and the native Postgres adapter hold their latency down for longer.

Next benchmark up is thundering herd. It connects a fixed number of users simultaneously, and each user subscribes, sends one ping, validates the pong, and then disconnects. It's testing how an adapter handles a sudden burst in connections, for example, after a deployment forces every client to reconnect at once.

The main thing we're looking at is how quickly each adapter works through the resulting backlog, which we measure through message round-trip latency and total completion time. At 250 users, the total completion times are all very close, right around 2 seconds. At 500 users, we start to see a little separation, but the adapter still clears the storm in roughly 3 seconds.

By a thousand users, though, the differences are a little more obvious. Redis finishes in 5.2 seconds and the native Postgres adapter in about five. Solid Cable on Postgres is still close at 5.7 seconds, while SQLite is even closer at 5.2, with MySQL bringing up the rear in around six. So even under a pretty aggressive reconnect storm, Solid Cable is still in the same general range as the push-based adapters.

And last up is the lecture benchmark. It subscribes a number of users to the same channel, then publishes seven messages per second for five minutes. It's designed to show how efficiently a single broadcast can fan out to a large number of subscribers at the same time.

These are the latency numbers. At 250 subscribers, Redis and the native Postgres adapter stay around 70 milliseconds on average, while Solid Cable is closer to 120. Since our polling interval is set to 100 milliseconds, that difference is mostly the cost of polling showing up. At 500 subscribers, everything is still holding together pretty well. Redis and the native Postgres adapter are around a 90-millisecond average, and Solid Cable is around 145.

At 750 subscribers, though, you start to see the system come under some real pressure. Redis and the native Postgres adapter are still under 200-millisecond P95, while Solid Cable climbs to the 400 to 500 range. I'll also note, at 10 messages per second instead of the seven we use at 750 subscribers, every adapter failed the benchmark. We'll come back to why that happened in a bit.

So in aggregate, Solid Cable stayed pretty competitive at the lower end of the load test, even with a 100-millisecond polling interval. But as we pushed the system harder and added more users, its weaknesses started to show. And after staring at those higher-end load numbers for a while and trying to find some positive way to put a spin on them, the immortal words of the great Dr. Ian Malcolm kept coming to mind: we can do better.

So I instrumented a bunch of different parts of Solid Cable to see where the time was actually going. And pretty quickly, a few obvious hotspots showed up that we could speed up. First up was configuration handling.

Every broadcast checks whether autotrimming is enabled, and every pass through the polling loop checks the polling interval and whether polling logs should be silenced. The problem was that each of those checks went back through reading the cable config, reparsing YAML and all the values, and allocating objects over and over again. Not super great under load.

That turned into a surprising amount of redundant work and GC pressure. So instead, we move all that parsing into a memoized configuration object. We read the cable config once, memoize all the values, and reuse that object everywhere.

Next up was how we tracked watched channels. When a channel's added, we add it to our watch list. We store the raw channel name as the key in our map. Then on every polling pass, we take all those channel names, hash them again, and use those hashes in the polling query. Now we hash the channel once when it's added and store that hash as the key in our channel map. Then when our listener polls, those hashes are ready to go straight into the query. No rehashing every channel on every pass through the loop.

Next up is batching writes. Right now, every broadcast results in its own insert into the messages table. That's simple, but under heavy load, it means a lot of individual writes hitting the database. We're already batching reads by polling for multiple messages at once. So we can apply the same idea on the write side: collect broadcasts together and insert them in batches to reduce write pressure.

The new broadcaster groups messages into batches based on either a maximum batch size or maximum delay, whichever happens first. When it starts up, it creates a queue for incoming broadcasts and spins

Up a dedicated thread to watch that queue. Each call to broadcast just wraps the channel and the payload in a small message object and pushes it onto that queue.

The thread that watches the queue starts by waiting for the first message to show up in the queue. Pop blocks the thread here until one arrives, so we're not spinning while there's nothing to do.

Once that first message comes in, we drain anything else that may already be waiting in the queue into our batch in case we've gotten behind.

Then we keep collecting messages until either the batch reaches its target size or the delay expires, whichever happens first. And then we stop collecting and flush the batch. Instead of doing one insert per broadcast, we use insert_all to write the entire batch in a single SQL statement. That cuts down on the number of writes in the database and reduces a lot of the overhead we were seeing under heavier load.

Lastly, like I said, while running the lecture benchmark at 750 subscribers and 10 messages per second instead of seven, Redis, Postgres and Solid Cable all crapped out. Since all the adapters failed around the same load, that led me to believe this was an issue inside Action Cable. And sure enough, it was.

When we call stream_from, Action Cable builds a handler for that stream. If you don't provide a custom callback, it uses the default handler whose job is basically just to take the broadcast and send it down the WebSocket.

Before this change, even that simple default handler was wrapped with a worker pool stream handler. So every incoming message took an extra trip through the connection worker thread pool before being sent to the client.

That extra pool is useful for custom callbacks because they can run arbitrary application code, and you don't want that work blocking the executor. But for the default handler, that extra hop just meant bouncing every message through another thread pool for no real benefit.

After this change, we only use the worker pool when there's actually a user-supplied callback. For the normal case, the default stream handler runs directly, removing that extra queue hop from every broadcast. This change will be included in the upcoming Rails release.

So let's rerun the benchmark and see where we landed after all these changes. I've removed the Postgres adapter from the upcoming charts to give us a little more room in the charts since its results were generally very close to Redis. I've also skipped the Thundering Herd benchmark here since it didn't really change after these updates, mainly because it had so little sustained traffic. For Solid Cable, I configured the batch broadcaster with a batch size of four and a 1 millisecond delay.

So the first benchmark up is constant load. These are throughput numbers. At 250 users, things are basically unchanged. The traffic just isn't high enough for these changes to make much difference.

At 500 and 750 users, though, we start seeing the gains. Postgres and MySQL close a lot of the gap with SQLite, and all three move much closer to Redis. At a thousand users, the improvement is much bigger. SQLite actually edges past Redis at ,48 messages per second. Postgres jumps from 69 to 1,000, and MySQL nearly doubles from 491 to 961.

It's an even better story with latency. Before these changes, Solid Cable's latency started falling apart as we pushed into the higher user counts. Now the curve is dramatically flatter from 500 to 1,000 users. MySQL's P95 drops from 412 milliseconds to 174 at 500 users and from 953 to 228 milliseconds at 750. Postgres shows the same pattern, dropping from 668 milliseconds to 231 at 750 users.

At 1,000 users, SQLite drops from a 398 millisecond average to 140. Postgres drops from 477 to 179, and MySQL goes from taking over a second to just 210 milliseconds.

And the tail latency is the real winner here. SQLite now has a dramatically better P95 than Redis at 750 and 1,000 users, which results in more throughput, while Postgres and MySQL have gone from being way behind to basically matching Redis at 1,000 users.

And finally, the lecture benchmark. Here we see solid improvements across the board as the subscriber count climbs. Latency stays much flatter than before instead of starting to run away at the higher end. And more importantly, all the adapters now make it through the 750 user, 10 messages per second test, which failed before. That works out to roughly a 35% uplift in messages being handled.

And these improvements mostly come from the stream handler changes inside Action Cable. So the benefit actually applies across adapters.

Solid Cable gives us a handful of knobs and dials to tune how it behaves. So let's see what those actually do to performance. For the next few benchmarks, I'm going to stick with just the constant load test since it gives us the most balanced view of throughput and latency.

Solid Cable's default polling interval is 100 milliseconds. So what happens if we drop that to 25? We'll be polling the database four times as often, but in exchange, we should be able to pick up new messages sooner. Let's see what that does to throughput and latency.

And sure enough, it gives us a pretty big latency improvement through 500 users and even into 750. At 250 users, Solid Cable drops from roughly 160 millisecond P95 to around 90. And at 500 users, we're down to around 100 milliseconds. Not quite down to Redis, but much closer.

Once we start saturating the system at 750,000 users, those extra reads start to show diminishing returns for Postgres and MySQL. SQLite, on the other hand, keeps stretching its legs, pulling further ahead.

Just to see how far we could push it, I also ran SQLite at a 10 millisecond polling interval with 250 users. That came in at 65 millisecond average and 78 millisecond P95. Closer still, but Redis still keeps the lead.

Throughput tells a similar story. Dropping the polling interval gives us a nice bump across most of the Solid Cable adapters. SQLite benefits the most, climbing to ,71 messages per second at 1,000 users. Postgres and MySQL show modest gains through 750 users, with MySQL giving a little bit back at the very top end.

So if your traffic is relatively low and delivery latency matters, it may be worth lowering your polling interval. But as your system gets more saturated, polling too aggressively can start to work against you depending on the database you're using. There isn't one perfect value, so tune it for what your infrastructure can handle and what your latency requirements are.

Now let's look at the cost of autotrimming. I'll be using a trim batch size of 100 messages and a retention period of 30 seconds. So for most of the five-minute benchmark, we'll be actively trimming old messages in the background. I'm also back to using the default 100 millisecond polling interval. So we're only changing one variable at a time.

At 250 and 500 users, autotrimming is basically free. So I've dropped those from the chart and zoomed in at the higher end. Even here, the throughput impact is tiny. At 1,000 users, SQLite is essentially unchanged. Postgres drops from 1,000 to 956 messages per second, and MySQL from 961 to 942.

Latency takes a somewhat bigger hit on Postgres and MySQL, but overall the cost of continuously cleaning up the table is pretty modest. So unless you're already operating right at the edge of capacity, autotrimming is probably a pretty reasonable trade-off, which is why it's enabled by default.

So that's Solid Cable. It offers a simpler infrastructure alternative while still delivering performance that can now stay surprisingly close to Redis even under fairly heavy load, and in some cases, like SQLite, exceed it. The biggest trade-off shows up at the low end when the system isn't under much load. Redis and the native Postgres adapter usually have a lower delivery latency because they can push messages immediately instead of waiting for the next poll.

If that low latency really matters, Solid Cable gives you room to tune the polling interval and close some of that gap. But for a lot of Rails apps, though, the database you already have is probably more than capable of handling Action Cable too.

All the improvements I've shown today are included in the 4.1 release of Solid Cable, which is out now. I did bring one more gift, though.

Action Cable currently doesn't guarantee message delivery order because messages are handed off to a thread pool. Two broadcasts can be received in one order but actually delivered in another depending on how those threads get scheduled. Most of the time that's not a big deal. If two notifications arrive slightly out of order, nobody notices.

But for something like streaming an LLM response, order matters a lot. If chunks of the response are delivered out of sequence, the text can be assembled incorrectly.

Here I have an example of that. I'm firing off a two-way stream that just appends incrementing numbers as fast as possible, and they appear out of order. Absolute chaos.

This is where Solid Cable has an advantage because every message is stored in the database. We can persist ordering information right alongside it. If we can give each message a sequence within its channel, then we have enough information to make sure that those messages are consumed in the same order that they were broadcast.

So I've been working on a new gem called ChronoCable, which is basically a collection of monkey patches for Action Cable and Solid Cable that allows a channel to keep its messages in order and can recover dropped messages. This is very much a work in progress, so lots may still change.

ChronoCable adds a new channel model that stores the channel hash, tracks the current sequence number for that channel, and records whether messages on that channel should be delivered in order.

You can opt into ordered delivery with a new deliver_in_order option on stream_from. When the stream is set up, ChronoCable creates the corresponding channel record and marks it for ordered delivery.

If you're using Turbo, you can pass deliver_in_order through the data attributes on turbo_stream_from to opt that stream into ordered delivery.

When we broadcast, ChronoCable first checks whether that channel uses ordered delivery. If it does, we lock the channel record, increment its sequence, and attach that channel-specific ID to the message before it gets queued for storage. ChronoCable also adds a new channel ID column on the messages table that stores this sequence ID.

The downside of this is it cuts into some of the benefit we were getting from batching because ordered channels now need extra database work on every broadcast. Still experimenting with ways to reduce that cost. But for now, let's look at how the front end works.

This is where a lot of the new work happens because the browser now has to keep a little bit of state for each ordered stream. The subscription starts like normal. The browser sends a subscribe message to the server with the channel it wants to listen to. When the subscription is confirmed, the server now sends back to us the current sequence ID for each stream in that subscription.

That tells the browser the last message on chat was one, so the next message it should expect is two.

When a message is broadcast, we now include the stream name along with that message sequence ID. We check that the message ID is the next one we're expecting. If it is, we consume the message and advance the expected ID to the next one in the sequence. But if a message arrives ahead of the one we're expecting, we don't consume it yet. Instead, we set it aside and ask the server for the stream's history starting from the last message ID that we successfully consumed.

While we're waiting for the missing message, another newer message might arrive. Since it is ahead as well, we set it aside with the others. When the history comes back, we consume the missing messages in order, three and then four, and then advance the expected ID as we go.

Then we work through the messages we set aside. We already recovered message four from the history response, so we discard it. Message five is new, so we consume it and advance the expected ID again. With the waiting queue empty, we're caught back up. So we can go right back to listening for new messages like normal.

If the website gets disconnected, say your internet drops, when the browser reconnects and resubscribes, the subscription confirmation will tell us that the stream is currently at message seven, but the browser was still expecting message six. So now we know we've missed something while the connection was down.

So we can ask the server for the stream history again, starting with the last message we saw, and we can use that to catch back up. We can see message six and then seven in order, advance what we're expecting next, and then go right back to normal streaming.

And that's ChronoCable. If you've got channels where message order really matters, give it a whirl and let me know.

And that's it. Here are the repo links if you want to check it out and give any feedback or issues. And thanks.

[music]

[music]