Following kamal deploy From SSH to Proxied Requests: Josef Strzibny's Kamal Deep Dive

Open on YouTube ↗
Overview

Josef Strzibny calls themself a "Kamalholic." It started with deploying a few apps and writing a blog post. Then came the first book on Kamal, a video course, a magazine article, and a series of conference talks. In this Rails World session, Strzibny asks what actually happens when you type kamal deploy, and follows that command through a set of "dives": SSH and Docker, building and pushing images, registering services with Kamal Proxy, zero-downtime deploys across multiple hosts, and finally into the Go code that proxies requests.

15 min read

Strzibny also mentioned joining SerpApi, which provides live search data as JSON or Markdown, and working on SerpTrail, an SEO checker that "last time I checked" still ran on Rails, not Rust, and ships with a Kamal configuration. The talk was framed by stories from a first trip to America: an In-N-Out order meant to be "a portion for one" that needed help from colleagues to finish, and an impulse drive in a Ferrari California in Hollywood after signing a waiver without reading it.

5:23

How Kamal sees an application

Before going deep, Strzibny laid out Kamal's mental model. Kamal is built around a Docker image of your application. The image runs in roles. The primary one is the web role, for example Puma serving your Rails app. Additional roles can run jobs or other processes from the same image. Accessories are separate services such as a database or a headless browser. They have different life cycles from the app roles, and the talk did not cover them.

There are two separate pieces of software involved. The Kamal CLI runs on your laptop when you type kamal deploy and issues commands to servers over SSH. Kamal Proxy runs on the servers themselves. The rest of the talk follows how the first drives the second.

6:33

Dive one: Thor, SSHKit, and commands as arrays

Strzibny said their favorite Kamal feature is that it prints every command it runs, so you can see exactly what is happening. To understand where those commands come from, they started with the CLI layer. Kamal uses Thor, the same library Rails uses for its command line. Kamal subclasses Thor and implements its commands. kamal deploy is made up of many subcommands that can also be called individually from the command line. Inside the code, they are chained through Thor's invoke method.

Actual remote execution goes through SSHKit, which Rubyists may remember from Capistrano. Kamal adds a small patch to SSHKit and a small DSL on top. SSHKit already has a DSL for running code on hosts and locally. Kamal extends it so it can target its own roles. The flow is: Kamal configuration goes in, and in the end mostly Docker commands are run on the servers. The code that produces those commands lives in Kamal's Commands namespace.

A simple example is a Docker version check to see whether Docker is installed. A more complex one is a full docker run. Strzibny pointed out that the Ruby code for these almost reads like shell, with a mix of symbols, strings, and method calls. That is because the docker helper just builds an array with the :docker symbol at the front. Kamal registers docker as an SSHKit command. According to Strzibny, this means SSHKit skips its usual /usr/bin/env wrapping, so the logs come out cleaner, even though simply shelling out to Docker would also work.

Sensitive values are handled with a dedicated "sensitive" class. When it is converted to a string, for example to go into a real command, you get the unredacted value. When it is inspected, as in log output, you get a redacted placeholder. The registry password is one value treated this way.

Because commands are arrays of tokens, combining them is just array manipulation: compacting, flattening, and joining. A shell pipe is the same idea, with the arrays joined by a | element. There are helpers for wrapping something in a new shell, similar to sh -c. Strzibny's example was building a Docker install command by piping the result of fetching the get-docker script into sh. The whole thing is still just arrays. Kamal keeps commands as arrays as long as possible and only converts them when it has to hand them to SSHKit.

When reading Kamal's source, you will see execute. This comes from SSHKit's Net::SSH backend and is what actually runs the command. The splat (*) is there because it accepts positional arguments, so the command array is passed in that way. Strzibny also explained why Kamal has a global KAMAL constant. Thor and SSHKit both use instance_exec, so self is lost inside those blocks. Kamal still needs to reach global configuration, so it exposes it through a constant.

Concurrency across hosts: the SSHKit patch and connection pooling

Running on many hosts at once brings up Kamal's SSHKit patch. Normally SSHKit runs threads and simply joins them all. Kamal still joins the threads, but it rescues exceptions from each one. Strzibny explained the goal: run the code on every host and collect the errors, rather than stopping at the first failure.

Threads are only part of the picture. TCP connections are expensive to create, so Kamal shares them. Many small commands, from docker login to docker stop, run across concurrent roles, and they can reuse connections from a pool instead of opening new ones each time.

11:55

Dive two: building and pushing images

The image itself is built by BuildKit, part of Docker or, more precisely, the Moby project that is upstream of Docker. Kamal runs docker buildx build. Strzibny called buildx "a bit weird" because it is a plugin to Docker, and that brings in builder drivers. The docker driver is the original, simple docker build. Kamal uses the docker-container driver because it needs more: multiple architectures, exports, multi-node builds, and so on.

A "builder" is a named instance of that driver. When you configure a builder in Kamal's YAML file, that is what you are naming. Strzibny admitted the naming is confusing at first, and walked through the chain. kamal deploy builds the image by running docker buildx build. That uses the buildx plugin, which starts a new container (hence the docker-container driver name). BuildKit inside that container runs your Dockerfile and produces the image. On a remote builder, the process is exactly the same. Kamal only needs Docker to be pre-installed on that host.

13:50

Registries, including a local one

Part of the deploy is uploading the image to a registry: Docker Hub, another provider, or, the case Strzibny found most interesting, a local registry. A local registry is just an extra container running the standard registry image under a Kamal-specific name. There is nothing special about it except for one question: how do remote servers reach a port on your laptop?

Strzibny compared it to local port forwarding, which you might use to expose a server's database port on your machine for inspection. Here it runs in the opposite direction, as remote forwarding, conceptually like ssh -R. Kamal does not shell out to ssh -R. It implements the forwarding in Net::SSH, but the idea is the same.

For the push, Kamal uses the same docker buildx build command, which builds and then pushes. For the pull, once the port forwarding is in place, the servers can see the laptop's registry port and pull the image directly.

15:35

Service registration with Kamal Proxy

Kamal has its YAML configuration file, but Kamal Proxy has no configuration file of its own. Each instruction to the proxy is a docker exec issued by the Kamal CLI. Strzibny described this as consistent with Kamal's imperative style. The core command is kamal-proxy deploy.

Since everything is passed in a single docker exec, every setting has to be passed as a flag: TLS, logging, buffering, and anything else. The shape of the command is docker exec kamal-proxy (run something in the proxy container), then kamal-proxy (the binary), then deploy with a service name such as web, then the flags. The most important flag is the target, the container ID and port to send traffic to.

Because configuration arrives imperatively, there is the question of what happens if the proxy goes down. Kamal Proxy keeps its state in a named configuration volume and reloads it on restart. Besides deploy, the proxy supports other service actions: removing a service, stopping it (maintenance mode), and resuming it.

Zero-downtime deploys on a single host

Strzibny called zero-downtime deployment essential to how Kamal works. The mechanism goes back to the same kamal-proxy deploy command. For a new version, Kamal runs it again with the same service name and a new target, meaning a new container ID and port. The new container is started first. Once it is running, the proxy can switch over.

Kamal uses blue-green deployment. In the simple single-host case, the new web container (Puma) boots, it is handed to Kamal Proxy, the proxy runs a health check, and once that passes, traffic switches to the new "green" container. The old container's in-flight requests are drained.

Assets need extra care because two versions are live at once. Kamal unpacks assets into the same directory and copies them into volumes for both the old and new containers. It uses a no-clobber copy so files that were copied first are not overwritten. As long as the order is right, both containers can serve both sets of assets. Strzibny noted that assets must be fingerprinted for this to work.

The deploy lock is "even easier." To stop concurrent deploys from interfering with each other, Kamal creates a directory on the server, and that directory is the lock.

19:26

Coordinating many hosts: the barrier

With multiple hosts, the same flow needs coordination. For this, Strzibny went "even deeper to the Great Barrier Reef" to look at Kamal's barrier, also called the web barrier, named for the role it protects. It lives in the health check namespace and is built on a concurrent "IVar," a variable that can be set only once. The barrier is either open or closed. When setting it, errors are ignored, because only the first set to true counts.

The process works like this. Kamal starts booting containers for the web role. The first one to boot correctly opens the barrier, and then everything else is allowed to boot. Only one healthy web container is needed before the rest proceed, and each switch happens as in the single-host case. In code, a non-web role waits at the barrier before starting its new version. A web role acts as the gatekeeper and releases the barrier.

Starting a new version does what you would expect. It uploads the environment with the new secrets, runs the new container, and, if a proxy is in use, has the proxy perform the switch with the proxy deploy step.

For rolling deploys, Kamal slices the servers into batches, which corresponds to the limit setting in the config. In code, the limit can be a number or a percentage.

21:29

Dive three: proxying requests in Go

The last dive was into Kamal Proxy, which Strzibny acknowledged is "Go territory" and not the usual subject at a Rails conference, before offering to "teach you Go just now." The two features Go was known for at launch were goroutines and channels.

Goroutines are concurrent functions scheduled by the Go runtime across available threads, on one CPU or many. They are started with the go keyword. Strzibny joked that this is maybe why the language has that name. Go's net/http package, simplified, does something like go serve for each new connection. In Kamal Proxy's own code, however, you will not find the go keyword in the request path at all. It relies on net/http, runs its middleware, and then calls a handler. That handler finds the right service for the incoming request and serves it.

Channels are how goroutines coordinate. Strzibny described them as roughly a first-in, first-out queue. Goroutines can write to them or read from them. You do not know which goroutine will pick up a given value, but when a channel is closed, every goroutine waiting on it finds out.

Health checks, buffering, draining, and maintenance mode

Health checks are the first channel-based example. A ticker sends a value on its channel (ticker.C) at the configured interval, and each tick triggers an actual check. The code creates a new ticker with the interval from the configuration, then loops forever around a select statement. Strzibny noted that select is everywhere in Go code. Each case waits for a new value on some channel. Here, a value on ticker.C means "run the check now."

Kamal Proxy also buffers requests, as a protection against slow clients. Part of a large request is held in memory and part is written to disk. Either way, the request is buffered completely and then sent on to Puma.

Draining uses the same select pattern. For each in-flight request, either the request context finishes, in which case it is allowed to complete, or a deadline is reached, in which case it is cancelled. Strzibny walked through the drain function. Hijacked in-flight requests, meaning WebSockets such as Action Cable connections, are killed immediately without further handling. For everything else, a labeled block waits in parallel for each request to either complete or hit the deadline. If the deadline is hit, execution breaks out of the labeled block, and any remaining requests are cancelled with a draining error.

Maintenance mode was the last example, and one Strzibny found especially interesting. It uses a "pause" channel. While paused, request goroutines wait on that channel. When the pause ends, the channel is closed, and every waiting goroutine sees it and continues. No value is ever sent. Closing the channel is the signal. Inside the request path, there is again a select waiting on the relevant channels.

What the dives add up to

Strzibny summarized the journey. Kamal is built on Thor and SSHKit. It keeps commands as arrays until the last moment, when they must be executed. It registers services with Kamal Proxy through imperative docker exec commands, which enables blue-green deploys, coordinated across many servers by the barrier. Kamal Proxy's request handling, health checks, draining, and maintenance mode come down to goroutines, channels, and select.

Strzibny closed by admitting they cannot stop talking about Kamal. They recalled explaining to someone at a California counter that Kamal is a simple deploy tool that can replace a Kubernetes cluster and save a lot of money, only to be told "Sir, this is a Wendy's." They ended with a promise to stick to Texas barbecue and a proper American muscle car instead of a Ferrari, and a final "long live Kamal."