Your Cluster Is Finally a Cluster
Introducing distributed runners in Miren: add machines to your cluster and spread stateless workloads across them.
Miren has always called the thing you install a cluster. It’s in the docs, it’s what miren cluster list prints, and for all of Miren’s life thus far it has described a single machine running everything you own, whether that’s a rented VPS, a box in your basement, or the laptop on your desk. It was a slightly uncomfortable word to keep typing, always with Inigo Montoya somewhere in the back of my head:
You keep using that word. I do not think it means what you think it means.
We kept using it because we were designing for it. Miren has always had a scheduler picking which node each sandbox runs on, it’s just been a scheduler with a very easy job so far. Our choice of a reactive data tier, which we wrote about at length back in June, was also made with clustering in mind. A single-machine install was the N=1 case of that design rather than the design itself, and now N finally gets to be bigger.
Adding a machine
Distributed runners, new in v0.13, let you point more machines at a cluster and have your stateless workloads spread across them: web services, workers, anything that doesn’t mount a disk. The machine you already have becomes the coordinator and keeps doing the things that have to happen in one place, which is holding cluster state, building and serving images, signing identities, and scheduling. It also keeps running workloads, so a two-node cluster is a coordinator plus a runner rather than a manager watching a worker.
To add one, mint a token from your workstation and hand it to the new machine:
# on your workstation
miren runner token create
# on the machine you're adding
miren runner install --token mren_...That enrolls the machine and installs it as a systemd service, so it comes back on its own after a reboot. miren runner list shows it arriving, and once it reports ready the scheduler starts placing sandboxes there. The best part: nothing about your apps changes, they just get more room to spread out.
How the overlay works
Start from a request. It arrives at the coordinator, which terminates TLS and knows which app it belongs to, and gets proxied to one of that app’s sandboxes on an internal network. The sandbox has an address there, the coordinator opens a connection to it, the response comes back. That has been the shape of it since long before there were two machines to choose between.
All distributed runners change is that the sandbox on the far end of that proxy might not be on the same machine anymore. Which means two things have to be true: a sandbox’s address has to mean the same thing everywhere in the cluster, and a packet aimed at one of those addresses has to be able to find the machine holding it.
Our old friend switched packet networking has answers for both. The cluster gets one flat range, 10.8.0.0/16, and each machine leases a distinct /24 out of it, so an address encodes which machine its sandbox lives on and no two sandboxes can collide. Each machine then registers itself as the next hop for its own slice, and a packet for 10.8.2.9 leaves the coordinator, meets a route saying that /24 lives over there, and goes. This is classic IP, just with some modern bookkeeping and crypto layered on top.
The bookkeeping is flannel’s subnet manager, keeping leases in the same etcd the rest of the control plane already uses. We wrap it in a library we naturally named grunge. The crypto is WireGuard: every machine publishes a public key and an endpoint alongside its lease, and since they all watch those leases, one machine joining is enough for the others to learn its subnet, its key, and where to reach it, and to install the route.
Two things in that picture are deliberate. Every pair of machines gets its own tunnel rather than routing through the coordinator, so two sandboxes on different runners talk to each other directly. And those tunnels are encrypted, which means a runner can safely join over an untrusted network, without needing a private link back to the coordinator.
What’s not here yet
Anything with a disk still pins to the coordinator. Disks and local storage are node-local, so a sandbox that mounts one can’t move between machines, which means distributed runners give your stateless workloads room to spread out and do nothing for your database.
The coordinator is also still a single point of failure. Every request enters the cluster through it before being proxied onward, so a sandbox happily running on a runner is unreachable while the coordinator is down, and deploys and scheduling wait on it too. Spreading your workloads across three machines buys you capacity, not availability. That’s a real limit and worth knowing before you plan around it.
We’ll keep chipping away at both. Letting stateful workloads move between machines is one piece, and making the coordinator something other than a singleton is another. In distributed systems, N=1 to N=2 is the expensive jump. We’ve made it for stateless workloads, and we’ll keep closing gaps until we can use the word cluster in every sense of it.
Try it
Upgrade to v0.13, then use the two commands above to point another machine at your cluster.
The full details, including the ports to open between machines and how to enroll runners from a provisioning script, live in the distributed runners docs.
Haven’t tried Miren yet, but you’ve read all the way down to this last line of the post? Sounds like a great time to get started.