RPC That Calls You Back

Almost every RPC system agrees on the shape of a conversation: a client calls, a server answers. You point a stub at an address, invoke a method, and a result comes back. gRPC, Thrift, JSON-RPC, plain HTTP over a socket: they differ in wire format and ceremony, but they all agree that requests flow one way and replies flow back along the same wire.

For most of what a deployment platform does, that shape is exactly right. The CLI asks the control plane to list apps; the control plane answers. But some of the most important things Miren does don’t fit in a single request and a single reply. You run miren deploy and a lot has to happen before it’s live: building an image, rolling out containers, waiting on health checks to pass. We work hard to make that fast, but fast isn’t instant, and a deploy is a sequence of real steps you’ll want to watch unfold. The interesting information is everything that happens while it runs, and the server is the one that has it, not the client.

The reflex fix is to bolt a second channel onto the side. The client opens a stream and the server writes progress into it. Or the client polls: “done yet? done yet?” Or, worse, the client stands up its own little server and hands the platform a webhook URL to call later. Each of these works. Each one is also a sign that the request/response model ran out somewhere and you’re now working around it. We did all three at various points, and none of them ever felt like the answer.

So when it came time to make the RPC layer something we’d actually build the whole platform on, we took the problem seriously. What we wanted wasn’t a better streaming API. We wanted the thing you called to be able to call back, as a first-class primitive.

Pass an object, not just data

In our schema, a method’s parameters and results aren’t limited to plain data like structs, ints, strings, and lists. A parameter can be an interface. And when you pass an interface, it doesn’t get serialized and copied to the other side. It becomes a capability: a live, typed reference to the object you passed, which the other side can now call methods on.

This makes the client/server relationship symmetric. Either side can hand the other a callable object, the same mechanism running both ways over the connection that’s already open.

Consider a “meter” service that reports temperature readings. The ordinary direction is easy: ask for a reading, get one back. But say the client wants to be notified whenever a new reading lands, rather than polling for it. In our IDL that’s one method:

interfaces:
  - name: UpdateReceiver
    methods:
      - name: update
        parameters:
          - name: reading
            type: Reading

  - name: MeterUpdates
    methods:
      - name: registerUpdates
        parameters:
          - name: recv
            type: UpdateReceiver

Read registerUpdates closely. Its one parameter, recv, is an UpdateReceiver, an interface. The client isn’t sending the server some data describing where to send updates. It’s handing the server an object, one the client itself implements, and the server can now call recv.update(...) on it every time a reading arrives. No callback URL, no second port: the client dialed out, as clients do, and in the same breath gave the server a way to call back in.

We didn’t invent this from nothing. Reference-passing RPC, where an argument can be a live object rather than a copy, is the object-capability model, and the system that made it famous is Cap’n Proto. We borrowed the core idea the same way our entity store borrowed from Datomic: take the one good primitive and build our own thing around it, fitted to our own constraints. The primitive here is that an interface is a first-class value on the wire.

Why not gRPC

We’d used gRPC plenty, and it’s very good at what it’s for, so it’s worth being precise about what pushed us off it. It wasn’t the usual grumbling about protobuf or code generation. We generate plenty of code ourselves, as you’ll see in a bit.

gRPC does have bidirectional streaming. But what it gives you is a stream of messages: two channels, one in each direction, that you push typed protobuf messages onto. That’s better than raw bytes, but it still isn’t a typed, callable object that you passed as an argument. If I want the server to be able to invoke three different methods on something the client owns, gRPC hands me one message stream and I get to invent a little dispatch protocol inside it: a tagged union of message types, a switch statement on the receiving end, a convention for matching a response to a request. I’m building a tiny RPC system inside my RPC system, by hand, untyped, once per feature.

And there’s a nastier flaw waiting in that stream, the kind you don’t see until it’s in production: gRPC streams aren’t acknowledged. When you send a message, the send completes as soon as it’s buffered into the stream, not when the peer has received it, and definitely not when the peer has acted on it. For a one-way firehose of log lines, fine. But the moment you’re using a stream as a protocol, where sending a message means asking the other side to do something, that missing ack is a real hole. A stream can drop with messages in flight and the two ends won’t agree on what got through, so you bolt on your own acknowledgments, sequence numbers, and retries, which is, again, more of the RPC system you were trying not to build.

The capability model deletes both problems. recv is an UpdateReceiver; the server calls recv.update(reading); the types are checked at generation time on both ends; the dispatch is generated, not hand-rolled. And because a capability call is a real call, completing it is the acknowledgment: the call returns when the other side has handled it, with a result or an error, instead of leaving you to hope the bytes landed. The thing I would have hand-built inside a gRPC stream is the RPC system itself, already sitting there, pointed the other way.

None of this makes gRPC wrong. It optimizes for a world of stable, addressable services calling each other. We have plenty of that too, but we also have ephemeral clients (a CLI invocation, a running deploy, a controller mid-reconcile) that need to be callable for as long as they’re around, without becoming servers. Capabilities fit that better than streams do.

On the wire: CBOR, and two ways to call

Underneath the model, the bytes are CBOR, the same compact, typed, binary format we encode entities with. It’s a close cousin of JSON, but it knows the difference between an int64 and a float, and it’s smaller. Generated message types carry integer field keys rather than string ones, so a field is tagged cbor:"0,keyasint,omitempty" instead of by name. The field’s index in the schema is its identity on the wire, which keeps encodings small and lets a message grow new fields over time without breaking old readers.

There are two ways a call travels. One is the ordinary one-way call; the other carries traffic in both directions.

A plain unary call, with no capabilities passed, is a CBOR body sent as an HTTP/3 POST:

POST /_rpc/call/{oid}/{method}

The {oid} is the object being called; the {method} is which of its methods. Request body in, response body out, over HTTP/3 on QUIC. Nothing exotic. If that were the whole system, “CBOR over HTTP/3” would be a complete description of it.

But a call that passes an object needs something the request/response shape can’t give it: a channel the receiver can start traffic on, so it can call back into the capability you handed over. For that, the client opens a WebTransport session with an HTTP CONNECT:

CONNECT /_rpc/callstream/{oid}/{method}

WebTransport rides on HTTP/3 and QUIC, and it gives us bidirectional QUIC streams within a single session. The client’s own request and response go out on one stream, and, crucially, the client sits in a loop accepting streams the server opens back toward it. When the server wants to invoke recv.update(...), it opens a fresh stream on that session, and the client’s loop picks it up and dispatches it into the object the client passed. That accept loop is the server calling the client.

So “CBOR over HTTP/3” is true for the simple case and only half the story for the interesting one. The bidirectional feature isn’t plain HTTP/3 request/response; it’s a WebTransport session over HTTP/3, with QUIC streams flowing in both directions. Under all of it is quic-go for QUIC and HTTP/3, webtransport-go for the sessions, and fxamacker/cbor for the bytes.

YAML schemas, generated Go

You don’t hand-write CBOR marshalers or capability plumbing, the same way you don’t hand-write entity descriptors. You write a schema and generate the rest. A schema is a YAML file, rpc.yml, declaring the types and interfaces for one API. Here’s the full version of the meter-updates example from earlier, types and all:

apiVersion: miren.dev/rpc/v1
kind: IDL
types:
  - type: Reading
    fields:
      - name: temperature
        type: float32
        index: 0
      - name: seconds
        type: int32
        index: 1
      - name: meter
        type: string
        index: 2

interfaces:
  - name: UpdateReceiver
    methods:
      - name: update
        index: 0
        parameters:
          - name: reading
            type: Reading
  - name: MeterUpdates
    methods:
      - name: registerUpdates
        index: 0
        parameters:
          - name: recv
            type: UpdateReceiver
        results:
          - name: sub
            type: Subscription
  - name: Subscription
    methods:
      - name: cancel
        index: 0

Types are structs, each field with an explicit integer index that becomes its CBOR key. (The IDL also handles unions, lists, generics, and cross-file imports, but the struct-and-interface core is the part you touch daily.) Interfaces are services, methods with parameters and results. This is the same registerUpdates from before, fully specified, and both ends are worth reading. The parameter recv is an interface: the UpdateReceiver the client implements, so the server can call back into it. The result sub is an interface too: a Subscription the server hands back, so the client can cancel() when it’s done listening. One method, an interface going in and an interface coming out. The schema treats them the same; an interface is an interface, wherever it shows up.

A generator turns this into Go. Each API package carries a go:generate directive:

//go:generate rpcgen -input rpc.yml -output rpc.gen.go

so one go generate ./... regenerates every service in the tree. For each interface you get three things: a Go interface you implement on the server side, a typed Client you call from the other side, and an adapter that wires your implementation into the dispatch machinery. The adapter is where “an interface is just a value” becomes concrete. It builds a table of the methods and their handlers:

func AdaptMeterUpdates(t MeterUpdates) *rpc.Interface {
	methods := []rpc.Method{
		{
			Name:          "registerUpdates",
			InterfaceName: "MeterUpdates",
			Index:         0,
			Params:        []string{"recv"},
			Handler: func(ctx context.Context, call rpc.Call) error {
				return t.RegisterUpdates(ctx, &MeterUpdatesRegisterUpdates{Call: call})
			},
		},
	}
	return rpc.NewInterface(methods, t)
}

That *rpc.Interface, a dispatch table bound to a concrete implementation, is what every passed object turns into. When you pass an object as an argument, this is the thing that gets registered so the other side’s calls can find their way to your methods.

What actually happens when you pass one

The generated client method for registerUpdates shows the concrete steps behind all of this:

func (v MeterUpdatesClient) RegisterUpdates(ctx context.Context, recv UpdateReceiver) (*MeterUpdatesClientRegisterUpdatesResults, error) {
	args := MeterUpdatesRegisterUpdatesArgs{}
	ret := MeterUpdatesRegisterUpdatesResults{}
	caps := map[rpc.OID]*rpc.InlineCapability{}
	{
		ic, oid, c := v.NewInlineCapability(AdaptUpdateReceiver(recv), recv)
		args.data.Recv = c
		caps[oid] = ic
	}
	err := v.CallWithCaps(ctx, "registerUpdates", &args, &ret, caps)
	// ...
}

recv is your own UpdateReceiver implementation. AdaptUpdateReceiver(recv) builds its dispatch table, and NewInlineCapability mints a capability for it: a wire reference (an OID, a random 16-byte id the system hands out) paired with the local dispatch table the id resolves to. The reference goes into the arguments in place of the object; the table goes into a side-map keyed by that id. Then CallWithCaps opens the WebTransport session, sends the call, and, on a background goroutine, accepts the streams the server opens back and routes each one, by its OID, into the matching dispatch table. The server, for its part, receives the argument as a live client bound to that same session, and calling a method on it opens a stream back toward you.

Capabilities are reference-counted, because the receiver can hold onto one past the original call. There are ref and deref operations to say “I’m still using this” and “I’m done,” and the object stays alive on the far side exactly as long as someone holds a live reference to it. It’s the same discipline any capability system needs: the objects have to be garbage-collected across the wire, not just within one process.

Where this leaves us

Writing your own RPC system is not a free afternoon. It’s a real bet, in the same weight class as choosing your own storage model, and when you take a swing that big there’s no guarantee it pays off. gRPC exists. It’s good. We could have leaned on it and worked around the two-way cases like everyone else does.

It has paid off, and in the best way: we mostly don’t think about it. Every component in Miren, the CLI, the control plane, the runtime pieces, speaks one typed protocol, and when one of them needs to call back into another, it passes an object and calls it. The cases that used to sprout webhooks and polling loops are ordinary methods now. Adding a new call across the whole platform is a few lines of YAML and a go generate. The two-way calls we wanted at the start show up in more places than we predicted.

There are edges left to sand. Capability lifetime across flaky connections is never quite finished, and running your own transport means you own every reconnection bug yourself. But the shape is settled, and it’s the one we wanted: CBOR on the wire, QUIC underneath, and objects rather than just data moving in both directions. Letting the client hand the server an object turned a whole category of awkward problems into ordinary method calls.