Adaptive Runtime

HTTP/3 with several instances — operation

Design: ADR-0039 (transport plane), ADR-0040 (connection IDs), ADR-0041 (lifecycle). TCP (HTTP/1.1, HTTP/2) is unchanged: one shared accept queue per address (listen: inherited).

How it works

 clients ──UDP :443──► kernel socket (opened once by scalws-controller)
                            │ read by
                            ▼
                 scalws-controller quic-lb  (demux process)
          DCID → owner instance · Initial → placement · unknown → any (stateless reset)
                            │ Unix SOCK_SEQPACKET, one connection per instance endpoint
             ┌──────────────┼──────────────┐
             ▼              ▼              ▼
          web-0          web-1          web-2     quinn endpoints with an abstract socket:
             └──────────────┴──────────────┘      replies go out directly on the shared
                     same UDP socket (send only)  UDP socket (GSO, correct source address)

Configure HTTP/3 as for one instance: http3: true on a TLS listener. Under the controller the instance receives the UDP socket with the TCP sockets; nothing else to configure.
The demux is started by the controller when a generation has HTTP/3 listeners, adopted after a controller restart, restarted if it dies or when the HTTP/3 sockets change.
Connection IDs are 12 bytes and carry the owning instance’s server id, encrypted (keys in <state_dir>/quic-keys.json, 0600). Server ids are never reused while a key generation lives (<state_dir>/quic-server-id).

Lifecycle

Event	HTTP/3 behaviour
scale-out	the new instance registers with the demux; it receives new connections after activation; existing connections stay where they are
scale-in / restart / rolling reload	the instance stops receiving new connections, sends GOAWAY on each connection, finishes accepted requests, closes with `H3_NO_ERROR`; datagrams of its connections keep reaching it until it exits
instance crash	datagrams for its connections go to another instance, which answers with a stateless reset (shared reset key): clients reconnect at once
demux crash	restarted by the controller with the same UDP socket; instances re-register; established connections continue
controller restart	demux and instances keep running and are adopted
client address change (NAT rebinding, migration)	routing follows the connection ID, not the address

Also with a single instance (no controller) a shutdown now drains HTTP/3 instead of closing every connection at once (ADR-0041).

Observability

scalws-controller status → quic: demux pid, its UDP sockets, restarts, next server id.
scalws-controller quic-lb-status --demux-socket <state_dir>/quic-lb.sock → datagrams received, routing reasons (Owner, Placed, Pinned, Orphan, VersionNegotiation, NotQuic, NoInstance), drops, and per instance: server id, placeable, datagrams, bytes, drops on a full instance socket.

Limits

The demux is one process: inbound HTTP/3 packets per second on a host are bounded by it — ≈ 8.4 µs CPU per inbound datagram, about 110 000 datagrams/s on one core of the development VM (docs/benchmarks/2026-10-07-remaining-gaps.md). Outbound bytes do not pass through it. Placing it behind the instances costs ~10 % on one instance; three instances serve 2.5× one.
No QUIC connection moves between processes; a connection lives and ends on its instance.
listen: reuseport still refuses HTTP/3 with several instances.

Tests

scripts/controller-h3.sh (scale-out/in, 1 MiB responses across scale-in, rebinding every second across scale-in, controller restart, demux crash, rolling reload, instance SIGKILL, malformed datagrams), using scalws-h3-probe (bench/h3probe).