Decision records

ADR-0039: HTTP/3 with several instances — a separate QUIC transport plane

Status: accepted (design, D7) — 2026-10-07

Context

Under scalws-controller, HTTP/1.1 and HTTP/2 share one accept queue (listen: inherited, ADR-0033 addendum). QUIC cannot use that model: UDP has no accept queue, every datagram of a connection must reach the one process that holds the connection’s state, and the client’s address may change during a connection (NAT rebinding, migration). With one socket per instance in a SO_REUSEPORT group the kernel hashes the 4-tuple, which breaks on every address change and on every change of the group. HTTP/3 has therefore been refused with more than one instance.

Requirements (handoff D7): keep the TCP plane unchanged; the controller stays out of the HTTP data path; established QUIC connections always reach their owner, also after address changes; new connections are placed on serving instances; scale-out and scale-in never move or break an established connection; no custom QUIC implementation; eBPF optional at most.

Decision

A QUIC plane next to the TCP plane:

                       scalws-controller (control plane)
             owns the TCP and UDP sockets, supervises, never touches packets
                  |                                   |
          TCP plane (inherited queue)        QUIC plane, UDP :443
                  |                          scalws-quic-lb (demux)
                  |                          inbound datagrams only:
                  |                          DCID -> owner, Initial -> placement
                  |                                   |  Unix SOCK_SEQPACKET
          scalwsd web-0 .. web-n  <-------------------+
             H1/H2 from the queue,  H3 endpoints receive from the demux and
             send replies directly on the shared UDP socket (no hop outbound)

1. Socket ownership. The controller opens the UDP socket of every HTTP/3 listener once (same address as the TLS listener) and hands duplicates (SCM_RIGHTS, ADR-0033 addendum) to the demux and to the instances. Only the demux ever reads it; instances only send on it. A demux or controller restart therefore never rebinds: the kernel socket, and the datagrams queued on it, survive.
2. Demux (scalws-controller quic-lb, a separate supervised process). Reads datagrams, parses the QUIC invariant header only (RFC 8999: header form, DCID length and DCID; short-header DCIDs have our fixed length), and forwards the datagram with its source address, destination address and ECN bits to the owner over a Unix SOCK_SEQPACKET connection. It never decrypts, never terminates TLS or HTTP. It runs outside the controller so that the controller stays off the data path and a demux crash does not take the control plane with it (the controller restarts it with the same sockets; instances reconnect).
3. Routing (details in ADR-0040):
   • datagrams whose DCID is a server-issued, routable CID go to the instance encoded in it — whatever the client’s address;
   • client Initial / 0-RTT packets (client-chosen DCID) are placed: rendezvous hash of the DCID over the instances that accept new connections, pinned for 10 s in a small table so that retransmissions and 0-RTT reach the same instance even if the placement set changes meanwhile;
   • a DCID that names no live instance (crashed, unknown key) goes to any serving instance, which answers with a stateless reset (shared reset key), so the client learns at once instead of waiting for its idle timeout;
   • datagrams too short for a QUIC header or with unknown long-header versions go to any instance (version negotiation is quinn’s) or are dropped.
4. Instances run quinn with an abstract socket (Endpoint::new_with_abstract_socket): receive = the demux connection, send = quinn-udp on the shared socket (GSO, PKTINFO source address). A routable CID generator, the shared reset key and the CID key come from the controller with the sockets. One instance alone (no controller) keeps today’s plain UDP endpoint.
5. No cross-process QUIC state migration in this version. A connection lives and ends on the instance that accepted it.
6. eBPF (SO_ATTACH_REUSEPORT_EBPF steering by CID) stays a possible later optimization, only if the demux is measured to be a bottleneck.

Consequences

• One extra hop (Unix seqpacket write/read) for inbound datagrams only; responses — the bulk of the bytes — leave directly. Must be measured (CPU per packet, packets/s, p99) against one instance with a plain endpoint.
• The demux is a single process per host: its packet rate bounds inbound H3. It is stateless apart from the 10 s Initial table, so restarting it costs at most in-flight handshakes.
• Instances need the demux to serve H3; until registered, an instance does not advertise HTTP/3 (Alt-Svc) — ADR-0041.
• The TCP plane, its defaults and its regression gates are untouched.

Alternatives rejected

• One UDP socket per instance in a SO_REUSEPORT group — 4-tuple hashing: breaks on rebinding and on every scale event.
• Shared UDP socket read by every instance — datagrams reach a random process.
• Per-instance UDP ports advertised through Alt-Svc — not :443, breaks HTTPS/SVCB records and clients that cached an advertisement of a removed instance.
• Demux inside the controller — puts the controller on the data path.
• eBPF first — kernel-feature dependent; not needed for correctness.