Decision records

ADR-0033: Adaptive Runtime — control plane, data plane and listener sharing

Status: accepted (design, D0) — 2026-10-07

Context

One scalwsd process uses all CPUs it is given (per-core event loops, ADR-0029). The Adaptive Runtime adds capacity on one host by running several scalwsd instances and deciding, from measurements, when more web-tier capacity helps. Rollback point: pre-adaptive-runtime-2026-10-06 (docs/pre-adaptive-runtime-backup.md).

Decision

Two roles, one host.

• scalws-controller (new binary, feature adaptive-runtime): observes instances, decides, and runs their lifecycle. It never accepts, proxies or terminates client traffic, holds no TLS keys in memory, and is not on the request path.
• scalwsd instances: the data plane, unchanged except for the opt-in listener mode below and an instance identity. Each instance terminates TLS itself, as today.

Listener sharing: SO_REUSEPORT. With server.listener_mode: reuseport (default shared, today’s behaviour) every instance binds its own reuseport socket per listener address; the kernel spreads new connections across the group by hash. A connection, its HTTP/1 keep-alive requests and its HTTP/2 streams stay on the instance that accepted it: scaling adds capacity for new connections; it never migrates in-flight requests.

Removing an instance without dropping queued connections. When a reuseport socket closes, connections in its accept queue are reset — unless net.ipv4.tcp_migrate_req=1 (Linux ≥ 5.14), which moves them to another socket of the group. The controller requires it (checks and reports; refuses scale-in without it unless an operator overrides) and drain closes the listening sockets first, then finishes the accepted connections.

Alternatives.

• Controller as reverse proxy: an extra hop for every byte and a single choke point — rejected by the handoff (stop condition).
• One socket shared by fd passing (SCM_RIGHTS/LISTEN_FDS): one accept queue, no migration problem, accept balancing like nginx workers. Not chosen for the MVP because every instance would depend on the controller for its sockets (controller restart, privileges for port 80/443 stay in one place) — kept as the documented fallback if reuseport distribution proves unfair for long-lived connections (measured in D1/D5).
• eBPF reuseport steering: future, only if D1/D5 measurements show hash imbalance.

Feature flags. adaptive-runtime cargo feature for the controller; the data-plane change (listener_mode) is plain configuration with today’s default. With the default, scalwsd is byte-for-byte the same request path: the regression gates of the handoff apply against the frozen baseline.

Consequences

• Each instance needs its own admin socket, metrics address, runtime directory and log identity: derived from an instance id (ADR-0035).
• Shared mutable state must be audited (ADR-0036): per-instance caches are acceptable; limits and quotas must not multiply with the instance count.
• The kernel decides placement; imbalance is measured, not assumed (D1 benchmark plan).

Addendum (2026-10-07): shared accept queue (listen: inherited)

The controller may open the listeners itself and hand duplicates to the instances it started (Unix socket, SCM_RIGHTS; only pids it spawned, checked with SO_PEERCRED). All instances then accept from one kernel queue instead of one SO_REUSEPORT queue each. It is the controller default; listen: reuseport remains available.

• Scale-in and reload: a leaving instance closes its duplicate, the queue stays open for the others: nothing to migrate, no reset, tcp_migrate_req not needed (also on kernels < 5.14 and in containers where the sysctl cannot be set). Measured: 0 failed requests over 3 × 40 rolling reloads under connection churn, against 1 connection error per 40 reloads with reuseport.
• Throughput: an idle instance takes the next connection. With a new connection per request: +17 % / +8 % / +15 % req/s at 8 / 64 / 256 connections, p99 −20…−28 %. With keep-alive: equal (64 connections, 5 interleaved runs).
• Not solved: connection share per instance is no more even than the kernel hash (busiest instance 0.53–0.62 of 2); long-lived connections stay where they were accepted.
• Constraints: the listeners are fixed while the controller runs (a generation that changes them is refused); after a controller restart new instances get the new controller’s sockets (same address, SO_REUSEPORT group) while adopted instances keep the old queue until they leave — draining those still benefits from tcp_migrate_req. HTTP/3 stays refused with several instances.

Benchmark: bench/listen-modes.sh.