Adaptive Runtime

Adaptive Runtime — final report (2026-10-07)

Design: ADR-0033 … 0036 and docs/adaptive-runtime.md (architecture diagram, configuration reference). Rollback point: pre-adaptive-runtime-2026-10-06 (VERIFIED, docs/pre-adaptive-runtime-backup.md). Benchmarks: docs/benchmarks/2026-10-07-adaptive-runtime.md.

Failure and recovery tests

scripts/controller-e2e.sh (19 checks, passed three times in a row) and scripts/controller-d5.sh (all checks passed), on the development host:

Drain defects found and fixed on the way (they also affected a single instance on every SIGTERM / restart):

1. a just-accepted connection was closed with its request unread (client: incomplete message);
2. accept loops were not tracked by the drain, so a just-accepted connection could be dropped (reset);
3. idle keep-alive connections were closed while the client sent its next request — HTTP/1 responses after the shutdown signal now carry Connection: close, idle connections wait up to 1 s for their next request;
4. rolling reload could drain two instances at once (capacity dropped by two) — now one drain at a time.

Definition of success (handoff) — status

Known limitations

• Multi-instance is not more CPU-efficient than one instance with the same CPUs (D1); its value is adding and removing capacity. Each instance costs ~26–30 MiB.
• Kernel hash distribution is uneven with few connections (64: up to 2:1). The shared queue (listen: inherited, below) does not even it out either (busiest 0.53–0.62 of 2 instances): long-lived connections stay where they were accepted. An instance that is idle does take the next connection, which helps connection-heavy traffic (+8…17 %).
• HTTP/3 with several instances — resolved (ADR-0039 … 0041): the controller’s QUIC demux routes datagrams by routable connection IDs; scripts/controller-h3.sh 12/12 (placement on 3 instances, scale-out keeps 16/16 connections, scale-in with 1 MiB responses 0 truncated, rebinding every second across scale-in, controller restart and demux crash without effect, rolling reload 0 errors). Still refused with listen: reuseport. Remaining limits, measured 2026-10-07 (docs/benchmarks/2026-10-07-remaining-gaps.md):
  • the demux is one process per host: ≈ 8.4 µs CPU per inbound datagram, ceiling about 110 000 inbound datagrams/s on one core (3 instances: 76 k req/s used ~0.85 of it). Receive is already batched (recvmmsg, ~32 per call); the cost is one Unix message per forwarded datagram — sendmmsg did not help (measured, rejected, docs/benchmarks/2026-10-08-demux-batching.md). Next step if ever needed: several datagrams per Unix message (protocol change, after 1.0);
  • behind the demux one instance serves ~10 % less than a standalone one; 3 instances serve 2.48× one (efficiency 0.83);
  • after an instance SIGKILL some of its clients see a reset, others close only at their idle timeout (the stateless reset reaches them only when they send);
  • the HTTP/3 drain grace (client close after GOAWAY) is fixed at 10 s.
• Managed PHP-FPM / Node / Python / container runtimes are refused under the controller; application tiers must be external endpoints. Application-tier scaling is recommendation-only (D6). Design proposal: docs/d8-managed-runtimes-proposal.md.
• Listener changes needed a controller restart — resolved (ADR-0037): listener generations; scripts/controller-listeners.sh 13/13 (add/remove port, HTTP/3 add/remove, bind-address change, HTTPS, IPv6, port in use rejected with nothing changed, invalid certificate rolled back to the serving generation, controller restart mid-transition with one socket per address), 0 failed requests on the address that stayed.
• Scale-in requires net.ipv4.tcp_migrate_req=1 — only with listen: reuseport now: the default listen: inherited shares one accept queue opened by the controller, which outlives every instance. Earlier fix, still valid for reuseport: the package ships /usr/lib/sysctl.d/90-scalws-adaptive-runtime.conf and applies it on install; the controller refuses scale-in without it (unless overridden) and says why.
• Limit sharing divides limits evenly; with uneven hashing one instance may reject a client earlier than a single process would (never later).
• Per-instance caches (response, open-file, TLS sessions): hit ratios start cold on new instances. Resolved for purges: scalws-controller purge QUERY sends the admin purge to every serving instance and reports per instance (e2e: entries 1/1 → 0/0).
• Slow-client drains, now tested conclusively (scripts/controller-slow-drain.sh, 16 MiB at 200 KiB/s, connections attributed to instances while downloading): with a 10 s window the 6 downloads on the leaving instance end incomplete, the 6 on the staying one complete; with a 100 s window all 12 complete. Note: data already in the kernel’s send buffer (several MB on loopback) is still delivered after the process exits — files smaller than that are not cut at all.
• A connection reset was seen once during a rolling reload (1 in ~4.5 million requests over three runs). Resolved with listen: inherited: reproduced as 1 connection error per 40 reloads under connection churn with reuseport (3 rounds: 3 errors), 0 with the shared queue (3 rounds) — the reset came from a per-instance queue being closed.
• WebSocket across scale-in — resolved: tunnels were cut without a close frame (clients saw 1006). They now end with 1001 Going Away at a frame boundary after the HTTP drain; scripts/controller-ws-drain.sh (20 clients, scale-in 2 → 1): before 8 × 1006, after all closes 1001, reconnected, no message lost, 10/10 runs in both listen modes. The test also exposed an event-loop shutdown race (loops dropped tasks right after the HTTP drain), fixed for every SIGTERM.
• Node.js 0.79× explained (not a proxy deficiency): the managed Node.js process ran ~30 % slower right after the PHP scenario in the same server process (identical syscalls per request; more futex/GC activity inside Node). With FRESH_SERVER=1 (fresh processes per scenario) Node is 0.98×, PHP 1.12×, Python 0.99× of nginx; in an isolated interleaved test scalws is 1.16× on Node.
• HTTP/2 clients report requests canceled by GOAWAY during scale-in / reload (oha: “http2 error”): these were never sent or were refused before processing (REFUSED_STREAM) — retry-safe; scripts/controller-h2-drain.sh classifies them.
• Connection imbalance is reported (status.distribution, ADR-0038), not acted on: no harmful imbalance has been measured (bench/imbalance.sh: 0 harmful seconds).
• rc1 soak: 10 HTTP/2 client timeouts in 12.4 M requests in clean windows (0.8 per million), each in the window after a controller restart. Not seen again: 0 in 192 000 with 23 restarts (bench/ctl-restart-h2.sh) and 0 in 14.2 M in the 6 h mixed soak (17 controller restarts). Cause unknown; the soak keeps checking for it.
• Mixed 6 h soak (dev build, eight streams including HTTP/3, 180 cycles, 11 fault kinds): 0 errors in 87 M requests/messages in clean windows, resources flat (docs/benchmarks/2026-10-07-remaining-gaps.md). Its memory-pressure probe was invalid (the hog was not yet full); fixed and re-checked: below min_free_ram no instance starts.

Reverted or rejected experiments