Benchmarks

2026-10-07 — Remaining gaps: soak, HTTP/3 with several instances, distribution

Host: development VM (8 vCPU EPYC 7642, Ubuntu 24.04, kernel 6.8). Binaries: scalws-adaptive-runtime-rc1 (tag, /opt/scalws-rc1) and the development build after it (/opt/scalws-dev). Only interleaved A/B numbers are compared across builds (the VM drifts by several percent within an hour).

rc1 soak, 6 h (`scripts/controller-soak.sh`, E1)

6.01 h, 175 cycles (1 → 2 → 3 → 2 → 1 instances) with one fault per cycle, 11 kinds × 16: controller restart, SIGKILL of an active / starting / draining instance, failed readiness, upstream dead / slow / resetting, memory pressure, scale beyond max, reload. Seven load streams in 60 s windows (HTTP/1.1 static, 1 MiB, proxy, new connection per request, HTTP/2 + cache, PHP, WebSocket).

Stream	requests in clean windows	errors	p99 ms (median of windows)
HTTP/1.1 static	25.5 M	0	0.37
HTTP/1.1 1 MiB	0.34 M	0	1.34
HTTP/1.1 proxy	7.8 M	0	0.72
HTTP/1.1 new connection	5.0 M	0	0.56
HTTP/2 + cache	12.4 M	10 (timeout)	0.51
PHP	2.5 M	0	1.01
WebSocket messages	10.0 M	0	–

Controller restarts: 16/16 adopted every running instance. No orphaned state record in 2127 samples; zombies at most 2, transient.
Resources: controller 7.4 → 7.2 MB RSS, 15 FDs, 3 threads; instances 32.5 → 31.8 MB RSS (mean), ~220 FDs. No growth.
Faults: all 11 recovered each time. The memory-pressure probe did not reach its threshold (defect in the soak script — fixed, re-checked by the mixed soak).
The host’s TCP “estab” count rose 1166 → 6558: CLOSE_WAIT sockets in the benchmark’s WebSocket echo server (Node HTTP servers are half-open; it never closed after a peer FIN). Fixed in bench/apps/ws/server.js; reproduced and verified (20 of 20 sockets stuck before the fix). ScalWS FDs stayed bounded throughout.
The 10 HTTP/2 timeouts (0.8 per million) fell in 4 windows, each the window right after a controller restart. bench/ctl-restart-h2.sh (same stream, 23 controller restarts in 4 min, rc1) produced 0 errors in 192 000 requests, with and without restarts: the restart alone does not cause them. Open; watched in the mixed soak.
A 42-minute period (10:00:30–10:42 UTC) is annotated: a parallel test blocked metrics ports of the soak’s instances.

HTTP/3 behind the QUIC demux (`bench/h3-instances.sh`, D7)

3 runs × 10 s, medians. Instances on CPUs 0–2 (one each), demux and controller on CPU 3, 4 × scalws-h3-probe (16 connections × 4 streams, /hello.txt) on CPUs 4–7. plain-1: one scalwsd with its own UDP socket, no controller, no demux.

Setup	req/s	p99 ms	instance CPU µs/req	demux CPU µs/req	demux µs/datagram	datagrams/s	efficiency
plain-1	33 840	3.08	29.1	–	–	–	–
demux, 1	30 633	3.40	32.2	14.5	14.1	31 600	1.00
demux, 2	51 766	3.06	38.1	12.8	10.1	65 600	0.84
demux, 3	75 947	2.62	38.9	11.1	8.4	101 000	0.83

The demux costs 9.5 % of one instance’s throughput; 3 instances serve 2.48× one.
The demux is the ceiling: at 3 instances it uses ~0.85 of its CPU (≈ 8.4 µs per inbound datagram). Correction (profile of 2026-10-08, docs/benchmarks/2026-10-08-demux-batching.md): receive is already batched (recvmmsg, ~32 per call); the cost is one SEQPACKET message per forwarded datagram, and sendmmsg did not reduce it. Several datagrams per message would be the next step if one host needs more than ~110 000 inbound datagrams/s. Outbound traffic does not pass through the demux.

Does connection imbalance hurt? (`bench/imbalance.sh`, E3)

Two instances (one CPU each); 8 HTTP/2 connections × 32 streams opened in a burst plus HTTP/1.1 at 2000 req/s; status.distribution sampled every second; 3 runs per mode.

Mode	HTTP/2 req/s (runs)	HTTP/2 p99 ms	busiest connection share	max pressure imbalance	harmful seconds
inherited	143 541 / 137 403 / 146 617	3.66–3.79	0.63	0.06–0.11	0
reuseport	148 582 / 111 540 / 160 352	3.39–4.73	0.58–0.67	0.10–0.22	0

No harmful second in any run: the policy of ADR-0038 holds — no mitigation is built.

Regression gate and cache-hit CPU

Cache-hit CPU flag (rc1 gate had 26.4 → 29.1 µs/request): interleaved release A/B, 7 × 30 s, one instance:

Build	req/s (median)	p99 ms	CPU µs/req
baseline	103 173	9.22	27.97
rc1	102 987	9.17	27.94
dev	102 121	9.17	27.85

No difference: the flag was drift between non-interleaved runs.

dev vs rc1 (MODE=release bench/multi.sh, 5 × 30 s; the three scenarios with the largest deltas repeated with 7 runs):

Scenario	req/s	CPU µs/req	7-run repeat (req/s, CPU)
static-small	−3.2 %	+7.1 %	+2.9 %, −1.3 %
static-1m	−2.2 %	−1.3 %
respond	−4.3 % (CV 5 %)	−2.6 %	+0.6 %, +8.6 %
proxy	+4.3 %	−2.9 %
cache-hit	−2.3 %	+2.6 %
TLS + HTTP/2	+0.7 %	+10.3 %	−0.8 %, +9.6 %
PHP	+1.5 %	−5.2 %

Throughput within ±5 % everywhere. CPU per request from a saturated closed loop is noisy (signs flip between repeats), so it was measured at a fixed rate (bench/cpu-fixed-rate.sh, 20 000 req/s, 6 alternating rounds, server on 2 CPUs):

Scenario	rc1 µs/req	dev µs/req
TLS + HTTP/2	31.77	30.77 (−3 %)
respond	30.73	30.84 (+0.4 %)

Gate passed. The only request-path changes since rc1 are an Option check on the Alt-Svc insert and atomic loads of the per-IP limits.

Controller test suite (dev build)

e2e 21/21, d5 (sweep, HTTP/2 scale-in, reload while scaling, max instances, memory guard, bursts), slow-drain, WebSocket drain (20 tunnels, all 1001), HTTP/2 drain 5/5 (only refused-before-processing streams), listener generations 13/13, HTTP/3 12/12 (instance SIGKILL: 3 of 15 connections reset — clients of the killed instance).

Mixed soak, 6 h, dev build (`H3=1 scripts/controller-soak.sh`)

2026-10-07 18:52 → 2026-10-08 00:53 UTC, 6.01 h, 180 cycles, 11 fault kinds × 16–17. Eight streams: the seven of the rc1 soak plus HTTP/3 through the QUIC demux (connections rebind to a new source port every 5 s).

Stream	requests in clean windows	unsent (retry-safe)	p99 ms (median / max)
HTTP/1.1 static	28.7 M	0	0.97 / 2.50
HTTP/1.1 1 MiB	0.38 M	0	1.42 / 4.04
HTTP/1.1 proxy	8.9 M	0	1.34 / 5.34
HTTP/1.1 new connection	5.7 M	0	0.93 / 4.61
HTTP/2 + cache	14.2 M	4 624 (GOAWAY)	1.16 / 5.83
HTTP/3	15.2 M	1 924	1.46 / 3.36
PHP	2.8 M	0	1.35 / 9.89
WebSocket messages	11.4 M	140 (in flight at 1001)	–

0 errors in 87 M requests/messages in clean windows. Excused windows (overlapping a SIGKILL of a serving instance, or a dead/resetting upstream for the proxy stream): proxy 160 050 (the upstream was dead), static 130, HTTP/2 88, HTTP/3 272, PHP 15, WebSocket 510 — the clients of a killed process.

The rc1 soak’s 10 HTTP/2 timeouts did not recur: 0 in 14.2 M, across 17 controller restarts. Cause unknown; the soak keeps checking for it.
Controller restarts 17/17 adopted every active instance (3.0 s). All faults recovered every time; no settle timeout.
Resources flat: controller 8.0 MB RSS / 16 FDs, QUIC demux 7.9 → 8.0 MB / 13 FDs, instances 33.0 → 32.5 MB / 82 FDs, TCP estab 210 → 206 (the WebSocket echo-server fix holds: rc1 soak 1166 → 6558), 0 orphaned sockets, zombies ≤ 2, no stray state record in 2125 samples. Generations 1 → 18.
One p99.9 outlier: the 1 MiB stream had one window at 3.2 s (median 2.2 ms).
Memory-pressure probe: still not a valid test. The probe’s memory hog filled pages one by one in Python and was not done when the probe scaled: MemAvailable was ~17 GiB, above min_free_ram (4 GiB), so the scale-out correctly happened. The controller did see the pressure later (MemoryPressure in 45 samples). Fixed: the hog writes 64 MiB chunks and the probe waits until MemAvailable is below the threshold; checked separately (below).

Memory-pressure re-check (FAULTS_ONLY=fault_memory H3=1 SOAK_SECONDS=420, dev build, 3 cycles, all eight streams): every probe ran with MemAvailable at 2.9 GiB (below 4 GiB); the scale request did not start an instance (active 1 → 1) and the controller reported MemoryPressure. After the hog ended the instance count recovered. Load during the run: 0 errors in 1.96 M requests/messages.