Internals: benchmark reports

Kubernetes ingress controllers on one k3s node: ScalWS, ingress-nginx, Traefik, 2026-10-14

ScalWS in ingress mode (chart deploy/helm/scalws) against ingress-nginx and Traefik on one k3s node: the same backend, the same Ingress objects, one controller installed at a time. Harness: bench/kubernetes/ (README there). Round 2 (below, the current figures) measures ScalWS at commit 4823999 with both accept policies and ingress-nginx with its defaults and tuned for this workload. The first section measures the fountain accept policy again after its connection cap (00ae108, 2026-10-15). Round 1 (ScalWS 115f5f2) and “After fixes” (d14639f) follow as history.

Fountain with the connection cap: ScalWS 00ae108, 2026-10-15

ADR-0044 now also counts a loop as full while it holds twice its fair share of the live connections (at least 8): with 16 loops and 256 connections a cap of 32 per loop. This section measures fountain with that change in the same harness, and ScalWS shared again as a same-day control. The round 2 figures below are unchanged; ingress-nginx and Traefik were not run again.

Image scalws:bench-00ae108 (sha256:561d3a0ae6c98ea90b6bad7560776198d983d3632422b8e8ec698cbe6ddedafb), values as in round 2 (values/scalws.yaml, values/scalws-fountain.yaml), all scenarios, 3 runs each, fixed rate 19000 req/s. Order: throughput fountain, throughput shared, fixed rate fountain, fixed rate shared, then a perf pass of fountain for the CPU per loop. No pod restarts. Raw results on the bench host, in its results directory: results/k8s-ingress-2026-10-15-fcap and ...-fcap-perf.

The same-day shared control repeats round 2 within 1 % in requests/s (h1-small 64482 against 64273, tls-h2 62209 against 62389) and within 0.1 ms in fixed-rate p99, so the two days compare.

Throughput, requests/s (median of 3 runs) and p99:

Fixed rate, 19000 req/s, 3 runs of 60 s, medians (clean percentiles as defined in round 2):

Raw p99 per run (ms): latency 2.88, 2.55, 2.86; churn 3.18, 3.08, 3.59; config 3.25, 2.78, 2.73 (round 2: churn 8.50, 16.88, 5.98; config 12.46, 11.47, 7.86). The worst one-second p99 per run is still higher than with shared (churn 7.1 to 10.3 ms against 3.7 to 4.0 ms).

CPU per event loop during the perf pass (% of one core, busiest first; perf itself costs CPU, so the figures are relative):

Findings:

1. The p99 penalty of fountain shrank. At the fixed rate fountain’s p99 is now 1.5x to 1.6x that of shared (round 2: 1.8x to 7x); the Ingress churn and backend churn scenarios lost their outliers (11.15 to 2.35 ms, 6.08 to 2.54 ms) and the runs no longer differ from each other by a factor of two to three. Requests/s went up to 0.92x to 0.98x of shared (round 2: 0.92x to 0.96x). The controller CPU saving at the fixed rate is the same as before (2.2 against 2.5 cores, 0.88x).
2. Against tuned ingress-nginx fountain now loses in five figures instead of ten: tls-h2 p99 (22.96 against 21.26 ms), latency p50 (0.77 against 0.69 ms), latency p99 clean (2.46 against 2.05 ms), latency p99.9 clean (3.17 against 2.80 ms) and config p50 (0.79 against 0.71 ms). It is ahead in the p99 of h1-small, h1-100k (35.87 against 36.12 ms, within noise), tls-h1 and tls-newconn, in every churn and config p99, and in requests/s, CPU and memory everywhere. The target (fountain’s p99 no longer behind tuned ingress-nginx) is met for the HTTP/1.1 throughput scenarios and the churn scenarios, not for tls-h2 and the quiet fixed-rate scenario.
3. Why the rest remains: the cap does not bind for the opening burst. At 19000 req/s the 256 connections still sit on four loops (84/78/44/35 %), on average 64 each against a cap of 32. Only connections accepted before the current 20 ms window count against the cap (ADR-0044: counting the current window throttled short-connection workloads), so connections that arrive together are placed by the surge estimate, and the loop that is awake keeps accepting while it is full. tls-h2 (64 connections, cap 8) shows the same: loops 0 and 1 at the ceiling. The core benchmark with all connections opened at once (ADR-0044) shows this in pure form. A loop that also steps back while full from a surge fixes the closed-loop placement but collapsed the accept rate under many new connections in a first experiment (ADR-0044, “Not done”).

Recommendation unchanged: shared for ingress mode. fountain remains the policy for hosting servers with many short connections, where it saves CPU.

Reproduce:

# on the bench host, as root, a checkout of 00ae108
SCALWS_TAG=bench-00ae108 bench/kubernetes/setup-host.sh
export SCALWS_TAG=bench-00ae108 OUT=results/k8s-ingress-2026-10-15-fcap
for c in scalws-fountain scalws; do SCENARIOS="h1-small h1-100k tls-h1 tls-h2 tls-newconn" bench/kubernetes/run.sh $c; done
for c in scalws-fountain scalws; do RATE=19000 SCENARIOS="latency churn config" bench/kubernetes/run.sh $c; done
OUT=$OUT-perf RATE=19000 SCENARIOS=perf PERF_SCENARIOS="latency h1-small h1-100k tls-h1 tls-h2" bench/kubernetes/run.sh scalws-fountain
bench/kubernetes/aggregate.py $OUT
systemctl stop k3s; /usr/local/bin/k3s-killall.sh

Round 2: ScalWS 4823999, both accept policies, ingress-nginx default and tuned

Raw results on the bench host, in its results directory: results/k8s-ingress-2026-10-14-r2 (pass 1: all scenarios, all five configurations), ...-r2-pass2 (throughput again in reverse order, ScalWS both policies and tuned ingress-nginx), ...-r2-diag (ingress-nginx with the keep-alive settings only), ...-r2-perf and ...-r2-perf2 (profiles). Run on the night of 2026-10-14 to 2026-10-15.

Summary

Throughput: median of 3 runs of 30 s, 256 requests in flight. “ScalWS shared / ingress-nginx tuned” is the ratio of requests/s.

Fixed rate, 19000 req/s, 3 runs of 60 s. Latency is the median over the runs of the clean p99 (see “Fixed rate”); failed requests are non-2xx plus client errors, summed over the 3 runs. “ingress-nginx tuned / ScalWS” is the ratio of p99 (above 1.00x: ScalWS lower).

Efficiency: controller pod CPU per request in the throughput scenarios (us), controller CPU at the fixed rate (cores) and peak RSS (MiB). “ingress-nginx tuned / ScalWS shared” above 1.00x means ScalWS uses less.

In this setup ScalWS with the default accept policy (shared) is ahead of both ingress-nginx configurations and Traefik in every scenario and every metric measured: requests/s, p50 and p99, failed requests, CPU per request and memory. Against tuned ingress-nginx it is 1.29x in the tested HTTP/1.1 small-response workload, 1.19x with 100 KiB responses, 1.36x (TLS, HTTP/1.1), 1.55x (TLS, HTTP/2) and 2.46x with a new TLS connection per request. The tuning helps ingress-nginx only in the 100 KiB scenario and costs it elsewhere (next section); against the best ingress-nginx configuration per scenario the margins are 1.10x (h1-small) to 2.44x (tls-newconn), see “ingress-nginx: which settings help”. With the fountain accept policy ScalWS still has more requests/s than every other controller, but loses to tuned ingress-nginx in ten latency figures (p99 of the four keep-alive throughput scenarios, p50 and p99 at the fixed rate); see “Where ScalWS loses and why”.

What changed since round 1

• ScalWS image scalws:bench-4823999 (ScalWS 1.0.0-rc.1 at 4823999, sha256:43e9582c6044): the round 1 findings fixed (pool retry of idempotent requests on a broken reused connection, listener backlog 4096, no default memory limit in the chart), the controller writes retries: 2 into its pools, only the namespace that owns a host adds paths to it, and idle-connection replay only for idempotent requests.
• ScalWS runs twice: server.accept_policy: shared (the default, values/scalws.yaml) and fountain (ADR-0044, values/scalws-fountain.yaml).
• ingress-nginx runs twice: chart defaults as in round 1 (values/ingress-nginx.yaml) and tuned (values/ingress-nginx-tuned.yaml, next section).
• Run order, throughput: ScalWS shared, ingress-nginx tuned, ScalWS fountain, ingress-nginx default, Traefik; the fixed-rate scenarios in the same order afterwards. Traefik was measured again in this order (all scenarios).
• The fixed rate stays at 19000 req/s, as in round 1, for comparability.

Everything else (host, k3s, pinning, backend, Ingresses, oha, scenarios, method) is as in round 1; see “Setup” and “Fairness choices” below.

ingress-nginx tuned: the settings

values/ingress-nginx-tuned.yaml, on top of the round 1 values (16 workers, access log off). The keys an operator would reach for after round 1 and the ingress-nginx ConfigMap reference:

Not changed: proxy-connect-timeout (5 s) and retries (proxy-next-upstream, 3 tries), TLS settings (session cache on, tickets off, the default curves), worker-cpu-affinity (ScalWS does not pin its loops either), max-worker-connections (16384) and reuse-port (on). The backend has no preStop hook, as in round 1; an operator would add one to the backend, which would help every controller in the churn scenario alike.

ingress-nginx: which settings help

The tuned values are slower than the defaults except at 100 KiB. A diagnostic run (...-r2-diag, values/ingress-nginx-keepalive.yaml: the tuned values without proxy-buffer-size) separates the two changes. Requests/s, median of 3 runs:

• With the 64 KiB proxy-buffer-size ingress-nginx reaches 0.85x of the keep-alive-only requests/s with 13-byte responses (CPU per request 205 to 254 us), 0.86x with TLS and HTTP/1.1 and 0.88x with HTTP/2; it doubles the 100 KiB throughput. nginx sets up the proxy buffer for every request; that the larger buffer is the cost per request is the likely reading, ingress-nginx was not profiled. An operator would set it per Ingress (annotation nginx.ingress.kubernetes.io/proxy-buffer-size) for the paths that return large bodies; here one Ingress carries both paths, so the global setting is what applies.
• The keep-alive settings help a little (1.01x to 1.04x). HTTP/2 still had about 1100 to 1300 failed streams per 3 runs with keep-alive-requests: 10000 (15560 with the default 1000): the GOAWAY comes ten times less often but still comes.
• tls-newconn was not part of the diagnostic run; both measured configurations are within 1 % there, so the best of the two is used.

Throughput (scenarios 1 to 4)

Median of 3 runs; min and max of the 3 runs; CPU and RSS of the controller pod (all its containers); “oha cores” is the load generator’s CPU (16 available); errors are non-2xx plus client errors.

The ingress-nginx default and Traefik rows repeat round 1 within 1 % (round 1: 58479, 12374, 51518, 44867, 5532 and 39623, 19482, 36691, 28064, 6013). ScalWS shared also repeats its round 1 and “After fixes” figures within 2 %. All controller pods ran without a restart.

Fixed rate (scenarios 5 to 7)

19000 req/s, HTTP/1.1 keep-alive, /small, 256 connections, 3 runs of 60 s, medians over the runs (failed requests and slow requests: sums). Definitions as in round 1: “clean” leaves out the first second and the seconds in which oha itself stalls; “worst 1 s p99” is the highest per-second p99 of all runs; “slow over 1 s” counts successful requests whose own round trip took more than 1 s.

Per run:

• Backend churn (4 -> 8 -> 2 -> 4). ScalWS shared had no failed request in 3 runs (round 1: 126, “After fixes”: 15). The pools now retry twice (retries: 2), and the 15 remaining 502s of “After fixes”, all at the 8 -> 2 step where the one retry went to another pod that was leaving, are gone. The 1 failure with fountain is the same kind (502 at the scale-down). ingress-nginx (both configurations) and Traefik behave as in round 1: thousands of 502s per run and 256 requests per run that wait for the 5 s connect timeout (ingress-nginx) or hit oha’s 10 s timeout (Traefik), which is what their p99 shows. The ingress-nginx tuning does not change this.
• Ingress churn. ScalWS shared: no failed request, p99 clean 1.60 ms against 1.56 quiet. ingress-nginx: 826 and 981 failed requests in 3 runs (connection error, connection closed before message completed, operation was canceled) from the reloads closing client keep-alive connections, as in round 1. Traefik: no failed request, p99 clean 3.48 ms.
• CPU at the same rate. ScalWS shared 2.4 cores, fountain 2.1, ingress-nginx 3.3 (default) and 3.8 (tuned: the 64 KiB buffer again), Traefik 5.8.

Accept policy for ingress mode: shared

Recommendation: keep shared (the chart default) for ingress mode. In this setup:

fountain does what ADR-0044 says it does, lower controller CPU at a moderate fixed rate (0.88x at 19000 req/s, with lower peak RSS too), but an ingress controller is judged on latency and throughput, and fountain is behind shared in every throughput scenario (0.92x to 0.96x) and has 1.8x to 7x the p99 at the fixed rate. Pass 2 repeated the throughput gap (0.91x to 0.97x). The CPU saving is worth having on a hosting server where PHP or Node.js share the cores (the case ADR-0044 targets), not in front of a Service. The cause is in the next section.

Where ScalWS loses and why

ScalWS shared is not behind tuned ingress-nginx (nor default ingress-nginx, the keep-alive-only variant or Traefik) in any scenario or metric of this run: requests/s, p50, p99 (throughput and fixed rate), failed requests, CPU per request and peak RSS. The narrowest margins are against the best ingress-nginx configuration per scenario: 1.10x in h1-small (keep-alive only) and 1.19x in h1-100k (tuned).

ScalWS fountain loses to tuned ingress-nginx in these figures:

(In churn its p99 is far below ingress-nginx’s, which is dominated by the 5 s connect timeouts.) Requests/s, CPU per request and RSS of fountain are still better than tuned ingress-nginx everywhere.

Profiles. run.sh scenario perf: the load of a scenario runs as in the measurement, and after the 10 s warm-up perf record -F 499 -g samples the scalwsd process for 20 s (perf pinned to the load CPUs), while the CPU time of every scalwsd thread is read from /proc before and after. Files: ...-r2-perf2/{scalws,scalws-fountain}/perf-<scenario>.{txt,data}. perf costs CPU itself (scalwsd used 12.4 cores in h1-small under the profiler against 10.4 without), so the thread figures are relative; values slightly above 100 % come from tick accounting over the 20 s window.

CPU per event loop thread (% of one core, busiest first, the 16 scalws-loop-N threads, separated by /):

The symbol profile of fountain is the same as shared’s (top self entries in the latency scenario: scalws_http::conn::drive 1.7 %, nft_do_chain 1.6 %, route_and_handle 1.4 %, PoolProxyHandler::forward 1.4 %, none above 2 %): fountain does not do more work per request, it does it on fewer loops. Findings:

1. fountain packs the connections onto the first loops and keeps them there. At 19000 req/s the 256 keep-alive connections end up on three loops: loops 0 and 1 at about 90 % of a core, loop 2 at 46 %, the other 13 at about 4 % each. By ADR-0044 a loop counts as full from 85 % busy and a later loop only takes connections while the earlier ones are full; established connections never move. A loop at 85 to 90 % busy is a queue: requests wait for the loop, which is the 1.8x p99 against shared (where 16 loops share the load at 16 to 27 %). Under the throughput load the same happens one level up: the first three loops at the ceiling, the rest at about 50 to 70 %; the requests of the connections on the saturated loops are the slow ones (p99) and, with a closed-loop client, they also cap requests/s (0.92x to 0.96x).
2. The placement is decided by the opening burst and then fixed for the run. Per-second p99 of the config scenario, run 1 (config-run1.persec.csv): fountain 11.4 to 13.2 ms in every second from the first one on, apart from the oha stall seconds and one second at 17.7 ms; shared 1.5 to 1.9 ms apart from the stall seconds. The Ingress events (every 2 s) do not show in either series; the level is set when oha opens its 256 connections at once at the start. That is why the fountain runs differ so much from each other (raw p99 of the three config runs 12.46, 11.47, 7.86 ms; churn 8.50, 16.88, 5.98 ms; latency about 2.5 ms per second in its run 1): the burst estimate (10 % per connection above the previous window’s count) decides how many loops get connections, and nothing evens it out later. The config and churn scenarios are not harder for fountain than the latency scenario; their runs drew a worse placement.
3. Fix candidates (not made here): for proxy workloads a much lower “full” threshold (busy share around 50 %) or a cap on connections per loop while later loops are idle; or moving idle keep-alive connections off a loop that stays above the threshold. Until then shared is the right policy for ingress mode (previous section). The cap on connections per loop was added on 2026-10-15; its measurement is the first section of this report.

Headroom in ScalWS shared (not a loss against ingress-nginx; where the profiles say the next gains are):

• h1-small: 49.8 % of scalwsd’s samples are in the kernel, 45.3 % in scalwsd, 4.1 % in libc. writev (sending responses and upstream requests) is 37.5 % inclusive, of which about 24 points are the softirq that delivers the packet through the veth, the cni0 bridge and bridge netfilter in the sender’s context (br_nf_pre_routing 16.9 %, nf_hook_slow 14.1 %, nft_do_chain 4.0 %, all inclusive). That is the pod network path of flannel and kube-proxy rules, paid by every controller per packet. In user space no symbol is above 1.9 % (conn::drive 1.90, route_and_handle 1.63, PoolProxyHandler::forward 1.25, Target::send 1.13, mimalloc mi_free plus _mi_page_malloc_zero 1.2): there is no single hot spot left, and fewer system calls or packets per request are the larger lever.
• h1-100k: 68.9 % kernel. rep_movs_alternative 10.8 % is copying the body: out of the upstream socket (recvfrom, skb_copy_datagram_iter) and into the client socket. kernel_init_pages 8.0 % is the kernel zeroing fresh pages for socket send buffers (tcp_sendmsg_locked -> sk_page_frag_refill -> alloc_pages -> post_alloc_hook): Ubuntu’s kernel is built with CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y, so every page TCP allocates for sent data is cleared first. Both costs are per byte relayed through user space and apply to nginx and Traefik as well; only a zero-copy relay (splice) would avoid them.
• tls-h2: the 64 HTTP/2 connections landed on 11 of the 16 loops, 9 of them at the ceiling, 5 loops idle. The HTTP/2 figure (62389 req/s, 1.55x tuned ingress-nginx) is reached with 11 loops; with 64 long-lived connections the shared accept queue leaves part of the CPU unused (the first-come concentration ADR-0044 describes). The throughput scenarios with 256 connections show the same in milder form (one loop idle or at 6 % in h1-small and h1-100k).

User-space call chains in these profiles are not usable (scalwsd and libc are built without frame pointers; frames above the system call show up as unrelated libc symbols); the kernel chains and the flat symbol profile are.

Run spread and order

Within each pass the 3 runs of a throughput scenario are within 1 % of each other for every configuration except ScalWS fountain (h1-small 57495 to 61256, 6 %) and ScalWS shared tls-h2 (59935 to 63612, 6 %). Pass 2 ran the throughput scenarios again in reverse order (ingress-nginx tuned, ScalWS fountain, ScalWS shared); this also serves as the A/A check for the same image and values:

So differences below about 2 % (4 % for fountain) are within the noise of this setup. The fixed-rate scenarios ran once per configuration (3 runs each); their run-to-run spread is in the per-run table above.

Round 2 limitations

Those of round 1 apply (one node, controller and backend on the same 16 CPUs, oha’s vector-growth stalls, self-signed RSA 2048, ingress-nginx retired upstream). In addition:

• “Tuned” is one operator’s reasonable global configuration, not a search for the best ingress-nginx configuration per scenario; the diagnostic run shows the global 64 KiB buffer costs it at small bodies, and the comparison against the best configuration per scenario is given next to it.
• ScalWS was not tuned beyond the accept policy.
• Traefik ran with its round 1 values in the new order; it repeats round 1 within 1 %.
• The oha SQLite files (2.9 GB) were reduced to per-second summaries (*.persec.csv: requests, p50, p99, p99.9 and max per second) after aggregation; aggregate.csv keeps the clean figures computed from them.

Reproduce round 2

# on the bench host, as root, a checkout of 4823999 with these bench files
SCALWS_TAG=bench-4823999 bench/kubernetes/setup-host.sh   # builds and imports the image
OUT=results/k8s-ingress-2026-10-14-r2 RATE=19000 \
  ORDER="scalws ingress-nginx-tuned scalws-fountain ingress-nginx traefik" bench/kubernetes/run-all.sh
for c in ingress-nginx-tuned scalws-fountain scalws; do
  OUT=results/k8s-ingress-2026-10-14-r2-pass2 SCENARIOS="h1-small h1-100k tls-h1 tls-h2 tls-newconn" bench/kubernetes/run.sh $c
done
OUT=results/k8s-ingress-2026-10-14-r2-diag SCENARIOS="h1-small h1-100k tls-h1 tls-h2" bench/kubernetes/run.sh ingress-nginx-keepalive
for c in scalws scalws-fountain; do
  OUT=results/k8s-ingress-2026-10-14-r2-perf2 RATE=19000 SCENARIOS=perf \
    PERF_SCENARIOS="latency h1-small h1-100k tls-h1 tls-h2" bench/kubernetes/run.sh $c
done
systemctl stop k3s; /usr/local/bin/k3s-killall.sh

Round 1: ScalWS 115f5f2 (history)

The first run, ScalWS at 115f5f2, all controllers with chart defaults. Raw results: results/k8s-ingress-2026-10-14 (main run), ...-pass2 (throughput repeated in reverse controller order), ...-probe (diagnostics). Findings 1 to 3 were fixed afterwards (see “After fixes” and round 2). Numbers in the tables are written without thousands separators like the round 2 tables; the text is the original.

Summary

ScalWS was ahead in every scenario of this setup: 1.10x ingress-nginx in the tested plain HTTP/1.1 small-body workload up to 2.44x in the connection-per-request workload. The margins and the conditions behind them are discussed below; the most important caveats: one node with the controller and the backend sharing 16 CPUs, chart defaults everywhere (ingress-nginx’s defaults cost it in the 100 KiB and HTTP/2 scenarios: with a 64 KiB proxy buffer it reaches 84 % of ScalWS at 100 KiB, see “Diagnostics”), and one finding against ScalWS: with the chart’s default memory limit scalwsd was OOM-killed in the 100 KiB scenario (fixed since, see “After fixes”).

Setup

Machine: xeon, 2x Xeon E5-2650 v2 (8 cores, 16 threads each, no AVX2), 121 GB RAM, performance governor, turbo off, Ubuntu 26.04, kernel 7.0.0-38-generic.

Fairness choices

• One node, one controller at a time. Each controller is installed with helm into the namespace ingress, measured, and removed with its namespace and IngressClass before the next one. The backend Deployment is restarted for every controller.
• CPU split. k3s, containerd and every pod (controller and backend) run on NUMA node 0 (CPUs 0-7,16-23): a CPUAffinity=0-7 16-23 drop-in for k3s.service and AllowedCPUs=0-7 16-23 on the kubelet’s kubepods.slice. The k3s drop-in alone did not reach the containers: attaching a task to a cgroup v2 cpuset resets its affinity to the cgroup’s CPUs (all 32), so the slice property is needed too; the kubelet creates the slice as a transient unit, so it is set at runtime (setup-host.sh, and again by run.sh). environment.txt of every controller shows all controller and backend processes on 0-7,16-23. oha runs on node 1 (taskset -c 8-15,24-31).
• Workers. No CPU limits. Every controller gets 16 workers: ingress-nginx worker-processes: "16" (16 nginx workers seen), Traefik GOMAXPROCS=16, ScalWS server.event_loops: 16 (scalwsd showed 34 threads: 16 event loops plus helpers). The defaults would pick the same from the CPU affinity; they are set so it is explicit.
• Controller and backend share node 0. The backend (4 replicas x 2 nginx workers) runs on the same 16 CPUs as the controller. Requests/s is therefore the throughput of “controller plus backend on 8 cores”; a controller that needs less CPU leaves more to the backend. Controller CPU per request (pod cgroup cpu.stat) is reported next to it and is the cleaner efficiency figure.
• Traffic path. oha connects to the controller pod IP from the host (--connect-to), so no Service, kube-proxy or NodePort path is involved and the path (host -> cni0 bridge -> veth) is the same for all three. hostNetwork was not used: the ScalWS chart has no option for it.
• Same Ingresses. manifests/ingress.yaml: bench.local (plain) and tls.bench.local (TLS from the Secret bench-tls, self-signed RSA 2048), path / Prefix to Service backend:80. Only ingressClassName differs. No annotations.
• Same backend. nginx, /small 13 bytes (return 200), /100k a 102 400-byte file, access log off, upstream keep-alive for 100 000 requests per connection. Its stub_status gives the upstream connections each controller opened (last column of the throughput table: every controller keeps upstream connections alive, 0.1 to 1 new connection per 1000 requests).
• Settings changed from the chart defaults (all in values/): one replica, ClusterIP Service, the worker counts above, access logs off for all three (ingress-nginx disable-access-log: "true", ScalWS server.logging.access_log: false; Traefik logs no access by default), and for ScalWS no memory limit (next point). Everything else is the chart default, including upstream keep-alive (ingress-nginx 320 idle connections per worker, Traefik 200 per host, ScalWS 32 per member, fixed by the controller), retries (ingress-nginx up to 3 tries on error or timeout with a 5 s connect timeout, ScalWS 1 retry, Traefik none) and ingress-nginx’s keep-alive-requests: 1000 and proxy-buffering: off (both matter below).
• ScalWS memory limit (finding 1, fixed in 5015c8b). The chart set resources.limits.memory: 256Mi for scalwsd. In the 100 KiB scenario at 256 connections scalwsd was OOM-killed twice (oom-kill ... task=scalwsd, anon RSS 254 MB at the limit) and the run answered mostly “connection refused”. ingress-nginx and Traefik have no memory limit by default, so the bench values remove it for ScalWS (limits: null); peak RSS then was 323 MiB in that scenario. This is a chart default that breaks under ordinary load (100 KiB responses), not a bench artefact; see “Findings”.
• Load. Throughput: 10 s warm-up, then 3 runs of 30 s, 256 requests in flight (HTTP/2: 64 connections x 4 streams), median and min to max. Fixed rate: 3 runs of 60 s at 19 000 req/s (half of the slowest controller’s scenario 1 median, rounded down to 1000), 256 connections, --latency-correction (no coordinated omission), request timeout 10 s. oha’s aborted due to deadline (requests in flight when a timed run ends, at most 256 per run) is not counted as an error.

Results

Throughput (scenarios 1 to 4)

Median of 3 runs. CPU and RSS are the controller pod (all its containers: for ScalWS scalwsd plus the scalws-ingress sidecar, for ingress-nginx the Go controller plus nginx). “oha cores” is the load generator’s CPU (16 available).

The spread between runs is below 2 % everywhere. No run answered anything but 200 apart from the ingress-nginx HTTP/2 errors (below). Peak RSS is the highest sample of the pod during the scenario; the pod is not restarted between throughput scenarios, so it carries the high-water mark of earlier scenarios (ScalWS 323 MiB in h1-100k, then 291 in tls-h1).

Pass 2 (...-pass2): the five throughput scenarios again, controller order reversed (Traefik, ingress-nginx, ScalWS), same method. Medians in req/s, change against the table above in brackets:

Order has no visible effect; CPU per request repeats within 2 %.

Fixed rate (scenarios 5 to 7)

19 000 req/s, HTTP/1.1 keep-alive, /small, 3 runs of 60 s each, medians. “clean” leaves out the first second (oha opens its 256 connections) and the seconds in which oha itself stalls (see Limitations); the raw percentiles are given as well. “failed” is non-2xx plus oha errors (timeouts, resets). “slow > 1 s” counts successful requests whose own round trip took over 1 s.

Per run: churn non-2xx ScalWS 49 / 31 / 46 (502), ingress-nginx 3 018 / 2 718 / 2 621 (502 and 27 to 41 504 each), Traefik 4 901 / 4 531 / 4 459 (502) plus 256 timeouts each. Config errors, ingress-nginx: 700 connection error, 97 connection closed before message completed, 92 operation was canceled, about 300 per run.

Controller CPU at the same 19 000 req/s: ScalWS 2.5 cores, ingress-nginx 3.2, Traefik 5.8.

Discussion

Plain HTTP/1.1, small body. ScalWS +10 % over ingress-nginx and +62 % over Traefik in requests/s, with 18 % less controller CPU per request than ingress-nginx and half of Traefik’s. The controller and backend together are what saturates node 0; oha used 4 of its 16 CPUs, so the client is not the limit.

100 KiB body. ScalWS 2.3x ingress-nginx and 1.5x Traefik. ingress-nginx used 13.8 cores for 12 374 req/s, 1.1 ms of CPU per request against 0.34 ms for ScalWS. ingress-nginx ships proxy-buffering: off with a 4 KiB proxy-buffer-size, so nginx relays a 100 KiB response in 4 KiB reads and writes; the diagnostics confirm it: with proxy-buffer-size: 64k ingress-nginx reaches 24 092 req/s, 84 % of ScalWS. The table shows a default install; against a tuned ingress-nginx the ScalWS lead at 100 KiB is about 1.2x, not 2.3x.

TLS, HTTP/1.1 and HTTP/2. ScalWS +19 % (HTTP/1.1) and +37 % (HTTP/2) over ingress-nginx, +67 % and +120 % over Traefik. ScalWS loses almost nothing for TLS (64.3k plain, 61.1k TLS) and its HTTP/2 is as fast as its HTTP/1.1. ingress-nginx answered about 0.4 % of the HTTP/2 requests with errors (http2 error, operation was canceled): its default keep-alive-requests: 1000 closes an HTTP/2 connection with GOAWAY after 1000 requests, and oha fails the streams it had queued on it. With keep-alive-requests: 100000 the errors disappear and requests/s rise by about 3 % (diagnostics). Browsers retry such streams, so the errors are partly a load-generator effect; the cost of the reconnects is in the requests/s figure.

New connection per request. ScalWS 13 486 req/s, 2.4x ingress-nginx and 2.2x Traefik, with less than half of their CPU per request. All three resume TLS 1.3 sessions and negotiate X25519MLKEM768 with OpenSSL 3.5’s default client; the server-to-client bytes per oha connection (1 710 to 1 998) are close to a resumed handshake plus the response and well below a full one (about 2 400), so oha very likely resumes on all three and every connection pays the hybrid post-quantum key exchange. Treat the ScalWS figure as a lower bound: oha used 14.1 of its 16 CPUs there, and the controller 14.8 cores, so both sides were close to saturation.

Latency at a fixed rate. At 19 000 req/s ScalWS has the lowest p50, p99 and p99.9 (clean p99 1.55 ms, ingress-nginx 2.17, Traefik 3.21) and uses the least CPU. ScalWS had 19 responses slower than 1 s, all in the first 35 ms of one run and each 1.015 s: 19 of the 256 connections oha opens at the start took one SYN retransmission (1 s), so some connection attempts of that burst were dropped by scalwsd’s listener. The other two controllers had none. Small, but a real difference in the connection-burst path; not investigated further here.

Backend churn. This scenario separates the controllers most. The backend is scaled 4 -> 8 -> 2 -> 4; the backend pods have no preStop hook, so a removed pod stops right after SIGTERM (nginx graceful quit) while controllers may still send to it.

• ScalWS: 126 non-2xx in 3 runs (0.004 %), p99 unchanged (1.73 ms clean), worst second 4.3 ms. The 502s are at the scale-down: the scalwsd log shows upstream request failed ... Connection reset by peer from pods being removed, on reused keep-alive connections. The single retry did not save those requests (it was never made: see “After fixes”).
• ingress-nginx: about 2 800 502s and 30 504s per run, and 256 requests per run that took 5 s (the default 5 s proxy connect timeout to a pod that is gone, then a retry). p99 is 4.6 s because oha’s 256 connections all end up waiting on such requests at the scale-down, and requests scheduled behind them queue (latency correction counts that).
• Traefik: about 4 600 502s per run and 256 requests that hit oha’s 10 s timeout: Traefik has no retry and its dial timeout (30 s) is longer than oha’s request timeout. p99 9.7 s for the same reason as above.

The p99 figures of ingress-nginx and Traefik are shaped by oha’s fixed connection count (a blocked connection blocks its queue); the counts of failed and slow requests are the robust figures. With more clients, more requests would wait.

Ingress churn. An unrelated Ingress is added or removed every 2 s and once a path is added to the measured host. ScalWS and Traefik: no failed request, p99 unchanged (ScalWS 1.58 ms against 1.55 quiet; Traefik 3.37 against 3.21). ingress-nginx: about 300 failed requests per run and clean p99 4.66 ms against 2.17 quiet: each new or removed host is an nginx reload, the old workers close their keep-alive client connections, and requests sent on them at that moment fail. ScalWS also reloads for a new host (the structure channel); its reload evidently keeps the established client connections serving.

Memory. ScalWS has the smallest footprint at small bodies (114 to 132 MiB against about 380 for ingress-nginx and 200 to 330 for Traefik) but reached 323 MiB in the 100 KiB scenario, about the level of the others. That figure includes the scalws-ingress sidecar.

Findings

1. Fixed in 5015c8b: chart default memory limit too low (ScalWS, chart). resources.limits.memory: 256Mi in deploy/helm/scalws/values.yaml gets scalwsd OOM-killed when proxying 100 KiB responses at 256 concurrent connections (peak RSS without the limit: 323 MiB). A controller that restarts under ordinary load is a production outage. Options: raise or drop the default limit in ingress mode, and/or find out why scalwsd needs several hundred KiB per in-flight 100 KiB response (allocator retention per event loop is a candidate). Not changed here; the bench values work around it. Fix: no memory limit by default, request 128Mi (see “After fixes”).
2. Fixed in 0358812: 502 on reset keep-alive upstream connections at pod removal (ScalWS). About 40 per scale-down. A retry of idempotent requests that failed on a reused connection before any response byte would likely remove them. Fix: such requests are now retried on another member (idempotent methods only); 15 instead of 126 in three runs (see “After fixes”).
3. Fixed in 2df2cba: listener drops in a connection burst (ScalWS). 19 of 256 simultaneous connection attempts needed a SYN retransmission in one of three runs. Cause: the shared listener socket was created by std’s TcpListener::bind, which listens with a backlog of 128; it now uses 4096 like the reuseport sockets.
4. Upstream keep-alive is not tunable in ingress mode (ScalWS). The controller writes keepalive: 32 into every pool; ingress-nginx exposes upstream-keepalive-connections. It did not matter here (0.17 to 0.55 new upstream connections per 1000 requests).
5. Traefik chart value names. service.type is ignored by Traefik chart 41.x (service.spec.type is the key) and its pods are labelled app.kubernetes.io/instance=<release>-<namespace>; only relevant to the harness.

Diagnostics (not part of the measurement)

run.sh scenario probe, after the measured runs, results/k8s-ingress-2026-10-14-probe:

ingress-nginx, 10 s runs on a fresh install, changes applied one after the other with kubectl patch of the controller ConfigMap:

So a 64 KiB proxy buffer nearly doubles ingress-nginx in the 100 KiB scenario, to 84 % of ScalWS’s 28 526 (and CPU per request drops by about half). The ScalWS lead in that scenario is mostly an ingress-nginx default, not a data-plane difference of 2.3x. The tls-h2 figure of these short runs (50 to 52k) is higher than in the measured sequence (44.9k); the probe runs are 10 s without warm-up on a fresh pod, and the difference is not explained here, so the measured 44.9k may understate ingress-nginx’s HTTP/2 by up to about 15 %.

Limitations

• One node, one machine. Controller, backend and load generator share one host (two NUMA nodes, the load on the other node). No network card, no kube-proxy path, one controller replica. Multi-node behaviour, endpoint propagation across nodes and real NIC interrupts are not covered.
• Controller and backend share 16 CPUs, so requests/s mixes controller and backend cost; the CPU-per-request column isolates the controller.
• Defaults. Each controller runs its chart defaults except the settings listed above. A tuned ingress-nginx (proxy buffers, keep-alive-requests) or Traefik (retries, timeouts) would do better in the scenarios that hurt them; ScalWS was not tuned either.
• oha artefacts. oha keeps every result in a growing vector; when it doubles (at 2^17, 2^18, … results, at 19 000 req/s around 7, 14, 28 and 55 s) the client stalls for 30 to 270 ms and every controller shows a latency spike in that second. The raw p99.9 values carry it for all three controllers alike; the “clean” columns leave those seconds out. In the connection-per-request scenario oha itself was close to its CPU limit with ScalWS.
• Self-signed RSA 2048 certificate, TLS 1.3 only in practice, cipher chosen by each server (ScalWS and ingress-nginx AES-256-GCM, Traefik AES-128-GCM).
• ingress-nginx is retired upstream (the Kubernetes project ended its maintenance in 2026), but it is still the most widely deployed controller and the one ScalWS’s annotation support targets, which is why it is the reference here.
• Single day, single host; the throughput scenarios were repeated in reverse controller order (pass 2) to check for order effects, the fixed-rate scenarios once.

Reproduce

# on the bench host, as root, in the repository checkout
bench/kubernetes/setup-host.sh
bench/kubernetes/run-all.sh                 # throughput, then fixed rate at half the slowest
SCENARIOS=probe bench/kubernetes/run.sh scalws   # diagnostics, per controller
systemctl stop k3s; /usr/local/bin/k3s-killall.sh

After fixes: ScalWS d14639f (history)

Commits on top of 115f5f2: 0358812 (pool retry), 2df2cba (listen backlog), 5015c8b (chart default), d14639f (test formatting only). Image scalws:bench-d14639f, ScalWS only, same host, harness and method as above; the ScalWS values now use the chart’s resources (no memory limit). Raw results, in the bench host’s results directory: results/k8s-ingress-fixes.

Finding 2, root cause. In a pool the outcome of an attempt whose connection broke after the request was written (Attempt::Broken, here Connection reset by peer on a reused keep-alive connection to a pod that is shutting down) was never retried, for any method; only connect failures, timeouts and 502 to 504 answers were. The “single retry” never happened for these requests. The connection-level replay in the HTTP/1 client only covers requests hyper had not written yet and bodyless requests on connections idle for at least 1 s, which is never the case at 19 000 req/s. Now a broken attempt is retried on another member for idempotent methods (as nginx’s proxy_next_upstream error does); POST and PATCH are still not repeated once written. Removing a member through the admin API already closed its idle connections (the member, its connection pool and the idle connections are dropped with the old member set); a test now pins that. Tests: reset_on_a_reused_connection_is_retried_for_get_on_another_member (failed before the fix with a 502, passes after), reset_on_a_reused_connection_is_not_retried_for_post_once_written, removed_members_close_their_idle_connections, balance::tests::retry_rules.

Finding 3, root cause. The default listener mode shares one socket across the event loops; it was created with std’s TcpListener::bind, which listens with a backlog of 128 (the reuseport sockets already used 4096). 256 clients connecting at once overflowed the accept queue and some retransmitted their SYN after 1 s. The socket now listens with 4096 (capped by net.core.somaxconn, 4096 by default in a pod’s network namespace; ingress-nginx uses net.core.somaxconn, plain nginx 511). Test: cores::tests::shared_listener_queues_a_connection_burst (129 of 300 connections before, 300 after).

Finding 1. The chart sets no memory limit by default (as the ingress-nginx and Traefik charts) and requests 128Mi. Memory under the 100 KiB load is per connection, not a leak or whole-response buffering by design: the response body is streamed frame by frame, but the client-side write queue holds up to 256 KiB per connection (more than a 100 KiB body) and each chunk read by the upstream HTTP/1 client keeps its read buffer allocation alive (hyper’s adaptive read buffer grows to a few hundred KiB on fast reads) until it is written to the client. That gives about 0.8 MiB per in-flight 100 KiB response at 256 connections over the baseline of about 110 MiB, in line with ingress-nginx and Traefik (361 to 378 MiB). Lowering it (a smaller write high-water mark or upstream read buffer) trades throughput and was not changed here.

The 15 remaining 502s all come from the 8 -> 2 step (25 s into each run), where 6 of 8 pods stop at once. The scalwsd log line of each is the last attempt, the retry, and it again ended in Connection reset by peer: the retry went to another pod of the same scale-down (6 of the 7 other members were leaving). More retries (the pool’s retries, up to 5; the controller writes 1) or a preStop delay on the backend pods remove these; neither was changed here. The 1 s SYN retransmissions of finding 3 occurred in one run of three originally, so “0 in 3 runs” is consistent with the fix but does not prove it alone; the unit test does.

Core regression gate (bench/multi.sh, MODE=release, 5 runs, base = 115f5f2, new = d14639f, both cargo build --release --locked -p scalwsd --features http3, k3s stopped; results/xeon-ingress-fixes-ab). The absolute figures are lower than in 2026-10-14-xeon-ingress-core-regression.md for both binaries alike; only the ratio counts:

CV of the runs 0.4 to 1.1 %. None of the changes is on the path of these scenarios (the pool retry rule is evaluated after an upstream attempt, the backlog is set once at bind), so the differences are within the 1 to 2 % that A/A runs of the same binary show.