ScalWSScalWS

    Get started
    ScalWSScalWS
    Get started

    Kubernetes ingress benchmark

    ScalWS in ingress mode against ingress-nginx and Traefik on one k3s node: the same backend, the same Ingress objects, one controller installed at a time. With its default accept policy ScalWS is 1.29× tuned ingress-nginx in the tested HTTP/1.1 small-response workload, 1.55× in the tested TLS + HTTP/2 workload and 2.46× with a new TLS connection per request. Against the best ingress-nginx setting for each scenario the range is 1.10× (HTTP/1.1, small response) to 2.44× (TLS, new connection per request).

    One machine: controller, backend and load generator share a host (the load generator on the other CPU socket), one node, one controller replica, traffic sent straight to the controller pod. ingress-nginx is retired upstream (2026); it is the reference because it is still the most widely deployed controller. ScalWS commit 4823999, measured 2026-10-14. Limits and where ScalWS loses → Methodology →

    1.29×

    tuned ingress-nginx · HTTP/1.1, small response

    1.55×

    tuned ingress-nginx · TLS, HTTP/2

    2.46×

    tuned ingress-nginx · TLS, new connection per request

    Limits and where ScalWS loses →

    Throughput

    Five scenarios at full load. ScalWS runs with the default accept policy (shared); the opt-in fountain policy is shown as a secondary variant. ingress-nginx runs with its chart defaults and tuned for this workload (settings); Traefik with its chart defaults.

    ScalWS vs ingress-nginx tuned

    • HTTP/1.1, small response1.29×
    • HTTP/1.1, 100 KiB response1.19×
    • TLS, HTTP/1.11.36×
    • TLS, HTTP/21.55×
    • TLS, new connection per request2.46×

    ScalWS (shared accept policy) requests/s divided by tuned ingress-nginx requests/s, as published. Right of the 1.00× line: ScalWS faster. Symmetric log scale.

    ScalWS vs the best ingress-nginx setting

    • HTTP/1.1, small response1.10×
    • HTTP/1.1, 100 KiB response1.19×
    • TLS, HTTP/1.11.17×
    • TLS, HTTP/21.36×
    • TLS, new connection per request2.44×

    The same, against whichever ingress-nginx configuration was fastest in each scenario (defaults, keep-alive settings only, or tuned), as published.

    Requests per second, every controller

    • HTTP/1.1, small responseScalWS64.3kScalWS fountain59.8kingress-nginx58.2kingress-nginx tuned49.9kTraefik39.6k
    • HTTP/1.1, 100 KiB responseScalWS29.0kScalWS fountain27.3kingress-nginx12.3kingress-nginx tuned24.3kTraefik19.5k
    • TLS, HTTP/1.1ScalWS61.2kScalWS fountain57.6kingress-nginx51.4kingress-nginx tuned45.1kTraefik36.9k
    • TLS, HTTP/2ScalWS62.4kScalWS fountain57.4kingress-nginx44.4kingress-nginx tuned40.2kTraefik28.1k
    • TLS, new connection per requestScalWS13.5kScalWS fountain13.0kingress-nginx5.53kingress-nginx tuned5.49kTraefik6.01k

    Requests per second, median of 3 runs of 30 s, 256 requests in flight. Higher is better. Each row is scaled to its own largest value.

    Requests/s, medians of 3 runs; ratios are ScalWS (shared) divided by the peer.
    ScenarioScalWS sharedScalWS fountainingress-nginx defaultingress-nginx tunedTraefikvs ingress-nginx defaultsvs ingress-nginx tunedvs best ingress-nginxvs Traefik
    HTTP/1.1, small response64.3k59.8k58.2k49.9k39.6k1.11×1.29×1.10×1.62×
    HTTP/1.1, 100 KiB response29.0k27.3k12.3k24.3k19.5k2.35×1.19×1.19×1.49×
    TLS, HTTP/1.161.2k57.6k51.4k45.1k36.9k1.19×1.36×1.17×1.66×
    TLS, HTTP/262.4k57.4k44.4k40.2k28.1k1.41×1.55×1.36×2.22×
    TLS, new connection per request13.5k13.0k5.53k5.49k6.01k2.44×2.46×2.44×2.25×

    “vs tuned” and “vs best” are as published in the report; “vs defaults” and “vs Traefik” are computed from the same requests/s.

    Latency at full load, p50 / p99 ms, and errors summed over the runs.
    ScenarioScalWS shared p50 / p99ScalWS fountain p50 / p99ingress-nginx default p50 / p99ingress-nginx tuned p50 / p99Traefik p50 / p99ScalWS shared errorsScalWS fountain errorsingress-nginx default errorsingress-nginx tuned errorsTraefik errors
    HTTP/1.1, small response2.74 / 16.741.25 / 22.003.46 / 16.994.00 / 19.965.24 / 22.9300000
    HTTP/1.1, 100 KiB response6.54 / 32.642.48 / 57.8317.78 / 70.498.56 / 36.1210.11 / 50.5600000
    TLS, HTTP/1.12.91 / 17.801.55 / 22.283.81 / 20.204.34 / 21.935.54 / 24.7800000
    TLS, HTTP/23.15 / 15.182.33 / 23.004.27 / 20.755.16 / 21.267.72 / 28.860015,5601,0920
    TLS, new connection per request17.22 / 46.7017.30 / 53.4535.22 / 183.9934.74 / 200.2826.74 / 166.7900000

    Latencies are medians of the runs; errors are non-2xx answers plus client errors. The load generator is closed-loop, so these latencies include queueing at full load. The ingress-nginx HTTP/2 errors are streams cut by its keep-alive request limit (GOAWAY).

    Latency, backend churn and Ingress churn

    Three scenarios at a fixed 19,000 req/s over HTTP/1.1 keep-alive. In the first nothing changes. In backend churn the backend Deployment is scaled 4 to 8 to 2 to 4 pods during each run, and the pods have no preStop delay, so a removed pod stops while controllers may still send to it. In Ingress churn an unrelated Ingress is added or removed every 2 s.

    • 0

      failed requests for ScalWS in backend churn (3 runs). ingress-nginx: 5,979 (defaults) and 8,512 (tuned); Traefik: 13,690.

    • 0

      failed requests for ScalWS in Ingress churn. ingress-nginx: 826 and 981 (its reloads close client keep-alive connections); Traefik: 0.

    • 1.56 ms

      ScalWS p99 at 19,000 req/s; 1.70 ms in backend churn and 1.60 ms in Ingress churn. Tuned ingress-nginx: 2.05 ms quiet.

    Fixed rate: p99 and failed requests

    • Fixed rate, quiet: p99ScalWS1.56 msScalWS fountain2.80 msingress-nginx2.24 msingress-nginx tuned2.05 msTraefik3.20 ms
    • Backend churn: failed requestsScalWS0ScalWS fountain1ingress-nginx5,979ingress-nginx tuned8,512Traefik13,690
    • Backend churn: p99ScalWS1.70 msScalWS fountain6.08 msingress-nginx4.65 singress-nginx tuned4.67 sTraefik9.71 s
    • Ingress churn: failed requestsScalWS0ScalWS fountain0ingress-nginx826ingress-nginx tuned981Traefik0
    • Ingress churn: p99ScalWS1.60 msScalWS fountain11.15 msingress-nginx5.02 msingress-nginx tuned7.85 msTraefik3.48 ms

    19,000 req/s, 3 runs of 60 s. p99 is the "clean" p99 (median over the runs, without the first second and the seconds in which the load generator itself stalled); failed requests are non-2xx answers plus client errors, summed over the runs. Lower is better.

    19,000 req/s, 3 runs of 60 s; medians, failed and slow requests summed.
    Scenario, controllerp50 msp99 msp99.9 msworst 1 s p99 msfailedover 1 sCPU corespeak RSS MiB
    Fixed rate, quiet ScalWS shared0.571.562.122.3002.4119
    ScalWS fountain0.822.803.665.1002.1106
    ingress-nginx default0.712.243.098.6003.3377
    ingress-nginx tuned0.692.052.803.0003.8376
    Traefik0.763.206.066.5005.8208
    Backend churn ScalWS shared0.571.703.483.5002.5128
    ScalWS fountain1.106.0812.2632.0102.1118
    ingress-nginx default0.884647.934948.885001.65,9797683.4381
    ingress-nginx tuned0.864668.454946.765001.58,5127683.9382
    Traefik1.069708.009941.459965.313,69005.6333
    Ingress churn ScalWS shared0.581.602.592.0002.5133
    ScalWS fountain1.0011.1513.1717.7002.2126
    ingress-nginx default0.725.0229.7734.982603.4379
    ingress-nginx tuned0.717.8533.1534.298104.0380
    Traefik0.763.486.327.4005.9298

    “Over 1 s” counts successful requests whose own round trip took more than a second. In backend churn the p99 of ingress-nginx and Traefik is set by requests that wait for a connect timeout to a pod that is gone (ingress-nginx) or hit the load generator's request timeout (Traefik); with a fixed number of connections a blocked one holds up its queue, so the failed and slow counts are the robust figures.

    CPU per request and memory

    Controller and backend share the same 16 CPUs, so requests/s measures both together. CPU per request of the controller pod isolates the controller: tuned ingress-nginx uses 1.51× the CPU of ScalWS per request in the HTTP/1.1 small-response workload and 2.38× with a new TLS connection per request. At 19,000 req/s ScalWS used 2.4 cores, tuned ingress-nginx 3.8 and Traefik 5.8.

    Controller CPU per request

    • HTTP/1.1, small responseScalWS168.5 µsScalWS fountain168.4 µsingress-nginx210.0 µsingress-nginx tuned254.4 µsTraefik330.7 µs
    • HTTP/1.1, 100 KiB responseScalWS341.8 µsScalWS fountain349.9 µsingress-nginx1127.4 µsingress-nginx tuned484.1 µsTraefik640.8 µs
    • TLS, HTTP/1.1ScalWS183.1 µsScalWS fountain180.8 µsingress-nginx241.5 µsingress-nginx tuned285.7 µsTraefik357.8 µs
    • TLS, HTTP/2ScalWS188.8 µsScalWS fountain196.4 µsingress-nginx274.2 µsingress-nginx tuned317.3 µsTraefik482.6 µs
    • TLS, new connection per requestScalWS1107.3 µsScalWS fountain1120.4 µsingress-nginx2622.5 µsingress-nginx tuned2635.1 µsTraefik2521.2 µs

    CPU time of the controller pod (all its containers) per request, from the pod cgroup. Lower is better.

    Last column: tuned ingress-nginx divided by ScalWS (shared), as published; above 1.00× ScalWS uses less.
    FigureScalWS sharedScalWS fountainingress-nginx defaultingress-nginx tunedTraefiktuned / ScalWS
    CPU µs/req, HTTP/1.1, small response168.5168.4210.0254.4330.71.51×
    CPU µs/req, HTTP/1.1, 100 KiB response341.8349.91127.4484.1640.81.42×
    CPU µs/req, TLS, HTTP/1.1183.1180.8241.5285.7357.81.56×
    CPU µs/req, TLS, HTTP/2188.8196.4274.2317.3482.61.68×
    CPU µs/req, TLS, new connection per request1107.31120.42622.52635.12521.22.38×
    CPU cores at 19,000 req/s2.42.13.33.85.81.58×
    Peak RSS MiB, HTTP/1.1, small response1301133823813502.93×
    Peak RSS MiB, HTTP/1.1, 100 KiB response3272613803803621.16×

    Peak resident memory is that of the whole pod (for ScalWS: scalwsd plus the scalws-ingress sidecar). ScalWS memory grows with the requests in flight and the size of proxied responses: 327 MiB with 100 KiB responses against 130 MiB with small ones. The chart sets no memory limit by default for that reason.

    Accept policy: shared, not fountain

    The opt-in fountain policy (ADR-0044) fills event loops in order to save CPU at moderate load. It does that here (0.88× of the CPU at 19,000 req/s), but it packs the keep-alive connections onto the first loops and keeps them there, which costs a proxy latency and throughput. For ingress mode keep the default, shared.

    fountain divided by shared, as published. Requests/s: below 1.00× fountain is slower. Latency and CPU: above 1.00× fountain is higher.
    FigureScalWS sharedScalWS fountainfountain / shared
    h1-small req/s64.3k59.8k0.93×
    h1-100k req/s29.0k27.3k0.94×
    tls-h1 req/s61.2k57.6k0.94×
    tls-h2 req/s62.4k57.4k0.92×
    tls-newconn req/s13.5k13.0k0.96×
    latency p99 clean ms1.562.801.79×
    churn p99 clean ms1.706.083.58×
    config p99 clean ms1.6011.156.97×
    latency CPU cores2.42.10.88×

    What ingress-nginx was given

    Every controller ran with 16 workers, one replica, no CPU limit and access logs off. ingress-nginx ran twice: with its chart defaults and tuned with the settings an operator would reach for. The tuning doubles its 100 KiB throughput but costs it in the other scenarios, so each scenario is also compared with the best of the three ingress-nginx configurations measured.

    Not changed: connect timeout, retries, TLS settings, worker CPU affinity, worker connections, reuse-port.
    KeyDefaultTunedWhy
    proxy-buffer-size4k64kwith proxy-buffering: off (the default, kept: turning it on was slower in round 1) nginx relays a response in reads and writes of this size
    keep-alive-requests100010000client side: HTTP/1.1 connections are closed and HTTP/2 connections get GOAWAY after this many requests
    upstream-keepalive-requests100010000fewer upstream reconnects
    upstream-keepalive-connections320320idle upstream connections per worker; the default, stated explicitly
    worker-processesauto16as round 1
    Requests/s, medians; the fastest ingress-nginx configuration of each row in bold.
    Scenarioingress-nginx defaultskeep-alive onlytunedScalWS sharedvs best ingress-nginx
    HTTP/1.1, small response58.2k58.6k49.9k64.3k1.10×
    HTTP/1.1, 100 KiB response12.3k12.5k24.3k29.0k1.19×
    TLS, HTTP/1.151.4k52.4k45.1k61.2k1.17×
    TLS, HTTP/244.4k45.9k40.2k62.4k1.36×
    TLS, new connection per request5.53kn/a5.49k13.5k2.44×

    “Keep-alive only” is the tuned configuration without the larger proxy buffer; it was not run with a new connection per request. The ratio is ScalWS (shared) against the fastest ingress-nginx configuration of the row, as published.

    Limits and where ScalWS loses

    ScalWS fountain loses to tuned ingress-nginx

    With the opt-in fountain policy ScalWS still has more requests/s, less CPU per request and less memory than tuned ingress-nginx in every scenario, but it is behind on latency in 10 figures:

    Milliseconds, lower is better.
    Scenario, metricScalWS fountainingress-nginx tunedScalWS shared
    HTTP/1.1, small response, p9922.0019.9616.74
    HTTP/1.1, 100 KiB response, p9957.8336.1232.64
    TLS, HTTP/1.1, p9922.2821.9317.80
    TLS, HTTP/2, p9923.0021.2615.18
    Fixed rate, quiet, p500.820.690.57
    Fixed rate, quiet, p992.802.051.56
    Fixed rate, quiet, p99.93.662.802.12
    Backend churn, p501.100.860.57
    Ingress churn, p501.000.710.58
    Ingress churn, p9911.157.851.60

    Read the results with these in mind

    • ScalWS shared is not behind ingress-nginx (either configuration) or Traefik in any of the 135 figures compared: requests/s, p50, p99, errors, CPU and peak memory, at full load and at the fixed rate. The narrowest margins are against the best ingress-nginx setting: 1.10× ingress-nginx in the tested HTTP/1.1 small-response workload (keep-alive settings only) and 1.19× in the tested 100 KiB workload (tuned).
    • With 100 KiB responses the lead over ingress-nginx defaults (2.35×) is mostly an ingress-nginx default: its proxy buffer of 4k. With 64k the lead is 1.19×.
    • One node on one machine. Controller, backend and the load generator share one host; the load generator runs on the other CPU socket. No network card, no kube-proxy or Service path, one controller replica. Multi-node behaviour and real NIC interrupts are not covered.
    • Controller and backend share the same 16 CPUs, so a controller that needs less CPU leaves more to the backend. CPU per request is the cleaner efficiency figure.
    • In the scenario “TLS, new connection per request” the load generator used 14.2 of its 16 CPUs against ScalWS, close to its limit: treat the ScalWS figure there as a lower bound.
    • ingress-nginx is retired upstream (2026). It is still the most widely deployed controller and the one ScalWS's annotation support targets.
    • “Tuned” is one operator's reasonable global configuration, not a search for the best ingress-nginx configuration per scenario; ScalWS was not tuned beyond the accept policy, and Traefik ran with its chart defaults (no retries).
    • ingress-nginx's HTTP/2 errors (15,560 with defaults, 1,092 tuned, over 3 runs) come from its keep-alive request limit closing connections with GOAWAY. Browsers retry such streams; the load generator counts them as failed.
    • Self-signed RSA 2048 certificate, TLS 1.3 in practice. The fixed-rate scenarios ran once per configuration (3 runs each).

    Methodology

    Host
    Bare metal: 2x Xeon E5-2650 v2 (8 cores, 16 threads each, no AVX2), 121 GB, Ubuntu 26.04, kernel 7.0.0-38-generic; performance governor, turbo off
    Cluster
    k3s v1.36.5+k3s1, one node (containerd 2.3.4-k3s1.36, flannel CNI, Traefik and ServiceLB disabled)
    CPU split
    k3s, containerd and every pod (controller and backend) on NUMA node 0, CPUs 0-7,16-23; the load generator (oha 1.16.0) on node 1, CPUs 8-15,24-31
    Traffic path
    oha connects from the host straight to the controller pod IP (host, cni0 bridge, veth): no Service, kube-proxy or NodePort, the same path for every controller
    Controllers
    One installed at a time with helm, then removed with its namespace. 16 workers each (ingress-nginx worker processes, Traefik GOMAXPROCS, ScalWS event loops), one replica, no CPU limit, access logs off
    Backend
    nginx:1.27.5-alpine, 4 replicas × 2 workers; a 13-byte and a 100 KiB response; the same Ingress objects for every controller, only ingressClassName differs
    Throughput
    10 s warm-up, then 3 runs of 30 s with 256 requests in flight (HTTP/2: 64 connections × 4 streams); medians
    Fixed rate
    3 runs of 60 s at 19,000 req/s with latency correction (no coordinated omission); “clean” percentiles leave out the first second and the seconds in which oha itself stalled
    Versions
    ScalWS 1.0.0-rc.1 at 4823999 (chart deploy/helm/scalws, image bench-4823999); ingress-nginx v1.15.1 (chart 4.15.1); Traefik v3.7.14 (chart 41.7.1)
    Spread
    Largest run-to-run spread (max minus min over the median): 5.9 % for ScalWS shared and the other controllers, 6.3 % for fountain. A second pass in reverse order repeated the throughput medians within 1.9 % (3.3 % for fountain).

    Sources

    History

    Round 1 (ScalWS 115f5f2) found three problems in ScalWS: the chart's default memory limit got scalwsd OOM-killed with 100 KiB responses, requests on a reused connection to a pod that was shutting down were not retried, and the shared listener's backlog was too short for a burst of new connections. They were fixed (d14639f), and round 2 (4823999) also lets the generated pools retry twice and only lets the namespace that owns a host add paths to it.

    Failed requests of ScalWS in backend churn, 3 runs each. Round 1 for comparison: ingress-nginx 8,357, Traefik 14,659.
    RunScalWS commitfailed requests
    Round 1115f5f2126
    After fixesd14639f15
    Round 248239990