ScalWSScalWS

    Get started
    ScalWSScalWS
    Get started

    Internals: benchmark reports

    Kubernetes ingress benchmark harness

    Compares the ScalWS ingress controller (chart deploy/helm/scalws in ingress mode) with ingress-nginx and Traefik on one k3s node. Report (round 2 on top, round 1 below): docs/benchmarks/2026-10-14-xeon-k8s-ingress.md.

    Requirements

    • A Linux host with two NUMA nodes (or two CPU sets that share nothing): node 0 runs k3s and every pod, node 1 the load generator. The scripts assume CPUs 0-7,16-23 for node 0 (NODE0_CPUS changes it) and 8-15,24-31 for oha. Run as root from the repository root, on a host that serves nothing else; k3s is restarted by the setup.
    • k3s with Traefik and ServiceLB disabled (--disable traefik --disable servicelb), so only the controller under test handles Ingresses; kubectl and helm 3.
    • oha 1.16 or later, jq, curl, openssl, python3, perf (only for the perf scenario) and Docker with buildx to build the ScalWS image from the checked-out tree (packaging/container/Dockerfile). The image is imported into k3s’s containerd; no registry is involved.
    • About 3 GB free for the oha result files of the fixed-rate scenarios (reduced to per-second summaries by aggregate.py).

    Files

    FilePurpose
    setup-host.shOne-time host preparation: k3s and all pods on NUMA node 0, ScalWS image import, client sysctls, helm repositories.
    run.sh CONTROLLERInstalls one controller (scalws, ingress-nginx, traefik, or a variant such as scalws-fountain), runs the scenarios, uninstalls it.
    run-all.shAll controllers, throughput then fixed rate, then the aggregate.
    aggregate.py DIRMedians and spreads per controller and scenario: DIR/aggregate.csv, DIR/aggregate.md.
    manifests/backend.yamlNamespace bench, the nginx backend (4 replicas), its Service.
    manifests/ingress.yamlThe measured Ingresses (bench.local plain, tls.bench.local TLS); only ingressClassName differs per controller.
    values/*.yamlHelm values per controller; each file says what differs from the chart defaults and why.
    values/<controller>-<variant>.yamlA variant: applied on top of the controller’s values, results under $OUT/<controller>-<variant>/. scalws-fountain (accept policy fountain), ingress-nginx-tuned (64k proxy buffer, keep-alive limits raised), ingress-nginx-keepalive (only the keep-alive limits).

    Method

    • One k3s node. k3s, containerd and every pod (controller and backend) run on NUMA node 0 (CPUs 0-7,16-23): a CPUAffinity drop-in for k3s.service plus AllowedCPUs on the kubelet’s kubepods.slice. oha runs on node 1 (taskset -c 8-15,24-31). One controller is installed at a time; the namespace is deleted before the next.
    • Every controller gets 16 workers (ingress-nginx worker-processes: 16, Traefik GOMAXPROCS=16, ScalWS server.event_loops: 16), no CPU limit, one replica, access logs off. environment.txt records the processes, thread counts and CPU affinity seen.
    • Load goes to the controller pod IP from the host (oha --connect-to), so no Service, kube-proxy or NodePort path is involved. Same Host headers and the same Ingress objects.
    • Backend: nginx:1.27.5-alpine, 4 replicas x 2 workers, /small (13 bytes), /100k (102 400 bytes), upstream keep-alive for 100 000 requests per connection. Its stub_status counts the upstream connections each controller opens.
    • Throughput scenarios: warm-up (WARMUP, 10 s), then RUNS (3) runs of DURATION (30 s) at 256 in flight. Fixed-rate scenarios: RUNS runs of FIXED_DURATION (60 s) at RATE req/s with --latency-correction; every successful request goes to an SQLite file for the worst one-second p99.
    • Per run: controller pod CPU (cgroup cpu.stat), peak RSS of the pod’s processes (sampled every 2 s), oha CPU, backend connections opened.
    ScenarioRequest
    h1-smallHTTP/1.1 keep-alive, GET /small, 256 connections
    h1-100ksame, GET /100k
    tls-h1HTTPS (TLS from the Ingress Secret, RSA 2048), HTTP/1.1, 256 connections
    tls-h2HTTPS, HTTP/2, 64 connections x 4 streams
    tls-newconnHTTPS, a new connection per request, 256 in flight
    latencyfixed rate, HTTP/1.1 /small
    churnas latency, the backend is scaled 4 -> 8 -> 2 -> 4 at 10 s, 25 s and 40 s
    configas latency, every 2 s an unrelated Ingress is added or removed; at the 15th step a path is added to the measured host
    perfprofile, not a measurement: the load of each of PERF_SCENARIOS (throughput scenarios or latency at RATE), perf record -g on the data-plane process for PERF_SECONDS (20) after the warm-up, CPU per thread from /proc; perf-<scenario>.{txt,data}

    Running

    bench/kubernetes/setup-host.sh          # once (restarts k3s)
nohup bench/kubernetes/run-all.sh > run-all.log 2>&1 &

    run-all.sh runs the throughput scenarios for every controller (ORDER, default scalws ingress-nginx traefik), sets RATE to half of the slowest controller’s h1-small median (rounded down to 1000 req/s) unless given, runs the fixed-rate scenarios and writes the aggregate. Single pieces:

    SCENARIOS="h1-small tls-h2" RUNS=5 bench/kubernetes/run.sh ingress-nginx
RATE=19000 SCENARIOS="churn" bench/kubernetes/run.sh traefik
SCENARIOS=probe bench/kubernetes/run.sh scalws      # diagnostics, not a measurement
RATE=19000 SCENARIOS=perf PERF_SCENARIOS="latency h1-small" bench/kubernetes/run.sh scalws-fountain
ORDER="scalws ingress-nginx-tuned scalws-fountain ingress-nginx traefik" bench/kubernetes/run-all.sh
bench/kubernetes/aggregate.py "$OUT"

    Results go to $OUT (default: bench-results/k8s-ingress-<date> under root’s home directory), one directory per controller. SCALWS_TAG (default the benchmarked commit’s tag) names the ScalWS image setup-host.sh builds and imports.

    probe records the negotiated TLS 1.3 group, the server bytes per connection of a connection-per-request run (to tell resumed from full handshakes), and for ingress-nginx the HTTP/2 errors and the 100 KiB throughput with a few configuration changes.

    Afterwards stop k3s (systemctl stop k3s; /usr/local/bin/k3s-killall.sh). The k3s CPUAffinity drop-in (/etc/systemd/system/k3s.service.d/bench-cpuaffinity.conf) only affects k3s; the kubepods.slice setting is runtime only and gone after a reboot.

    Reading the results

    aggregate.md in $OUT has one table per scenario group: requests/s as the median of the runs with min and max, p50 and p99, errors (non-2xx plus client errors), controller CPU in cores and microseconds per request, peak RSS of the controller pod, the load generator’s CPU and the backend connections opened per 1000 requests. For the fixed-rate scenarios the “clean” percentiles leave out the first second and the seconds in which oha itself stalls (its result vector grows at about 7, 14, 28 and 55 s at 19 000 req/s). Check before comparing:

    • environment.txt of each controller: all controller and backend processes on node 0, the worker or event loop counts as configured, no restarts.
    • The spread of the runs (min to max): within about 2 % on a quiet host. A run order effect shows when the throughput scenarios are repeated in reverse order (ORDER).
    • oha cores well below 16: if the client is near its limit (the connection-per-request scenario can get there), the figure is a lower bound for the faster controller.