Benchmarks

Benchmark harness

Reproducible, comparative HTTP benchmarks (handoff §14).

What is measured

Runtime scenarios php, node, python put each server in front of identical application servers (see docs/benchmarks/2026-10-05-m2-runtimes.md).

For every scenario and server the harness records requests/s, latency p50/p95/p99 (from oha), success rate, server CPU seconds consumed during the measured window (/proc/<pid>/stat, process tree), and peak RSS (VmHWM, process tree).

Method

• One container, three CPU sets: server (SERVER_CPUS, default 0-3), upstream (UPSTREAM_CPUS, 4-5), load generator (LOAD_CPUS, 6-11), enforced with taskset. Roles never share CPUs. Each server gets as many workers/threads as it has CPUs (--worker-threads, worker_processes, GOMAXPROCS).
• Same document root, same certificate (P-256), access logs off everywhere, upstream keep-alive on everywhere.
• Load generator: oha 1.16.0, fixed concurrency (CONNECTIONS, default 64), fixed duration (DURATION, default 15 s) after a warm-up (WARMUP, default 3 s).
• Security and correctness features stay on for scalws (limits, timeouts, path confinement); nothing is disabled to improve numbers.
• environment.txt in each result directory records kernel, CPU model, CPU sets, versions of every server and tool, and the virtualisation type.

Running

docker build -f bench/Dockerfile -t scalws-bench .
# document root on a real filesystem (volume), not the container's overlayfs
# SYS_NICE lets the harness move application processes (incl. PHP-FPM workers running
# as nobody) to UPSTREAM_CPUS; without it runtime scenarios are not comparable.
docker run --rm --cpuset-cpus 0-11 --cap-add SYS_NICE -v scalws-bench-www:/bench/www -v "$PWD/bench/results:/results" scalws-bench
# options
docker run --rm -e SERVERS="scalws nginx" -e DURATION=30 -e CONNECTIONS=256 \
  -v "$PWD/bench/results:/results" scalws-bench

Results land in bench/results/<UTC timestamp>/ (summary.csv, summary.txt, raw oha JSON per run, environment.txt).

Repeated runs, A/B builds, regression check

One run decides nothing on a shared or virtual machine (on the development VM the same build varies by ±10–20 % between runs). bench/multi.sh repeats run.sh and aggregates:

MODE=quick   bench/multi.sh   # 3 s warm-up, 10 s measurement, 3 runs (iteration)
MODE=release bench/multi.sh   # 10 s warm-up, 30 s measurement, 5 runs (reported numbers)
# two scalws builds against each other (and nginx), order rotating from run to run
VARIANTS="old=/opt/scalws-a new=/opt/scalws-b" SERVERS="scalws nginx" SCENARIOS=node MODE=release bench/multi.sh
# regression rules against a stored baseline (exit status 1 on failure)
bench/compare.py BASELINE/aggregate.csv CURRENT/aggregate.csv

aggregate.csv holds per server and scenario the median, mean, standard deviation, coefficient of variation and min/max of requests/s, median p50/p95/p99, server CPU per request (cpu_us_per_req, much steadier than requests/s on a VM), application CPU (PHP-FPM/Node/uvicorn processes) and peak RSS. compare.py flags requests/s down > 5 % (failure on cache-hit and tls-h2-*), p99 up > 15 % and peak RSS up > 15 %.

Every process of a run gets the same open-file limit (NOFILE, default 65535); the environment record adds ulimit -n and each server’s effective limit, CPU governor, THP, NUMA nodes, loopback MTU, kTLS module and allocator.

Stage timings

A scalwsd built with --features perf-instrumentation times the request stages (HTTP service, pipeline checks, host/route, .htaccess, handler, finish; proxy prepare, checkout, connect, upstream head; FastCGI queue wait, params, connect, upstream head; static path checks, open-file-cache lookup, response headers) and counts pool events (connections created/reused, idle connections on another event loop). With SCALWS_PERF_REPORT=/tmp/scalws-perf.json it writes them once per second, and run.sh stores <label>-<scenario>.perf.txt per scenario (bench/perfdiff.py: per-request averages, p50/p99 bounds, counters, reuse ratios). Production builds compile the instrumentation out entirely.

Interpreting results

• Numbers are only comparable within one result directory (same machine, same run).
• Docker Desktop (Windows/macOS) runs inside a VM with virtualised networking and shared host CPUs; treat them as smoke tests, not performance claims. Publishable numbers come from bare-metal Linux with frequency scaling pinned (cpupower frequency-set -g performance) and SMT siblings kept within one role.
• nginx and Caddy are the Debian bookworm packages (versions in environment.txt); for a fair publication pin current upstream releases. OpenLiteSpeed (SERVERS=openlitespeed, installed from its release tarball under LSWS, default /usr/local/lsws) uses bench/configs/openlitespeed; it has no fixed-response directive, so respond is skipped for it. LiteSpeed Enterprise is not included (licence).
• Application-bound scenarios (php, node, python) leave the server cores partly idle; firmware power management may then lower the clock regardless of the governor (seen on HP ProLiant: 1.2 GHz instead of 2.6 GHz in some runs). Check /proc/cpuinfo MHz during a run and set the BIOS power profile to maximum performance.
• Profile before optimising (handoff §14): scripts/profile.sh --docker <scenario> builds with frame pointers, runs the scenario and writes perf reports to profile/<scenario>/.

Acceptance gate 6: soak and overload

Two pass/fail tests in the same image (handoff §16: bounded memory under sustained load; overload produces rejection, not unbounded queues):

# Soak: every serving path at once (static, 1 MiB files, proxy with and without
# keep-alive, cache hits, TLS+HTTP/2, PHP-FPM, Node.js, Python) for 30 minutes, sampling
# scalwsd RSS, file descriptors and threads every 10 s.
docker run --rm --cpuset-cpus 0-11 --cap-add SYS_NICE --ulimit nofile=65536 \
  -e SOAK_DURATION=1800 -v scalws-bench-www:/bench/www -v "$PWD/bench/results:/results" \
  --entrypoint /bench/soak.sh scalws-bench

# Overload: 2000 clients against a tenant limited to 64 concurrent requests in front of a
# 50 ms application, while a probe loads another tenant.
docker run --rm --cpuset-cpus 0-11 --ulimit nofile=65536 \
  -v scalws-bench-www:/bench/www -v "$PWD/bench/results:/results" \
  --entrypoint /bench/overload.sh scalws-bench

Requests still in flight when oha’s timer ends are reported by oha as “aborted due to deadline”; both scripts exclude them from errors. Results: samples.csv / rss.csv, load.txt, raw oha JSON and verdict.txt; the exit status is 0 only on PASS.