Internals: benchmark reports

Performance gates for 1.0.0 on the Xeon host, 2026-11-06 to 08

The rule: every scenario at parity or faster than the version it replaces, median of at least three rounds (method of 2026-10-27-el8-build.md). Three gates:

1. the EL8 release build of the current tree against the published 1.0.0~rc.1-11;
2. the in-process sandbox (docs/design/2026-10-28-sandbox-design.md), same binary, off against on;
3. the static musl build (Alpine) against the glibc release build.

Gate 1 passes. Gates 2 and 3 failed as first built. Gate 3 passes with the musl memcpy fix, and again with both builds profile-guided. Gate 2 cannot pass as defined (the sandbox’s cost is the kernel’s, below); the user’s ruling keeps the sandbox as it is and asks instead that a new install, sandbox on, be at least as fast as rc.1-11 in every scenario. With profile-guided release builds (ADR-0047, now wired into the release scripts) it is, by 0.0 % to 45 % (“What a new install gets”, at the end).

Method

Machine, pinning and harness as in 2026-10-27-el8-build.md: 2x Xeon E5-2650 v2 (no AVX2), performance governor, turbo off, kernel 7.0.0-38 (Ubuntu 26.04), Landlock in the LSM list. Server on CPUs 0-3 (socket 0, 4 workers), upstream and applications on 4-7, oha 1.16.0 on 8-15 and 24-31 (socket 1), 64 connections. bench/multi.sh MODE=release: 10 s warm-up, 30 s measurement, 5 runs per round with rotating variant order, three rounds. No VM or container ran; the 1-minute load was under 2 before each round (the drivers waited for it). Every measurement answered 2xx only; no run was excluded.

Gate value, per scenario: the median of the three per-round ratios (new / old requests per second) is at least 1.000, or below it by less than the round spread (the larger CV of the two variants in a round, median over the rounds). p99 latency and peak RSS may not rise by more than 15 % (bench/compare.py). Tables come from bench/musl_gate.py --base <old> --new <new>; cpu_us is server CPU per request.

Binaries, all scalwsd, rustc 1.99.1, workspace release profile (thin LTO, codegen-units = 1), mimalloc, no PGO, no http3 feature unless noted. Packaged builds run through a wrapper that drops http3: true from the bench configuration.

Gate 1: EL8 release build against rc.1-11 (PASS)

Rounds 2026-11-06 21:00 to 2026-11-07 00:12, results/xeon-gate1-{1,2,3}.

Worst: python 0.998, inside its spread (rounds in both directions). static-1m is bimodal on this host (rounds sit at about 9 580 or 10 300 req/s for either build), which is the 3.5 % spread.

Gate 2: sandbox off against on (FAIL as built)

Same el8 binary. bench/sandbox-variants.sh makes the two variants. As written for task 8f it ran scalwsd as root, where the sandbox is never applied (fix round 3), so it would have compared the same program twice; it now runs both variants as an unprivileged user, and the on processes were checked every 20 s during the whole gate (Seccomp: 2 in 886 samples, 0 for every off sample; one sample caught an on process before its filter). PHP goes through an external pool as under the controller, so php is in the gate. Standalone mode (reads not limited) through bench/multi.sh; instance mode (an instance id, so reads are limited as under the controller) through bench/instances.sh, 3 instances, multi (one CPU each, reuseport) and single (one process, 3 loops), 5 runs of 5 s + 15 s per round, order rotating.

Standalone, OPEN_FILE_CACHE=off (every static request opens its file), results/xeon-gate2-sa-{1,2,3}:

Standalone, OPEN_FILE_CACHE=check (the default cache), static-small: 0.982 / 0.982 / 0.974, median 0.982, spread 1.8: FAIL by the margin. HTTP/3 (h3 build), h3-static-small, cache off: 0.968 / 0.968 / 0.981, median 0.968, spread 1.0: FAIL.

Instance mode, results/xeon-gate2-inst-{1,2,3}:

(the instance configuration has the open-file cache on, so few opens there.)

Cause

perf record of CPUs 0-3 at full load, on against off (static-small with the cache off, and respond), and the same binary built with switches that leave out one part:

• seccomp, every system call: about 0.7 % of the server’s CPU. __secure_computing, __seccomp_filter, populate_seccomp_data and seccomp_run_filters appear only with the filter. The kernel’s action cache works: no BPF program runs (no bpf_prog symbol), the hot calls are not argument-checked (CORE_ARG_CHECKED: clone, socket, fallocate, prlimit64, personality, bpf), two stacked filters cost nothing extra (the kernel checks only the newest filter’s cache bitmap, which already includes the older one’s), SPEC_ALLOW holds (Speculation_Store_Bypass: thread vulnerable with and without the filter, the host’s prctl default), and LOG applies to refusals only. What remains is the fixed entry work a filtered task pays on each call; the server makes 2 calls per request on respond (recvfrom, writev), 3 on cached static, 6 on uncached static.
• Landlock, every file open: about 0.6 % more on static-small with the cache off. Standalone mode handles no read right, but TRUNCATE is handled and never granted, and the kernel checks the optional TRUNCATE right on every open of a regular file, so each open walks the path to the root (is_access_to_paths_allowed, find_rule, landlock_init_layer_masks, lockref on each parent). With TRUNCATE not handled the walk disappears (sandbox symbols 1.59 % of samples on, 1.02 % without TRUNCATE, 0.71 % seccomp alone, 0.93 % Landlock alone, 0.03 % off).

Neither is the generator’s fault, and neither can be removed without dropping a protection: no TRUNCATE means open(O_RDONLY | O_TRUNC) could truncate any file the service user may write outside the write fence (truncate/ftruncate are refused by seccomp, but openat2 cannot be argument-checked there), and no seccomp means no filter. Pending a ruling.

The user’s ruling (2026-11-07): TRUNCATE stays handled and the seccomp filter stays; the sandbox’s cost is to be made up elsewhere, so that a new install (sandbox on) is no slower than rc.1-11. See the last section.

Gate 3: musl build against the glibc release build (FAIL as built)

bench/musl-gate.sh, rounds 2026-11-07 07:07 to 10:20, results/xeon-gate3-{1,2,3}.

Cause and fix

Profiles of both builds (unstripped) on respond, cache-hit and php: musl’s memcpy takes 2.0 to 3.3 % of the server’s CPU where glibc’s __memmove_sse2_unaligned_erms takes 1.9 to 2.2 %; memset and memcmp add a little. musl’s x86_64 memcpy copies with rep movsq, slow for the 16 to 512 byte copies of header buffers, response encoding and FastCGI records. musl’s malloc does not appear: the request path allocates through mimalloc already, so lever 1 of the design (mimalloc for the C allocations) has nothing to act on and was skipped; lever 2 applies.

cmd/scalwsd/musl/mem.c (built by cmd/scalwsd/build.rs for the musl target only, linked as a whole archive): memcpy, memmove, memset, memcmp and bcmp after glibc’s SSE2 strategy (overlapping 16-byte loads and stores up to 128 bytes, a 64-byte loop with aligned stores, rep movsb/rep stosb from 2 KiB). Checked against byte loops in 4.7 million cases (every size 0 to 1100 and larger ones, every alignment, every memmove overlap distance, a difference at every position for the compares). With it the copy functions take 2.1 % on respond (glibc 1.9 %) and 2.0 % on php (glibc 2.2 %), memset 1.1 % (glibc 1.0 %).

Gate 3 rerun with the fix (PASS)

The four failing scenarios and static-small as a control, rhel-final glibc (EL8 image) against rhel-final musl with mem.c, both without PGO, rounds 2026-11-07 13:12 to 14:59, results/xeon-gate3b-{1,2,3}. Another task’s container (a preparation container, pinned to CPUs 0-3 and 16-23) ran dnf from about 13:12 to 13:18, during the first run of round 1; that round’s ratios match rounds 2 and 3 and were kept.

respond stays 1.1 % below in all three rounds, inside its spread. With both builds profile-guided (below) musl is ahead there.

Making up the sandbox’s cost

System calls per request

perf stat on the sandboxed server (rhel-final, sandbox wrapper configuration), every syscall tracepoint, one scenario at a time:

No clock_gettime or getrandom reaches the kernel per request, epoll_ctl is not re-armed per request, and responses go out in one writev. The static paths make exactly the calls their semantics need: the cache-off configuration opens every file by definition, and the default cache checks identity with one statx per hit (valid unset: changes are served at once). Two findings:

• TCP_NODELAY was set on every accepted socket. Linux copies the listener’s option to accepted sockets, so it is now set once on the listener (commit 738de85, with a test for shared and per-loop listeners): one call less per connection.
• FastCGI to external pools uses one connection per request. Managed pools already keep connections (FCGI_KEEP_CONN, bounded by the pool’s worker count); external pools do not, so that idle connections cannot starve other clients of a pool scalwsd does not size. Under the controller (ADR-0036) and in the sandboxed standalone setup PHP is always external, so this is where the sandbox meets PHP. Keeping connections to an external pool needs its worker count (a setting, or the controller’s knowledge of the pool): a behaviour change, left as a proposal.

Profile-guided release builds

ADR-0047 (accepted 2026-10-10) decided that release binaries of scalwsd are built with tools/pgo-build.sh, but scripts/release-build.sh still built them plainly, as did rc.1-11. An estimate on the host toolchain (one round of 3 runs, sandbox on against rc.1-11) gave 1.21 static-small, 1.36 respond, 1.42 proxy, 1.46 cache-hit, 1.35 tls-h2, 1.11 php, against 0.96 to 1.006 for the plain build. Both release scripts now run tools/pgo-build.sh in their build container (commits 7886446, e9cfadd); the images gain llvm-tools, oha 1.16.0, openssl and PHP-FPM for the training load. Two compatibility findings on the way, outside this work: EL8’s default PHP 7.2 refuses the decorate_workers_output line scalwsd writes into a managed pool (exit 78), so the image uses the php:8.2 stream; and on Alpine scalwsd looks for php-fpm and php-fpm8.x but the binary is php-fpm83, and it needs util-linux setpriv (the apk depends on it), so the image links php-fpm and installs setpriv. Both trainings ran without an error line, PHP included.

Binaries: pgo-el8 = scripts/release-build.sh (SCALWS_RELEASE=12preP2), pgo-musl = scripts/release-build-apk.sh --binaries-only, rhel-final with the commits above, rustc 1.99.1, no http3 feature.

Gate 3 with both builds profile-guided (PASS)

Rounds 2026-11-07 23:05 to 2026-11-08 01:14, results/xeon-gate3p-{1,2,3}.

What a new install gets: sandbox on, PGO, against rc.1-11 (PASS)

The user’s rule as it applies to a new install: pgo-el8 with the sandbox on against rc.1-11 (which has no sandbox), both through the sandbox wrapper’s configuration (run as an unprivileged user, PHP through an external pool, open-file cache off), so the comparison differs only in the build. Rounds 2026-11-07 19:32 to 22:20, results/xeon-rcon-{1,2,3}; every on server sampled showed Seccomp: 2 (253 samples), every rc.1-11 one 0 (250).

static-1m’s first round (0.964) is the host’s bimodal 1 MiB result (rounds sit near 9 600 or 10 300 req/s for any build); the other two rounds are above 1.

Node.js and Python start processes, so they are never sandboxed; for them the release build is compared as is: pgo-el8 against rc.1-11, standard bench configuration, results/xeon-rcapps2-{1,2,3} (2026-11-08 01:16 to 01:58).

(Requests per second there are bound by the applications; scalwsd’s own CPU per request is 18 to 23 % lower.)

Incident: a duplicate runner (2026-11-07)

The driver for these runs was queued twice (a waiter thought gone was still alive), so two copies ran: instance 1 from 16:41:40 and instance 2 from 17:00:15. Each held the lock only around its own run and waited for a quiet host (1-minute load under 2) before the next.

• rcon: instance 1 measured rounds 1 to 3 from 16:47 to 19:30 (its first round overlapped the musl build until 17:00; its last write, the aggregate, is at 19:30:34); instance 2 then measured rounds 1 to 3 again from 19:32:18 to 22:20:29 into the same directories, while instance 1 waited in its quiet check. Every summary.csv of rcon (30 files, 8 scenarios each) is dated 19:37:53 to 22:20:29, so the table above is instance 2’s run alone; its logs show no bind failure, and instance 1 started its next run (apps) only at 22:22.
• apps (node, python): instance 1 measured from 22:22; instance 2 started the same run at the same time, its servers failed on the taken ports, and its end removed the lock. That run (xeon-rcapps-*) was discarded and repeated alone (xeon-rcapps2-*, above) after the duplicate was stopped by PID at 23:02.
• gate 3 with PGO ran from 23:05 with the duplicate gone.

Still to run

The HTTP/3 gates, after rhel-h3 (HTTP/3 in the packages) is merged:

1. http3 feature against no feature, TCP scenarios. Two release builds of the same tree (SCALWS_FEATURES=scalwsd/http3 and empty, both PGO), bench/multi.sh MODE=release, three rounds, scenarios static-small, static-1m, respond, proxy, cache-hit, tls-h2-static-small, php, node, python, both through the wrapper that drops http3: true (the listeners stay TCP only, so only the feature’s presence differs). Gate: bench/musl_gate.py --base noh3 --new h3, same rule.
2. h3 with the sandbox off and on. The http3 build, bench/sandbox-variants.sh with SBX_HTTP3=1, scenario h3-static-small with the open-file cache off and with OPEN_FILE_CACHE=check, three rounds, bench/musl_gate.py --base off --new on. rc.1-11 has no HTTP/3, so for h3 the comparison is within the build (expect the TCP result: about 3 % below with the cache off, from seccomp and the Landlock walk); the http3 PGO build against the plain build of the first gate run (h3 0.968 of off there) shows whether the new install is ahead.