# Performance gates for 1.0.0 on the Xeon host, 2026-11-06 to 08

URL: https://scalws.com/docs/benchmarks/2026-11-06-100-gates/
Source: `docs/benchmarks/2026-11-06-1.0.0-gates.md` in the ScalWS repository

The rule: every scenario at parity or faster than the version it replaces, median of at
least three rounds (method of `2026-10-27-el8-build.md`). Three gates:

1. the EL8 release build of the current tree against the published 1.0.0~rc.1-11;
2. the in-process sandbox (`docs/design/2026-10-28-sandbox-design.md`), same binary, off
   against on;
3. the static musl build (Alpine) against the glibc release build.

Gate 1 passes. Gates 2 and 3 failed as first built. Gate 3 passes with the musl `memcpy`
fix, and again with both builds profile-guided. Gate 2 cannot pass as defined (the
sandbox's cost is the kernel's, below); the user's ruling keeps the sandbox as it is and
asks instead that a new install, sandbox on, be at least as fast as rc.1-11 in every
scenario. With profile-guided release builds (ADR-0047, now wired into the release
scripts) it is, by 0.0 % to 45 % ("What a new install gets", at the end).

## Method

Machine, pinning and harness as in `2026-10-27-el8-build.md`: 2x Xeon E5-2650 v2 (no
AVX2), `performance` governor, turbo off, kernel 7.0.0-38 (Ubuntu 26.04), Landlock in the
LSM list. Server on CPUs 0-3 (socket 0, 4 workers), upstream and applications on 4-7,
`oha` 1.16.0 on 8-15 and 24-31 (socket 1), 64 connections. `bench/multi.sh MODE=release`:
10 s warm-up, 30 s measurement, 5 runs per round with rotating variant order, three
rounds. No VM or container ran; the 1-minute load was under 2 before each round (the
drivers waited for it). Every measurement answered 2xx only; no run was excluded.

Gate value, per scenario: the median of the three per-round ratios (new / old requests
per second) is at least 1.000, or below it by less than the round spread (the larger CV of
the two variants in a round, median over the rounds). p99 latency and peak RSS may not
rise by more than 15 % (`bench/compare.py`). Tables come from `bench/musl_gate.py --base
<old> --new <new>`; `cpu_us` is server CPU per request.

Binaries, all `scalwsd`, rustc 1.99.1, workspace release profile (thin LTO,
`codegen-units = 1`), mimalloc, no PGO, no `http3` feature unless noted. Packaged builds
run through a wrapper that drops `http3: true` from the bench configuration.

| label | from | C library, C compiler |
|---|---|---|
| rc1-11 | published `scalws_1.0.0~rc.1-11_amd64.deb` (`/opt/scalws-rc1-11`) | glibc 2.39, Ubuntu gcc 13.3 |
| el8 | `scripts/release-build.sh`, rhel-family at c89b145, `SCALWS_RELEASE=12preG1` | glibc 2.28 (newest symbol GLIBC_2.28), gcc-toolset-13 |
| musl | `scripts/release-build-apk.sh --binaries-only`, same tree | musl 1.2.6 (alpine:3.24), static-pie |
| h3 | EL8 image, same tree, `--features http3` | as el8 |

## Gate 1: EL8 release build against rc.1-11 (PASS)

Rounds 2026-11-06 21:00 to 2026-11-07 00:12, `results/xeon-gate1-{1,2,3}`.

| scenario | rps el8 / rc1-11 per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 1.002 / 1.011 / 1.009 | 1.009 | 1.0 | 0.901 | 0.991 | 0.977 | PASS |
| static-1m | 1.001 / 1.073 / 0.999 | 1.001 | 3.5 | 1.023 | 0.999 | 0.977 | PASS |
| respond | 1.018 / 1.020 / 1.017 | 1.018 | 1.3 | 0.936 | 0.982 | 1.000 | PASS |
| proxy | 1.011 / 1.006 / 1.006 | 1.006 | 0.9 | 0.937 | 0.994 | 1.000 | PASS |
| cache-hit | 1.007 / 1.013 / 1.012 | 1.012 | 1.0 | 1.000 | 0.988 | 1.000 | PASS |
| tls-h2-static-small | 1.011 / 1.005 / 0.992 | 1.005 | 1.2 | 0.987 | 0.995 | 0.981 | PASS |
| php | 1.018 / 1.003 / 1.034 | 1.018 | 4.2 | 0.986 | 0.985 | 0.981 | PASS |
| node | 1.005 / 0.996 / 1.001 | 1.001 | 1.0 | 0.996 | 0.998 | 0.981 | PASS |
| python | 0.993 / 0.998 / 1.003 | 0.998 | 0.6 | 0.966 | 0.998 | 0.981 | PASS |

Worst: python 0.998, inside its spread (rounds in both directions). static-1m is bimodal
on this host (rounds sit at about 9 580 or 10 300 req/s for either build), which is the
3.5 % spread.

## Gate 2: sandbox off against on (FAIL as built)

Same el8 binary. `bench/sandbox-variants.sh` makes the two variants. As written for task
8f it ran scalwsd as root, where the sandbox is never applied (fix round 3), so it would
have compared the same program twice; it now runs both variants as an unprivileged user,
and the `on` processes were checked every 20 s during the whole gate (`Seccomp: 2` in 886
samples, `0` for every `off` sample; one sample caught an `on` process before its filter).
PHP goes through an external pool as under the controller, so php is in the gate.
Standalone mode (reads not limited) through `bench/multi.sh`; instance mode (an instance
id, so reads are limited as under the controller) through `bench/instances.sh`, 3
instances, `multi` (one CPU each, reuseport) and `single` (one process, 3 loops), 5 runs
of 5 s + 15 s per round, order rotating.

Standalone, `OPEN_FILE_CACHE=off` (every static request opens its file),
`results/xeon-gate2-sa-{1,2,3}`:

| scenario | rps on / off per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 0.975 / 0.962 / 0.960 | 0.962 | 1.1 | 0.945 | 1.040 | 1.000 | FAIL |
| static-1m | 0.998 / 0.995 / 0.995 | 0.995 | 3.2 | 1.022 | 1.005 | 1.026 | PASS |
| respond | 1.019 / 0.979 / 1.007 | 1.007 | 1.6 | 1.013 | 0.994 | 1.000 | PASS |
| proxy | 0.990 / 0.981 / 0.998 | 0.990 | 1.2 | 1.004 | 1.011 | 1.049 | PASS |
| proxy-churn | 0.990 / 0.983 / 0.992 | 0.990 | 0.8 | 1.011 | 1.010 | 0.980 | FAIL |
| cache-hit | 1.006 / 0.980 / 0.998 | 0.998 | 1.0 | 0.950 | 1.002 | 1.000 | PASS |
| tls-h2-static-small | 0.960 / 0.966 / 0.962 | 0.962 | 1.8 | 1.057 | 1.040 | 1.000 | FAIL |
| php | 0.983 / 0.956 / 0.942 | 0.956 | 3.9 | 1.055 | 1.015 | 1.000 | FAIL |

Standalone, `OPEN_FILE_CACHE=check` (the default cache), static-small: 0.982 / 0.982 /
0.974, median 0.982, spread 1.8: FAIL by the margin. HTTP/3 (h3 build), h3-static-small,
cache off: 0.968 / 0.968 / 0.981, median 0.968, spread 1.0: FAIL.

Instance mode, `results/xeon-gate2-inst-{1,2,3}`:

| mode | scenario | rps on / off per round | median | spread % | p99 | cpu_us | RSS | gate |
|---|---|---|---:|---:|---:|---:|---:|---|
| multi | respond | 0.988 / 0.985 / 0.982 | 0.985 | 0.6 | 0.979 | 1.015 | 1.000 | FAIL |
| multi | static-small | 0.987 / 0.983 / 0.997 | 0.987 | 0.8 | 0.879 | 1.013 | 1.000 | FAIL |
| multi | static-1m | 0.996 / 0.997 / 0.999 | 0.997 | 0.2 | 0.971 | 1.003 | 1.000 | FAIL |
| multi | cache-hit | 0.988 / 0.988 / 0.987 | 0.988 | 0.6 | 1.059 | 1.012 | 1.000 | FAIL |
| multi | tls-h2-static-small | 0.977 / 0.988 / 0.995 | 0.988 | 0.7 | 1.038 | 1.012 | 1.000 | FAIL |
| multi | proxy | 0.980 / 0.984 / 0.990 | 0.984 | 0.5 | 1.050 | 1.017 | 0.990 | FAIL |
| multi | php | 0.994 / 1.046 / 1.025 | 1.025 | 4.1 | 0.987 | 1.007 | 1.010 | PASS |
| single | respond | 0.983 / 0.971 / 0.980 | 0.980 | 1.1 | 1.000 | 1.020 | 0.973 | FAIL |
| single | static-small | 0.986 / 0.981 / 0.980 | 0.981 | 1.2 | 0.977 | 1.019 | 1.027 | FAIL |
| single | static-1m | 1.000 / 0.995 / 0.997 | 0.997 | 0.2 | 0.911 | 1.003 | 1.027 | FAIL |
| single | cache-hit | 0.982 / 0.972 / 0.987 | 0.982 | 1.2 | 0.992 | 1.019 | 1.000 | FAIL |
| single | tls-h2-static-small | 0.982 / 0.988 / 0.982 | 0.982 | 0.9 | 1.005 | 1.018 | 1.000 | FAIL |
| single | proxy | 0.979 / 0.979 / 0.983 | 0.979 | 0.7 | 0.970 | 1.021 | 1.024 | FAIL |
| single | php | 0.982 / 0.947 / 0.911 | 0.947 | 6.0 | 1.045 | 1.024 | 1.000 | FAIL |

(the instance configuration has the open-file cache on, so few opens there.)

### Cause

`perf record` of CPUs 0-3 at full load, on against off (static-small with the cache off,
and respond), and the same binary built with switches that leave out one part:

* **seccomp, every system call: about 0.7 % of the server's CPU.** `__secure_computing`,
  `__seccomp_filter`, `populate_seccomp_data` and `seccomp_run_filters` appear only with
  the filter. The kernel's action cache works: no BPF program runs (no `bpf_prog` symbol),
  the hot calls are not argument-checked (`CORE_ARG_CHECKED`: clone, socket, fallocate,
  prlimit64, personality, bpf), two stacked filters cost nothing extra (the kernel checks
  only the newest filter's cache bitmap, which already includes the older one's),
  `SPEC_ALLOW` holds (`Speculation_Store_Bypass: thread vulnerable` with and without the
  filter, the host's prctl default), and `LOG` applies to refusals only. What remains is
  the fixed entry work a filtered task pays on each call; the server makes 2 calls per
  request on respond (recvfrom, writev), 3 on cached static, 6 on uncached static.
* **Landlock, every file open: about 0.6 % more on static-small with the cache off.**
  Standalone mode handles no read right, but `TRUNCATE` is handled and never granted, and
  the kernel checks the optional `TRUNCATE` right on every open of a regular file, so each
  open walks the path to the root (`is_access_to_paths_allowed`, `find_rule`,
  `landlock_init_layer_masks`, `lockref` on each parent). With `TRUNCATE` not handled the
  walk disappears (sandbox symbols 1.59 % of samples on, 1.02 % without `TRUNCATE`, 0.71 %
  seccomp alone, 0.93 % Landlock alone, 0.03 % off).

Neither is the generator's fault, and neither can be removed without dropping a
protection: no `TRUNCATE` means `open(O_RDONLY | O_TRUNC)` could truncate any file the
service user may write outside the write fence (`truncate`/`ftruncate` are refused by
seccomp, but `openat2` cannot be argument-checked there), and no seccomp means no filter.
Pending a ruling.

The user's ruling (2026-11-07): `TRUNCATE` stays handled and the seccomp filter stays; the
sandbox's cost is to be made up elsewhere, so that a new install (sandbox on) is no slower
than rc.1-11. See the last section.

## Gate 3: musl build against the glibc release build (FAIL as built)

`bench/musl-gate.sh`, rounds 2026-11-07 07:07 to 10:20,
`results/xeon-gate3-{1,2,3}`.

| scenario | rps musl / glibc per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 1.003 / 1.000 / 1.002 | 1.002 | 1.5 | 0.945 | 0.998 | 0.927 | PASS |
| static-1m | 0.999 / 0.998 / 0.998 | 0.998 | 3.3 | 0.905 | 1.002 | 0.927 | PASS |
| respond | 0.984 / 0.978 / 0.991 | 0.984 | 1.5 | 1.081 | 1.017 | 0.950 | FAIL |
| proxy | 0.991 / 0.983 / 0.994 | 0.991 | 0.9 | 1.000 | 1.009 | 0.955 | FAIL |
| cache-hit | 0.985 / 0.985 / 0.985 | 0.985 | 1.1 | 1.030 | 1.015 | 0.955 | FAIL |
| tls-h2-static-small | 0.979 / 0.995 / 0.988 | 0.988 | 2.4 | 1.079 | 1.012 | 0.960 | PASS |
| php | 0.956 / 0.964 / 0.912 | 0.956 | 3.3 | 1.041 | 1.043 | 0.960 | FAIL |
| node | 0.991 / 0.991 / 0.994 | 0.991 | 1.2 | 1.011 | 0.998 | 0.960 | PASS |
| python | 0.995 / 1.002 / 1.004 | 1.002 | 0.7 | 0.993 | 1.003 | 0.960 | PASS |

### Cause and fix

Profiles of both builds (unstripped) on respond, cache-hit and php: musl's `memcpy` takes
2.0 to 3.3 % of the server's CPU where glibc's `__memmove_sse2_unaligned_erms` takes 1.9 to
2.2 %; `memset` and `memcmp` add a little. musl's x86_64 `memcpy` copies with `rep movsq`,
slow for the 16 to 512 byte copies of header buffers, response encoding and FastCGI
records. musl's `malloc` does not appear: the request path allocates through mimalloc
already, so lever 1 of the design (mimalloc for the C allocations) has nothing to act on
and was skipped; lever 2 applies.

`cmd/scalwsd/musl/mem.c` (built by `cmd/scalwsd/build.rs` for the musl target only, linked
as a whole archive): `memcpy`, `memmove`, `memset`, `memcmp` and `bcmp` after glibc's SSE2
strategy (overlapping 16-byte loads and stores up to 128 bytes, a 64-byte loop with
aligned stores, `rep movsb`/`rep stosb` from 2 KiB). Checked against byte loops in 4.7
million cases (every size 0 to 1100 and larger ones, every alignment, every `memmove`
overlap distance, a difference at every position for the compares). With it the copy
functions take 2.1 % on respond (glibc 1.9 %) and 2.0 % on php (glibc 2.2 %), memset 1.1 %
(glibc 1.0 %).

### Gate 3 rerun with the fix (PASS)

The four failing scenarios and static-small as a control, rhel-final glibc (EL8 image)
against rhel-final musl with `mem.c`, both without PGO, rounds 2026-11-07 13:12 to 14:59,
`results/xeon-gate3b-{1,2,3}`. Another task's container (a preparation container,
pinned to CPUs 0-3 and 16-23) ran `dnf` from about 13:12 to 13:18, during the first run of
round 1; that round's ratios match rounds 2 and 3 and were kept.

| scenario | rps musl / glibc per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 1.007 / 1.006 / 0.997 | 1.006 | 1.6 | 1.016 | 0.994 | 0.927 | PASS |
| respond | 0.989 / 0.987 / 0.989 | 0.989 | 2.2 | 1.013 | 1.011 | 0.950 | PASS |
| proxy | 0.995 / 1.001 / 0.996 | 0.996 | 1.0 | 0.978 | 1.004 | 0.955 | PASS |
| cache-hit | 0.996 / 0.997 / 0.991 | 0.996 | 1.4 | 0.980 | 1.005 | 0.955 | PASS |
| php | 1.061 / 1.071 / 1.025 | 1.061 | 4.3 | 0.943 | 0.988 | 0.955 | PASS |

respond stays 1.1 % below in all three rounds, inside its spread. With both builds
profile-guided (below) musl is ahead there.

## Making up the sandbox's cost

### System calls per request

`perf stat` on the sandboxed server (rhel-final, sandbox wrapper configuration), every
syscall tracepoint, one scenario at a time:

| scenario | calls per request |
|---|---|
| respond, cache-hit | 2.0 (recvfrom, writev) |
| proxy (kept-alive upstream) | 4.0 |
| static-small, open-file cache on (default) | 3.0 (statx identity check, recvfrom, writev) |
| static-small and tls-h2, cache off | 5.9 to 6.0 (openat2, statx, preadv2, close, recvfrom, writev) |
| static-1m, cache off | 5.9 (sendfile, the rest as above) |
| proxy-churn (no keep-alive) | 11.3 (accept4, setsockopt, two epoll_ctl, shutdown, close, ...) |
| php (external pool) | 11.2 (socket, connect, getsockopt, close, two epoll_ctl per request) |

No `clock_gettime` or `getrandom` reaches the kernel per request, `epoll_ctl` is not
re-armed per request, and responses go out in one `writev`. The static paths make exactly
the calls their semantics need: the cache-off configuration opens every file by
definition, and the default cache checks identity with one `statx` per hit (`valid`
unset: changes are served at once). Two findings:

* **TCP_NODELAY** was set on every accepted socket. Linux copies the listener's option to
  accepted sockets, so it is now set once on the listener (commit 738de85, with a test for
  shared and per-loop listeners): one call less per connection.
* **FastCGI to external pools** uses one connection per request. Managed pools already keep
  connections (`FCGI_KEEP_CONN`, bounded by the pool's worker count); external pools do
  not, so that idle connections cannot starve other clients of a pool scalwsd does not
  size. Under the controller (ADR-0036) and in the sandboxed standalone setup PHP is
  always external, so this is where the sandbox meets PHP. Keeping connections to an
  external pool needs its worker count (a setting, or the controller's knowledge of the
  pool): a behaviour change, left as a proposal.

### Profile-guided release builds

ADR-0047 (accepted 2026-10-10) decided that release binaries of `scalwsd` are built with
`tools/pgo-build.sh`, but `scripts/release-build.sh` still built them plainly, as did
rc.1-11. An estimate on the host toolchain (one round of 3 runs, sandbox on against
rc.1-11) gave 1.21 static-small, 1.36 respond, 1.42 proxy, 1.46 cache-hit, 1.35 tls-h2,
1.11 php, against 0.96 to 1.006 for the plain build. Both release scripts now run
`tools/pgo-build.sh` in their build container (commits 7886446, e9cfadd); the images gain
llvm-tools, oha 1.16.0, openssl and PHP-FPM for the training load. Two compatibility
findings on the way, outside this work: EL8's default PHP 7.2 refuses the
`decorate_workers_output` line scalwsd writes into a managed pool (exit 78), so the image
uses the `php:8.2` stream; and on Alpine scalwsd looks for `php-fpm` and `php-fpm8.x` but
the binary is `php-fpm83`, and it needs util-linux `setpriv` (the apk depends on it), so
the image links `php-fpm` and installs `setpriv`. Both trainings ran without an error
line, PHP included.

Binaries: `pgo-el8` = `scripts/release-build.sh` (`SCALWS_RELEASE=12preP2`), `pgo-musl` =
`scripts/release-build-apk.sh --binaries-only`, rhel-final with the commits above, rustc
1.99.1, no `http3` feature.

### Gate 3 with both builds profile-guided (PASS)

Rounds 2026-11-07 23:05 to 2026-11-08 01:14, `results/xeon-gate3p-{1,2,3}`.

| scenario | rps musl / glibc per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 1.038 / 1.021 / 1.051 | 1.038 | 2.3 | 1.000 | 0.963 | 0.974 | PASS |
| respond | 1.031 / 1.032 / 1.038 | 1.032 | 3.1 | 1.000 | 0.969 | 0.974 | PASS |
| proxy | 0.997 / 0.990 / 1.015 | 0.997 | 1.7 | 1.013 | 1.003 | 1.000 | PASS |
| cache-hit | 1.042 / 1.019 / 1.035 | 1.035 | 2.6 | 0.944 | 0.966 | 1.000 | PASS |
| tls-h2-static-small | 0.981 / 1.018 / 1.020 | 1.018 | 4.4 | 0.971 | 0.983 | 0.979 | PASS |
| php | 1.010 / 1.012 / 0.991 | 1.010 | 2.3 | 0.995 | 1.003 | 0.979 | PASS |

## What a new install gets: sandbox on, PGO, against rc.1-11 (PASS)

The user's rule as it applies to a new install: `pgo-el8` with the sandbox on against
rc.1-11 (which has no sandbox), both through the sandbox wrapper's configuration (run as
an unprivileged user, PHP through an external pool, open-file cache off), so the
comparison differs only in the build. Rounds 2026-11-07 19:32 to 22:20,
`results/xeon-rcon-{1,2,3}`; every `on` server sampled showed
`Seccomp: 2` (253 samples), every rc.1-11 one `0` (250).

| scenario | rps on / rc1-11 per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| static-small | 1.219 / 1.243 / 1.248 | 1.243 | 1.0 | 0.780 | 0.804 | 0.897 | PASS |
| static-1m | 0.964 / 1.021 / 1.019 | 1.019 | 3.4 | 0.901 | 0.982 | 0.897 | PASS |
| respond | 1.365 / 1.376 / 1.376 | 1.376 | 1.4 | 0.734 | 0.727 | 0.921 | PASS |
| proxy | 1.392 / 1.404 / 1.418 | 1.404 | 1.4 | 0.695 | 0.712 | 0.909 | PASS |
| proxy-churn | 1.224 / 1.233 / 1.246 | 1.233 | 0.9 | 0.781 | 0.811 | 0.938 | PASS |
| cache-hit | 1.448 / 1.442 / 1.455 | 1.448 | 1.4 | 0.692 | 0.691 | 0.917 | PASS |
| tls-h2-static-small | 1.344 / 1.328 / 1.361 | 1.344 | 3.5 | 0.770 | 0.744 | 0.945 | PASS |
| php | 1.063 / 1.176 / 1.146 | 1.146 | 5.3 | 0.891 | 0.854 | 0.945 | PASS |

static-1m's first round (0.964) is the host's bimodal 1 MiB result (rounds sit near
9 600 or 10 300 req/s for any build); the other two rounds are above 1.

Node.js and Python start processes, so they are never sandboxed; for them the release
build is compared as is: `pgo-el8` against rc.1-11, standard bench configuration,
`results/xeon-rcapps2-{1,2,3}` (2026-11-08 01:16 to 01:58).

| scenario | rps pgo / rc1-11 per round | median | spread % | p99 | cpu_us | peak RSS | gate |
|---|---|---:|---:|---:|---:|---:|---|
| node | 1.007 / 0.996 / 1.006 | 1.006 | 0.7 | 0.997 | 0.773 | 0.936 | PASS |
| python | 0.995 / 1.000 / 1.000 | 1.000 | 0.6 | 0.906 | 0.818 | 0.915 | PASS |

(Requests per second there are bound by the applications; scalwsd's own CPU per request
is 18 to 23 % lower.)

### Incident: a duplicate runner (2026-11-07)

The driver for these runs was queued twice (a waiter thought gone was still alive), so two
copies ran: instance 1 from 16:41:40 and instance 2 from 17:00:15. Each held the lock only
around its own run and waited for a quiet host (1-minute load under 2) before the next.

* rcon: instance 1 measured rounds 1 to 3 from 16:47 to 19:30 (its first round overlapped
  the musl build until 17:00; its last write, the aggregate, is at 19:30:34); instance 2
  then measured rounds 1 to 3 again from 19:32:18 to 22:20:29 into the same directories,
  while instance 1 waited in its quiet check. Every `summary.csv` of rcon (30 files, 8
  scenarios each) is dated 19:37:53 to 22:20:29, so the table above is instance 2's run
  alone; its logs show no bind failure, and instance 1 started its next run (apps) only at
  22:22.
* apps (node, python): instance 1 measured from 22:22; instance 2 started the same run at
  the same time, its servers failed on the taken ports, and its end removed the lock. That
  run (`xeon-rcapps-*`) was discarded and repeated alone (`xeon-rcapps2-*`, above) after
  the duplicate was stopped by PID at 23:02.
* gate 3 with PGO ran from 23:05 with the duplicate gone.

## Still to run

The HTTP/3 gates, after rhel-h3 (HTTP/3 in the packages) is merged:

1. **http3 feature against no feature, TCP scenarios.** Two release builds of the same
   tree (`SCALWS_FEATURES=scalwsd/http3` and empty, both PGO), `bench/multi.sh
   MODE=release`, three rounds, scenarios static-small, static-1m, respond, proxy,
   cache-hit, tls-h2-static-small, php, node, python, both through the wrapper that drops
   `http3: true` (the listeners stay TCP only, so only the feature's presence differs).
   Gate: `bench/musl_gate.py --base noh3 --new h3`, same rule.
2. **h3 with the sandbox off and on.** The http3 build, `bench/sandbox-variants.sh` with
   `SBX_HTTP3=1`, scenario h3-static-small with the open-file cache off and with
   `OPEN_FILE_CACHE=check`, three rounds, `bench/musl_gate.py --base off --new on`. rc.1-11
   has no HTTP/3, so for h3 the comparison is within the build (expect the TCP result:
   about 3 % below with the cache off, from seccomp and the Landlock walk); the http3 PGO
   build against the plain build of the first gate run (h3 0.968 of off there) shows
   whether the new install is ahead.