Benchmarks

2026-10-09 — Where scalws trailed nginx outside the proxy: causes

Follow-up to 2026-10-09-final-vs-nginx.md. Four areas were behind nginx 1.24–1.31.6 there: small static files (0.95–0.98×), 1 MiB files (0.95×), TLS + HTTP/2 (0.96–0.97×) and p99 tails (cache hits 9.2 ms vs 3.3–6.8 ms). Each was measured before changing anything.

1. Small static files — no gap

Syscalls per request (strace, 20 000 requests): scalws 6 (recvfrom, openat2, statx, preadv2, writev, close), nginx 1.30.5 8 (recvfrom, openat, fstat, 2 × setsockopt for tcp_nopush, writev, sendfile, close). Profiled on the same load (perf, CPUs 0-3, all work on them): scalws 97 819 req/s at 40.9 µs per request (kernel 29.8, user 9.8), nginx 92 364 req/s at 43.3 µs (kernel 37.6, user 4.7). scalws spends 5 µs more in user space and 8 µs less in the kernel (small files are read into memory instead of sendfile). The 2–5 % in the comparison is within this scenario’s run-to-run spread. Nothing changed.

2. 1 MiB files — connection distribution across event loops

Transmission is equivalent: one sendfile of the whole file per request; the response head goes out with MSG_MORE (what nginx gets from TCP_CORK), 6 syscalls vs nginx’s 8. CPU per request is lower than nginx’s. What limits throughput is how connections are spread over the event loops (next section): with ~300 µs of work per request the most loaded loop sets the ceiling.

3. TLS + HTTP/2 — parity on average, higher spread

Means over the runs: scalws 63 137 / 62 455, nginx 62 665–63 879 req/s; CPU per request equal (41–43 vs 42.6–44.2 µs); p99 4× lower for scalws (2.5–2.9 vs 10.4–11.2 ms). The lower median comes from a run-to-run spread of 5–7 % (nginx 0.5–2.6 %): scalws’s best run (68 083) is above every nginx run. The spread follows the connection distribution.

4. p99 tails — the event loops are unevenly loaded

All loops wait on one shared accept queue; a loop that wakes first takes a whole burst of new connections and keeps them. Per-loop CPU under 64 keep-alive connections:

Tried: count-balanced accept (ADR-0038 step 3a — a loop clearly ahead of the least loaded one leaves new connections to the others for up to 2 ms). The loops became even (52–76 %), but (results/2026-10-09-loop-balance-ab/):

Spreading the work evenly makes every loop partly idle and wake more often; concentrated loops batch more work per wake-up. Only the heavy 1 MiB requests gained, everything else cost CPU, and TLS p99 got four times worse. By ADR-0038’s rule (adopt only if p99 or throughput improve) it was not adopted.

nginx avoids both effects with EPOLLEXCLUSIVE on the shared listener: the kernel wakes only a worker that is idle in epoll_wait, so new connections go to idle workers without any delay or bookkeeping. Tokio/mio do not expose that flag; trying it means owning the listener’s epoll registration — a separate piece of work, with the per-loop distribution measured the same way.

Summary