Internals: benchmark reports

Node.js, Python and fixed responses: where the remaining gaps come from: 2026-10-12

Follow-up to 2026-10-12-xeon-release-run6.md (respond 0.95×, python 0.94×, node 1.00× nginx 1.31 and 0.91× OpenLiteSpeed; static-1m 0.99×). Bare metal was unavailable, so this was measured locally in Docker (Ubuntu 24.04, release build of the run-6 code, nothing pinned). Throughput there is noisy; the measures used are the application’s CPU time per request (/proc/<pid>/stat, 8 s under wrk -t4 -c64), syscall counts (strace -c), the bytes the application receives, and user-space instructions per request (callgrind).

static-1m: no work item

Every server is at the same ceiling (scalws 11 024, nginx 11 123, OpenLiteSpeed 10 177 req/s ≈ 11.5 GB/s over loopback) with equal CPU per request (363 vs 360 µs). Not pursued.

Node.js: OpenLiteSpeed forwards fewer headers

OpenLiteSpeed 1.9.3 (same bench/configs/openlitespeed) sends the application Host, User-Agent, Accept, X-Forwarded-Host, Accept-Encoding: gzip; nginx (bench config) and scalws also send X-Forwarded-For, X-Forwarded-Proto and X-Request-Id. Syscalls per request are identical for all servers (node: 1 read, 1 writev; uvicorn: 1 recvfrom, 3 sendto). nginx with OpenLiteSpeed’s header set, 3 interleaved rounds:

So OpenLiteSpeed’s lead is information it does not pass on (client address, scheme, request id), not a faster proxy. Behind the same external node process scalws (either HTTP/1 client) and nginx are level within the noise.

Python, not the bytes, not the connections

• Behind the same external uvicorn (2 workers), uvicorn needs more user time per request behind scalws than behind nginx in every round: 158–167 µs (user 125–132) with either client versus 147–150 (user 119).
• The request bytes are not the cause: wrk replaying scalws’s exact request (lower-case names, 24-hex request id) and nginx’s (title case, 32-hex id) directly against uvicorn costs the same (161–172 µs for all four variants, 3 rounds).
• Upstream connections are the same: 64 from every server, spread alike over the workers.
• uvicorn sometimes switches a worker into allocating each 256 KiB receive buffer with mmap/mremap/munmap (glibc heap trimmed below the request size); this happened behind nginx as well and costs nothing measurable. Fixing the threshold (MALLOC_MMAP_THRESHOLD_) made uvicorn slower (more kernel time) and is not used.
• The own HTTP/1 client and hyper cost the application the same (3 interleaved rounds: 170–176 vs 173–177 µs).

Left: timing or CPU sharing (in Docker the servers share cores with the application; scalws runs 8 event loops, nginx 4 workers). Needs bare metal with pinned cores and perf stat on the uvicorn workers (cycles, IPC, cache misses, context switches).

Fixed responses: spread over the pipeline

callgrind, one event loop, 500 warm-up + 4 000 requests (ab -k -c 4), user space only: scalws 16 364 instructions per request, nginx 1.24 10 941. scalws by area: request parsing ~2.8k (scalws_h1::request::parse 1.5k, httparse 1.1k), http header maps ~1.9k, pipeline (route_and_handle, handle, finish) ~1.9k, h1 server loop ~0.8k, response head 0.5k, tokio ~1.5k, memcpy 1.3k (0.5k moving the handler future into its pinned slot; rep movsb iterations inflate this), host/path normalisation ~0.5k, request id 0.16k, RespondHandler 0.34k. On bare metal the difference is ~0.9 µs of 19 µs per request (the kernel’s share is equal). No single item closes it; it needs a structural change such as a fast path that answers fixed responses without building http header maps.

Bare-metal plan

1. Python: scalws final / hyper / nginx in front of the same pinned uvicorn; perf stat -p on the workers; if IPC or misses differ, compare scheduling (perf sched) and pacing.
2. Node: nginx with OpenLiteSpeed’s header set next to OpenLiteSpeed, to confirm that the 9 % is the header set.
3. Respond: perf record scalws vs nginx under load, to rank the user-space items by cycles rather than instructions.

Bare metal results (same day, afternoon)

Xeon host as in 2026-10-12-xeon-release-run6.md (server CPUs 0-3, applications 4-7, oha on socket 1, 64 connections). perf stat on the application processes.

Python: glibc’s mmap threshold, not the client

Per request inside uvicorn, round 1 / round 2 (rps, cycles, syscalls):

strace -c: the slow cases make 0.44 mmap, mremap and munmap per request (plus a page fault); the fast ones none. CPython reads each socket into a 256 KiB buffer and shrinks it; below glibc’s mmap threshold (128 KiB at start) that buffer is an mmap. glibc raises the threshold when an mmapped chunk is freed whole, which happens on the first EOF a worker reads (the buffer is resized to zero). 40 connect-and-close calls to the uvicorn socket ended the mode at once (strace: 4 076 → 4 457 req/s). Whether a run gets the fast mode depends on which front closes a connection first: the same app behind nginx ran at 6 277 in one round and 5 688 in the next.

Fix (scalws-runtime-python): managed processes start with MALLOC_MMAP_THRESHOLD_=524288 and MALLOC_TRIM_THRESHOLD_=1048576, the values glibc would adapt to, unless the application sets MALLOC_*. Three rounds: 6 342 / 6 271 / 6 394 req/s with 4.1 syscalls and 0.04 page faults per request (before: 5 962 / 5 939 / 6 363), RSS unchanged (27.5 MiB per worker). The earlier attempt in Docker set only the mmap threshold, so glibc trimmed the heap on every free (brk), which made it slower.

Node.js: header set confirmed

Instructions per request inside node (3 rounds, nearly identical):

With the same information scalws costs node less than nginx does; OpenLiteSpeed’s lead is the three headers it does not send (client address, scheme, request id). A larger V8 young generation (--max-semi-space-size 32 or 64) changed nothing.

Fixed responses

perf stat under load (cycles per request, user / total): scalws 15.8k / 50.8k, nginx 15.9k / 51.7k at 201 259 vs 197 923 req/s: the kernel share is two thirds and equal, and user-space cycles are level although scalws executes 13.6k instructions against 9.7k. The run-6 gap (0.95×) was mostly nginx’s spread (205k–221k over its 5 runs). The pipeline now answers respond routes from Handler::fixed() after all checks, without a request context or handler future: +6 % in a plain release build (140 918 → 149 318, 5 interleaved quick runs each).