# Node.js, Python and fixed responses: where the remaining gaps come from: 2026-10-12

URL: https://scalws.com/docs/benchmarks/2026-10-12-app-respond-analysis/
Source: `docs/benchmarks/2026-10-12-app-respond-analysis.md` in the ScalWS repository

Follow-up to `2026-10-12-xeon-release-run6.md` (respond 0.95×, python 0.94×, node 1.00×
nginx 1.31 and 0.91× OpenLiteSpeed; static-1m 0.99×). Bare metal was unavailable, so this
was measured locally in Docker (Ubuntu 24.04, release build of the run-6 code, nothing
pinned). Throughput there is noisy; the measures used are the application's CPU time per
request (`/proc/<pid>/stat`, 8 s under `wrk -t4 -c64`), syscall counts (`strace -c`), the
bytes the application receives, and user-space instructions per request (callgrind).

## static-1m: no work item

Every server is at the same ceiling (scalws 11 024, nginx 11 123, OpenLiteSpeed 10 177
req/s ≈ 11.5 GB/s over loopback) with equal CPU per request (363 vs 360 µs). Not pursued.

## Node.js: OpenLiteSpeed forwards fewer headers

OpenLiteSpeed 1.9.3 (same `bench/configs/openlitespeed`) sends the application
`Host, User-Agent, Accept, X-Forwarded-Host, Accept-Encoding: gzip`; nginx (bench config)
and scalws also send `X-Forwarded-For`, `X-Forwarded-Proto` and `X-Request-Id`. Syscalls
per request are identical for all servers (node: 1 `read`, 1 `writev`; uvicorn: 1
`recvfrom`, 3 `sendto`). nginx with OpenLiteSpeed's header set, 3 interleaved rounds:

| nginx headers | node app µs/req | python app µs/req |
|---|---:|---:|
| all (bench config) | 50.9 / 52.7 / 47.3 | 156.7 / 149.5 / 149.9 |
| without X-Request-Id | 51.4 / 49.9 / 49.6 | 149.4 / 146.7 / 147.9 |
| OpenLiteSpeed's set | 47.6 / 49.4 / 40.4 | 141.4 / 145.4 / 143.8 |

So OpenLiteSpeed's lead is information it does not pass on (client address, scheme,
request id), not a faster proxy. Behind the same external node process scalws (either
HTTP/1 client) and nginx are level within the noise.

## Python, not the bytes, not the connections

* Behind the same external uvicorn (2 workers), uvicorn needs more user time per request
  behind scalws than behind nginx in every round: 158–167 µs (user 125–132) with either
  client versus 147–150 (user 119).
* The request bytes are not the cause: wrk replaying scalws's exact request (lower-case
  names, 24-hex request id) and nginx's (title case, 32-hex id) directly against uvicorn
  costs the same (161–172 µs for all four variants, 3 rounds).
* Upstream connections are the same: 64 from every server, spread alike over the workers.
* uvicorn sometimes switches a worker into allocating each 256 KiB receive buffer with
  `mmap`/`mremap`/`munmap` (glibc heap trimmed below the request size); this happened
  behind nginx as well and costs nothing measurable. Fixing the threshold
  (`MALLOC_MMAP_THRESHOLD_`) made uvicorn slower (more kernel time) and is not used.
* The own HTTP/1 client and hyper cost the application the same (3 interleaved rounds:
  170–176 vs 173–177 µs).

Left: timing or CPU sharing (in Docker the servers share cores with the application;
scalws runs 8 event loops, nginx 4 workers). Needs bare metal with pinned cores and
`perf stat` on the uvicorn workers (cycles, IPC, cache misses, context switches).

## Fixed responses: spread over the pipeline

callgrind, one event loop, 500 warm-up + 4 000 requests (`ab -k -c 4`), user space only:
scalws 16 364 instructions per request, nginx 1.24 10 941. scalws by area: request parsing
~2.8k (`scalws_h1::request::parse` 1.5k, httparse 1.1k), `http` header maps ~1.9k,
pipeline (`route_and_handle`, `handle`, `finish`) ~1.9k, h1 server loop ~0.8k, response
head 0.5k, tokio ~1.5k, `memcpy` 1.3k (0.5k moving the handler future into its pinned slot;
`rep movsb` iterations inflate this), host/path normalisation ~0.5k, request id 0.16k,
`RespondHandler` 0.34k. On bare metal the difference is ~0.9 µs of 19 µs per request (the
kernel's share is equal). No single item closes it; it needs a structural change such as
a fast path that answers fixed responses without building `http` header maps.

## Bare-metal plan

1. Python: scalws final / hyper / nginx in front of the same pinned uvicorn; `perf stat -p`
   on the workers; if IPC or misses differ, compare scheduling (`perf sched`) and pacing.
2. Node: nginx with OpenLiteSpeed's header set next to OpenLiteSpeed, to confirm that the
   9 % is the header set.
3. Respond: `perf record` scalws vs nginx under load, to rank the user-space items by
   cycles rather than instructions.

## Bare metal results (same day, afternoon)

Xeon host as in `2026-10-12-xeon-release-run6.md` (server CPUs 0-3, applications 4-7, `oha`
on socket 1, 64 connections). `perf stat` on the application processes.

### Python: glibc's mmap threshold, not the client

Per request inside uvicorn, round 1 / round 2 (rps, cycles, syscalls):

| front | round 1 | round 2 |
|---|---|---|
| scalws final | 5 878, 876k, 5.6 | 5 880, 880k, 5.5 |
| scalws final-hyper | 5 939, 872k, 5.6 | 5 902, 876k, 5.5 |
| nginx 1.31 | 6 277, 824k, 5.0 | 5 688, 909k, 7.1 |

`strace -c`: the slow cases make 0.44 `mmap`, `mremap` and `munmap` per request (plus a
page fault); the fast ones none. CPython reads each socket into a 256 KiB buffer and
shrinks it; below glibc's mmap threshold (128 KiB at start) that buffer is an mmap. glibc
raises the threshold when an mmapped chunk is freed whole, which happens on the first
EOF a worker reads (the buffer is resized to zero). 40 connect-and-close calls to the
uvicorn socket ended the mode at once (strace: 4 076 → 4 457 req/s). Whether a run gets
the fast mode depends on which front closes a connection first: the same app behind
nginx ran at 6 277 in one round and 5 688 in the next.

Fix (scalws-runtime-python): managed processes start with `MALLOC_MMAP_THRESHOLD_=524288`
and `MALLOC_TRIM_THRESHOLD_=1048576`, the values glibc would adapt to, unless the
application sets `MALLOC_*`. Three rounds: 6 342 / 6 271 / 6 394 req/s with 4.1 syscalls
and 0.04 page faults per request (before: 5 962 / 5 939 / 6 363), RSS unchanged
(27.5 MiB per worker). The earlier attempt in Docker set only the mmap threshold, so
glibc trimmed the heap on every free (`brk`), which made it slower.

### Node.js: header set confirmed

Instructions per request inside node (3 rounds, nearly identical):

| front | rps | instructions | cycles |
|---|---:|---:|---:|
| scalws final | 10 915–11 182 | 83.4k | 234–239k |
| nginx 1.31, bench headers | 11 036–11 113 | 85.9k | 237–239k |
| nginx 1.31, OpenLiteSpeed's header set | 11 310–11 550 | 77.0k | 228–233k |

With the same information scalws costs node less than nginx does; OpenLiteSpeed's lead is
the three headers it does not send (client address, scheme, request id). A larger V8
young generation (`--max-semi-space-size` 32 or 64) changed nothing.

### Fixed responses

`perf stat` under load (cycles per request, user / total): scalws 15.8k / 50.8k, nginx
15.9k / 51.7k at 201 259 vs 197 923 req/s: the kernel share is two thirds and equal, and
user-space cycles are level although scalws executes 13.6k instructions against 9.7k.
The run-6 gap (0.95×) was mostly nginx's spread (205k–221k over its 5 runs). The
pipeline now answers `respond` routes from `Handler::fixed()` after all checks, without
a request context or handler future: +6 % in a plain release build (140 918 → 149 318,
5 interleaved quick runs each).