Decision records

ADR-0029: Per-core event loops

• Status: Accepted (2026-10-06)
• Revisits the “measured, not adopted” note of docs/benchmarks/2026-10-06-proxy-profile.md.

Context

With the file system out of the way (ADR-0027, ADR-0028) the remaining gap to nginx is per-request cost and CPU utilisation: under load the work-stealing runtime used 3.15–3.3 of 4 CPUs (one I/O driver; other workers are woken through futexes), nginx all four. An upper-bound experiment — four single-threaded scalws processes pinned to one CPU each, SO_REUSEPORT — served respond at 157.5k / 149.6k req/s against 113.7k / 108.5k for one process with four work-stealing threads (+38 %). For the proxy scenario the same experiment gave +10 % (upstream-bound), which is why it was not adopted then.

Decision

• server.event_loops: unset = auto (one per CPU the process may run on, available_parallelism, which honours affinity and cgroup quotas); N; 0 = the previous shared work-stealing runtime. Per-core loops are Linux-only; elsewhere 0. scalwsd --worker-threads N sets the count.
• Each event loop is an OS thread running a current-thread Tokio runtime, pinned to one CPU of the process affinity mask (when there are no more loops than CPUs). Every listener is bound once per loop with SO_REUSEPORT; the kernel spreads connections and each loop has its own epoll. A connection lives on one loop for its whole life.
• A small multi-threaded control runtime keeps everything that is not a client connection: supervisors, admin API, metrics, ACME, diagnostics, HTTP/3.
• Shared state stays shared (Arc, atomics, mutexes); tasks spawned while serving a request run on that request’s loop. I/O objects created on one loop (e.g. a pooled upstream connection) can be used from another — correct, but woken across threads; per-loop pools are a follow-up.
• Shutdown: accept loops stop on the shared token, connections drain through the shared TaskTracker; after the drain timeout the loops are stopped and their remaining tasks dropped.

Consequences

• A CPU-heavy request delays the other connections of its loop (as in nginx); blocking work already goes to the blocking pool.
• SO_REUSEPORT lets another process of the same user join the port group; scalws’s own user is the trust boundary there anyway.