Decision records

ADR-0044: Fountain accept policy (opt-in)

Status: accepted, opt-in — 2026-10-09

Context

All event loops of one scalwsd wait on a shared accept queue (ADR-0029, ADR-0033). The loop that wakes first takes a new connection, so under a burst one loop can take most of them and keep them for their lifetime: measured 0 % / 98 % / 29 % / 94 % CPU across four loops under 64 keep-alive connections. Balancing connection counts evenly (ADR-0038, step 3a) made the loops even but cost CPU per request (+5–12 %) and TLS p99 (2.65 → 10.26 ms): concentrated loops batch more work per wake-up. Neither “first come” nor “even” is best.

Decision

server.accept_policy: shared | fountain (default shared, unchanged behaviour).

With fountain the loops fill in order, like the tiers of a fountain: loop 0 takes new connections; loop i takes one only while every loop before it is full, and otherwise leaves it to them for up to 2 ms (a stuck loop never strands a connection).

• Full is the loop’s busy share, read every 20 ms from Tokio’s own busy-time counter (no cost on the request path), smoothed over about two samples, with hysteresis: full from 85 %, free again below 70 %.
• Bursts: each connection accepted in a 20 ms window beyond the previous window’s count adds an estimated 10 % until the next sample. A steady stream of new connections adds nothing (its load is in the measured share); 64 connections opened at once spread over the loops (about nine per loop before the next one starts).
• Established connections never move.

Measurements (development VM, 4 loops; bench/fountain.sh, bench/multi.sh)

Open loop, a new connection per request (the case the policy is for), 3 rounds:

Mixed load (TLS + HTTP/2 small files with HTTP/1 1 MiB files at the same time): HTTP/2 +20 %, 1 MiB −15 % — capacity moves between the two, no net gain.

Closed loop, 64 keep-alive connections, 5 runs (spread 3–12 %): static −4 %, 1 MiB −2 %, respond −5 %, proxy −7 %, cache hits +3 %, TLS + HTTP/2 +6 %, PHP +6 % (CPU per request −7 %).

Consequences

• fountain lowers server CPU per request by up to ~20 % at moderate load with many short-lived connections — CPU left to PHP-FPM, Node.js, Python or databases on the same cores (hosting servers). Throughput and latency stay the same there.
• It is not the default: with few long-lived connections it is at best even with shared, and it can shift capacity between workloads of different weight.
• A first version that counted 2 % per new connection regardless of the steady rate left two of four loops without connections in the closed-loop benchmark (−6 … −19 %); the surge-only estimate fixed that.
• EPOLLEXCLUSIVE (nginx’s way: the kernel wakes one waiter that is idle in epoll_wait) was examined and is not usable here. Tokio registers the listener in its own epoll without the flag and does not expose that registration; nesting a per-loop epoll that holds the listener with EPOLLEXCLUSIVE does not help — measured on Linux 6.8 (bench/probes/epollexclusive-nested.py), one connection wakes 1 of 4 threads waiting directly but 4 of 4 when the exclusive epoll is nested, because the kernel only stops waking at an epoll with a thread blocked on it. Patching mio is not enough either: it always adds EPOLLRDHUP, which the kernel rejects with EPOLLEXCLUSIVE, and Tokio’s edge-triggered readiness makes the one woken loop take a whole burst. A dedicated acceptor thread per loop would not see the loop’s load and would add a hand-off per connection.