Decision records

ADR-0043: Upstream connection telemetry

Status: accepted — 2026-10-09

Context

Comparing ScalWS with nginx on real applications raised questions the existing metrics could not answer: are upstream connections reused, how long does a request wait for a connection or a PHP-FPM worker, how long does the upstream take to answer, how often is a request replayed or abandoned. scalws_upstream_failures_total counts only synthesised error responses. The proxy (HTTP/1.1, including the managed Node.js, Python and container runtimes) and the FastCGI client manage connections differently, but these questions are the same for both.

Decision

1. One telemetry shape for every upstream protocol. scalws_observe::UpstreamTelemetry holds, per application and protocol (http1, fastcgi):

   Metric	Meaning
   scalws_upstream_connections_opened_total	new connections to the upstream
   scalws_upstream_connections_reused_total	requests sent on a kept-alive connection
   scalws_upstream_retries_total	requests sent again after a pooled connection turned out stale
   scalws_upstream_cancelled_total	exchanges abandoned before the response was complete (client gone, response timeout, queued request given up)
   scalws_upstream_wait_seconds	from entering the upstream engine until the request is written: pool checkout and connect, or the FastCGI queue
   scalws_upstream_ttfb_seconds	from writing the request until the response head (FastCGI: first output)

2. Pre-resolved handles, atomics on the request path. The handles are created when an application is prepared (RuntimeMetrics::upstream, on demand per protocol, so static applications export no such series) and handed to the proxy pool and the FastCGI pool. A request costs a few relaxed atomic adds and two clock reads; no label lookup, no allocation. Histograms are the lock-free FastHistogram; buckets 50 µs … 30 s.

3. Engines keep their own connection management. The telemetry is an observer, not a shared abstraction every request passes through: the HTTP/1 pool (hyper’s protocol code, per-event-loop pools) and the FastCGI drivers (shared queue, persistent connections) stay as they are.

4. Series follow the application’s lifecycle (Owned families): removed applications’ series are dropped after a reload.

Consequences

Connection reuse ratio = reused / (reused + opened). A rising wait on fastcgi means PHP-FPM workers are the bottleneck (queueing), on http1 it means connects; a rising ttfb points at the application itself. Measured on the development VM the overhead is within the run-to-run spread of the proxy benchmark.