# ADR-0049: Upstream pools and run-time members URL: https://scalws.com/docs/adr/0049-upstream-pools-and-runtime-members/ Source: `docs/adr/0049-upstream-pools-and-runtime-members.md` in the ScalWS repository Status: accepted, 2026-10-14 ## Context A `proxy` runtime took one `upstream` address. There were no pools of members, no load balancing, no session affinity, no passive health tracking, no retries to another member and no weighted split between backends; a plain load balancer in front of several application servers needed another product. Kubernetes phase 2 makes ScalWS an ingress controller (phase 2b: Ingress, phase 2c: Gateway API). In a cluster, pod addresses change all the time, and the only way to change an address was a full configuration reload, which rebuilds every changed proxy and used to reset per-application rate-limit state. Phase 2a adds the data-plane features both phases need, without any Kubernetes code. ## Decision * **Pools by name.** A top-level `upstreams` map defines pools (names follow the tenant and application name rule). A proxy runtime uses one with `upstream: {pool: name}`, or several with `backends: [{pool, weight}]` (weights 0 to 1000, at most 64, at least one above 0, chosen per request by smooth weighted round robin). `upstream` and `backends` are mutually exclusive. The single-address form (`http://`, `https://`, `unix:`) keeps the existing `ProxyHandler` path: no retries, no ejection, no new allocations per request. `preserve_host`, the timeouts and `websocket` stay on the runtime; `client` is ignored on pool routes with a warning (members are shared between routes, so they use `server.upstream_client`), and `tls_ca_file` is refused there (pools have `tls`). * **Members.** `ip:port`, `[ipv6]:port`, `host:port` (resolved at connect time) or `unix:/path` (only as the single member of a plain-HTTP pool); weight 1 to 1000; no duplicates; at most 10,000 per pool. Each member has its own connection pool (`keepalive` idle connections, default 32, at most 1024). * **`members_from`.** `file` (default): the configuration's `members`. `api`: the member list is replaced at run time through the admin API; the file's `members` only seed a new pool. * **Selection.** `round_robin` is smooth weighted round robin (nginx). `least_conn` picks the fewest in-flight requests divided by weight, ties round robin. `hash` is weighted rendezvous hashing (`weight / -ln(u)`, `u` from a stable 64-bit hash of key and member), so removing a member moves only its keys; the key is an affinity cookie, a request header or the client address after `trusted_proxies`. A missing key falls back to round robin. * **Affinity cookie.** The value is the first 16 bytes of HMAC-SHA256 of the member's canonical address under a random key generated per process, in lower-case hex. It reveals nothing and cannot name an arbitrary member. Attributes: `Path=/; HttpOnly; SameSite=Lax`, `Max-Age` from `cookie_max_age` (0: session cookie), `Secure` on HTTPS requests. Set only when a member answered and the request did not already hold it; appended next to the member's own cookies. * **Passive health.** A failure is a connect error or timeout, no response head within `response_timeout`, a connection broken after the request was written, or a `502`, `503` or `504`. Failures caused by the client's request body (over the body limit, a body read timeout, an aborted upload, or an upload still waiting for client data when the response timeout passes) are neutral: no health change, no retry. After `failures` consecutive failures (default 3, 0 turns it off, at most 100) the member is ejected for `cooldown` (default 10 s, 1 ms to 1 h), then it is on probation: one success restores it, one failure ejects it again. Results of requests that were in flight when the member was ejected are ignored, so an ejection is never extended or ended by stale results. When every member is ejected, the one ejected longest ago is used: passive health alone never refuses traffic. No active checks. * **Retries.** At most `retries` (default 1, 0 to 5) further attempts, each on a member not tried yet, and only when the pool has more than one member, nothing was sent to the client, and either the connection failed before the request was written (any method), or the method is idempotent (GET, HEAD, OPTIONS, PUT, DELETE) and the member answered `502`, `503` or `504` or timed out. The body must be replayable: bodies with a `Content-Length` of at most 64 KiB are kept in memory; larger bodies and bodies without a length (also short chunked ones) are sent once. A connection broken after the request was written counts for health but is not retried. WebSocket upgrades go to one member, without retry. * **Run-time members.** `PUT /v1/upstreams/{name}` with `{"members": [...]}` on the admin socket or the mTLS listener replaces the list of an `api` pool: `404` unknown pool, `409` file pool, `400` invalid list (nothing changes), `413` body over 2 MiB, `200` with the pool's state. It takes the same lock as reloads (`Control.lock`), so a reload never plans member changes against a set a `PUT` is replacing. A new member set is built beside the current one and swapped in atomically; members whose canonical address stays are the same objects, with their connections, load and health. Planning (new members' connectors) and pruning metric series run on the blocking pool while the lock is held; `https` members of a pool share one TLS client configuration, so the system roots are read once per pool, not per member. `GET /v1/upstreams` and `GET /v1/upstreams/{name}` show pools, members, weights and state; these views are not served on the unauthenticated metrics listener. * **Reload.** Pools are matched by name. A pool whose settings (everything except `members`) are unchanged is kept; a kept file pool gets its new member list planned during prepare and installed after the generation swap, under the reload lock. A pool whose settings changed is rebuilt; an api pool keeps its run-time members in either case (also when it switches from `file` to `api`). Per-application rate-limit and quota state is kept when the limit's definition is unchanged. After a process restart an api pool starts with the file's `members`, or empty. * **Metrics.** `scalws_upstream_requests_total{pool, member, class}`, `scalws_upstream_request_duration_seconds{pool}`, `scalws_upstream_member_ejected{pool, member}`, `scalws_upstream_pool_retries_total{pool}` and `scalws_upstream_members{pool}`. The retries family is not named `scalws_upstream_retries_total`: that name already counts stale-connection retries by tenant, application and protocol (ADR-0043). Series of removed members and pools, and of pools a refused reload built, are dropped; member series are bounded by the 10,000 member limit. The access log gains `upstream_member` and `upstream_attempts`. * **Tools.** `scalwsctl upstreams`, `upstreams show ` and `upstreams set addr[=weight]...`; a read-only Upstreams view in the console. ## Consequences * The affinity secret is per process: a restart moves every cookie client once, and several instances behind `SO_REUSEPORT` or scalws-controller do not share affinity, balancing or health state. `by: header` and `by: client_ip` give the same member on every instance. * Under scalws-controller (`listener_mode: inherited` or `reuseport`) the `PUT` answers `501`, on the instances and on the controller's control socket, because the instances of a generation would disagree about the members; file pools work. The ingress pods of phase 2b run `scalwsd` alone. Sharing run-time members across instances is left for a later phase. * Members of api pools are lost on restart; whatever maintains them must set them again. * A request body of up to 64 KiB is buffered when a retry is possible; nothing is buffered for single-member pools or `retries: 0`. * A configuration that keeps the single-address form behaves and performs as before.