# Load balancing URL: https://scalws.com/docs/guide/load-balancing/ Source: `docs/guide/load-balancing.md` in the ScalWS repository A `proxy` runtime can send its requests to a named **pool** of members instead of one address. ScalWS then balances requests across the members, can keep a client on one member (affinity), takes members that keep failing out for a while (passive health), retries idempotent requests on another member, splits traffic by weight across pools, and lets you replace the member list at run time through the admin API, without a reload. Design record: [ADR-0049](https://scalws.com/docs/adr/0049-upstream-pools-and-runtime-members/). Route matching and route policies (rewrites, CORS, basic authentication, address lists) are in [configuration.md](https://scalws.com/docs/guide/configuration/#route-matching) and [ADR-0050](https://scalws.com/docs/adr/0050-route-matching-and-policies/). Contents: * [Pools and members](#pools-and-members) * [Choosing a member](#choosing-a-member) * [Passive health](#passive-health) * [Retries](#retries) * [Weighted backends](#weighted-backends) * [TLS to members](#tls-to-members) * [Changing members at run time](#changing-members-at-run-time) * [Observability](#observability) * [Under scalws-controller](#under-scalws-controller) * [Complete example](#complete-example) ## Pools and members Pools are defined once, in the top-level `upstreams` section of the main configuration file, and used by name from any `proxy` runtime (an application's `runtime` or a route's): ```yaml upstreams: api: members: - { address: "192.0.2.5:8080", weight: 1 } - { address: "192.0.2.6:8080" } # weight defaults to 1, range 1..=1000 members_from: file # file (default) | api balance: round_robin # round_robin (default) | least_conn | hash passive_health: failures: 3 # consecutive failures before a member is ejected cooldown: 10s # how long it stays out retries: 1 # 0..=5, to a different member keepalive: 32 # idle connections kept per member scheme: http # http | https ``` ```yaml tenants: - name: acme applications: - name: api domains: [api.example.com] runtime: {type: proxy, upstream: {pool: api}} ``` Several applications and routes may use the same pool; they share its members, their connections, their load and their health. Every key with its default is listed in [configuration.md](https://scalws.com/docs/guide/configuration/#upstreams). A member `address` is one of: | Form | Example | Notes | |---|---|---| | `ip:port` | `192.0.2.5:8080` | IPv4 address. | | `[ipv6]:port` | `[2001:db8::5]:8080` | IPv6 address in brackets. | | `host:port` | `api-1.internal:8080` | Resolved when a connection is made, like a single `upstream`. | | `unix:/absolute/path` | `unix:/run/api/api.sock` | Only as the single member of a plain-HTTP pool. | The port is required; a scheme (`http://`) or a path is refused. A member may appear only once in a pool, and a pool holds at most 10,000 members. A file pool without members loads with a warning and answers every request with `503` and `Retry-After: 1`. The single-address form, `upstream: "http://192.0.2.5:8080"` (or `https://...`, `unix:/...`), works as before: one address, no retries, no ejection, the same code path. `upstream` and `backends` are mutually exclusive. `preserve_host`, `connect_timeout`, `response_timeout` and `websocket` stay on the proxy runtime and apply to every member; the timeouts apply to each attempt. `tls_ca_file` belongs to the single `https://` address (a pool has `tls.ca_file`), and `client` is ignored on pool routes with a warning at load: pools use [`server.upstream_client`](https://scalws.com/docs/guide/configuration/#upstream_client). Site files cannot define pools, but their `proxy` (or `runtime`) can use one that the main file defines: `proxy: {upstream: {pool: api}}`. ## Choosing a member `balance` selects the algorithm. A member's `weight` (1 to 1000) is its relative share of requests, and ejected members are skipped by every algorithm. * **`round_robin`** (default): smooth weighted round robin, the nginx algorithm. Weights 5, 1 and 1 give the sequence `a a b a c a a` rather than five `a` in a row. * **`least_conn`**: the member with the fewest requests in flight divided by its weight; ties go round robin. Requests count until their response body is finished. WebSocket tunnels are not counted once the handshake is done. * **`hash`**: weighted rendezvous hashing of a key, so removing a member moves only the keys that pointed at it. `hash.by` is required: * `cookie`: affinity by cookie. A request without a valid affinity cookie gets a member by round robin, and the response sets the cookie (`hash.name`, default `SCALWS_AFFINITY`). Later requests with the cookie go to the same member as long as it is in the pool and not ejected; a cookie naming a member that is gone or ejected is replaced the same way. * `header`: the value of the request header `hash.name` (required). Requests without the header go round robin. * `client_ip`: the client address after [`trusted_proxies`](https://scalws.com/docs/guide/configuration/#trusted_proxies) (an IPv4-mapped IPv6 address hashes like the IPv4 address). `hash.name` is not used. ```yaml upstreams: shop: members: [{address: "198.51.100.10:8080"}, {address: "198.51.100.11:8080"}] balance: hash hash: by: cookie name: SHOP_NODE cookie_max_age: 1d # 0s (default): a session cookie ``` The affinity cookie value is 32 lower-case hex digits: the first 16 bytes of an HMAC-SHA256 of the member's address under a random key generated when the process starts. It does not reveal the address, and a client cannot use it to pick an arbitrary member. The cookie is sent as `Set-Cookie: SHOP_NODE=; Path=/; HttpOnly; SameSite=Lax`, plus `Max-Age=` when `cookie_max_age` is above zero and `Secure` when the request came over HTTPS. It is added next to the member's own `Set-Cookie` headers, and only when a member answered. WebSocket handshakes follow the cookie too. Affinity is per process. After a restart the key changes, so every client moves once to a member chosen anew. Several scalwsd instances behind `SO_REUSEPORT` or under scalws-controller do not share the key or the balancing state: a client that reaches another instance gets a new cookie there. Use `by: header` or `by: client_ip` when several instances must agree on the member. ## Passive health A **failure** is a connect error or connect timeout, no response head within `response_timeout`, a connection that broke after the request was written (reset, malformed response), or a `502`, `503` or `504` answer from the member. Any other answer is a success and resets the member's count. Failures caused by the client's request body are not the member's: a body over the route's `max_body` (or the global limit), a gap between body chunks longer than `body_read_timeout`, an aborted upload, or an upload still waiting for client data when `response_timeout` passes. They leave the member's health unchanged and are never retried; the client gets `413`, `408`, `400` or `504`. After `passive_health.failures` consecutive failures (default 3) the member is **ejected** for `passive_health.cooldown` (default `10s`): new requests skip it. When the cooldown ends the member is on probation: the next success brings it back fully, the next failure ejects it again at once. Requests that were already in flight when the member was ejected do not extend or end the ejection. * `failures: 0` turns passive health off: members are never ejected. * `failures` is at most 100; `cooldown` is between `1ms` and `1h`. * When every member of a pool is ejected, the one ejected longest ago is tried anyway, so passive health alone never makes a pool refuse traffic. A success there brings it back at once. There are no active health checks: ScalWS does not poll members. In Kubernetes, pod readiness and the member list maintained through the admin API play that role. ## Retries A request is sent again, to a member it has not tried yet, only when all of these hold: * nothing has been sent to the client yet; * the pool has more than one member and `retries` (default 1, at most 5) is not used up; * either the connection failed or timed out before the request was written (any method), or the method is idempotent (`GET`, `HEAD`, `OPTIONS`, `PUT`, `DELETE`) and the member answered `502`, `503` or `504`, sent no response head within `response_timeout`, or broke the connection before the response head (for example a reset keep-alive connection to a pod that is shutting down); * the request body can be sent again: no body, or a body with a `Content-Length` of at most 64 KiB, which is kept in memory for this. Larger bodies and bodies without a length (chunked uploads, even short ones) are streamed once and never retried. A connection that breaks after the request was written counts against the member's health. For other methods (`POST`, `PATCH`) such a request is never retried: the member may have acted on it. Once the response head was passed on, nothing is retried. WebSocket upgrades go to one member and are not retried. Separately from pool retries, a request sent on a kept-alive connection that the member had already closed (its idle timeout fired) is sent once more on a fresh connection to the same member: for any method when none of the request was written, and after it was written only for a bodyless idempotent request on a connection that sat idle for at least 1 s. A written `POST` or `PATCH` is never sent again, with or without a body. These resends count in `scalws_upstream_retries_total`, not in the pool retries. When every attempt failed, the client gets the last answer (the member's `502`/`503`/`504`, or ScalWS's own `502` or `504`). ## Weighted backends `backends` splits requests by weight across several pools, for example a canary release that sends 10 % of the traffic to a new version: ```yaml upstreams: api-v1: members: [{address: "192.0.2.5:8080"}, {address: "192.0.2.6:8080"}] api-v2: members: [{address: "192.0.2.7:8080"}] ``` ```yaml runtime: type: proxy backends: # weighted split per request; weights 0..=1000 - { pool: api-v1, weight: 90 } - { pool: api-v2, weight: 10 } ``` The pool is chosen per request by smooth weighted round robin over the backends; then the pool chooses its member as usual (affinity applies inside the chosen pool, retries stay in that pool). A weight of 0 sends no traffic, which keeps a pool configured but idle; at least one backend needs a weight above 0, and a route has at most 64 backends. A pool without members is skipped while another backend has members. Move the split by changing the weights and reloading. An entry can also be `{status: 500, weight: 1}` instead of `{pool, weight}`: that share of the requests is answered with the status (500 to 599) and a short text body, without contacting a pool. It takes part in the split like any other entry, is never skipped for lack of members, and does not count as an upstream failure in the metrics. Exactly one of `pool` and `status` is allowed per entry. ## TLS to members `scheme: https` connects to every member over TLS: ```yaml upstreams: secure-api: scheme: https members: [{address: "192.0.2.5:8443"}, {address: "192.0.2.6:8443"}] tls: ca_file: /etc/scalws/upstream-ca.pem # trusted in addition to the system roots server_name: api.internal # SNI and certificate name; default: member host ``` Certificates are always verified, against the system roots plus `tls.ca_file`; there is no switch to turn verification off. Without `tls.server_name` the member's host is the server name, which for members given by IP address means the certificate must name that address. `tls` is only allowed with `scheme: https`, and `unix:` members only in plain-HTTP pools. The system roots and `tls.ca_file` are read once per pool, when its first `https` member is created; every member of the pool shares that TLS configuration. A pool's TLS settings, a rotated `tls.ca_file` included, are therefore read when the pool is built and read again only when the pool's settings change (a reload with an unchanged pool keeps them). A changed `tls` or `scheme` setting builds a new pool (and reads them again); a new member list does not. ## Changing members at run time A pool with `members_from: api` takes its member list from the admin API instead of the configuration file. Its `members` in the file are optional and only seed the pool when it is created. ```yaml upstreams: api: members_from: api balance: least_conn ``` `PUT /v1/upstreams/{name}` replaces the whole member list, without a reload, on the admin socket or the [mTLS remote admin listener](https://scalws.com/docs/guide/configuration/#admin): ```sh curl --unix-socket /run/scalws/admin.sock -X PUT \ --data '{"members": [{"address": "192.0.2.5:8080", "weight": 2}, {"address": "192.0.2.6:8080"}]}' \ http://localhost/v1/upstreams/api ``` or with [`scalwsctl`](https://scalws.com/docs/reference/programs/scalwsctl/#upstreams): ```sh scalwsctl upstreams set api 192.0.2.5:8080=2 192.0.2.6:8080 scalwsctl upstreams # every pool with its members and their state scalwsctl upstreams show api ``` The answer is the pool as it is now (the same JSON as `GET /v1/upstreams/api`). The request is refused, and nothing changes, with: | Status | When | |---|---| | `400` | The body is not `{"members": [...]}`, or an address, weight (1 to 1000) or duplicate is invalid, or the list has more than 10,000 members. The message names the entry, for example `members[1].weight: must be between 1 and 1000`. | | `404` | No pool of that name. | | `409` | The pool has `members_from: file`. | | `413` | The body is larger than 2 MiB. | | `501` | The process runs under scalws-controller ([below](#under-scalws-controller)). | An empty list is accepted: the pool then answers `503` with `Retry-After: 1`. Members that stay in the list (same address) keep their connections, load and health; requests in flight to a removed member finish, its idle connections are closed and its metric series disappear. Changes are serialised with reloads and audited like every admin change. The full API is in the [admin API reference](https://scalws.com/docs/reference/admin-api/). **What survives what:** * **Reload.** Pools are matched by name. A pool whose settings (everything except `members`) are unchanged is kept with its connections and health; a file pool gets its new member list, and connections to members that stay are kept. A pool whose settings changed is rebuilt with fresh connections and health. An api pool keeps the members it has at run time in both cases (its `members` in the file are ignored); a pool that switches from `file` to `api` keeps the members it had. A pool removed from the file is gone, with its series. * **Restart.** Members of an api pool live in memory only. After a restart the pool starts with the file's `members` (or empty) until the next `PUT`, so whatever maintains the list must set it again after a restart. Per-application rate-limit and quota state is kept across a reload as well, as long as the limit's definition is unchanged. ## Observability Every request through a pool is visible in the access log and in five metric families on both metrics endpoints ([logging-metrics.md](https://scalws.com/docs/guide/logging-metrics/#upstreams)): | Metric | Labels | Meaning | |---|---|---| | `scalws_upstream_requests_total` | `pool`, `member`, `class` | Attempts by member and status class (`2xx`, `3xx`, `4xx`, `5xx`, `error`). | | `scalws_upstream_request_duration_seconds` | `pool` | Time to the response head, retries included. | | `scalws_upstream_member_ejected` | `pool`, `member` | 1 while passive health keeps the member out. | | `scalws_upstream_pool_retries_total` | `pool` | Requests sent to another member. | | `scalws_upstream_members` | `pool` | Members of the pool. | `scalws_upstream_pool_retries_total` counts retries to another member. It is not `scalws_upstream_retries_total`, which counts requests sent again because a kept-alive connection was stale, by tenant, application and protocol ([ADR-0043](https://scalws.com/docs/adr/0043-upstream-telemetry/)). The access log line has two fields for pools: `upstream_member` (the member address that answered, `-` for requests that did not go through a pool) and `upstream_attempts` (members tried, `0` without a pool, more than 1 after retries). `GET /v1/upstreams` shows the live state: requests in flight, consecutive failures and the end of an ejection per member. The console shows the same data under Monitor, Upstreams. ## Under scalws-controller Pools with `members_from: file` work under scalws-controller like any other configuration change: a reload starts a new generation. Each instance balances and tracks health on its own. Run-time members are not supported there in this version: `PUT /v1/upstreams/{name}` answers `501` on the controller's control socket and on the instances' admin sockets (`listener_mode: inherited` or `reuseport`), because the instances of one generation would disagree about the member list. Use `members_from: file` and reload, or run `scalwsd` on its own when members change at run time. `GET /upstreams` on the control socket shows the pools of one serving instance. ## Pools made by the ingress controller The [ingress controller](https://scalws.com/docs/guide/ingress-controller/) generates one pool per Service port (`p___`) with `members_from: api`, and keeps the members up to date with `PUT /v1/upstreams/{name}` from the Service's EndpointSlices, so scaling a backend needs no reload. Annotations such as `affinity`, `load-balance` and `backend-protocol` set the pool's `balance`, `hash` and `scheme`; canary annotations become weighted `backends`. Pools it generates should not be edited by hand: the next synchronisation overwrites them. ## Complete example Two applications share a pool with cookie affinity; an API route sends 10 % of its traffic to a canary pool. ```yaml server: listeners: - name: http address: "0.0.0.0:80" upstreams: web: members: - {address: "192.0.2.5:8080", weight: 2} - {address: "192.0.2.6:8080"} balance: hash hash: {by: cookie} passive_health: {failures: 3, cooldown: 30s} api-stable: members: [{address: "198.51.100.5:9000"}, {address: "198.51.100.6:9000"}] retries: 2 api-canary: members_from: api tenants: - name: acme applications: - name: www domains: [example.com, www.example.com] runtime: {type: proxy, upstream: {pool: web}, response_timeout: 30s} routes: - path: /api runtime: type: proxy backends: - {pool: api-stable, weight: 90} - {pool: api-canary, weight: 10} - name: shop domains: [shop.example.com] runtime: {type: proxy, upstream: {pool: web}} ``` Until the canary's members are set with `scalwsctl upstreams set api-canary ...`, the `/api` route sends everything to `api-stable`.