Users' guide

Load balancing

A proxy runtime can send its requests to a named pool of members instead of one address. ScalWS then balances requests across the members, can keep a client on one member (affinity), takes members that keep failing out for a while (passive health), retries idempotent requests on another member, splits traffic by weight across pools, and lets you replace the member list at run time through the admin API, without a reload.

Design record: ADR-0049. Route matching and route policies (rewrites, CORS, basic authentication, address lists) are in configuration.md and ADR-0050.

Contents:

• Pools and members
• Choosing a member
• Passive health
• Retries
• Weighted backends
• TLS to members
• Changing members at run time
• Observability
• Under scalws-controller
• Complete example

Pools and members

Pools are defined once, in the top-level upstreams section of the main configuration file, and used by name from any proxy runtime (an application’s runtime or a route’s):

upstreams:
  api:
    members:
      - { address: "192.0.2.5:8080", weight: 1 }
      - { address: "192.0.2.6:8080" }  # weight defaults to 1, range 1..=1000
    members_from: file               # file (default) | api
    balance: round_robin             # round_robin (default) | least_conn | hash
    passive_health:
      failures: 3                    # consecutive failures before a member is ejected
      cooldown: 10s                  # how long it stays out
    retries: 1                       # 0..=5, to a different member
    keepalive: 32                    # idle connections kept per member
    scheme: http                     # http | https

tenants:
  - name: acme
    applications:
      - name: api
        domains: [api.example.com]
        runtime: {type: proxy, upstream: {pool: api}}

Several applications and routes may use the same pool; they share its members, their connections, their load and their health. Every key with its default is listed in configuration.md.

A member address is one of:

The port is required; a scheme (http://) or a path is refused. A member may appear only once in a pool, and a pool holds at most 10,000 members. A file pool without members loads with a warning and answers every request with 503 and Retry-After: 1.

The single-address form, upstream: "http://192.0.2.5:8080" (or https://..., unix:/...), works as before: one address, no retries, no ejection, the same code path. upstream and backends are mutually exclusive. preserve_host, connect_timeout, response_timeout and websocket stay on the proxy runtime and apply to every member; the timeouts apply to each attempt. tls_ca_file belongs to the single https:// address (a pool has tls.ca_file), and client is ignored on pool routes with a warning at load: pools use server.upstream_client.

Site files cannot define pools, but their proxy (or runtime) can use one that the main file defines: proxy: {upstream: {pool: api}}.

Choosing a member

balance selects the algorithm. A member’s weight (1 to 1000) is its relative share of requests, and ejected members are skipped by every algorithm.

• round_robin (default): smooth weighted round robin, the nginx algorithm. Weights 5, 1 and 1 give the sequence a a b a c a a rather than five a in a row.
• least_conn: the member with the fewest requests in flight divided by its weight; ties go round robin. Requests count until their response body is finished. WebSocket tunnels are not counted once the handshake is done.
• hash: weighted rendezvous hashing of a key, so removing a member moves only the keys that pointed at it. hash.by is required:
  • cookie: affinity by cookie. A request without a valid affinity cookie gets a member by round robin, and the response sets the cookie (hash.name, default SCALWS_AFFINITY). Later requests with the cookie go to the same member as long as it is in the pool and not ejected; a cookie naming a member that is gone or ejected is replaced the same way.
  • header: the value of the request header hash.name (required). Requests without the header go round robin.
  • client_ip: the client address after trusted_proxies (an IPv4-mapped IPv6 address hashes like the IPv4 address). hash.name is not used.

upstreams:
  shop:
    members: [{address: "198.51.100.10:8080"}, {address: "198.51.100.11:8080"}]
    balance: hash
    hash:
      by: cookie
      name: SHOP_NODE
      cookie_max_age: 1d     # 0s (default): a session cookie

The affinity cookie value is 32 lower-case hex digits: the first 16 bytes of an HMAC-SHA256 of the member’s address under a random key generated when the process starts. It does not reveal the address, and a client cannot use it to pick an arbitrary member. The cookie is sent as Set-Cookie: SHOP_NODE=<value>; Path=/; HttpOnly; SameSite=Lax, plus Max-Age=<seconds> when cookie_max_age is above zero and Secure when the request came over HTTPS. It is added next to the member’s own Set-Cookie headers, and only when a member answered. WebSocket handshakes follow the cookie too.

Affinity is per process. After a restart the key changes, so every client moves once to a member chosen anew. Several scalwsd instances behind SO_REUSEPORT or under scalws-controller do not share the key or the balancing state: a client that reaches another instance gets a new cookie there. Use by: header or by: client_ip when several instances must agree on the member.

Passive health

A failure is a connect error or connect timeout, no response head within response_timeout, a connection that broke after the request was written (reset, malformed response), or a 502, 503 or 504 answer from the member. Any other answer is a success and resets the member’s count.

Failures caused by the client’s request body are not the member’s: a body over the route’s max_body (or the global limit), a gap between body chunks longer than body_read_timeout, an aborted upload, or an upload still waiting for client data when response_timeout passes. They leave the member’s health unchanged and are never retried; the client gets 413, 408, 400 or 504.

After passive_health.failures consecutive failures (default 3) the member is ejected for passive_health.cooldown (default 10s): new requests skip it. When the cooldown ends the member is on probation: the next success brings it back fully, the next failure ejects it again at once. Requests that were already in flight when the member was ejected do not extend or end the ejection.

• failures: 0 turns passive health off: members are never ejected.
• failures is at most 100; cooldown is between 1ms and 1h.
• When every member of a pool is ejected, the one ejected longest ago is tried anyway, so passive health alone never makes a pool refuse traffic. A success there brings it back at once.

There are no active health checks: ScalWS does not poll members. In Kubernetes, pod readiness and the member list maintained through the admin API play that role.

Retries

A request is sent again, to a member it has not tried yet, only when all of these hold:

• nothing has been sent to the client yet;
• the pool has more than one member and retries (default 1, at most 5) is not used up;
• either the connection failed or timed out before the request was written (any method), or the method is idempotent (GET, HEAD, OPTIONS, PUT, DELETE) and the member answered 502, 503 or 504, sent no response head within response_timeout, or broke the connection before the response head (for example a reset keep-alive connection to a pod that is shutting down);
• the request body can be sent again: no body, or a body with a Content-Length of at most 64 KiB, which is kept in memory for this. Larger bodies and bodies without a length (chunked uploads, even short ones) are streamed once and never retried.

A connection that breaks after the request was written counts against the member’s health. For other methods (POST, PATCH) such a request is never retried: the member may have acted on it. Once the response head was passed on, nothing is retried. WebSocket upgrades go to one member and are not retried.

Separately from pool retries, a request sent on a kept-alive connection that the member had already closed (its idle timeout fired) is sent once more on a fresh connection to the same member: for any method when none of the request was written, and after it was written only for a bodyless idempotent request on a connection that sat idle for at least 1 s. A written POST or PATCH is never sent again, with or without a body. These resends count in scalws_upstream_retries_total, not in the pool retries. When every attempt failed, the client gets the last answer (the member’s 502/503/504, or ScalWS’s own 502 or 504).

Weighted backends

backends splits requests by weight across several pools, for example a canary release that sends 10 % of the traffic to a new version:

upstreams:
  api-v1:
    members: [{address: "192.0.2.5:8080"}, {address: "192.0.2.6:8080"}]
  api-v2:
    members: [{address: "192.0.2.7:8080"}]

runtime:
  type: proxy
  backends:                 # weighted split per request; weights 0..=1000
    - { pool: api-v1, weight: 90 }
    - { pool: api-v2, weight: 10 }

The pool is chosen per request by smooth weighted round robin over the backends; then the pool chooses its member as usual (affinity applies inside the chosen pool, retries stay in that pool). A weight of 0 sends no traffic, which keeps a pool configured but idle; at least one backend needs a weight above 0, and a route has at most 64 backends. A pool without members is skipped while another backend has members. Move the split by changing the weights and reloading.

An entry can also be {status: 500, weight: 1} instead of {pool, weight}: that share of the requests is answered with the status (500 to 599) and a short text body, without contacting a pool. It takes part in the split like any other entry, is never skipped for lack of members, and does not count as an upstream failure in the metrics. Exactly one of pool and status is allowed per entry.

TLS to members

scheme: https connects to every member over TLS:

upstreams:
  secure-api:
    scheme: https
    members: [{address: "192.0.2.5:8443"}, {address: "192.0.2.6:8443"}]
    tls:
      ca_file: /etc/scalws/upstream-ca.pem   # trusted in addition to the system roots
      server_name: api.internal              # SNI and certificate name; default: member host

Certificates are always verified, against the system roots plus tls.ca_file; there is no switch to turn verification off. Without tls.server_name the member’s host is the server name, which for members given by IP address means the certificate must name that address. tls is only allowed with scheme: https, and unix: members only in plain-HTTP pools.

The system roots and tls.ca_file are read once per pool, when its first https member is created; every member of the pool shares that TLS configuration. A pool’s TLS settings, a rotated tls.ca_file included, are therefore read when the pool is built and read again only when the pool’s settings change (a reload with an unchanged pool keeps them). A changed tls or scheme setting builds a new pool (and reads them again); a new member list does not.

Changing members at run time

A pool with members_from: api takes its member list from the admin API instead of the configuration file. Its members in the file are optional and only seed the pool when it is created.

upstreams:
  api:
    members_from: api
    balance: least_conn

PUT /v1/upstreams/{name} replaces the whole member list, without a reload, on the admin socket or the mTLS remote admin listener:

curl --unix-socket /run/scalws/admin.sock -X PUT \
  --data '{"members": [{"address": "192.0.2.5:8080", "weight": 2}, {"address": "192.0.2.6:8080"}]}' \
  http://localhost/v1/upstreams/api

or with scalwsctl:

scalwsctl upstreams set api 192.0.2.5:8080=2 192.0.2.6:8080
scalwsctl upstreams              # every pool with its members and their state
scalwsctl upstreams show api

The answer is the pool as it is now (the same JSON as GET /v1/upstreams/api). The request is refused, and nothing changes, with:

An empty list is accepted: the pool then answers 503 with Retry-After: 1. Members that stay in the list (same address) keep their connections, load and health; requests in flight to a removed member finish, its idle connections are closed and its metric series disappear. Changes are serialised with reloads and audited like every admin change. The full API is in the admin API reference.

What survives what:

• Reload. Pools are matched by name. A pool whose settings (everything except members) are unchanged is kept with its connections and health; a file pool gets its new member list, and connections to members that stay are kept. A pool whose settings changed is rebuilt with fresh connections and health. An api pool keeps the members it has at run time in both cases (its members in the file are ignored); a pool that switches from file to api keeps the members it had. A pool removed from the file is gone, with its series.
• Restart. Members of an api pool live in memory only. After a restart the pool starts with the file’s members (or empty) until the next PUT, so whatever maintains the list must set it again after a restart.

Per-application rate-limit and quota state is kept across a reload as well, as long as the limit’s definition is unchanged.

Observability

Every request through a pool is visible in the access log and in five metric families on both metrics endpoints (logging-metrics.md):

scalws_upstream_pool_retries_total counts retries to another member. It is not scalws_upstream_retries_total, which counts requests sent again because a kept-alive connection was stale, by tenant, application and protocol (ADR-0043).

The access log line has two fields for pools: upstream_member (the member address that answered, - for requests that did not go through a pool) and upstream_attempts (members tried, 0 without a pool, more than 1 after retries).

GET /v1/upstreams shows the live state: requests in flight, consecutive failures and the end of an ejection per member. The console shows the same data under Monitor, Upstreams.

Under scalws-controller

Pools with members_from: file work under scalws-controller like any other configuration change: a reload starts a new generation. Each instance balances and tracks health on its own.

Run-time members are not supported there in this version: PUT /v1/upstreams/{name} answers 501 on the controller’s control socket and on the instances’ admin sockets (listener_mode: inherited or reuseport), because the instances of one generation would disagree about the member list. Use members_from: file and reload, or run scalwsd on its own when members change at run time. GET /upstreams on the control socket shows the pools of one serving instance.

Pools made by the ingress controller

The ingress controller generates one pool per Service port (p_<namespace>_<service>_<port>) with members_from: api, and keeps the members up to date with PUT /v1/upstreams/{name} from the Service’s EndpointSlices, so scaling a backend needs no reload. Annotations such as affinity, load-balance and backend-protocol set the pool’s balance, hash and scheme; canary annotations become weighted backends. Pools it generates should not be edited by hand: the next synchronisation overwrites them.

Complete example

Two applications share a pool with cookie affinity; an API route sends 10 % of its traffic to a canary pool.

server:
  listeners:
    - name: http
      address: "0.0.0.0:80"

upstreams:
  web:
    members:
      - {address: "192.0.2.5:8080", weight: 2}
      - {address: "192.0.2.6:8080"}
    balance: hash
    hash: {by: cookie}
    passive_health: {failures: 3, cooldown: 30s}
  api-stable:
    members: [{address: "198.51.100.5:9000"}, {address: "198.51.100.6:9000"}]
    retries: 2
  api-canary:
    members_from: api

tenants:
  - name: acme
    applications:
      - name: www
        domains: [example.com, www.example.com]
        runtime: {type: proxy, upstream: {pool: web}, response_timeout: 30s}
        routes:
          - path: /api
            runtime:
              type: proxy
              backends:
                - {pool: api-stable, weight: 90}
                - {pool: api-canary, weight: 10}
      - name: shop
        domains: [shop.example.com]
        runtime: {type: proxy, upstream: {pool: web}}

Until the canary’s members are set with scalwsctl upstreams set api-canary ..., the /api route sends everything to api-stable.