ScalWS
Get started
Get started

Decision records

ADR-0021: Multi-node control plane (design only)

  • Status: Proposed (2026-10-06) — design only; nothing is implemented
  • Handoff M10: “multi-node control plane only after earlier acceptance gates pass” — they pass; this ADR fixes the direction before any code.

Context

Today one scalwsd serves a host. Operators with several hosts need: one source of configuration, coordinated rollouts and rollback, certificates that are ordered once and used everywhere, and one view of status and diagnostics. The data plane must keep the properties of a single node: validate-before-apply, atomic swap, no dependence on remote services on the request path, and continued service when anything central is down.

Goals

  • Declarative desired state for a fleet: which nodes serve which tenants/applications.
  • Safe rollout: validate centrally and on every node, canary first, automatic halt on failure, one-step rollback to the previous fleet revision.
  • Nodes keep serving their last good configuration when the control plane is unreachable.
  • Shared ACME certificates (one order per name for the fleet).
  • Aggregated status, diagnostics and optimizer journals.

Non-goals (first phases)

  • Consensus inside the data plane, shared caches, or cross-node rate limits (all stay per node, documented as such).
  • Scheduling applications onto nodes automatically (placement is declared).
  • A new distributed database; no etcd/Consul dependency (handoff: avoid unnecessary infrastructure).

Options

  1. Shared files (git/rsync/object storage) + per-node scalwsctl reload: simplest, but no rollout control, no status feedback, secrets copied by hand.
  2. Pull-based control plane (scalws-controld): nodes poll for signed configuration bundles over mTLS and report back; the controller orchestrates rollout by publishing revisions per node group.
  3. Push-based (controller connects to every node’s admin API): needs inbound access to nodes and long-lived connections; rollout coupled to controller uptime.
  4. Raft cluster of scalws nodes: strongest consistency, highest complexity, failure modes on the serving path.

Decision (direction)

Option 2, in phases.

  • Bundle: a fleet revision = the validated configuration for a node group + referenced files (certificates from the controller’s ACME, CA bundles) + metadata (revision number, parent, author, time). Bundles are signed (Ed25519) by the controller; nodes verify the signature, then run the existing validate → prepare → swap path. Secrets never leave the controller unencrypted at rest; transport is mTLS with per-node client certificates.
  • Node agent: lives in scalwsd (no extra daemon on nodes): polls GET /v1/fleet/desired every N seconds with its current revision (long-poll), applies newer revisions through Control::apply, and reports status, apps, certs, diagnose summaries. Failure to apply is reported and the node stays on its revision — exactly today’s reload semantics. Control plane unreachable: keep serving, retry with backoff.
  • Controller (scalws-controld, new binary): stores revisions (SQLite, file-backed, replicated by the operator’s backup — not a distributed store), runs ACME once per name (reusing scalws-server::acme logic) and distributes certificates in bundles, exposes an admin API (mTLS) and scalwsctl --fleet ….
  • Rollout: per node group: canary node(s) → wait for healthy reports (no new critical diagnostic findings, apply succeeded) → remaining nodes in batches; any failure halts the rollout; rollback publishes the parent revision. Rollback is a new revision, like the single-node controller history (ADR-0017).
  • Observability: nodes keep exposing Prometheus metrics locally (federation by the operator’s Prometheus); the controller aggregates status/diagnostics for scalwsctl --fleet.

Consequences and open questions

  • Per-node semantics stay explicit: caches, per-IP limits, tenant quotas and optimizer decisions are per node; fleet-wide quotas would need a separate design.
  • Controller availability: an outage blocks changes, not serving. A warm standby restored from SQLite backups is the first availability answer; HA controller later.
  • Open: bundle size limits for many tenants (per-tenant partial bundles?), certificate private key distribution policy (per-node keys vs shared), node identity bootstrap (join tokens), and how the AI advisor and optimizer behave fleet-wide (likely: per node, aggregated view only).

Implementation prerequisites

  1. A stable admin API surface (ADR-0017) — done.
  2. Bundle format and signature verification as a library with tests.
  3. Node agent behind a feature/config switch, end-to-end tested with two nodes and a controller in containers before any production use.