Decision records

ADR-0035: Instance lifecycle, draining, configuration generations and recovery

Status: accepted (design, D0) — 2026-10-07

Lifecycle

Stopped → Starting → Warming → Ready → Active → Draining → Stopped
             │          │        │        │          │
             └──────────┴────────┴────────┴──────────┴──→ Failed (exit, start timeout,
                                                          drain timeout → forced stop)

• Starting: process spawned with its instance id, generation and CPU set. It binds its reuseport sockets only when told to activate — so Starting/Warming receive no traffic.
• Warming: configuration prepared, runtimes and caches initialised; readiness probe answers {"generation": N, "state": "warming"}.
• Ready: readiness passed; still not bound. Active: listeners bound (joins the reuseport group); the kernel starts sending new connections.
• Draining: listening sockets closed first (queued connections migrate, tcp_migrate_req), keep-alive connections told to close (Connection: close, HTTP/2 GOAWAY), in-flight requests finish until drain_timeout (default 30 s), then graceful shutdown; forced kill only after drain_timeout + kill_grace.
• Liveness ≠ readiness ≠ saturation: a saturated instance is healthy; restarts happen only on exit or failed liveness, never because of load.

Instances have stable ids (web-0, web-1, …), and the controller records PID, CPU set, generation, state and timestamps in its state directory (/run/scalws-controller/instances/<id>.json, 0600, written atomically).

Configuration generations

A generation is a validated, immutable copy of the configuration: <state>/generations/<n>/scalws.yaml written to a temporary name, fsynced and renamed; never edited in place. Instances start against an explicit generation path and report it in readiness. A reload validates first, then rolls instances one by one (start new generation instance → Ready → Active → drain one old), so capacity never drops below the current count. Rollback is the same procedure towards the previous generation. An instance is never started against a partially written file.

Failure recovery

• Instance crash: Failed, restarted with exponential backoff (1 s → 60 s); more than max_restarts (5) in restart_window (5 min) → stays Failed, alert, no fork storm.
• Controller restart: reads the state directory, verifies each recorded PID (process start time and executable), re-adopts healthy instances and reconciles towards the desired count — it never stops healthy instances to “start clean”.
• Host limits: max_instances, cpus_per_instance, reserved system/controller and application CPUs, min_free_ram; optional cgroup v2 CPU/memory/pids per instance. If cgroup delegation is unavailable the controller says so in its status and logs (limits unenforced), it does not pretend.
• Emergency switch: autoscaling: off (config or scalws-controller disable) freezes the instance count immediately; lifecycle management continues.

Anti-thrashing

Scale-out window 3–5 s above threshold, scale-in window 60 s below, cooldown after any action (default 30 s), start and drain timeouts, at most one action in flight, hysteresis between scale_out_above and scale_in_below.