Decision records

ADR-0014: Deterministic optimizer (policy engine)

• Status: Accepted (2026-10-05)

Context

Handoff §8: a policy engine fed with rolling telemetry; bounded actions (runtime worker targets, connection-pool bounds, selected cache parameters); every rule has min/max, cooldown, hysteresis, preconditions and rollback conditions; recommend-only by default, automatic mode opt-in per application; every change audited (measurements, rule, old/new value, reason, observed outcome). The local LLM (M8) is not the optimizer.

Decision

• scalws-policy is pure: it receives one telemetry snapshot per application and evaluation window and returns decisions. No I/O, no clocks of its own, no randomness; identical input sequences give identical decisions (unit-tested with synthetic telemetry).
• Telemetry per application (window = server.optimizer.interval, default 10 s): requests, 5xx ratio, latency p50/p95/p99 (power-of-two microsecond buckets: accurate to a factor of two, enough for the generous rollback thresholds), average in-flight requests (sampled every second), ready/target workers, tenant memory pressure (memory.current / memory.max from cgroups, when limits are enforced), cache hit ratio. Latency is time to response headers.
• Worker utilisation = average in-flight / (ready workers × workers.target_concurrency; default 4, or 1 for synchronous WSGI).
• Rules (parameters in scalws-policy, documented in its source):
  • scale_up: utilisation ≥ 0.75 in 3 consecutive windows, ≥ 20 requests per window, memory pressure < 0.85, below workers.max → target + 1; cooldown 60 s.
  • scale_down: utilisation ≤ 0.25 in 6 consecutive windows, above workers.min → target − 1; cooldown 120 s.
  • Hysteresis: the up/down thresholds are far apart and a streak resets as soon as a window leaves the band.
  • Rollback: 3 windows after an applied change, if the 5xx ratio rose by more than 5 points or p95 grew by more than 50 % (and above 50 ms) compared with the window before the change, the old value is restored and the rule is suppressed for 10 minutes.
  • Advisory only (never acted on): memory_pressure (≥ 0.9 for 3 windows), error_rate (≥ 20 % 5xx with ≥ 20 requests).
• Modes from workers.autoscale: off, recommend (default), auto.
• Actions in this stage: live worker targets of managed Node.js worker pools. Endpoints for workers.max are reserved when the instance is prepared; scaling up starts workers in free slots, scaling down takes the highest slot out of rotation, waits for its in-flight requests (up to stop_timeout) and stops it. Python receives recommendations only when it runs as one server with a fixed --workers count; with process_pool: true (since 2026-10-06) scalws runs single-worker processes in a scalable pool and acts on Python like on Node.js. PHP-FPM sizes its own pool (pm = dynamic) and receives advisories only. Connection-pool and cache parameters are not changed automatically yet.
• A configuration reload that changes the runtime spec creates a new instance at workers.min; an unchanged spec keeps the instance and its current worker count.
• server.optimizer.enabled and interval apply on reload; journal_size at start.
• Audit: every decision (recommended, applied, rolled back, outcome) is logged under target scalws::audit and kept in a bounded in-memory journal served as JSON at GET /optimizer on the loopback admin listener; scalwsctl recommendations reads it.

Consequences

• Telemetry is application-wide; an application with several runtimes is optimised on its default runtime.
• The journal is lost on restart (the structured log is the durable record).