Decision records

ADR-0015: Deterministic diagnostics

• Status: Accepted (2026-10-05)

Context

Handoff §10/M7: diagnostics with root-cause correlation views; eBPF is optional enrichment for process, network and block-I/O correlation and basic diagnosis must never depend on it. Diagnostic enrichment happens off the hot path. scalwsctl diagnose is part of the CLI surface (handoff §9).

Decision

• scalws-diag is pure, like scalws-policy: it maps one evidence snapshot per application to ordered findings. No I/O, no clock, no AI.
• Evidence per window (collected by the server’s analysis task, which also runs the optimizer):
  • request telemetry: requests, 5xx, latency to response headers (p50/p95/p99), upstream failures by reason (connect, timeout, unavailable, other), tenant quota rejections, cache hit ratio;
  • runtime: lifecycle state, ready/target workers, restarts in the window, utilisation;
  • cgroup v2 (when enforced): memory usage and limit, OOM kills in the window, pressure stall information some avg10 for memory, CPU and I/O (*.pressure), CPU throttling.
• Phases: scalws does not queue requests (quotas reject, they do not wait), so the phases are time to response headers (routing + handler + upstream until headers) and transfer (body until finished). Upstream connect time inside pooled clients is not measured separately. TLS handshake duration is a metric per listener.
• Findings carry a stable code, severity (critical, warning, info), a summary, evidence lines and a suggestion. Resource causes (crash_loop, runtime_down, oom_kill, memory_pressure, cpu_pressure, io_pressure) are linked to symptom findings in the same window (upstream_timeouts, upstream_unreachable, app_errors, saturated, slow) as related causes — correlation, stated as such, not proof. Thresholds are constants in scalws-diag.
• Recent requests: a bounded ring buffer per application keeps the last server.diagnostics.recent_requests (default 50) requests that failed with 5xx or took longer than server.diagnostics.slow_request (default 1 s): time, method, path without query, status, handler, failure reason, cache result, time to headers, total time, bytes, completion.
• Exposure: GET /diagnose (all applications) and GET /diagnose?app=tenant/app on the loopback admin listener; scalwsctl diagnose [tenant/app]; findings counted in scalws_diagnostic_findings_total{tenant,app,code} when they first appear.
• eBPF and OpenTelemetry trace export are not part of this stage. Both stay optional later additions; every finding above is produced from cgroup files, process state and in-process counters.

Consequences

• Findings lag by up to one window (server.optimizer.interval).
• The ring buffer holds request paths (no query strings, no headers), like the access log.