Console
ScalWS Console: architecture
ScalWS Console is the web administration of a ScalWS host, shipped with ScalWS the way
LiteSpeed WebAdmin ships with LiteSpeed Web Server: the same packages and release artifacts
(scalwsd, scalws-controller, scalwsctl, scalws-console), its own systemd unit
(scalws-console.service, enabled by the installer) and port 8090 by default. It stays a
separate process: its failure, restart or removal never touches client traffic.
Handoff: ScalWS-Console-HANDOFF.md. Status: C0 (design), C1 (foundation), C2 (safe
configuration) implemented; live traffic statistics and schema-driven settings for every
field of scalws.yaml and controller.yaml added; C3-C7 partly (see “Gaps”).
Topology
Browser ──HTTPS :8090──▶ scalws-console (management plane only)
│ Unix socket, HTTP/1.1 (controller API, ADR-0035)
├────────────▶ scalws-controller ──▶ scalwsd instances
│ status, scale, reload, autoscaling, cache purge
├─ reads/replaces /etc/scalws/scalws.yaml (validated, atomic)
├─ runs `scalwsd --check` on drafts
└─ /var/lib/scalws-console: history, audit log, TLS material
Client traffic ──▶ scalwsd instances directly (the console is never on this path)- The browser talks to the console only. The console talks to the controller only through the controller’s existing local API; it has no hook into scalwsd and no code in the data plane.
- Lifecycle truth stays in the controller (generations, rollouts, automatic rollback, listener generations, QUIC demux). The console is a narrow gateway: it adds authentication, roles, drafts, validation, diff, an audit trail and an operations view.
- The configuration file remains the single source of truth. The console writes it only
through the transaction in
docs/console-config-workflow.mdand lets the controller turn it into a generation. No competing store.
Process
cmd/scalws-console (Rust, same workspace and toolchain as the server):
| Module | Role |
|---|---|
app.rs | HTTPS listener (rustls, aws-lc-rs, h2 + HTTP/1.1), security headers, routing of /api/v1, static assets |
auth.rs | credentials file (argon2id), server-side sessions, login throttling, re-authentication |
rbac.rs | roles → capabilities, checked in every handler |
controller.rs | controller API client, rollout health of a generation |
txn.rs | drafts, validation, impact, apply with health check and automatic rollback, history, operations |
diff.rs, redact.rs | line diff; secret placeholders for views, drafts and audit |
audit.rs | append-only JSON-lines audit log |
tls.rs | operator certificate or a generated self-signed one |
stats.rs | live traffic statistics: scrapes instance /metrics, rates from counter deltas, 15-minute ring, server-sent events |
Resource use: 2 worker threads, at most 2 concurrent argon2 verifications, bounded
request bodies (5 MiB), bounded log/audit reads (2-4 MiB tails), at most 64 drafts and 200
operations in memory. The systemd unit sets Nice=10, CPUWeight=20, MemoryMax=256M.
Frontend
TypeScript without a framework (cmd/scalws-console/web, ~45 KB minified JS + 14 KB CSS),
bundled with esbuild and embedded in the binary (assets/, committed so building the
binary needs no Node.js). One telemetry request every 2 s feeds every page; polling stops
while the tab is hidden. Hash routing, no third-party runtime code, no external fonts or
CDNs (the CSP allows 'self' only). All server text is inserted with textContent.
Why no React/Preact: the console is a set of tables, forms and a text editor; plain DOM
helpers (dom.ts) keep the bundle small and the CSP strict, and there is no build-time
dependency besides esbuild and TypeScript.
Live statistics
Every 2 s the console reads /metrics of each serving instance (addresses from the
controller’s status, 127.0.0.1 only), turns counter increases into rates (a decrease is a
process restart: the new value counts), sums the instances and keeps 450 samples (15 min)
in memory. Browsers get one server-sent event stream: a snapshot (compact history and the
latest full sample), then one sample every 2 s; the stream closes while the tab is hidden
and when the session ends.
| Shown | From (instance metrics) |
|---|---|
| requests/s total, per site, per listener, per method, per status class | scalws_http_requests_total{tenant,app,listener,method,status} |
| latency p50/p95/p99 total and per site | scalws_http_request_duration_seconds_bucket (bucket increases, linear interpolation) |
| bandwidth | scalws_http_response_body_bytes_total |
| in flight, open connections, accepts/s | scalws_http_requests_in_flight, scalws_connections_open, scalws_connections_accepted_total |
| TLS handshakes/s and p95 per listener | scalws_tls_handshake_seconds |
| cache hit ratio per site | scalws_cache_lookups_total{result} (hit + stale over hit + stale + miss) |
| upstream failures, security blocks, tenant quota refusals, refusals before routing | scalws_upstream_failures_total, scalws_security_blocked_total, scalws_tenant_limited_total, scalws_rejections_total |
| HTTP/3 GOAWAY and connection closes | scalws_h3_goaway_sent_total, scalws_h3_connection_closes_total |
Sites are applications; their domains come from the configuration (each application lists its domains). Per-domain counts come from the counters below; the data plane does not count per HTTP version. Cost: one scrape per instance per 2 s (the controller already scrapes every 1 s), parsed in the console process.
Per-domain counters (metrics.hosts, default on)
The data plane counts per configured domain pattern (scalws_core::host_metrics):
| Series | Labels |
|---|---|
scalws_http_host_requests_total | tenant, app, host, status (1xx … 5xx) |
scalws_http_host_response_bytes_total | tenant, app, host |
scalws_http_host_request_bytes_total | tenant, app, host (body bytes read) |
scalws_http_host_cache_total | tenant, app, host, result (hit includes stale, miss) |
scalws_http_host_requests_in_flight | tenant, app, host |
hostis a configured domain pattern (a wildcard is one series), never the request’s Host header: cardinality is set by the configuration. Hosts that match no domain, and domains pastmetrics.max_hosts(default 1000, at most 16383), are counted astenant="_unmatched",host="_other".- Request path: the host router already resolves the domain; its entry carries a small
integer id, and the counters are plain atomics indexed by it, sharded per event loop
(each loop writes its own cache lines;
/metricssums the shards). No label hashing, no lock, no allocation per request. Memory: about 5 KB per 64 domains per event loop that serves them. - A reload keeps the counters of domains that stay; removed domains disappear once the old generation drained (their ids are reused).
- Latency is not per domain (a histogram per domain costs memory and update time with thousands of domains); it stays per application. A per-domain histogram could be added later as an opt-in.
- Not split by HTTP version yet.
The console’s Traffic page shows a Domains tab (and the Overview “Top domains”): requests/s and its 3-minute trend, 2xx/3xx/4xx/5xx shares, bytes out and in, cache hit ratio, in-flight requests, and totals since each instance started. The Applications tab keeps the per-application view with latency percentiles.
Configuration forms from the schema
The settings pages are generated from JSON Schemas of the configuration structs:
scalws_config::json_schema() (feature schema of scalws-config: derives
schemars::JsonSchema on the model; parsing and validation are unchanged) and
scalws-controller config-schema (derived from ControllerConfig). Field names, types,
defaults, units (x-scalws-unit: duration, bytes, cpu, seconds, milliseconds) and doc
comments come from the code, so the forms cannot offer a field the parser does not know and
cannot miss one: every server field not placed in a named section appears under
Advanced. scalws-controller check-config --config FILE validates controller.yaml drafts.
API (/api/v1, JSON)
| Method, path | Capability | Notes |
|---|---|---|
POST login | none | same-origin check, throttled; sets __Host-scalws_console |
GET session, POST logout, POST reauth | session | reauth opens a 5-minute window for sensitive actions |
GET telemetry | server.read | controller status + limits + running operations |
GET about | server.read | versions, paths, capability matrix |
GET config/current, GET config/history[/<seq>] | config.read | redacted; ?target=controller for controller.yaml |
GET/POST drafts, GET/PUT/DELETE drafts/<id>, GET/PUT drafts/<id>/model | config.read / config.draft | owner or administrator edits |
POST drafts | config.draft | {target: "scalws" or "controller", from?: seq} |
POST drafts/<id>/validate | config.draft | validation, changed settings (path, before, after; secrets as “(secret)”), redacted diff, impact, conflict flag |
POST drafts/<id>/apply | config.apply + re-auth | {base_hash, idempotency_key} → 202 {operation} |
POST config/rollback | config.rollback + re-auth | {seq, base_hash, idempotency_key} |
GET operations[/<id>] | config.read | states and steps |
POST runtime/scale, POST runtime/autoscaling | runtime.manage | within instances.min..max |
POST cache/purge | cache.purge + re-auth | all or host (+ path prefix) |
GET tenants/usage | server.read | CPU, memory, disk I/O of each tenant over one second, with its limits (Resources page) |
POST tenants/<tenant>/limits | runtime.manage | {cpu?, memory_percent?, io_weight?, io_bandwidth_mib?} or {clear: all|cpu|memory|io}: run-time limits on every instance until the controller restarts |
GET logs | logs.read | controller log tail, filtered, text only |
GET stats, GET stats/stream | server.read | live statistics: snapshot; server-sent events |
GET schema | config.read | JSON Schemas of scalws.yaml and controller.yaml |
GET audit | audit.read | search |
GET users | users.read | accounts (read-only) |
Every state-changing request needs the session cookie, the X-CSRF-Token of the session
and a same-origin Origin/Referer. The API is versioned by path; the UI reads the
capability list of its session and the capability matrix of about and hides what is not
available.
Controller API gap report
What the console needs and what scalws-controller offers today:
| Need | Controller today | Console behaviour |
|---|---|---|
| State, instances, decisions, distribution, listeners, QUIC | GET /status | used as is |
| Apply a configuration | POST /reload re-reads controller.config | console writes the file first (atomic, validated), then reloads |
| Know whether a generation is healthy | instance phases + last_rollback in status | console polls and decides (controller.rs::rollout) |
| Desired instances, autoscaling mode, cache purge | POST /scale/N, /autoscaling/M, /cache/purge | used as is |
| Tenant usage and run-time limits | GET /tenants/usage, POST /tenants/<tenant>/limits (kept for instances that start later) | used as is |
| Drain or restart one instance | none | hidden in the UI |
Change instances.*, CPUs, autoscaling thresholds | controller.yaml, read at start | edited like scalws.yaml, validated by check-config, applied by restarting the controller (--controller-restart-command) with automatic restore if it does not settle |
| Per-instance p99 | sampled internally, not in status | not shown; pressure “latency” share shown instead |
| Validate a configuration without applying | none (reload validates and rejects) | console runs scalwsd --check itself |
| Apply with an expected base generation | none | console checks the file hash (optimistic concurrency) |
| Unprivileged management | socket is 0600 for the controller user | console runs as root with a fenced systemd unit |
Added to scalws-controller (additive, control plane only): config-schema (hidden)
and check-config --config FILE. Proposed further additive endpoints: POST /config (validate + write + reload in one step, with an expected base
hash) so the console can run as an unprivileged user; per-instance POST /instances/<id>/drain;
per-instance p99 in status.
Failure behaviour
- Console down: no effect on traffic or on the controller. Drafts are lost (in memory); history, audit log and operations already finished are on disk.
- Controller down or restarting: pages show “Controller unavailable” with the error; actions fail with nothing changed; an apply that was waiting for health keeps polling until its timeout and then restores the previous file.
- Credentials file unreadable or too open: logins are refused, the reason is logged.
Gaps (later phases)
- C3: certificate inventory with expiry (needs a certificate-info endpoint), route editor with conflict detection (routes are editable in the raw editor only).
- C5: application runtime pages beyond the structured static/proxy/respond editor.
- C6: QUIC connection, stream and drain metrics (the controller exposes demux status only).
- C7: WebAuthn/TOTP, per-site scopes, journald log reader for packaged installs
(
--controller-logreads a file; under systemd the controller logs to the journal).