Security
Threat Model
Status: living document, revised at every milestone’s security review. Scope today: M1 serving core (listeners, TLS, HTTP/1.1 + HTTP/2, routing, static files, reverse proxy, metrics endpoint, config reload) and M2 managed runtimes (process supervision, PHP-FPM/FastCGI, Node, Python, WebSocket tunnels, HTTPS upstreams).
1. Assets
| Asset | Why it matters |
|---|---|
| Availability of the server and of unrelated tenants | Shared process serves many tenants. |
| TLS private keys | Impersonation of hosted domains. |
| Files outside configured document roots | Source code, .env, credentials, other tenants’ data. |
| Upstream applications | Must only receive requests the operator routed to them; must not be fooled by spoofed forwarding headers. |
| Configuration and its integrity | A bad or malicious config can expose files or take the server down. |
| Logs and metrics | May leak secrets or become a DoS vector via cardinality. |
2. Trust boundaries
- Network clients → listener. Fully untrusted: bytes, timing, TLS ClientHello, headers, bodies.
- scalwsd → upstream applications. Semi-trusted: apps are tenant code. Their responses are forwarded but must not crash or stall the server.
- Filesystem. Document roots are tenant-controlled content; tenants may create symlinks.
- Operator → configuration / signals / admin socket. Trusted principal, but input is still validated (typos are a threat to availability).
- Metrics endpoint. Exposes operational data; bound to loopback by default.
3. Threats and mitigations
Legend: ✅ implemented and tested · 🟡 partial · ⏳ planned (milestone)
Network / protocol
| Threat | Mitigation | Status |
|---|---|---|
| Slowloris (slow headers) | h1 header_read_timeout (also bounds keep-alive idle); TLS handshake timeout | ✅ |
| Slow request body | Per-frame body read timeout in LimitedBody | ✅ |
| Slow response reader | Write timeout below TLS (limits.write_timeout) closes connections whose writes stall | ✅ tested |
| Connection exhaustion | Global max_connections semaphore; accept pauses instead of queueing in user space | ✅ |
| One source exhausting connections or request capacity | Per-source (IPv4 / IPv6 /64) connection limit at accept and request token bucket (429); sharded, size-capped state that cannot grow with the number of attacking addresses | ✅ tested (incl. 100k-address flood) |
| Per-tenant request floods | Tenant concurrency and rate quotas (M3) | ✅ tested |
| Oversized headers / URI / body | max_header_bytes, max_headers, h2 max_header_list_size, URI length → 414, Content-Length precheck → 413, streaming cap | ✅ |
| HTTP/2 rapid reset (CVE-2023-44487) and stream floods | max_concurrent_streams, max_pending_accept_reset_streams, max_local_error_reset_streams | ✅ (config) |
| Request smuggling (CL/TE ambiguity, obs-fold, invalid chunking) | hyper’s strict h1 parser; proxy always re-frames the upstream request and strips hop-by-hop headers; never forwards Transfer-Encoding from the client verbatim | ✅ tested for CL+TE |
| Host header ambiguity | Multiple Host headers → 400; HTTP/1.1 without Host → 400; absolute-form authority wins and must agree with Host | ✅ |
| SNI / Host mismatch (domain fronting across tenants) | strict_sni listeners answer 421 when Host differs from SNI | ✅ tested (opt-in) |
| TLS downgrade / weak ciphers | rustls defaults (TLS 1.2+ only, AEAD suites only) | ✅ |
| Header injection into upstream | Inbound X-Forwarded-*, Forwarded, X-Real-IP removed and replaced; X-Request-Id regenerated | ✅ |
| WebSocket tunnel exhaustion | Server-wide tunnel limit (max_upgraded_connections, 503 when exhausted) and idle timeout; only Upgrade: websocket on HTTP/1.1 GET is honoured, other upgrades are stripped | ✅ (tunnel + idle pump tested) |
| Upstream TLS spoofing | https:// upstreams verify the certificate against system roots (+ optional CA file) and the configured host name | ✅ tested |
Filesystem
| Threat | Mitigation | Status |
|---|---|---|
Path traversal (.., encoded %2e%2e, %2f, backslash, NUL) | Decode once, reject .. / NUL / \ / encoded slash; reject before touching the filesystem | ✅ tested |
| Symlink escape | Canonicalise and require the result to stay beneath the canonical root | ✅ tested (Linux) |
| TOCTOU symlink swap between check and open | Linux: openat2(RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS) from a root dir fd (kernel-enforced); fallback platforms keep the residual risk | ✅ Linux / 🟡 fallback |
| Opening a FIFO/device blocks a thread | O_NONBLOCK on open, type checked with fstat on the opened fd | ✅ |
Dotfile exposure (.env, .git) | Any path segment starting with . is 404 except .well-known | ✅ tested |
| Directory listing | Never generated | ✅ |
Managed applications (M2)
| Threat | Mitigation | Status |
|---|---|---|
| Shell injection through configured commands | argv is executed directly, never through a shell | ✅ |
| Secrets in scalws’s environment leaking to applications | Child environment is cleared; only PATH, LANG, HOME, configured env and SCALWS_* listen variables | ✅ tested |
| Injection into the generated PHP-FPM pool file | ini keys/values, env values and runtime_dir restricted to safe character sets at validation | ✅ |
| PHP source disclosure | .php files are never served by the static handler next to a PHP runtime | ✅ tested |
| Executing scripts outside the document root | SCRIPT_FILENAME only from the confined lookup (ADR-0006); security.limit_extensions = .php | ✅ tested |
httpoxy (Proxy: header → HTTP_PROXY) | Proxy header never forwarded to FastCGI; underscore header names dropped | ✅ tested |
| Spoofed client identity towards applications | REMOTE_ADDR/X-Forwarded-* set from the connection; client values discarded | ✅ tested |
| Other local users reaching application sockets | Socket directory 0700, FPM socket 0600 | ✅ |
| Response mix-up on reused FastCGI connections | Connection returned to the pool only after END_REQUEST; stale detection; replay limited to buffered requests | ✅ tested (worker recycling under concurrency) |
| A crashing / crash-looping application | Supervisor restarts with backoff; Failed after 5 crashes/minute; 503 + Retry-After; unrelated apps unaffected | ✅ tested |
| Orphaned application processes if scalwsd is killed | Own process group per worker, group killed on stop/crash; on SIGKILL of scalwsd the systemd unit’s control-group kill is required | 🟡 |
| Applications of different tenants reading each other’s files | All applications run as the scalws user (PHP workers as nobody when scalws runs as root); per-tenant users designed in ADR-0010, not implemented | ⏳ hardening |
| Runaway application memory / fork bombs / CPU hogging | cgroup v2 memory.max (+ memory.swap.max), pids.max, cpu.max per tenant; OOM kill confined to the tenant’s group; workers join their group before their first instruction | ✅ tested (OOM isolation, live limit change) |
| A tenant exhausting the shared server (request floods) | Per-tenant max_concurrent_requests (503) and token-bucket rate (429) with bounded state | ✅ tested |
| Limits silently not enforced | server.cgroups.mode: required refuses to start without delegation; auto logs a warning and exports scalws_cgroups_enabled 0 | ✅ |
Cache and request rules (M5)
| Threat | Mitigation | Status |
|---|---|---|
| Serving one user’s personalised response to another | Only explicitly cacheable responses; never with Set-Cookie, private, no-store, no-cache; requests with Authorization or personalisation cookies (profile rules) bypass the cache; no AI involvement | ✅ tested |
| Cache poisoning via unkeyed headers | Vary variants keyed by request header values; Vary: * not stored | ✅ tested |
| Cache memory exhaustion | Size-weighted budget with eviction, per-entry size cap | ✅ tested |
| Brute force on specific paths (login, password reset) | security.rate_limits per client address and path | ✅ tested |
| Cache stampede (many concurrent misses hitting a slow upstream) | Miss coalescing: one upstream request per URL, bounded 5 s wait | ✅ tested |
| Orphaned application processes after a crash of scalwsd | setpriv --pdeathsig TERM; cgroup leaves reaped at start | ✅ tested |
| Cache disk exhaustion / stale files | Disk tier bounded by size (LRU), expired and partial files removed at start, writes dropped when the queue is full | ✅ tested |
| Unauthenticated purge | Purge only on the admin socket (ADR-0017) | ✅ |
Sensitive files of known frameworks (wp-config.php, .env, uploaded PHP) | Profile deny rules + configured deny paths, case-insensitive on the normalised path | ✅ tested |
Detection (M4)
| Threat | Mitigation | Status |
|---|---|---|
A hostile application tree (symlinks to /etc, huge files, deep trees) abusing scalwsctl detect | Symlinks never followed; reads capped at 256 KiB per file, depth 3, 4000 entries; dependency dirs skipped; nothing executed | ✅ tested (symlink escape) |
| Detected values injecting configuration | Facts are escaped into quoted YAML strings or JSON arrays and the result is parsed and validated like any configuration | ✅ |
| A deploy silently changing how an application runs | Configuration loading never inspects application files; detection only proposes | ✅ tested |
Optimizer (M6)
| Threat | Mitigation | Status |
|---|---|---|
| Load spike makes the optimizer exhaust host resources | Worker targets bounded by configured min/max; scale-up blocked at ≥ 85 % tenant memory; tenant cgroup limits still apply | ✅ tested |
| Oscillation (flapping) | Separate up/down bands, consecutive-window streaks, per-rule cooldowns; property test proves single steps and cooldown spacing for arbitrary telemetry | ✅ tested |
| A change makes things worse | Outcome observed for 3 windows; regression in 5xx ratio or p95 rolls back and suppresses the rule for 10 min | ✅ tested |
| Unexpected automatic changes | Recommend-only by default; autoscale: auto per application; every decision audited | ✅ |
| Journal disclosure | Served only on the loopback admin listener; contains tenant/app names and aggregate numbers, no request data | ✅ |
Diagnostics (M7)
| Threat | Mitigation | Status |
|---|---|---|
| Leaking request data through diagnostics | Ring buffer keeps method, path without query (truncated to 256 bytes), status and timings; no headers, cookies or bodies | ✅ |
| Unbounded diagnostic memory | Ring buffer capped per application (recent_requests, max 10000); one evidence/finding set per application | ✅ |
Exposure of /diagnose | Read-only, loopback metrics listener and admin socket | ✅ |
| Misleading root cause | Findings state correlation (related), carry their evidence and are fully deterministic and unit-tested | ✅ |
Local AI advisor (M8)
| Threat | Mitigation | Status |
|---|---|---|
| Telemetry or customer data leaving the host | Disabled by default; loopback endpoint unless allow_remote: true; only aggregated diagnostics, recent request paths and optimizer decisions are sent | ✅ tested |
| Secrets in the model input | Every string redacted (query strings, userinfo, Bearer/Basic credentials, secret key=value, JWTs, e-mail addresses, opaque tokens); secret-named fields dropped; no headers, cookies or bodies collected | ✅ tested (unit, property, end to end) |
| Prompt injection via request paths | Model output is advisory text only; strict schema, bounded lengths, causes must reference real finding codes; the advisor cannot act | ✅ |
| Terminal escape injection via model output | Control characters removed from every output field | ✅ tested |
| LLM-generated commands executed | Never: investigations are displayed only; no code path executes them | ✅ |
| Model down/slow/hostile degrades serving | Separate task, bounded queue (429 when full), per-call timeout, size-capped response (256 KiB); request path never waits | ✅ tested |
| API key exposure | Read from an environment variable named in the configuration, never stored in YAML or logs | ✅ |
ACME (M10)
| Threat | Mitigation | Status |
|---|---|---|
| Theft of private keys / account key | state_dir 0700, keys and account 0600, atomic writes; unit StateDirectoryMode=0750 | ✅ tested |
| Challenge endpoint abuse | Only tokens of pending orders are answered; everything else falls through to the application; plain HTTP only | ✅ |
| Issuance for names the operator did not configure | Only domains of applications with acme: true; wildcards and IPs rejected at validation | ✅ tested |
| CA outage / rate limits | Failures back off per application (5 min doubling to 24 h); existing certificates keep being served; renewal starts 30 days before expiry | ✅ |
| A bad issued file breaks serving | Managed certificates are verified (key match, names) at prepare; invalid ones are skipped with a warning | ✅ tested |
HTTP/3 (M10, opt-in feature)
| Threat | Mitigation | Status |
|---|---|---|
| QUIC connection floods | Global connection permits and per-source admission before the handshake completes (refuse); stream limit and idle timeout from limits | ✅ |
| Amplification / address spoofing | quinn’s address validation and anti-amplification limits (RFC 9000 §8) | ✅ (library) |
| Replay via 0-RTT | 0-RTT disabled (max_early_data_size = 0) | ✅ |
| Immature HTTP/3 implementation | Off by default and not in packages; requests pass the same pipeline checks as TCP | 🟡 experimental |
Containers (M10)
| Threat | Mitigation | Status |
|---|---|---|
Option injection into the engine CLI (--privileged as image, mounts with ,) | argv only, no shell; image, user, mount paths and env keys validated | ✅ tested |
| Container escaping tenant limits | podman: --cgroup-parent = application group (tenant memory/CPU/pids apply); docker: daemon-placed, warning at validation | ✅ tested (podman) / 🟡 docker |
| Privilege escalation inside the container | no-new-privileges always; drop_capabilities, read_only, user options | ✅ |
| Orphaned containers keep serving or consuming resources | --init for signal delivery; rm --force after every worker exit; names unique per start | ✅ tested |
| Exposure of the published port | Published on 127.0.0.1 only; reached through scalws | ✅ |
Client address (trusted proxies)
| Threat | Mitigation | Status |
|---|---|---|
Client spoofs X-Forwarded-For to evade per-IP limits or fake log entries | Header used only from trusted_proxies peers; right-most untrusted entry wins; otherwise replaced | ✅ tested |
| Forged PROXY headers | Only from trusted peers; malformed/late headers close the connection; parser never reads past the header and is property-tested | ✅ tested |
| Balancer throttled as one client | Trusted peers exempt from the per-source connection limit; requests limited per reported client | ✅ tested |
Configuration and operations
| Threat | Mitigation | Status |
|---|---|---|
| Typo / invalid config takes server down | deny_unknown_fields, semantic validation, full prepare before atomic swap; failure keeps active state | ✅ tested |
| Unimplemented features silently ignored | Runtime types not available in this build are rejected at validation | ✅ |
| Metrics endpoint exposure | Loopback by default; separate listener | ✅ |
| Admin API abuse | Unix socket (0660, directory 0750, owner/group scalws); optional remote API only with mutual TLS (client certificate from a configured CA, TLS 1.3); mutations refused (403) on the TCP metrics listener; every mutation audited (peer uid/gid/pid or certificate subject); a live socket of another instance is never replaced | ✅ tested (incl. package smoke test: outsider denied, group member allowed) |
| Running as root | Packaged unit runs as scalws with only CAP_NET_BIND_SERVICE; warning when started as root; no in-process privilege drop (avoids multi-threaded setuid) | ✅ |
| Bad configuration applied by an operator | Same validate/prepare path as reload; rollback to any of the last 10 configurations; restore is a dry run unless --apply and keeps .bak files | ✅ tested |
| Tampered backup manifest writing outside intended paths | Only absolute paths without .. are accepted; sizes verified; dry run by default | ✅ tested |
Logging / metrics
| Threat | Mitigation | Status |
|---|---|---|
| Secrets in logs | Access log records path without query string; no headers or bodies logged | ✅ |
| Metric cardinality explosion | Labels limited to configured tenant/app names, listener name, normalised method and status class; unmatched hosts collapse to _unmatched | ✅ |
Supply chain
| Threat | Mitigation | Status |
|---|---|---|
| Vulnerable / malicious dependencies | cargo deny (advisories, licences, sources, bans) and cargo audit in CI; Cargo.lock committed | ✅ (CI config) |
| Unsafe code bugs | unsafe_code = "forbid" workspace-wide | ✅ |
| Parser bugs | Property-based fuzzing (proptest) of path and host normalisation, range, Cache-Control, CGI headers, units and the configuration parser; coverage-guided fuzzing (cargo-fuzz, nightly) not set up | 🟡 |
4. Residual risks accepted for M1
- TOCTOU on symlink replacement inside a document root on platforms/kernels without
openat2(fallback path only). - No write/response timeout for slow readers while a response is streaming.
- No per-client rate limiting; a single client can consume up to
max_connections. - No WAF.
- Applications of different tenants share a Unix user (per-tenant users are a designed but unimplemented hardening step).
- The cgroup trampoline relies on
/bin/sh(a fixed script; user argv stays positional).