Users' guide

Security

This guide describes the security features of ScalWS and how to configure them: request and connection limits, per-client rate limits, tenant quotas, per-application rules, trusted proxies, path handling, isolation of tenants, and access to the admin API. It ends with a hardening checklist.

The full list of threats and their mitigations is in THREAT_MODEL.md. TLS is covered in tls.md.

Order of checks

For each request ScalWS applies its rules in a fixed order, before any application code runs and before the response cache:

ACME HTTP-01 challenge answers (plain HTTP, pending tokens only).
Per-client request rate (server.security.per_ip) → 429.
CONNECT → 405; URI longer than max_uri_bytes → 414.
Host check: missing, duplicated or conflicting Host → 400.
Path normalisation; malformed paths → 400. A host that no application serves → 404.
Strict SNI mismatch → 421.
Denied paths (deny_paths) → 404.
Per-path rate limits (rate_limits) → 429.
https_redirect → 301.
.htaccess rules (when enabled); denied paths are checked again on a rewritten path.
Request body size (Content-Length above max_request_body_bytes) → 413.
Tenant quotas → 429 or 503.

Every refusal by steps 2, 6, 7 and 8 is counted in scalws_security_blocked_total{tenant,app,reason} with the reasons ip_rate, misdirected, denied_path and route_rate. 429 answers carry Retry-After.

Request and connection limits (`server.limits`)

The defaults are conservative production values; most installations do not need to change them.

Every key of server.limits, with its syntax, default and reload behaviour: reference. What the limits do at their boundary:

At max_connections (HTTP/3 connections count too) the server stops accepting until a connection closes; there is no queueing in user space.
max_header_bytes limits the request head (HTTP/1), header list (HTTP/2) or field section and any frame but DATA (HTTP/3), from 8 KiB to 1 MiB; max_headers is 1 to 1024.
A path and query longer than max_uri_bytes gets 414; the value is at least 256 bytes and not more than max_header_bytes.
A Content-Length above max_request_body_bytes gets 413 before the body is read; a streamed body is cut off at the limit.
header_read_timeout is also the keep-alive idle time between HTTP/1 requests (slowloris protection).
h2_max_concurrent_streams (1 to 10000) applies per HTTP/2 and per HTTP/3 connection.
Beyond max_upgraded_connections, new WebSocket tunnels get 503.

server:
  limits:
    max_request_body_bytes: 64MiB     # e.g. for media uploads
    header_read_timeout: 10s

Timeouts must be greater than zero. Changing max_connections requires a restart; the other limits change on reload.

Built in, without settings: HTTP/2 rapid-reset protection, strict HTTP/1 parsing (requests with conflicting framing are refused), removal of client-supplied X-Forwarded-*, Forwarded and X-Real-IP headers before proxying, and only Upgrade: websocket on HTTP/1.1 GET is honoured as an upgrade.

Per-client limits (`server.security.per_ip`)

Limits per client address. IPv4 addresses are counted individually, IPv6 addresses per /64 (one subscriber’s allocation).

server:
  security:
    per_ip:
      max_connections: 100
      requests_per_second: 50
      burst: 200

Every key of per_ip: reference. Over max_connections, a new connection is closed immediately at accept; over requests_per_second (a token bucket of size burst, which needs requests_per_second), requests get 429 with Retry-After. table_size must be at least 64.

Memory is bounded by table_size, whatever the number of attacking addresses: when the table is full, the least recently seen idle address is forgotten, and if every tracked address is busy, a new address is admitted without tracking (fail open, counted). The global max_connections still applies.

Values of 0 are refused. Changing the limits takes effect on reload; adding or removing per_ip, or changing table_size, needs a restart.

Behind a load balancer, configure trusted proxies: otherwise every request appears to come from the balancer and all clients share one limit.

Tenant quotas

A tenant’s resources can limit the work the server does for all its applications together (ADR-0010):

tenants:
  - name: shop
    resources:
      max_concurrent_requests: 200   # excess: 503 with Retry-After
      requests_per_second: 500       # excess: 429 with Retry-After
      burst: 1000                    # default: equal to requests_per_second
    applications: [...]

These quotas protect other tenants from one tenant’s traffic peak. CPU, memory, process and disk limits for the tenant’s own processes (cpu, memory, swap, pids, io_weight, io_bandwidth) are covered under cgroups.

Per-application rules (`security:`)

      - name: blog
        domains: [blog.example.com]
        root: /srv/www/blog
        profile: wordpress
        security:
          deny_paths: [/backup, "/wp-content/uploads/*.phar"]
          rate_limits:
            - {path: /wp-login.php, requests_per_second: 1, burst: 5}
            - {path: /xmlrpc.php, requests_per_second: 1}

Every key of an application’s security:, with its syntax, default and reload behaviour: reference. deny_paths are answered with 404 and added to the profile’s deny rules; ignore_profile_rules drops the profile’s rules (only your own apply); rate_limits takes at most 64 rules.

Each rate_limits entry has path (same patterns as deny_paths), requests_per_second (greater than zero) and burst (default: equal to the rate). The first matching rule applies, so put specific rules before general ones. Requests over the rate get 429 with Retry-After.

Pattern rules for deny_paths and rate_limits[].path:

Patterns start with /, contain printable ASCII only, and no ? or #.
They are matched case-insensitively against the normalised path (decoded, no ./.. or duplicate slashes), so /WP-CONFIG.PHP, /%77p-config.php and /x/../wp-config.php all match /wp-config.php.
A pattern without * matches the path itself and everything below it, on segment boundaries: /backup matches /backup and /backup/db.sql, but not /backups.
* matches any sequence of characters, including /. A * pattern also covers paths below a match, so /uploads/*.php matches /uploads/a/b.php and /uploads/x.php/extra (PATH_INFO).
An encoded slash (%2F) is also checked in its decoded form, as an upstream that decodes it would see the path.
For a path ending in /, the application’s index files are checked too: denying /admin/index.php also denies /admin/.

Denied requests get 404 so that the response does not confirm that the file exists.

Profile rules

Profiles (applications.md) bring deny rules for well-known sensitive files, for example profile: wordpress denies /wp-config.php, /wp-content/debug.log, /wp-content/uploads/*.php, /readme.html and /license.txt; laravel denies /.env, /storage/logs and /vendor; node denies /node_modules, /.env and /package-lock.json. Use ignore_profile_rules: true only when you have a reason to serve one of these paths, and then list the rules you still want yourself.

Trusted proxies and PROXY protocol

When ScalWS runs behind a load balancer or CDN, tell it which peers may report the real client address (ADR-0022):

server:
  trusted_proxies: ["192.0.2.0/24", "192.0.2.10"]

Entries of trusted_proxies are IP addresses or CIDR networks, IPv4 or IPv6. The list changes on reload.

X-Forwarded-For (any listener): for a request whose TCP peer is in trusted_proxies, the client is the right-most X-Forwarded-For entry that is not itself a trusted proxy; entries further left could have been written by the client and are not believed. If that entry cannot be parsed, the client is treated as unknown (never as the proxy itself). From untrusted peers the header is ignored and replaced. Forwarded (RFC 7239) is not read.

PROXY protocol (proxy_protocol: true on a listener): every connection must start with a PROXY protocol v1 or v2 header, read before TLS and HTTP within header_read_timeout.

server:
  trusted_proxies: ["192.0.2.10"]
  listeners:
    - name: https
      address: "0.0.0.0:443"
      proxy_protocol: true
      tls: {acme: true}

proxy_protocol: true requires trusted_proxies; the check refuses it otherwise.
Connections from peers outside trusted_proxies are closed at once; a missing or malformed header also closes the connection.
Turning proxy_protocol on or off for a listener needs a restart.

The effective client address is used by the per-client limits, rate_limits, .htaccess Require ip, the access log, PHP’s REMOTE_ADDR and the X-Forwarded-For sent to applications. Connections from trusted proxies are exempt from per_ip.max_connections (a balancer carries many clients); their requests are still rate limited per real client.

Paths, files and dotfiles

Normalisation: request paths are percent-decoded once, . and .. segments and repeated slashes are resolved. A path that climbs above /, contains a NUL byte or an invalid escape is answered 400. Routing, deny rules and the cache all see the same normalised path.
Confinement of static files (ADR-0006): on Linux files are opened with openat2(RESOLVE_BENEATH) relative to the document root, so neither .. nor symlinks can leave the root, even if a symlink is swapped during the request. Consequence: absolute symlinks are refused even when they point inside the root; use relative symlinks. On a kernel without openat2 (older than 5.6, or a seccomp policy that blocks it) the path is opened one component at a time from the root without following links, so no symbolic link is served there. Encoded slashes (%2F) and backslashes in file paths are refused. systemd’s RestrictSUIDSGID= is such a policy (it answers openat2 with ENOSYS), so the packaged server units do not set it and pass --restrict-suid-sgid instead: the same refusal of set-user-ID and set-group-ID modes on open, openat, creat, mkdir, mkdirat, mknod, mknodat and the chmod calls, inherited by the instances and the applications, with openat2 left working. The filter cannot read openat2’s mode, so a compromised process could still create a file with such a bit through openat2; it is owned by the service user, and its first write by that user (without CAP_FSETID) clears the bit. On kernels whose ext4 or xfs fallocate does not clear set-ID bits (before the 2022 file_modified fixes, unless the distribution backported them), fallocate and a shared mmap write could give such a file content and keep the bit; that needs code execution in the service user already. io_uring, whose open and mkdir operations take a mode no filter sees, is refused (ENOSYS, callers fall back). Environment=SCALWS_RESTRICT_SUID_SGID=false in a drop-in turns the flag off (the former RestrictSUIDSGID=false).
Dotfiles: static files and PHP scripts never serve or run a path with a segment starting with . (.env, .git/config, .htaccess), 404, except below the first segment /.well-known/. Proxied and managed Node.js/Python applications receive such paths unchanged; protect them there or with deny_paths.
No directory listings are ever generated.
PHP: .php files are never served as source by the static handler next to a PHP runtime, and scripts are only run from inside the document root.

`.htaccess`

With htaccess: true an application reads Apache .htaccess files below its root (ADR-0031). These files are written by the tenant, so they are treated as untrusted:

proxy rules ([P]), RewriteMap, SetHandler/AddHandler and php_admin_* are ignored and reported;
regular expressions use a linear-time engine; patterns it cannot express make the rule inactive;
hard limits: 64 KiB per file, 1000 directives, 10 rewrite rounds, bounded regular expression size and scanning per request, and at most 100 ms of parsing per second per application;
rewrites stay inside the application, and deny rules apply to the rewritten path too;
an access rule ScalWS cannot evaluate (for example AuthType/Require valid-user, or access rules in <RequireAll>, <Limit> or <If>) denies the directory with 403 instead of being ignored, so protected content is not published by accident.

Ignored lines are logged with the file name: journalctl -u scalwsd | grep htaccess. Changes are picked up within 2 seconds.

Tenant isolation

Tenant Unix users

By default all managed application processes (PHP-FPM, Node.js, Python) run as the scalws user, so one tenant’s code could read another tenant’s files. On Linux, tenants[].user runs a tenant’s processes as its own account (ADR-0026):

tenants:
  - name: web1
    user: web1
    applications: [...]

The account must exist (resolved through NSS) and must not be root. Container applications are refused for such tenants.
scalwsd needs four more capabilities, granted with a systemd drop-in; it refuses the configuration at activation when one is missing. This makes scalwsd root-equivalent if it is compromised: the same trade-off as a root PHP-FPM master. The drop-in and the file permissions are in OPERATIONS.md.
Each such tenant gets a private socket directory that other tenants cannot list.
Under scalws-controller, tenant users need server.sandbox: off: with on or audit, which a new install has, the reload is refused (sandbox.md). Change the line in /etc/scalws/scalws.yaml when you add user.
Document roots must be readable by the tenant and by group scalws (for static files), e.g. chown -R web1:scalws /srv/web1 && chmod -R g+rX,o-rwx /srv/web1.

With or without user, application processes start without any of scalwsd’s capabilities and with no_new_privs, and their environment is cleared except for PATH, LANG, HOME, the configured env and ScalWS’s own variables.

Managed runtimes run under scalwsd.service; scalws-controller refuses them (see OPERATIONS.md).

cgroups

With cgroup v2 delegation (the packaged unit sets Delegate=yes), each tenant’s processes run in their own cgroup with the limits from resources (ADR-0010):

server:
  cgroups:
    mode: required        # auto (default) | required | off
tenants:
  - name: web1
    resources:
      cpu: "200%"         # two CPUs
      memory: 2GiB
      pids: 512
      io_bandwidth: 50MiB

mode: auto enforces limits when delegation works and otherwise only warns (scalws_cgroups_enabled 0); required refuses to start without it. Use required in production so limits are never silently missing.
io_bandwidth is written for the disks that hold the tenant’s application roots. A root on a file system without a block device (btrfs, ZFS, overlay) has no disk to write it for: scalwsd warns once per tenant, mode: required refuses to start, and scalwsctl limit shows the rate as not enforced. io_weight still applies.
swap defaults to 0 when memory is set, so the memory limit is not absorbed by swap. An out-of-memory kill stays inside the tenant’s group.
Processes are placed in their group before their first instruction; a worker is never started unconfined.
Limits change on reload without restarting applications; scalws-limitcpu, scalws-limitram and scalws-limitio change them at run time (see cli.md).
CPU that scalwsd itself spends on a tenant’s requests (TLS, proxying, static files) is not part of the tenant’s cgroup; the request quotas above limit that.

Admin API

scalwsctl, the console and scalws-controller manage the server through its admin API (ADR-0017).

Unix socket <runtime_dir>/admin.sock (default /run/scalws/admin.sock, mode 0660, owner scalws:scalws, directory 0750). It accepts every operation, including reloads, rollbacks, cache purges and tenant limits. Membership in the scalws group is administrative access: add only operators. Every change is logged under the target scalws::audit with the caller’s uid, gid and pid. server.admin.enabled: false turns the socket off; server.admin.socket moves it (absolute path, at most 100 bytes).
Metrics listener server.metrics.address (default 127.0.0.1:9901; the packaged configuration uses 127.0.0.1:9900) serves /metrics, /healthz, read-only views and a read-only dashboard, without authentication. Keep it on loopback; the check warns when it is not. server.metrics.enabled: false turns it off.

Remote administration (mutual TLS)

Remote administration is off unless server.admin.remote is configured (ADR-0023):

server:
  admin:
    remote:
      address: "0.0.0.0:9443"
      cert: /etc/scalws/admin.pem          # server certificate chain
      key: /etc/scalws/admin.key
      client_ca: /etc/scalws/operators-ca.pem

All four fields are required (absolute paths); there is no password or token mode.
TLS 1.3 only. A client without a certificate issued by client_ca fails the handshake and never reaches the API; refused handshakes are logged under scalws::audit.
Authenticated clients have the same rights as the Unix socket. Mutations are audited with the client certificate’s subject and address.
Revocation lists are not checked: to revoke an operator, re-issue the CA bundle without the old issuer, or rotate the CA. Restrict address with a firewall as well.
Changing admin.remote needs a restart.

scalwsctl --remote admin.example.com:9443 --ca server-ca.pem \
  --client-cert alice.pem --client-key alice.key status

Console

The ScalWS Console is a separate process that talks only to scalws-controller. Its accounts, sessions, CSRF protection, re-authentication for destructive actions and audit trail are described in console-security.md.

Configuration files

scalws.yaml is installed as 0640 root:scalws. Keep it that way: it can contain secrets (application env), and whoever can write it controls the server.
Site files under sites.d/ must belong to root and must not be writable by group or others: whoever can write there can serve any domain from any directory the server can read (sites.md).
Unknown fields are errors, so a misspelt security setting fails the check instead of being silently ignored. Run scalwsctl check before every reload.

Hardening checklist

Network and TLS

Only the listeners you need; plain HTTP only for redirects and ACME (https_redirect: true on applications).
strict_sni: true on HTTPS listeners shared by several tenants.
Certificate keys readable only by root and group scalws (0640).
Behind a load balancer: trusted_proxies set to exactly its addresses; with PROXY protocol, the listener reachable only from the balancer.

Limits

server.security.per_ip with max_connections and requests_per_second suited to your traffic.
rate_limits for login, password-reset and XML-RPC endpoints.
Tenant quotas (max_concurrent_requests, requests_per_second) on shared hosts.
max_request_body_bytes no larger than your applications need.

Applications

A matching profile (WordPress, Laravel, …) so its deny rules apply; add your own deny_paths for backups, logs and exports inside document roots.
No secrets, VCS directories or build artefacts in document roots of proxied or Node.js/Python applications (dotfile protection covers static files and PHP only).
Relative symlinks only inside document roots.
Cached applications: personalisation cookies listed in bypass_cookies (caching.md).

Tenants

server.cgroups.mode: required and resource limits per tenant.
tenants[].user for tenants that must not read each other’s files (accepting the extra capabilities). Under scalws-controller this needs server.sandbox: off.
Document roots not world-readable (o-rwx).

Administration

server.sandbox: on unless tenant users need it off (sandbox.md), checked with scalwsd --sandbox-selftest.
Only operators in group scalws.
Metrics listener on loopback.
Remote admin off, or mutual TLS with a private CA and a firewall rule.
Console bound to loopback or restricted with --allow (see console-security.md).
Audit logs (scalws::audit) collected and kept.
scalws.yaml 0640 root:scalws; sites.d owned by root, not group-writable.
Packages kept current (apt upgrade), see OPERATIONS.md.