# Security

URL: https://scalws.com/docs/guide/security/
Source: `docs/guide/security.md` in the ScalWS repository

This guide describes the security features of ScalWS and how to configure them: request
and connection limits, per-client rate limits, tenant quotas, per-application rules,
trusted proxies, path handling, isolation of tenants, and access to the admin API. It ends
with a [hardening checklist](#hardening-checklist).

The full list of threats and their mitigations is in
[THREAT_MODEL.md](https://scalws.com/docs/threat-model/). TLS is covered in [tls.md](https://scalws.com/docs/guide/tls/).

## Order of checks

For each request ScalWS applies its rules in a fixed order, before any application code
runs and before the response cache:

1. ACME HTTP-01 challenge answers (plain HTTP, pending tokens only).
2. Per-client request rate (`server.security.per_ip`) → `429`.
3. `CONNECT` → `405`; URI longer than `max_uri_bytes` → `414`.
4. Host check: missing, duplicated or conflicting `Host` → `400`.
5. Path normalisation; malformed paths → `400`. A host that no application serves →
   `404`.
6. Strict SNI mismatch → `421`.
7. Denied paths (`deny_paths`) → `404`.
8. Per-path rate limits (`rate_limits`) → `429`.
9. `https_redirect` → `301`.
10. `.htaccess` rules (when enabled); denied paths are checked again on a rewritten path.
11. Request body size (`Content-Length` above `max_request_body_bytes`) → `413`.
12. Tenant quotas → `429` or `503`.

Every refusal by steps 2, 6, 7 and 8 is counted in
`scalws_security_blocked_total{tenant,app,reason}` with the reasons `ip_rate`,
`misdirected`, `denied_path` and `route_rate`. `429` answers carry `Retry-After`.

## Request and connection limits (`server.limits`)

The defaults are conservative production values; most installations do not need to
change them.

Every key of `server.limits`, with its syntax, default and reload behaviour:
[reference](https://scalws.com/docs/reference/server-limits/). What the limits do at their boundary:

* At [`max_connections`](https://scalws.com/docs/reference/server-limits/#server.limits.max_connections) (HTTP/3 connections count too) the server
  stops accepting until a connection closes; there is no queueing in user space.
* [`max_header_bytes`](https://scalws.com/docs/reference/server-limits/#server.limits.max_header_bytes) limits the request head (HTTP/1), header
  list (HTTP/2) or field section and any frame but DATA (HTTP/3), from 8 KiB to 1 MiB; [`max_headers`](https://scalws.com/docs/reference/server-limits/#server.limits.max_headers) is 1 to 1024.
* A path and query longer than [`max_uri_bytes`](https://scalws.com/docs/reference/server-limits/#server.limits.max_uri_bytes) gets `414`; the
  value is at least 256 bytes and not more than `max_header_bytes`.
* A `Content-Length` above [`max_request_body_bytes`](https://scalws.com/docs/reference/server-limits/#server.limits.max_request_body_bytes) gets
  `413` before the body is read; a streamed body is cut off at the limit.
* [`header_read_timeout`](https://scalws.com/docs/reference/server-limits/#server.limits.header_read_timeout) is also the keep-alive idle time
  between HTTP/1 requests (slowloris protection).
* [`h2_max_concurrent_streams`](https://scalws.com/docs/reference/server-limits/#server.limits.h2_max_concurrent_streams) (1 to 10000) applies per
  HTTP/2 and per HTTP/3 connection.
* Beyond [`max_upgraded_connections`](https://scalws.com/docs/reference/server-limits/#server.limits.max_upgraded_connections), new WebSocket
  tunnels get `503`.

```yaml
server:
  limits:
    max_request_body_bytes: 64MiB     # e.g. for media uploads
    header_read_timeout: 10s
```

Timeouts must be greater than zero. Changing `max_connections` requires a restart; the
other limits change on reload.

Built in, without settings: HTTP/2 rapid-reset protection, strict HTTP/1 parsing (requests
with conflicting framing are refused), removal of client-supplied `X-Forwarded-*`,
`Forwarded` and `X-Real-IP` headers before proxying, and only `Upgrade: websocket` on
HTTP/1.1 `GET` is honoured as an upgrade.

## Per-client limits (`server.security.per_ip`)

Limits per client address. IPv4 addresses are counted individually, IPv6 addresses per
`/64` (one subscriber's allocation).

```yaml
server:
  security:
    per_ip:
      max_connections: 100
      requests_per_second: 50
      burst: 200
```

Every key of [`per_ip`](https://scalws.com/docs/reference/server-security/#server.security.per_ip):
[reference](https://scalws.com/docs/reference/server-security/). Over
[`max_connections`](https://scalws.com/docs/reference/server-security/#server.security.per_ip.max_connections), a new connection is closed immediately at
accept; over [`requests_per_second`](https://scalws.com/docs/reference/server-security/#server.security.per_ip.requests_per_second) (a token bucket of size
[`burst`](https://scalws.com/docs/reference/server-security/#server.security.per_ip.burst), which needs `requests_per_second`), requests get `429` with
`Retry-After`. [`table_size`](https://scalws.com/docs/reference/server-security/#server.security.per_ip.table_size) must be at least 64.

Memory is bounded by `table_size`, whatever the number of attacking addresses: when the
table is full, the least recently seen idle address is forgotten, and if every tracked
address is busy, a new address is admitted without tracking (fail open, counted). The
global `max_connections` still applies.

Values of 0 are refused. Changing the limits takes effect on reload; adding or removing
`per_ip`, or changing `table_size`, needs a restart.

Behind a load balancer, configure [trusted proxies](#trusted-proxies-and-proxy-protocol):
otherwise every request appears to come from the balancer and all clients share one
limit.

## Tenant quotas

A tenant's [`resources`](https://scalws.com/docs/reference/tenants/#tenants-.resources) can limit the work the server does for all its applications
together ([ADR-0010](https://scalws.com/docs/adr/0010-cgroups-and-tenant-limits/)):

```yaml
tenants:
  - name: shop
    resources:
      max_concurrent_requests: 200   # excess: 503 with Retry-After
      requests_per_second: 500       # excess: 429 with Retry-After
      burst: 1000                    # default: equal to requests_per_second
    applications: [...]
```

These quotas protect other tenants from one tenant's traffic peak. CPU, memory, process
and disk limits for the tenant's own processes (`cpu`, `memory`, `swap`, `pids`,
`io_weight`, `io_bandwidth`) are covered under [cgroups](#tenant-isolation).

## Per-application rules (`security:`)

```yaml
      - name: blog
        domains: [blog.example.com]
        root: /srv/www/blog
        profile: wordpress
        security:
          deny_paths: [/backup, "/wp-content/uploads/*.phar"]
          rate_limits:
            - {path: /wp-login.php, requests_per_second: 1, burst: 5}
            - {path: /xmlrpc.php, requests_per_second: 1}
```

Every key of an application's `security:`, with its syntax, default and reload
behaviour: [reference](https://scalws.com/docs/reference/applications/#tenants-.applications-.security).
[`deny_paths`](https://scalws.com/docs/reference/applications/#tenants-.applications-.security.deny_paths) are answered with `404` and added to the profile's deny
rules; [`ignore_profile_rules`](https://scalws.com/docs/reference/applications/#tenants-.applications-.security.ignore_profile_rules) drops the profile's rules (only
your own apply); [`rate_limits`](https://scalws.com/docs/reference/applications/#tenants-.applications-.security.rate_limits) takes at most 64 rules.

Each `rate_limits` entry has `path` (same patterns as `deny_paths`),
`requests_per_second` (greater than zero) and `burst` (default: equal to the rate). The
**first** matching rule applies, so put specific rules before general ones. Requests over
the rate get `429` with `Retry-After`.

Pattern rules for `deny_paths` and `rate_limits[].path`:

* Patterns start with `/`, contain printable ASCII only, and no `?` or `#`.
* They are matched case-insensitively against the normalised path (decoded, no `.`/`..`
  or duplicate slashes), so `/WP-CONFIG.PHP`, `/%77p-config.php` and `/x/../wp-config.php`
  all match `/wp-config.php`.
* A pattern without `*` matches the path itself and everything below it, on segment
  boundaries: `/backup` matches `/backup` and `/backup/db.sql`, but not `/backups`.
* `*` matches any sequence of characters, including `/`. A `*` pattern also covers paths
  below a match, so `/uploads/*.php` matches `/uploads/a/b.php` and
  `/uploads/x.php/extra` (PATH_INFO).
* An encoded slash (`%2F`) is also checked in its decoded form, as an upstream that
  decodes it would see the path.
* For a path ending in `/`, the application's index files are checked too: denying
  `/admin/index.php` also denies `/admin/`.

Denied requests get `404` so that the response does not confirm that the file exists.

### Profile rules

Profiles ([applications.md](https://scalws.com/docs/guide/applications/)) bring deny rules for well-known sensitive
files, for example `profile: wordpress` denies `/wp-config.php`,
`/wp-content/debug.log`, `/wp-content/uploads/*.php`, `/readme.html` and `/license.txt`;
`laravel` denies `/.env`, `/storage/logs` and `/vendor`; `node` denies `/node_modules`,
`/.env` and `/package-lock.json`. Use `ignore_profile_rules: true` only when you have a
reason to serve one of these paths, and then list the rules you still want yourself.

## Trusted proxies and PROXY protocol

When ScalWS runs behind a load balancer or CDN, tell it which peers may report the real
client address ([ADR-0022](https://scalws.com/docs/adr/0022-trusted-proxies/)):

```yaml
server:
  trusted_proxies: ["192.0.2.0/24", "192.0.2.10"]
```

Entries of [`trusted_proxies`](https://scalws.com/docs/reference/server/#server.trusted_proxies) are IP
addresses or CIDR networks, IPv4 or IPv6. The list changes on reload.

**`X-Forwarded-For`** (any listener): for a request whose TCP peer is in
`trusted_proxies`, the client is the right-most `X-Forwarded-For` entry that is not itself
a trusted proxy; entries further left could have been written by the client and are not
believed. If that entry cannot be parsed, the client is treated as unknown (never as the
proxy itself). From untrusted peers the header is ignored and replaced. `Forwarded`
(RFC 7239) is not read.

**PROXY protocol** ([`proxy_protocol: true`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.proxy_protocol) on a listener): every connection must start
with a PROXY protocol v1 or v2 header, read before TLS and HTTP within
`header_read_timeout`.

```yaml
server:
  trusted_proxies: ["192.0.2.10"]
  listeners:
    - name: https
      address: "0.0.0.0:443"
      proxy_protocol: true
      tls: {acme: true}
```

* `proxy_protocol: true` requires `trusted_proxies`; the check refuses it otherwise.
* Connections from peers outside `trusted_proxies` are closed at once; a missing or
  malformed header also closes the connection.
* Turning `proxy_protocol` on or off for a listener needs a restart.

The effective client address is used by the per-client limits, `rate_limits`, `.htaccess`
`Require ip`, the access log, PHP's `REMOTE_ADDR` and the `X-Forwarded-For` sent to
applications. Connections from trusted proxies are exempt from
`per_ip.max_connections` (a balancer carries many clients); their requests are still rate
limited per real client.

## Paths, files and dotfiles

* **Normalisation**: request paths are percent-decoded once, `.` and `..` segments and
  repeated slashes are resolved. A path that climbs above `/`, contains a NUL byte or an
  invalid escape is answered `400`. Routing, deny rules and the cache all see the same
  normalised path.
* **Confinement** of static files ([ADR-0006](https://scalws.com/docs/adr/0006-static-file-confinement/)):
  on Linux files are opened with `openat2(RESOLVE_BENEATH)` relative to the document
  root, so neither `..` nor symlinks can leave the root, even if a symlink is swapped
  during the request. Consequence: **absolute symlinks are refused even when they point
  inside the root**; use relative symlinks. On a kernel without `openat2` (older than
  5.6, or a seccomp policy that blocks it) the path is opened one component at a time
  from the root without following links, so no symbolic link is served there. Encoded
  slashes (`%2F`) and backslashes in file paths are refused. systemd's
  `RestrictSUIDSGID=` is such a policy (it answers openat2 with `ENOSYS`), so the packaged
  server units do not set it and pass `--restrict-suid-sgid` instead: the same refusal of
  set-user-ID and set-group-ID modes on open, openat, creat, mkdir, mkdirat, mknod,
  mknodat and the chmod calls, inherited by the instances and the applications, with
  openat2 left working. The filter cannot read openat2's mode, so a compromised process
  could still create a file with such a bit through openat2; it is owned by the service
  user, and its first write by that user (without `CAP_FSETID`) clears the bit. On kernels
  whose ext4 or xfs `fallocate` does not clear set-ID bits (before the 2022 `file_modified`
  fixes, unless the distribution backported them), `fallocate` and a shared `mmap` write
  could give such a file content and keep the bit; that needs code execution in the service
  user already. io_uring, whose open and mkdir operations take a mode no filter sees, is
  refused (`ENOSYS`, callers fall back). `Environment=SCALWS_RESTRICT_SUID_SGID=false` in a
  drop-in turns the flag off (the former `RestrictSUIDSGID=false`).
* **Dotfiles**: static files and PHP scripts never serve or run a path with a segment
  starting with `.` (`.env`, `.git/config`, `.htaccess`), `404`, except below the
  first segment `/.well-known/`. Proxied and managed Node.js/Python applications receive
  such paths unchanged; protect them there or with `deny_paths`.
* **No directory listings** are ever generated.
* **PHP**: `.php` files are never served as source by the static handler next to a PHP
  runtime, and scripts are only run from inside the document root.

## `.htaccess`

With [`htaccess: true`](https://scalws.com/docs/reference/applications/#tenants-.applications-.htaccess) an application reads Apache `.htaccess` files below its `root`
([ADR-0031](https://scalws.com/docs/adr/0031-htaccess/)). These files are written by the tenant, so they are
treated as untrusted:

* proxy rules (`[P]`), `RewriteMap`, `SetHandler`/`AddHandler` and `php_admin_*` are
  ignored and reported;
* regular expressions use a linear-time engine; patterns it cannot express make the rule
  inactive;
* hard limits: 64 KiB per file, 1000 directives, 10 rewrite rounds, bounded regular
  expression size and scanning per request, and at most 100 ms of parsing per second
  per application;
* rewrites stay inside the application, and deny rules apply to the rewritten path too;
* an access rule ScalWS cannot evaluate (for example `AuthType`/`Require valid-user`, or
  access rules in `<RequireAll>`, `<Limit>` or `<If>`) **denies** the directory with `403`
  instead of being ignored, so protected content is not published by accident.

Ignored lines are logged with the file name:
`journalctl -u scalwsd | grep htaccess`. Changes are picked up within 2 seconds.

## Tenant isolation

### Tenant Unix users

By default all managed application processes (PHP-FPM, Node.js, Python) run as the
`scalws` user, so one tenant's code could read another tenant's files. On Linux,
[`tenants[].user`](/docs/reference/tenants/#tenants-.user) runs a tenant's processes as its own account
([ADR-0026](https://scalws.com/docs/adr/0026-tenant-unix-users/)):

```yaml
tenants:
  - name: web1
    user: web1
    applications: [...]
```

* The account must exist (resolved through NSS) and must not be `root`. Container
  applications are refused for such tenants.
* scalwsd needs four more capabilities, granted with a systemd drop-in; it refuses the
  configuration at activation when one is missing. This makes scalwsd root-equivalent if
  it is compromised: the same trade-off as a root PHP-FPM master. The drop-in and the
  file permissions are in [OPERATIONS.md](https://scalws.com/docs/operations/#tenant-users-adr-0026-optional).
* Each such tenant gets a private socket directory that other tenants cannot list.
* Under scalws-controller, tenant users need `server.sandbox: off`: with `on` or `audit`,
  which a new install has, the reload is refused ([sandbox.md](https://scalws.com/docs/guide/sandbox/#when-it-applies)).
  Change the line in `/etc/scalws/scalws.yaml` when you add `user`.
* Document roots must be readable by the tenant and by group `scalws` (for static files),
  e.g. `chown -R web1:scalws /srv/web1 && chmod -R g+rX,o-rwx /srv/web1`.

With or without `user`, application processes start without any of scalwsd's
capabilities and with `no_new_privs`, and their environment is cleared except for `PATH`,
`LANG`, `HOME`, the configured `env` and ScalWS's own variables.

Managed runtimes run under `scalwsd.service`; scalws-controller refuses them (see
[OPERATIONS.md](https://scalws.com/docs/operations/#install)).

### cgroups

With cgroup v2 delegation (the packaged unit sets `Delegate=yes`), each tenant's processes
run in their own cgroup with the limits from `resources`
([ADR-0010](https://scalws.com/docs/adr/0010-cgroups-and-tenant-limits/)):

```yaml
server:
  cgroups:
    mode: required        # auto (default) | required | off
tenants:
  - name: web1
    resources:
      cpu: "200%"         # two CPUs
      memory: 2GiB
      pids: 512
      io_bandwidth: 50MiB
```

* [`mode: auto`](https://scalws.com/docs/reference/server-cgroups/#server.cgroups.mode) enforces limits when delegation works and otherwise only warns
  (`scalws_cgroups_enabled 0`); `required` refuses to start without it. Use `required` in
  production so limits are never silently missing.
* `io_bandwidth` is written for the disks that hold the tenant's application roots. A root
  on a file system without a block device (btrfs, ZFS, overlay) has no disk to write it
  for: scalwsd warns once per tenant, `mode: required` refuses to start, and
  `scalwsctl limit` shows the rate as not enforced. `io_weight` still applies.
* `swap` defaults to 0 when `memory` is set, so the memory limit is not absorbed by swap.
  An out-of-memory kill stays inside the tenant's group.
* Processes are placed in their group before their first instruction; a worker is never
  started unconfined.
* Limits change on reload without restarting applications; `scalws-limitcpu`,
  `scalws-limitram` and `scalws-limitio` change them at run time (see [cli.md](https://scalws.com/docs/guide/cli/)).
* CPU that scalwsd itself spends on a tenant's requests (TLS, proxying, static files) is
  not part of the tenant's cgroup; the request quotas above limit that.

## Admin API

`scalwsctl`, the console and scalws-controller manage the server through its admin API
([ADR-0017](https://scalws.com/docs/adr/0017-admin-api-and-operations/)).

* **Unix socket** `<runtime_dir>/admin.sock` (default `/run/scalws/admin.sock`, mode
  `0660`, owner `scalws:scalws`, directory `0750`). It accepts every operation, including
  reloads, rollbacks, cache purges and tenant limits. **Membership in the `scalws` group
  is administrative access**: add only operators. Every change is logged under the
  target `scalws::audit` with the caller's uid, gid and pid.
  [`server.admin.enabled: false`](https://scalws.com/docs/reference/server-admin/#server.admin.enabled) turns the socket off; [`server.admin.socket`](https://scalws.com/docs/reference/server-admin/#server.admin.socket) moves it
  (absolute path, at most 100 bytes).
* **Metrics listener** [`server.metrics.address`](https://scalws.com/docs/reference/server-metrics/#server.metrics.address) (default `127.0.0.1:9901`; the packaged
  configuration uses `127.0.0.1:9900`) serves `/metrics`, `/healthz`, read-only views and
  a read-only dashboard, **without authentication**. Keep it on loopback; the check warns
  when it is not. `server.metrics.enabled: false` turns it off.

### Remote administration (mutual TLS)

Remote administration is off unless [`server.admin.remote`](https://scalws.com/docs/reference/server-admin/#server.admin.remote)
is configured ([ADR-0023](https://scalws.com/docs/adr/0023-remote-admin-mtls/)):

```yaml
server:
  admin:
    remote:
      address: "0.0.0.0:9443"
      cert: /etc/scalws/admin.pem          # server certificate chain
      key: /etc/scalws/admin.key
      client_ca: /etc/scalws/operators-ca.pem
```

* All four fields are required (absolute paths); there is no password or token mode.
* TLS 1.3 only. A client without a certificate issued by `client_ca` fails the handshake
  and never reaches the API; refused handshakes are logged under `scalws::audit`.
* Authenticated clients have the same rights as the Unix socket. Mutations are audited
  with the client certificate's subject and address.
* Revocation lists are not checked: to revoke an operator, re-issue the CA bundle without
  the old issuer, or rotate the CA. Restrict `address` with a firewall as well.
* Changing `admin.remote` needs a restart.

```sh
scalwsctl --remote admin.example.com:9443 --ca server-ca.pem \
  --client-cert alice.pem --client-key alice.key status
```

### Console

The ScalWS Console is a separate process that talks only to scalws-controller. Its
accounts, sessions, CSRF protection, re-authentication for destructive actions and audit
trail are described in [console-security.md](https://scalws.com/docs/console/security/).

## Configuration files

* `scalws.yaml` is installed as `0640 root:scalws`. Keep it that way: it can contain
  secrets (application `env`), and whoever can write it controls the server.
* Site files under `sites.d/` must belong to root and must not be writable by group or
  others: whoever can write there can serve any domain from any directory the server can
  read ([sites.md](https://scalws.com/docs/sites/#rules)).
* Unknown fields are errors, so a misspelt security setting fails the check instead of
  being silently ignored. Run `scalwsctl check` before every reload.

## Hardening checklist

Network and TLS

- [ ] Only the listeners you need; plain HTTP only for redirects and ACME
      (`https_redirect: true` on applications).
- [ ] `strict_sni: true` on HTTPS listeners shared by several tenants.
- [ ] Certificate keys readable only by root and group `scalws` (`0640`).
- [ ] Behind a load balancer: `trusted_proxies` set to exactly its addresses; with PROXY
      protocol, the listener reachable only from the balancer.

Limits

- [ ] `server.security.per_ip` with `max_connections` and `requests_per_second` suited to
      your traffic.
- [ ] `rate_limits` for login, password-reset and XML-RPC endpoints.
- [ ] Tenant quotas (`max_concurrent_requests`, `requests_per_second`) on shared hosts.
- [ ] `max_request_body_bytes` no larger than your applications need.

Applications

- [ ] A matching `profile` (WordPress, Laravel, …) so its deny rules apply; add your own
      `deny_paths` for backups, logs and exports inside document roots.
- [ ] No secrets, VCS directories or build artefacts in document roots of proxied or
      Node.js/Python applications (dotfile protection covers static files and PHP only).
- [ ] Relative symlinks only inside document roots.
- [ ] Cached applications: personalisation cookies listed in `bypass_cookies`
      ([caching.md](https://scalws.com/docs/guide/caching/)).

Tenants

- [ ] `server.cgroups.mode: required` and resource limits per tenant.
- [ ] `tenants[].user` for tenants that must not read each other's files (accepting the
      extra capabilities). Under scalws-controller this needs `server.sandbox: off`.
- [ ] Document roots not world-readable (`o-rwx`).

Administration

- [ ] `server.sandbox: on` unless tenant users need it off ([sandbox.md](https://scalws.com/docs/guide/sandbox/)), checked with
      `scalwsd --sandbox-selftest`.
- [ ] Only operators in group `scalws`.
- [ ] Metrics listener on loopback.
- [ ] Remote admin off, or mutual TLS with a private CA and a firewall rule.
- [ ] Console bound to loopback or restricted with `--allow` (see
      [console-security.md](https://scalws.com/docs/console/security/)).
- [ ] Audit logs (`scalws::audit`) collected and kept.
- [ ] `scalws.yaml` `0640 root:scalws`; `sites.d` owned by root, not group-writable.
- [ ] Packages kept current (`apt upgrade`), see [OPERATIONS.md](https://scalws.com/docs/operations/).

## See also

* [THREAT_MODEL.md](https://scalws.com/docs/threat-model/): threats, mitigations and accepted risks.
* [tls.md](https://scalws.com/docs/guide/tls/): certificates, ACME, HTTPS redirects.
* [sandbox.md](https://scalws.com/docs/guide/sandbox/): Landlock and seccomp confinement of scalwsd.
* [configuration.md](https://scalws.com/docs/guide/configuration/): file layout, reload and restart rules.
* [troubleshooting.md](https://scalws.com/docs/guide/troubleshooting/): `403`/`404`/`429`/`421` answers you did not
  expect.