# HTTPS and TLS

URL: https://scalws.com/docs/guide/tls/
Source: `docs/guide/tls.md` in the ScalWS repository

This guide covers how ScalWS serves HTTPS: TLS listeners and certificate files,
certificates from Let's Encrypt (ACME), HTTP/2, HTTP/3, kernel TLS, and redirects to
HTTPS. The basics of the configuration file are in [configuration.md](https://scalws.com/docs/guide/configuration/),
and applications are covered in [applications.md](https://scalws.com/docs/guide/applications/).

TLS is implemented with [rustls](https://github.com/rustls/rustls) and the aws-lc-rs
crypto provider ([ADR-0002](https://scalws.com/docs/adr/0002-rustls-aws-lc-rs/)).

## TLS listeners

A listener becomes a TLS listener when it has a [`tls:`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls) block. A TLS listener
needs at least one certificate, or [`acme: true`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.acme) to also serve the
ACME-managed certificates of applications with `acme: true` (see
[Let's Encrypt](#lets-encrypt-acme)):

```yaml
server:
  listeners:
    - name: http
      address: "0.0.0.0:80"
    - name: https
      address: "0.0.0.0:443"
      tls:
        certificates:
          - cert: /etc/scalws/tls/example.com/fullchain.pem
            key: /etc/scalws/tls/example.com/privkey.pem
            server_names: [example.com, www.example.com]
            default: true
          - cert: /etc/scalws/tls/example.net/fullchain.pem
            key: /etc/scalws/tls/example.net/privkey.pem
            server_names: ["*.example.net", example.net]
```

The other listener fields that relate to TLS are [`strict_sni`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.strict_sni) (only
valid on TLS listeners, see [Strict SNI](#strict-sni)) and [`http3`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.http3) (see
[HTTP/3](#http3-quic)). Every key of a listener, including `tls` and its certificate
entries, with its syntax, default and reload behaviour:
[reference](https://scalws.com/docs/reference/server-listeners/).

### Certificate entries

Each entry of [`certificates`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.certificates) needs `cert`, `key` and at least one
name in `server_names`, except the `default` entry, which may have `server_names: []`
and is then served only to clients that send no SNI name or one no other entry lists:

* [`cert`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.certificates-.cert) holds the end-entity (leaf) certificate **first**,
  followed by the intermediate certificates. Do not include the root. A `fullchain.pem`
  as written by most ACME clients has this order.
* [`key`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.certificates-.key) is PKCS#8 (`BEGIN PRIVATE KEY`), PKCS#1
  (`BEGIN RSA PRIVATE KEY`) or SEC1 (`BEGIN EC PRIVATE KEY`).
* [`server_names`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.certificates-.server_names) are exact names (`example.com`) or
  wildcards (`*.example.com`).
* [`default`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.tls.certificates-.default) (at most one per listener) is used when the
  client sends no SNI or a name no entry lists.

When the configuration is checked (`scalwsctl check`, a reload, or start-up), every
certificate is loaded and verified:

* the PEM files can be read and the chain is not empty;
* the private key matches the certificate;
* every name in `server_names` is covered by the certificate's subject alternative
  names. A wildcard entry needs a matching wildcard SAN in the certificate. A wildcard
  SAN directly under one label, such as `*.org`, is accepted for one label below it
  (`example.org`), as RFC 6125 and clients such as Go do; browsers refuse such
  certificates, so the check only decides what ScalWS may serve, not what a client trusts;
* a name may be assigned to only one certificate per listener.

If anything fails, a reload is refused and the running configuration stays active; the
error names the file and the problem, for example
`…/fullchain.pem: certificate is not valid for server name "shop.example.com"`.

The server runs as the `scalws` user, so the certificate and key files must be readable by
that user (for example `root:scalws`, mode `0640` for the key).

### Certificates from a file

A listener can read more certificate entries from a separate YAML file, for example one
that a tool generates. The file is a list with the keys of `certificates`; its entries are
appended to the listener's `certificates` when the configuration is loaded from a file
(like `include` for site files), so a stored generation is self-contained.

```yaml
# fragment
server:
  listeners:
    - name: https
      address: "0.0.0.0:443"
      tls:
        acme: true
        certificates_file: "-/etc/scalws/panel/certificates.yaml"
```

A relative path is resolved against the directory of the configuration file. A leading `-`
makes a missing file mean "no entries"; without it a missing file is an error. An empty
file or `[]` is an empty list.

### How a certificate is chosen (SNI)

For each new TLS connection ScalWS looks at the SNI name the client sends:

1. an exact match in `server_names`;
2. otherwise a wildcard: `*.example.net` matches exactly **one** extra label
   (`shop.example.net`, but not `example.net` and not `a.shop.example.net`). List the bare
   domain separately if it should use the same certificate, as in the example above;
3. otherwise the `default` certificate;
4. otherwise the handshake fails.

A client that sends no SNI at all (for example a request to the IP address) gets the
`default` certificate, or a failed handshake when no certificate is marked `default`.
Names are compared case-insensitively and without a trailing dot.

Certificate selection is independent of routing: the certificate only has to be valid for
the name; which application answers is decided by the request's `Host` (see
[applications.md](https://scalws.com/docs/guide/applications/)).

### Replacing certificate files

Certificates are reloadable. After you replace the files (for example after renewing
them with an external tool), run:

```sh
sudo scalwsctl reload
```

The new files are checked first; existing connections keep their certificate and new
handshakes get the new one. Adding or removing a TLS listener, or changing its address,
`http3` or `proxy_protocol`, is a listener change and needs a restart (the reload says
"requires a restart").

The expiry of configured certificates is exported as the metric
`scalws_tls_certificate_expiry_seconds` (Unix time).

### Strict SNI

Browsers may reuse one HTTP/2 connection for several host names when the certificate
covers them all (connection coalescing). With [`strict_sni: true`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.strict_sni) on a listener, a request
whose `Host` differs from the SNI name of its connection is answered with
`421 Misdirected Request`; the client then opens a new connection for that host. This
prevents one tenant's connection from being used to reach another tenant's site
(domain fronting).

```yaml
    - name: https
      address: "0.0.0.0:443"
      strict_sni: true
      tls: {acme: true}
```

Requests on connections without an SNI name are not affected. Refused requests are counted
in `scalws_security_blocked_total{reason="misdirected"}`.

## TLS versions, ciphers and ALPN

There are no settings for protocol versions or cipher suites. What ScalWS uses:

| | HTTPS listeners (TCP) | HTTP/3 (QUIC) |
|---|---|---|
| TLS versions | TLS 1.2 and TLS 1.3 | TLS 1.3 only |
| Cipher suites | rustls defaults for the aws-lc-rs provider (AEAD suites with forward secrecy only) | same, TLS 1.3 suites |
| ALPN | `h2`, then `http/1.1` | `h3` |
| Session resumption | stateless TLS 1.3 session tickets | n/a |
| 0-RTT (early data) | not enabled | disabled |

Client certificates are not requested on application listeners (mutual TLS exists only for
the remote admin API, see [security.md](https://scalws.com/docs/guide/security/#admin-api)). OCSP stapling is not
implemented, and ScalWS does not add an `Strict-Transport-Security` header by itself;
applications can send it, or with `htaccess: true` you can set it with
`Header set Strict-Transport-Security "max-age=31536000"` in an `.htaccess` file.

With several instances under scalws-controller, session tickets are per instance, so a
resumption may fall back to a full handshake when the client lands on another instance.

## Let's Encrypt (ACME)

ScalWS can obtain and renew certificates itself ([ADR-0018](https://scalws.com/docs/adr/0018-acme/)). It uses
the HTTP-01 challenge only, so:

* the domain's DNS must point to this server, and port 80 must reach a plain-HTTP
  listener of ScalWS;
* wildcard domains (`*.example.com`) cannot get an ACME certificate (they need DNS-01,
  which is not implemented): use a certificate file for them;
* IP addresses and `localhost` are refused.

Three things have to be switched on:

```yaml
server:
  listeners:
    - {name: http, address: "0.0.0.0:80"}                       # HTTP-01 challenges
    - {name: https, address: "0.0.0.0:443", tls: {acme: true}}  # serves ACME certificates
  acme:
    enabled: true
    contact: ["mailto:ops@example.com"]
    terms_of_service_agreed: true

tenants:
  - name: shop
    applications:
      - name: shop
        domains: [shop.example.com, www.shop.example.com]
        acme: true                       # one certificate for all domains of this app
        root: /srv/www/shop
        runtime: {type: static}
```

With [site files](https://scalws.com/docs/sites/), `tls: auto` is the default as soon as `server.acme` is
enabled, so `scalwsctl site add` needs no extra option.

### `server.acme` settings

Every key of `server.acme`, with its syntax, default and reload behaviour:
[reference](https://scalws.com/docs/reference/server-acme/). The
[`directory`](https://scalws.com/docs/reference/server-acme/#server.acme.directory) must be an `https://`
URL, each [`contact`](https://scalws.com/docs/reference/server-acme/#server.acme.contact) a `mailto:`
address, and [`check_interval`](https://scalws.com/docs/reference/server-acme/#server.acme.check_interval)
at least `1m`. [`ca_bundle`](https://scalws.com/docs/reference/server-acme/#server.acme.ca_bundle) is meant
for test CAs such as Pebble; [`state_dir`](https://scalws.com/docs/reference/server-acme/#server.acme.state_dir)
is described [below](#state-directory).

`scalwsctl check` refuses the configuration when `enabled` is true but
`terms_of_service_agreed` is not, when no TLS listener has `acme: true`, when there is no
plain-HTTP listener, or when an application uses `acme: true` while ACME is disabled.

Try a new setup against the Let's Encrypt staging directory first, to stay clear of the
production rate limits:

```yaml
  acme:
    enabled: true
    directory: https://acme-staging-v02.api.letsencrypt.org/directory
    contact: ["mailto:ops@example.com"]
    terms_of_service_agreed: true
```

### What happens

* The certificate manager checks every application with `acme: true` at start, every
  `check_interval`, and after every configuration change. An application gets a new order
  when it has no certificate yet, when its `domains` changed, or when its certificate
  expires within `renew_before`.
* The CA's validation request `GET /.well-known/acme-challenge/<token>` is answered on
  plain-HTTP listeners for pending tokens, before routing and before `https_redirect`.
  Other `/.well-known/` paths go to the application as usual.
* A new certificate is activated like a reload: existing connections keep the old one,
  new handshakes get the new one. No restart is needed.
* A failed order is retried with a back-off per application, starting at 5 minutes and
  doubling up to 24 hours.
* Until an application has its first certificate, handshakes for its names get the
  listener's `default` certificate if there is one, and fail otherwise.
* When a configured certificate file and an ACME certificate cover the same name, the
  configured file wins.

### State directory

```text
/var/lib/scalws/acme/
  account.json                              ACME account credentials (0600)
  certs/<tenant>/<application>/cert.pem      certificate chain
  certs/<tenant>/<application>/key.pem       private key (0600)
  certs/<tenant>/<application>/meta.json     names and expiry
```

Files are written atomically. The packaged systemd unit makes `/var/lib/scalws` writable
(`StateDirectory=scalws`); keep `state_dir` below it. `scalwsctl backup` copies the
certificate files referenced by the configuration, not the ACME state directory: back
that up separately, or let ScalWS order new certificates after a restore (mind the CA's
rate limits). See [cli.md](https://scalws.com/docs/guide/cli/).

Several instances under scalws-controller share the state directory: one instance places
the order (a lock file in the directory), challenge tokens are shared through the
directory so every instance can answer the CA, and every instance loads the stored
certificates.

### Checking certificates

```sh
scalwsctl certs          # application, state, days left, names or the last error
scalwsctl certs --json
```

The same data is available from the admin API (`GET /v1/certs`) and as metrics:
`scalws_acme_orders_total{result}` and `scalws_acme_certificate_expiry_seconds{tenant,app}`.
Log messages use the target `scalws::acme`:

```sh
journalctl -u scalwsd | grep scalws::acme       # or -u scalws-controller
```

If an order fails, check in this order: DNS points to this host, port 80 is reachable from
the internet and ends at ScalWS (not at another web server or a load balancer that
redirects), and the domain is not a wildcard. More in
[troubleshooting.md](https://scalws.com/docs/guide/troubleshooting/).

## HTTP/2

HTTP/2 needs no configuration:

* on TLS listeners it is negotiated with ALPN (`h2` preferred over `http/1.1`); clients
  that send no ALPN get HTTP/1.1;
* on plain-HTTP listeners, clients that start with the HTTP/2 preface ("prior knowledge",
  h2c) are served HTTP/2; everything else is HTTP/1.1.

Limits that apply to HTTP/2 are in `server.limits` (see [security.md](https://scalws.com/docs/guide/security/)):
[`h2_max_concurrent_streams`](https://scalws.com/docs/reference/server-limits/#server.limits.h2_max_concurrent_streams) (default 128), `max_header_bytes` (also the HTTP/2 header list
size), and `idle_timeout`. Rapid-reset protection (CVE-2023-44487) is built in.

## HTTP/3 (QUIC)

HTTP/3 is in every package (`.deb`, `.rpm`, `.apk`) and off by default
([ADR-0019](https://scalws.com/docs/adr/0019-http3/)). Turn it on per TLS listener:

```yaml
server:
  listeners:
    - name: https
      address: "0.0.0.0:443"
      http3: true
      tls: {acme: true}
    - name: https6
      address: "[::]:443"
      http3: true
      tls: {acme: true}
```

```sh
sudo scalwsctl check && sudo scalwsctl reload      # under scalws-controller
sudo scalwsctl check && sudo systemctl restart scalwsd   # under scalwsd.service
```

Under `scalwsd.service` a listener's `http3` is a listener change, which a reload refuses
with "configuration change requires a restart"
([reload or restart](https://scalws.com/docs/guide/configuration/#reload-or-restart)).

* [`http3: true`](https://scalws.com/docs/reference/server-listeners/#server.listeners-.http3) is only valid on TLS listeners. ScalWS then also listens on the same port
  number over **UDP**. The package does not open UDP 443 in the firewall: open it yourself
  ([installation.md](https://scalws.com/docs/guide/installation/#http3-udp-443) has the commands for ufw, firewalld,
  nftables and awall), and in the provider's firewall or security group. Until UDP 443 is
  reachable, browsers keep using HTTP/2 and nothing breaks.
* A `scalwsd` built from source has HTTP/3 only with the Cargo feature `http3`
  (`cargo build --release -p scalwsd --features scalwsd/http3`); without it, `http3: true`
  is refused at start with an error.
* QUIC uses the listener's certificates (configured and ACME), and they rotate on reload
  and renewal like those of the TCP listener. TLS 1.3 only, ALPN `h3`, no 0-RTT.
* HTTPS responses on that listener carry `Alt-Svc: h3=":443"; ma=86400` so browsers can
  switch to HTTP/3.
* Requests go through the same pipeline as HTTP/1.1 and HTTP/2: routing, limits, security
  rules, quotas, cache and logs are the same. Each connection takes a slot of
  `limits.max_connections`; streams per connection follow `h2_max_concurrent_streams`.
* Not supported over HTTP/3: WebSockets (extended CONNECT; they stay on HTTP/1.1 and
  HTTP/2) and 0-RTT (early data is off on purpose: it can be replayed).

With several instances (scalws-controller), HTTP/3 needs the controller's default
`listen: inherited`; the controller then runs a QUIC demultiplexer that routes each
connection to the instance that owns it. `server.listener_mode: reuseport` refuses
`http3: true`. Lifecycle, observability and limits of this setup are described in
[http3-multi-instance.md](https://scalws.com/docs/adaptive-runtime/http3-multi-instance/).

## Kernel TLS (kTLS)

On Linux, [`server.ktls: true`](https://scalws.com/docs/reference/server/#server.ktls) lets the kernel encrypt HTTPS/1.1 connections, so large
static files can be sent with `sendfile(2)` over TLS as well
([ADR-0030](https://scalws.com/docs/adr/0030-ktls/)). It is off by default.

```sh
sudo modprobe tls
echo tls | sudo tee /etc/modules-load.d/tls.conf    # load it at boot too
```

```yaml
server:
  ktls: true
```

* At start ScalWS probes which cipher suites the kernel supports. Without the `tls` module
  (or without any supported suite) it logs a warning and keeps TLS in user space.
* The handshake is always done by rustls. Only connections that negotiated HTTP/1.1 and a
  kernel-supported cipher suite switch to kTLS; HTTP/2 connections stay in user space.
* Encryption time then shows as system CPU time of the process.
* On older kernels that do not handle TLS 1.3 key updates, such a connection is closed
  when the client requests a key update; clients reconnect.

## Redirecting to HTTPS

### `https_redirect`

[`https_redirect: true`](https://scalws.com/docs/reference/applications/#tenants-.applications-.https_redirect) on an application answers every plain-HTTP request for its domains
with `301 Moved Permanently` to the same host, path and query over `https://`. The target
has no port, so it always points to port 443. ACME HTTP-01 challenges are still answered
over plain HTTP.

```yaml
tenants:
  - name: shop
    applications:
      - name: shop
        domains: [shop.example.com]
        acme: true
        https_redirect: true
        root: /srv/www/shop
        runtime: {type: static}
```

In a [site file](https://scalws.com/docs/sites/) the field has the same name (`https_redirect: true`), and
`scalwsctl site add … --https-redirect` sets it.

Security rules run before the redirect: a denied path is answered `404` over plain HTTP
as well, and rate limits count redirected requests.

### `www.` redirects

There is no separate `www` option for applications in `scalws.yaml`; a redirect from
`www.example.com` to `example.com` is an application of its own with a `redirect`
runtime:

```yaml
      - name: example-www
        domains: [www.example.com]
        acme: true
        runtime: {type: redirect, to: "https://example.com", status: 301, keep_path: true}
```

[`status`](https://scalws.com/docs/reference/runtime-redirect/#tenants-.applications-.runtime.redirect.status)
(default `301`) may be 301, 302, 303, 307 or 308;
[`keep_path`](https://scalws.com/docs/reference/runtime-redirect/#tenants-.applications-.runtime.redirect.keep_path)
(default `true`) appends the request's path and query.

Site files do this for you with [`www: redirect`](https://scalws.com/docs/reference/site-file/#www): the site gets a second application that
redirects `www.<domain>` to `https://<domain>` (or `http://` when the server has neither
ACME nor a TLS listener), keeping path and query, with the same `tls` setting as the site.
`www: alias` serves `www.<domain>` as part of the site instead, and `none` (the default in
the file) does not serve it. `scalwsctl site add` uses `--www redirect` by default for a
name with one dot such as `example.com`. See [sites.md](https://scalws.com/docs/sites/).

## See also

* [security.md](https://scalws.com/docs/guide/security/): limits, strict SNI, trusted proxies, admin API.
* [OPERATIONS.md](https://scalws.com/docs/operations/): installation, ACME quick start, performance settings.
* [THREAT_MODEL.md](https://scalws.com/docs/threat-model/): TLS and ACME threats and mitigations.