Users' guide

HTTPS and TLS

This guide covers how ScalWS serves HTTPS: TLS listeners and certificate files, certificates from Let’s Encrypt (ACME), HTTP/2, HTTP/3, kernel TLS, and redirects to HTTPS. The basics of the configuration file are in configuration.md, and applications are covered in applications.md.

TLS is implemented with rustls and the aws-lc-rs crypto provider (ADR-0002).

TLS listeners

A listener becomes a TLS listener when it has a tls: block. A TLS listener needs at least one certificate, or acme: true to also serve the ACME-managed certificates of applications with acme: true (see Let’s Encrypt):

server:
  listeners:
    - name: http
      address: "0.0.0.0:80"
    - name: https
      address: "0.0.0.0:443"
      tls:
        certificates:
          - cert: /etc/scalws/tls/example.com/fullchain.pem
            key: /etc/scalws/tls/example.com/privkey.pem
            server_names: [example.com, www.example.com]
            default: true
          - cert: /etc/scalws/tls/example.net/fullchain.pem
            key: /etc/scalws/tls/example.net/privkey.pem
            server_names: ["*.example.net", example.net]

The other listener fields that relate to TLS are strict_sni (only valid on TLS listeners, see Strict SNI) and http3 (see HTTP/3). Every key of a listener, including tls and its certificate entries, with its syntax, default and reload behaviour: reference.

Certificate entries

Each entry of certificates needs cert, key and at least one name in server_names, except the default entry, which may have server_names: [] and is then served only to clients that send no SNI name or one no other entry lists:

cert holds the end-entity (leaf) certificate first, followed by the intermediate certificates. Do not include the root. A fullchain.pem as written by most ACME clients has this order.
key is PKCS#8 (BEGIN PRIVATE KEY), PKCS#1 (BEGIN RSA PRIVATE KEY) or SEC1 (BEGIN EC PRIVATE KEY).
server_names are exact names (example.com) or wildcards (*.example.com).
default (at most one per listener) is used when the client sends no SNI or a name no entry lists.

When the configuration is checked (scalwsctl check, a reload, or start-up), every certificate is loaded and verified:

the PEM files can be read and the chain is not empty;
the private key matches the certificate;
every name in server_names is covered by the certificate’s subject alternative names. A wildcard entry needs a matching wildcard SAN in the certificate. A wildcard SAN directly under one label, such as *.org, is accepted for one label below it (example.org), as RFC 6125 and clients such as Go do; browsers refuse such certificates, so the check only decides what ScalWS may serve, not what a client trusts;
a name may be assigned to only one certificate per listener.

If anything fails, a reload is refused and the running configuration stays active; the error names the file and the problem, for example …/fullchain.pem: certificate is not valid for server name "shop.example.com".

The server runs as the scalws user, so the certificate and key files must be readable by that user (for example root:scalws, mode 0640 for the key).

Certificates from a file

A listener can read more certificate entries from a separate YAML file, for example one that a tool generates. The file is a list with the keys of certificates; its entries are appended to the listener’s certificates when the configuration is loaded from a file (like include for site files), so a stored generation is self-contained.

# fragment
server:
  listeners:
    - name: https
      address: "0.0.0.0:443"
      tls:
        acme: true
        certificates_file: "-/etc/scalws/panel/certificates.yaml"

A relative path is resolved against the directory of the configuration file. A leading - makes a missing file mean “no entries”; without it a missing file is an error. An empty file or [] is an empty list.

How a certificate is chosen (SNI)

For each new TLS connection ScalWS looks at the SNI name the client sends:

an exact match in server_names;
otherwise a wildcard: *.example.net matches exactly one extra label (shop.example.net, but not example.net and not a.shop.example.net). List the bare domain separately if it should use the same certificate, as in the example above;
otherwise the default certificate;
otherwise the handshake fails.

A client that sends no SNI at all (for example a request to the IP address) gets the default certificate, or a failed handshake when no certificate is marked default. Names are compared case-insensitively and without a trailing dot.

Certificate selection is independent of routing: the certificate only has to be valid for the name; which application answers is decided by the request’s Host (see applications.md).

Replacing certificate files

Certificates are reloadable. After you replace the files (for example after renewing them with an external tool), run:

sudo scalwsctl reload

The new files are checked first; existing connections keep their certificate and new handshakes get the new one. Adding or removing a TLS listener, or changing its address, http3 or proxy_protocol, is a listener change and needs a restart (the reload says “requires a restart”).

The expiry of configured certificates is exported as the metric scalws_tls_certificate_expiry_seconds (Unix time).

Strict SNI

Browsers may reuse one HTTP/2 connection for several host names when the certificate covers them all (connection coalescing). With strict_sni: true on a listener, a request whose Host differs from the SNI name of its connection is answered with 421 Misdirected Request; the client then opens a new connection for that host. This prevents one tenant’s connection from being used to reach another tenant’s site (domain fronting).

    - name: https
      address: "0.0.0.0:443"
      strict_sni: true
      tls: {acme: true}

Requests on connections without an SNI name are not affected. Refused requests are counted in scalws_security_blocked_total{reason="misdirected"}.

TLS versions, ciphers and ALPN

There are no settings for protocol versions or cipher suites. What ScalWS uses:

	HTTPS listeners (TCP)	HTTP/3 (QUIC)
TLS versions	TLS 1.2 and TLS 1.3	TLS 1.3 only
Cipher suites	rustls defaults for the aws-lc-rs provider (AEAD suites with forward secrecy only)	same, TLS 1.3 suites
ALPN	`h2`, then `http/1.1`	`h3`
Session resumption	stateless TLS 1.3 session tickets	n/a
0-RTT (early data)	not enabled	disabled

Client certificates are not requested on application listeners (mutual TLS exists only for the remote admin API, see security.md). OCSP stapling is not implemented, and ScalWS does not add an Strict-Transport-Security header by itself; applications can send it, or with htaccess: true you can set it with Header set Strict-Transport-Security "max-age=31536000" in an .htaccess file.

With several instances under scalws-controller, session tickets are per instance, so a resumption may fall back to a full handshake when the client lands on another instance.

Let’s Encrypt (ACME)

ScalWS can obtain and renew certificates itself (ADR-0018). It uses the HTTP-01 challenge only, so:

the domain’s DNS must point to this server, and port 80 must reach a plain-HTTP listener of ScalWS;
wildcard domains (*.example.com) cannot get an ACME certificate (they need DNS-01, which is not implemented): use a certificate file for them;
IP addresses and localhost are refused.

Three things have to be switched on:

server:
  listeners:
    - {name: http, address: "0.0.0.0:80"}                       # HTTP-01 challenges
    - {name: https, address: "0.0.0.0:443", tls: {acme: true}}  # serves ACME certificates
  acme:
    enabled: true
    contact: ["mailto:[email protected]"]
    terms_of_service_agreed: true

tenants:
  - name: shop
    applications:
      - name: shop
        domains: [shop.example.com, www.shop.example.com]
        acme: true                       # one certificate for all domains of this app
        root: /srv/www/shop
        runtime: {type: static}

With site files, tls: auto is the default as soon as server.acme is enabled, so scalwsctl site add needs no extra option.

`server.acme` settings

Every key of server.acme, with its syntax, default and reload behaviour: reference. The directory must be an https:// URL, each contact a mailto: address, and check_interval at least 1m. ca_bundle is meant for test CAs such as Pebble; state_dir is described below.

scalwsctl check refuses the configuration when enabled is true but terms_of_service_agreed is not, when no TLS listener has acme: true, when there is no plain-HTTP listener, or when an application uses acme: true while ACME is disabled.

Try a new setup against the Let’s Encrypt staging directory first, to stay clear of the production rate limits:

  acme:
    enabled: true
    directory: https://acme-staging-v02.api.letsencrypt.org/directory
    contact: ["mailto:[email protected]"]
    terms_of_service_agreed: true

What happens

The certificate manager checks every application with acme: true at start, every check_interval, and after every configuration change. An application gets a new order when it has no certificate yet, when its domains changed, or when its certificate expires within renew_before.
The CA’s validation request GET /.well-known/acme-challenge/<token> is answered on plain-HTTP listeners for pending tokens, before routing and before https_redirect. Other /.well-known/ paths go to the application as usual.
A new certificate is activated like a reload: existing connections keep the old one, new handshakes get the new one. No restart is needed.
A failed order is retried with a back-off per application, starting at 5 minutes and doubling up to 24 hours.
Until an application has its first certificate, handshakes for its names get the listener’s default certificate if there is one, and fail otherwise.
When a configured certificate file and an ACME certificate cover the same name, the configured file wins.

State directory

/var/lib/scalws/acme/
  account.json                              ACME account credentials (0600)
  certs/<tenant>/<application>/cert.pem      certificate chain
  certs/<tenant>/<application>/key.pem       private key (0600)
  certs/<tenant>/<application>/meta.json     names and expiry

Files are written atomically. The packaged systemd unit makes /var/lib/scalws writable (StateDirectory=scalws); keep state_dir below it. scalwsctl backup copies the certificate files referenced by the configuration, not the ACME state directory: back that up separately, or let ScalWS order new certificates after a restore (mind the CA’s rate limits). See cli.md.

Several instances under scalws-controller share the state directory: one instance places the order (a lock file in the directory), challenge tokens are shared through the directory so every instance can answer the CA, and every instance loads the stored certificates.

Checking certificates

scalwsctl certs          # application, state, days left, names or the last error
scalwsctl certs --json

The same data is available from the admin API (GET /v1/certs) and as metrics: scalws_acme_orders_total{result} and scalws_acme_certificate_expiry_seconds{tenant,app}. Log messages use the target scalws::acme:

journalctl -u scalwsd | grep scalws::acme       # or -u scalws-controller

If an order fails, check in this order: DNS points to this host, port 80 is reachable from the internet and ends at ScalWS (not at another web server or a load balancer that redirects), and the domain is not a wildcard. More in troubleshooting.md.

HTTP/2

HTTP/2 needs no configuration:

on TLS listeners it is negotiated with ALPN (h2 preferred over http/1.1); clients that send no ALPN get HTTP/1.1;
on plain-HTTP listeners, clients that start with the HTTP/2 preface (“prior knowledge”, h2c) are served HTTP/2; everything else is HTTP/1.1.

Limits that apply to HTTP/2 are in server.limits (see security.md): h2_max_concurrent_streams (default 128), max_header_bytes (also the HTTP/2 header list size), and idle_timeout. Rapid-reset protection (CVE-2023-44487) is built in.

HTTP/3 (QUIC)

HTTP/3 is in every package (.deb, .rpm, .apk) and off by default (ADR-0019). Turn it on per TLS listener:

server:
  listeners:
    - name: https
      address: "0.0.0.0:443"
      http3: true
      tls: {acme: true}
    - name: https6
      address: "[::]:443"
      http3: true
      tls: {acme: true}

sudo scalwsctl check && sudo scalwsctl reload      # under scalws-controller
sudo scalwsctl check && sudo systemctl restart scalwsd   # under scalwsd.service

Under scalwsd.service a listener’s http3 is a listener change, which a reload refuses with “configuration change requires a restart” (reload or restart).

http3: true is only valid on TLS listeners. ScalWS then also listens on the same port number over UDP. The package does not open UDP 443 in the firewall: open it yourself (installation.md has the commands for ufw, firewalld, nftables and awall), and in the provider’s firewall or security group. Until UDP 443 is reachable, browsers keep using HTTP/2 and nothing breaks.
A scalwsd built from source has HTTP/3 only with the Cargo feature http3 (cargo build --release -p scalwsd --features scalwsd/http3); without it, http3: true is refused at start with an error.
QUIC uses the listener’s certificates (configured and ACME), and they rotate on reload and renewal like those of the TCP listener. TLS 1.3 only, ALPN h3, no 0-RTT.
HTTPS responses on that listener carry Alt-Svc: h3=":443"; ma=86400 so browsers can switch to HTTP/3.
Requests go through the same pipeline as HTTP/1.1 and HTTP/2: routing, limits, security rules, quotas, cache and logs are the same. Each connection takes a slot of limits.max_connections; streams per connection follow h2_max_concurrent_streams.
Not supported over HTTP/3: WebSockets (extended CONNECT; they stay on HTTP/1.1 and HTTP/2) and 0-RTT (early data is off on purpose: it can be replayed).

With several instances (scalws-controller), HTTP/3 needs the controller’s default listen: inherited; the controller then runs a QUIC demultiplexer that routes each connection to the instance that owns it. server.listener_mode: reuseport refuses http3: true. Lifecycle, observability and limits of this setup are described in http3-multi-instance.md.

Kernel TLS (kTLS)

On Linux, server.ktls: true lets the kernel encrypt HTTPS/1.1 connections, so large static files can be sent with sendfile(2) over TLS as well (ADR-0030). It is off by default.

sudo modprobe tls
echo tls | sudo tee /etc/modules-load.d/tls.conf    # load it at boot too

server:
  ktls: true

At start ScalWS probes which cipher suites the kernel supports. Without the tls module (or without any supported suite) it logs a warning and keeps TLS in user space.
The handshake is always done by rustls. Only connections that negotiated HTTP/1.1 and a kernel-supported cipher suite switch to kTLS; HTTP/2 connections stay in user space.
Encryption time then shows as system CPU time of the process.
On older kernels that do not handle TLS 1.3 key updates, such a connection is closed when the client requests a key update; clients reconnect.

Redirecting to HTTPS

`https_redirect`

https_redirect: true on an application answers every plain-HTTP request for its domains with 301 Moved Permanently to the same host, path and query over https://. The target has no port, so it always points to port 443. ACME HTTP-01 challenges are still answered over plain HTTP.

tenants:
  - name: shop
    applications:
      - name: shop
        domains: [shop.example.com]
        acme: true
        https_redirect: true
        root: /srv/www/shop
        runtime: {type: static}

In a site file the field has the same name (https_redirect: true), and scalwsctl site add … --https-redirect sets it.

Security rules run before the redirect: a denied path is answered 404 over plain HTTP as well, and rate limits count redirected requests.

`www.` redirects

There is no separate www option for applications in scalws.yaml; a redirect from www.example.com to example.com is an application of its own with a redirect runtime:

      - name: example-www
        domains: [www.example.com]
        acme: true
        runtime: {type: redirect, to: "https://example.com", status: 301, keep_path: true}

status (default 301) may be 301, 302, 303, 307 or 308; keep_path (default true) appends the request’s path and query.

Site files do this for you with www: redirect: the site gets a second application that redirects www.<domain> to https://<domain> (or http:// when the server has neither ACME nor a TLS listener), keeping path and query, with the same tls setting as the site. www: alias serves www.<domain> as part of the site instead, and none (the default in the file) does not serve it. scalwsctl site add uses --www redirect by default for a name with one dot such as example.com. See sites.md.