Operations

Operations guide

Install

From the ScalWS apt repository (Ubuntu 24.04 or later, Debian 13 or later, amd64):

curl -fsSL https://scalws.com/apt/scalws-archive-keyring.gpg   -o /usr/share/keyrings/scalws-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/scalws-archive-keyring.gpg] https://scalws.com/apt stable main'   > /etc/apt/sources.list.d/scalws.list
apt update && apt install scalws

The repository is signed with the “ScalWS Archive Signing Key” (fingerprint 0B49 CE44 88DC 3A3E 4F63 1218 CBBB F99C 69B9 9EF0). Upgrades come with apt upgrade. ScalWS is not in the distributions’ official archives yet.

From a package file:

apt install ./scalws_<version>_amd64.deb      # Debian/Ubuntu
dnf install ./scalws-<version>-1.x86_64.rpm   # Fedora/RHEL family (glibc ≥ 2.36)

The package installs /usr/bin/scalwsd, /usr/bin/scalwsctl, the systemd unit, shell completions, the system user/group scalws and a loopback-only /etc/scalws/scalws.yaml (0640 root:scalws, kept on upgrade). The full reference configuration is in /usr/share/doc/scalws/examples/scalws.yaml.

scalwsctl check                       # validate + prepare exactly like a reload
systemctl enable --now scalwsd
usermod -aG scalws alice              # operators use scalwsctl via the admin socket

Packages for amd64 and arm64 (ARCH=arm64) are built with scripts/package.sh and verified with scripts/package.sh --verify (clean Debian and Fedora containers: user, files, permissions, configuration check, server start, scalwsctl over the socket, access denied outside group scalws).

Privileges

The unit runs as scalws with only CAP_NET_BIND_SERVICE, Delegate=yes for cgroup v2 (tenant limits) and a hardened sandbox. scalws never needs root; it logs a warning when started as root.

Application processes never inherit scalws’s capabilities (setpriv --inh-caps=-all --ambient-caps=-all --no-new-privs) and receive SIGTERM if scalwsd dies, even when it is killed outside systemd (--pdeathsig); processes left in cgroup leaves by a crashed instance are killed at the next start.

Tenant users (ADR-0026, optional)

tenants[].user: web1 runs that tenant’s PHP-FPM, Node.js and Python processes as the Unix account web1 (not root; container applications are refused for such tenants). scalwsd then needs four more capabilities — this makes it root-equivalent if compromised, the same trade-off as a root PHP-FPM master:

# /etc/systemd/system/scalwsd.service.d/tenant-users.conf
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_CHOWN CAP_KILL
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_CHOWN CAP_KILL

Each such tenant gets its own socket directory (<runtime_dir>/t/<hash>, owner web1:scalws, mode 2770); other tenants cannot list it. Document roots must be readable by the tenant (PHP, Node, Python) and by group scalws (static files served by scalwsd), e.g. chown -R web1:scalws /srv/web1 && chmod -R g+rX,o-rwx /srv/web1. Changing a tenant’s user restarts its applications on reload.

Admin API and scalwsctl (ADR-0017)

/run/scalws/admin.sock (mode 0660 scalws:scalws, directory 0750) accepts everything; every change is logged under scalws::audit with the caller’s uid/gid/pid. The loopback metrics listener (127.0.0.1:9900 in the default configuration) serves /metrics, /healthz, read-only views and a read-only dashboard at /dashboard.

A reload that changes listeners, the metrics/admin endpoints, connection limits, cgroups, cache memory, per-IP limits or the AI advisor is refused with “requires a restart”: systemctl restart scalwsd.

Tenant resource limits (ADR-0010)

With cgroups enabled (server.cgroups), the busiest tenants and their limits:

scalws-highcpu            # CPU, percent of one CPU, over one second
scalws-highram            # memory in use (and share of the host)
scalws-highio             # disk read/write MiB/s and I/O pressure
scalws-highcpu -n 5 --json

Limit a tenant, named by its tenant name or its Unix user (tenants[].user):

scalws-limitcpu ali 50%          # half a CPU (200% = two CPUs)
scalws-limitram ali 30%          # 30% of the host's memory, or a size: 2GiB
scalws-limitio  ali 50%          # half a default tenant's share of disk time
scalws-limitio  ali 20MiB/s      # at most 20 MiB/s of reads and 20 MiB/s of writes
scalws-limitcpu ali --clear      # back to the configuration
scalws-limitram ali 2GiB --persist

A limit applies at once and lasts until scalwsd restarts (under scalws-controller, until its instances are replaced); it survives reloads. --persist also writes it to the tenant’s resources in scalws.yaml (comments are kept; the previous file stays as scalws.yaml.bak.<time>) and reloads. These are scalwsctl limit cpu|ram|io and scalwsctl high cpu|ram|io under other names; they use the admin socket, so they run as root or a member of the scalws group.

Remote administration (optional)

server:
  admin:
    remote: {address: "0.0.0.0:9443", cert: /etc/scalws/admin.pem, key: /etc/scalws/admin.key,
             client_ca: /etc/scalws/operators-ca.pem}

Only clients with a certificate issued by client_ca can connect (mutual TLS): scalwsctl --remote host:9443 --ca server-ca.pem --client-cert alice.pem --client-key alice.key status.

Certificates (ACME)

server:
  listeners:
    - {name: http, address: "0.0.0.0:80"}                 # HTTP-01 challenges
    - {name: https, address: "0.0.0.0:443", tls: {acme: true}}
  acme:
    enabled: true
    contact: ["mailto:[email protected]"]
    terms_of_service_agreed: true
tenants:
  - name: acme
    applications:
      - {name: shop, domains: [shop.example.com, www.shop.example.com], acme: true, ...}

scalwsctl certs shows each application’s names, days left and errors. Renewal starts 30 days before expiry (renew_before) and is activated without a restart. Try the configuration against the Let’s Encrypt staging directory first (directory: https://acme-staging-v02.api.letsencrypt.org/directory).

Containers

runtime: {type: container} runs images with podman (default) or docker; install podman and catatonit on the host. With cgroup enforcement and podman, containers are created under the tenant’s cgroup, so tenant limits include them. scalwsctl scale and scalwsctl restart work as for Node.js pools.

Performance settings

• server.event_loops (ADR-0029): unset = one event loop per CPU scalwsd may run on (honours taskset/cgroup CPU limits), 0 = one shared work-stealing runtime. scalwsd --worker-threads N sets the count.
• server.open_file_cache (ADR-0028): {max_entries: 10000} by default, every hit is checked against the file; valid: 60s trusts cached files for that long (like nginx’s open_file_cache_valid), max_entries: 0 turns it off.
• server.sendfile (ADR-0027, default on): large static files on plain HTTP/1.
• server.ktls: true (ADR-0030): kernel TLS for HTTPS/1.1 so large files use sendfile over TLS too. Load the module first (modprobe tls, persist it in /etc/modules-load.d/tls.conf); without it scalwsd logs a warning and keeps TLS in user space.
• server.http1_server: hyper | experimental (ADR-0046, default hyper, read at start-up): experimental serves HTTP/1.1 with scalws’s own server (strict framing, differential-tested against hyper, fuzzed, audited). Opt-in until it has passed the real-application soak.
• server.upstream_client: hyper | experimental (ADR-0045, default hyper) and, per proxy route, client: overriding it: the HTTP/1 client to upstreams and to managed Node/Python applications. Same opt-in status.
• Release binaries are profile-guided (ADR-0047): tools/pgo-build.sh builds, trains and rebuilds scalwsd (PGO_BOLT=1 adds a BOLT pass on hosts with last-branch records). A plain cargo build --release works too, only slower.

.htaccess (ADR-0031)

applications[].htaccess: true reads .htaccess files from the application’s root down to the requested directory (mod_rewrite, Redirect/RedirectMatch, Require/Order/ Allow/Deny, <Files>/<FilesMatch>, ErrorDocument, DirectoryIndex, Header, Expires). Edits apply within 2 seconds. Unsupported or refused lines (php_value, [P] proxy rules, SetHandler, look-around regexes) are skipped and logged with the file name (journalctl -u scalwsd | grep htaccess).

Backup and restore

scalwsctl backup /var/backups/scalws-$(date +%F)     # config + certificates/keys + manifest
scalwsctl restore /var/backups/scalws-2026-10-05     # dry run: what would change
scalwsctl restore /var/backups/scalws-2026-10-05 --apply && scalwsctl reload

Restore keeps every replaced file as <file>.bak.<unix time> and validates the restored configuration. Document roots and application data are not part of the backup.

Upgrade

Install the new package; a running service is restarted by the package scripts. The configuration file is never replaced. Check scalwsctl status for the version.