# Files, directories and services

URL: https://scalws.com/docs/guide/files-and-services/
Source: `docs/guide/files-and-services.md` in the ScalWS repository

Reference for everything the `scalws` package puts on a system and for the day-to-day
mechanics: where configuration and state live, the systemd units and their sandbox,
logs, reloads, backups and upgrades. Installation itself is covered in
[installation](https://scalws.com/docs/guide/installation/).

## Configuration: `/etc/scalws`

`/etc/scalws` is `0750 root:scalws`: the server (group `scalws`) can read it, only root can
write it.

| Path | Owner, mode | Content |
|---|---|---|
| `/etc/scalws/scalws.yaml` | `root:scalws 0640` | Server configuration: listeners, limits, ACME, tenants and applications. Installed with ports 80 and 443, ACME and the welcome page ([first start](https://scalws.com/docs/guide/installation/#first-start)); never replaced on upgrade. See [configuration](https://scalws.com/docs/guide/configuration/). |
| `/etc/scalws/controller.yaml` | `root:scalws 0640` | `scalws-controller` configuration: state directory, control socket, instance count and CPUs, autoscaling. Never replaced on upgrade. Reference: [docs/adaptive-runtime.md](https://scalws.com/docs/adaptive-runtime/). |
| `/etc/scalws/sites.d/` | `root:scalws 0750` | One file per site, included by `include: sites.d/*.yaml` ([site files](https://scalws.com/docs/sites/)). |
| `/etc/scalws/console/` | `root:root 0700` | `credentials`: console accounts (`0600`). |
| `/etc/scalws/tls/` | not created by the package | Conventional place for certificate and key files you manage yourself (used by the example configuration). |

### `scalws.yaml`

The packaged file:

```yaml
# Default configuration installed by the scalws package: ready to publish sites.
#
#   1. Point your domain's DNS (A, and AAAA if the server has IPv6) at this server.
#   2. sudo scalwsctl site add example.com
#      (static files in /srv/www/example.com; --php-fastcgi, --proxy, --redirect: see
#      `scalwsctl site add --help` and https://scalws.com/docs/sites/)
#
# Each site gets a Let's Encrypt certificate automatically (server.acme below). Ports 80
# and 443 must be reachable from the internet; with ufw: sudo ufw allow 'ScalWS'.
# Full reference: /usr/share/doc/scalws/examples/scalws.yaml and https://scalws.com/docs/
include: sites.d/*.yaml
server:
  listeners:
    - name: http
      address: "0.0.0.0:80"
    - name: https
      address: "0.0.0.0:443"
      tls:
        acme: true
    # IPv6 (the package comments these out on hosts without IPv6).
    - name: http6
      address: "[::]:80"
    - name: https6
      address: "[::]:443"
      tls:
        acme: true
  metrics:
    address: "127.0.0.1:9900"
  # Certificates from Let's Encrypt for every site with `tls: auto` (the default while this
  # is enabled). Using it accepts the Let's Encrypt Subscriber Agreement
  # (https://letsencrypt.org/repository/). Add `contact: ["mailto:you@example.com"]` to get
  # expiry notices.
  acme:
    enabled: true
    terms_of_service_agreed: true
  # Admin API (scalwsctl): /run/scalws/admin.sock, group scalws.
tenants:
  # Answers every host name that no site serves (the server's IP address, a domain whose
  # site is not added yet) with a short page. Remove it to answer such requests with 404.
  - name: default
    applications:
      - name: welcome
        domains: ["*"]
        root: /usr/share/scalws/welcome
        runtime: {type: static, index: [index.html]}
```

The welcome page is `/usr/share/scalws/welcome/index.html`. On a host without IPv6 the
package comments out the `http6` and `https6` listeners on installation.

Every section rejects unknown keys, so a typo fails the check instead of being ignored.
The fully commented reference is `/usr/share/doc/scalws/examples/scalws.yaml`.

### Site files

`include:` takes a path or a list of paths, relative to the directory of `scalws.yaml`;
an entry may have one `*` in the file name. Only the main file may include. The rules the
loader enforces:

* files are read in name order; names starting with `.` and files not matching the
  pattern are ignored (editor backups, `site remove` backups);
* each site file and the directory holding it must belong to root (or to the owner of the
  main configuration file) and must not be writable by group or others. Whoever can write
  there can serve any domain from any directory the server can read;
* a domain may be served once across all files.

`scalwsctl site add` writes files as root, mode 0640, group `scalws`. `site remove` renames
a file to `.<name>.removed-<time>`; `migrate` replaces a file only after keeping it as
`.<name>.replaced-<time>`.

### Certificate files

Certificates you manage yourself are referenced from a TLS listener
(`tls: {certificates: [{cert, key, server_names, default}]}`, see [TLS](https://scalws.com/docs/guide/tls/)). The server
runs as `scalws`, so it needs read access to them, for example:

```sh
sudo install -d -m 0750 -o root -g scalws /etc/scalws/tls
sudo install -m 0640 -o root -g scalws example.com.pem example.com.key /etc/scalws/tls/
```

## State: `/var/lib`

| Path | Owner, mode | Content |
|---|---|---|
| `/var/lib/scalws` | `scalws:scalws 0750` | Home of the `scalws` user; default parent of the ACME state. |
| `/var/lib/scalws/acme` | written by the server | `server.acme.state_dir`: `account.json`, `certs/<tenant>/<app>/{cert.pem,key.pem,meta.json}`, `challenges/`, `order.lock`. Shared by all instances under the controller (one of them orders, the others wait). |
| `/var/lib/scalws-controller` | `scalws:scalws` | Controller state (`controller.state_dir`): `generations/` (every configuration it applied, with site files expanded into it), `instances/`, `logs/` (one log per instance, `web-<n>.log`), `shares/` (`limit-share`, `tenant-limits.json`: what the instances read), `quic-lb/` (the QUIC demux socket). |
| `/var/lib/scalws-console` | `root:root 0700` | Console state: `audit.jsonl` (append-only audit log, 0600), `history/` (every configuration applied or found), `tls/` (generated certificate and key, 0600). |

Because the controller stores each generation with the site files expanded, a rollback to
an earlier generation restores exactly the sites of that generation, even after site files
changed.

## Runtime directories

| Path | Content |
|---|---|
| `/run/scalws` | `server.runtime_dir` (default `/run/scalws`): sockets and generated configuration of managed runtimes. Under `scalwsd.service` it is mode 0750 and holds `admin.sock`. |
| `/run/scalws/admin.sock` | Admin API of `scalwsd.service` (mode `0660 scalws:scalws`), used by `scalwsctl`. Members of group `scalws` may use it; every change is logged under `scalws::audit` with the caller's uid, gid and pid. |
| `/run/scalws/web-<n>/` | Under the controller: each instance's own runtime directory, with its `admin.sock`. |
| `/run/scalws-controller/control.sock` | Controller control API (mode 0600, user `scalws`): `scalws-controller status|scale|reload|autoscaling|purge`, and the console. Use `sudo`. |

`scalwsctl` finds the admin socket from `scalws.yaml` (`server.admin.socket`, default
`<runtime_dir>/admin.sock`). Under the controller that path does not exist; name an
instance's socket for read commands, for example:

```sh
sudo scalwsctl --socket /run/scalws/web-0/admin.sock status
```

`scalwsctl reload`, `site`, `migrate`, `limit` and `high` discover a running controller
themselves through `/etc/scalws/controller.yaml` and its control socket
(`--controller-config` to change the file, except for `reload`) and reload or apply limits
through it.

## Metrics and health

| Server | Address |
|---|---|
| `scalwsd.service` | `server.metrics.address`: `127.0.0.1:9900` in the packaged configuration (`127.0.0.1:9901` when the key is omitted). |
| `scalws-controller` | Instance *i* listens on `127.0.0.1:(metrics_base_port + i)`, default `9910`, `9911`, … |

Each serves `/metrics` (Prometheus), `/healthz`, read-only views and a read-only
dashboard at `/dashboard`. Keep these on loopback.

## Binaries

| Path | Purpose |
|---|---|
| `/usr/bin/scalwsd` | The web server. `scalwsd --config FILE [--check]`. |
| `/usr/bin/scalwsctl` | Operator CLI ([CLI](https://scalws.com/docs/guide/cli/)). |
| `/usr/bin/scalws-limitcpu`, `scalws-limitram`, `scalws-limitio` | `scalwsctl limit cpu|ram|io` |
| `/usr/bin/scalws-highcpu`, `scalws-highram`, `scalws-highio` | `scalwsctl high cpu|ram|io` |
| `/usr/bin/scalws-controller` | Adaptive Runtime controller. |
| `/usr/bin/scalws-console` | Web console. |

## systemd units

Two units run the web server and exclude each other; enable one. The console is
independent. When to use which: [installation](https://scalws.com/docs/guide/installation/#the-three-services).

### `scalwsd.service`

```ini
User=scalws
Group=scalws
ExecStartPre=/usr/bin/scalwsd --config /etc/scalws/scalws.yaml --check
ExecStart=/usr/bin/scalwsd --config /etc/scalws/scalws.yaml --restrict-suid-sgid
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=45
Restart=on-failure
LimitNOFILE=1048576
Delegate=yes
KillMode=control-group
RuntimeDirectory=scalws
RuntimeDirectoryMode=0750
StateDirectory=scalws
StateDirectoryMode=0750
```

* The configuration is checked before every start; a broken file keeps the unit from
  starting instead of failing half-way.
* `Delegate=yes`: the server manages a cgroup v2 subtree for tenant limits. It moves itself
  to `<unit>/server` and places applications in `<unit>/tenants/<tenant>/<app>`.
  Stopping the unit kills the whole control group, including application processes.
* Application processes never inherit the server's capabilities and receive SIGTERM if
  `scalwsd` dies.

### `scalws-controller.service`

```ini
User=scalws
Group=scalws
ExecStart=/usr/bin/scalws-controller run --config /etc/scalws/controller.yaml --restrict-suid-sgid
Conflicts=scalwsd.service
KillMode=process
Restart=on-failure
LimitNOFILE=1048576
Delegate=yes
DelegateSubgroup=controller
RuntimeDirectory=scalws-controller scalws
RuntimeDirectoryPreserve=yes
StateDirectory=scalws-controller scalws
StateDirectoryMode=0750
```

* `KillMode=process`: stopping or restarting the unit stops only the controller. Its
  `scalwsd` instances keep serving and a restarted controller adopts them. To take the
  whole service down:

  ```sh
  sudo systemctl stop scalws-controller
  sudo pkill -TERM -f 'scalwsd .*--instance-id'
  ```

  (`scalws-controller scale 0` is not allowed.)
* `DelegateSubgroup=controller` (systemd 254 or later): the controller runs in
  `<unit cgroup>/controller`, so instances can enable the cgroup controllers for tenant
  limits in the unit's group. Older systemd (EL8, EL9, Debian 12, Ubuntu 22.04) starts the
  controller in the unit's group itself, which cgroup v2 refuses while instances there use
  those controllers. On such a system the package adds
  `/usr/lib/systemd/system/scalws-controller.service.d/10-systemd-before-254.conf` with
  `KillMode=control-group`: the instances stop with the controller, and a restart starts
  new ones (a short interruption instead of a handover).
* Instances are started with `taskset` on their CPU slice and inherit the right to bind
  ports 80 and 443.

### `scalws-console.service`

```ini
ExecStart=/usr/bin/scalws-console run \
    --controller-config /etc/scalws/controller.yaml \
    --credentials /etc/scalws/console/credentials \
    --state-dir /var/lib/scalws-console \
    --listen 0.0.0.0:8090 \
    --controller-restart-command "systemctl restart scalws-controller"
After=network-online.target scalws-controller.service
Restart=on-failure
UMask=0077
Nice=10
CPUWeight=20
MemoryMax=256M
```

It runs as root, because it must reach the controller socket and replace
`/etc/scalws/scalws.yaml` atomically with its owner and mode, with a reduced capability set
and a read-only file system except `/etc/scalws` and `/var/lib/scalws-console`. It starts
after the controller but does not depend on it. Its low CPU weight and memory limit keep it
from competing with the web server. Options: [console operations](https://scalws.com/docs/console/operations/).

### Sandbox summary

| Setting | `scalwsd` | `scalws-controller` | `scalws-console` |
|---|---|---|---|
| User | `scalws` | `scalws` | root |
| Capabilities | `CAP_NET_BIND_SERVICE` (ambient and bounding) | `CAP_NET_BIND_SERVICE` (ambient and bounding) | bounding set `CAP_DAC_OVERRIDE CAP_CHOWN CAP_FOWNER` |
| `NoNewPrivileges` | yes | yes | yes |
| `ProtectSystem` | `strict` | `strict` | `strict` (`ReadWritePaths=/etc/scalws /var/lib/scalws-console /srv/www /home`) |
| `ProtectHome` | `read-only` | `read-only` | `read-only` (`/home` writable, `/root` and `/run/user` not) |
| `PrivateTmp` | yes | yes | yes |
| `PrivateDevices` | yes | – | yes |
| `ProtectKernelTunables` | yes | – | yes |
| `ProtectKernelModules` | yes | yes | yes |
| `ProtectControlGroups` | no (would break delegation) | – | yes |
| `RestrictNamespaces` | yes | – | yes |
| `RestrictSUIDSGID` | `--restrict-suid-sgid` instead (below) | `--restrict-suid-sgid` instead (below) | yes |
| `LockPersonality` | yes | yes | yes |
| `MemoryDenyWriteExecute` | no (Node.js V8 and PHP's opcache JIT need it) | – | yes |
| `SystemCallArchitectures=native` | yes | yes | yes |
| `SystemCallFilter` | – | – | `@system-service` and `pidfd_open`; other calls fail with `ENOSYS` |
| `ReadOnlyPaths=/srv` | yes | yes | – |

Consequences:

* Document roots under `/home` (for example `/home/<user>/public_html`) can be served:
  `/home` is read-only for the server. User `scalws` needs search (`x`) permission on every
  directory on the way and read permission on the files; `scalwsctl site add` warns when
  the mode bits do not allow it. `/root` and `/run/user` are not visible.
* `/srv` is read-only for the server and for managed applications. If an application
  started by `scalwsd.service` must write there, add a drop-in:

  ```ini
  # /etc/systemd/system/scalwsd.service.d/writable.conf
  [Service]
  ReadWritePaths=/srv/www/blog/wp-content
  ```

  then `sudo systemctl daemon-reload && sudo systemctl restart scalwsd`.
* `scalwsd.service` and `scalws-controller.service` start their binaries with
  `--restrict-suid-sgid` instead of setting `RestrictSUIDSGID=`: the same seccomp rules
  (no set-user-ID or set-group-ID mode on a file or directory created by open, openat,
  creat, mkdir, mkdirat, mknod or mknodat, or set by chmod, fchmod, fchmodat or
  fchmodat2), inherited by the instances, the QUIC demux and the applications, but openat2
  stays available. systemd cannot read openat2's mode and refuses the whole call, which
  would leave the static files without the kernel's `RESOLVE_BENEATH` lookups (every
  instance would log `openat2 unavailable`). The one creation the rules cannot see is
  openat2 with `O_CREAT` and such a mode (see [security](https://scalws.com/docs/guide/security/)).
* Running tenants' processes as their own Unix users (`tenants[].user`) needs four more
  capabilities in a drop-in; see [security](https://scalws.com/docs/guide/security/) and
  [operations](https://scalws.com/docs/operations/#tenant-users-adr-0026-optional).

Never edit the unit files in `/usr/lib/systemd/system`; upgrades replace them. Use
drop-ins (`sudo systemctl edit scalwsd`).

## Kernel setting: `/usr/lib/sysctl.d/90-scalws-adaptive-runtime.conf`

```ini
net.ipv4.tcp_migrate_req = 1
```

When a `scalwsd` instance leaves a `SO_REUSEPORT` group, connections still queued on its
listening sockets move to another instance instead of being reset (Linux 5.14 or later).
It matters for the controller's `listen: reuseport` mode. The package applies it on
install and systemd loads it at every boot. Override it, if needed, with a file of the
same name in `/etc/sysctl.d/`.

## Logs

All three units write to the journal:

```sh
journalctl -u scalwsd -f
journalctl -u scalws-controller -f
journalctl -u scalws-console -f
```

Under the controller, each instance's output goes to its own file instead:
`/var/lib/scalws-controller/logs/web-<n>.log`. The packages rotate these files with
`/etc/scalws-logrotate-instances.conf`: daily, and within the hour once a file passes
100 MB, keeping 7 (compressed). Only the hourly run reads that file, with a state file of
its own (it is not in `/etc/logrotate.d`, so the system's daily run leaves these logs
alone): `scalws-logrotate.timer` on systemd hosts and `/etc/periodic/hourly/scalws-logrotate`
on Alpine (crond's periodic jobs; `rc-update add crond` if it is not in a runlevel). The
package enables the timer once; `systemctl disable --now scalws-logrotate.timer` stays
across upgrades. The deb and rpm recommend `logrotate`, the apk depends on it; without it
the files grow until the file system is full. Under another `controller.state_dir`, change
the path in `/etc/scalws-logrotate-instances.conf` (it keeps `su scalws scalws`). The file
lies outside `/etc/scalws` on purpose: logrotate runs as root and runs the scripts of its
configuration, and the console (root) may write everything in `/etc/scalws`.

`scalwsd` logs JSON lines by default, including one access-log line per request. Settings
in `server.logging`:

| Key | Default | Meaning |
|---|---|---|
| `format` | `json` | `json` or `text`. |
| `level` | `info` | `tracing` filter, e.g. `info,scalws_proxy=debug`. |
| `access_log` | `true` | Log every request. |

Useful log targets: `scalws::access` (requests), `scalws::audit` (admin API changes),
`scalws::acme` (certificates). The console keeps its own audit log in
`/var/lib/scalws-console/audit.jsonl`.

## How reload works

A reload validates the configuration (with all site files), prepares it (certificates,
document roots, runtimes) and swaps it in atomically. If anything fails, the active
configuration keeps serving and the error is reported and logged. Run the same check
without touching the server with:

```sh
sudo scalwsctl check          # alias: scalwsctl config-test
```

| Server | Reload | What happens |
|---|---|---|
| `scalwsd.service` | `scalwsctl reload`, `sudo systemctl reload scalwsd` (SIGHUP) | In-process swap. Changes to listeners, the metrics/admin endpoints, connection limits, cgroups, cache memory, per-IP limits or the AI advisor are refused with "requires a restart": `sudo systemctl restart scalwsd`. |
| `scalws-controller` | `sudo scalws-controller reload` | The controller validates `scalws.yaml` as a new generation, binds any new listening addresses first, then rolls instances onto it and drains the old ones. A bind failure rejects the reload and leaves the current sockets as they were. |
| Console | Apply in the browser | The console validates the draft (with the site files), writes `scalws.yaml` and reloads through the controller, then waits up to 120 s (`--health-timeout-secs`) for the new generation. |

`scalwsctl site add|remove`, `migrate` and `limit --persist` reload for you, through the
controller when it runs.

Under `scalwsd.service` the server keeps earlier configurations for comparison and
rollback:

```sh
scalwsctl history
scalwsctl diff                       # previous -> active
scalwsctl rollback                   # back to the previous one (or --generation G)
```

## Backup and restore

`scalwsctl backup` copies the configuration file, its site files and the files they
reference (TLS certificates and keys, `tls_ca_file` bundles) into a new directory with a
`manifest.json`:

```sh
sudo scalwsctl backup /var/backups/scalws-$(date +%F)
sudo scalwsctl restore /var/backups/scalws-2026-10-05            # dry run: what would change
sudo scalwsctl restore /var/backups/scalws-2026-10-05 --apply
```

* The target directory must not exist or must be empty. Files are stored under
  `files/<absolute path>`.
* `restore` without `--apply` only lists each file as `unchanged`, `changed` or `new`.
  With `--apply` it keeps every replaced file as `<file>.bak.<unix time>`, writes the
  restored files with their original mode (and the replaced file's owner), then checks the
  restored configuration. Apply it with `scalwsctl reload` or `sudo scalws-controller
  reload`.

What `scalwsctl backup` does **not** contain: ACME certificates and the
account in `/var/lib/scalws/acme`, `controller.yaml`, console state and credentials,
document roots and application data. For a full configuration backup, also archive the
directories:

```sh
sudo tar -C / -czf /var/backups/scalws-files-$(date +%F).tar.gz \
    etc/scalws var/lib/scalws var/lib/scalws-console
```

## Upgrades

Upgrading the package, what happens to running services, and rollback: [Upgrading](https://scalws.com/docs/guide/upgrading/).

## See also

* [Operations](https://scalws.com/docs/operations/): admin API commands, tenant limits, remote
  administration, performance settings.
* [Troubleshooting](https://scalws.com/docs/guide/troubleshooting/) and [FAQ](https://scalws.com/docs/guide/faq/).