# Upgrading

URL: https://scalws.com/docs/guide/upgrading/
Source: `docs/guide/upgrading.md` in the ScalWS repository

This page explains how to upgrade the `scalws` package, what the upgrade does to running services and configuration files, and how to go back if something goes wrong.

## Before you upgrade: back up

Take a backup of the configuration first:

```sh
sudo scalwsctl backup /var/backups/scalws-$(date +%F)
```

It copies `/etc/scalws/scalws.yaml`, its site files and the certificate and key files they
reference into a new directory. It does not contain `controller.yaml`, the ACME account
and certificates, or the console's state and credentials; archive those directories as
well:

```sh
sudo tar -C / -czf /var/backups/scalws-files-$(date +%F).tar.gz \
    etc/scalws var/lib/scalws var/lib/scalws-console
```

Note the version you are running, so you can return to it:

```sh
scalwsd --version
```

Details: [Files and services](https://scalws.com/docs/guide/files-and-services/#backup-and-restore) and
[scalwsctl backup](https://scalws.com/docs/reference/programs/scalwsctl/#backup-and-restore).

## Upgrade the package

From the apt repository:

```sh
sudo apt update && sudo apt upgrade
```

`sudo apt install scalws` upgrades only ScalWS.

From the dnf repository (RHEL family, Fedora):

```sh
sudo dnf upgrade scalws
```

From the same repository with zypper (openSUSE Leap, SLES):

```sh
sudo zypper up scalws
```

From the apk repository (Alpine Linux; OpenRC services, see
[Alpine Linux](https://scalws.com/docs/guide/installation/#logs-cgroups-and-diskless-systems)):

```sh
doas apk upgrade scalws
```

From a package file, install the newer file the same way as the first one
([Installation](https://scalws.com/docs/guide/installation/#from-a-package-file)).

## What happens to running services

The package scripts (`preinstall.sh`, `postinstall.sh`, `preremove.sh`) do the following on
an upgrade, with apt, dnf and zypper:

* Binaries, systemd unit files, shell completions and
  `/usr/lib/sysctl.d/90-scalws-adaptive-runtime.conf` are replaced. systemd re-reads the
  unit files and the sysctl setting is applied again.
* Ownership is set again on `/etc/scalws` (`root:scalws`, mode 0750),
  `/etc/scalws/scalws.yaml` and `/etc/scalws/controller.yaml` (`root:scalws`).
* **scalwsd.service**, if it is running, is restarted onto the new binary. The old process
  drains its connections and exits; the new one checks the configuration
  (`ExecStartPre=... --check`) and starts. While one has stopped and the other has not yet
  started, no process accepts connections. If the new version refuses the configuration,
  the unit does not start.
* **scalws-controller.service**, if it is running, is restarted. This restarts only the
  controller (`KillMode=process`): its `scalwsd` instances keep serving on the old binary
  while the new controller adopts them. The new controller then
  [rolls them onto the new binary](#roll-the-instances-onto-the-new-binary) by itself, one
  at a time, unless `instances.roll_on_upgrade` is `false` or every instance slot up to
  `instances.max` is taken. With systemd before 254 (EL8, EL9, Debian 12, Ubuntu 22.04)
  the restart stops the instances too and starts new ones on the new binary: a short
  interruption instead of the handover
  ([systemd before 254](https://scalws.com/docs/guide/installation/#systemd-before-254)).
* Where SELinux is enabled (RHEL family, Fedora, openSUSE Leap 16, SLES 16), a changed SELinux policy module is loaded and the package's files
  are relabelled before the services restart. Coming from an `.rpm` without the module,
  the controller's instances run unconfined, so the package restarts them in their confined
  domain (they drain for up to 30 seconds).
* **scalws-console.service**, if it is running, is restarted. Then the scripts run
  `systemctl enable --now scalws-console` on every install and upgrade, so the console is
  enabled and started even if you had disabled it; the output then says so
  (`scalws-console was disabled and is enabled again ...`) and that the console listens on
  port 8090. To keep it off, mask it: `sudo systemctl mask --now scalws-console`; a masked
  console is never enabled or mentioned by an upgrade. Console
  sessions end; history and audit log stay.
* A unit that was not running is not started (except the console, as above).
* The console's credentials file is created only if it does not exist.

A failed restart does not make the package installation fail, so check the services
afterwards:

```sh
systemctl status scalwsd scalws-controller scalws-console
journalctl -u scalwsd -n 50
```

## Configuration files are kept

`/etc/scalws/scalws.yaml` and `/etc/scalws/controller.yaml` are configuration files of the
package (`config|noreplace`):

* On Debian and Ubuntu they are dpkg conffiles. A file you changed is kept. If the package
  ships a changed version of a file you also changed, dpkg asks what to do; keeping your
  version is the default. A file you never changed is replaced by the new packaged version.
* On RPM systems (`%config(noreplace)`) a file you changed is kept and the new packaged
  version is written next to it as `<file>.rpmnew`. A file you never changed is replaced.
  When the package is removed, a file you changed is saved as `<file>.rpmsave`.

The package's scripts edit `scalws.yaml` only on a first install: they add `sandbox: on`
([sandbox.md](https://scalws.com/docs/guide/sandbox/)) and, on a host without IPv6, comment out the IPv6 listeners. An
upgrade or a reinstall never edits it, so an existing installation keeps the sandbox mode it
has. dpkg and rpm treat a file edited that way like one you changed.

Files the package does not ship are never touched: the site files in `/etc/scalws/sites.d/`,
the console credentials in `/etc/scalws/console/`, your certificates, and everything under
`/var/lib/scalws`, `/var/lib/scalws-controller` and `/var/lib/scalws-console`.

Unit files in `/usr/lib/systemd/system` are replaced. Keep your changes in drop-ins
(`sudo systemctl edit scalwsd`), which upgrades leave alone.

New defaults in the packaged configuration are not merged into a file you changed. Compare
your file with the commented example in `/usr/share/doc/scalws/examples/scalws.yaml` after
an upgrade, then validate it:

```sh
sudo scalwsctl check
```

## Site files on older installations

Installations that predate site files have no `include:` line, and `scalwsctl site add` and
`scalwsctl migrate` stop with:

```text
/etc/scalws/scalws.yaml has no `include:` for site files; add `include: sites.d/*.yaml` and create /etc/scalws/sites.d
```

The package creates `/etc/scalws/sites.d` (`root:scalws`, mode 0750). If it is missing,
create it:

```sh
sudo install -d -m 0750 -o root -g scalws /etc/scalws/sites.d
```

Then add the line at the top level of `/etc/scalws/scalws.yaml` (under scalws-controller:
the file named by [`controller.config`](https://scalws.com/docs/reference/controller/#controller.config), which
is the same file by default):

```yaml
include: sites.d/*.yaml
```

Check and reload:

```sh
sudo scalwsctl check
sudo scalwsctl reload
```

The rules for site files and the directory are in [Sites](https://scalws.com/docs/sites/) and
[`include`](https://scalws.com/docs/reference/top-level/#include).

## Roll the instances onto the new binary

Under scalws-controller, the restarted controller adopts the running instances, which still
run the previous `scalwsd` (their `/proc/<pid>/exe` is the deleted file that
[`controller.webd_binary`](https://scalws.com/docs/reference/controller/#controller.webd_binary) named). It
then rolls them onto the binary on disk by itself. It creates a new generation that is a
copy of the current one. The configuration file is not read again, so edits you have not
applied stay unapplied. The instances are replaced one at a time: a new instance becomes
active before an old one drains, so capacity does not drop.

The first new instance needs a free slot: the controller rolls only while fewer instances
run than `instances.max`. When every slot is taken (for example `min: 1, max: 1`, or two
instances with `max: 2`), it does not roll, because it would have to stop a serving
instance before the new binary has proved it can start. It logs a warning instead
(`not rolled: instances.max leaves no spare slot`). Raise `instances.max` by one, or roll
them yourself with `sudo scalws-controller reload` (which replaces an instance in place, so
capacity dips by one while it does).

`scalws-controller status` shows `"binary": "replaced"` for an instance that still runs the
old file and `"binary": "current"` for one that runs the new file. It also shows the phases
and the active generation:

```sh
sudo scalws-controller status
```

If the new binary cannot become ready, the controller goes back to the older generation
(see below) and the old instances keep serving from the old, deleted file. An old instance
that crashes meanwhile is replaced by an instance started from the new file.

To turn the automatic roll off, set `roll_on_upgrade: false` in the `instances` section of
`/etc/scalws/controller.yaml`. The controller then only logs a warning that names the
instances. Roll them yourself with a reload, which also retries a roll that went back:

```sh
sudo scalws-controller reload
sudo scalws-controller status
```

This reload reads the configuration file again, so it applies your pending edits too. Each
instance's version is the first line of its status view:

```sh
sudo scalwsctl --socket /run/scalws/web-0/admin.sock status
```

If an instance of the new generation fails to start while an older one still serves, the
controller returns to the older generation by itself and reports it as `last_rollback` in
`status`. Check the instance log in `/var/lib/scalws-controller/logs/web-<n>.log`.

`scalwsctl reload` does the same while the controller runs. See
[scalws-controller](https://scalws.com/docs/reference/programs/scalws-controller/#reload).

The standalone `scalwsd.service` is not affected by this: it still restarts with the short
gap described above.

## Rollback

### Going back to an earlier configuration

On `scalwsd.service`, `scalwsctl rollback` re-applies the configuration before the active
one, or `--generation G`. It uses the configurations the running process keeps in memory
(at most 10, since it started), so after an upgrade, which restarts the process, it cannot
reach configurations from before the upgrade. It also does not change the file on disk: the
next reload reads the file again. To return to the configuration of your backup:

```sh
sudo scalwsctl restore /var/backups/scalws-2026-10-05            # dry run: what would change
sudo scalwsctl restore /var/backups/scalws-2026-10-05 --apply
sudo scalwsctl reload
```

scalws-controller has no rollback command. Restore the files from a backup (or restore an
entry of the console's [history](https://scalws.com/docs/console/config-workflow/#history-and-rollback)) and run
`sudo scalws-controller reload`; this makes a new generation with the earlier content. The
generations it applied are kept in `/var/lib/scalws-controller/generations/<n>/scalws.yaml`,
readable by root, with the site files expanded into them, which helps to compare what
changed.

### Going back to an earlier package version

Install the earlier version again, from the repository if it still offers it, or from the
package file:

```sh
apt list -a scalws                        # versions the repository offers
sudo apt install scalws=<version>         # apt asks you to confirm the downgrade
```

With dnf:

```sh
dnf --showduplicates list scalws          # versions the repository offers
sudo dnf downgrade scalws-<version>
```

With zypper:

```sh
zypper search -s scalws                   # versions the repository offers
sudo zypper install --oldpackage scalws-<version>
```

The package scripts restart the running services as on an upgrade. A downgrade also
replaces the binary, so under scalws-controller the restarted controller rolls the
instances back onto the earlier binary by itself, when a slot is free as described above.
Otherwise, or with `roll_on_upgrade: false`, run `sudo scalws-controller reload` to put them
there.

Every section of the configuration rejects unknown keys, so a configuration that uses
settings added in the newer version is refused by the older one. Restore the backup you
took before the upgrade first, then downgrade.

## See also

* [Installation](https://scalws.com/docs/guide/installation/)
* [Files and services](https://scalws.com/docs/guide/files-and-services/)
* [scalws-controller](https://scalws.com/docs/reference/programs/scalws-controller/) and [scalwsctl](https://scalws.com/docs/reference/programs/scalwsctl/)