# Troubleshooting

URL: https://scalws.com/docs/guide/troubleshooting/
Source: `docs/guide/troubleshooting.md` in the ScalWS repository

Where to look when something is wrong, the built-in diagnostics, and common problems
with their real error messages, causes and fixes.

When something is wrong, start here:

1. **Check the configuration.** Run `sudo scalwsctl check`. It finds most problems before
   they reach the server.
2. **Read the logs.** See [Logs](#logs).
3. **Look at what the server sees.** Run `scalwsctl status`, `apps` and `diagnose`. See
   [Built-in diagnostics](#built-in-diagnostics).
4. **Find the error.** Look up the exact message under [Common problems](#common-problems).

Every command used on this page is described in [cli.md](https://scalws.com/docs/guide/cli/).

## Contents

* [Logs](#logs)
* [Built-in diagnostics](#built-in-diagnostics)
* [Health and metrics endpoints](#health-and-metrics-endpoints)
* [Common problems](#common-problems)
  * [Configuration and reload](#configuration-and-reload)
  * [scalwsctl cannot reach the server](#scalwsctl-cannot-reach-the-server)
  * [scalws-controller](#scalws-controller)
  * [Ports and listeners](#ports-and-listeners)
  * [Files and permissions](#files-and-permissions)
  * [Certificates and ACME](#certificates-and-acme)
  * [HTTP errors clients see](#http-errors-clients-see)
  * [PHP-FPM](#php-fpm)
  * [Tenant limits and cgroups](#tenant-limits-and-cgroups)
  * [SELinux](#selinux)
  * [ScalWS Console](#scalws-console)

---

## Logs

All ScalWS processes log to the systemd journal. The default format is one JSON object per
line.

| Unit | What it logs |
|---|---|
| `scalws-controller` | The controller: instance starts, activation, draining, scaling, rejected generations. |
| `scalwsd` | The single-process server, if you run `scalwsd.service` instead of the controller. |
| `scalws-console` | ScalWS Console. |

```sh
journalctl -u scalws-controller -f                    # follow the controller
journalctl -u scalwsd --since "10 min ago"            # single-process server
journalctl -u scalws-controller -u scalwsd -p warning # warnings and errors only
```

**Instances under the controller.** The controller writes each instance's own output to
`<state_dir>/logs/web-<n>.log`, which is `/var/lib/scalws-controller/logs/web-0.log` and so on by
default. That file holds the request, application and ACME logs of the instance.

```sh
sudo tail -f /var/lib/scalws-controller/logs/web-0.log
```

When an instance fails to start, the controller also logs the last lines of that file with
the failure.

**Log targets.** Each JSON line has a `target` field. These are the useful ones:

| Target | Content |
|---|---|
| `scalws::app` | Output of managed applications (PHP-FPM, Node.js, Python, containers). |
| `scalws::access` | Access log, one line per request. On by default (`server.logging.access_log`). |
| `scalws::audit` | Every change made through the admin API, with the caller's uid, gid and pid. Also the optimizer's actions. |
| `scalws::acme` | Certificate orders and renewals. |
| `scalws::diag` | Diagnostic findings. |

```sh
journalctl -u scalwsd -o cat | jq -c 'select(.target == "scalws::app")'
sudo grep '"scalws::acme"' /var/lib/scalws-controller/logs/web-0.log
```

**Log level.** Set `server.logging.level` (a `tracing` filter such as `info` or
`info,scalws_proxy=debug`) and `server.logging.format: json | text`. The environment
variable `SCALWS_LOG` overrides the configured filter. For a one-off debug session, add a
systemd drop-in with `Environment=SCALWS_LOG=debug`. See
[configuration.md](https://scalws.com/docs/guide/configuration/#logging).

**.htaccess.** Lines that are not supported or are refused are skipped and logged with the
file name:

```sh
journalctl -u scalwsd | grep htaccess
```

---

## Built-in diagnostics

### `scalwsctl check`

```sh
sudo scalwsctl check                      # /etc/scalws/scalws.yaml and its site files
sudo scalwsctl check /tmp/new.yaml        # a draft
```

This runs the same validation and preparation as a reload. Certificates are loaded,
document roots and runtimes are checked, and site files are included. All problems are
reported at once, each with its location:

```text
/etc/scalws/scalws.yaml: 1 configuration error(s):
  - server.listeners[1].tls.certificates: a TLS listener needs at least one certificate (or acme: true)
```

Warnings start with `warning:`. For example, tenant limits with cgroups turned off give
`cpu/memory/pids/io limits are not enforced because server.cgroups.mode is off`.

`scalwsd --check` does the same check from the daemon and prints `configuration OK`. The
`scalwsd.service` unit runs it before every start. For `controller.yaml` itself, use
`scalws-controller check-config`.

### `status`, `apps`, `tenants`, `top`

```sh
scalwsctl status     # version, generation, uptime, listeners, cgroups/cache/optimizer on or off
scalwsctl apps       # runtime state, workers ready/target, in flight, p95, findings
scalwsctl tenants    # memory and CPU per tenant (admin socket only)
scalwsctl top        # apps, refreshed every 2 s
```

Check these fields in `status`:

* **`generation`** goes up with every accepted reload. If it did not change after your
  edit, the reload was rejected. Look in the log, or run `scalwsctl reload` to see the
  reason.
* **`cgroups off`** means that tenant limits and per-tenant usage are not available. See
  [Tenant limits and cgroups](#tenant-limits-and-cgroups).
* **`DRAINING`** means the server is shutting down.

In `apps`, `STATE` shows whether an application's runtime is ready. `FINDINGS` lists the
diagnostic codes, with `!` for a warning and `!!` for critical.

### `diagnose`

```sh
scalwsctl diagnose               # every application
scalwsctl diagnose shop/web      # one application, with its recent failed and slow requests
```

`diagnose` exits with 2 when a finding is critical, so you can use it in health checks.
Diagnostics are computed per window. Right after a start you get
`no data yet (diagnostics appear after one window)`.

Each finding has a code, evidence, possibly related causes, and a suggestion:

| Code | Meaning | Suggestion it gives |
|---|---|---|
| `runtime_down` | No worker of the application is ready. | Check the application log for start-up errors (target `scalws::app`). |
| `crash_loop` | Workers restarted repeatedly in one window. | Check the application's log and its memory limit. |
| `oom_kill` | The tenant hit its memory limit. | Raise the tenant memory limit, or reduce workers or per-worker memory. |
| `memory_pressure` | Memory close to the limit, or memory stalls (PSI). | Raise the tenant memory limit or reduce workers. |
| `cpu_pressure` | The tenant's tasks wait for CPU. | Raise `resources.cpu` or reduce load. |
| `io_pressure` | The tenant's tasks wait for block I/O. | Check disk load; raise `io_weight` for this tenant. |
| `upstream_timeouts` | Requests timed out waiting for the application. | The application is too slow or stuck: raise workers or `response_timeout`. |
| `upstream_unreachable` | The application is not listening. | Check that it binds `PORT`. |
| `app_errors` | The application answered with 5xx. | These errors come from the application, not from ScalWS. |
| `saturated` | Workers are saturated. | Raise `workers.max` (or `target_concurrency`), or enable `autoscale: auto`. |
| `slow` | p95 time to response headers is above the slow threshold. | See the recent slow requests for the affected paths. |
| `quota_rejections` | Tenant concurrency or rate quotas rejected requests. | Raise `max_concurrent_requests` / `requests_per_second` if the load is legitimate. |

The recent-requests list shows the status and timing of each request, and, for requests
that failed at the upstream, the reason (`upstream connect`, `upstream protocol`,
`upstream timeout`, `upstream unavailable`):

```text
Recent failed or slow requests (2):
  1760000000 GET /slow -> 504 headers 60000 ms, total 60000 ms, upstream timeout
  1760000004 POST /api -> 502 headers 1 ms, total 1 ms, upstream connect
```

### `explain`

```sh
scalwsctl explain shop/web --refresh
```

This asks the optional local AI advisor to explain an application's diagnostics in plain
language. It is off by default. Without it you get `AI advisor disabled (server.ai.enabled)`. Treat the
advice as a hint: it is labelled `advisory only, verify before acting`, and its suggested
investigations are never executed by ScalWS.

### `recommendations`

```sh
scalwsctl recommendations --last 50
```

Shows the optimizer's view of each application (requests, 5xx %, p95, workers) and its
recent decisions, such as `scale_up applied workers 1 -> 2`, each with the reason. Use it when
worker counts change and you want to know why.

---

## Health and metrics endpoints

The metrics listener is set by `server.metrics.address`. The package default is
`127.0.0.1:9900`. Under scalws-controller, each instance listens on `127.0.0.1:9910 + n` by
default (`instances.metrics_base_port`). The listener has no authentication. Keep it on
loopback: the check warns `metrics endpoint is not bound to loopback; it has no authentication`.

| Path | Answer |
|---|---|
| `/healthz` | `200 ok`, or `503 draining` during shutdown. Liveness. |
| `/readyz` | JSON `{"state": ..., "generation": ..., "instance": ..., "pid": ...}`. `state` is `active` (accepting connections), `ready` (prepared, waiting for activation by the controller) or `draining` (status 503). |
| `/metrics` | Prometheus text format, for example `scalws_http_requests_total`, and per-domain counters. |
| `/dashboard` (also `/`) | A read-only HTML dashboard. |
| `/v1/status`, `/v1/apps`, `/v1/diagnose`, `/v1/optimizer`, `/v1/explain` | The JSON behind `scalwsctl status`, `apps`, `diagnose`, `recommendations` and `explain`. |

```sh
curl -s 127.0.0.1:9900/healthz
curl -s 127.0.0.1:9900/readyz
curl -s 127.0.0.1:9900/metrics | grep scalws_http_requests_total
```

---

## Common problems

Each entry lists the symptom (the exact message where there is one), the cause, and the fix.

### Configuration and reload

**`<file>: N configuration error(s):` followed by `- <path>: <message>` lines**

* **Cause:** the file parsed, but values are invalid. `<path>` points at the key, for
  example `tenants[0].applications[2].domains[1]`.
* **Fix:** correct each listed key, then run `scalwsctl check` again. The keys are
  documented in [configuration.md](https://scalws.com/docs/guide/configuration/) and
  [applications.md](https://scalws.com/docs/guide/applications/).

**`syntax error: ...`**

* **Cause:** the YAML is malformed, or it has an unknown key. Every section rejects
  unknown keys, so a typo is an error.
* **Fix:** check the spelling and indentation at the reported position.

**`domain "example.com" is already used by tenant/app`**, or from `site add`:
**`example.com is already served by tenant/app`**

* **Cause:** a domain may be served only once. That includes another site's alias and its
  `www.` name.
* **Fix:** remove the domain from one of the two, or edit the existing site. To see where it
  is defined, run `scalwsctl site list`.

**`scalwsctl: reload rejected: <reason>`**

* **Cause:** the new file failed validation or preparation. The server keeps serving the
  active configuration.
* **Fix:** run `scalwsctl check` for the full error list.

**`reload rejected: configuration change requires a restart: ...`**

* **Cause:** some changes cannot be applied by a reload on a single `scalwsd`: listeners,
  the metrics or admin endpoints, connection limits, cgroups, cache memory, per-IP limits,
  and the AI advisor.
* **Fix:** run `systemctl restart scalwsd`. See
  [configuration.md](https://scalws.com/docs/guide/configuration/#reload-or-restart).

**`<config> has no `include:` for site files; add `include: sites.d/*.yaml` and create /etc/scalws/sites.d`**

* **Cause:** `scalwsctl site add` or `migrate` needs an `include:` entry with a `*`.
  Installations from before site files existed do not have one.
* **Fix:** add `include: sites.d/*.yaml` to `scalws.yaml`, then run
  `sudo install -d -m 0750 -o root -g scalws /etc/scalws/sites.d`.

**`the new site does not pass the check, nothing was kept:`** or
**`the server refused the reload, the site file was removed again:`**

* **Cause:** `site add` rolled itself back. The message after the colon gives the reason.
* **Fix:** fix that reason. Use `--dry-run` to see the file that `site add` would write.

### scalwsctl cannot reach the server

**`scalwsctl: /run/scalws/admin.sock: Permission denied (os error 13)`**

* **Cause:** the admin socket is `0660 scalws:scalws`.
* **Fix:** run the command with `sudo`, or add your account to the group with
  `sudo usermod -aG scalws $USER`, then log in again.

**`scalwsctl: /run/scalws/admin.sock: No such file or directory (os error 2)`**

* **Cause:** either `scalwsd` is not running, or the server runs under scalws-controller,
  where each instance has its own socket (`/run/scalws/web-<n>/admin.sock`).
* **Fix:** check with `systemctl status scalws-controller scalwsd`. Under the controller:
  * name an instance: `scalwsctl --socket /run/scalws/web-0/admin.sock apps`
  * or use its metrics port: `scalwsctl --address 127.0.0.1:9910 status`
  * reload with `scalws-controller reload`

  `site`, `migrate`, `limit` and `high` find the controller on their own.

**`this view is only available on the admin socket (server.admin.socket)`**

* **Cause:** you used `--address`, the metrics listener, for a view that exposes
  configuration content: `tenants`, `certs`, `history` or `diff`.
* **Fix:** use the admin socket (`--socket`) or `--remote`.

**`... (use the admin socket: --socket or server.admin.socket)`**

* **Cause:** you sent a change (`reload`, `purge`, `restart`, and so on) to the read-only
  metrics listener.
* **Fix:** use the admin socket.

**`scalwsctl: neither the admin socket nor the metrics listener is enabled`**

* **Cause:** the configuration disables both.
* **Fix:** enable `server.admin` or `server.metrics`, or pass `--socket` or `--address`
  explicitly.

**`scalwsctl: "web": use tenant/app`**

* **Cause:** `restart`, `scale`, `diagnose` and `explain` name an application as
  `tenant/app`.
* **Fix:** take the name from the first column of `scalwsctl apps`.

### scalws-controller

**`tenant/app: managed php runtimes are not supported with several instances (ADR-0036); use an external endpoint (php: fastcgi, others: proxy)`**

The same message appears with `node`, `python` and `container`.

* **Cause:** every instance would start its own application pool. So under the controller,
  applications must be reached as external endpoints. The controller refuses such a
  generation at start and on `scalws-controller reload`.
* **Fix:** choose one:
  * Point PHP at an existing PHP-FPM pool: `php: {fastcgi: "unix:/run/php/php8.3-fpm.sock"}`,
    or in a site file `php: unix:/run/php/php8.3-fpm.sock`.
  * Run Node.js, Python or containers yourself (for example as systemd services) and use
    `proxy: http://127.0.0.1:PORT`.
  * Keep the managed runtimes and run the single-process server instead:
    `systemctl disable --now scalws-controller && systemctl enable --now scalwsd`. The two
    units exclude each other.

  `scalwsctl site add --php|--node|--python` refuses these runtimes up front:
  `scalws-controller cannot run a managed PHP-FPM pool; use --php-fastcgi ...`.

**`HTTP/3 is not supported with server.listener_mode: reuseport (use the controller's default listen: inherited)`**

* **Cause:** HTTP/3 with several instances needs the controller's QUIC demux. That works
  only with inherited listeners.
* **Fix:** remove `listen: reuseport` from `controller.yaml`, so the default `inherited`
  applies. See [../http3-multi-instance.md](https://scalws.com/docs/adaptive-runtime/http3-multi-instance/).

**`net.ipv4.tcp_migrate_req is 0: scale-in would reset queued connections; scale-in is refused until it is 1`** (warning)

* **Cause:** you use `listen: reuseport`, and the sysctl is not set.
* **Fix:** run `sysctl -w net.ipv4.tcp_migrate_req=1`. The package installs it in
  `/usr/lib/sysctl.d/90-scalws-adaptive-runtime.conf`.

**`instances must be between N and M`**

* **Cause:** `scalws-controller scale` asked for a count outside `instances.min..max`.
* **Fix:** request a count within that range. `scale 0` is never allowed. To take the
  service down, stop the unit.

**`instance not ready within the start timeout; killed`**, or
**`rolling reload failed: an instance of the new generation did not start; rolled back`**

* **Cause:** a new instance did not become ready. The old generation keeps serving.
* **Fix:** read the end of `/var/lib/scalws-controller/logs/web-<n>.log`. The
  controller's log line includes its last lines.

### Ports and listeners

**`cannot bind listener https on 0.0.0.0:443: Address already in use (os error 98)`**

* **Cause:** another process holds the port. Usually it is nginx or Apache during a
  migration, or both `scalwsd` and `scalws-controller` were started by hand.
* **Fix:** find the process with `sudo ss -ltnp 'sport = :443'`, then stop it. ScalWS's
  two units declare `Conflicts=` with each other. Do not start a second copy outside
  systemd.

**`cannot bind ... : Permission denied (os error 13)`** on ports below 1024

* **Cause:** you started `scalwsd` by hand as an unprivileged user.
* **Fix:** use the systemd unit, which grants `CAP_NET_BIND_SERVICE`. Running as root is not
  needed, and the server logs a warning if you do.

**`duplicate listener address ...`**, **`address is also used by server.metrics`**,
**`address is already used by a listener or the metrics endpoint`** (remote admin)

* **Cause:** two endpoints in the configuration use the same address.
* **Fix:** give each one its own port.

**`cannot bind metrics endpoint on 127.0.0.1:9900: Address already in use`**

* **Cause:** another program, or another `scalwsd`, uses the metrics port.
* **Fix:** change `server.metrics.address`.

### Files and permissions

**`/etc/scalws/sites.d/example.com.yaml: must not be writable by group or others (chmod go-w)`**,
**`...: must belong to root (or the owner of the main configuration file)`**

* **Cause:** anyone who can write a site file can serve any domain from any directory the
  server can read. So site files and the `sites.d` directory must be owned by root and not
  writable by group or others.
* **Fix:**
  ```sh
  sudo chown root:scalws /etc/scalws/sites.d /etc/scalws/sites.d/*.yaml
  sudo chmod 0750 /etc/scalws/sites.d && sudo chmod 0640 /etc/scalws/sites.d/*.yaml
  ```

**`document root /srv/www/site: No such file or directory`**, or **`document root is not a directory`**

* **Cause:** the path is wrong, or the systemd sandbox hides it. The units use
  `ProtectHome=read-only`: `/home` is visible, `/root` and `/run/user` are not.
* **Fix:** check the path, and keep document roots out of `/root`. Units from before
  1.0.0~rc.1-11 hide `/home` too: upgrade the package.
* `scalwsctl check` and the Console report this as an error. A starting server does not
  fail for it, nor does a reload that leaves the application's root and runtimes as they
  were (a root that became unopenable): only that application answers `503`, the log has
  the error, and the next reload tries the root again. A reload that adds such an
  application, or changes one's root or runtimes, is refused; under scalws-controller
  the new generation's instances fail to start and the controller rolls it back
  (`scalws-controller status`, `last_rollback`).

**403 or 404 for files that exist, or a managed application cannot write to its directory**

* **Cause:** `scalwsd` serves static files as user `scalws`, so it may not be able to read
  them. Managed PHP-FPM, Node.js and Python processes run inside the unit's sandbox, and
  the units set `ReadOnlyPaths=/srv`.
* **Fix:** make document roots readable by group `scalws`:
  `chown -R web1:scalws /srv/web1 && chmod -R g+rX,o-rwx /srv/web1`. For directories a
  managed application must write to, widen the sandbox with a unit drop-in, for example
  `ReadWritePaths=/srv/web1/uploads`. See [../OPERATIONS.md](https://scalws.com/docs/operations/#privileges)
  and [security.md](https://scalws.com/docs/guide/security/).

**`/etc/scalws/scalws.yaml: cannot read ...: Permission denied`** when you run `scalwsctl check`

* **Cause:** the configuration is `0640 root:scalws`.
* **Fix:** use `sudo`, or a member of group `scalws`.

### Certificates and ACME

Run `scalwsctl certs` first. It shows each ACME application's state (`ok`, `pending`,
`error`, `missing`), days left, and the last error with the time of the next retry. The
`scalws::acme` log target has the details.

**`ACME HTTP-01 needs a plain HTTP listener (port 80 in production)`**

* **Cause:** certificates are validated with HTTP-01 challenges.
* **Fix:** add a listener without `tls` on port 80, and make sure port 80 is reachable from
  the internet: check the firewall, NAT, and that DNS points to this host.

**`ACME is enabled but no TLS listener has acme: true`**,
**`applications use acme: true but ACME is disabled`**

* **Fix:** set `tls: {acme: true}` on the HTTPS listener, and `server.acme.enabled: true`.

**`must be true (the CA's terms of service are accepted on account creation)`**

* **Fix:** set `server.acme.terms_of_service_agreed: true`.

**`use mailto:address`**

* **Fix:** write the contact as `server.acme.contact: ["mailto:ops@example.com"]`.

**`wildcard domains need DNS-01, which is not supported; use a static certificate`**,
**`ACME certificates need public DNS names`**

* **Cause:** only HTTP-01 is implemented. `localhost` and IP addresses cannot get a
  certificate this way.
* **Fix:** use a certificate file on the TLS listener for these names.

**`state` stays `error` in `scalwsctl certs`**

* **Cause:** usually the CA cannot reach `http://<domain>/.well-known/acme-challenge/` on
  this server, the domain does not resolve here, or you hit the CA's rate limits.
* **Fix:** correct DNS and port 80. While you test, use the Let's Encrypt staging
  directory: `server.acme.directory: https://acme-staging-v02.api.letsencrypt.org/directory`.
  See [tls.md](https://scalws.com/docs/guide/tls/).

### HTTP errors clients see

**421 Misdirected Request**

* **Cause:** the listener has `strict_sni: true`, and the request's `Host` differs from the
  name the client sent in TLS SNI. This often happens when HTTP/2 clients reuse one
  connection for several names that share a certificate. Browsers retry on a new
  connection automatically.
* **Fix:** if a non-browser client fails, make it send matching SNI and `Host`, or turn
  `strict_sni` off for that listener. `strict_sni` applies only to TLS listeners.

**404 Not Found for a whole site**

* **Cause:** no application serves the requested host.
* **Fix:** check `scalwsctl site list`, and whether the client sends the expected `Host`.
  The packaged default configuration answers every host that no site serves with the
  welcome page (the application `welcome` of the tenant `default`, `domains: ["*"]`);
  without it, such hosts get 404.

**502 Bad Gateway**

* **Cause:** the upstream (proxy target, PHP-FPM, Node.js or Python application) refused
  the connection or broke the protocol. The log shows `upstream request failed` (proxy) or
  `fastcgi request failed` (PHP), with `reason` `connect` or `protocol` and the OS error.
  `diagnose` reports `upstream_unreachable`.
* **Fix:** for `connect`, check that the application runs and listens where the
  configuration says. Managed Node.js and Python applications must bind `$PORT`. For an
  external upstream, test it with `curl` from the server. For `protocol`, the upstream sent
  something that is not valid HTTP or FastCGI. Check its own log.

**504 Gateway Timeout**

* **Cause:** the upstream did not send response headers within `response_timeout`
  (default 60s). The log shows `upstream response timeout` or `php response timeout`.
  `diagnose` reports `upstream_timeouts`, often together with `saturated` or
  `cpu_pressure`.
* **Fix:** find the slow paths with `scalwsctl diagnose tenant/app`. Then add workers,
  fix the application, or raise `response_timeout` on the runtime.

**503 Service Unavailable with `Retry-After: 1`**

* **Cause:** either no worker of the application is ready (it is starting, restarting or
  crashed: `runtime_down` or `crash_loop`), or the tenant's `max_concurrent_requests`
  quota is reached (`quota_rejections`).
* **Fix:** run `scalwsctl apps`, then `diagnose`, then read the application's log. If the
  load is legitimate, raise the quota.

**429 Too Many Requests**

* **Cause:** a rate limit applied: tenant `requests_per_second`, per-IP limits, or an
  application's `security.rate_limits`.
* **Fix:** raise the limit if the traffic is legitimate. See [security.md](https://scalws.com/docs/guide/security/).

**WebSocket connections fail through a proxy**

* **Cause:** proxy runtimes tunnel WebSocket upgrades by default (`websocket: true`).
  Failure usually means `websocket: false` is set on that route, or the
  `server.limits.max_upgraded_connections` limit is reached.
* **Fix:** check both settings in [configuration.md](https://scalws.com/docs/guide/configuration/#limits).

### PHP-FPM

**502 with `fastcgi request failed` and `Permission denied` for an external pool (`php: {fastcgi: "unix:..."}`)**

* **Cause:** the PHP-FPM pool's socket does not let user `scalws` connect. Distribution pools
  usually make the socket owned by `www-data`.
* **Fix:** in the pool file (for example `/etc/php/8.3/fpm/pool.d/www.conf`), set:
  ```ini
  listen.owner = www-data
  listen.group = scalws
  listen.mode = 0660
  ```
  Then run `systemctl restart php8.3-fpm`. Alternatively, add `scalws` to the socket's group.

**502 with `No such file or directory` for the socket**

* **Cause:** the socket path in the configuration does not match the pool's `listen =`, or
  PHP-FPM is not running.
* **Fix:** compare the two paths and start PHP-FPM.

**PHP pages download, or `index.php` is not found**

* **Fix:** check `root` and `index`. For front-controller applications (WordPress,
  Laravel), set `front_controller: /index.php`, or use a profile such as
  `profile: wordpress`. See [applications.md](https://scalws.com/docs/guide/applications/).

**The managed pool does not start (`runtime_down`)**

* **Cause:** with `php: true`, ScalWS starts PHP-FPM itself. By default it uses the first
  `php-fpm` found on `PATH`, else the newest versioned one (`php-fpm8.4`, `php-fpm84`).
* **Fix:** install PHP-FPM, or set `fpm: {binary: /usr/sbin/php-fpm8.3}`. The pool's
  start-up errors are logged under target `scalws::app`.

### Tenant limits and cgroups

**`tenant limits need cgroup v2, which is not active on this server (server.cgroups is off, or the startup warning "cgroup v2 enforcement unavailable" says why; see journalctl -u scalws-controller -u scalwsd)`**

`high` reports the same problem as `per-tenant usage needs cgroup v2, ...`.

* **Cause:** limits and usage need cgroup v2 delegation.
* **Fix:**
  1. Check that `server.cgroups.mode` is not `off`.
  2. Check that the host uses cgroup v2: `stat -fc %T /sys/fs/cgroup` should print
     `cgroup2fs`. EL8 (AlmaLinux, Rocky Linux, Oracle Linux 8) uses cgroup v1 by default;
     switch it with `sudo grubby --update-kernel=ALL --args=systemd.unified_cgroup_hierarchy=1`
     and a reboot ([cgroup v1](https://scalws.com/docs/guide/installation/#cgroup-v1-el8)).
  3. Check that the server runs from the packaged unit, which sets `Delegate=yes`.
  4. Search the log for `cgroup v2 enforcement unavailable`. That line gives the reason.

  `status` shows `cgroups on` when enforcement works.

**`no tenant or tenant user named "ali"`**, or **`user "ali" runs several tenants (...); name the tenant`**

* **Fix:** name the tenant as it appears in `scalwsctl tenants`.

**`invalid percentage "..."`**, **`a memory percentage is at most 100`**, **`an I/O share is at most 10000%`**

* **Fix:** see the value formats under [scalwsctl](https://scalws.com/docs/reference/programs/scalwsctl/#tenant-resources-limit-and-high).

### SELinux

On the RHEL family, Fedora, openSUSE Leap 16 and SLES 16, ScalWS runs confined by its
policy module, which the package loads with `semodule` on installation
([SELinux](https://scalws.com/docs/guide/installation/#selinux)). On SUSE 15 no module is loaded. Start with the
denials:

```sh
sudo ausearch -m avc -ts recent
```

**403 or 404 for files that exist, with a denial for `scalws_t` on a document root**

* **Cause:** the directory has no web content type (it was created outside `/srv/www` and
  `/var/www`, or moved there with `mv`, which keeps the old label).
* **Fix:** give it one: `sudo semanage fcontext -a -t httpd_sys_content_t '/data/www(/.*)?'`
  and `sudo restorecon -R /data/www`. For a directory moved into `/srv/www`,
  `sudo restorecon -R /srv/www` is enough. A directory an application writes to needs
  `httpd_sys_rw_content_t`.

**`502` from a reverse proxy, with a denial for `scalws_t` on `name_connect`**

* **Cause:** the boolean `scalws_can_network_connect` is off, so only HTTP ports may be
  reached.
* **Fix:** `sudo setsebool -P scalws_can_network_connect on`.

**`ERROR: SELinux is enforcing but semodule is missing: ScalWS runs unconfined.`** during
the installation

* **Cause:** `policycoreutils` is not installed, so the module could not be loaded.
* **Fix:** run the commands at the end of the message: `sudo dnf install policycoreutils`,
  then the `semodule`, `restorecon` and `systemctl try-restart` commands it names.

**`applications under SELinux need setpriv (util-linux) for their context scalws_app_t`**

* **Cause:** `/usr/bin/setpriv` is missing, so `scalwsd` cannot start applications in
  their domain.
* **Fix:** `setpriv` comes from util-linux, which the `.rpm` requires through
  `/usr/bin/setpriv`; install it again with `sudo dnf install /usr/bin/setpriv`.

**Container applications (`runtime: {type: container}`) under SELinux enforcing**

* **Cause:** `runtime: {type: container}` (podman) is not supported under SELinux enforcing
  in this release.
* **Fix:** run the container yourself (for example as its own systemd unit) and reach it
  through `proxy`.

### ScalWS Console

**The console shows `controller unavailable`, and its actions are disabled**

* **Cause:** the console manages the server only through scalws-controller.
* **Fix:** run `systemctl status scalws-controller`. If you run `scalwsd.service` instead
  of the controller, the console's management actions stay disabled.

**`sign-in is disabled: the credentials file is invalid (see the console log)`**

* **Cause:** the credentials file has a bad line, or it is readable by group or others.
* **Fix:** run `sudo chmod 0600 /etc/scalws/console/credentials`, check each line
  (`user:role:password`), then run `journalctl -u scalws-console`.

**Lost the admin password**

* **Fix:** the password is in the credentials file:
  `sudo grep '^admin:' /etc/scalws/console/credentials | cut -d: -f3`. To set a new one,
  replace the third field. `scalws-console hash-password` prints a hash you can use there.
  See [../console-operations.md](https://scalws.com/docs/console/operations/).

---

## Still stuck?

Collect this information before you ask for help:

* `scalwsctl status`, `scalwsctl apps`, `scalwsctl diagnose`
* `sudo scalwsctl check`
* `scalws-controller status`, if you use the controller
* the relevant log lines (`journalctl -u scalws-controller -u scalwsd --since ...`, and
  the instance logs)

Remove secrets from the configuration before you share it.