Help

Troubleshooting

Where to look when something is wrong, the built-in diagnostics, and common problems with their real error messages, causes and fixes.

When something is wrong, start here:

1. Check the configuration. Run sudo scalwsctl check. It finds most problems before they reach the server.
2. Read the logs. See Logs.
3. Look at what the server sees. Run scalwsctl status, apps and diagnose. See Built-in diagnostics.
4. Find the error. Look up the exact message under Common problems.

Every command used on this page is described in cli.md.

Contents

• Logs
• Built-in diagnostics
• Health and metrics endpoints
• Common problems
  • Configuration and reload
  • scalwsctl cannot reach the server
  • scalws-controller
  • Ports and listeners
  • Files and permissions
  • Certificates and ACME
  • HTTP errors clients see
  • PHP-FPM
  • Tenant limits and cgroups
  • SELinux
  • ScalWS Console

Logs

All ScalWS processes log to the systemd journal. The default format is one JSON object per line.

journalctl -u scalws-controller -f                    # follow the controller
journalctl -u scalwsd --since "10 min ago"            # single-process server
journalctl -u scalws-controller -u scalwsd -p warning # warnings and errors only

Instances under the controller. The controller writes each instance’s own output to <state_dir>/logs/web-<n>.log, which is /var/lib/scalws-controller/logs/web-0.log and so on by default. That file holds the request, application and ACME logs of the instance.

sudo tail -f /var/lib/scalws-controller/logs/web-0.log

When an instance fails to start, the controller also logs the last lines of that file with the failure.

Log targets. Each JSON line has a target field. These are the useful ones:

journalctl -u scalwsd -o cat | jq -c 'select(.target == "scalws::app")'
sudo grep '"scalws::acme"' /var/lib/scalws-controller/logs/web-0.log

Log level. Set server.logging.level (a tracing filter such as info or info,scalws_proxy=debug) and server.logging.format: json | text. The environment variable SCALWS_LOG overrides the configured filter. For a one-off debug session, add a systemd drop-in with Environment=SCALWS_LOG=debug. See configuration.md.

.htaccess. Lines that are not supported or are refused are skipped and logged with the file name:

journalctl -u scalwsd | grep htaccess

Built-in diagnostics

scalwsctl check

sudo scalwsctl check                      # /etc/scalws/scalws.yaml and its site files
sudo scalwsctl check /tmp/new.yaml        # a draft

This runs the same validation and preparation as a reload. Certificates are loaded, document roots and runtimes are checked, and site files are included. All problems are reported at once, each with its location:

/etc/scalws/scalws.yaml: 1 configuration error(s):
  - server.listeners[1].tls.certificates: a TLS listener needs at least one certificate (or acme: true)

Warnings start with warning:. For example, tenant limits with cgroups turned off give cpu/memory/pids/io limits are not enforced because server.cgroups.mode is off.

scalwsd --check does the same check from the daemon and prints configuration OK. The scalwsd.service unit runs it before every start. For controller.yaml itself, use scalws-controller check-config.

status, apps, tenants, top

scalwsctl status     # version, generation, uptime, listeners, cgroups/cache/optimizer on or off
scalwsctl apps       # runtime state, workers ready/target, in flight, p95, findings
scalwsctl tenants    # memory and CPU per tenant (admin socket only)
scalwsctl top        # apps, refreshed every 2 s

Check these fields in status:

• generation goes up with every accepted reload. If it did not change after your edit, the reload was rejected. Look in the log, or run scalwsctl reload to see the reason.
• cgroups off means that tenant limits and per-tenant usage are not available. See Tenant limits and cgroups.
• DRAINING means the server is shutting down.

In apps, STATE shows whether an application’s runtime is ready. FINDINGS lists the diagnostic codes, with ! for a warning and !! for critical.

diagnose

scalwsctl diagnose               # every application
scalwsctl diagnose shop/web      # one application, with its recent failed and slow requests

diagnose exits with 2 when a finding is critical, so you can use it in health checks. Diagnostics are computed per window. Right after a start you get no data yet (diagnostics appear after one window).

Each finding has a code, evidence, possibly related causes, and a suggestion:

The recent-requests list shows the status and timing of each request, and, for requests that failed at the upstream, the reason (upstream connect, upstream protocol, upstream timeout, upstream unavailable):

Recent failed or slow requests (2):
  1760000000 GET /slow -> 504 headers 60000 ms, total 60000 ms, upstream timeout
  1760000004 POST /api -> 502 headers 1 ms, total 1 ms, upstream connect

explain

scalwsctl explain shop/web --refresh

This asks the optional local AI advisor to explain an application’s diagnostics in plain language. It is off by default. Without it you get AI advisor disabled (server.ai.enabled). Treat the advice as a hint: it is labelled advisory only, verify before acting, and its suggested investigations are never executed by ScalWS.

recommendations

scalwsctl recommendations --last 50

Shows the optimizer’s view of each application (requests, 5xx %, p95, workers) and its recent decisions, such as scale_up applied workers 1 -> 2, each with the reason. Use it when worker counts change and you want to know why.

Health and metrics endpoints

The metrics listener is set by server.metrics.address. The package default is 127.0.0.1:9900. Under scalws-controller, each instance listens on 127.0.0.1:9910 + n by default (instances.metrics_base_port). The listener has no authentication. Keep it on loopback: the check warns metrics endpoint is not bound to loopback; it has no authentication.

curl -s 127.0.0.1:9900/healthz
curl -s 127.0.0.1:9900/readyz
curl -s 127.0.0.1:9900/metrics | grep scalws_http_requests_total

Common problems

Each entry lists the symptom (the exact message where there is one), the cause, and the fix.

Configuration and reload

<file>: N configuration error(s): followed by - <path>: <message> lines

• Cause: the file parsed, but values are invalid. <path> points at the key, for example tenants[0].applications[2].domains[1].
• Fix: correct each listed key, then run scalwsctl check again. The keys are documented in configuration.md and applications.md.

syntax error: ...

• Cause: the YAML is malformed, or it has an unknown key. Every section rejects unknown keys, so a typo is an error.
• Fix: check the spelling and indentation at the reported position.

domain "example.com" is already used by tenant/app, or from site add: example.com is already served by tenant/app

• Cause: a domain may be served only once. That includes another site’s alias and its www. name.
• Fix: remove the domain from one of the two, or edit the existing site. To see where it is defined, run scalwsctl site list.

scalwsctl: reload rejected: <reason>

• Cause: the new file failed validation or preparation. The server keeps serving the active configuration.
• Fix: run scalwsctl check for the full error list.

reload rejected: configuration change requires a restart: ...

• Cause: some changes cannot be applied by a reload on a single scalwsd: listeners, the metrics or admin endpoints, connection limits, cgroups, cache memory, per-IP limits, and the AI advisor.
• Fix: run systemctl restart scalwsd. See configuration.md.

<config> has no include:for site files; addinclude: sites.d/*.yaml and create /etc/scalws/sites.d

• Cause: scalwsctl site add or migrate needs an include: entry with a *. Installations from before site files existed do not have one.
• Fix: add include: sites.d/*.yaml to scalws.yaml, then run sudo install -d -m 0750 -o root -g scalws /etc/scalws/sites.d.

the new site does not pass the check, nothing was kept: or the server refused the reload, the site file was removed again:

• Cause: site add rolled itself back. The message after the colon gives the reason.
• Fix: fix that reason. Use --dry-run to see the file that site add would write.

scalwsctl cannot reach the server

scalwsctl: /run/scalws/admin.sock: Permission denied (os error 13)

• Cause: the admin socket is 0660 scalws:scalws.
• Fix: run the command with sudo, or add your account to the group with sudo usermod -aG scalws $USER, then log in again.

scalwsctl: /run/scalws/admin.sock: No such file or directory (os error 2)

• Cause: either scalwsd is not running, or the server runs under scalws-controller, where each instance has its own socket (/run/scalws/web-<n>/admin.sock).

• Fix: check with systemctl status scalws-controller scalwsd. Under the controller:

  • name an instance: scalwsctl --socket /run/scalws/web-0/admin.sock apps
  • or use its metrics port: scalwsctl --address 127.0.0.1:9910 status
  • reload with scalws-controller reload

  site, migrate, limit and high find the controller on their own.

this view is only available on the admin socket (server.admin.socket)

• Cause: you used --address, the metrics listener, for a view that exposes configuration content: tenants, certs, history or diff.
• Fix: use the admin socket (--socket) or --remote.

... (use the admin socket: --socket or server.admin.socket)

• Cause: you sent a change (reload, purge, restart, and so on) to the read-only metrics listener.
• Fix: use the admin socket.

scalwsctl: neither the admin socket nor the metrics listener is enabled

• Cause: the configuration disables both.
• Fix: enable server.admin or server.metrics, or pass --socket or --address explicitly.

scalwsctl: "web": use tenant/app

• Cause: restart, scale, diagnose and explain name an application as tenant/app.
• Fix: take the name from the first column of scalwsctl apps.

scalws-controller

tenant/app: managed php runtimes are not supported with several instances (ADR-0036); use an external endpoint (php: fastcgi, others: proxy)

The same message appears with node, python and container.

• Cause: every instance would start its own application pool. So under the controller, applications must be reached as external endpoints. The controller refuses such a generation at start and on scalws-controller reload.

• Fix: choose one:

  • Point PHP at an existing PHP-FPM pool: php: {fastcgi: "unix:/run/php/php8.3-fpm.sock"}, or in a site file php: unix:/run/php/php8.3-fpm.sock.
  • Run Node.js, Python or containers yourself (for example as systemd services) and use proxy: http://127.0.0.1:PORT.
  • Keep the managed runtimes and run the single-process server instead: systemctl disable --now scalws-controller && systemctl enable --now scalwsd. The two units exclude each other.

  scalwsctl site add --php|--node|--python refuses these runtimes up front: scalws-controller cannot run a managed PHP-FPM pool; use --php-fastcgi ....

HTTP/3 is not supported with server.listener_mode: reuseport (use the controller's default listen: inherited)

• Cause: HTTP/3 with several instances needs the controller’s QUIC demux. That works only with inherited listeners.
• Fix: remove listen: reuseport from controller.yaml, so the default inherited applies. See ../http3-multi-instance.md.

net.ipv4.tcp_migrate_req is 0: scale-in would reset queued connections; scale-in is refused until it is 1 (warning)

• Cause: you use listen: reuseport, and the sysctl is not set.
• Fix: run sysctl -w net.ipv4.tcp_migrate_req=1. The package installs it in /usr/lib/sysctl.d/90-scalws-adaptive-runtime.conf.

instances must be between N and M

• Cause: scalws-controller scale asked for a count outside instances.min..max.
• Fix: request a count within that range. scale 0 is never allowed. To take the service down, stop the unit.

instance not ready within the start timeout; killed, or rolling reload failed: an instance of the new generation did not start; rolled back

• Cause: a new instance did not become ready. The old generation keeps serving.
• Fix: read the end of /var/lib/scalws-controller/logs/web-<n>.log. The controller’s log line includes its last lines.

Ports and listeners

cannot bind listener https on 0.0.0.0:443: Address already in use (os error 98)

• Cause: another process holds the port. Usually it is nginx or Apache during a migration, or both scalwsd and scalws-controller were started by hand.
• Fix: find the process with sudo ss -ltnp 'sport = :443', then stop it. ScalWS’s two units declare Conflicts= with each other. Do not start a second copy outside systemd.

cannot bind ... : Permission denied (os error 13) on ports below 1024

• Cause: you started scalwsd by hand as an unprivileged user.
• Fix: use the systemd unit, which grants CAP_NET_BIND_SERVICE. Running as root is not needed, and the server logs a warning if you do.

duplicate listener address ..., address is also used by server.metrics, address is already used by a listener or the metrics endpoint (remote admin)

• Cause: two endpoints in the configuration use the same address.
• Fix: give each one its own port.

cannot bind metrics endpoint on 127.0.0.1:9900: Address already in use

• Cause: another program, or another scalwsd, uses the metrics port.
• Fix: change server.metrics.address.

Files and permissions

/etc/scalws/sites.d/example.com.yaml: must not be writable by group or others (chmod go-w), ...: must belong to root (or the owner of the main configuration file)

• Cause: anyone who can write a site file can serve any domain from any directory the server can read. So site files and the sites.d directory must be owned by root and not writable by group or others.
• Fix:

  sudo chown root:scalws /etc/scalws/sites.d /etc/scalws/sites.d/*.yaml
  sudo chmod 0750 /etc/scalws/sites.d && sudo chmod 0640 /etc/scalws/sites.d/*.yaml

document root /srv/www/site: No such file or directory, or document root is not a directory

• Cause: the path is wrong, or the systemd sandbox hides it. The units use ProtectHome=read-only: /home is visible, /root and /run/user are not.
• Fix: check the path, and keep document roots out of /root. Units from before 1.0.0~rc.1-11 hide /home too: upgrade the package.
• scalwsctl check and the Console report this as an error. A starting server does not fail for it, nor does a reload that leaves the application’s root and runtimes as they were (a root that became unopenable): only that application answers 503, the log has the error, and the next reload tries the root again. A reload that adds such an application, or changes one’s root or runtimes, is refused; under scalws-controller the new generation’s instances fail to start and the controller rolls it back (scalws-controller status, last_rollback).

403 or 404 for files that exist, or a managed application cannot write to its directory

• Cause: scalwsd serves static files as user scalws, so it may not be able to read them. Managed PHP-FPM, Node.js and Python processes run inside the unit’s sandbox, and the units set ReadOnlyPaths=/srv.
• Fix: make document roots readable by group scalws: chown -R web1:scalws /srv/web1 && chmod -R g+rX,o-rwx /srv/web1. For directories a managed application must write to, widen the sandbox with a unit drop-in, for example ReadWritePaths=/srv/web1/uploads. See ../OPERATIONS.md and security.md.

/etc/scalws/scalws.yaml: cannot read ...: Permission denied when you run scalwsctl check

• Cause: the configuration is 0640 root:scalws.
• Fix: use sudo, or a member of group scalws.

Certificates and ACME

Run scalwsctl certs first. It shows each ACME application’s state (ok, pending, error, missing), days left, and the last error with the time of the next retry. The scalws::acme log target has the details.

ACME HTTP-01 needs a plain HTTP listener (port 80 in production)

• Cause: certificates are validated with HTTP-01 challenges.
• Fix: add a listener without tls on port 80, and make sure port 80 is reachable from the internet: check the firewall, NAT, and that DNS points to this host.

ACME is enabled but no TLS listener has acme: true, applications use acme: true but ACME is disabled

• Fix: set tls: {acme: true} on the HTTPS listener, and server.acme.enabled: true.

must be true (the CA's terms of service are accepted on account creation)

• Fix: set server.acme.terms_of_service_agreed: true.

use mailto:address

• Fix: write the contact as server.acme.contact: ["mailto:[email protected]"].

wildcard domains need DNS-01, which is not supported; use a static certificate, ACME certificates need public DNS names

• Cause: only HTTP-01 is implemented. localhost and IP addresses cannot get a certificate this way.
• Fix: use a certificate file on the TLS listener for these names.

state stays error in scalwsctl certs

• Cause: usually the CA cannot reach http://<domain>/.well-known/acme-challenge/ on this server, the domain does not resolve here, or you hit the CA’s rate limits.
• Fix: correct DNS and port 80. While you test, use the Let’s Encrypt staging directory: server.acme.directory: https://acme-staging-v02.api.letsencrypt.org/directory. See tls.md.

HTTP errors clients see

421 Misdirected Request

• Cause: the listener has strict_sni: true, and the request’s Host differs from the name the client sent in TLS SNI. This often happens when HTTP/2 clients reuse one connection for several names that share a certificate. Browsers retry on a new connection automatically.
• Fix: if a non-browser client fails, make it send matching SNI and Host, or turn strict_sni off for that listener. strict_sni applies only to TLS listeners.

404 Not Found for a whole site

• Cause: no application serves the requested host.
• Fix: check scalwsctl site list, and whether the client sends the expected Host. The packaged default configuration answers every host that no site serves with the welcome page (the application welcome of the tenant default, domains: ["*"]); without it, such hosts get 404.

502 Bad Gateway

• Cause: the upstream (proxy target, PHP-FPM, Node.js or Python application) refused the connection or broke the protocol. The log shows upstream request failed (proxy) or fastcgi request failed (PHP), with reason connect or protocol and the OS error. diagnose reports upstream_unreachable.
• Fix: for connect, check that the application runs and listens where the configuration says. Managed Node.js and Python applications must bind $PORT. For an external upstream, test it with curl from the server. For protocol, the upstream sent something that is not valid HTTP or FastCGI. Check its own log.

504 Gateway Timeout

• Cause: the upstream did not send response headers within response_timeout (default 60s). The log shows upstream response timeout or php response timeout. diagnose reports upstream_timeouts, often together with saturated or cpu_pressure.
• Fix: find the slow paths with scalwsctl diagnose tenant/app. Then add workers, fix the application, or raise response_timeout on the runtime.

503 Service Unavailable with Retry-After: 1

• Cause: either no worker of the application is ready (it is starting, restarting or crashed: runtime_down or crash_loop), or the tenant’s max_concurrent_requests quota is reached (quota_rejections).
• Fix: run scalwsctl apps, then diagnose, then read the application’s log. If the load is legitimate, raise the quota.

429 Too Many Requests

• Cause: a rate limit applied: tenant requests_per_second, per-IP limits, or an application’s security.rate_limits.
• Fix: raise the limit if the traffic is legitimate. See security.md.

WebSocket connections fail through a proxy

• Cause: proxy runtimes tunnel WebSocket upgrades by default (websocket: true). Failure usually means websocket: false is set on that route, or the server.limits.max_upgraded_connections limit is reached.
• Fix: check both settings in configuration.md.

PHP-FPM

502 with fastcgi request failed and Permission denied for an external pool (php: {fastcgi: "unix:..."})

• Cause: the PHP-FPM pool’s socket does not let user scalws connect. Distribution pools usually make the socket owned by www-data.
• Fix: in the pool file (for example /etc/php/8.3/fpm/pool.d/www.conf), set:

  listen.owner = www-data
  listen.group = scalws
  listen.mode = 0660

  Then run systemctl restart php8.3-fpm. Alternatively, add scalws to the socket’s group.

502 with No such file or directory for the socket

• Cause: the socket path in the configuration does not match the pool’s listen =, or PHP-FPM is not running.
• Fix: compare the two paths and start PHP-FPM.

PHP pages download, or index.php is not found

• Fix: check root and index. For front-controller applications (WordPress, Laravel), set front_controller: /index.php, or use a profile such as profile: wordpress. See applications.md.

The managed pool does not start (runtime_down)

• Cause: with php: true, ScalWS starts PHP-FPM itself. By default it uses the first php-fpm found on PATH, else the newest versioned one (php-fpm8.4, php-fpm84).
• Fix: install PHP-FPM, or set fpm: {binary: /usr/sbin/php-fpm8.3}. The pool’s start-up errors are logged under target scalws::app.

Tenant limits and cgroups

tenant limits need cgroup v2, which is not active on this server (server.cgroups is off, or the startup warning "cgroup v2 enforcement unavailable" says why; see journalctl -u scalws-controller -u scalwsd)

high reports the same problem as per-tenant usage needs cgroup v2, ....

• Cause: limits and usage need cgroup v2 delegation.

• Fix:

  1. Check that server.cgroups.mode is not off.
  2. Check that the host uses cgroup v2: stat -fc %T /sys/fs/cgroup should print cgroup2fs. EL8 (AlmaLinux, Rocky Linux, Oracle Linux 8) uses cgroup v1 by default; switch it with sudo grubby --update-kernel=ALL --args=systemd.unified_cgroup_hierarchy=1 and a reboot (cgroup v1).
  3. Check that the server runs from the packaged unit, which sets Delegate=yes.
  4. Search the log for cgroup v2 enforcement unavailable. That line gives the reason.

  status shows cgroups on when enforcement works.

no tenant or tenant user named "ali", or user "ali" runs several tenants (...); name the tenant

• Fix: name the tenant as it appears in scalwsctl tenants.

invalid percentage "...", a memory percentage is at most 100, an I/O share is at most 10000%

• Fix: see the value formats under scalwsctl.

SELinux

On the RHEL family, Fedora, openSUSE Leap 16 and SLES 16, ScalWS runs confined by its policy module, which the package loads with semodule on installation (SELinux). On SUSE 15 no module is loaded. Start with the denials:

sudo ausearch -m avc -ts recent

403 or 404 for files that exist, with a denial for scalws_t on a document root

• Cause: the directory has no web content type (it was created outside /srv/www and /var/www, or moved there with mv, which keeps the old label).
• Fix: give it one: sudo semanage fcontext -a -t httpd_sys_content_t '/data/www(/.*)?' and sudo restorecon -R /data/www. For a directory moved into /srv/www, sudo restorecon -R /srv/www is enough. A directory an application writes to needs httpd_sys_rw_content_t.

502 from a reverse proxy, with a denial for scalws_t on name_connect

• Cause: the boolean scalws_can_network_connect is off, so only HTTP ports may be reached.
• Fix: sudo setsebool -P scalws_can_network_connect on.

ERROR: SELinux is enforcing but semodule is missing: ScalWS runs unconfined. during the installation

• Cause: policycoreutils is not installed, so the module could not be loaded.
• Fix: run the commands at the end of the message: sudo dnf install policycoreutils, then the semodule, restorecon and systemctl try-restart commands it names.

applications under SELinux need setpriv (util-linux) for their context scalws_app_t

• Cause: /usr/bin/setpriv is missing, so scalwsd cannot start applications in their domain.
• Fix: setpriv comes from util-linux, which the .rpm requires through /usr/bin/setpriv; install it again with sudo dnf install /usr/bin/setpriv.

Container applications (runtime: {type: container}) under SELinux enforcing

• Cause: runtime: {type: container} (podman) is not supported under SELinux enforcing in this release.
• Fix: run the container yourself (for example as its own systemd unit) and reach it through proxy.

ScalWS Console

The console shows controller unavailable, and its actions are disabled

• Cause: the console manages the server only through scalws-controller.
• Fix: run systemctl status scalws-controller. If you run scalwsd.service instead of the controller, the console’s management actions stay disabled.

sign-in is disabled: the credentials file is invalid (see the console log)

• Cause: the credentials file has a bad line, or it is readable by group or others.
• Fix: run sudo chmod 0600 /etc/scalws/console/credentials, check each line (user:role:password), then run journalctl -u scalws-console.

Lost the admin password

• Fix: the password is in the credentials file: sudo grep '^admin:' /etc/scalws/console/credentials | cut -d: -f3. To set a new one, replace the third field. scalws-console hash-password prints a hash you can use there. See ../console-operations.md.

Still stuck?

Collect this information before you ask for help:

• scalwsctl status, scalwsctl apps, scalwsctl diagnose
• sudo scalwsctl check
• scalws-controller status, if you use the controller
• the relevant log lines (journalctl -u scalws-controller -u scalwsd --since ..., and the instance logs)

Remove secrets from the configuration before you share it.