Users' guide

Logging and metrics

This page describes the server log and the access log of ScalWS, the endpoints of the metrics listener, and every metric family it exports.

Design record: ADR-0005; per-domain counters: ADR-0042; upstream telemetry: ADR-0043.

Logging

server:
  logging:
    format: json        # json (default) | text
    level: info         # default
    access_log: true    # default

The environment variable SCALWS_LOG overrides level when it is set (same syntax). An invalid filter stops the start with invalid log filter: ....

All logs go to standard output. The scalwsd unit writes them to the journal (journalctl -u scalwsd -f); under scalws-controller each instance’s output goes to its own file (see files-and-services.md).

Log targets worth knowing:

• scalws::access: the access log (level info). A level of warn or higher therefore also hides the access log; info,scalws::access=off keeps the server log and drops the access log.
• scalws::audit: one line per change made through the admin API ("admin request"), with the method, path, query, status, and the caller’s uid, gid and pid (admin socket) or client certificate (remote admin).

The access log line

One line is written per request that reached the request pipeline, when the response body has been sent completely or was dropped (for example because the client went away). Requests refused while the connection was set up or the request head was parsed (TLS failures, 400 or 431 from the HTTP parser, timeouts) have no access log line; they are counted in scalws_rejections_total.

Example with format: json (one line, wrapped here):

{"timestamp":"2026-10-12T09:15:02.481233Z","level":"INFO","request_id":"5f0c2a9be41d7730000004d2",
 "listener":"https","remote":"192.0.2.10","scheme":"https","version":"HTTP/2.0","method":"GET",
 "host":"shop.example.com","path":"/products/42","tenant":"shop","app":"storefront",
 "handler":"proxy","upstream_member":"192.0.2.5:8080","upstream_attempts":1,
 "user_agent":"Mozilla/5.0 (X11; Linux x86_64)","status":200,
 "bytes":5120,"complete":true,"duration_ms":12.7,
 "target":"scalws::access"}

With format: text the same fields follow the time, level and target as key=value pairs.

The query string is not logged. The path contains only characters valid in a URI, so it cannot break a log line.

The metrics listener

server:
  metrics:
    enabled: true              # default
    address: "127.0.0.1:9901"  # default; the packaged scalws.yaml uses 127.0.0.1:9900
    hosts: true                # per-domain counters, default
    max_hosts: 1000            # default

enabled and address need a restart; hosts and max_hosts apply on reload.

The listener has no authentication. Keep it on loopback: on another address the check warns metrics endpoint is not bound to loopback; it has no authentication. It answers GET only and serves a read-only set of paths; each path may also be written with a /v1 prefix.

server.health.address serves the same two endpoints (and /metrics with server.health.metrics: true) on an address of its own, for orchestrator probes; it serves nothing else. During limits.shutdown_delay, /readyz answers 503 with "state":"draining" while requests are still served.

Other admin views answer 403 with this view is only available on the admin socket (server.admin.socket), a POST answers 403 with changes are only accepted on the admin socket (server.admin.socket), other methods 405 (use GET or POST), unknown paths 404.

curl -s 127.0.0.1:9901/healthz
curl -s 127.0.0.1:9901/metrics | grep scalws_http_requests_total

Metric families

All names carry the prefix scalws. Counters end in _total; histograms are exposed with the usual _bucket, _sum and _count series. A labelled family appears in /metrics once it has at least one series. Series of tenants and applications that a reload removed are dropped.

Common labels:

• tenant, app: the tenant and application names; _unmatched for requests refused before routing.
• listener: the listener name.
• status (requests): the status class 1xx to 5xx.
• method: GET, HEAD, POST, PUT, DELETE, PATCH, OPTIONS, CONNECT, TRACE or OTHER.
• handler: static, proxy, respond, cache or none (as in the access log).
• protocol (upstream): http1 (reverse proxy and managed HTTP runtimes) or fastcgi (PHP).

Latency histograms (http_request_duration, http_response_transfer, tls_handshake) use buckets from 0.5 ms to 60 s; the upstream histograms from 50 µs to 30 s.

Requests and connections

Upstreams

Security and quotas

Response cache

See caching.md.

Tenant and application resources

These are sampled from cgroup v2 (ADR-0010) and eBPF (ADR-0025) when those are enabled (server.cgroups, server.ebpf).

TLS and ACME

Configuration, diagnostics and advisors

Per-domain counters

These counters are controlled by server.metrics.hosts. They are on by default; hosts: false turns them off completely (nothing is recorded or exported).

• host is the configured domain pattern, not the request’s host: all hosts matched by *.example.com count under host="*.example.com". The number of series is therefore bounded by the configuration, never by clients.
• Requests whose host matches no domain count under host="_other" (with tenant and app _unmatched), and so do domains beyond server.metrics.max_hosts (default 1000, at most 16383).
• Domains without requests are left out. A reload keeps the counters of every domain that stays.

See also

• request-processing.md: when each counter is incremented.
• troubleshooting.md: health checks.
• security.md: limits and rate limits behind the reason labels.