# Logging and metrics

URL: https://scalws.com/docs/guide/logging-metrics/
Source: `docs/guide/logging-metrics.md` in the ScalWS repository

This page describes the server log and the access log of ScalWS, the endpoints of the
metrics listener, and every metric family it exports.

Design record: [ADR-0005](https://scalws.com/docs/adr/0005-observability/); per-domain counters:
[ADR-0042](https://scalws.com/docs/adr/0042-per-domain-counters/); upstream telemetry:
[ADR-0043](https://scalws.com/docs/adr/0043-upstream-telemetry/).

## Logging

```yaml
server:
  logging:
    format: json        # json (default) | text
    level: info         # default
    access_log: true    # default
```

| Key | Default | Reload | Meaning |
|---|---|---|---|
| [`format`](https://scalws.com/docs/reference/server-logging/#server.logging.format) | `json` | read when the process starts | `json`: one JSON object per line. `text`: human-readable lines. |
| [`level`](https://scalws.com/docs/reference/server-logging/#server.logging.level) | `info` | read when the process starts | A `tracing` filter directive: a level (`error`, `warn`, `info`, `debug`, `trace`), optionally with per-target levels such as `info,scalws::access=off`. |
| [`access_log`](https://scalws.com/docs/reference/server-logging/#server.logging.access_log) | `true` | applies on reload | One line per request. |

The environment variable `SCALWS_LOG` overrides `level` when it is set (same syntax). An
invalid filter stops the start with `invalid log filter: ...`.

All logs go to standard output. The `scalwsd` unit writes them to the journal
(`journalctl -u scalwsd -f`); under scalws-controller each instance's output goes to its
own file (see [files-and-services.md](https://scalws.com/docs/guide/files-and-services/#logs)).

Log targets worth knowing:

* `scalws::access`: the access log (level `info`). A `level` of `warn` or higher therefore
  also hides the access log; `info,scalws::access=off` keeps the server log and drops the
  access log.
* `scalws::audit`: one line per change made through the admin API (`"admin request"`),
  with the method, path, query, status, and the caller's uid, gid and pid (admin socket) or
  client certificate (remote admin).

### The access log line

One line is written per request that reached the request pipeline, when the response body
has been sent completely or was dropped (for example because the client went away). Requests
refused while the connection was set up or the request head was parsed (TLS failures, `400`
or `431` from the HTTP parser, timeouts) have no access log line; they are counted in
`scalws_rejections_total`.

| Field | Content |
|---|---|
| `request_id` | The request id, also sent as the `x-request-id` response header: 24 hexadecimal characters. |
| `listener` | Name of the listener (`server.listeners[].name`). |
| `remote` | Client address: the peer, or the address reported by a [trusted proxy](https://scalws.com/docs/guide/security/). |
| `scheme` | `http` or `https`. |
| `version` | `HTTP/1.0`, `HTTP/1.1`, `HTTP/2.0` or `HTTP/3.0`. |
| `method` | Request method. |
| `host` | Normalised host: the configured domain for an exact match, otherwise the request's host; `-` when the request was refused before the host was read. |
| `path` | The raw path as received (still percent-encoded, without the query). |
| `tenant`, `app` | The application that answered, or `_unmatched` for requests refused before routing or for unknown hosts. |
| `handler` | `static`, `proxy` (also PHP, Node.js, Python and containers), `respond` (also `redirect`), `cache` (answered from the response cache) or `none` (no handler ran). |
| `upstream_member` | The upstream pool member that answered (`host:port` or `unix:/path`); `-` when the request did not go through an upstream pool. |
| `upstream_attempts` | Members of the pool the request was sent to (more than 1 after retries); `0` without a pool. |
| `user_agent` | The request's `User-Agent` header, at most 256 bytes (bytes that are not UTF-8 are replaced); `-` without the header. |
| `status` | Response status code. |
| `bytes` | Response body bytes sent. |
| `complete` | `false` when the body was not sent completely. |
| `duration_ms` | Milliseconds from the start of the request to the end of the response body. |

Example with `format: json` (one line, wrapped here):

```json
{"timestamp":"2026-10-12T09:15:02.481233Z","level":"INFO","request_id":"5f0c2a9be41d7730000004d2",
 "listener":"https","remote":"192.0.2.10","scheme":"https","version":"HTTP/2.0","method":"GET",
 "host":"shop.example.com","path":"/products/42","tenant":"shop","app":"storefront",
 "handler":"proxy","upstream_member":"192.0.2.5:8080","upstream_attempts":1,
 "user_agent":"Mozilla/5.0 (X11; Linux x86_64)","status":200,
 "bytes":5120,"complete":true,"duration_ms":12.7,
 "target":"scalws::access"}
```

With `format: text` the same fields follow the time, level and target as `key=value` pairs.

The query string is not logged. The path contains only characters valid in a URI, so it
cannot break a log line.

## The metrics listener

```yaml
server:
  metrics:
    enabled: true              # default
    address: "127.0.0.1:9901"  # default; the packaged scalws.yaml uses 127.0.0.1:9900
    hosts: true                # per-domain counters, default
    max_hosts: 1000            # default
```

[`enabled`](https://scalws.com/docs/reference/server-metrics/#server.metrics.enabled) and
[`address`](https://scalws.com/docs/reference/server-metrics/#server.metrics.address) need a restart;
[`hosts`](https://scalws.com/docs/reference/server-metrics/#server.metrics.hosts) and
[`max_hosts`](https://scalws.com/docs/reference/server-metrics/#server.metrics.max_hosts) apply on reload.

The listener has no authentication. Keep it on loopback: on another address the check warns
`metrics endpoint is not bound to loopback; it has no authentication`. It answers `GET` only
and serves a read-only set of paths; each path may also be written with a `/v1` prefix.

| Path | Answer |
|---|---|
| `/metrics` | All metric families below, Prometheus text format (`text/plain; version=0.0.4`). |
| `/healthz` | `200 ok`; `503 draining` once shutdown has started. Liveness. |
| `/readyz` | JSON `{"state", "generation", "instance", "pid"}`. `state` is `active` (accepting connections), `ready` (prepared, waiting for activation by scalws-controller) or `draining` (with status `503`). |
| `/dashboard` and `/` | A read-only HTML status page that refreshes every 5 seconds from `/v1/status` and `/v1/apps`. |
| `/status`, `/apps`, `/diagnose`, `/optimizer`, `/explain` | The JSON views behind `scalwsctl status`, `apps`, `diagnose`, `recommendations` and `explain`. |

`server.health.address` serves the same two endpoints (and `/metrics` with
`server.health.metrics: true`) on an address of its own, for orchestrator probes; it serves
nothing else. During `limits.shutdown_delay`, `/readyz` answers 503 with
`"state":"draining"` while requests are still served.

Other admin views answer `403` with
`this view is only available on the admin socket (server.admin.socket)`, a `POST` answers
`403` with `changes are only accepted on the admin socket (server.admin.socket)`, other
methods `405` (`use GET or POST`), unknown paths `404`.

```sh
curl -s 127.0.0.1:9901/healthz
curl -s 127.0.0.1:9901/metrics | grep scalws_http_requests_total
```

## Metric families

All names carry the prefix `scalws`. Counters end in `_total`; histograms are exposed with the
usual `_bucket`, `_sum` and `_count` series. A labelled family appears in `/metrics` once it
has at least one series. Series of tenants and applications that a reload removed are
dropped.

Common labels:

* `tenant`, `app`: the tenant and application names; `_unmatched` for requests refused
  before routing.
* `listener`: the listener name.
* `status` (requests): the status class `1xx` to `5xx`.
* `method`: `GET`, `HEAD`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`, `CONNECT`, `TRACE`
  or `OTHER`.
* `handler`: `static`, `proxy`, `respond`, `cache` or `none` (as in the access log).
* `protocol` (upstream): `http1` (reverse proxy and managed HTTP runtimes) or `fastcgi`
  (PHP).

Latency histograms (`http_request_duration`, `http_response_transfer`, `tls_handshake`)
use buckets from 0.5 ms to 60 s; the upstream histograms from 50 µs to 30 s.

### Requests and connections

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_http_requests_total` | counter | `tenant`, `app`, `listener`, `method`, `status`, `handler` | HTTP requests handled. |
| `scalws_http_request_duration_seconds` | histogram | `tenant`, `app`, `handler` | Time from request head received to response head sent. |
| `scalws_http_response_transfer_seconds` | histogram | `tenant`, `app` | Time from the response head to the end of the response body. |
| `scalws_http_response_body_bytes_total` | counter | `tenant`, `app` | Response body bytes sent. |
| `scalws_http_requests_in_flight` | gauge | `tenant`, `app` | Requests being processed. |
| `scalws_connections_open` | gauge | `listener` | Open client connections. |
| `scalws_connections_accepted_total` | counter | `listener` | Accepted client connections. |
| `scalws_rejections_total` | counter | `listener`, `reason` | Connections or requests refused before routing. `reason`: `tls_handshake`, `tls_handshake_timeout`, `bad_request` (malformed or too large request head), `header_timeout`, `connection_limit`, `source_connection_limit`, `untrusted_proxy`, `proxy_protocol`. |
| `scalws_tls_handshake_seconds` | histogram | `listener` | Duration of successful TLS handshakes. |
| `scalws_h3_goaway_sent_total` | counter | `listener` | HTTP/3 GOAWAY frames sent while draining ([ADR-0041](https://scalws.com/docs/adr/0041-http3-graceful-lifecycle/)). |
| `scalws_h3_connection_closes_total` | counter | `listener`, `how` | HTTP/3 connections ended. `how`: `client` (closed by the client), `drained` (closed by the client after a GOAWAY), `forced` (still open when the drain grace period ended). |
| `scalws_event_loop_lag_seconds` | gauge | `event_loop` | Highest delay of a 100 ms timer on each event loop over the last second. Only with [`server.loop_lag_probe`](https://scalws.com/docs/reference/server/#server.loop_lag_probe). |

### Upstreams

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_upstream_failures_total` | counter | `tenant`, `app`, `reason` | Responses synthesised because an upstream failed (`502`, `503`, `504`). `reason`, for example `connect`, `timeout`, `protocol`, `unavailable`. |
| `scalws_upstream_connections_opened_total` | counter | `tenant`, `app`, `protocol` | New connections opened to upstreams. |
| `scalws_upstream_connections_reused_total` | counter | `tenant`, `app`, `protocol` | Upstream requests sent on a kept-alive connection. |
| `scalws_upstream_retries_total` | counter | `tenant`, `app`, `protocol` | Upstream requests sent again after a pooled connection turned out stale. |
| `scalws_upstream_cancelled_total` | counter | `tenant`, `app`, `protocol` | Upstream exchanges abandoned before the response was complete (client gone, response timeout). |
| `scalws_upstream_wait_seconds` | histogram | `tenant`, `app`, `protocol` | Time until an upstream request is written (pool checkout, connect, FastCGI queue). |
| `scalws_upstream_ttfb_seconds` | histogram | `tenant`, `app`, `protocol` | Time from writing an upstream request to its response head. |
| `scalws_upstream_requests_total` | counter | `pool`, `member`, `class` | Requests answered through upstream pools by member and status class (`2xx` to `5xx`, `error` when the member gave no response). Every attempt counts, retries included. |
| `scalws_upstream_request_duration_seconds` | histogram | `pool` | Time to the response head of pool requests, retries included. |
| `scalws_upstream_member_ejected` | gauge | `pool`, `member` | 1 while passive health keeps the member out. |
| `scalws_upstream_pool_retries_total` | counter | `pool` | Requests sent to another member of the pool ([retries](https://scalws.com/docs/guide/load-balancing/#retries)). Not to be confused with `scalws_upstream_retries_total` above, which counts requests sent again on a fresh connection to the same upstream. |
| `scalws_upstream_members` | gauge | `pool` | Members of the pool. Series of removed members and pools disappear; see [load-balancing.md](https://scalws.com/docs/guide/load-balancing/#observability). |
| `scalws_runtime_ready_workers` | gauge | `tenant`, `app` | Managed application workers that are ready. |
| `scalws_runtime_restarts_total` | counter | `tenant`, `app` | Managed application process restarts. |

### Security and quotas

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_security_blocked_total` | counter | `tenant`, `app`, `reason` | Requests refused by security rules. `reason`: `ip_rate` (`429`), `misdirected` (`421`), `denied_path` (`404`), `route_rate` (`429`), `route_access` (`403`, a route's `allow`/`deny`), `basic_auth` (`401`, missing or wrong credentials on a route with `basic_auth`), `basic_auth_busy` (`503`, 64 password checks already running). See [request-processing.md](https://scalws.com/docs/guide/request-processing/). |
| `scalws_tenant_limited_total` | counter | `tenant`, `reason` | Requests refused by tenant quotas. `reason`: `rate` (`429`) or `concurrency` (`503`). |

### Response cache

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_cache_lookups_total` | counter | `tenant`, `app`, `result` | Response cache lookups. `result`: `hit`, `stale`, `miss`, `bypass`. |
| `scalws_cache_entries` | gauge | none | Cached URLs (updated when `/metrics` is read). |
| `scalws_cache_bytes` | gauge | none | Memory used by cached responses. |
| `scalws_cache_disk_entries` | gauge | none | Responses held by the disk tier of the cache. |
| `scalws_cache_disk_bytes` | gauge | none | Bytes used by the disk tier of the cache. |

See [caching.md](https://scalws.com/docs/guide/caching/).

### Tenant and application resources

These are sampled from cgroup v2 ([ADR-0010](https://scalws.com/docs/adr/0010-cgroups-and-tenant-limits/)) and
eBPF ([ADR-0025](https://scalws.com/docs/adr/0025-ebpf-network-accounting/)) when those are enabled
([`server.cgroups`](https://scalws.com/docs/reference/server/#server.cgroups),
[`server.ebpf`](https://scalws.com/docs/reference/server/#server.ebpf)).

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_cgroups_enabled` | gauge | none | `1` when cgroup v2 limits are enforced. |
| `scalws_tenant_cpu_seconds` | gauge | `tenant` | CPU time used by the tenant's application processes. |
| `scalws_tenant_memory_bytes` | gauge | `tenant` | Memory used by the tenant's application processes. |
| `scalws_tenant_pids` | gauge | `tenant` | Processes and threads in the tenant's cgroup. |
| `scalws_tenant_oom_kills` | gauge | `tenant` | OOM kills in the tenant's cgroup since it was created. |
| `scalws_app_cpu_seconds` | gauge | `tenant`, `app` | CPU time used by the application's processes. |
| `scalws_app_memory_bytes` | gauge | `tenant`, `app` | Memory used by the application's processes. |
| `scalws_app_pids` | gauge | `tenant`, `app` | Processes and threads in the application's cgroup. |
| `scalws_app_network_bytes` | gauge | `tenant`, `app`, `direction` | Network bytes of the application's processes since start. `direction`: `ingress` or `egress`, from the application's point of view. |
| `scalws_app_network_packets` | gauge | `tenant`, `app`, `direction` | Network packets of the application's processes since start. |

### TLS and ACME

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_tls_certificate_expiry_seconds` | gauge | `listener`, `certificate` | Expiry (Unix time) of configured listener certificates; `certificate` is the first configured server name. |
| `scalws_acme_certificate_expiry_seconds` | gauge | `tenant`, `app` | Expiry (Unix time) of ACME-managed certificates. |
| `scalws_acme_orders_total` | counter | `result` | ACME certificate orders. `result`: `success` or `failure`. |

### Configuration, diagnostics and advisors

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_config_reloads_total` | counter | `result` | Configuration activation attempts. `result`: `success` or `failure`. |
| `scalws_config_generation` | gauge | none | Generation number of the active configuration. |
| `scalws_config_last_success_timestamp_seconds` | gauge | none | Unix time of the last successful configuration activation. |
| `scalws_diagnostic_findings_total` | counter | `tenant`, `app`, `code` | Diagnostic findings, counted when they appear ([ADR-0015](https://scalws.com/docs/adr/0015-diagnostics/)). |
| `scalws_optimizer_decisions_total` | counter | `tenant`, `app`, `rule`, `disposition` | Optimizer decisions ([ADR-0014](https://scalws.com/docs/adr/0014-deterministic-optimizer/)). `disposition`: `recommended`, `applied`, `rolled_back`, `kept`, `advisory`. |
| `scalws_ai_requests_total` | counter | `result` | Local AI advisor requests by result ([ADR-0016](https://scalws.com/docs/adr/0016-local-ai-advisor/)). |
| `scalws_ai_queue_depth` | gauge | none | Pending local AI advisor requests. |
| `scalws_build_info` | gauge | `version` | Always `1`; `version` is the ScalWS version. |

### Per-domain counters

| Name | Type | Labels | Meaning |
|---|---|---|---|
| `scalws_http_host_requests_total` | counter | `tenant`, `app`, `host`, `status` | HTTP requests by configured domain and status class. |
| `scalws_http_host_response_bytes_total` | counter | `tenant`, `app`, `host` | Response body bytes by configured domain. |
| `scalws_http_host_request_bytes_total` | counter | `tenant`, `app`, `host` | Request body bytes read by configured domain. |
| `scalws_http_host_cache_total` | counter | `tenant`, `app`, `host`, `result` | Response cache lookups by configured domain. `result`: `hit` (stale answers included) or `miss`. |
| `scalws_http_host_requests_in_flight` | gauge | `tenant`, `app`, `host` | Requests being handled by configured domain. |

These counters are controlled by
[`server.metrics.hosts`](https://scalws.com/docs/reference/server-metrics/#server.metrics.hosts). They are on
by default; `hosts: false` turns them off completely (nothing is recorded or exported).

* `host` is the configured domain pattern, not the request's host: all hosts matched by
  `*.example.com` count under `host="*.example.com"`. The number of series is therefore
  bounded by the configuration, never by clients.
* Requests whose host matches no domain count under `host="_other"` (with tenant and app
  `_unmatched`), and so do domains beyond
  [`server.metrics.max_hosts`](https://scalws.com/docs/reference/server-metrics/#server.metrics.max_hosts)
  (default 1000, at most 16383).
* Domains without requests are left out. A reload keeps the counters of every domain that
  stays.

## See also

* [request-processing.md](https://scalws.com/docs/guide/request-processing/): when each counter is incremented.
* [troubleshooting.md](https://scalws.com/docs/guide/troubleshooting/#health-and-metrics-endpoints): health checks.
* [security.md](https://scalws.com/docs/guide/security/): limits and rate limits behind the `reason` labels.