# Ingress controller

URL: https://scalws.com/docs/guide/ingress-controller/
Source: `docs/guide/ingress-controller.md` in the ScalWS repository

`scalws-ingress` turns Kubernetes `networking.k8s.io/v1` Ingress resources and the common
`nginx.ingress.kubernetes.io/*` annotations into ScalWS configuration, so ScalWS can take
the place of ingress-nginx. It runs as a second container next to `scalwsd` in every
ingress pod. This guide covers the install, how the controller works, TLS, the supported
annotations with every difference from ingress-nginx, what is refused on purpose, the
Events and metrics, the flags, RBAC and a migration checklist.

Design record: [ADR-0051](https://scalws.com/docs/adr/0051-ingress-controller/). The data plane (pools, route
matching and policies) is described in [load-balancing.md](https://scalws.com/docs/guide/load-balancing/) and
[configuration.md](https://scalws.com/docs/guide/configuration/#route-matching). General Kubernetes notes are in
[kubernetes.md](https://scalws.com/docs/guide/kubernetes/). The same controller also serves the Gateway API (Gateway,
HTTPRoute), alone or next to Ingresses: [gateway-api.md](https://scalws.com/docs/guide/gateway-api/).

> **Note:** the container image `ghcr.io/xsmtx/scalws` and the Helm chart
> `oci://ghcr.io/xsmtx/charts/scalws` are not yet published for public download. Until they
> are, the steps on this page describe how the Kubernetes deployment works but cannot be
> followed as written; the Linux packages are the supported way to install 1.0.0.

## Install

The controller is a mode of the Helm chart, in chart versions after 1.0.0-rc.1 (the
1.0.0-rc.1 chart has no ingress controller mode):

```sh
helm install ingress oci://ghcr.io/xsmtx/charts/scalws --version <chart version> \
    --set ingressController.enabled=true
```

This creates a Deployment (or a DaemonSet with `ingressController.kind=DaemonSet`) with the
two containers, a `LoadBalancer` Service on ports 80 and 443, the IngressClass `scalws`, a
ServiceAccount and the RBAC the controller needs. The values:

| Value | Default | Meaning |
|---|---|---|
| `ingressController.enabled` | `false` | Turn the mode on. |
| `ingressController.ingress.enabled` | `true` | Serve Ingress resources (`--ingress`); `false` for a Gateway-only install, which also leaves out the IngressClass. |
| `ingressController.gatewayAPI.*` | off | The Gateway API input, see [gateway-api.md](https://scalws.com/docs/guide/gateway-api/#chart-values). |
| `ingressController.kind` | `Deployment` | `Deployment` or `DaemonSet`. Autoscaling needs `Deployment`. |
| `ingressController.className` | `scalws` | Name of the IngressClass. |
| `ingressController.controllerValue` | `scalws.com/ingress-controller` | `spec.controller` of the IngressClass; the controller serves classes with this value. |
| `ingressController.default` | `false` | Mark the IngressClass as the cluster default. |
| `ingressController.watchNamespace` | empty | Watch one namespace instead of the cluster. |
| `ingressController.watchIngressWithoutClass` | `false` | Also serve Ingresses that name no class. |
| `ingressController.crossNamespacePaths` | `false` | Let Ingresses of every namespace add paths to a host, as ingress-nginx (see Which Ingress wins); applies to HTTPRoutes too. |
| `ingressController.defaultCertificate` | empty | `namespace/name` of a TLS Secret for hosts without their own certificate. |
| `ingressController.publishService` | the release's Service | `namespace/name` of the Service whose address goes into the Ingress status. |
| `ingressController.http3` | `false` | Publish 443/UDP. Needs a custom image, see below. |
| `ingressController.service.type`, `.annotations` | `LoadBalancer`, none | The Service in front of the pods. |
| `ingressController.resources` | none | Resources of the controller container (`resources` is for `scalwsd`). |

Some chart values are refused with an error in this mode because they have no meaning
there: `sites`, `existingConfigMap`, `php.enabled`, `tls.enabled`, `content.image` and
`content.existingClaim`. The sites come from the Ingress resources.

`http3: true` needs an image whose `scalwsd` is built with `--features http3`. The
published image has no HTTP/3, and `scalwsd` refuses to start with the setting.

An Ingress is served when `spec.ingressClassName` names an IngressClass whose
`spec.controller` is the controller value. An Ingress without a class is served when the
IngressClass is marked default
(`ingressclass.kubernetes.io/is-default-class: "true"`), or when
`watchIngressWithoutClass` is set. The deprecated `kubernetes.io/ingress.class` annotation
is honoured with the same rule. To serve an Ingress, set `ingressClassName: scalws`:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 8m
spec:
  ingressClassName: scalws
  tls:
    - hosts: [www.example.com]
      secretName: www-example-com
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: {name: web, port: {number: 80}}
```

### Check the install

With the release named `ingress` the objects are called `ingress-scalws`:

```sh
kubectl get pods -l app.kubernetes.io/instance=ingress    # 2/2 Ready: scalws and controller
kubectl get ingressclass scalws
kubectl get service ingress-scalws                        # EXTERNAL-IP of the LoadBalancer
kubectl get ingress web                                   # ADDRESS is set by the leader pod
curl -H 'Host: www.example.com' http://<EXTERNAL-IP>/
```

A pod becomes ready only after the controller has loaded its first configuration, so a
pod that stays at 1/2 points at the controller: `kubectl logs deploy/ingress-scalws -c
controller` (RBAC errors with `watchNamespace`, an API server it cannot reach). A host
that answers `404` is not served yet; see [Troubleshooting](#troubleshooting).

### Resources and performance

`scalwsd` (the chart's `resources`) requests `cpu: 100m` and `memory: 128Mi` and has no
memory limit by default, like the ingress-nginx and Traefik charts. Its memory grows with
the requests in flight and the size of the proxied responses: about 330 MiB for 256
concurrent 100 KiB responses. If you set a limit, leave that headroom, or `scalwsd` is
OOM-killed under load. `ingressController.resources` applies to the controller container
only, which needs little.

Settings of `scalwsd` go under `server:` in the values, as outside ingress mode. Keep
`server.accept_policy` at its default, `shared`: the opt-in `fountain` policy saves CPU at
moderate load but places keep-alive connections on fewer event loops, which costs a proxy
latency (1.5x to 1.6x the p99 at a fixed rate in the benchmark, with the connection cap
of 2026-10-15). `server.event_loops`
defaults to one loop per CPU the pod may use. The measured comparison with ingress-nginx
and Traefik is in [the Kubernetes ingress benchmark](https://scalws.com/docs/benchmarks/2026-10-14-xeon-k8s-ingress/).

## How it works

```text
 API server --watch--> scalws-ingress --scalws.yaml, certificates, htpasswd--> emptyDir
                            |                                                    |
                            +--POST /v1/reload---------------------------------> scalwsd
                            +--PUT /v1/upstreams/{pool}--------------------------> (admin socket)
```

The two containers share the admin socket (`/run/scalws`) and a memory-backed volume
(`/etc/scalws/generated`) that holds the generated configuration, certificates, keys and
htpasswd files. Keys never touch a disk. Both containers run as user 65532 with a read-only
root file system, no capabilities and the `RuntimeDefault` seccomp profile; the admin
socket is reachable only inside the pod. Secret contents are never logged.

**Structure channel.** The controller watches IngressClass, Ingress, Service,
EndpointSlice and Secret resources (cluster-wide, or one namespace with
`--watch-namespace`). A change marks the state dirty; after a quiet time of 1 s it
translates the whole state, validates it with the same validation `scalwsd` runs, and when
the rendered `scalws.yaml` differs from the last one it writes it atomically and calls
`POST /v1/reload`. The call answers synchronously, so a refused configuration is known
immediately: the previous configuration stays active, the controller writes the last
configuration that `scalwsd` accepted back to `scalws.yaml` (so a restarted `scalwsd` never
starts from a file it refused), `scalws_ingress_reload_failures_total` rises and the reload
is retried on the next change or after 60 s. Certificate, key, CA
and htpasswd files are content-addressed, written with mode 0600 before the configuration
that references them, and removed only after a reload no longer references them.
`server.reload.watch` stays off in this mode.

**Member channel.** Every pool the controller generates is `members_from: api`. When the
ready addresses of a pool change, the controller sends `PUT /v1/upstreams/{pool}` with the
full member list; no reload happens, so scaling a backend does not touch the
configuration. Every 30 s it sends every pool again, which repairs a missed update and a
restarted `scalwsd` container. The rendered configuration also carries each pool's current
members as its `members` seed, so a pool created by a reload, and a `scalwsd` that
restarts, start with members instead of answering 503. Member-only changes never rewrite
the file; the seed is refreshed with the next structural write.

**Start-up.** An init container (`scalws-ingress bootstrap`) writes a first configuration
with the listeners, the health listener, the admin socket and no applications, so every
request gets 404. The controller waits for its caches, writes the first real
configuration, reloads, pushes the members and only then reports ready on `/readyz`. The
pod's readiness probe points at the controller, so a pod receives traffic only with a real
configuration.

**Status and leadership.** The holder of a Lease (named `<fullname>-leader` in the chart,
`scalws-ingress-leader` by default, in the controller's namespace; 15 s duration) writes
`status.loadBalancer` of the Ingresses it serves, with the address of the publish Service.
Serving traffic needs no leadership: every pod renders its own configuration. The Lease is
released at shutdown.

### Generated names

One tenant, `ingress`, holds one application per host, named `h_` plus the host with `.`
turned into `_` and `*` removed (`www.example.com` gives `h_www_example_com`,
`*.example.com` gives `h__example_com`). Rules without a host and `spec.defaultBackend`
go to the application `default`, which has the catch-all domain `"*"` and answers hosts no
other application serves. Pools are named `p_<namespace>_<service>_<port>`, plus `_<6
hex>` of the pool settings when annotations change balance, hash or scheme, so two
Ingresses that use one Service with different affinity get different pools. Names over 63
characters are cut and get 8 hex digits of a hash of the full name. Kubernetes names never
contain `_`, so generated names do not collide by accident; the controller still records
the owner of every generated name and refuses a second owner with a `PathConflict` Event.

### Which Ingress wins

A host belongs to the namespace of its oldest Ingress with a rule for it, served or not
(Ingresses of another class do not count; the rules without a host count as one host, the
catch-all). Several Ingresses of that namespace may define the host: their paths are
merged into one application, as in ingress-nginx. A rule of an Ingress of another
namespace for the host is not served: that Ingress gets a `PathConflict` Event saying
which namespace the host belongs to, and its rules for other hosts are still served. So
another namespace can neither add a path to a host nor catch, with a broad prefix or a
regex, the paths the owner does not serve (or no longer serves because its Ingress was
refused). ingress-nginx merges the Ingresses of every namespace;
`ingressController.crossNamespacePaths` (`--cross-namespace-paths`) restores that, see
below.

When two Ingresses define the same host, path type, path and matchers, the oldest
(`creationTimestamp`, then namespace/name) wins and the other gets a `PathConflict` Event.
The same rule picks the one whose `spec.defaultBackend` is used.

Only Ingresses of the namespace the host belongs to may set the host's certificate and
make it a TLS host (with the HTTPS redirect); among them the oldest with a `spec.tls`
entry for the host wins. A `spec.tls` entry for the host from another namespace changes
nothing and gets a `TLSSecret` Event naming the owning namespace, so a newer Ingress
elsewhere cannot take over the certificate or the HTTPS behaviour of a host.

An Ingress that is not served (refused access annotations, an invalid annotation, a
missing basic-auth Secret, or left out because it broke the configuration) keeps its
paths and its TLS hosts: a newer Ingress with the same path gets a `PathConflict` Event
and the path answers `404`, instead of sending the users of the older Ingress to another
backend. Paths it does not have may still be served by other Ingresses of its namespace.

With `crossNamespacePaths: true` Ingresses of every namespace merge their paths into the
host, as in ingress-nginx; the oldest still wins a path, and the certificate, TLS and
regex rules of the owning namespace stay as described here. An unserved Ingress then also
keeps its paths against newer Ingresses of other namespaces. Turn it on only when every
namespace that can create Ingresses of this class is trusted with every host.

A host under an older wildcard Ingress of another namespace is not served either: when
namespace `a` has the oldest Ingress for `*.example.com`, a newer Ingress of namespace `b`
for `shop.example.com` gets a `PathConflict` Event naming namespace `a`, because the
wildcard owner would otherwise lose a host it already serves. This is a change from the
first version of the controller, which compared host patterns only as written; Ingresses
of the wildcard's own namespace are not affected, and `crossNamespacePaths: true` restores
the old behaviour among Ingresses. A `spec.defaultBackend` counts as a claim on the
catch-all, like the rules without a host.

With the Gateway API input on, the same ownership rule covers Gateways and HTTPRoutes: on
each listener the oldest claim wins, whatever its input, and an Ingress and an HTTPRoute
never serve one host together (not even with `crossNamespacePaths`, which lifts only the
namespace rule among Ingresses). A newer Ingress for a host an HTTPRoute serves gets
`PathConflict`; a newer HTTPRoute for an Ingress host gets `HostnameConflict`. See
[gateway-api.md](https://scalws.com/docs/guide/gateway-api/#ownership-across-ingress-and-gateway-api), which also
describes moving a host from an Ingress to an HTTPRoute.

### Paths

| Ingress `pathType` | Route |
|---|---|
| `Exact` | `exact` |
| `Prefix` | `prefix` on whole segments (the Kubernetes definition) |
| `ImplementationSpecific` | `prefix` on whole segments; ingress-nginx treats the value as a plain prefix of the string |

With `use-regex: "true"` or a `rewrite-target`, `Prefix` and `ImplementationSpecific` paths
become regular expressions, as in ingress-nginx. They get `(?i)` prepended, because
ingress-nginx regex locations are case-insensitive. The expressions are linear-time, with
no look-around or back-references (ADR-0050).

Regex mode covers the whole host, as in ingress-nginx, but only the namespace the host
belongs to (see Which Ingress wins) can switch it on: when a served Ingress of that
namespace with a rule for the host asks for regex paths, every `Prefix` and
`ImplementationSpecific` path of that host, from all its Ingresses and namespaces, becomes
a regex. Paths of Ingresses that do not ask for regex paths themselves are matched
literally (their regex characters are escaped), so they keep their meaning and another
Ingress cannot make them invalid. `Exact` paths stay exact. The regex routes of such a host
are tried in two groups, first those of the owning namespace, then those of every other
namespace, each group longest path (as written in the Ingress) first, so another namespace
can never shadow a path of the owner. So a frontend Ingress with `Prefix /` and an API
Ingress of the same namespace with `rewrite-target: /$2` and path `/api(/|$)(.*)` work
together: `/api/x` reaches the API as `/x`, everything else the frontend.

A regex Ingress of another namespace does not switch the host: its own paths are regexes,
which the router tries after the exact and prefix routes, so it cannot take the paths of
the owner. Hosts without a regex Ingress of the owning namespace keep plain prefix routes.
Precedence inside a host is exact, then longest prefix, then regex in that order.

### Backends

A backend `service.name` and `service.port` (number or name) becomes a pool. Its members
are the ready addresses (IPv4 and IPv6, `serving` and not `terminating`) of the Service's
EndpointSlices for the matching port: by name for a named port, by the Service's
`targetPort` otherwise. With `service-upstream: "true"` the pool has one member, the
ClusterIP and port. An `ExternalName` Service gives one member, `externalName:port`. A
missing Service, port or EndpointSlice leaves the pool empty: the route answers 503, as
with ingress-nginx, and the Ingress gets a `ServiceNotFound` or `NoEndpoints` Event.

A request makes up to three tries on different members (the pool's `retries: 2`), the
ingress-nginx default of `proxy-next-upstream-tries`. A request is tried again only while
nothing has been sent to the client: after a failed connect, or, for an idempotent method
(`GET`, `HEAD`, `OPTIONS`, `PUT`, `DELETE`), after a `502`, `503` or `504`, a timeout or a
connection broken before the response head (a pod that is shutting down). A `POST` that
reached a member is never sent again. The number of tries is not configurable per
Ingress.

## TLS

For every `spec.tls` entry the controller loads the Secret of type `kubernetes.io/tls` from
the Ingress's namespace and serves the entry's hosts with it. The Secret is checked with the
same code `scalwsd` loads certificates with: a missing Secret, a wrong type, a `tls.crt`
with any PEM block that does not parse, a key that does not match the certificate, or a
certificate whose names do not cover a host leaves that host on the default certificate
and gives a `TLSSecret` Event. A TLS
host only counts when the same Ingress has a rule for it (the same as ingress-nginx).

Hosts without a certificate of their own get the default certificate:
`ingressController.defaultCertificate` (`--default-ssl-certificate`), or else a
self-signed certificate that the controller generates at start (ECDSA P-256, common name
`ScalWS Ingress Controller Fake Certificate`). With `watchNamespace` set, the default
certificate must be in that namespace.

cert-manager works as with any controller: annotate the Ingress for cert-manager (or create
a `Certificate`), name the Secret in `spec.tls`, and the controller picks the Secret up
when cert-manager has written it and again when it renews it. No reload is needed beyond
the one the new files cause. ACME inside ScalWS is not used in this mode.

Hosts with TLS get an HTTPS redirect for plain HTTP (see `ssl-redirect`). The redirect
answers `301`; ingress-nginx answers `308`.

## Annotations

All keys have the prefix `nginx.ingress.kubernetes.io/` and apply to the routes generated
from the Ingress that carries them. Booleans are `true` or `false`; durations are seconds
or a duration such as `60s`; sizes take an optional `k`, `m` or `g` suffix. A value over
4096 bytes, or with control characters, is invalid.

| Annotation | ScalWS | Notes |
|---|---|---|
| `rewrite-target` | route `rewrite.path` | Replaces the whole path, as nginx `rewrite` does. `$1` to `$9` are the captures of the regex path; a target without captures sends every matching request to that path. Makes the paths regexes. Must start with `/`. |
| `use-regex` | regex paths | See Paths. |
| `app-root` | `exact` route `/` with a 302 redirect to the value | |
| `ssl-redirect` | route `https_redirect` | On by default for hosts with TLS. Answers 301 where ingress-nginx answers 308. |
| `force-ssl-redirect` | route `https_redirect: true` | Also for hosts without TLS. |
| `permanent-redirect`, `permanent-redirect-code` | `redirect` runtime | Default 301. Codes: 301, 302, 303, 307, 308. |
| `temporal-redirect`, `temporal-redirect-code` | `redirect` runtime | Default 302. |
| `backend-protocol` | `HTTP` (default) or `HTTPS` (pool `scheme: https`) | `GRPC`, `GRPCS`, `AJP` and `FCGI` are not supported. HTTPS backends are verified, see below. |
| `proxy-ssl-name` | pool `tls.server_name` | Only with `backend-protocol: HTTPS`. |
| `proxy-ssl-secret` | pool `tls.ca_file` from the key `ca.crt` of the Secret | Only with HTTPS; the Secret must be in the Ingress's namespace. |
| `proxy-body-size` | route `max_body` | `0` means no limit. |
| `proxy-connect-timeout` | `connect_timeout` | |
| `proxy-read-timeout` | `response_timeout` | Bounds only the time until the response head, not the whole body. |
| `proxy-send-timeout` | ignored | No counterpart; an Event says so. |
| `upstream-vhost` | `rewrite.host` | |
| `service-upstream` | one member, the ClusterIP | |
| `enable-cors`, `cors-allow-origin`, `cors-allow-methods`, `cors-allow-headers`, `cors-expose-headers`, `cors-allow-credentials`, `cors-max-age` | route `cors` | ingress-nginx defaults. Credentials are refused with the origin `*`. |
| `auth-type: basic`, `auth-secret`, `auth-secret-type` (`auth-file`, `auth-map`), `auth-realm` | route `basic_auth` | The Secret's `auth` key (or, for `auth-map`, each key as user and value as hash) becomes an htpasswd file. Accepted hashes: bcrypt, SHA-256 crypt, SHA-512 crypt and Apache MD5. |
| `whitelist-source-range`, `allowlist-source-range` | route `allow` | At most 1024 entries. |
| `denylist-source-range` | route `deny` | |
| `affinity: cookie`, `session-cookie-name`, `session-cookie-max-age` (or `session-cookie-expires`) | pool `balance: hash`, `hash.by: cookie` | Default cookie `INGRESSCOOKIE`. |
| `affinity-mode` | ignored | No counterpart. |
| `upstream-hash-by` | `$remote_addr`: `client_ip`; `$http_<name>`: `header`; `$cookie_<name>`: `cookie` | Other expressions are invalid. Ignored when `affinity` is set. |
| `load-balance` | `round_robin`; `ewma` | `ewma` is served as `least_conn`, with an Event. |
| `canary`, `canary-weight`, `canary-weight-total` | weighted `backends` on the main route | See Canary. |
| `canary-by-header`, `canary-by-header-value`, `canary-by-header-pattern` | extra route with a header matcher | Without a value or pattern the header must be `always`. |
| `canary-by-cookie` | extra route with a regex matcher on `Cookie` | Matches `name=always`. |
| `configuration-snippet`, `server-snippet`, `stream-snippet` | not interpreted | `SnippetIgnored` Event, see below. |

An invalid value of an annotation that is not an access annotation is ignored with an
`InvalidAnnotation` Event, and the route keeps the default. Unknown keys under the prefix
get one `UnknownAnnotation` Event each.

### Canary

A canary Ingress (`canary: "true"`) is merged into the non-canary Ingress that has the same
host, path type and path. Unlike ingress-nginx, the two must be in the same namespace: a
canary never joins a main Ingress of another namespace (`CanaryWithoutMain`). A canary with
no main route is not served.

Header and cookie canaries become extra routes that copy the main route, so they keep its
access control, CORS, rewrite and timeouts; they take precedence over the weight, as in
ingress-nginx. The weight splits the main route between the two pools. A canary that sets
its own basic authentication or source ranges different from the main Ingress's is not
served (`InvalidAnnotation`); one that leaves them out takes the main's.

### Backends over HTTPS

ScalWS verifies the certificate of an HTTPS backend; there is no switch to turn that off.
ingress-nginx does not verify by default. A backend with a certificate from a private CA
needs `proxy-ssl-secret` (a Secret with the key `ca.crt`), and `proxy-ssl-name` when the
certificate names a host other than the Service address.

## Refused on purpose

### Access control that ScalWS does not provide

These annotations ask for protection ScalWS cannot give in this release: `auth-url`,
`auth-signin`, every `auth-tls-*`, `satisfy`, `enable-modsecurity`,
`enable-owasp-core-rules` and every `modsecurity-*`. An Ingress that carries any of them is
**not served** (its hosts and paths are left out) and gets an `UnsupportedAccessControl`
Event. Serving it without the control would expose a backend that its owner meant to
protect, so the controller fails closed.

The same holds for invalid values of the access annotations (`auth-type`, `auth-secret`,
`auth-secret-type`, `auth-realm`, the three source-range annotations): a bad CIDR, an
`auth-type` other than `basic`, a missing Secret, or a Secret without the key `auth`. A
weak or malformed htpasswd entry (plain text, `{SHA}`, crypt) also makes the Ingress
unserved; the Event names the line, never the content. A `spec.defaultBackend` of an
Ingress with basic authentication or source ranges is not served, because the catch-all
has no access control.

### Snippets

`configuration-snippet`, `server-snippet` and `stream-snippet` are never interpreted: the
controller builds the configuration from typed fields and does not paste annotation text
into any configuration, which rules out the injection class that affected ingress-nginx.
ingress-nginx disables snippets by default since v1.9, so the behaviour matches a default
install. An Ingress with a snippet is still served; the Event is `SnippetIgnored`.

## Isolation of bad Ingresses

Each Ingress is translated on its own, and problems stay with it. The whole configuration
is then validated. When that fails, the controller rebuilds it by adding Ingresses one at a
time, oldest first, validating after each, and leaves out the ones that make it fail
(`InvalidConfiguration` Event with the validator's message). This handles conflicts that
only appear together, such as the regex budget. `scalwsd` never receives an invalid
configuration from the controller; if it still refuses one, the previous configuration
stays active.

## Events

Events are `Warning` events on the Ingress. A reason is reported again only when the
problem is new; after a leader change an Event may be emitted again once.

| Reason | Meaning |
|---|---|
| `PathConflict` | Another Ingress (served or not), the default backend, or a generated name has a prior owner; this one lost. Also: the host belongs to another namespace, so this Ingress's rules for it are not served. |
| `ServiceNotFound` | The Service, its port or its EndpointSlices were not found; the route answers 503. |
| `NoEndpoints` | The Service has no ready endpoints; the route answers 503. |
| `UnsupportedAccessControl` | A fail-closed annotation is present; the Ingress is not served. |
| `InvalidAnnotation` | A value is invalid. Access annotations make the Ingress unserved; others are ignored. Also used for a canary that differs from its main in access annotations, `ewma`, and HTTPS-only annotations on an HTTP backend. |
| `UnknownAnnotation` | The key is under the prefix but unknown, or has no counterpart (`proxy-send-timeout`, `affinity-mode`). |
| `SnippetIgnored` | A snippet annotation is present and not interpreted. |
| `TLSSecret` | The TLS Secret is missing or unusable, the host is already served by another Ingress's certificate, or the host belongs to another namespace. |
| `InvalidConfiguration` | The Ingress was left out because it made the configuration invalid. |
| `CanaryWithoutMain` | The canary has no main route in its namespace, or no paths. |

## Metrics and health

The controller serves `/healthz`, `/readyz` and `/metrics` (OpenMetrics) on `:10254`.
`/readyz` answers 503 until the first configuration is loaded.

| Metric | Meaning |
|---|---|
| `scalws_ingress_syncs_total{result}` | Synchronisations, `ok` or `error`. |
| `scalws_ingress_reload_failures_total` | Reloads of `scalwsd` that failed or were refused. |
| `scalws_ingress_last_success_timestamp_seconds` | Unix time of the last successful synchronisation. |
| `scalws_ingress_ingresses{state}` | Ingresses of this controller, `served` or `rejected`. |
| `scalws_ingress_member_updates_total{result}` | Pool member updates, `ok` or `error`. |

The metrics of `scalwsd` are unchanged. Alert on a growing
`scalws_ingress_reload_failures_total` and on a stale
`scalws_ingress_last_success_timestamp_seconds`.

`scalwsd` serves its own metrics (`scalws_upstream_*` per pool and member, see
[logging-metrics.md](https://scalws.com/docs/guide/logging-metrics/)) on the health port, 8081. To look at both by hand:

```sh
kubectl port-forward deploy/ingress-scalws 10254 8081
curl -s localhost:10254/metrics    # controller
curl -s localhost:8081/metrics     # scalwsd
```

## Troubleshooting

Start with the Events of the Ingress and the two logs:

```sh
kubectl describe ingress web                     # Events at the end
kubectl get events -A --field-selector involvedObject.kind=Ingress,type=Warning
kubectl logs deploy/ingress-scalws -c controller # translation, reloads, API errors
kubectl logs deploy/ingress-scalws -c scalws     # requests, upstream failures
kubectl exec deploy/ingress-scalws -c scalws -- scalwsctl upstreams   # pools and members
```

| Symptom | Likely cause | Check |
|---|---|---|
| Every request to a host answers `404` | The Ingress is not served by this controller: `ingressClassName` names another class, the class has no `scalws.com/ingress-controller` controller value, or the Ingress has no class and the IngressClass is not the default. Or the host belongs to another namespace. | `kubectl get ingress -A` (CLASS column), `PathConflict` Events. |
| One path answers `404` | Another Ingress owns the path (older one wins), or its Ingress is not served and keeps the path. | `PathConflict`, `UnsupportedAccessControl`, `InvalidAnnotation` Events; `scalws_ingress_ingresses{state="rejected"}`. |
| `503` | The Service, its port or ready endpoints are missing. | `ServiceNotFound` or `NoEndpoints` Events; `kubectl get endpointslices -l kubernetes.io/service-name=<service>`; `scalwsctl upstreams show <pool>`. |
| Browser shows the certificate `ScalWS Ingress Controller Fake Certificate` | The host's Secret is missing or unusable, the Secret's names do not cover the host, or the Ingress with `spec.tls` has no rule for the host. | `TLSSecret` Events. |
| A few `502` while a backend scales down | Pods stop while the controller still sends to them. Idempotent requests are retried on other members (three tries); a `POST` that was sent is not. | Give the backend pods a `preStop` delay of a few seconds, which helps every controller. |
| A change of an Ingress has no effect | The reload was refused, or the Ingress was left out. | `scalws_ingress_reload_failures_total`, `InvalidConfiguration` Events, the controller log. |
| `ADDRESS` of the Ingress stays empty | No pod holds the leader Lease, or the publish Service has no address (a `LoadBalancer` Service without a load balancer in the cluster). | `kubectl get lease -n <namespace>`, `kubectl get service ingress-scalws`. |
| `scalwsd` restarts with `OOMKilled` | A memory limit below what the load needs. | See [Resources and performance](#resources-and-performance). |

## Flags of `scalws-ingress run`

The chart sets these; `POD_NAME` and `POD_NAMESPACE` come from the environment.

| Flag | Default | Meaning |
|---|---|---|
| `--base` | required | Base configuration (listeners, admin API, logging). |
| `--out` | required | Directory for the generated files (`/etc/scalws/generated` in the chart). |
| `--admin-socket` | required | Admin socket of the `scalwsd` in the pod. |
| `--https-listener` | `https` | Name of the listener that serves HTTPS. |
| `--controller-class` | `scalws.com/ingress-controller` | IngressClass `spec.controller` value to serve. |
| `--watch-namespace` | all | Watch one namespace (IngressClasses stay cluster-wide). |
| `--watch-ingress-without-class` | off | Also serve Ingresses that name no class. |
| `--cross-namespace-paths` | off | Let Ingresses of every namespace add paths to a host (HTTPRoutes: routes of every namespace a listener allows). |
| `--default-ssl-certificate` | generated | `namespace/name` of the default TLS Secret. |
| `--publish-service` | none | `namespace/name` of the Service whose address goes to the Ingress status. |
| `--election-id` | `scalws-ingress-leader` | Name of the leader election Lease. |
| `--listen` | `0.0.0.0:10254` | Address of `/healthz`, `/readyz` and `/metrics`. |
| `--debounce` | `1s` | Quiet time after a change before rebuilding. |
| `--resync` | `30s` | Interval of the member resynchronisation. |
| `--retry` | `60s` | Interval of the retry after a failed reload. |
| `--ingress` | `true` | Serve Ingress resources; `--ingress=false` needs `--gateway-api`. |
| `--gateway-api` | off | Also serve the Gateway API, see [gateway-api.md](https://scalws.com/docs/guide/gateway-api/#flags-of-scalws-ingress-run). |
| `--gateway-controller-name` | `scalws.com/gateway-controller` | GatewayClass `controllerName` to serve. |
| `--gateway-listener` | `80=http`, `443=https` | `PORT=NAME`, repeated: Gateway listener port to `scalwsd` listener. |

`scalws-ingress bootstrap --base F --out D [--https-listener N]` writes the first
configuration and exits; the chart runs it as an init container.

## RBAC

Without `watchNamespace` the chart creates a ClusterRole with `get`, `list` and `watch` on
`services`, `secrets`, `ingresses`, `ingressclasses` and `endpointslices`, `update` and
`patch` on `ingresses/status`, and `create` and `patch` on `events`. With `watchNamespace`
the same rules are a Role in that namespace, and only the IngressClass read stays
cluster-wide. The Gateway API rules are listed in [gateway-api.md](https://scalws.com/docs/guide/gateway-api/#rbac).
A Role in the release namespace allows `get`, `create` and `update` on
`leases` and `get` on `services` (the publish Service).

A `publishService` outside the release namespace needs an extra Role there that allows
`get` on that Service; the chart does not create it. Reading Secrets cluster-wide is
needed because cert-manager Secrets carry no label to select on; restrict the controller
with `watchNamespace` where that is too wide.

## Not supported

- External authentication (`auth-url`, `auth-signin`), client certificate authentication
  (`auth-tls-*`), ModSecurity: Ingresses using them are not served.
- Rate-limit annotations (`limit-*`) and other keys not listed above: reported as
  `UnknownAnnotation` and ignored.
- TCP and UDP services (the `tcp-services` and `udp-services` ConfigMaps), gRPC backends
  (`backend-protocol: GRPC`), the global ingress-nginx ConfigMap (the chart values take its
  place), an admission webhook (problems are reported as Events).
- Gateway API: served by the same controller, see [gateway-api.md](https://scalws.com/docs/guide/gateway-api/).

## Migrating from ingress-nginx

1. Install the chart in ingress mode next to ingress-nginx. Use another Service address
   (or `watchNamespace`) so both can run during the move.
2. Change `ingressClassName` of an Ingress to `scalws` (or drop the class and mark the
   IngressClass default). Leave the annotations as they are.
3. Check the Events: `kubectl get events --field-selector involvedObject.kind=Ingress`
   and `kubectl describe ingress <name>`. Fix every `UnsupportedAccessControl`,
   `InvalidAnnotation`, `TLSSecret`, `PathConflict` and `CanaryWithoutMain`. An Ingress with
   `UnsupportedAccessControl` is not served at all.
4. Check `scalws_ingress_ingresses{state="rejected"}` is zero for the Ingresses you moved.
5. Review the differences that can change behaviour:
   - HTTPS backends are verified; add `proxy-ssl-secret` for a private CA.
   - Only the namespace of a host's oldest Ingress may add paths to it; set
     `ingressController.crossNamespacePaths: true` if your Ingresses spread one host over
     several namespaces.
   - Canaries must be in the same namespace as the main Ingress.
   - `ImplementationSpecific` prefixes match whole segments; HTTP-to-HTTPS redirects
     answer 301.
   - `proxy-read-timeout` bounds the response head only; `proxy-send-timeout` and
     `affinity-mode` are ignored; `ewma` is `least_conn`.
   - Snippets are ignored; anything they did must move to annotations or the backend.
   - A TLS host needs a rule for it in the same Ingress.
6. Move the DNS name or the external address, then remove the old Ingress controller.

### Global ingress-nginx settings

ingress-nginx reads cluster-wide settings from its ConfigMap; ScalWS takes them from the
chart values. The common ones:

| ingress-nginx ConfigMap key | ScalWS chart value |
|---|---|
| `worker-processes` | `server.event_loops` (default: one per CPU the pod may use) |
| `disable-access-log` | `server.logging.access_log: false` |
| `proxy-body-size` (default `1m`) | `server.limits.max_request_body_bytes` (default `10MiB`); the annotation still sets it per Ingress |
| `keep-alive` (client idle timeout) | `server.limits.idle_timeout` (default `60s`) |
| `use-forwarded-headers`, `proxy-real-ip-cidr` | `server.trusted_proxies`: the addresses of the load balancer in front |
| `ssl-redirect` | always on for TLS hosts; the annotation turns it off per Ingress |
| `upstream-keepalive-connections`, `proxy-next-upstream-tries` | not configurable: 32 idle connections per member and three tries per request |

Other keys have no counterpart; per-Ingress annotations from the table above still apply.