# ADR-0051: Ingress controller as a sidecar that writes configuration and members

URL: https://scalws.com/docs/adr/0051-ingress-controller/
Source: `docs/adr/0051-ingress-controller.md` in the ScalWS repository

Status: accepted, 2026-10-14

## Context

Kubernetes phase 2a (ADR-0049, ADR-0050) gave the data plane pools, run-time members,
route matching and route policies. Phase 2b adds the control plane: something has to read
Ingress resources, follow pod endpoints, load Secrets as certificates and translate the
common `nginx.ingress.kubernetes.io/*` annotations, so that ingress-nginx users move by
changing `ingressClassName`. The translation handles input from every tenant of a cluster,
and a mistake in it either exposes a backend that was meant to be protected or lets one
Ingress break the others. ingress-nginx showed the cost of building configuration text
from annotation values (snippet and template injection).

## Decision

* **A separate binary in a sidecar.** `scalws-ingress` runs as a second container in every
  ingress pod, next to `scalwsd`, from the same image. `scalwsd` gains no Kubernetes
  dependency and no code path that is only used in a cluster. Every pod renders its own
  configuration; serving needs no leadership. A Lease elects the pod that writes
  `status.loadBalancer`.
* **Files plus `POST /v1/reload`.** The controller writes `scalws.yaml`, certificates,
  keys, CA bundles and htpasswd files to a memory-backed volume shared with `scalwsd`, and
  calls the admin API reload, which answers synchronously with the generation or the
  reason. `server.reload.watch` stays off: a file watcher cannot report the outcome. Files
  are content-addressed, written (mode 0600) before the configuration that names them, and
  removed only after a reload no longer references them.
* **Members by `PUT`.** Every generated pool is `members_from: api`. A change of ready
  addresses is one `PUT /v1/upstreams/{pool}`; no reload. A full resend every 30 s repairs
  missed updates and a restarted `scalwsd`. The rendered file carries the current members
  as a seed so a new pool or a restarted `scalwsd` does not answer 503 until the first
  `PUT`; the seed is ignored when deciding whether the structure changed. Generated pools
  have `retries: 2` (three tries, the ingress-nginx default), which the pool spends only
  on safe retries (a failed connect, or an idempotent request that failed before any
  response was sent), so a scale-down that stops several pods at once is not met with
  `502`s when the single retry hits another stopping pod.
* **Typed rendering.** Translation is a pure function from cached objects to a
  `scalws_config::Config`, serialised with `serde`. No annotation value is ever formatted
  into configuration text. Snippet annotations are not interpreted. Generated names (`h_`
  applications, `p_` pools) are derived from Kubernetes names, which never contain `_`,
  and the translator records one owner per generated name and refuses a second.
* **Fail closed.** An Ingress that asks for access control ScalWS does not provide
  (`auth-url`, `auth-signin`, `auth-tls-*`, `satisfy`, ModSecurity) or has an invalid
  value of an access annotation is not served, and the reason is an Event. A protected
  Ingress never serves its `spec.defaultBackend`. HTTPS backends are always verified.
* **Isolation by incremental validation.** Each Ingress is translated alone; the whole
  result is validated with the validation `scalwsd` uses. When that fails, Ingresses are
  added one at a time, oldest first, validating after each, and the ones that make it fail
  are left out with an Event. If the reload is still refused, the previous configuration
  stays active, the last accepted `scalws.yaml` is written back (a restarted `scalwsd`
  never reads a refused file) and the reload is retried. TLS Secrets are checked with the
  `scalws-tls` code `scalwsd` loads certificates with, so the two cannot disagree.
* **Core additions.** `domains: ["*"]` (at most one application) answers hosts no other
  domain matches, and routes gain `https_redirect`, which overrides the application's
  value. Both are optional; configurations without them take the old code paths.
* **Conflicts follow ingress-nginx where it is safe.** The oldest Ingress wins a path and
  the default backend. A host belongs to the namespace of its oldest Ingress (served or
  not; the rules without a host count as one host), and only Ingresses of that namespace
  may add paths to it: a rule of another namespace for the host is not served and gets a
  `PathConflict` Event naming the owning namespace. ingress-nginx merges every namespace;
  that lets one team add a path to another team's host, or catch with a broad prefix or a
  regex the paths the owner does not serve, so it is opt-in (`--cross-namespace-paths`,
  chart `ingressController.crossNamespacePaths`). Only Ingresses of the owning namespace
  may set the host's certificate or make it a TLS host, with the opt-in too, so a newer
  Ingress elsewhere cannot take over a host's certificate or HTTPS behaviour. An Ingress that is
  not served keeps its paths (they answer 404) and its TLS hosts, so refusing it never
  hands its users to a newer Ingress. Regex mode is host-wide, as in ingress-nginx, but
  only the owning namespace switches it on, its routes are tried before those of other
  namespaces, and the paths of Ingresses that do not ask for regexes are matched
  literally. A canary
  merges only with a main Ingress of its own namespace, which differs from ingress-nginx
  and is documented, because a cross-namespace canary would let one team redirect another
  team's traffic.

## Alternatives

* **Embedded in `scalwsd`.** One process, no files and no socket. Rejected: it puts a
  Kubernetes client, its watches and its credentials in the data-plane process, which
  serves untrusted traffic and runs in non-cluster installs too; a controller bug or a
  slow API server could affect request handling, and the data plane would build for
  clusters only.
* **A central controller serving many proxies.** One watch for the cluster and a push to
  every pod. Rejected for this phase: it needs a protocol and authentication between the
  controller and the pods, a place to keep Secrets in transit, and a state that every pod
  must reach; it belongs to the multi-node work (ADR-0021, phase 3). The sidecar needs
  only a pod-local socket and a memory volume, and N pods cost N watches.
* **Template-based configuration with snippets.** Rejected for the injection class above.

## Consequences

* Each ingress pod watches the API server on its own, so a large cluster pays for N
  watches of Secrets and EndpointSlices; `watchNamespace` and few replicas limit it.
* The controller reads Secrets cluster-wide by default (cert-manager Secrets carry no
  label to select on); `watchNamespace` narrows it.
* Per-pod state (limits, caches, affinity) is not shared, as in ADR-0049.
* After a leader change an Event may be emitted again once.
* Ingresses using unsupported access control are unserved until their owners change them;
  the migration guide lists the differences that need a decision.
* Phase 2c (Gateway API) can reuse the translation target, the file and member channels
  and the isolation scheme.

See also: [ADR-0052](https://scalws.com/docs/adr/0052-gateway-api/), the Gateway API as a second input of this
sidecar, which extends the host ownership rule to both inputs.