Internals: decision records

ADR-0051: Ingress controller as a sidecar that writes configuration and members

Status: accepted, 2026-10-14

Context

Kubernetes phase 2a (ADR-0049, ADR-0050) gave the data plane pools, run-time members, route matching and route policies. Phase 2b adds the control plane: something has to read Ingress resources, follow pod endpoints, load Secrets as certificates and translate the common nginx.ingress.kubernetes.io/* annotations, so that ingress-nginx users move by changing ingressClassName. The translation handles input from every tenant of a cluster, and a mistake in it either exposes a backend that was meant to be protected or lets one Ingress break the others. ingress-nginx showed the cost of building configuration text from annotation values (snippet and template injection).

Decision

• A separate binary in a sidecar. scalws-ingress runs as a second container in every ingress pod, next to scalwsd, from the same image. scalwsd gains no Kubernetes dependency and no code path that is only used in a cluster. Every pod renders its own configuration; serving needs no leadership. A Lease elects the pod that writes status.loadBalancer.
• Files plus POST /v1/reload. The controller writes scalws.yaml, certificates, keys, CA bundles and htpasswd files to a memory-backed volume shared with scalwsd, and calls the admin API reload, which answers synchronously with the generation or the reason. server.reload.watch stays off: a file watcher cannot report the outcome. Files are content-addressed, written (mode 0600) before the configuration that names them, and removed only after a reload no longer references them.
• Members by PUT. Every generated pool is members_from: api. A change of ready addresses is one PUT /v1/upstreams/{pool}; no reload. A full resend every 30 s repairs missed updates and a restarted scalwsd. The rendered file carries the current members as a seed so a new pool or a restarted scalwsd does not answer 503 until the first PUT; the seed is ignored when deciding whether the structure changed. Generated pools have retries: 2 (three tries, the ingress-nginx default), which the pool spends only on safe retries (a failed connect, or an idempotent request that failed before any response was sent), so a scale-down that stops several pods at once is not met with 502s when the single retry hits another stopping pod.
• Typed rendering. Translation is a pure function from cached objects to a scalws_config::Config, serialised with serde. No annotation value is ever formatted into configuration text. Snippet annotations are not interpreted. Generated names (h_ applications, p_ pools) are derived from Kubernetes names, which never contain _, and the translator records one owner per generated name and refuses a second.
• Fail closed. An Ingress that asks for access control ScalWS does not provide (auth-url, auth-signin, auth-tls-*, satisfy, ModSecurity) or has an invalid value of an access annotation is not served, and the reason is an Event. A protected Ingress never serves its spec.defaultBackend. HTTPS backends are always verified.
• Isolation by incremental validation. Each Ingress is translated alone; the whole result is validated with the validation scalwsd uses. When that fails, Ingresses are added one at a time, oldest first, validating after each, and the ones that make it fail are left out with an Event. If the reload is still refused, the previous configuration stays active, the last accepted scalws.yaml is written back (a restarted scalwsd never reads a refused file) and the reload is retried. TLS Secrets are checked with the scalws-tls code scalwsd loads certificates with, so the two cannot disagree.
• Core additions. domains: ["*"] (at most one application) answers hosts no other domain matches, and routes gain https_redirect, which overrides the application’s value. Both are optional; configurations without them take the old code paths.
• Conflicts follow ingress-nginx where it is safe. The oldest Ingress wins a path and the default backend. A host belongs to the namespace of its oldest Ingress (served or not; the rules without a host count as one host), and only Ingresses of that namespace may add paths to it: a rule of another namespace for the host is not served and gets a PathConflict Event naming the owning namespace. ingress-nginx merges every namespace; that lets one team add a path to another team’s host, or catch with a broad prefix or a regex the paths the owner does not serve, so it is opt-in (--cross-namespace-paths, chart ingressController.crossNamespacePaths). Only Ingresses of the owning namespace may set the host’s certificate or make it a TLS host, with the opt-in too, so a newer Ingress elsewhere cannot take over a host’s certificate or HTTPS behaviour. An Ingress that is not served keeps its paths (they answer 404) and its TLS hosts, so refusing it never hands its users to a newer Ingress. Regex mode is host-wide, as in ingress-nginx, but only the owning namespace switches it on, its routes are tried before those of other namespaces, and the paths of Ingresses that do not ask for regexes are matched literally. A canary merges only with a main Ingress of its own namespace, which differs from ingress-nginx and is documented, because a cross-namespace canary would let one team redirect another team’s traffic.

Alternatives

• Embedded in scalwsd. One process, no files and no socket. Rejected: it puts a Kubernetes client, its watches and its credentials in the data-plane process, which serves untrusted traffic and runs in non-cluster installs too; a controller bug or a slow API server could affect request handling, and the data plane would build for clusters only.
• A central controller serving many proxies. One watch for the cluster and a push to every pod. Rejected for this phase: it needs a protocol and authentication between the controller and the pods, a place to keep Secrets in transit, and a state that every pod must reach; it belongs to the multi-node work (ADR-0021, phase 3). The sidecar needs only a pod-local socket and a memory volume, and N pods cost N watches.
• Template-based configuration with snippets. Rejected for the injection class above.

Consequences

• Each ingress pod watches the API server on its own, so a large cluster pays for N watches of Secrets and EndpointSlices; watchNamespace and few replicas limit it.
• The controller reads Secrets cluster-wide by default (cert-manager Secrets carry no label to select on); watchNamespace narrows it.
• Per-pod state (limits, caches, affinity) is not shared, as in ADR-0049.
• After a leader change an Event may be emitted again once.
• Ingresses using unsupported access control are unserved until their owners change them; the migration guide lists the differences that need a decision.
• Phase 2c (Gateway API) can reuse the translation target, the file and member channels and the isolation scheme.

See also: ADR-0052, the Gateway API as a second input of this sidecar, which extends the host ownership rule to both inputs.