Users' guide

Ingress controller

scalws-ingress turns Kubernetes networking.k8s.io/v1 Ingress resources and the common nginx.ingress.kubernetes.io/* annotations into ScalWS configuration, so ScalWS can take the place of ingress-nginx. It runs as a second container next to scalwsd in every ingress pod. This guide covers the install, how the controller works, TLS, the supported annotations with every difference from ingress-nginx, what is refused on purpose, the Events and metrics, the flags, RBAC and a migration checklist.

Design record: ADR-0051. The data plane (pools, route matching and policies) is described in load-balancing.md and configuration.md. General Kubernetes notes are in kubernetes.md. The same controller also serves the Gateway API (Gateway, HTTPRoute), alone or next to Ingresses: gateway-api.md.

Note: the container image ghcr.io/xsmtx/scalws and the Helm chart oci://ghcr.io/xsmtx/charts/scalws are not yet published for public download. Until they are, the steps on this page describe how the Kubernetes deployment works but cannot be followed as written; the Linux packages are the supported way to install 1.0.0.

Install

The controller is a mode of the Helm chart, in chart versions after 1.0.0-rc.1 (the 1.0.0-rc.1 chart has no ingress controller mode):

helm install ingress oci://ghcr.io/xsmtx/charts/scalws --version <chart version> \
    --set ingressController.enabled=true

This creates a Deployment (or a DaemonSet with ingressController.kind=DaemonSet) with the two containers, a LoadBalancer Service on ports 80 and 443, the IngressClass scalws, a ServiceAccount and the RBAC the controller needs. The values:

Some chart values are refused with an error in this mode because they have no meaning there: sites, existingConfigMap, php.enabled, tls.enabled, content.image and content.existingClaim. The sites come from the Ingress resources.

http3: true needs an image whose scalwsd is built with --features http3. The published image has no HTTP/3, and scalwsd refuses to start with the setting.

An Ingress is served when spec.ingressClassName names an IngressClass whose spec.controller is the controller value. An Ingress without a class is served when the IngressClass is marked default (ingressclass.kubernetes.io/is-default-class: "true"), or when watchIngressWithoutClass is set. The deprecated kubernetes.io/ingress.class annotation is honoured with the same rule. To serve an Ingress, set ingressClassName: scalws:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 8m
spec:
  ingressClassName: scalws
  tls:
    - hosts: [www.example.com]
      secretName: www-example-com
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: {name: web, port: {number: 80}}

Check the install

With the release named ingress the objects are called ingress-scalws:

kubectl get pods -l app.kubernetes.io/instance=ingress    # 2/2 Ready: scalws and controller
kubectl get ingressclass scalws
kubectl get service ingress-scalws                        # EXTERNAL-IP of the LoadBalancer
kubectl get ingress web                                   # ADDRESS is set by the leader pod
curl -H 'Host: www.example.com' http://<EXTERNAL-IP>/

A pod becomes ready only after the controller has loaded its first configuration, so a pod that stays at 1/2 points at the controller: kubectl logs deploy/ingress-scalws -c controller (RBAC errors with watchNamespace, an API server it cannot reach). A host that answers 404 is not served yet; see Troubleshooting.

Resources and performance

scalwsd (the chart’s resources) requests cpu: 100m and memory: 128Mi and has no memory limit by default, like the ingress-nginx and Traefik charts. Its memory grows with the requests in flight and the size of the proxied responses: about 330 MiB for 256 concurrent 100 KiB responses. If you set a limit, leave that headroom, or scalwsd is OOM-killed under load. ingressController.resources applies to the controller container only, which needs little.

Settings of scalwsd go under server: in the values, as outside ingress mode. Keep server.accept_policy at its default, shared: the opt-in fountain policy saves CPU at moderate load but places keep-alive connections on fewer event loops, which costs a proxy latency (1.5x to 1.6x the p99 at a fixed rate in the benchmark, with the connection cap of 2026-10-15). server.event_loops defaults to one loop per CPU the pod may use. The measured comparison with ingress-nginx and Traefik is in the Kubernetes ingress benchmark.

How it works

 API server --watch--> scalws-ingress --scalws.yaml, certificates, htpasswd--> emptyDir
                            |                                                    |
                            +--POST /v1/reload---------------------------------> scalwsd
                            +--PUT /v1/upstreams/{pool}--------------------------> (admin socket)

The two containers share the admin socket (/run/scalws) and a memory-backed volume (/etc/scalws/generated) that holds the generated configuration, certificates, keys and htpasswd files. Keys never touch a disk. Both containers run as user 65532 with a read-only root file system, no capabilities and the RuntimeDefault seccomp profile; the admin socket is reachable only inside the pod. Secret contents are never logged.

Structure channel. The controller watches IngressClass, Ingress, Service, EndpointSlice and Secret resources (cluster-wide, or one namespace with --watch-namespace). A change marks the state dirty; after a quiet time of 1 s it translates the whole state, validates it with the same validation scalwsd runs, and when the rendered scalws.yaml differs from the last one it writes it atomically and calls POST /v1/reload. The call answers synchronously, so a refused configuration is known immediately: the previous configuration stays active, the controller writes the last configuration that scalwsd accepted back to scalws.yaml (so a restarted scalwsd never starts from a file it refused), scalws_ingress_reload_failures_total rises and the reload is retried on the next change or after 60 s. Certificate, key, CA and htpasswd files are content-addressed, written with mode 0600 before the configuration that references them, and removed only after a reload no longer references them. server.reload.watch stays off in this mode.

Member channel. Every pool the controller generates is members_from: api. When the ready addresses of a pool change, the controller sends PUT /v1/upstreams/{pool} with the full member list; no reload happens, so scaling a backend does not touch the configuration. Every 30 s it sends every pool again, which repairs a missed update and a restarted scalwsd container. The rendered configuration also carries each pool’s current members as its members seed, so a pool created by a reload, and a scalwsd that restarts, start with members instead of answering 503. Member-only changes never rewrite the file; the seed is refreshed with the next structural write.

Start-up. An init container (scalws-ingress bootstrap) writes a first configuration with the listeners, the health listener, the admin socket and no applications, so every request gets 404. The controller waits for its caches, writes the first real configuration, reloads, pushes the members and only then reports ready on /readyz. The pod’s readiness probe points at the controller, so a pod receives traffic only with a real configuration.

Status and leadership. The holder of a Lease (named <fullname>-leader in the chart, scalws-ingress-leader by default, in the controller’s namespace; 15 s duration) writes status.loadBalancer of the Ingresses it serves, with the address of the publish Service. Serving traffic needs no leadership: every pod renders its own configuration. The Lease is released at shutdown.

Generated names

One tenant, ingress, holds one application per host, named h_ plus the host with . turned into _ and * removed (www.example.com gives h_www_example_com, *.example.com gives h__example_com). Rules without a host and spec.defaultBackend go to the application default, which has the catch-all domain "*" and answers hosts no other application serves. Pools are named p_<namespace>_<service>_<port>, plus _<6 hex> of the pool settings when annotations change balance, hash or scheme, so two Ingresses that use one Service with different affinity get different pools. Names over 63 characters are cut and get 8 hex digits of a hash of the full name. Kubernetes names never contain _, so generated names do not collide by accident; the controller still records the owner of every generated name and refuses a second owner with a PathConflict Event.

Which Ingress wins

A host belongs to the namespace of its oldest Ingress with a rule for it, served or not (Ingresses of another class do not count; the rules without a host count as one host, the catch-all). Several Ingresses of that namespace may define the host: their paths are merged into one application, as in ingress-nginx. A rule of an Ingress of another namespace for the host is not served: that Ingress gets a PathConflict Event saying which namespace the host belongs to, and its rules for other hosts are still served. So another namespace can neither add a path to a host nor catch, with a broad prefix or a regex, the paths the owner does not serve (or no longer serves because its Ingress was refused). ingress-nginx merges the Ingresses of every namespace; ingressController.crossNamespacePaths (--cross-namespace-paths) restores that, see below.

When two Ingresses define the same host, path type, path and matchers, the oldest (creationTimestamp, then namespace/name) wins and the other gets a PathConflict Event. The same rule picks the one whose spec.defaultBackend is used.

Only Ingresses of the namespace the host belongs to may set the host’s certificate and make it a TLS host (with the HTTPS redirect); among them the oldest with a spec.tls entry for the host wins. A spec.tls entry for the host from another namespace changes nothing and gets a TLSSecret Event naming the owning namespace, so a newer Ingress elsewhere cannot take over the certificate or the HTTPS behaviour of a host.

An Ingress that is not served (refused access annotations, an invalid annotation, a missing basic-auth Secret, or left out because it broke the configuration) keeps its paths and its TLS hosts: a newer Ingress with the same path gets a PathConflict Event and the path answers 404, instead of sending the users of the older Ingress to another backend. Paths it does not have may still be served by other Ingresses of its namespace.

With crossNamespacePaths: true Ingresses of every namespace merge their paths into the host, as in ingress-nginx; the oldest still wins a path, and the certificate, TLS and regex rules of the owning namespace stay as described here. An unserved Ingress then also keeps its paths against newer Ingresses of other namespaces. Turn it on only when every namespace that can create Ingresses of this class is trusted with every host.

A host under an older wildcard Ingress of another namespace is not served either: when namespace a has the oldest Ingress for *.example.com, a newer Ingress of namespace b for shop.example.com gets a PathConflict Event naming namespace a, because the wildcard owner would otherwise lose a host it already serves. This is a change from the first version of the controller, which compared host patterns only as written; Ingresses of the wildcard’s own namespace are not affected, and crossNamespacePaths: true restores the old behaviour among Ingresses. A spec.defaultBackend counts as a claim on the catch-all, like the rules without a host.

With the Gateway API input on, the same ownership rule covers Gateways and HTTPRoutes: on each listener the oldest claim wins, whatever its input, and an Ingress and an HTTPRoute never serve one host together (not even with crossNamespacePaths, which lifts only the namespace rule among Ingresses). A newer Ingress for a host an HTTPRoute serves gets PathConflict; a newer HTTPRoute for an Ingress host gets HostnameConflict. See gateway-api.md, which also describes moving a host from an Ingress to an HTTPRoute.

Paths

With use-regex: "true" or a rewrite-target, Prefix and ImplementationSpecific paths become regular expressions, as in ingress-nginx. They get (?i) prepended, because ingress-nginx regex locations are case-insensitive. The expressions are linear-time, with no look-around or back-references (ADR-0050).

Regex mode covers the whole host, as in ingress-nginx, but only the namespace the host belongs to (see Which Ingress wins) can switch it on: when a served Ingress of that namespace with a rule for the host asks for regex paths, every Prefix and ImplementationSpecific path of that host, from all its Ingresses and namespaces, becomes a regex. Paths of Ingresses that do not ask for regex paths themselves are matched literally (their regex characters are escaped), so they keep their meaning and another Ingress cannot make them invalid. Exact paths stay exact. The regex routes of such a host are tried in two groups, first those of the owning namespace, then those of every other namespace, each group longest path (as written in the Ingress) first, so another namespace can never shadow a path of the owner. So a frontend Ingress with Prefix / and an API Ingress of the same namespace with rewrite-target: /$2 and path /api(/|$)(.*) work together: /api/x reaches the API as /x, everything else the frontend.

A regex Ingress of another namespace does not switch the host: its own paths are regexes, which the router tries after the exact and prefix routes, so it cannot take the paths of the owner. Hosts without a regex Ingress of the owning namespace keep plain prefix routes. Precedence inside a host is exact, then longest prefix, then regex in that order.

Backends

A backend service.name and service.port (number or name) becomes a pool. Its members are the ready addresses (IPv4 and IPv6, serving and not terminating) of the Service’s EndpointSlices for the matching port: by name for a named port, by the Service’s targetPort otherwise. With service-upstream: "true" the pool has one member, the ClusterIP and port. An ExternalName Service gives one member, externalName:port. A missing Service, port or EndpointSlice leaves the pool empty: the route answers 503, as with ingress-nginx, and the Ingress gets a ServiceNotFound or NoEndpoints Event.

A request makes up to three tries on different members (the pool’s retries: 2), the ingress-nginx default of proxy-next-upstream-tries. A request is tried again only while nothing has been sent to the client: after a failed connect, or, for an idempotent method (GET, HEAD, OPTIONS, PUT, DELETE), after a 502, 503 or 504, a timeout or a connection broken before the response head (a pod that is shutting down). A POST that reached a member is never sent again. The number of tries is not configurable per Ingress.

TLS

For every spec.tls entry the controller loads the Secret of type kubernetes.io/tls from the Ingress’s namespace and serves the entry’s hosts with it. The Secret is checked with the same code scalwsd loads certificates with: a missing Secret, a wrong type, a tls.crt with any PEM block that does not parse, a key that does not match the certificate, or a certificate whose names do not cover a host leaves that host on the default certificate and gives a TLSSecret Event. A TLS host only counts when the same Ingress has a rule for it (the same as ingress-nginx).

Hosts without a certificate of their own get the default certificate: ingressController.defaultCertificate (--default-ssl-certificate), or else a self-signed certificate that the controller generates at start (ECDSA P-256, common name ScalWS Ingress Controller Fake Certificate). With watchNamespace set, the default certificate must be in that namespace.

cert-manager works as with any controller: annotate the Ingress for cert-manager (or create a Certificate), name the Secret in spec.tls, and the controller picks the Secret up when cert-manager has written it and again when it renews it. No reload is needed beyond the one the new files cause. ACME inside ScalWS is not used in this mode.

Hosts with TLS get an HTTPS redirect for plain HTTP (see ssl-redirect). The redirect answers 301; ingress-nginx answers 308.

Annotations

All keys have the prefix nginx.ingress.kubernetes.io/ and apply to the routes generated from the Ingress that carries them. Booleans are true or false; durations are seconds or a duration such as 60s; sizes take an optional k, m or g suffix. A value over 4096 bytes, or with control characters, is invalid.

An invalid value of an annotation that is not an access annotation is ignored with an InvalidAnnotation Event, and the route keeps the default. Unknown keys under the prefix get one UnknownAnnotation Event each.

Canary

A canary Ingress (canary: "true") is merged into the non-canary Ingress that has the same host, path type and path. Unlike ingress-nginx, the two must be in the same namespace: a canary never joins a main Ingress of another namespace (CanaryWithoutMain). A canary with no main route is not served.

Header and cookie canaries become extra routes that copy the main route, so they keep its access control, CORS, rewrite and timeouts; they take precedence over the weight, as in ingress-nginx. The weight splits the main route between the two pools. A canary that sets its own basic authentication or source ranges different from the main Ingress’s is not served (InvalidAnnotation); one that leaves them out takes the main’s.

Backends over HTTPS

ScalWS verifies the certificate of an HTTPS backend; there is no switch to turn that off. ingress-nginx does not verify by default. A backend with a certificate from a private CA needs proxy-ssl-secret (a Secret with the key ca.crt), and proxy-ssl-name when the certificate names a host other than the Service address.

Refused on purpose

Access control that ScalWS does not provide

These annotations ask for protection ScalWS cannot give in this release: auth-url, auth-signin, every auth-tls-*, satisfy, enable-modsecurity, enable-owasp-core-rules and every modsecurity-*. An Ingress that carries any of them is not served (its hosts and paths are left out) and gets an UnsupportedAccessControl Event. Serving it without the control would expose a backend that its owner meant to protect, so the controller fails closed.

The same holds for invalid values of the access annotations (auth-type, auth-secret, auth-secret-type, auth-realm, the three source-range annotations): a bad CIDR, an auth-type other than basic, a missing Secret, or a Secret without the key auth. A weak or malformed htpasswd entry (plain text, {SHA}, crypt) also makes the Ingress unserved; the Event names the line, never the content. A spec.defaultBackend of an Ingress with basic authentication or source ranges is not served, because the catch-all has no access control.

Snippets

configuration-snippet, server-snippet and stream-snippet are never interpreted: the controller builds the configuration from typed fields and does not paste annotation text into any configuration, which rules out the injection class that affected ingress-nginx. ingress-nginx disables snippets by default since v1.9, so the behaviour matches a default install. An Ingress with a snippet is still served; the Event is SnippetIgnored.

Isolation of bad Ingresses

Each Ingress is translated on its own, and problems stay with it. The whole configuration is then validated. When that fails, the controller rebuilds it by adding Ingresses one at a time, oldest first, validating after each, and leaves out the ones that make it fail (InvalidConfiguration Event with the validator’s message). This handles conflicts that only appear together, such as the regex budget. scalwsd never receives an invalid configuration from the controller; if it still refuses one, the previous configuration stays active.

Events

Events are Warning events on the Ingress. A reason is reported again only when the problem is new; after a leader change an Event may be emitted again once.

Metrics and health

The controller serves /healthz, /readyz and /metrics (OpenMetrics) on :10254. /readyz answers 503 until the first configuration is loaded.

The metrics of scalwsd are unchanged. Alert on a growing scalws_ingress_reload_failures_total and on a stale scalws_ingress_last_success_timestamp_seconds.

scalwsd serves its own metrics (scalws_upstream_* per pool and member, see logging-metrics.md) on the health port, 8081. To look at both by hand:

kubectl port-forward deploy/ingress-scalws 10254 8081
curl -s localhost:10254/metrics    # controller
curl -s localhost:8081/metrics     # scalwsd

Troubleshooting

Start with the Events of the Ingress and the two logs:

kubectl describe ingress web                     # Events at the end
kubectl get events -A --field-selector involvedObject.kind=Ingress,type=Warning
kubectl logs deploy/ingress-scalws -c controller # translation, reloads, API errors
kubectl logs deploy/ingress-scalws -c scalws     # requests, upstream failures
kubectl exec deploy/ingress-scalws -c scalws -- scalwsctl upstreams   # pools and members

Flags of scalws-ingress run

The chart sets these; POD_NAME and POD_NAMESPACE come from the environment.

scalws-ingress bootstrap --base F --out D [--https-listener N] writes the first configuration and exits; the chart runs it as an init container.

RBAC

Without watchNamespace the chart creates a ClusterRole with get, list and watch on services, secrets, ingresses, ingressclasses and endpointslices, update and patch on ingresses/status, and create and patch on events. With watchNamespace the same rules are a Role in that namespace, and only the IngressClass read stays cluster-wide. The Gateway API rules are listed in gateway-api.md. A Role in the release namespace allows get, create and update on leases and get on services (the publish Service).

A publishService outside the release namespace needs an extra Role there that allows get on that Service; the chart does not create it. Reading Secrets cluster-wide is needed because cert-manager Secrets carry no label to select on; restrict the controller with watchNamespace where that is too wide.

Not supported

• External authentication (auth-url, auth-signin), client certificate authentication (auth-tls-*), ModSecurity: Ingresses using them are not served.
• Rate-limit annotations (limit-*) and other keys not listed above: reported as UnknownAnnotation and ignored.
• TCP and UDP services (the tcp-services and udp-services ConfigMaps), gRPC backends (backend-protocol: GRPC), the global ingress-nginx ConfigMap (the chart values take its place), an admission webhook (problems are reported as Events).
• Gateway API: served by the same controller, see gateway-api.md.

Migrating from ingress-nginx

1. Install the chart in ingress mode next to ingress-nginx. Use another Service address (or watchNamespace) so both can run during the move.
2. Change ingressClassName of an Ingress to scalws (or drop the class and mark the IngressClass default). Leave the annotations as they are.
3. Check the Events: kubectl get events --field-selector involvedObject.kind=Ingress and kubectl describe ingress <name>. Fix every UnsupportedAccessControl, InvalidAnnotation, TLSSecret, PathConflict and CanaryWithoutMain. An Ingress with UnsupportedAccessControl is not served at all.
4. Check scalws_ingress_ingresses{state="rejected"} is zero for the Ingresses you moved.
5. Review the differences that can change behaviour:
   • HTTPS backends are verified; add proxy-ssl-secret for a private CA.
   • Only the namespace of a host’s oldest Ingress may add paths to it; set ingressController.crossNamespacePaths: true if your Ingresses spread one host over several namespaces.
   • Canaries must be in the same namespace as the main Ingress.
   • ImplementationSpecific prefixes match whole segments; HTTP-to-HTTPS redirects answer 301.
   • proxy-read-timeout bounds the response head only; proxy-send-timeout and affinity-mode are ignored; ewma is least_conn.
   • Snippets are ignored; anything they did must move to annotations or the backend.
   • A TLS host needs a rule for it in the same Ingress.
6. Move the DNS name or the external address, then remove the old Ingress controller.

Global ingress-nginx settings

ingress-nginx reads cluster-wide settings from its ConfigMap; ScalWS takes them from the chart values. The common ones:

Other keys have no counterpart; per-Ingress annotations from the table above still apply.