Routing design¶

One routing mode applies to the whole platform, set by the Helm value routing.mode. A routing backend in core/src/ecmwf/sites/core/kubernetes/ routing/ hides the difference; the operator uses it to create, update, and delete the route of a site. The Hub only maintains the read-only state and the master-Ingress snippet through the data access object (DAO). Installation and configuration procedures are in sys-admin: routing and networking.

Modes¶

flowchart LR
    subgraph ingress["ingress mode"]
        I1["client"] --> M["master Ingress (host, TLS, platform snippets)"] --> MN["minion Ingress per site (/space/site)"] --> SVC["site Service"]
        M --> H["Hub: master defaultBackend + minion path /"] --> HS["hub Service"]
    end
    subgraph gateway["gateway mode"]
        G1["client"] --> GW["Gateway (listener + certificate)"] --> HR["HTTPRoute Hub"] --> HS2["hub Service"]
        GW --> SR["HTTPRoute per site"] --> SS["site Service"]
    end

The site path layout is identical in both modes (01-overview). Each site gets exactly one route object. Public sub-domain routing is not implemented: sites are reachable by path under the platform host only.

routing.host and routing.scheme (chart values) drive route hosts and the gateway redirect scheme. Site routes use routing.host; in ingress mode a site route without routing.host falls back to the host of the master Ingress — specifically the first rule of that Ingress (ingress_utils.extract_ingress_host, result cached with the cache.timeout_cold time to live (TTL)) — while gateway mode fails when no host is available (core/.../routing/endpoint.py; it never reads an Ingress). The fallback applies only to a legacy or hand-written configuration that omits routing.host: chart rendering requires the value in every mode (below).

The chart rejects an unknown routing.mode at render time, rejects sitesConfig.routing ("use the top-level routing values"), and rejects the moved keys ingress.* (top level), routing.hstsEnabled, sitesConfig.kubernetes.ingress_class_name, and sitesConfig.kubernetes.ingress_extra_annotations. The Python schema applies the same rules to the rendered configuration, so a hand-edited ConfigMap cannot carry them either (_helpers.tpl, core/.../config/schema/models.py).

Routing values (chart)¶

Route shapes in ingress mode¶

Three kinds of Ingress objects exist. All carry ingressClassName: routing.ingress.className.

sites-ingress-master (chart): the mergeable master for routing.host. It carries defaultBackend: sites-hub:8080, so the Hub answers every path no minion claims. Its fixed annotations set nginx.org/redirect-to-https, a 600 s connect/read/send proxy timeout, 64k buffers, unlimited client body size, buffering off, and a location-snippet with proxy_cache off and request buffering off. The TLS block renders only with routing.tls.enabled, and references routing.tls.secretName — the chart never creates the Secret; a certificate controller or the operator must provide it. routing.ingress.hubAnnotations merges on top.

sites-ingress Hub minion (chart): the mergeable minion with the single path / → sites-hub:8080. Its nginx.org/location-snippets carry the platform behaviour that document 10 attributes to the "minion snippet": the five security headers, Content Security Policy (CSP) from trusted_domains, HTTP Strict Transport Security (HSTS) when routing.hsts, Cache-Control: no-cache, and X-Sites-Proxy. The same snippet denies /hub/metrics/ (deny all) and rewrites /<space>/<site> without a trailing slash to the path with a trailing slash (redirect) — the browser-friendly site entry.

Site minions (operator-built, one per site, named after the site id): mergeable minion, single path /<space>/<site>, host from routing.host or the master Ingress. Annotations merge in this order, so later layers win: site metadata annotations → generated annotations (mergeable-ingress-type: minion, websocket-services: <site id>, and lb-method: hash $remote_addr consistent when site.replicas > 1) → application annotations → routing.ingress.routeAnnotations. A disabled site gets the snippet return 302 /hub/error/page/410/; appended to its location-snippets (ingress_utils.archive_conf_snippet).

Route shapes in gateway mode¶

Gateway. The chart creates routing.gateway.name only when routing.gateway.create is true. Listener selection, in order:

1. routing.gateway.listeners (a raw listener list) replaces everything.
2. With routing.tls.enabled: an HTTPS listener on port 443 named https, hostname routing.host, certificateRefs to routing.tls.secretName (reference only), plus a port-80 http listener when routing.gateway.httpRedirect is true.
3. Otherwise: one HTTP listener on port 80 named http.

Every chart-managed listener sets allowedRoutes.namespaces.from: Same. With create: false the chart renders no Gateway and cannot check these conditions — the existing Gateway must accept routes from the release namespace.

HTTP→HTTPS redirect. The chart renders sites-http-redirect (an HTTPRoute on the http listener with a RequestRedirect filter, status 301) only when all of: create is true, routing.tls.enabled, httpRedirect, and no custom listeners. Setting listeners silently drops the redirect route, and an existing Gateway gets no redirect at all — there, redirecting is the platform operator's job. Ingress mode does not need either: the master Ingress always carries nginx.org/redirect-to-https.

Hub HTTPRoute (sites-hub): hostnames: [routing.host], parentRef from the chart helper, routing.gateway.routeAnnotations, and two rules: /hub/metrics (Prefix) redirects to /hub/error/page/403/ (status 302 — monitoring reads the Service directly), and / (Prefix) → sites-hub:8080.

Site HTTPRoutes (operator-built through SiteExporter.to_http_route, named after the site id): one parentRefs entry and hostnames: [routing.host]; annotations combine the policy provider's annotations with routing.gateway.routeAnnotations; updates carry over the previous resourceVersion and ownerReferences. The parentRef includes namespace only when the Gateway lives in another namespace and sectionName only when routing.gateway.listenerName is set; the chart helper and the Python get_parent_ref build the same reference.

Rule conversion translates the application's Ingress paths (core/.../gateway_utils.py):

Site Service (operator-built through SiteExporter.to_service, named after the site id): both modes route to it. It selects the site pod, uses ClientIP session affinity, and sets publishNotReadyAddresses: true. The pod stays in the Service endpoints when it is not Ready. The site-custom readiness probe (custom applications) only reports whether the custom application accepts connections. It does not remove the pod from routing, so the static site, the Site API, and the other site containers stay reachable while the custom application does not accept connections. During pod start-up, a request can reach the pod before its containers listen; the gateway then answers 502, not 503. An existing Service gets the field at the next Site update or forced upgrade.

Route ownership¶

Every route object created for a site carries the sites.ecmwf.int/id label, and its name is the site id in both modes. Two rules follow from this:

1. The operator owns site routes. The routing backend reads the object by the site id and only changes routes whose sites.ecmwf.int/id label equals the site id; a same-named object with any other label raises RouteOwnershipError (core/.../routing/backend.py). A site update (update_routes in operator/.../site_update.py) creates or replaces the route of the active mode — and deletes the active-mode route when the site has no public path — then deletes any route of the other mode that carries the site's label. The other-mode deletion is skipped for a missing or foreign route, and a deletion failure retries the whole update after 30 s. The create path (create_routes) does not delete other-mode routes; cleanup therefore depends on an update, which is exactly what the staged mode switch exploits. New routes and policy objects are adopted (kopf.adopt) so they are removed with the Site.
2. The Helm release owns platform routes. The Hub route, the Gateway (when chart-created), and the HTTP→HTTPS redirect route (routing.gateway.httpRedirect, default true) are Helm-managed.

Gateway policy resources (from a policy provider) are synced with the site route: the operator creates missing objects, replaces owned ones, lists by the label selector sites.ecmwf.int/id=<site id> and deletes stale ones, and deletes all policy objects of the provider when the route is deleted. A policy object that exists with a foreign label raises RouteOwnershipError (core/.../routing/gateway.py). The chart can grant the operator role-based access control (RBAC) for extra policy resource kinds through routing.gateway.policyRbacRules; the operator Role already allows httproutes get/create/update/delete in both modes, so a mode switch never needs an RBAC change. The Hub has no HTTPRoute RBAC at all — it only touches the master Ingress and the platform-state ConfigMap.

Policy providers¶

Standard Gateway API resources do not cover features such as session affinity. A policy provider converts the requested features into controller-specific resources (core/.../routing/policies.py):

• The only shipped provider is none (StandardPolicyProvider): no annotations, no policy objects.
• routing.gateway.policies lists required features (currently session_affinity); the configuration schema rejects a feature the selected provider does not support.
• When site.replicas > 1 and no policy supplies session affinity, the export logs a warning (in ingress mode the lb-method annotation supplies it automatically).
• New providers register at import time through register_policy_provider; an unknown provider name is a configuration error.

Mode switch design¶

A mode switch is deliberately staged, because the Helm upgrade changes only the platform routes. Sites keep serving through the old mode until the operator touches each one, and the operator touches a site only on an update. The procedure exploits this:

1. Upgrade to a version that supports both modes, keeping the current mode. The Hub copies the legacy read-only state from the master Ingress into the sites-platform-state ConfigMap.
2. Open the maintenance window.
3. helm upgrade with the new routing.mode.
4. If the new entry point has a different external address, move the public DNS record to it (integration or manual). When both modes share one address, the record does not change. The platform does not manage DNS records.
5. Verify the Hub.
6. Force an upgrade of all sites: "Force Upgrade All Sites" or POST /hub/api/v2/sites/upgrade/?force=true (09). Each site update installs the new-mode route and removes the old-mode route.
7. Verify all sites.

Rollback: run helm upgrade with the previous mode (or helm rollback). Sites you did not upgrade become reachable again through their retained old-mode route; upgraded sites need one more forced upgrade. Never edit sites-platform-state manually.

The routing backend rejects ambiguous ownership: taking over a route labelled for a different site raises an error, and the operator retries after 60 s rather than deleting the other route (LIFE-07). The same 60 s deferral also protects a new site whose name already has a foreign route (site_handler.__has_route before the create flow).

Three gotchas for step 3 and step 6:

• Gateway mode needs a non-empty routing.host before any site update: the operator fails the update with a kopf.TemporaryError ("Public host name required") when it builds the site files ConfigMap.
• The chart renders gateway mode only when the cluster serves the gateway.networking.k8s.io/v1 HTTPRoute CRD (render-time check; nothing checks later removal).
• Routing values live in the sites-config ConfigMap, whose hash is an annotation on the Hub and operator pod templates. A routing change rolls those pods; that is intentional and needs no extra restart.

Read-only state¶

Read-only mode is a routing-adjacent platform switch stored under the read-only key of the sites-platform-state ConfigMap. Helm does not manage that ConfigMap, so the state survives upgrades and mode switches (core/.../kubernetes/platform_state.py; concurrent writes retry up to five times on a 409 conflict, and the API contract is in 12-hub-api).

State resolution (SitesDAO.is_system_read_only): the ConfigMap value wins; without one, ingress mode falls back to the presence of the legacy snippet on the master Ingress, and gateway mode reports false. The toggle writes the ConfigMap and re-syncs the snippet; the Hub startup runs the same sync in a background thread, because a helm upgrade can recreate the master Ingress without the snippet, and imports the legacy state once.

The NGINX snippet is the ingress-mode enforcement point. Its exact logic: methods outside GET|OPTIONS|HEAD return 405 /hub/error/page/405/, except the exact URI /hub/api/v2/read-only/ and any path under /hub/list/ — the same allow-list the gateway-mode Hub middleware applies (the middleware behaviour, including the 5-second per-replica state cache and its immediate clear after a toggle through that Hub, is owned by 12-hub-api). Site-level read-only enforcement does not exist; the switch is for the Hub API surface.

Behaviour requirements¶