Site lifecycle and reconciliation¶

This page describes exactly what the operator does during the life of a site. The Site custom resource is the intent; the operator turns it into Kubernetes objects and reports pod state back into it. All handlers are in operator/src/ecmwf/sites/operator/site_handler.py. The image starts the process with kopf run --standalone --namespace "$NAMESPACE" --module ecmwf.sites.operator.site_handler (operator/Dockerfile), so the operator watches only the platform namespace. Module import loads ecmwf.sites.website.applications, which registers the built-in application classes the model needs.

Reconciler configuration¶

The startup handler (configure) and the chart set these values:

On resume (operator restart) the handler only logs (at debug) and records a trivial result. Progress annotations make kopf replay handlers not marked complete — in practice the create sequence, because update retries are capped; every create step tolerates HTTP 409, so replaying create is the recovery path. There is no separate converge loop and no periodic resync.

What triggers work¶

The operator acts on a Site update only when the kopf diff touches a restart-triggering property (operator/src/ecmwf/sites/operator/kopf_diff.py checks these prefixes):

• spec.configuration.applications, .enabled, .headers, .quota, .retention_date, .share, .settings.resource_limits, .settings.error_handling, .space
• spec.name, spec.owner, spec.type

The match is a substring test against the diff path, so a change to a deeper field such as spec.configuration.share.administrator_users triggers on the registered prefix spec.configuration.share. A code comment warns that a new prefix must not be a substring of an unrelated property path.

An added or changed, non-empty metadata.annotations["sites.ecmwf.int/upgrade-request"] also triggers the full update path. Removing the annotation does not trigger work. Unrelated annotations do not trigger work.

The comparison is order-insensitive: lists of application dicts are sorted by their name field before comparison, so a reorder without content change causes no work. There is no version comparison in the operator: any qualifying change re-renders every object and force-restarts the pods. A change to audit fields (description, updated_by, dates) causes no work.

Create flow¶

sequenceDiagram
    participant U as Admin
    participant H as Hub
    participant K as Kubernetes API
    participant O as Operator
    U->>H: Create site (UI or Hub API)
    H->>K: Create Site CR (id, owner, apps, quota...)
    K-->>O: on.create(Site)
    O->>O: Build model from spec
    O->>K: Secret, 2 ConfigMaps, PVC, StatefulSet, Service, Route (in order)
    Note over O: each step tolerates 409. A route owned by another site raises TemporaryError (retry in 60 s)
    O-->>K: background, after 20-40 s: label the bound PV
    K-->>O: on.field(Pod, status.conditions)
    O->>K: patch Site status

Existence gate. If both Secrets (<id> and <id>-oauth2), both ConfigMaps, PVC, StatefulSet, Service all exist and the route check passes, the create handler logs "already exists" and finishes without creating anything and without starting the persistent volume (PV) patch task. The route check treats a site with no public path as satisfied; a route of this site's name owned by another site raises TemporaryError (retry in 60 s) — the operator never overwrites a foreign route (14-route-ownership).

Steps. Otherwise the full sequence runs inside kopf.execute, in this order. Each step is idempotent: HTTP 409 logs "already exists... might be a RESUME event" and continues; any other API error aborts the sequence and retries it from the start after 60 s.

Background PV patch. After the sequence, an un-awaited task sleeps a random 20–40 s (jitter lets the PVC bind, and spreads the load when many sites are created at once), then labels the bound PV with the site labels. The PV is found by claim → volume name, then by site labels, then by the old claim names; a not-yet-bound volume logs "not found" and is skipped. A failure in this background task is logged by a done-callback — nothing else awaits it. The same patch runs after updates, with a fixed 5 s delay.

Update flow¶

On a qualifying update (see "What triggers work") the operator runs the update wrapper, in order:

1. update_secret, update_oauth2_secret, update_config_map — both ConfigMaps re-render from scratch (fresh host, branding, OpenID Connect (OIDC) endpoints, nginx configuration). update_oauth2_secret creates <id>-oauth2 when it does not exist (a site from an earlier release), else writes the cookie secret again from the current session.secret_key.
2. update_volume_claim — refreshes the claim labels and annotations; the storage request is the only spec change, and it applies only when kubernetes.storage_expansion_supported is true.
3. update_stateful_set(force_restart=True) — sets the kubectl.kubernetes.io/restartedAt pod-template annotation to the current UTC time, so every qualifying change rolls the pods. For an enabled site, it then deletes each site pod that is not Running and Ready and is not terminating (replace_not_ready_pods). The StatefulSet uses the default OrderedReady pod management policy. With this policy, the controller does not replace a pod that never becomes ready: it waits for that pod and stops the rollout (the Kubernetes "forced rollback" limitation). For example, a custom application that does not accept connections on port 8000 fails its readiness check. If the user then removes or changes the image, the old pod stays and the Site status stays not running. The deletion removes this wait, and the controller creates the pod again from the updated template. A failure to list or delete pods is logged as a warning and does not stop the update.
4. update_service, update_routes — the route is rebuilt for the active routing mode, and any route of the other mode owned by this site (sites.ecmwf.int/id label) is deleted (14-route-ownership).
5. A background task re-patches the PV label after 5 s.

Implemented details:

• Self-healing. Every update step first reads the object; a missing object (for example deleted by hand) is recreated through the create function instead of failing.
• Owner references are cleared and re-appended on every updated object. A code comment explains why: kopf.adopt does not refresh the apiVersion of existing references, and UID-based removal fails after a site restore gives the Site a new UID. The code clears all references and appends the current one — safe because kopf is the only controller that owns these objects.
• Route update: an API error on the route read is treated as "missing" (recreate); a route owned by another site logs an error and raises TemporaryError (retry in 60 s); a site that lost its public path has its route deleted. The replacement reuses the existing object, then sync_site_policies reconciles the gateway policies (14-routing).
• Other-mode cleanup deletes only routes owned by this site. A foreign resource of the other mode stays: the routing backend checks ownership before deleting and returns silently when it does not own the object; the handler additionally keeps and warns if a backend raises RouteOwnershipError. A delete API error other than 404 raises TemporaryError (retry in 30 s).
• If no diff qualifies, the handler returns without touching anything.

Site status monitoring¶

The operator watches pods, not just Sites. Every pod handler acts only on pods that carry the sites.ecmwf.int/id label and the sites.ecmwf.int/operator_bound annotation. Pods inherit both from the StatefulSet pod template, so the operator reconciles only pods it created. No pod handler returns a result: kopf writes handler results into the watched object's status, and the operator must not write status fields on pods.

• status.conditions or status.containerStatuses change: the handler finds the Site from the pod labels, maps the pod status with resolve_pod_status, and patches the Site's status subresource. The container-statuses watcher exists because the conditions stop changing while a container keeps failing its readiness check: a container that starts and never becomes ready leaves Ready and ContainersReady False from the first kubelet update, and the readiness flag moves only in status.containerStatuses. Without that watcher the stored status would keep the state the last condition flip computed. After patching, the handler deletes the cached site entries (13-key-space): a pod's phase does not change when a probe starts or stops failing, so the pages cannot rely on the phase watcher alone to clear them. If the Site was deleted meanwhile the handler logs a warning and returns. Each write logs one INFO line: the pod, the field that triggered it, the resolved phase, and the container problems now stored on the Site. Cost of the two watchers: kopf keeps the watched fields in the kopf.zalando.org/last-handled-configuration annotation of each site pod, so it patches that annotation when status.containerStatuses changes. When one pod event changes both fields, both handlers run, each with one Site read, one status patch, one INFO line, and one cache delete (a Redis SCAN on the site key pattern). After an operator upgrade that adds a watched field, the first event of each pod runs the handler once.
• status.phase change: the handler logs the new phase, and when the phase is not Running it clears the site's status-cache entries so the Hub and health views fetch fresh data (13-key-space).

resolve_pod_status (core/.../kubernetes/utils.py) builds the Site status:

• Conditions are ordered by lastTransitionTime; the first failed one wins, otherwise the last successful one.
• When the winning condition is not successful, the container statuses are scanned for waiting or terminated entries, and for running containers that report ready: false (recorded as kind running, reason NotReady). Any problem stores the phase as Pending plus a plain list of per-container problems (name, kind, reason, message) in the Site status — the Hub renders it.
• A disabled site always reports phase Disabled, whatever the pod says.
• Otherwise the pod phase is used, defaulting to Pending.
• all_steps_started is true only when the phase is Running and the selected condition is successful.

Disable and archive¶

Disabling a site (Hub toggle "Archive Site", API POST /hub/api/v2/sites/<space>/<site>/toggle/) flips spec.configuration.enabled to false. It is a qualifying change, so the update flow runs. Effects:

• StatefulSet replicas become 0 (re-render sets site.replicas only for enabled sites): no pods, content unmodifiable.
• The route is kept. Requests redirect with HTTP 302 to /hub/error/page/410/ (NGINX location snippet in ingress mode, RequestRedirect filter in gateway mode).
• Data on the PVC is untouched. Restarting a disabled site is a no-op.
• The retention job "archives" a site by performing the same toggle (15-retention-job).

Upgrade flow¶

Upgrades are Hub-side. POST /hub/api/v2/sites/upgrade/ ("Upgrade All Sites", platform admin) or the per-site variant rewrites the resource:

sequenceDiagram
    participant A as Platform admin
    participant H as Hub
    participant K as Kubernetes API
    participant O as Operator
    A->>H: POST /hub/api/v2/sites/upgrade/
    H->>H: per site: rebuild model with upgrade=true (rewrite built-in images to site.version, keep custom images)
    H->>K: Patch Site CR (thread pool)
    K-->>O: on.update: applications diff
    O->>K: Re-render ConfigMaps + StatefulSet (forced restart)

The Hub replies while the work continues per site. No version marker is stored in status; the applications list itself is the record.

For a same-version update, use Force Upgrade All Sites or POST /hub/api/v2/sites/upgrade/?force=true. Force defaults to false. For each forced request, the Hub writes a new UUID to the sites.ecmwf.int/upgrade-request Site annotation in the same write as the application updates. Kopf tracks the annotation and the existing update handler regenerates all resources and restarts active site pods. The annotation is not part of the Site model. The Sites data access object (DAO) adds it to the custom object only for a forced upgrade. Site writes use JSON merge patches, so later edits do not remove it. It is not a completion marker. Each later forced request uses a different UUID. Archived sites are included and remain at zero replicas. Custom images, site settings, and data are preserved. No Custom Resource Definition (CRD) schema change is required.

Migration mirror¶

When migration.enabled is true, the platform adds site-mirror to every site. The Hub and operator must restart to load the setting, and the sites must then be upgraded so their StatefulSets include the new container. The mirror writes to the local Site PVC and has no Service, route, or source-side write operation. It runs independently of the operator reconcile handler. Disable the setting, restart the Hub and operator, and force an upgrade of all sites again. Sites whose resources already contain a site-mirror application entry keep that entry and need it removed manually. The complete cycle algorithm, deletion safeguards, retry behaviour, and rollout procedure are in content mirror.

Delete flow¶

There is no delete handler and no finalizer. Every namespaced object was created with an owner reference to the Site resource, so kubectl delete site/<id> (or the Hub API) cascades through Kubernetes garbage collection: Secret, both ConfigMaps, PVC, StatefulSet, Service, route, and gateway policy objects.

The PVC is deleted with the site. When the Site StorageClass has the Retain reclaim policy — the convention the platform recovery design assumes — the backing PersistentVolume becomes Released, and an administrator cleans it up from the Hub administration pages (the Hub ServiceAccount may list and delete released PVs). A site recreated with the same identity re-binds the retained volume (13-DAT-10).

Site state machine¶

stateDiagram-v2
    [*] --> Provisioning: create Site CR
    Provisioning --> Running: operator created objects, pods ready
    Running --> Running: qualifying change (re-render + restart)
    Running --> Archived: disable (Hub toggle, or the retention job archives the site)
    Archived --> Running: enable
    Archived --> [*]: the retention job deletes an archived site
    Running --> [*]: delete (CR removed, objects cascade)
    Archived --> [*]: delete

The archive and delete delays are configuration settings, jobs.retention_expired_archive_after_days and jobs.retention_expired_delete_after_days; 15 lists the defaults.

Monitored sites¶

spec.configuration.settings.monitored (set by Hub administrators only) does two things: pods use priority class medium-priority instead of low-priority, and the Hub health report includes the site when it is also enabled. It defines no service-level objective.

Operator logging¶

• operator.dependency_log_level (default INFO) sets the level of the kopf logger. The HTTP client loggers (urllib3, kubernetes) stay at WARNING unless this level is DEBUG. Sites loggers are unchanged.
• The startup log line reports the Python, kopf, and Sites package versions; the full installed-package list is logged at debug level.
• Update logs list the changed field paths at info level and old/new values at debug level. Values of fields whose names look like secrets are replaced with a mask, and long values are truncated to 120 characters before logging.

Behaviour requirements¶