Troubleshooting¶

Start with Kubernetes events and workload status. Check application logs before changing configuration.

kubectl -n sites-stage get pods
kubectl -n sites-stage get events --sort-by=.lastTimestamp
kubectl -n sites-stage logs deployment/sites-hub --tail=200
kubectl -n sites-stage logs deployment/sites-operator --tail=200

Symptom	Checks and actions
Hub or Operator pod does not start	Check pod events, image pull errors, Secret names, required Secret keys, ConfigMap parsing, resource requests, and ServiceAccount permissions.
Hub pod stays in `Init:0/1`	The `sites-hub-initializer` container creates the `backups/` and `analytics/` directories under `/sites-data/<namespace>`, then resolves the OpenID Connect (OIDC) endpoints and writes the client secrets file `secrets.json` there. Read its log with `kubectl -n sites-stage logs <hub-pod> -c sites-hub-initializer`. A failure usually means the data volume is not mounted or not writable, or the OIDC discovery endpoint is unreachable.
`ImagePullBackOff`	Check image tag, registry, node architecture, and node access to the registry. If the configured repository requires authentication, check `serviceAccount.imagePullSecretName` and confirm the pull Secret exists in the release namespace. For a Site pod, check `sitesConfig.kubernetes.private_registry_image_pull_secrets_name`, and for a custom application image also `sitesConfig.kubernetes.custom_application_image_pull_secrets_name`. Site pods do not use `serviceAccount.imagePullSecretName`.
persistent volume claim (PVC) stays pending	Check the StorageClass name, access mode, capacity, Container Storage Interface (CSI) provisioner, and available storage. Check both the platform data PVC and the Site PVC.
Hub is ready but cannot be reached	Check public DNS, Ingress or HTTPRoute status, the controller namespace, listener/host match, certificate, and firewall rules.
OIDC login fails	Check issuer URL, callback URL registration, certificate validation, provider discovery, and `oidc-secrets`. Confirm the client ID and secret keys are present.
Operator cannot create Site resources	Check the Operator ServiceAccount and chart role-based access control (RBAC). Confirm the Site Custom Resource Definition (CRD) is installed and its schema matches the application version.
A site is not available after an upgrade	Check the Site status, Operator logs, pod readiness, and route status. Use Force Upgrade All Sites to regenerate resources when application versions are unchanged; see upgrades and rollback.
Gateway HTTPRoute is not accepted	Check the Gateway listener, allowed route namespaces, `parentRefs`, `status.parents` conditions, DNS, certificate, and `networkPolicy.gatewayNamespace`.
Redis cannot be reached	Check the Redis Service and endpoints, pod readiness, NetworkPolicy selectors, and application cache host and port.
Scheduled jobs fail	Check the CronJob schedule, job and pod events, `sites-jobs` permissions, Secrets, data PVC mounts, network access, and logs.
Monitoring has no metrics	Check the provider setting, ServiceMonitor enablement, Prometheus selectors, namespace selection, trusted IPs, and NetworkPolicy access.
Site status stays stale, or a Site is never reconciled	The Operator has no liveness probe, so a stalled Operator is not restarted automatically. Check the Operator logs and the Site resource as described in Check Operator reconciliation. Restart the deployment if the logs stop.
A Site creation repeats every minute	A create step failed; the Operator retries the whole sequence after 60 seconds with no attempt limit. Read the `[CREATE]` error lines. A route owned by another Site raises a conflict and also retries after 60 seconds; the Operator never overwrites a foreign route.
An update stops after a few retries	Operator update retries are limited by `sitesConfig.operator.retries` (default 5). After the limit the Operator waits for the next qualifying change. Use Force Upgrade Site to trigger the full update path again.
The PVC is bound but the persistent volume (PV) has no Site labels	The Operator labels the PV in a background task 20-40 seconds after creation, or 5 seconds after an update. A `[PATCH] PV ... not found` line means the claim was not bound yet; the labels are applied on the next qualifying change.
Users cannot save; writes answer HTTP 405	The platform is in read-only mode. Check the badge on the Hub administration index page or the `sites-platform-state` ConfigMap; see Maintenance windows and read-only mode.

Check Operator reconciliation¶

When a Site does not change as expected:

Read the Site resource. The Operator writes pod conditions and per-container problems into its status, and records handler progress in its annotations:

kubectl -n sites-stage get site <site-id> -o yaml

Filter the Operator logs by their prefixes: [CREATE], [UPDATE], [EVENT], [RESUME], and [PATCH]. The line already exists... This might be a RESUME event is normal during replay after an Operator restart.
Compare the behaviour with site lifecycle and reconciliation.
To restart the Operator, use:

kubectl -n sites-stage rollout restart deployment/sites-operator

This is safe. On start, the Operator replays only incomplete create handlers, and every create step tolerates an object that already exists.

Render and compare¶

Render the chart with the same values used for the release:

helm template sites oci://eccr.ecmwf.int/sites-oss/ecmwf-sites \
  --version "2.7.1" \
  -f /secure/path/values.yaml \
  -n sites-stage

Replace 2.7.1 with the deployed chart version. Do not share rendered output if it contains credentials or other private configuration. Review configuration warnings in the Hub dependency page and application logs. For known platform failure modes, see the observability and triage guide.

Escalate¶

When you ask for help, provide the chart and image versions, relevant resource status, sanitized events, and sanitized logs. Remove tokens, passwords, user data, and private endpoints before sharing diagnostics.