Skip to content

Operations

Use Kubernetes health, rollout, log, and storage checks alongside the Hub's health and dependency status. Keep operational procedures for the cluster, storage, registry, DNS, and identity provider with the teams that own them.

Daily checks

kubectl -n sites-stage get deployments,pods,cronjobs,pvc
kubectl -n sites-stage get events --sort-by=.lastTimestamp
kubectl -n sites-stage logs deployment/sites-hub --tail=100
kubectl -n sites-stage logs deployment/sites-operator --tail=100

Check that:

  • Hub and Operator pods are ready.
  • Redis is ready and reachable by the platform.
  • The data persistent volume claim (PVC) is bound and mounted where required.
  • Scheduled jobs complete within their expected time.
  • The public Hub and a representative site are reachable.
  • The Hub dependency page reports expected status for configured services.

Use your cluster's logs and metrics system for longer history. Avoid relying only on in-pod logs because Pods can be replaced.

Scheduled jobs and data

The chart can create analytics, backup, quota, and retention CronJobs. Schedules use standard five-field cron syntax and the Kubernetes controller's time zone. suspend: true prevents new jobs from starting.

The platform backup job exports Site custom resources to the shared /sites-data volume. It does not back up website content stored on each site's PVC. Use separate backup and restore procedures for:

  • Site resources and platform configuration
  • website content volumes
  • chart values and external Secrets

Test restores; use Restore site definitions for the procedure. Check that the data volume has enough space and that the backup retention meets the site's recovery requirements.

A platform administrator can run a backup outside the schedule, for example before a maintenance window. POST /hub/api/v2/backup/ runs a fresh export and returns the file; GET /hub/api/v2/backup/ returns the current file without a new export. Store downloaded files according to your secret and data handling policy; they contain every Site definition. The ${ACCESS_TOKEN} value in this guide's examples is the platform administration token; see Platform administration token.

curl --fail-with-body -X POST \
  -H "Authorization: ******" \
  -o /secure/path/sites-backup.yaml \
  "https://<domain>/hub/api/v2/backup/"

The quota job checks storage use and sends notifications at the configured threshold. The retention job can archive and delete sites according to their retention dates. Confirm the configured notification providers and administrator assignee before enabling these actions.

Restore site definitions

The backup job writes one file per day to /sites-data/<namespace>/backups/YYYYMMDD.yaml on the platform data volume. A second run on the same day replaces the file. The file contains Site resources only, without status or resource version fields. The job deletes backup files older than jobs.backup_keep_files. When an export fails and jobs.backup_notifications_email is configured, the job sends a failure email and the run fails.

The backups live on the platform data PVC. If you lose that PVC, you lose the backups with it. Include the data PVC in your storage-system backup.

To restore Site definitions:

  1. Open the Hub administration page /hub/admin/backup-restore/. You need platform administrator access. The list shows the newest backup file first.
  2. Select the Sites to restore, or use full restore to restore every resource in the selected file.
  3. Confirm that each restored Site appears in the Hub list and that the Operator creates its resources. On success, the restore clears the Site caches and the administration messages.

A restore does not copy site content. When the PersistentVolume of the deleted Site still exists, the Operator binds the new claim to that volume and the Site recovers its content. When the volume was deleted, restore the content from your storage backup. See deletion and recovery boundaries.

Test a restore on a non-production platform before you need it.

Reclaim storage after site deletion

Deleting a Site resource removes its namespaced objects: the Secret, both ConfigMaps, the PVC, the StatefulSet, the Service, and the route. Kubernetes garbage collection does this through owner references; the Operator runs no delete code. The backing PersistentVolume keeps the Retain policy of the StorageClass and becomes Released. Its data survives, so a new Site with the same space and name re-binds the volume and recovers the content.

Released volumes accumulate and consume cluster storage. To clean them up:

  1. Open the Hub administration page /hub/admin/storage-management/. It lists the Released PersistentVolumes of the namespace that have the label sites.ecmwf.int/app: site. The admission policy sites-hub-volume-deletion lets the Hub delete only these volumes. Delete other volumes with kubectl as a cluster administrator.
  2. Confirm the Site owners and the retention rules. Deleting a released volume destroys its content permanently.
  3. Delete the selected volumes. Each deletion writes an audit entry.

Site deletion does not remove every platform-side file. The analytics report files under /sites-data/<namespace>/analytics/<site-id>/ and the ad-hoc Site creation templates remain on the data volume. Remove them only through your approved storage retention process.

Maintenance windows and read-only mode

Use platform read-only mode to block write operations during a maintenance window. Read operations continue to work. Enable or disable the mode with the action on the Hub administration index page, or with the API. The same request toggles the state in both directions:

curl --fail-with-body -X POST \
  -H "Authorization: ******" \
  "https://<domain>/hub/api/v2/read-only/"

The response reports the new state. The state is stored in the read-only key of the sites-platform-state ConfigMap. Helm does not manage that ConfigMap, so the state survives upgrades and restarts. Check the state with:

kubectl -n sites-stage get configmap sites-platform-state -o yaml

Do not edit the ConfigMap by hand. In read-only mode, write requests to the Hub answer with an HTTP 405 error page. The read-only toggle endpoint stays reachable, so you can always disable the mode. Whether site content writes are also blocked depends on the routing mode; see read-only state. Do not treat the mode as a storage freeze; stop site application work separately when you maintain shared storage.

Use this procedure for a planned window:

  1. Announce the window. Use the administration messages page (/hub/admin/messages/) to show a global notice to users.
  2. Enable read-only mode shortly before the window starts.
  3. Perform the platform work. Confirm the mode is still enabled after a Helm upgrade or workload restart, because the state persists.
  4. Disable read-only mode and confirm normal writes.
  5. Delete the notice when the window ends.

Manage Site creation templates

Site creation templates fill in the Hub create-Site form. The template sources, the siteTemplates chart value, and the file format are in Site creation templates.

Change platform templates

Platform templates come from the Helm value siteTemplates. Change them in your environment values file and upgrade the release. The Hub reads the changed files after the kubelet sync period, without a restart. See Apply changes.

Manage ad-hoc templates

Manage ad-hoc templates through the Hub API:

  • GET /hub/api/v2/sites/templates/ lists the template names (staff).
  • GET /hub/api/v2/sites/templates/<name>/ returns one template as YAML. The query parameter asAttachment=true downloads the file (staff).
  • PUT /hub/api/v2/sites/templates/<name>/ creates or replaces a template from the request body (platform administrator).
  • DELETE /hub/api/v2/sites/templates/<name>/ removes a template (platform administrator).

For example:

curl --fail-with-body -X PUT \
  -H "Authorization: ******" \
  --data-binary @/secure/path/data-exchange.yaml \
  "https://<domain>/hub/api/v2/sites/templates/data-exchange/"

A PUT or DELETE on the name of a platform template answers 403. Change the Helm value instead.

Ad-hoc templates are not part of the platform Site backup. Include the data volume in your storage-system backup, as described in Scheduled jobs and data.

Remove a shadowed ad-hoc template

A platform template shadows an ad-hoc template with the same name. The Hub writes a warning to its log once per Hub process and keeps the ad-hoc file. The Hub API answers 403 to a DELETE on that name. To remove the ad-hoc file, do one of these:

  • Remove the platform template from siteTemplates, upgrade the release, then delete the ad-hoc template through the Hub API.
  • Delete the file from the data volume, for example kubectl -n <namespace> exec deploy/sites-hub -- rm /sites-data/<namespace>/templates/<name>.yaml.

Clearing caches and restarting

Use this procedure when a configuration change must reach the caches or be re-read at startup:

  1. Apply the change with Helm. A changed sites-config ConfigMap checksum restarts the Hub and the Operator automatically.
  2. Clear the platform caches with the Clear cache action on the Hub administration index page, or with DELETE /hub/api/v2/cache/ and a platform administrator token. This clears the Site, specification, and statistics caches, and refreshes the directory cache.
  3. Restart a workload that reads a setting only at startup. For example, the Hub and the Operator read migration.enabled at startup; see configuration. Restart the Operator with kubectl -n sites-stage rollout restart deployment/sites-operator; the procedure in Check Operator reconciliation confirms this is safe.
  4. Reconcile existing sites that must pick up the change with Force Upgrade All Sites; see upgrades and rollback.

Configuration and branding changes

Use Helm to change chart values and application settings. Do not edit sites-config by hand when Helm owns it. A configuration checksum restarts the Hub and Operator when the ConfigMap changes. Some exported settings need site reconciliation before existing sites use them. Use Force Upgrade All Sites to regenerate site resources when application versions are unchanged. See upgrades and rollback.

The Hub also provides an in-band settings editor, /hub/admin/hub-settings/ (platform administrator). It edits the live sites-config ConfigMap: administrator, staff, access, and view users and groups, the Site applications version, and the platform master token in HUB_MASTER_TOKEN. Saving restarts the Hub Deployment. These edits live only in the cluster: the next helm upgrade overwrites them with the chart values. Use the editor for urgent changes, and record every edit in your values file so the next upgrade reproduces it.

For branding, trusted domains, OpenID Connect (OIDC) endpoints, and routing changes, follow the specific procedure in the related guide or the chart README, and use Clearing caches and restarting where a change must reach the caches or be re-read at startup. Confirm the effect on existing sites before applying platform-wide changes.

Scaling and resources

Set resource requests and limits for the Hub, Operator, Redis, and initializer based on measured use. The chart provides a Hub Horizontal Pod Autoscaler (HPA) and a namespace ResourceQuota. Confirm that:

  • the cluster has capacity within the quota
  • the metrics API is available when the HPA is enabled
  • hub.hpa.minReplicas and hub.hpa.maxReplicas match the service capacity
  • Hub concurrency and replicas are suitable for the application and session behaviour

Do not scale site replicas above one in Gateway mode unless your Gateway provides session affinity.

Monitoring

The chart can create ServiceMonitor resources for the Hub and site metrics. Install a Prometheus Operator and configure its selector and namespace selection before enabling serviceMonitor.enabled. Site metrics also depend on sitesConfig.integrations.metrics and the network policy paths.

The Hub exposes platform health and dependency status. The Hub API specification is at /hub/api/v2/spec/, with the OpenAPI document at /hub/api/v2/spec/swagger.json. Use them and the configured trusted IP ranges to monitor those endpoints. Avoid exposing internal health or metrics endpoints to untrusted networks.

The Hub writes an AUDIT log entry for administration and API actions: create, update, delete, access, restart, upgrade, archive, export, and backup. Entries use the form username=<user>;action=<action>;resource=<detail>; access entries add the method, endpoint, sanitized form data, remote address, and referrer. Secret-shaped values are masked before anything is written. Requests made with the platform administration token appear as user api-administration-token. The entries go to the Hub container log; send that log to your log system and retain it per your policy.

Maintenance records

Keep a record of the deployed chart version, application image tag, values revision, Custom Resource Definition (CRD) version, and external Secret references. Do not record Secret values. Use the procedure in upgrades and rollback for release changes.