Skip to content

Upgrades and rollback

Plan upgrades for the platform, Site Custom Resource Definition (CRD), and site applications as related changes. Review release notes and test the upgrade with a non-production release that uses representative storage, routing, and integrations.

Release channels

The ecmwf-sites repository publishes two channels:

  • Releases: images tagged <VERSION> and the matching SemVer chart, from the Release workflow. Use these in production.
  • Develop: images tagged develop and a <VERSION>-develop pre-release chart, published automatically after CI passes on the develop branch (CI/CD). Install the pre-release chart in test environments with --version <VERSION>-develop and images.tag: develop. The develop channel moves without notice and is not a supported release.

Before an upgrade

  1. Confirm the target chart and image versions. ECMWF Sites uses lockstep application versions; use the release's supported image tag.
  2. Save the current Helm values and chart version in a protected location. Do not save plaintext Secrets.
  3. Confirm recent backups and test the required restore path.
  4. Review chart and application changes, especially CRD schema, routing, storage, configuration, and required Secret keys.
  5. Render and inspect the proposed resources:
CHART_VERSION="2.7.1" # Replace with the release you have approved.
helm template sites oci://eccr.ecmwf.int/sites-oss/ecmwf-sites \
  --version "${CHART_VERSION}" \
  -f /secure/path/values.yaml -n sites-prod
  1. Schedule a maintenance window if users or site resources may be affected.

Apply CRD changes

Helm installs the Site CRD on the first install, but does not upgrade it automatically. If the release changes the CRD, apply the CRD from the target published chart before upgrading the Hub or Operator:

helm show crds oci://eccr.ecmwf.int/sites-oss/ecmwf-sites \
  --version "${CHART_VERSION}" | kubectl apply -f -

Check the resulting CRD and any validation warnings. Do not remove the CRD to force an update; it may contain definitions used by existing Site resources.

Upgrade the Helm release

If you set routing.hubHost for the first time, do the steps in Separate Hub host first. The Hub session cookie is renamed in this release, so all users log in again.

helm upgrade sites oci://eccr.ecmwf.int/sites-oss/ecmwf-sites \
  --version "${CHART_VERSION}" \
  -f /secure/path/values.yaml \
  -n sites-prod

Use --install only when the release may not exist:

helm upgrade --install sites oci://eccr.ecmwf.int/sites-oss/ecmwf-sites \
  --version "${CHART_VERSION}" \
  -f /secure/path/values.yaml \
  -n sites-prod --create-namespace

Wait for the Hub and Operator rollouts:

kubectl -n sites-prod rollout status deployment/sites-hub
kubectl -n sites-prod rollout status deployment/sites-operator
kubectl -n sites-prod get pods

Check the logs, Hub health, OpenID Connect (OIDC) login, and a test Site resource. Then confirm scheduled jobs, metrics, and site storage.

Upgrade sites

An upgrade to Hub and Operator does not always change existing websites. If the release includes new site application versions or changes to generated site configuration, use the Hub's Upgrade All Sites action or the administrator API. Test representative sites before upgrading all sites.

If site application versions are unchanged, an ordinary upgrade can do no work. Use Force Upgrade All Sites on the Hub administration page to apply the current platform configuration to every site. This action requires a platform administrator. The administrator API supports the same option. Use the platform administration token as ${ACCESS_TOKEN}; see Platform administration token:

curl --fail-with-body -X POST \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  "https://<domain>/hub/api/v2/sites/upgrade/?force=true"

The force query parameter accepts true or false and defaults to false. Other values and repeated force parameters return HTTP 400. The v1 bulk upgrade endpoint supports the same option. The response confirms that work is queued, not that all updates have finished. The Hub logs each failed site upgrade and one summary line when all queued upgrades are complete, for example Bulk forced upgrade complete: 41 succeeded, 1 failed (space/site).

Only one bulk upgrade, ordinary or forced, can run at a time. While a bulk upgrade runs, a new bulk upgrade request returns HTTP 409. The lock is released when all queued upgrades are complete. If the Hub stops before then, the lock expires after 30 minutes. With Redis, the lock applies to all Hub replicas. Without Redis, it applies only to one Hub process.

To test on one site before you upgrade all sites, force a single site. Use Force Upgrade Site in the Actions menu on the site page, or the API. This also requires a platform administrator:

curl --fail-with-body -X POST \
  -H "Authorization: ******" \
  "https://<domain>/hub/api/v2/sites/<space>/<site>/upgrade/?force=true"

A forced upgrade updates built-in application images and regenerates site resources, including configuration and routes, even when image versions have not changed. It preserves custom images, site settings, and data. It includes archived sites but keeps them archived with zero replicas. Active sites may be unavailable briefly while their pods restart. Restart all sites only restarts pods; it does not regenerate resources.

Deploy the updated Operator before using the force option in the Hub. An older Operator does not act on the upgrade request annotation. No CRD schema change is needed for this option.

A site Service gets publishNotReadyAddresses: true at its next Site update or forced upgrade. From then on, a site pod that is not Ready (for example, a custom application that does not accept connections on port 8000) stays in the routing, and its other containers stay reachable. See Routing.

Monitor Operator logs and site rollout state. Check route status and application health after the update.

Switch routing modes

Switching between Ingress and Gateway API affects all sites. First install a release that supports the target routing mode while keeping the current mode. Then, in a maintenance window:

  1. Set the new routing mode.
  2. Upgrade the Helm release.
  3. Confirm which external address serves the new entry point. If it differs from the current one, point the public DNS record at the new address (integration or manual). If the same address serves both modes, leave the record unchanged.
  4. Confirm the Hub is available.
  5. Use Force Upgrade All Sites so the Operator creates routes of the new type, including sites already at the current application versions.
  6. Confirm all sites are available and routes are accepted.

For routing prerequisites and behavior, see routing and networking.

Rollback

For a failed Helm change, restore the previous chart and values revision:

helm history sites -n sites-prod
helm rollback sites <revision> -n sites-prod

Then check the Hub and Operator rollout, routes, and representative sites. Helm rollback does not undo CRD changes, storage changes, external Secret changes, or every change already applied to a Site resource. Plan those separately and restore data only through the approved recovery process.

Keep the old routing entry point available until the Hub and all required sites pass validation. For a mode switch, sites already updated may need another site update to return to the earlier route type. Use Force Upgrade All Sites after the rollback to apply the earlier routing configuration even when application versions are unchanged.