Skip to content

ECMWF Sites — design and development

This section documents the platform for people who build and change it: contributors, integrators, and platform engineers. It describes the implemented architecture, the behaviour of each component, and the requirements the code meets today.

For task guides, use the other sections:

Document map

Document Covers
01-overview.md Repository layout, workspace and test entry points, built image set, lockstep versioning checks, release flow, endpoint anatomy
02-architecture.md System context, per-Site runtime and loopback model, chart object inventory, role-based access control (RBAC) model with exact Role names and verbs, data stores, shared core
03-domain-model.md Site Custom Resource Definition (CRD) fields and constraints, identity and label hashing, applications registry and selection rules, resource tiers, derived objects, SiteBuilder paths, status
04-site-applications/ Site applications. Index: registry attribute and method contract, application matrix with Service and route exposure, shared container contract, image and selection rules. Sub-pages: built-in applications (sets, Service behaviour, resource tiers, build metadata), one page for each application (initializer, OAuth2, proxy, API, admin, WebDAV, S3), custom applications, content mirror
05-lifecycle-and-reconciliation.md Operator handlers and configuration, create/update/delete flows, site status monitoring, disable, upgrade, retention actions
06-security.md Authentication surfaces, token types, .htpasswd enforcement, security and identity headers, OpenID Connect (OIDC) discovery, introspection, and logout
07-site-api/ Go Site API. Index: endpoints and boundary. Sub-pages: files, uploads, and integrity; tokens and shared URLs; server, errors, and health
08-hub-api-and-administration/ Hub API. Index: application assembly, route groups, tests. Sub-pages: authentication and authorization; site actions, credentials, and requests; platform and operations endpoints
09-data-storage-and-recovery.md Store inventory, site and platform volume layouts, signed Redis cache and key space, platform state, Site backups, token history, persistent volume claim (PVC) deletion and persistent volume (PV) re-binding, recovery boundaries
10-routing.md Routing modes, route objects per mode, route ownership, policy providers, mode switch design, read-only state storage
11-quotas-backups-retention.md Quota enforcement, the notification crawler framework, backup/quota/retention/analytics jobs and their defaults and failure semantics
12-platform-configuration-and-branding.md Configuration pipeline and load order, secret injection, upgrade guards and cross-field validators, consistency checks, branding schema, SVG validation and delivery
13-integrations.md Provider registry and probe/status semantics, degradation contract, directory settings, request flows, notification dispatch and templates, analytics backends
14-local-development.md Local cluster deployment from a checkout: runtime detection, image build/load/verify, TLS, post-renderer, local branding assets
15-ci-cd/ GitHub Actions and the Makefile. Index: goals, runners, workflow map, permissions, registry. Sub-pages: Makefile targets; continuous integration; releases; security scanning; documentation site; local testing
16-testing.md Test layers and how they verify the requirements in this section
17-observability-and-triage.md Health endpoints and payloads, pod probes, Site status fields, logs and audit formats, metrics exposition and scraping, cross-layer triage
18-hub-web-interface/ Hub browser UI. Index: blueprints, routes, page shell, JavaScript layer. Sub-pages: site lists and creation; the Site view; administration pages
19-contributing.md Contributor path: environment, licence and SPDX headers, change and pull-request workflow, documentation rules, reporting problems
20-alternatives.md External products and approaches compared with this platform: what each one is, what it is not, its strengths, and where it stops

Documentation conventions

Implemented behaviour only. Every statement describes the code in this repository. The VERSION file states the release these documents describe (lockstep versioning, see 01-overview). Do not document planned features here. The one exception is 24-alternatives: it describes external products for context, not implemented behaviour.

Requirement tables. Each design document ends with a behaviour requirements table, except 24-alternatives, which verifies nothing in this repository; a document split into a directory keeps one table in its index.md for the whole set. Each requirement has:

  • A stable ID: a document prefix and a number, for example API-03.
  • One behaviour statement in STE100 Simplified Technical English.
  • Implemented in: the code that provides the behaviour.
  • Verified by: the test that checks it, or — when no automated test covers it. 16-testing.md lists which tests run in CI.

Requirement IDs do not change or get reused. When behaviour is removed, its requirement is removed with it.

Page size. A page that grows past roughly 350 lines is split into a directory with an index.md and named sub-pages; the parent URL is kept. Every page must be listed in the nav of utilities/release/docs/zensical.toml, and every internal link and anchor must resolve: make build-docs checks navigation coverage first (check-docs-nav), then fails on broken links and anchors.

Diagrams. Diagrams are Mermaid source rendered by the documentation site. Keep them in sync with the code they describe.

API reference. The OpenAPI specifications are the complete endpoint reference:

  • hub/src/ecmwf/sites/hub/static/specs/hub-api-spec.yaml
  • hub/src/ecmwf/sites/hub/static/specs/site-api-spec.yaml

hub/tests/test_api_spec_coverage.py checks both specifications against the route modules (hub/src/ecmwf/sites/hub/app/api/, site/.../applications/api/main.go). When you change a route, update the specification in the same pull request.

Where to look when changing behaviour

When you change any of these, update this section in the same pull request:

Change Update
Site CRD or model (core/src/ecmwf/sites/core/model/, helm/ecmwf-sites/crds/) domain model, lifecycle
Site application configuration or image (site/src/ecmwf/sites/website/applications/, Makefile) site applications, built-in applications, site proxy, custom applications, content mirror, architecture, domain model, lifecycle
Operator handlers lifecycle
Authentication, headers, .htpasswd security
Hub API Hub API, domain model, lifecycle, security
Hub UI templates, wizard, and JavaScript behaviour Hub web interface, configuration and branding, Hub API
Go Site API Site API, data storage
Lua proxy or NGINX templates site proxy, security, Site API, routing
Routing code or chart routing values routing
Job defaults or cron schedules quotas, backups, retention, and jobs
Redis, backups, token history, platform state, or PVC recovery data storage, architecture, quotas, backups, retention, and jobs, observability
Configuration schema, consistency rules, branding configuration and branding, integrations
Deploy script or Makefile local development, CI/CD
Workflows, actions, or tests CI/CD, testing
Health endpoints or metrics observability