Architecture¶

This page describes the implemented system: what runs, how the pieces connect, and what the Helm chart deploys. Detailed designs are linked from each section.

System context¶

graph LR
    Browser["Browser"]
    Client["API client / sitesctl / SDK"]
    IdP["OIDC identity provider"]
    subgraph Cluster["Kubernetes namespace"]
        subgraph HubPod["Hub + platform"]
            HUB["sites-hub"]
            OP["sites-operator"]
            REDIS["redis"]
            JOBS["CronJobs: backup, quota, retention, analytics"]
        end
        SITE["Site StatefulSet (oauth2, proxy, api, optional apps; initializer, mirror)"]
        CRD[("Site CR")]
        DATA[("/sites-data PVC")]
    end
    GW["Ingress controller or Gateway"]

    Browser --> GW --> HubPod
    Browser --> GW --> SITE
    Client --> GW --> SITE
    HUB -- "writes/updates" --> CRD
    OP -- "watches + reconciles" --> CRD
    OP -- "creates/updates" --> SITE
    HUB --- REDIS
    OP --- REDIS
    JOBS --- REDIS
    HUB --> DATA
    JOBS --> DATA
    HUB -- "OIDC" --> IdP
    SITE -- "token + session auth" --> IdP

The Hub and the operator never proxy site content. The platform is control-plane only: the Hub records intent in the Site resource, and the operator turns that intent into running objects (lifecycle and reconciliation).

Per-Site runtime¶

One Site is one StatefulSet. Its replica count is the platform setting site.replicas (default 1; 0 while the Site is disabled). The selected applications run as containers of the same Pod, so they share one network namespace, and calls between them use 127.0.0.1. All replicas mount the same content persistent volume claim (PVC), which is a single object named after the Site id (data storage and recovery). site-oauth2 is the front door for browser traffic; it passes requests to site-proxy, which routes by path to the other containers.

graph LR
    IN["Route (Ingress or HTTPRoute)"] --> O2["site-oauth2 :4180"]
    O2 --> PX["site-proxy :8080 (OpenResty + Lua)"]
    PX -->|"/s/api"| API["site-api :10000 (Go)"]
    PX -->|"/s/admin"| AD["site-admin :9000 (File Browser)"]
    PX -->|"/s/dav"| DAV["site-webdav :7000"]
    PX -->|"/s/s3"| S3["site-s3 :6000"]
    PX -->|content| APP["site-custom :8000 (optional) or local files"]
    INIT["site-initializer (init)"] --> VOL[("site PVC")]
    MIRROR["site-mirror (no listener)"] --> VOL
    PX --> VOL

Facts that matter to developers:

• The Go API has no authentication of its own: its route middleware is a pass-through, and the service performs no credential check (site/src/ecmwf/sites/website/applications/api/main.go, api/api/parse_middleware.go). Authorization comes from the proxy: the authenticated API locations run admin_auth.lua (the session-files location runs session_files_auth.lua) before proxy_pass to http://127.0.0.1:10000; the public read-only files location carries no access module by design (site/.../applications/proxy/nginx/api.conf). The API listens only on 127.0.0.1:10000 in the shared Pod network namespace, and gin trusts X-Forwarded-* headers only from the loopback peer (SetTrustedProxies(["127.0.0.1"])). The application contract, and the limits of this isolation, are in Site API and security.
• The per-site Service exposes a single port: site-oauth2 on 4180. Every other application returns no Service port and no route path (site application architecture).
• When networkPolicy.enabled is true, the site-internal-traffic policy limits ingress to Site Pods: port 4180 from the routing namespace (ingressNamespace in ingress mode, gatewayNamespace in gateway mode), and all ports from the monitoring and metrics-gateway namespaces; Pods inside the platform namespace match every rule. It does not filter traffic inside one Pod (that is the shared loopback namespace).
• When networkPolicy.enabled is true, the hub-internal-traffic policy limits ingress to the Hub Pods: port 8080 from the routing namespace and from the monitoring and metrics-gateway namespaces. It does not filter egress, because the Hub reaches Redis, the Kubernetes API, the identity provider, and the other integrations. The policy keeps the unauthenticated /hub/metrics/ route and the rest of the Hub away from other Pods (SEC-29).
• site-oauth2, site-proxy, site-api, and site-initializer are essential applications: the platform always configures them. site-mirror is platform-controlled (migration.enabled). The other applications are enabled per site through spec.configuration.applications (03-domain-model).
• site-initializer runs before the application containers. site-mirror is a migration container in the same Pod; it has no listener, Service port, or route. See content mirror.
• Every container receives the per-site Secret <id> (holds MASTER_TOKEN) and the per-site environment ConfigMap <id> through envFrom. Rendered files arrive through the files ConfigMap <id>-files, mounted as the site-config volume (shared container contract).

What the Helm chart deploys¶

One Helm release installs the whole platform into one namespace. The chart (helm/ecmwf-sites/) renders:

The sites-platform-state ConfigMap is deliberately not Helm-managed. It stores read-only platform state so it survives upgrades (core/src/ecmwf/sites/core/kubernetes/platform_state.py).

RBAC model¶

The chart creates three ServiceAccounts with fixed names.

The operator Role has no wildcard resource or verb. Gateway policy providers can add rules to the operator Role through routing.gateway.policyRbacRules. The Redis Pod mounts no ServiceAccount token. The Hub cannot read per-Site Secrets (its Secret rule is restricted by name) and cannot create workloads; only the operator ServiceAccount can.

Kopf also makes Kubernetes API calls. It patches Pod annotations for handler progress and change tracking, posts Events, and observes Namespaces and CustomResourceDefinitions — these match the ClusterRole comments and rules. These permissions are required in addition to the calls made directly by application code.

The Hub uses sites-hub, not the operator account sites. Released-volume cleanup uses the Hub ClusterRole's PersistentVolume list and delete permissions. RBAC cannot limit delete by label, so the chart also creates the ValidatingAdmissionPolicy sites-hub-volume-deletion when the cluster serves that API. It denies a Hub deletion unless the volume has the label sites.ecmwf.int/app: site, its claimRef namespace is the release namespace, and its phase is Released. The operator can read and patch volumes for site creation and recovery; it cannot delete them.

Data stores¶

The complete ownership and recovery rules for these stores are in data storage and recovery.

Each Site's content stays on its own PVC. The platform backup never copies site content.

Shared Python core¶

All four Python packages build on sites-core:

• core/.../config/: the configuration schema, loader, and consistency checks (16).
• core/.../model/: the Site model, applications registry, and the SiteExporter that renders every per-site Kubernetes object and ConfigMap (03).
• core/.../kubernetes/: the data access object (DAO) layer and the routing backends (14).
• core/.../integrations/: analytics, metrics, directory, notifications (17).
• core/.../notifications/, core/.../backup/, core/.../analytics/: the CronJob implementations (15).

The Hub, the operator, and the jobs share this code. No Site container runs Python from core: the Go API, the Lua proxy, the Alpine shell initializer, and the shell-based mirror are independent implementations of their protocols, built from site/*.Dockerfile. The mirror's sitesctl comes from the separate Sites CLI repository (content mirror).

Behaviour requirements¶