Data storage and recovery design¶

This page describes the data stores used by the platform and the ownership and recovery rules for each store. The architecture page gives the system context; this page gives the data lifecycle. Job schedules and failure handling are in 15-jobs, the Site API contracts are in 07-site-api, and route-adjacent state is in 14-routing.

Store inventory¶

Site storage layout¶

Site PVC. SiteExporter.to_volume_claim builds one claim per site, named after the site id, with the site labels, size {quota}Gi, storageClassName from kubernetes.storage_class_name (null uses the cluster default) and a single access mode from kubernetes.storage_access_mode (default ReadWriteMany). On update the claim is only resized when kubernetes.storage_expansion_supported is true; the form rejects a quota decrease while the cluster reports expansion support, and Kubernetes never shrinks a claim (09).

Site pod mounts. The StatefulSet mounts the claim as site-volume and the per-site files ConfigMap as site-config. Every container — proxy, API, admin, OAuth2, WebDAV, S3, custom, and the init containers — mounts site-volume at /www; there are no subPath splits inside /www and no emptyDir volumes in site pods. The layout under /www is by convention: html/ (content, also the S3 backend base directory), database/ (the token SQLite database and its backup). A disabled site scales the StatefulSet to 0 replicas; the PVC and its data are untouched (09-disable-and-archive).

Platform data PVC. The chart renders one ReadWriteMany PVC (default name nfs-data-sites-pvc, 100 Gi, class freenas-nfs-csi-retain in the shipped values). The Hub and the jobs mount it at /sites-data; the quota and retention jobs do not mount it by default (cron.<job>.mountsData). Under /sites-data/<namespace>/ live backups/ (platform backups), analytics/ (per-site report directories, keyed by site id), templates/ (ad-hoc site templates, written by the Hub template API and created on demand; the platform-provided site templates come from the read-only sites-templates ConfigMap instead), and secrets.json (the OIDC client-credentials and endpoints file, written by the Hub init container at every start, hub/.../initializer/oidc/ensurer.py).

Redis cache¶

The chart deploys Redis as a single-replica Deployment with the data on an emptyDir and persistence switched off (appendonly no, save ""), plus maxmemory 200mb with maxmemory-policy allkeys-lru. Redis is therefore volatile by design: it loses everything on restart, and entries can be evicted under memory pressure even before their TTL. Every caller must treat Redis as non-authoritative.

RedisCacheService uses the configured host, port, database, and password values (cache.*, 16). It reconnects when Redis is unavailable; cache callers receive a miss instead of a cache-service failure when the connection cannot be established.

The Redis NetworkPolicy admits only the Hub and Operator pods. When CACHE_REDIS_PASSWORD is present in sites-secrets, the Redis container start script also sets requirepass from it. The script passes the value on standard input, not on the command line, and it refuses a value with ", \ or a line break. Redis adds standard input to the end of the configuration file with no line break, so the script starts standard input with an empty line. Then requirepass stays on its own line also when the redisConfig value has no final line break. The readiness probe authenticates through REDISCLI_AUTH. Without the key, Redis runs without a password.

Values are stored with a namespace prefix and expiry (default cache.timeout_cold, 7 days). Integers use a plain ASCII representation. Other values use a signed serialized payload:

1. Pickle serializes the value.
2. An HMAC-SHA-256 signature is added. The signing key is itself derived: HMAC(session_secret_key, "sites-redis-cache") with SHA-256. All components share the same session secret, so they can read each other's entries and an attacker without the secret cannot forge one. The configuration refuses a session secret shorter than 32 bytes, and the cache refuses to sign or verify with an empty key.
3. The marker, signature, and payload are stored with the configured TTL.
4. Reads verify the signature with hmac.compare_digest before using the payload.
5. A restricted unpickler allows only these names:
6. safe built-in primitives (no type, object, eval, open or __import__);
7. the datetime classes datetime, date, time, timedelta and timezone, and the enum base classes;
8. classes that an ecmwf.sites.* module defines, including nested classes such as SiteStatus.Condition.

Pickle protocol 4 accepts dotted names. Thus each part of a dotted name must be a class that the named module defines, and a part must not start with __. A name that reaches a function, an imported module or an imported class (for example pickle.loads through an ecmwf.sites module) is rejected. The module must already be loaded in the process, so an entry cannot make the process import a module. Any other reference raises before an object is built.

The unpickler cannot check the BUILD opcode, so a signed entry can still set attributes on an allowed class. This is an accepted risk: only a holder of the session secret key can sign an entry.

An unsigned, incorrectly signed, foreign-signed, or unpicklable payload is ignored — the caller sees a miss. Cache data is not authoritative data; callers must work when an entry is missing or expired.

Key space¶

Prefixes are defined in core/src/ecmwf/sites/core/config/redis.py:

The Hub cache-clear action (misc_clear_cache) removes the site, spec, and stats caches by wildcard and refreshes the directory-user cache. It does not touch OIDC credentials, discovery metadata, or locks. Site templates are not cached; the Hub reads them from disk on demand.

The distributed lock service (core/src/ecmwf/sites/core/service/lock_service.py) provides two primitives: try_lock (atomic Redis set-if-absent with a TTL, falling back to a process-local lock when Redis is unavailable — cross-replica exclusion is then lost) and lock (an unconditional marker write, used as a "generation in progress" flag). Users: the Hub bulk-upgrade lock (sites-bulk-upgrade, TTL 1800 s, 12) and analytics report generation (TTL 180 s, 15).

OIDC discovery adds a second validation layer: it checks the issuer and the endpoint URLs when it reads a Redis entry, before it returns the metadata.

Platform state¶

sites-platform-state is deliberately not Helm-managed. The Hub creates it when needed and updates it with resource-version conflict retries (up to five). A value of None removes a key.

The read-only key is used by the Hub and routing integrations. It survives a Helm upgrade and a workload restart. The Hub has access only to this ConfigMap and the named platform configuration objects described in the role-based access control (RBAC) design. Enforcement per routing mode is in 14-routing.

Platform backup¶

The backup system writes one dated YAML file under /sites-data/<namespace>/backups/. It lists all Site resources and removes status and resourceVersion from each exported object. It does not copy Site PVC content. Pruning is by modification time only (jobs.backup_keep_files days); there is no size cap, and subdirectories are kept. The schedule, the failure email, and the failure semantics are in 15. Note that the backups live on the platform PVC: a platform-PVC loss takes the content definitions' backup with it, so the storage system must back this PVC up as well.

Restoring Site definitions is a Hub administration operation. Restoring the backing storage is an operation of the Kubernetes storage system and is not performed by the platform backup job.

Token history¶

The Go Site API stores HTTP access token history in SQLite on the Site volume: /www/database/tokens.db, opened with WAL mode, a shared cache, and a 5 s busy timeout. The database stores token hashes and metadata such as identifier, permission, expiration, last use, and revocation state. Raw access tokens are not stored as token-history records.

History writes use a bounded asynchronous worker queue of 256 entries. A full queue or an unavailable database is logged and does not block the token request. Cleanup removes expired history entries after a 7-day grace, daily. Revocation updates the history record and uses the token hash. Every token decode first checks an in-memory Bloom filter of revoked hashes, seeded from the database at startup (api/token_revocation_bloom.go); the filter is memory-only and rebuilt on every restart. A hash that the loaded filter does not contain is not revoked, so the decode accepts it without a database read. Otherwise the database gives the authoritative answer, and a decode fails closed when the database is unavailable, so a revoked token cannot be used during the outage. When the database does not open, the API retries it in the background every 30 s and rebuilds the filter and the history endpoints when it opens, so revocation recovers without a restart; a snapshot is also written to the backup file at startup and once per day. The API contract (create/decode/revoke/backup/restore shapes) is in 07-site-api.

The API can export token history to a gzipped JSON backup with a SHA-256 checksum (/www/database/token-history-backup.json.gz) and restore it after validating the backup. Normal restore and the database recovery path are both reported in the API logs. The backup sits on the same PVC as the database: it survives a corrupted database file, but not a lost volume. Losing the PVC loses both copies.

Deletion and recovery boundaries¶

Deleting a Site resource deletes the namespaced objects through Kubernetes owner references; there is no delete handler and no finalizer (09). The owned set: Secret, both ConfigMaps, PVC, StatefulSet, Service, route, and gateway policy objects. The backing PersistentVolume keeps the reclaim policy of the Site storage class (kubernetes.storage_class_name; the shipped default is null, so the cluster default class decides), and can remain Released for an administrator to clean up. The class name freenas-nfs-csi-retain in the shipped values belongs to the platform data PVC, not to site volumes: site volumes accumulate only when their own class retains.

Site recreation re-binds a retained volume: on create, the operator looks for a PersistentVolume matching the site metadata or the old claim names (account or site id). When it finds one, the new PVC is created with volumeName pinned to that PV, and the operator patches the PV's labels and claim reference. This is how a restored Site definition picks up its old data.

The platform therefore has separate recovery boundaries:

• Site definitions: platform backup and Hub restore.
• Site content: storage-system backup and PVC/PV re-binding.
• Token history: Site API JSON backup and restore (same-PVC caveat above).
• Analytics: weekly preloader recomputation and on-demand generation. Report files are never pruned — every generated weekly period accumulates on the platform PVC — and deleting a Site does not remove its /sites-data/<namespace>/analytics/<site-id>/ directory.
• Templates: explicit Hub template deletes only; Site deletion leaves template files and the site's own /www content to the PV lifecycle.
• Redis: cache rebuild from authoritative data; no recovery operation exists or is needed.

Behaviour requirements¶