Quotas, backups, retention, and jobs¶

This page documents the four scheduled jobs and the quota model, with the implemented defaults. The jobs live in core/ (backup/, notifications/quota/, notifications/retention/, analytics/) and run as CronJobs from the chart with ServiceAccount sites-jobs. Operations and configuration procedures: sys-admin: operations.

Store ownership, backup contents, token history, and recovery boundaries are defined in data storage and recovery.

Job inventory¶

Each CronJob runs the core image with /opt/python/sites/bin/python3 -m ecmwf.sites.core.<module> (backup, notifications.quota, notifications.retention, analytics), reads the same sites-config configuration and secrets as the services, and mounts /sites-data from the platform data persistent volume claim (PVC) when its mountsData value is true (defaults as in the table).

Shared job wiring (all four templates): concurrencyPolicy: Forbid (a run never overlaps the previous one), history limits 1/1, backoffLimit: 0 (a failed run is not retried until the next schedule), activeDeadlineSeconds: 9000 for the backup, quota, and retention jobs and 18000 (5 hours) for the analytics job (a run is killed at its deadline), restartPolicy: Never, priorityClassName: high-priority, ServiceAccount sites-jobs, and resource requests/limits of 256Mi/100m and 1Gi. Schedules, suspension, host-network use, and PVC mounting are chart values under cron.*.

Quota model¶

• The Site stores quota as an integer number of GiB. The Custom Resource Definition (CRD) schema (v2) sets default 1 and bounds 1..5000. The Hub caps the selectable value at site.storage_max_quota (default 500); a request above the cap goes through the storage-request notification flow (17-integrations).
• The per-site PVC requests exactly {quota}Gi (site_exporter.to_volume_claim), and the site writes into that volume, so the filesystem is the hard limit. The Go API adds a statfs pre-check and maps EDQUOT/ENOSPC to HTTP 507 (11).
• The Hub shows current usage from the site health data.

The notification crawler framework¶

Both notification jobs extend AbstractNotificationCrawler (core/.../notifications/common/notification_crawler.py). The constructor runs the whole job in three steps:

1. Clean: dispatcher.clean(subject, validation_fn) on the default provider resolves previously raised notifications whose condition no longer holds (the Jira provider searches open tickets whose summary contains the crawler title; the email interface default is a no-op). Failures are logged and swallowed — cleanup retries next run.
2. Generate: list all Site CRs directly through the Kubernetes API (the service layer needs cluster state the jobs do not have) and group the sites that fail validation_fn by owner.
3. Process: with process_individually=True — which is what both job entry points use — render one notification per site and dispatch it through the default provider. Titles follow Sites - <crawler title> - <site path> and digests follow md5(<prefix><site_id>). Templates come from core/.../notifications/templates/<provider ID>/, so each provider has its own copy of each message.

Dispatch failures never fail the job: _notify and the dispatcher catch every provider exception, log it, and continue. A notification that could not be sent is simply retried on the next scheduled run.

Quota notification job¶

For each enabled site the crawler fetches the site health endpoint through the public proxy path (authenticated with the site master token, health_response_from_site). A site is valid — no notification — when its data.storage.used_percent is below jobs.quota_trigger_percentage (default 85; validated 1..100 at startup). Notifications start at exactly the threshold. Title Storage quota getting full, digest md5('QUOTA-'+site_id) — stable per site across runs.

A digest-aware provider de-duplicates repeats: the Jira provider searches for an open notification with the same digest. The email provider is fire-and-forget: it ignores the digest, re-notifies every run while the site stays above the threshold, and offers no way for the job to close a sent email.

Two fail-open rules in the validation (quota/crawler.py):

• a disabled site is valid without a health call;
• a site whose health request fails or whose response is not a JSON object (restarting, unreachable) is treated as valid and skipped — the code comment explains that a transient outage must not page the owner.

The cleanup step closes stale quota tickets when usage drops below the threshold or the site is disabled — again only on providers that implement clean (Jira).

Backup job¶

The backup job lists every Site resource through the Kubernetes API and writes one file, /sites-data/<namespace>/backups/YYYYMMDD.yaml, one ----separated document per site (SiteExporter.to_crd without status and without resourceVersion). The file is opened in write mode, so a second run on the same day replaces it. After a successful export it deletes files in the backups directory whose modification time is older than jobs.backup_keep_files (default 60 days); subdirectories are kept. Pruning only runs on success. On export failure it emails jobs.backup_notifications_email — only when that setting is configured, otherwise it logs a warning — and re-raises, so the CronJob run fails (and is not retried, backoffLimit: 0).

Two properties are deliberate:

• Backups contain site definitions only, never site content. Content backup is the storage system's job.
• Restore is a Hub administration function for site definitions (sys-admin: operations).

Retention job¶

A site carries a retention_date. The weekly job applies this sequence:

flowchart TD
    A["retention_date ≤ today + trigger_before_days"] --> B["notify owner: 'Retention date expiring'"]
    C["today > retention_date + archive_after_days and site enabled"] --> D["archive: set enabled=false, notify owner + broadcast channels"]
    E["today > retention_date + delete_after_days and site archived"] --> F["delete Site resource, notify owner"]

Implemented details (core/.../notifications/retention/crawler.py):

• The notification rule runs first and covers every site at or inside the trigger window, including sites already past their retention date. Archive and delete are extra_actions with strict > comparisons against the thresholds.
• Archiving calls the same sites_dao.toggle_site as the Hub (09); data stays on the PVC until deletion. "Archived" means enabled=false — it does not record who disabled it. A site the owner disabled manually is therefore deleted once it passes the delete threshold.
• Deletion happens only for already-disabled sites and uses sites_dao.delete_site: a run cannot archive and delete the same site (the archive branch continues).
• Archive and delete sleep 1 s per action to spare the API server, and after any action the job deletes all entries of the platform sites Redis cache (SITES_CACHE_KEY_PREFIX wildcard delete) so the UI shows the change immediately.
• Notifications for archive and delete go to the owner and broadcast to jobs.retention_broadcast_channels: with a non-empty list the targets are the default provider plus those channels, deduplicated by provider id; an empty list fans out to every configured provider. Each dispatch also passes with_copy (a backup-channel copy for auditing), comment, and tag_username to the provider.
• When integrations.notifications is none, archive and delete still run and log a warning that the owner is not notified; a provider that raises is caught by the framework as described above.
• Because the job runs weekly, each action lands on the first Monday after its threshold. The cleanup step resolves "Retention date expiring" tickets when the owner extends the retention date.

Defaults (configurable under jobs.): retention_expired_trigger_before_days = 31, retention_expired_archive_after_days = 183, retention_expired_delete_after_days = 365. The consistency checker reports an error (which blocks startup) when archive is not strictly earlier than delete (core/.../config/consistency.py; see 16).

Analytics preloader¶

The CronJob renders only when an analytics provider other than none is configured (QBR-06) and is suspended by default (cron.analytics.suspend = true): enabling analytics without un-suspending leaves precomputation off. The job skips again at run time when no analytics provider is active, computes reports for the previous week (span 2 hours per site) for every enabled site with clean_report=True, and writes them under /sites-data/<namespace>/analytics/. A per-site failure logs a warning and does not stop the run. Users normally notice the job only as faster analytics pages.

The concurrency contract (core/.../analytics/report.py, core/.../service/lock_service.py):

• When generation starts, Report writes a Redis marker key (per site and period) with time to live (TTL) 180 s and deletes it in a finally block. The TTL bounds a crash mid-generation.
• ReportService.exists() — the readiness check — is true only when the marker is absent and the JSON data file exists. While a report is being generated, the API therefore serves "no report yet" and queues regeneration (12).
• The queue is the Hub's single-worker serial pool, so within one Hub process generations run one at a time. The marker is a flag, not an atomic lock (lock() is a plain set-ex; the atomic try_lock exists but report generation does not use it): concurrent requests, or other Hub replicas, can enqueue overlapping generations. The serial pool and the final writer make the result consistent; wasted work is the accepted cost.

Behaviour requirements¶