Integrations design¶

Four platform features depend on external backends. Each is a provider feature under an integrations config key. Setup steps: sys-admin: secrets and integrations. The probe display on the Dependencies page is in 22; the health view of integrations is in 21.

This document uses the path shortcuts of 22: core/.../ is core/src/ecmwf/sites/core/.

Provider model¶

integrations:
  analytics: { providers: [splunk], default: splunk, required: false }
  metrics:   { providers: [prometheus], default: prometheus, required: false }
  directory: { providers: [ldap], default: ldap, required: false }
  notifications: { providers: [email, jira], default: email, required: false }

Model rules (core/.../config/schema/models.py):

• providers is non-empty; default must be one of them. providers: [none] + default: none disables the feature.
• Providers self-register at import time with the IntegrationProviderRegistry singleton (core/.../integrations/registry.py); each provider class declares ID, while FEATURE comes from the per-feature interface it implements (the base class itself declares FEATURE = None). Modules live under core/.../integrations/<feature>/ (an <id>.py module or an <id>/ package); none always exists.
• get_configured(feature) returns the singleton of the default provider, or None when the default is none or an unregistered ID (logged as an error). get_all_configured(feature) resolves every listed provider, skips none silently, and logs an error before skipping an unregistered ID.
• required: true marks the feature as mission-critical. It does not change response codes or degradation: consumers gate on integration_status.is_active or on the provider configuration and answer 503 whenever the feature is off or unreachable, or serve their fallback where one exists (the user lists), independently of the flag. The flag selects the probe log severity (ERROR instead of WARNING) and the display on the Dependencies page.
• Notifications may list several providers and select one default; broadcast dispatch can fan out to all of them (Dispatch semantics).

Probes and status¶

IntegrationStatusService (core/.../service/integration_status.py) probes each configured provider once at process start. A provider probe() returns (ok, detail); an unexpected exception marks the provider unavailable and stores the message as the last probe error. A disabled feature (provider none) is recorded as unavailable with an empty error. The service exposes per-provider status records {feature, provider, enabled, available, active, required, last_probe_error} through get_status() (active is enabled and available); disabled features produce no rows. The Hub Dependencies page (/hub/admin/dependencies/) re-probes asynchronously on visit (refresh()) without blocking the render (22).

is_active(feature) is true when at least one provider other than none is configured and the last probe of any configured provider succeeded: with a multi-provider notification set, one reachable provider keeps the feature active even when the default provider is down. Analytical and directory consumers gate on it, so a configured but unreachable backend degrades exactly like a disabled one. Other consumers gate on configuration only: the request endpoints and the requests_available flag use the dispatcher's is_enabled() (a default provider must be configured), the weekly preloader checks get_configured, and the site proxy renders metrics code when the configured provider reports is_enabled — a configured but unreachable backend leaves those paths enabled.

Probe semantics per provider:

Graceful-degradation contract¶

Observed behaviour when a feature is off or unreachable (is_active false):

• Analytics:
• the admin analytics dashboard redirects to the administration index with a flash;
• the per-Site monthly endpoint and both platform-wide analytics endpoints answer 503 Analytics integration is not available.;
• the per-Site weekly endpoint intentionally has no availability gate (sites_analytics in hub/.../app/api/common.py): a report that is already generated keeps being served while the backend is unreachable. A missing report still runs the pipeline: with a dead backend the data collector returns nothing, the first call queues generation and answers 202, and later calls answer 404 No data found to produce a report.;
• the Site view analytics step, its tab, and the admin-index Web analytics entry are hidden (the analytics CronJob object is not a UI element: the chart renders it only when the analytics integration is configured, helm/.../templates/cronjob-analytics.yaml);
• the weekly preloader job exits immediately when no default provider is configured; with a configured default that is unreachable, it runs and swallows the per-site failures into its log.
• Metrics: /hub/metrics/ answers 503, and the per-site proxy configuration contains no metrics code at all — no shared dict, no log_by_lua, no /s/metrics location (the files ConfigMap is rendered without them; see 11, 21, and the Lua metrics.conf).
• Notifications none: access, publish, and storage request endpoints answer 503 regardless of required (_require_notifications); the requests_available context flag hides every request button in the UI (22); the quota crawler logs "skipping notifications for N site(s)" and dispatches nothing; the retention crawler still archives and deletes and logs that the owner is not notified (QBR-05). A dispatch with no targets logs No notification provider available for dispatch. and returns None. Nothing is queued or retried.
• Directory none: the user list is directory.default_users plus the Hub staff and administrator users; directory.validate_usernames is coerced to false by a consistency rule (16).
• Directory configured but unreachable: the users service still returns the directory.default_users fallback list, independently of required; only the probe log severity differs.

Directory settings¶

The directory section applies to all providers (core/.../config/schema/models.py):

• default_users: entries [username, display name, email] that are always selectable. With provider none they are the user list, extended with (username, username, username) entries for staff and administrator users (core/.../integrations/directory/none.py).
• validate_usernames: when true, access forms accept only listed usernames (UserValid); coerced to false with provider none.
• policies_list: the roles/groups offered in access filters. The LDAP provider never queries groups; the list is the only source.
• With provider ldap, an ldap:// connection with ldap.start_tls false sends the bind password in clear text, so the platform reports the ldap-cleartext consistency warning. start_tls: true upgrades the connection before the bind, and ldaps:// is already encrypted (core/.../config/consistency.py; see 10-security, SEC-25).

The users service caches the retrieved list; the Hub autocomplete and the forms read it through users_service (22).

Request flows¶

Three user actions create platform requests through the notifications feature (hub/.../app/api/common.py):

Every flow: _require_notifications() (503 when no default provider), then _create_request renders the template with requester, site, form, host, branding data, and a digest — the MD5 of summary.site-path.assignee.reporter, the deduplication key. The response message carries one line per targeted provider ([id] result, or [id] Unable to send notification when the provider failed); even if every provider failed the endpoint answers 200 with those lines, because the joined report is not empty. The 500 answer fires only when the dispatcher has no target provider at all. The UI shows these buttons only when requests_available (a default notification provider is configured).

Notifications¶

Dispatch semantics¶

NotificationDispatcher (core/.../notifications/common/notification_dispatcher.py) resolves providers at call time:

• default (dispatch(...)): the default provider only.
• notifier_id='<id>': that provider only; takes precedence over broadcast.
• broadcast=True without channels: every configured provider.
• broadcast=True with broadcast_channels=['x', …]: the default provider plus the listed IDs, deduplicated by provider ID.

A provider exception is caught per provider; the per-provider result line is [provider_id] <message> or [provider_id] Unable to send notification, and the dispatcher returns the joined lines, or None when there were no targets. clean(subject, validation_fn) delegates to the default provider.

Providers¶

• email (core/.../integrations/notifications/email.py): resolves a non-email assignee through users_service.email_from_username, then sends HTML mail. The with_copy=True kwarg adds Cc: email.reporter (core/.../service/email.py).
• jira (core/.../integrations/notifications/jira.py): creates or updates an issue; the Request ID: <digest> line in the body enables duplicate detection. When jira.initial_transition is set, the provider transitions the new ticket; jira.final_transition and clean(validation_fn) support closing resolved tickets.
• none: no dispatch targets; see the degradation contract.

Templates¶

core/.../notifications/templates/<provider-id>/<name>.jinja2 — the path is resolved from the default provider ID, so the list must contain the default. Email uses HTML; Jira uses wiki markup. Seven templates per provider:

The backup-failure mail from the backup job bypasses the integrations and calls the email service directly (13).

Analytics backends¶

• Splunk: index apps_kubernetes, field filter k8s.container.name = site-proxy (both configurable).
• OpenSearch: index nginx-logs-*, field filter kubernetes.container.name = site-proxy, timestamp field @timestamp.
• Reports are precomputed by the weekly preloader into /sites-data/<namespace>/analytics/ and served from there (15).
• The analytics API (hub/.../app/api/common.py:_get_report) works on demand: a missing report queues generation in the background thread pool and answers 202 "The report is being generated in the background..."; a generated report without data answers 404; an existing report is served as JSON, HTML, or PDF (reportlab) selected by the Accept header (core/.../analytics/report.py). The Site view loaders and the admin web-analytics dashboard use the same endpoints (22).

Behaviour requirements¶