Observability and triage¶

This page maps symptoms to the component that decides the behaviour. It lists the health signals each component exposes, where to read them, and how to triage cross-layer failures. User actions: end-user troubleshooting. Platform procedures: sys-admin troubleshooting. The UI surfaces (Site view tabs, administration pages) are in 18-hub-web-interface.

This document uses the same path shortcuts as 22: hub/.../ is hub/src/ecmwf/sites/hub/, core/.../ is core/src/ecmwf/sites/core/.

Hub health endpoints¶

Platform health: GET /hub/api/v2/health/¶

Platform administrator credentials only (/hub/api/v1/ mirrors the route). The endpoint always answers HTTP 200; the health verdict is in the body, never in the status code. It checks three subsystems and returns JSON {status, message, data} with data.redis, data.kubernetes, and data.nfs:

• redis: cache round-trip probe (a get of a non-existent cache key), with the configured URL and response_time, or status: NOK with the error.
• kubernetes: API reachability probe, with the namespace and response_time.
• nfs: a latency probe on the platform data directory. The check_storage_and_elapsed_time helper (core/.../misc/storage_utils.py) submits os.listdir to a dedicated single-worker daemon-thread executor and returns (elapsed, timed_out). It measures read latency, not write staleness. The timeout is hub.health_timeout_seconds (default 5.0); ?deep=true waits unbounded for diagnostics. A timeout reports timed_out: true and the status set by hub.health_timeout_status (WARN by default — informational; ERROR flips the overall health). mount carries the /proc/mounts line for /sites-data, or null.

With Accept: text/plain or Accept: application/nagios, the same data returns as a Nagios plugin string (core/.../service/opsview.py):

<message> - Redis = <S>, Kubernetes = <S>, NFS = <S> | redis_response_time=<t>, kubernetes_response_time=<t>, nfs_response_time=<t>, nfs_timed_out=<b>

Monitoring systems that key on HTTP status cannot use this endpoint; key on the body or the Nagios prefix instead. overall_sites_health is the status-code-friendly alternative below.

Sites health: GET /hub/api/v2/sites/health/¶

Platform administrator credentials only. The Hub probes every enabled and operationally monitored Site (spec.configuration.settings.monitored) in parallel through the site health path below, with a shared 45-second pool timeout. The body lists {url, status, response} per Site. If any monitored Site is unhealthy, the endpoint deliberately returns HTTP 500 with Some operational sites are currently down! — the only Hub API endpoint that fails on purpose, so a plain status-code monitor can watch it. Unmonitored Sites never affect it.

Read-only state¶

POST /hub/api/v2/read-only/ (platform administrator) toggles the state in the sites-platform-state ConfigMap and answers System ReadOnly set to <bool>.. There is no GET endpoint; read the state with kubectl get cm sites-platform-state, or observe the badge on the administration index page and the flash on the lists (22). The ConfigMap also holds the in-band platform configuration; never edit either key by hand (13).

Route status¶

HTTPRoute resources report controller acceptance in status.parents conditions; a networking Ingress reports through status.loadBalancer and status.conditions instead. Check them when traffic does not reach the Hub or a Site (14). The Hub pod itself has no HTTP readiness probe; an initializer failure shows in the init container (08 — Hub pod initialisation).

Site health signals¶

The Site metrics families are sites_http_requests_total (labels site_path, status, method, handler), request- and response-size and duration histograms, sites_http_authenticated_requests_total (labels site_path, user, handler; the user label is truncated at 32 characters), sites_http_4xx_total, and sites_http_5xx_total (proxy/lua/metrics.lua). The handler label maps the path prefix to api, admin, webdav, or web. Requests to /s/metrics and /s/healthz are not counted. The proxy log_format main line is what the analytics integrations parse (17). In that line, the request and referer fields contain REDACTED in place of the signed URL signature value and the session URL token. The Site API request log also redacts the signature value, and the OAuth2 Proxy request log does not record the request URI. Use the proxy access log to find the path of a request.

Pod probes¶

Status written to the Site resource¶

The operator writes a condensed status into the Site Custom Resource Definition (CRD) (core/.../kubernetes/utils.py:resolve_pod_status): a single condition (the first failed condition, or the last true one, with last_transition_time), all_steps_started, phase (a synthetic Disabled phase stands in when the Site is archived; container problems override the pod phase), start_time, message, reason, and containers (one item per waiting, terminated, or running-but-unready container, with name, kind, reason and message; an unready running container reports kind: running with reason: NotReady). The top-level message and reason exist on the model but the operator's pod-status writer never fills them, so they stay empty on current resources. All fields are plain text, with no HTML. The Hub template escapes them and adds the formatting. The field site_enabled is not stored; the Hub data access object (DAO) injects it when reading, because enabled is a spec field (09). Nothing in the control loop consumes the stored status beyond SiteStatus.are_all_pods_running(); it exists for display and for kubectl get site -o yaml.

Logs and audit¶

• Operator logs prefix events with [EVENT], [CREATE], [UPDATE], [RESUME], and [PATCH] (persistent volume (PV) patch path, operator/.../site_create.py). Update summaries log changed field paths at INFO and the full diff at DEBUG; operator/.../logging_policy.py masks secret-bearing values and keeps key names only.
• Hub audit entries use a dedicated log level and the format username=<user>;action=<action>;resource=<detail>. The @audit.log_access decorator additionally records action=access with method, endpoint, sanitized form data, remote_addr, and referrer (core/.../misc/audit.py). Six high-frequency API routes omit the decorator and produce no action=access lines: the Site status poll and the user, group, site-owner, site-group, and application lookups (08 — Application assembly). A MaskingFilter and sanitize_struct mask tokens, cookies, and secret-shaped strings before anything is written (core/.../misc/logging_filter_utils.py).
• Hub and operator pod logs are readable without kubectl: GET /hub/api/v2/logs/?name=<sites-hub|sites-operator>&limit=<n> (platform administrator; defaults to the Hub pods).
• Site pod logs: GET /hub/api/v2/sites/<space>/<site>/logs/?name=<APPLICATION>&limit=<n> (Site administrator). The response maps container name to a log block; multi-pod apps prefix chunks with [pod-name]:. The Operations tab console calls this endpoint (22).
• Bulk upgrade failures are searchable in the Hub logs: Bulk forced upgrade failed names each Site the bulk action could not update; each forced upgrade writes an audit action=upgrade line (12).

Platform metrics¶

GET /hub/metrics/ serves Prometheus text only while the metrics integration is active; otherwise it answers 503. The exporter (core/.../integrations/metrics/prometheus.py) renders siteshub_sites_total and siteshub_http_requests_total; the request counter is an after_request hook registered when the provider is configured (hub/.../server.py).

The endpoint label of siteshub_http_requests_total is the Flask route template (request.url_rule.rule), for example /hub/api/v2/sites/<space>/<name>, and never the request path. A request that matches no route (404, or 405 for a method that is not allowed) gets the fixed label <unmatched>. The number of label values is therefore limited by the number of Hub routes, and a client cannot add time series with random paths. The routes that end in /healthz, /metrics, or /ping are not counted.

The path is blocked at the edge in both routing modes: the ingress minion adds location = /hub/metrics/ { deny all; }, and the gateway HTTPRoute redirects /hub/metrics* to the 403 error page. Monitoring must scrape the sites-hub Service directly.

Inside the cluster, the NetworkPolicy limits the Hub port to the routing, monitoring, and metrics-gateway namespaces (SEC-29). The endpoint has no authentication by default, because Prometheus scrapes it without a credential; the payload contains only site totals by state and type, and request counts by method, route template, and status class, with no site names, request paths, users, or secrets. Set hub.metrics_scrape_token (env HUB_METRICS_SCRAPE_TOKEN) to require a bearer token; when it is set, the route answers 401 without the token, and serviceMonitor.hub.bearerTokenSecret points the scrape at the Secret key that holds it. The generated metrics are cached for a few seconds, so the full site listing runs at most once per scrape interval.

When serviceMonitor.enabled is set, the chart creates Prometheus Operator ServiceMonitors: the Site monitor scrapes Service port 4180 at /s/metrics (the oauth2-proxy container lets that route through authentication), and the Hub monitor scrapes port 8080 at /hub/metrics/. Site scraping needs the monitoring namespace in networkPolicy.monitoringNamespace and the scraper IP in trusted_ips.

Integration probes¶

The Dependencies page renders the provider probe results and refreshes them in a background thread behind a non-blocking lock, so the page never waits on an integration. The provider contract and the graceful-degradation rules are in 13 — Probes and status. Probe state also gates UI features (the analytics sections and the request modals, 22).

Triage table¶

Common failure modes¶

• CRD drift. The chart installs the Site CRD but never upgrades it. The Hub or operator rejects a schema the CRD does not know. Apply helm show crds oci://... | kubectl apply -f - before upgrading (sys-admin: upgrades).
• Missing Secrets. sites-secrets, oidc-secrets, or the referenced TLS Secret missing in the release namespace causes CrashLoops or routing failures.
• RBAC changes outside the chart. Hand-edited Roles are out of sync with the chart-managed bindings; compare against helm/.../templates/rbac-*.yaml.
• Stale branding or configuration on a site. Expected until the site's next update (16).
• Released PersistentVolumes. After deletion the PV survives as Released; clean it from the Hub administration pages. It never rebinds automatically.
• Site not upgrading. A site already at the platform application versions can do no work on an ordinary upgrade. Use Force Upgrade All Sites or POST /hub/api/v2/sites/upgrade/?force=true to regenerate configuration and routes (14). To force one site, use POST /hub/api/v2/sites/<space>/<site>/upgrade/?force=true. Search the Hub logs for Bulk forced upgrade failed to find sites that a bulk upgrade did not update.
• Monitoring against the wrong signal. The Hub health endpoint always returns 200; the Site CRD status is display data, not a control signal; and /hub/metrics/ is 503 whenever the metrics integration is off. Wire status-code monitors to GET /hub/api/v2/sites/health/ and to the Service endpoints, not to the browser routes.

Escalation checklist¶

Escalate to the platform team when the Hub is down for everyone, the operator is not reconciling, data loss is suspected, or 5xx errors repeat. Include: space and site name, timestamps, exact endpoint and status code, and relevant log or event snippets.

Behaviour requirements¶