Security design¶
This page describes the implemented authentication and authorisation model, the enforcement points, and the security headers per routing mode. For user steps, see end-user: managing access.
Container isolation, custom application boundaries, and cache data protection are described in site application architecture, custom applications, and data storage and recovery.
Trust model¶
Sites is an internal service for an organisation that operates it in a trusted environment. It is not an open hosting service where anonymous people create sites. The security design depends on these trust levels:
| Actor | Trust | Reason |
|---|---|---|
Platform administrators (hub.administrator_users, hub.administrator_groups) |
Trusted | They operate the platform and have full control. |
Staff (hub.staff_users, hub.staff_groups) |
Trusted | Only staff can create sites: POST /hub/api/v2/sites/ requires the staff policy (staff_oidc_api_auth in hub/.../api/v2/views.py). |
| Site owners and site administrators | Trusted, and responsible for their site | A site administrator can publish any HTML and JavaScript, and can configure a custom application. The owner and the administrators can give administrator rights to any user or group of the identity provider, also to people outside the organisation. The person who gives these rights is responsible for this decision. |
| Users with view access, token holders, holders of signed or session URLs | Not trusted | They get only the access that the site gives them. |
| Anonymous visitors | Not trusted | They get only public content. |
The platform enforces these boundaries:
- A user without access to a site cannot read or change it.
- A site cannot read or change the files, Secrets, or Kubernetes objects of
another site. Between site Pods, this needs
networkPolicy.enabled(enforcement surfaces). - A site cannot use the Hub session when
routing.hubHostis set.
The platform accepts a risk that needs a site administrator who acts in bad faith, for example a script that a site administrator publishes on purpose. The accepted risks section lists these risks.
Enforcement surfaces¶
Authentication is enforced where traffic enters. The Go API performs no
authentication at all: it binds only 127.0.0.1:8090, and no Service or
Ingress routes to it. The proxy is its only client, reaching it over
127.0.0.1:8090 (proxy/nginx/api.conf). Other Pods cannot connect to the
API. The liveness probe runs /api healthcheck in the container, because the
kubelet cannot reach a loopback port. When networkPolicy.enabled is set, the
site NetworkPolicy permits ingress only on port 4180 (OAuth2 Proxy), from the
routing namespace, the monitoring and metrics namespaces, and same-namespace
pods (helm/ecmwf-sites/templates/networkpolicy-site.yaml). All sites share
one namespace, so this port limit also stops one site Pod from calling the
unauthenticated ports (8080, 7000, 6000, 9000) of another site Pod.
| Surface | Enforced by | What is accepted |
|---|---|---|
| Site content and listing UI (private site) | site-oauth2 (OAuth2 Proxy) + Lua private_auth |
OIDC browser session, or any chain credential (master token, Branca token, signed or session URL) |
| Site content and listing UI (public site) | Lua public_auth + basic.conf |
Anonymous, or HTTP Basic where a .htpasswd protects the path |
Site API /s/api/ |
Lua admin_auth |
Site API tokens (see chain below). JWT sessions must carry a site-admin entitlement; master and Branca tokens are accepted without a role check (read-only Branca tokens are limited to read methods) |
/s/api/v2/session-files/ |
Lua private_auth |
Site-admin or staff |
/s/api/v2/pub/ (public sites) |
None (no Lua) | Anonymous; the Go public router applies .htpasswd hiding |
/s/signed, /s/session (private sites) |
Lua auth chain, then the Go API verifies the credential | Signed/session URL parameters only |
/s/admin/ |
Lua admin_auth |
Site administrators |
/s/dav/ |
Lua (webdav_auth) |
Basic auth with a Site token as password (master token or Branca token), validated against the token system, not against .htpasswd file contents. The validated session lasts 1 hour: the token is kept in a Secure, HttpOnly, SameSite=Lax cookie scoped to the Site, and after expiry the proxy rotates the challenge realm nonce and forces a new prompt. On public sites, paths without a .htpasswd also allow anonymous read-only methods |
/s/s3 |
Lua (s3_auth, x-amz-security-token header) |
Issued S3 credentials |
/<space>/<site>/s/healthz |
NGINX allow/deny | IPv4 loopback (allow 127.0.0.1, the Site API probe) always, plus trusted_ips matched against the direct peer address; everything else is denied |
/s/metrics |
NGINX allow/deny | trusted_ips only, evaluated against the real IP taken from the leftmost X-Forwarded-For entry, trusted from any source (real_ip_recursive off, set_real_ip_from 0.0.0.0/0); an empty list denies all; the location renders only when metrics are enabled |
| Hub UI + Hub API | Flask session or bearer token | OIDC login (Authorization header only; bearer values in the query string or form are ignored). Unsafe requests with the session cookie must present the CSRF token (X-CSRF-Token header or csrf_token form field) |
The Lua auth chain tries, in order: session URL → signed URL → master token
(hex-64) → JWT → Branca token
(site/src/ecmwf/sites/website/applications/proxy/lua/auth_chain.lua).
Token prefixes: Bearer, Branca, Master. A failed token decode is
negatively cached for 150 s; successful Branca decodes are cached for at
most 10 s. Failures return 403 with a JSON body for API, _m2m, and S3
paths.
flowchart LR
R["Request to site"] --> C1{"session URL?"}
C1 -- yes --> OK["allow"]
C1 -- no --> C2{"signed URL valid?"}
C2 -- yes --> OK
C2 -- no --> C3{"master token (hex64)?"}
C3 -- yes --> OK
C3 -- no --> C4{"JWT (OAuth2 session)?"}
C4 -- yes --> OK
C4 -- no --> C5{"Branca token valid + allowed?"}
C5 -- yes --> OK
C5 -- no --> N["403 (JSON for API paths)"]
Credential types¶
| Credential | Created by | Default lifetime | Scope |
|---|---|---|---|
| Site master token | Hub at site creation (stored in spec.configuration.share.master_token and the per-site Secret) |
until rotated | Full access to one site on every surface |
| HTTP access token (Branca) | Site API POST /s/api/v2/token/ or Hub UI |
90 days (site.token_default_duration); the Hub token-history view flags tokens expiring within jobs.http_access_token_expiration_warning_seconds (604800) — no notification job exists |
Configured permissions on one site |
| Signed URL | POST /s/api/v2/signed-urls/ |
7 days (site.signed_url_default_duration) |
One path and method set |
| Session URL | POST /s/api/v2/session-urls/ |
7 days (site.session_url_default_duration) |
Time-boxed browsing session |
| OAuth2 Proxy session cookie | OAuth2 Proxy login | 12 h expiry, 2 h refresh (oidc.cookie_expire/cookie_refresh), per-request CSRF cookie (5 m) |
Browsing one private site |
| Hub session cookie | Hub login | Browser session cookie: the configured session.permanent_lifetime (72 h) applies only to permanent sessions, and the Hub never sets session.permanent |
Hub UI |
| Hub bearer token | Identity provider via Hub API | provider-dependent | Hub API |
| Hub platform master token | Hub administrator page (writes HUB_MASTER_TOKEN in the sites-secrets Secret; hub.master_token validates it) |
until regenerated | Hub API as the api-administration-token identity, carrying the configured administrator and staff group policies |
Treat the master token as a break-glass credential: store it in secret managers, rotate after exposure, and prefer short-lived scoped tokens for automation.
Roles: allowed users/groups may view a private site; staff
(default_view_groups, default_view_users) may read site content and use
session-files; administrators manage the site and use the full Site
API. The Hub enforces these from the Site resource on every request. Both
staff lists are empty by default, so an operator must set them explicitly.
.htpasswd protection¶
A file named .htpasswd protects its directory and everything below it.
Two independent mechanisms enforce this:
- Go API (public router only).
baseOrAncestorProtected()makes any request under a protected tree return 404 — not 401/403 — so path names are not disclosed. Listings prune protected entries, including cached continuation pages, and noRepr-Digestis returned for hidden paths (site/.../api/api/public_files.go,file_utils.go). - Lua proxy (browser traffic on public sites). An NGINX
mapresolves each URI to the applicable.htpasswd— including/s/dav/paths — andauth_basicrequests credentials against that file (proxy/nginx/basic.conf,nginx.conf).auth_basicis rendered only for public sites. WebDAV never uses the file check: the/s/dav/location runs onlyproxy/lua/basic.lua, which ignores the username and validates the password against the token system (64-hex master token or Branca token). Accounts listed in a.htpasswdare therefore not accepted on WebDAV; the request keeps getting 401 challenges until it presents a valid token (Site tokens have full access). The.htpasswdstill gates WebDAV indirectly: on public sites the Lua anonymous fallback applies only when themapfinds no.htpasswdfor the path and the method is read-only.
On private sites .htpasswd is inert: auth_basic is not rendered and the
Go public router is not registered.
The download-folder shortcut in the listing UI follows these rules: it
applies the site type's access module, and on public sites it goes through
the Go public router, so protected trees answer 404
(proxy/nginx/api.conf; see 06-site-proxy).
Because hiding returns 404, a missing file and a hidden file are indistinguishable on public endpoints. The authenticated API sees everything.
On public sites, the content locations run lua/public_auth.lua: anonymous
requests outside GET, HEAD, OPTIONS, PROPFIND, SEARCH are rejected with
403, so anonymous write requests are never accepted. The Go public router
behind /s/api/v2/pub/ registers only GET, HEAD, and OPTIONS;
requests with other methods answer 404 there, because the router has no
method-not-allowed handling.
Security headers per routing mode¶
When routing.hub_host is set, the Hub always adds the headers itself on the Hub host (in both modes), with frame-ancestors 'self' and X-Frame-Options: SAMEORIGIN. The Hub session cookie is __Host-sites-hub, and every Hub API route answers CORS without credentials. Together with the separate host, a site script cannot use the Hub session. See 14 — Routing and the system administrator guide.
The platform-level headers come from different components per mode; the site proxy adds its own in both modes.
| Header | Ingress mode | Gateway mode | Site proxy |
|---|---|---|---|
X-Content-Type-Options, X-XSS-Protection, Expect-CT, Feature-Policy, Referrer-Policy |
NGINX minion snippet | Hub middleware | — |
Content-Security-Policy (frame-ancestors from trusted_domains, each domain plus its subdomains; 'self' is always included) |
minion snippet | Hub middleware | own CSP in cors.conf (adds cors.extra_domains only when CORS is enabled) |
| HSTS | minion snippet (routing.hsts) |
Hub middleware (routing.hsts) |
max-age=63072000 when CORS is disabled, together with X-Permitted-Cross-Domain-Policies: none |
Cache-Control: no-cache |
minion/middleware | middleware | expires -1 per headers.no_cache |
X-Robots-Tag: noindex |
— | — | when headers.no_robots |
X-Sites-Proxy / X-Site-Proxy-Server |
minion/middleware | middleware | proxy identity header |
In gateway mode the Hub middleware adds the five security headers,
Cache-Control, and the CSP with setdefault, so explicit application
headers win. X-Sites-Proxy and, when routing.hsts is on, the HSTS header
are assigned directly and always overwrite application values. The middleware
is registered only in gateway mode (hub/src/ecmwf/sites/hub/routing_rules.py).
The concrete gateway values: X-Sites-Proxy is the request host; the HSTS
value is max-age=31536000; includeSubDomains; preload; the CSP is
frame-ancestors 'self' https://<domain> https://*.<domain> … plus the fixed
Hub sources HUB_CSP_SOURCES (the frame-ancestors list grows with
trusted_domains; with an empty list only 'self' is allowed to frame):
default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';
img-src 'self' data:; font-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'self';
The NGINX minion snippet of the Hub Ingress sends the same sources. The Hub
loads its scripts, styles, and fonts only from its own origin
(static/vendor), so a browser does not load code from a third-party CDN.
The templates have inline scripts, inline event handlers, and javascript:
links, so 'unsafe-inline' stays for scripts and styles. The site proxy CSP
is a different policy and does not change.
trusted_ips (root config, default empty) restricts the site health
location (/<space>/<site>/s/healthz, NGINX stub status) and the global
/s/metrics endpoint, and is passed to OAuth2 Proxy. The health location
always allows the IPv4 loopback address (127.0.0.1), so the Go API probe
needs no configuration. There is no IP restriction on signed or session
URLs.
Hub frontend packages¶
The Hub does not keep third-party frontend files in git. hub/frontend
contains the npm manifest, the lockfile, and the allow-list
vendor-files.json:
package.jsonsets an exact version for each package.package-lock.jsonrecords theregistry.npmjs.orgURL and thesha512integrity of each package.npm cistops when a download does not match.- The
frontendstage of the Hub image (nodeimage with a digest) runsnpm ci --ignore-scripts, so no package install script runs, and thencopy-vendor.mjs. The script copies only the allow-listed files tostatic/vendorand the licence files to/usr/share/licenses/sites-hub/. It stops when a version does not match the lockfile, when a path leaves its package, or when a file or licence is missing. The final Hub image does not contain Node.js ornode_modules. - The Hub sends the
static/vendorfiles unchanged. The response minifier (flask-minify) skips them, because a second minification can break upstream files (for example TomSelect and Swagger UI). This also keeps the licence headers. hub/tests/test_vendor_assets.pyfails when a template references an external script, style, or font, when a template uses a vendor file that is not in the allow-list, or when a package licence is not MIT, BSD-3-Clause, or Apache-2.0.- The Security workflow runs
npm audit --package-lock-onlyon the lockfile (advisory, like the Python and Go audits). Dependabot proposes npm updates.
OIDC provider integration¶
The platform needs oidc.issuer_url, account_url (the account page, not
the userinfo endpoint), and redirect_uri (the registered callback). Two
modes exist; issuer_url is required in both:
- Discovery (default): metadata is fetched from
<issuer_url>/.well-known/openid-configuration. Single endpoint overrides replace discovered values. - Explicit endpoints: when all six of
authorization_url,token_url,user_info_url,introspection_url,revocation_url,jwks_urlare set, no discovery document is fetched.logout_urlstays optional.
Keep client IDs and secrets in the oidc-secrets Secret, never in
ConfigMaps or values files.
OAuth2 Proxy secrets¶
The per-site OAuth2 Proxy needs a cookie secret and the OIDC client secret.
Neither value is in the files ConfigMap <id>-files, because a ConfigMap
is not a Secret for Kubernetes RBAC, audit, and encryption at rest.
- Cookie secret: the operator writes the Secret
<id>-oauth2, keycookie-secret. The value isHMAC-SHA256(session.secret_key, "ecmwf-sites/oauth2-cookie-secret/v1/" + site_id), URL-safe base64 encoded; OAuth2 Proxy decodes it to a 32-byte AES key. Each site has a different key, and a site key does not disclosesession.secret_key, which also protects the Hub session and the Redis payload signature. Thesite-oauth2container reads it fromOAUTH2_PROXY_COOKIE_SECRET. -
OIDC client secret: the volume
oidc-client-secretprojects only theOIDC_CLIENT_SECRETkey ofoidc-secretsas a file.site-oauth2mounts it read-only and reads it throughclientSecretFile. The value must not end with a newline, because OAuth2 Proxy does not trim the file. -
OIDC client values for the proxy: the
site-proxycontainer gets onlyOIDC_CLIENT_IDandOIDC_CLIENT_SECRET, each as an environment variable with asecretKeyRefto one key ofoidc-secrets. Its Lua code uses them for token introspection.
No site container gets the whole oidc-secrets Secret through envFrom.
The site-api, site-admin, site-webdav, site-s3, site-mirror,
site-initializer, and custom application containers get no value from
oidc-secrets. The custom application container also has no site-config
or oidc-client-secret mount (site/tests/test_oidc_secret_scope.py).
A rotation of session.secret_key changes every site cookie secret at the
next site update; users of private sites sign in again.
Discovery metadata service¶
core/src/ecmwf/sites/core/service/oidc_discovery.py is shared by the Hub
and the operator. It fetches metadata only when an operation needs it.
flowchart TD
A["need metadata"] --> B{"in process memory?"}
B -- yes, unexpired --> U["use it"]
B -- "expired, fetch fails" --> S["keep last good, warn, retry after cache.timeout_hot (300 s)"]
B -- no --> C{"in Redis (oidc-discovery_<sha256 key>)?"}
C -- yes --> V{"re-validates (issuer match, URL checks)?"}
V -- yes --> U
V -- no --> P
C -- no --> P["fetch discovery URL (5 s timeout, TLS verified)"]
P -- "issuer != issuer_url" --> E["OIDCDiscoveryError"]
P -- ok --> M["cache in memory + Redis, TTL cache.timeout_cold (7 d)"]
M --> U
Details that matter when changing this code:
- The memory cache key is
(discovery URL, expected issuer). Redis stores the same entry under a SHA-256 key. Redis data is never trusted before re-validation. - The returned
issuermust equalissuer_urlexactly. Endpoint values must be HTTP(S) URLs without userinfo or fragments. - A 60-second clock-skew tolerance applies to expiry checks.
- If all required endpoints are configured explicitly, the service is not
called. Providers may omit
introspection_endpointorend_session_endpoint; the platform requires them only for operations that use them. - Failed fetches are never cached. Discovery failures stop the operation that needs metadata; they never bypass authentication.
OAuth2 Proxy endpoint wiring¶
OAuth2 Proxy normally runs its own native discovery from the issuer. When
authorization_url, token_url, user_info_url, or jwks_url is
overridden, the operator disables native discovery and exports the
resolved URLs (OIDC_SKIP_DISCOVERY, OIDC_AUTHORIZATION_URL,
OIDC_TOKEN_URL, OIDC_USER_INFO_URL, OIDC_JWKS_URL in the site files
ConfigMap; core/.../model/site_exporter.py). OAuth2 Proxy still verifies
tokens and the issuer. An introspection_url override alone does not
disable native discovery; Lua reads OIDC_INTROSPECTION_URL, which the
operator exports when the URL is explicitly configured or discovery
provides it — if neither supplies it, per-site export fails. Lua performs
no discovery itself. When a JWT fails introspection, the proxy caches the
failure negatively for cache_ttl (150 s) and redirects the browser to the
Site's s/oauth2/sign_out?rd=<site_path> path (proxy/lua/jwt.lua).
Hub logout reads the stored OIDC credentials under oidc_<sub> in Redis. If
they contain an ID token, the platform calls the provider's end-session
endpoint: the discovered URL with an appended id_token_hint query
parameter, or, when oidc.logout_url is configured explicitly, that value
rendered as a sandboxed Jinja template with id_token available to the
template. A provider failure is logged and never blocks the local logout:
the local token cookie and the Flask session are always cleared, and the
browser is redirected to /hub/. The Redis credentials entry is not
deleted; it expires after cache.timeout_cold (default 7 days). The
account_url setting is unrelated to logout; it is the account page that
the Hub UI links to.
OAuth2 Proxy identity wiring¶
The per-site OAuth2 Proxy config (site/.../oauth2/oauth2-proxy/) defines
how a browser session becomes an identity the proxy Lua layer can use:
- Browser JWT to Lua:
injectRequestHeadersrewrites theAuthorizationheader with the session'saccess_tokenclaim, prefixedBearer, andpreserveRequestValue: truekeeps a caller-supplied header (master token or HTTP access token) when the session carries no claim. This is how the Lua introspection path receives the provider JWT for browser requests (06 — Authentication modules). - User header:
X-Remote-Useris set from thepreferred_usernameclaim; the proxy trusts it under the same rule as SEC-11. - Claim mapping: groups come from
entitlements, the user ID frompreferred_username, the email fromemail; audience validation acceptsaudandazpclaims and the extra audiencesites;insecureAllowUnverifiedEmailisfalse. - Email domains:
email_domains = ["*"]accepts any provider-verified email. The proxy does not filter people by domain: access decisions come from the user and group lists on the Site resource and the Lua checks. - CORS preflight:
skip_auth_preflightis on, so browserOPTIONSrequests pass without a login even on private Sites. Preflight carries no cookies and reads no content; the real request is still authenticated. - Sessions: the proxy cookie is named
s, scoped to the site path on the platform host,Secure,HttpOnly, with per-request CSRF cookies.
Changing OIDC endpoints¶
After changing the effective endpoints: apply the Helm change, delete
oidc-discovery_* keys from Redis, restart Hub and operator, reconcile the
affected sites, and restart site pods that still hold old values. The full
operational procedure is in
sys-admin: secrets and integrations.
Accepted risks¶
These risks are accepted under the trust model. Each one needs a trusted actor (a site administrator or a platform operator), or it has a documented mitigation.
- The optional
site-customcontainer runs tenant code in the same Pod as the Site API. It shares the Pod loopback address, so it can call the API with no authentication. A NetworkPolicy cannot stop loopback traffic. This is an accepted risk: the container already mounts the site volume and receives the site Secret, so the API gives it no additional access. The API confines every path to the site root. - Sites share one origin. All sites use the host
routing.host, and only the path makes them different. The OAuth2 Proxy cookieshas the site path as itsPath, and the WebDAV cookies have<site path>s/. A cookie path is not a security boundary in a browser. A script on one site can send a request to another site, and the browser adds the visitor's cookies for that site. The script can read the response and call the Site API of the other site as the visitor, because the proxy accepts the cookiesas a credential.HttpOnlystops the theft of the cookie value, but it does not stop these requests. All sites also sharelocalStorageandsessionStorage. The attack works only against a visitor who has an active session on the other site. A site administrator must publish the script, so the trust model accepts this risk. One host per site would remove it, because each site would then be a separate origin. Sites does not implement this (14 — Routing). - The chart default
routing.hubHost: ""keeps the Hub and the websites on one origin, so a website script can act with the Hub session. This is an accepted, documented risk for local development and tests. Production installations must setrouting.hubHost. The chart release notes (templates/NOTES.txt) and the Hub start log warn when it is empty. - The proxy reads the JWT claims (
preferred_username,policiesorentitlements) from the token payload without a local signature check. It uses the claims only after the IdP introspection endpoint returnsactive: true. The IdP introspects the full token string, thus a token with a changed payload is not active. The proxy does not use the claims of the introspection response, because IdPs do not always return custom claims such asentitlementsthere (proxy/lua/jwt.lua). This is an accepted design. - There is no rate limiter in the platform. Documented mitigations are the negative auth cache (150 s), bounded Go caches (3 min TTL), bounded token-history queue, and NGINX connection settings.
- A signed Redis cache entry can set class attributes. The restricted
unpickler of the Redis cache controls which classes an entry can name, but
the C unpickler does not let it check the
BUILDopcode. Thus a signed entry can set attributes on an allowed class, for exampleSiteStatus. The change stays in the reading process until it restarts. To make such an entry, an attacker needs write access to Redis and the session secret key, because each entry must have a valid HMAC signature. The same key signs the Hub session cookie, so this attacker can already sign in to the Hub as an administrator. The risk thus adds little. The Redis NetworkPolicy allows only Hub and Operator Pods to connect to Redis. A pure-Python unpickler or a JSON format would remove the risk, but each cached read would then be 4 to 9 times slower (13 — Data storage).
Hardening details¶
- The platform workloads (the Hub and its initializer, the operator, Redis,
and the CronJobs) run as non-root users with the
RuntimeDefaultseccomp profile, a read-only root filesystem, no Linux capabilities, and no privilege escalation (sites.podSecurityContextandsites.containerSecurityContextinhelm/ecmwf-sites/templates/_helpers.tpl). The Python containers write temporary files to anemptyDirat/tmp. The Hub image builds the asset bundles at build time, and the Hub keeps the webassets cache in/tmp/webassets-cache. Redis runs as UID 999, because the chart command replaces the image entrypoint that changes to theredisuser. The chart sets nofsGroup, so that Kubernetes does not change the ownership of the data volume each time that a pod mounts it. - The Hub ClusterRole can delete PersistentVolumes, because the admin storage
page removes the volumes of deleted Sites. RBAC cannot limit the
deleteverb by label. The chart adds theValidatingAdmissionPolicysites-hub-volume-deletion(helm/ecmwf-sites/templates/rbac-hub.yaml). It lets thesites-hubServiceAccount delete a volume only when the volume has the labelsites.ecmwf.int/app: site, itsclaimRefnamespace is the release namespace, and its phase isReleased. Thus a stolen Hub token cannot delete Bound volumes or volumes that Sites does not own. The chart renders the policy only when the cluster servesadmissionregistration.k8s.io/v1ValidatingAdmissionPolicy(Kubernetes 1.30 or later). On older clusters the RBAC rule is the only limit. - Signed URLs and session URLs are capability URLs: the
signaturequery value and the/s/session/<token>/path segment give access without a login. The logs do not record these values. The proxy access log writesREDACTEDin their place in therequestandrefererfields (proxy/nginx/nginx.conf). The Site API request log writessignature=REDACTED(api/api/logger.go). The OAuth2 Proxy request log does not record the request URI (request_logging_formatinoauth2-proxy/oauth2.cfg). - The proxy compares the master token in constant time. It compares
HMAC-SHA1 digests that use the master token as the key, and it reads every
byte of the digests (
Hex64.constant_time_equalsinproxy/lua/hex_64.lua). - The OAuth2 Proxy error page never shows debug details
(
show_debug_on_error = "false"inoauth2-proxy/oauth2.cfg), also when the platformdebugsetting is on. The container log has the details. - CSRF: OAuth2 Proxy uses per-request CSRF cookies. The Hub enforces a
session-bound CSRF token (
hub/csrf.py): an unsafe request that arrives with the Hub session cookie must present the token in theX-CSRF-Tokenheader or thecsrf_tokenform field, or it is rejected with 403. Bearer requests and requests without the session cookie are exempt. Session cookies areSameSite=Lax, a second layer of defence. - Session cookies are
Secure,HttpOnly,SameSite=Laxthroughout (OAuth2 Proxy:cookie_secure/cookie_httponlyinoauth2.cfg; Hub:SESSION_COOKIE_SECURE/SESSION_COOKIE_SAMESITEincore/config/flask.pyplus the FlaskHttpOnlydefault). - Identity reaches backends only through the proxy-controlled
X-Remote-Userheader: the server context resets$x_remote_userto the empty string on every request (server.conf) andproxy.confalways sendsX-Remote-User: $x_remote_user, so a client-forged header never reaches a site application. Only authentication handlers set the variable: JWT sessions contribute the username, Branca and session-URL tokens their identifier, master-token and signed-URL requests the fixed valuesmaster-tokenandSignedUrl, and anonymous public reads the valueanonymous. The same access log records the client-supplied header (user="$http_x_remote_user"), so that log field is forgeable and is not an audit trail. JWT validation additionally setsX-Remote-User,X-Remote-User-Name, andX-Remote-User-Adminon the response; the WebDAV location strips those headers from upstream responses (webdav.conf). - Master token comparisons avoid raw string timing: the Lua check compares
HMAC-SHA1 digests keyed by the master token (
hex_64.lua; the code comment explains why digest timing does not leak the token), and the Hub compares withhmac.compare_digest(core/.../auth/authn.py). - S3 proxying strips the issued credentials before the request reaches the
backend application:
s3.confsetsAuthorizationandX-Amz-Security-Tokento empty. - The Hub
session.permanent_lifetimevalue is currently inert: no code path setssession.permanent, so Hub sessions are browser-session cookies. Turning the 72 h lifetime on is a deliberate behaviour change, not a bug fix.
Behaviour requirements¶
| ID | Requirement | Implemented in | Verified by |
|---|---|---|---|
| SEC-01 | The Go API performs no authentication. It binds only 127.0.0.1:8090; only the proxy routes to it, over the pod loopback address, and the liveness probe runs /api healthcheck in the container. The site NetworkPolicy (when enabled) permits ingress only on port 4180, from the routing namespace, the monitoring namespace, the metrics-gateway namespace, and same-namespace pods. |
site/.../api/main.go, site/.../api/app_config.py, proxy/nginx/api.conf, helm/ecmwf-sites/templates/networkpolicy-site.yaml |
api/main_test.go (TestListenAddressIsLoopbackOnly, TestHealthcheck), site/tests/test_api_app_config.py (exec probe), core/tests/test_helm_configuration.py (test_site_network_policy_opens_only_the_oauth2_proxy_port) |
| SEC-02 | Public API requests into a .htpasswd-protected tree return 404 and are pruned from listings, including Repr-Digest. |
site/.../api/api/public_files.go, file_utils.go |
api/public_files_test.go (filtering and pruning cases), api/file_metadata_handlers_test.go (404, digest present/absent, including public responses) |
| SEC-03 | Anonymous requests on public sites are limited to GET, HEAD, OPTIONS, PROPFIND, SEARCH. |
proxy/lua/public_auth.lua |
— |
| SEC-04 | The health location allows the pod loopback address and trusted_ips only; /s/metrics allows trusted_ips only; an empty list denies all metrics requests. |
proxy/nginx/health.conf, metrics.conf |
site/tests/test_proxy_app_config.py (health render) |
| SEC-05 | OIDC discovery validates the issuer exactly, caches successes in memory and Redis with TTL 7 d, and never caches failures. | core/.../service/oidc_discovery.py |
core/tests/test_oidc_discovery.py |
| SEC-06 | Any of the four endpoint overrides (authorization, token, user_info, jwks) switches OAuth2 Proxy from native discovery to exported URLs. | core/.../model/site_exporter.py, oauth2/app_config.py |
core/tests/test_oidc_site_export.py |
| SEC-07 | CSP frame-ancestors is derived from trusted_domains (domain + subdomains); an empty list yields 'self' only. |
_helpers.tpl, routing_rules.py, cors.conf |
site/tests/test_proxy_app_config.py (site-proxy component) |
| SEC-08 | Failed token validations are cached negatively for 150 s; successful Branca decodes at most 10 s. | proxy/lua/auth_utils.lua, auth_cache.lua |
site/tests/test_proxy_lua_branca.py, test_proxy_lua_jwt.py (cache behaviour; the production 150 s value is stubbed in tests) |
| SEC-09 | The download-folder shortcut applies the site type's access module; public sites use the Go public router, so .htpasswd-protected trees answer 404, and private sites require the site authentication chain. |
proxy/nginx/api.conf |
site/tests/test_proxy_app_config.py (render wiring for both site types) |
| SEC-10 | A Hub request with an unsafe method and the session cookie must present the session CSRF token via the X-CSRF-Token header or the csrf_token form field, compared with hmac.compare_digest; failures answer 403. Safe methods, bearer requests, and cookie-less requests are exempt. |
hub/csrf.py |
hub/tests/test_csrf.py |
| SEC-11 | The X-Remote-User header sent to site applications is proxy-controlled: the server context resets the identity variable on every request and the proxy layer always overwrites the header, so client-forged values never reach a backend; only authentication handlers set it. |
proxy/nginx/server.conf, proxy.conf |
— (the render rule is not asserted) |
| SEC-12 | Master token checks avoid raw timing comparisons: Lua compares HMAC-SHA1 digests keyed by the master token; the Hub compares with hmac.compare_digest. |
proxy/lua/hex_64.lua, core/.../auth/authn.py |
site/tests/test_proxy_lua_hex64.py (digest comparison), core/tests/test_master_token_compare.py |
| SEC-13 | S3 requests reach the backend application without the issued credentials: the proxy blanks Authorization and X-Amz-Security-Token. |
proxy/nginx/s3.conf |
— (stripping is not asserted; test_s3_app_config.py covers the app config) |
| SEC-14 | WebDAV basic-auth sessions are remembered for 1 hour in a Secure, HttpOnly, SameSite=Lax cookie scoped to the site; on expiry the proxy rotates the realm nonce and forces a fresh challenge. The username and any .htpasswd file accounts are ignored: only master tokens and Branca tokens pass, and the public-site anonymous fallback requires a path with no applicable .htpasswd. |
proxy/lua/basic.lua, webdav_auth.lua |
site/tests/test_proxy_lua_basic.py (authorization decisions and .htpasswd challenge; the cookie expiry and realm rotation are not asserted) |
| SEC-15 | The files ConfigMap <id>-files holds no platform secret. The OAuth2 Proxy cookie secret is a per-site HMAC derivation of session.secret_key in the Secret <id>-oauth2, and the OIDC client secret is a read-only file from oidc-secrets; only site-oauth2 receives them. |
core/.../model/site_exporter.py (derive_site_oauth2_cookie_secret, to_oauth2_secret, to_files_config_map), site/.../oauth2/app_config.py, operator/.../site_create.py, site_update.py |
core/tests/test_site_oauth2_secret.py, site/tests/test_oauth_app_config.py, site/tests/test_custom_app_config.py, operator/tests/test_site_create.py, test_site_update.py, test_site_create_handler.py |
| SEC-16 | Configuration loading fails when session.secret_key is missing or shorter than 32 UTF-8 bytes; the Redis cache never derives its signing key from an empty secret. When CACHE_REDIS_PASSWORD is set, the chart Redis server requires it, and the value never appears in the process arguments. |
core/.../config/schema/models.py (validate_session_secret_key), core/.../service/redis_cache.py (_cache_mac_key), helm/.../templates/deployment-redis.yaml |
core/tests/test_config_schema.py (TestSessionSecretKeyValidation), core/tests/test_redis_cache_load_object.py, core/tests/test_helm_configuration.py (Redis password tests) |
| SEC-17 | The Hub (with its initializer), operator, Redis, and CronJob pods run as non-root (UID and GID 65534; Redis 999) with the RuntimeDefault seccomp profile. Every container has a read-only root filesystem, drops all capabilities, and does not allow privilege escalation. The Python containers get an emptyDir at /tmp. The Hub image contains the built asset bundles, and the Hub does not write to its package directory. |
helm/ecmwf-sites/templates/_helpers.tpl, deployment-hub.yaml, deployment-operator.yaml, deployment-redis.yaml, cronjob-*.yaml, hub/.../assets.py, hub/.../server.py, hub/Dockerfile |
core/tests/test_helm_configuration.py (test_platform_pods_run_hardened), hub/tests/test_assets.py, utilities/tests/dockerfile-policy.test.sh |
| SEC-18 | The sites-hub ServiceAccount can delete a PersistentVolume only when the volume has the label sites.ecmwf.int/app: site, its claimRef namespace is the release namespace, and its phase is Released. The admin storage page lists only these volumes. The chart renders the admission policy only when the cluster serves ValidatingAdmissionPolicy. |
helm/ecmwf-sites/templates/rbac-hub.yaml (sites-hub-volume-deletion), core/.../kubernetes/kubernetes_dao.py (get_released_volumes) |
core/tests/test_helm_configuration.py (test_hub_volume_deletion_policy), core/tests/test_kubernetes_api_dao.py (test_get_released_volumes_keeps_only_site_volumes_of_the_namespace) |
| SEC-19 | The request logs do not record capability URL credentials. The proxy access log and the Site API request log replace the signed URL signature value and the session URL token with REDACTED. The OAuth2 Proxy request log does not record the request URI. |
proxy/nginx/nginx.conf (log_format main), api/api/logger.go (GinLoggerFormatter), oauth2-proxy/oauth2.cfg (request_logging_format) |
site/tests/test_proxy_app_config.py (test_access_log_redacts_signed_url_signatures_and_session_tokens), site/tests/test_oauth_app_config.py (test_request_log_does_not_contain_the_request_uri), api/api/logger_test.go |
| SEC-20 | The proxy compares the master token digest in constant time. The OAuth2 Proxy error page never shows debug details. | proxy/lua/hex_64.lua (constant_time_equals), oauth2-proxy/oauth2.cfg (show_debug_on_error) |
site/tests/test_proxy_lua_hex64.py (test_constant_time_equals_reads_every_byte), site/tests/test_oauth_app_config.py (test_error_page_never_shows_debug_details) |
| SEC-21 | Site containers do not show the release namespace or the Kubernetes API server address. Every site container and init container sets NAMESPACE and the KUBERNETES_* variables of the kubernetes Service to empty values. Kubelet always injects the KUBERNETES_* variables, also with enableServiceLinks: false, and a container variable with the same name replaces them. The masked variables come after extra_env, and the Hub form rejects these names in extra_env. This does not block network access: site pods can still connect to the API server on TCP 443, but they have no service-account token. |
core/.../config/constants.py (SITE_MASKED_ENVIRONMENT_NAMES), core/.../model/site_app_config.py (_container, masked_env_var_list), site/.../initializer/app_config.py, site/.../mirror/app_config.py, core/.../model/site_exporter.py (SITE_PLATFORM_ENVIRONMENT_NAMES) |
site/tests/test_site_container_environment.py, site/tests/test_mirror_app_config.py, core/tests/test_site_applications_validation.py (test_masked_variables_are_reserved) |
| SEC-22 | The Hub pages load scripts, styles, and fonts only from the Hub origin. The Hub CSP allows only 'self' (and data: images); it blocks plugins and other base URLs. The image build installs the frontend packages from the npm lockfile (exact versions, sha512 integrity) without install scripts, and copies only the allow-listed files. |
hub/frontend/ (package-lock.json, vendor-files.json, copy-vendor.mjs), hub/Dockerfile, hub/.../routing_rules.py (HUB_CSP_SOURCES), helm/.../templates/ingress-minion.yaml, hub/.../templates/ |
hub/tests/test_vendor_assets.py, hub/tests/test_routing_rules.py (test_gateway_csp_allows_only_the_hub_origin, test_ingress_minion_snippet_has_the_hub_csp), hub/frontend/tests/copy-vendor.test.mjs, utilities/tests/dockerfile-policy.test.sh |