Skip to content

Site proxy design

site-proxy is the OpenResty gateway inside each Site Pod. It serves the default listing UI, applies request policy, and routes traffic to local applications. Its Python configuration class renders the NGINX and HTML files that the container uses.

Configuration assembly

site/src/ecmwf/sites/website/applications/proxy/app_config.py renders the files ConfigMap with Jinja: 23 NGINX files (all proxy/nginx/ templates, including custom-mime.types) and 3 HTML files (header.html, footer.html, retention-banner.html) — 26 keys total. The two Jinja environments differ: NGINX templates render without autoescaping; HTML templates render with autoescaping on. The rendered header.html comes from the template header-proxy.html, which includes the shared listing header (also used by the WebDAV listing) and adds the hidden index title.

The container mounts the rendered .conf files into /usr/local/openresty/nginx/conf/, server.conf into /etc/nginx/conf.d/, and the listing templates into /opt/proxy/html/template/. Every mount is read-only except location.conf (app_config.py, extra_volume_mounts). The fancyindex module reads header.html and footer.html from /opt/proxy/html/template/ one time, when NGINX starts (fancyindex_header ... local). On startup, bin/entrypoint.sh removes a legacy /www/template directory or symlink, exports the image BUILD_DATE, and starts NGINX in the foreground with nginx.conf.

The renderer provides these inputs:

Input Use
site_type Select public or private request rules.
site_path Prefix routes and the Site content root.
configuration.applications Add locations for optional and custom applications.
configuration.headers Configure CORS, cache, robots, and response headers.
configuration.settings.error_handling Configure custom error redirects.
trusted_ips Restrict health and metrics endpoints.
trusted_domains Build CORS and CSP allowlists.
proxy_app and context_root Select the custom backend integration.
proxy_header, log_level, and DOCUMENT_ROOT Set proxy identity, logging, and FastCGI values.
hostname, metrics_enabled, site_type (type) Set the server name, and select the metrics and public/private configuration.
js_code, css_code Minified listing scripts and styles, inlined into the listing templates (core/.../misc/asset_utils.py).

The renderer validates the values that become NGINX settings. remove_unsafe_nginx_settings deep-copies the configuration (the Site resource is never mutated), removes invalid entries, and logs one WARNING per affected key: Site <path>: ignored N invalid value(s) in <key>. The validated lists are then used for rendering. This also protects the proxy when a Site resource was changed directly in Kubernetes. The accepted formats are (core/.../config/constants.py):

Setting Format Example
settings.error_handling.custom_errors[] A code 300–599, =, then a path starting with / or an http(s) URL. 404=/errors/404.html
headers.no_cache.file_extensions[] 1–32 letters, digits, ., _, or -; first character a letter or digit. tar.gz
headers.cors.extra_domains[] A host name with an optional port; no scheme, path, or wildcard. www.example.org

The env declarations in nginx.conf pass these process variables to the Lua modules: OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_INTROSPECTION_URL, AUTHORIZED_ADMIN_USERS, AUTHORIZED_ADMIN_GROUPS, AUTHORIZED_STAFF_USERS, AUTHORIZED_STAFF_GROUPS, MASTER_TOKEN, RETENTION_DATE, SITE_PATH, SITE_TYPE, and IMAGE_TAG.

Request path

flowchart LR
    R["Ingress or HTTPRoute"] --> O["OAuth2 Proxy :4180"]
    O --> P["OpenResty proxy :8080"]
    P --> A["Go API :8090"]
    P --> U["Admin :9000"]
    P --> W["WebDAV :7000"]
    P --> S["S3 :6000"]
    P --> C["Custom application :8000"]
    P --> F[("Static Site files")]

Browser traffic normally enters OAuth2 Proxy first and holds an OAuth2 session there. The API, WebDAV, S3, _m2m, and signed or session URL traffic uses the skip-auth list of site-oauth2: it crosses the site-oauth2 hop unauthenticated (the Site route points only at that container; nothing exposes site-proxy as a Service port — it is reached at 127.0.0.1:8080 inside the Pod) and is authenticated by the proxy's own Lua modules. The proxy selects the authentication module and backend from the request path.

Always-on proxy behaviour

The proxy always:

  • listens on port 8080 with server_name set to the site's public host (absolute_redirect off, server_tokens off);
  • serves /www/html and the generated directory-listing templates (index index.html);
  • includes the API, admin, WebDAV, S3, health, metrics, CORS, error, and main Site location configuration files;
  • writes access logs to stdout in the main format (which records the X-Forwarded-For client and the X-Remote-User identity, and writes REDACTED in place of the signed URL signature value and the session URL token in the request and referer fields) and error logs to stderr at the platform LOG_LEVEL; the health and metrics locations set access_log off, and the metrics log phase also skips their URIs;
  • disables response buffering on the way to and from backends (proxy_buffering off, proxy_request_buffering off, proxy_cache off) and uses sendfile, TCP push, and TCP nodelay;
  • sizes body and header buffers at 512 KiB and 4×64 KiB;
  • supports proxy and FastCGI timeouts of up to 600 seconds;
  • compresses responses with gzip (minimum 1 KiB, level 5, Vary set, listed text/font/SVG types), and serves pre-compressed .gz files (gzip_static) when they exist;
  • overrides the response charset to UTF-8 and accepts underscores in request header names (underscores_in_headers);
  • keeps client connections alive for 30 seconds or 1000 requests, and upgrades Websocket connections to the backend ($connection_upgrade map);
  • forwards X-Real-IP, X-Forwarded-For/Proto/Host/Port, and X-Remote-User to backends (proxy.conf), always declaring the external scheme as https and port 443;
  • follows a symlink in the Site content only when the link and its target have the same owner (disable_symlinks if_not_owner from=$document_root). The site containers create links and files as UID 65534, so links between site files work. A link to a file of another owner, for example /etc, a mounted Secret, or a file that root copied into the volume, gives 403. This also closes the race between a check and the open of the file, because NGINX checks each path component when it opens it;
  • resolves names with the pod's own resolver (resolver local=on ipv6=off);
  • uses the prometheus_metrics Lua shared dictionary (10 MiB) for metrics data when the metrics integration is active;
  • serves the custom MIME type map nginx/custom-mime.types: ECMWF data formats (grib, grib1, grib2, bufr, odb, tfrecord) and csv as text/plain;
  • verifies TLS certificates for Lua HTTPS requests with the system CA bundle (lua_ssl_trusted_certificate, depth 2).

The proxy does not impose a client body size limit. Upload limits and backpressure are enforced by the Go API.

Site type scenarios

Scenario Public Site Private Site
Anonymous content Allowed for permitted read methods OAuth2 session is required
.htpasswd mapping Enabled per directory Not used for the public router
Signed URL map Not used Enabled for /s/signed/r/ and /s/signed/rw/
Session URL map and rewrite Not used Enabled. A request to s/session/<token>/… is rewritten by method: read-only methods (GET, HEAD, OPTIONS, PROPFIND, SEARCH) go to the Site browsing path, other methods to the session-files API path (map $request_method $session_rewrite_prefix in nginx.conf, rewrite in server.conf)
Machine-to-machine prefix _m2m Not special-cased: the rewrite is rendered only for private Sites Stripped by rewrite so the normal private locations and auth apply (server.conf)
Auth error format /s/api/vN, /s/_m2m, and /s/s3 paths answer authentication failures as JSON (map $want_json matches the versioned API prefix); other paths use the HTML/redirect treatment Same rule
CORS Rendered from the configured allowlist Rendered from the configured allowlist
Content Security Policy 'self' plus configured trusted domains; adds CORS extra_domains only when CORS is enabled 'self' plus configured trusted domains; adds CORS extra_domains only when CORS is enabled
Default errors Hub error redirects only when custom handling and CORS are both off Hub error redirects only when custom handling and CORS are both off

The authentication chain and status-code rules are in security design. Public API path hiding is also enforced by the Go public router.

Backend selection

The default Site path serves static content. A configured site-custom application changes the backend:

proxy_type Generated configuration Upstream path
absent or default Static alias /www/html/
proxy proxy_pass to 127.0.0.1:8000 Site path when context_root=site_context; / otherwise
fastcgi fastcgi_pass to 127.0.0.1:8000 FastCGI request parameters and configured document root

The full custom application contract is in custom applications.

Endpoint locations

Location Configuration Purpose
/s/api/ api.conf Forward Site API requests through the admin authorization module; return JSON auth errors.
/s/api/v2/session-files/ api.conf Forward session-URL file operations through the private authorization module (site administrators and staff pass; admin_users/staff_users and their groups).
.../download-folder api.conf GET/HEAD/OPTIONS folder-zip shortcut for the listing UI; applies the site type's access module, then forwards with recursive=false&list=false&archive=zip — through the public router on public sites (protected trees answer 404), through the authenticated router on private sites. Other methods receive 405.
/s/api/v2/pub/ api.conf Public-site anonymous API; rendered only for public Sites.
/s/admin admin.conf Forward the File Browser UI after admin authorization.
/s/dav webdav.conf Forward WebDAV requests after WebDAV authorization; the bare path without trailing slash answers 301 to /s/dav/.
/s/s3 s3.conf Forward S3 requests after S3 authorization.
/<space>/<site>/s/healthz health.conf NGINX stub status; loopback and the configured trusted_ips only.
/s/metrics metrics.conf Prometheus metrics when the metrics integration is active. This path is global, not site-scoped.
Site content location.conf Static content, custom backend, headers, and error handling.

The health location always allows the pod loopback address, so the Go API probe passes whatever trusted_ips holds; all other sources need trusted_ips. An empty trusted_ips list denies metrics requests. The metrics allowlist evaluates the X-Forwarded-For real IP; the health location uses the direct peer address. The metrics configuration is not rendered when the metrics integration is disabled (see 11-site-api).

Authentication modules

Each location loads one entry module with access_by_lua_file. The modules and their policies (proxy/lua/):

Module Used by Policy
public_auth.lua content on public Sites retention check, then restricts to read-only methods (GET, HEAD, OPTIONS, PROPFIND, SEARCH); other methods answer 403
private_auth.lua content on private Sites, session-files API retention check, then the authentication chain with the admin_users/staff_users and group policies
admin_auth.lua /s/api/ and /s/admin/, except the session-files and pub locations of api.conf, which match on longer prefixes chain with the admin_users/admin_groups policy
webdav_auth.lua /s/dav/ BasicAuth.handle(true): Basic password must be the master or a Branca token; public read-only fallback (05)
s3_auth.lua /s/s3/ chain with no group or user requirement, reading the token from X-Amz-Security-Token

All chain calls share one token extraction path: the configured header (Authorization by default, flattened by OAuth2 Proxy into <token>,Bearer <access-token> — the first comma-separated part wins; X-Amz-Security-Token for S3), then the OAuth2 session cookie s, then the BasicAuth cookie basic. Typed prefixes Bearer, Branca, and Master select the token kind inside the header. The chain order, JWT introspection, and the authentication caches (150-second negative entries, at most 10-second positive Branca entries) are specified in 10-security; this page only maps modules to locations.

Site browsing user interface

When no site-custom application replaces it, the proxy serves a file-listing UI built from the fancyindex templates (proxy/html/template/) and browser scripts (proxy/html/js/). The configuration renderer minifies the scripts and inlines them: styles into header.html, scripts into footer.html (core/src/ecmwf/sites/core/misc/asset_utils.py). The listing uses fancyindex with date_desc sorting, human-readable sizes, and the parent-directory entry hidden (server.conf).

The scripts cooperate through two shared primitives: 0.js (named to load first) defines the AppReady promises for DOM and site-information readiness, and utilities.js publishes the site information (path, space, name, user identity, default signed/session URL durations in days, which the header template injects from platform configuration) and the Site API endpoint builders. Several features render as custom elements (<user-name>, <main-menu>, <file-toolbar>).

  • Top menu and breadcrumbs render the Site branding, links, and current path; the username script shows the signed-in user from the identity headers that OAuth2 Proxy injects.
  • The toolbar (file_toolbar.js, a <file-toolbar> element) adds a search box and sort controls. Search is a client-side, case-insensitive substring filter on entry names that applies to both the table and the grid view; it never calls the server. The sort controls write the fancyindex query parameters in the URL: C for the column (D date, N name, S size) and O for the order (D descending, A ascending), then reload the page; with no parameters the listing uses the configured default date_desc.
  • Edit Mode (Ctrl+E) is visible to site administrators only. It adds delete buttons to entries, a directory-creation modal, and the upload drop area. The state persists in localStorage.
  • Dropzone upload accepts files and directories by drag-and-drop and uploads them with PUT to the Site API, using resumable chunks (HEAD plus Content-Range, 4 MiB default chunk, 4 concurrent uploads).
  • The right-click menu generates signed and session URLs in the browser: it calls s/api/v2/signed-urls/ and s/api/v2/session-urls/ with a scope and expiry, and copies the result to the clipboard. Default durations come from platform configuration through the header template.
  • Thumbnails and grid layout render previews and an alternative layout; toasts report upload and action results; a retention banner is injected when the retention date has passed.

Headers, errors, and retention

  • Server-level add_header directives — X-Site-Proxy-Server with the proxy identity line (<account>; <site version>; <namespace>;, custom-headers.conf, added with always) and X-Robots-Tag when no-robots is on — reach every response except the WebDAV locations: webdav.conf declares its own add_header directives, and nginx does not inherit server-level add_header into a location that sets them.
  • CORS maps and headers are rendered only when CORS is enabled; OPTIONS then answers 204 preflights directly. When CORS is disabled, the response instead clears any upstream CORS headers and adds Strict-Transport-Security (two years, includeSubdomains, preload) and X-Permitted-Cross-Domain-Policies: none.
  • trusted_domains contributes the platform domains and their subdomains to the CSP frame-ancestor policy; CORS extra_domains extend it only when CORS is enabled.
  • headers.no_cache adds expires -1 for all content or the configured file extensions.
  • headers.no_robots adds X-Robots-Tag: noindex.
  • Custom error entries use CODE=URL_OR_PATH; root-relative targets receive the Site path prefix; each entry becomes an error_page to a named location that answers 302. Without custom handling and without CORS, the defaults redirect 403/404/405/410/500/502/503 to https://$host/hub/error/page/<code>/.
  • A request for favicon.ico under the Site path or the admin UI redirects (301) to the platform-root favicon, which the Hub serves from the branding (favicon.conf, included by location.conf and admin.conf).
  • Retention: the retention module runs in every content authentication path. It always sets X-Site-Retention-Date-Expired to true or false (result cached 150 seconds per worker). When the date has passed, it loads the branded retention-banner.html from the files ConfigMap and nginx sub_filter injects it before the closing </html> tag of responses. A missing or empty banner file logs an ERROR and injects nothing; an invalid RETENTION_DATE logs a WARNING and behaves like no date (lua/retention_date_notification.lua).

Behaviour requirements

ID Requirement Implemented in Verified by
PRX-01 The proxy renders NGINX and listing-template files from the Site and platform configuration. site/.../applications/proxy/app_config.py, proxy/nginx/, proxy/html/ site/tests/test_proxy_app_config.py
PRX-02 Invalid values that would become NGINX settings are removed before configuration is rendered. proxy/app_config.py (remove_unsafe_nginx_settings) site/tests/test_proxy_app_config.py
PRX-03 Public and private Sites use different request maps for .htpasswd, signed URLs, session URLs, and anonymous methods. proxy/nginx/, proxy/lua/ Lua tests (public_auth.lua itself is not covered)
PRX-04 API, admin, WebDAV, S3, health, and metrics requests use separate locations and authentication handlers. proxy/nginx/*.conf, proxy/lua/*.lua site/tests/test_proxy_app_config.py, Lua tests (per-location wiring is not asserted)
PRX-05 The default content backend serves /www/html, while custom applications select proxy or FastCGI integration. proxy/nginx/location.conf, proxy/nginx/*-app.conf —
PRX-06 The health location allows the pod loopback address and trusted_ips; the metrics location allows trusted_ips only, evaluated against the real IP; an empty list denies all metrics requests. proxy/nginx/health.conf, metrics.conf site/tests/test_proxy_app_config.py (health render)
PRX-07 CORS, CSP, cache, robots, custom error, and retention settings change only the generated locations that use them. proxy/nginx/*.conf, proxy/app_config.py site/tests/test_proxy_app_config.py (CORS/CSP and unsafe settings; the X-Robots-Tag effect is not asserted)
PRX-08 The listing UI adds administrator-only Edit Mode, dropzone uploads to the Site API, and in-browser signed and session URL generation. proxy/html/js/, proxy/html/template/ —
PRX-09 Authentication failures answer as JSON on the /s/api/vN, /s/_m2m, and /s/s3 paths, and with the HTML or redirect treatment elsewhere. proxy/nginx/nginx.conf (map $want_json), proxy/lua/auth_utils.lua —
PRX-10 Content authentication paths always set X-Site-Retention-Date-Expired; an expired retention date injects the branded banner via sub_filter; a missing banner or an invalid RETENTION_DATE logs and degrades without injecting. proxy/lua/retention_date_notification.lua, proxy/nginx/server.conf, proxy/nginx/location.conf site/tests/test_proxy_lua_retention_date.py
PRX-11 When the metrics integration is active, the proxy collects counters in the log phase and serves /s/metrics from the shared dictionary; no metrics code is rendered when it is inactive. proxy/lua/metrics.lua, proxy/nginx/nginx.conf, server.conf, metrics.conf site/tests/test_proxy_app_config.py (render wiring on and off; metrics.lua logic is not covered, see 21)