Site proxy design¶
site-proxy is the OpenResty gateway inside each Site Pod. It serves the
default listing UI, applies request policy, and routes traffic to local
applications. Its Python configuration class renders the NGINX and HTML
files that the container uses.
Configuration assembly¶
site/src/ecmwf/sites/website/applications/proxy/app_config.py renders the
files ConfigMap with Jinja: 23 NGINX files (all proxy/nginx/ templates,
including custom-mime.types) and 3 HTML files (header.html,
footer.html, retention-banner.html) — 26 keys total. The two Jinja
environments differ: NGINX templates render without autoescaping; HTML
templates render with autoescaping on. The rendered header.html comes from
the template header-proxy.html, which includes the shared listing header
(also used by the WebDAV listing) and adds the hidden index title.
The container mounts the rendered .conf files into
/usr/local/openresty/nginx/conf/, server.conf into
/etc/nginx/conf.d/, and the listing templates into
/opt/proxy/html/template/. Every mount is read-only except location.conf
(app_config.py, extra_volume_mounts). The fancyindex module reads
header.html and footer.html from /opt/proxy/html/template/ one time,
when NGINX starts (fancyindex_header ... local). On startup,
bin/entrypoint.sh removes a legacy /www/template directory or symlink,
exports the image BUILD_DATE, and starts NGINX in the foreground with
nginx.conf.
The renderer provides these inputs:
| Input | Use |
|---|---|
site_type |
Select public or private request rules. |
site_path |
Prefix routes and the Site content root. |
configuration.applications |
Add locations for optional and custom applications. |
configuration.headers |
Configure CORS, cache, robots, and response headers. |
configuration.settings.error_handling |
Configure custom error redirects. |
trusted_ips |
Restrict health and metrics endpoints. |
trusted_domains |
Build CORS and CSP allowlists. |
proxy_app and context_root |
Select the custom backend integration. |
proxy_header, log_level, and DOCUMENT_ROOT |
Set proxy identity, logging, and FastCGI values. |
hostname, metrics_enabled, site_type (type) |
Set the server name, and select the metrics and public/private configuration. |
js_code, css_code |
Minified listing scripts and styles, inlined into the listing templates (core/.../misc/asset_utils.py). |
The renderer validates the values that become NGINX settings.
remove_unsafe_nginx_settings deep-copies the configuration (the Site
resource is never mutated), removes invalid entries, and logs one WARNING
per affected key: Site <path>: ignored N invalid value(s) in <key>. The
validated lists are then used for rendering. This also protects the proxy
when a Site resource was changed directly in Kubernetes. The accepted
formats are (core/.../config/constants.py):
| Setting | Format | Example |
|---|---|---|
settings.error_handling.custom_errors[] |
A code 300–599, =, then a path starting with / or an http(s) URL. |
404=/errors/404.html |
headers.no_cache.file_extensions[] |
1–32 letters, digits, ., _, or -; first character a letter or digit. |
tar.gz |
headers.cors.extra_domains[] |
A host name with an optional port; no scheme, path, or wildcard. | www.example.org |
The env declarations in nginx.conf pass these process variables to the
Lua modules: OIDC_CLIENT_ID, OIDC_CLIENT_SECRET,
OIDC_INTROSPECTION_URL, AUTHORIZED_ADMIN_USERS,
AUTHORIZED_ADMIN_GROUPS, AUTHORIZED_STAFF_USERS,
AUTHORIZED_STAFF_GROUPS, MASTER_TOKEN, RETENTION_DATE, SITE_PATH,
SITE_TYPE, and IMAGE_TAG.
Request path¶
flowchart LR
R["Ingress or HTTPRoute"] --> O["OAuth2 Proxy :4180"]
O --> P["OpenResty proxy :8080"]
P --> A["Go API :8090"]
P --> U["Admin :9000"]
P --> W["WebDAV :7000"]
P --> S["S3 :6000"]
P --> C["Custom application :8000"]
P --> F[("Static Site files")]
Browser traffic normally enters OAuth2 Proxy first and holds an OAuth2
session there. The API, WebDAV, S3, _m2m, and signed or session URL
traffic uses the skip-auth list of site-oauth2: it crosses the
site-oauth2 hop unauthenticated (the Site route points only at that
container; nothing exposes site-proxy as a Service port — it is reached at
127.0.0.1:8080 inside the Pod) and is authenticated by the proxy's own Lua
modules. The proxy selects the authentication module and backend from the
request path.
Always-on proxy behaviour¶
The proxy always:
- listens on port
8080withserver_nameset to the site's public host (absolute_redirect off,server_tokens off); - serves
/www/htmland the generated directory-listing templates (index index.html); - includes the API, admin, WebDAV, S3, health, metrics, CORS, error, and main Site location configuration files;
- writes access logs to stdout in the
mainformat (which records theX-Forwarded-Forclient and theX-Remote-Useridentity, and writesREDACTEDin place of the signed URLsignaturevalue and the session URL token in the request and referer fields) and error logs to stderr at the platformLOG_LEVEL; the health and metrics locations setaccess_log off, and the metrics log phase also skips their URIs; - disables response buffering on the way to and from backends
(
proxy_buffering off,proxy_request_buffering off,proxy_cache off) and uses sendfile, TCP push, and TCP nodelay; - sizes body and header buffers at 512 KiB and 4×64 KiB;
- supports proxy and FastCGI timeouts of up to 600 seconds;
- compresses responses with gzip (minimum 1 KiB, level 5,
Varyset, listed text/font/SVG types), and serves pre-compressed.gzfiles (gzip_static) when they exist; - overrides the response charset to UTF-8 and accepts underscores in request
header names (
underscores_in_headers); - keeps client connections alive for 30 seconds or 1000 requests, and
upgrades Websocket connections to the backend
(
$connection_upgrademap); - forwards
X-Real-IP,X-Forwarded-For/Proto/Host/Port, andX-Remote-Userto backends (proxy.conf), always declaring the external scheme ashttpsand port443; - follows a symlink in the Site content only when the link and its target
have the same owner (
disable_symlinks if_not_owner from=$document_root). The site containers create links and files as UID 65534, so links between site files work. A link to a file of another owner, for example/etc, a mounted Secret, or a file thatrootcopied into the volume, gives 403. This also closes the race between a check and the open of the file, because NGINX checks each path component when it opens it; - resolves names with the pod's own resolver (
resolver local=on ipv6=off); - uses the
prometheus_metricsLua shared dictionary (10 MiB) for metrics data when the metrics integration is active; - serves the custom MIME type map
nginx/custom-mime.types: ECMWF data formats (grib,grib1,grib2,bufr,odb,tfrecord) andcsvastext/plain; - verifies TLS certificates for Lua HTTPS requests with the system CA bundle
(
lua_ssl_trusted_certificate, depth 2).
The proxy does not impose a client body size limit. Upload limits and backpressure are enforced by the Go API.
Site type scenarios¶
| Scenario | Public Site | Private Site |
|---|---|---|
| Anonymous content | Allowed for permitted read methods | OAuth2 session is required |
.htpasswd mapping |
Enabled per directory | Not used for the public router |
| Signed URL map | Not used | Enabled for /s/signed/r/ and /s/signed/rw/ |
| Session URL map and rewrite | Not used | Enabled. A request to s/session/<token>/… is rewritten by method: read-only methods (GET, HEAD, OPTIONS, PROPFIND, SEARCH) go to the Site browsing path, other methods to the session-files API path (map $request_method $session_rewrite_prefix in nginx.conf, rewrite in server.conf) |
Machine-to-machine prefix _m2m |
Not special-cased: the rewrite is rendered only for private Sites | Stripped by rewrite so the normal private locations and auth apply (server.conf) |
| Auth error format | /s/api/vN, /s/_m2m, and /s/s3 paths answer authentication failures as JSON (map $want_json matches the versioned API prefix); other paths use the HTML/redirect treatment |
Same rule |
| CORS | Rendered from the configured allowlist | Rendered from the configured allowlist |
| Content Security Policy | 'self' plus configured trusted domains; adds CORS extra_domains only when CORS is enabled |
'self' plus configured trusted domains; adds CORS extra_domains only when CORS is enabled |
| Default errors | Hub error redirects only when custom handling and CORS are both off | Hub error redirects only when custom handling and CORS are both off |
The authentication chain and status-code rules are in security design. Public API path hiding is also enforced by the Go public router.
Backend selection¶
The default Site path serves static content. A configured site-custom
application changes the backend:
proxy_type |
Generated configuration | Upstream path |
|---|---|---|
| absent or default | Static alias | /www/html/ |
proxy |
proxy_pass to 127.0.0.1:8000 |
Site path when context_root=site_context; / otherwise |
fastcgi |
fastcgi_pass to 127.0.0.1:8000 |
FastCGI request parameters and configured document root |
The full custom application contract is in custom applications.
Endpoint locations¶
| Location | Configuration | Purpose |
|---|---|---|
/s/api/ |
api.conf |
Forward Site API requests through the admin authorization module; return JSON auth errors. |
/s/api/v2/session-files/ |
api.conf |
Forward session-URL file operations through the private authorization module (site administrators and staff pass; admin_users/staff_users and their groups). |
.../download-folder |
api.conf |
GET/HEAD/OPTIONS folder-zip shortcut for the listing UI; applies the site type's access module, then forwards with recursive=false&list=false&archive=zip — through the public router on public sites (protected trees answer 404), through the authenticated router on private sites. Other methods receive 405. |
/s/api/v2/pub/ |
api.conf |
Public-site anonymous API; rendered only for public Sites. |
/s/admin |
admin.conf |
Forward the File Browser UI after admin authorization. |
/s/dav |
webdav.conf |
Forward WebDAV requests after WebDAV authorization; the bare path without trailing slash answers 301 to /s/dav/. |
/s/s3 |
s3.conf |
Forward S3 requests after S3 authorization. |
/<space>/<site>/s/healthz |
health.conf |
NGINX stub status; loopback and the configured trusted_ips only. |
/s/metrics |
metrics.conf |
Prometheus metrics when the metrics integration is active. This path is global, not site-scoped. |
| Site content | location.conf |
Static content, custom backend, headers, and error handling. |
The health location always allows the pod loopback address, so the Go API
probe passes whatever trusted_ips holds; all other sources need
trusted_ips. An empty trusted_ips list denies metrics requests. The
metrics allowlist evaluates the X-Forwarded-For real IP; the health
location uses the direct peer address. The metrics configuration is not
rendered when the metrics integration is disabled (see
11-site-api).
Authentication modules¶
Each location loads one entry module with access_by_lua_file. The modules
and their policies (proxy/lua/):
| Module | Used by | Policy |
|---|---|---|
public_auth.lua |
content on public Sites | retention check, then restricts to read-only methods (GET, HEAD, OPTIONS, PROPFIND, SEARCH); other methods answer 403 |
private_auth.lua |
content on private Sites, session-files API | retention check, then the authentication chain with the admin_users/staff_users and group policies |
admin_auth.lua |
/s/api/ and /s/admin/, except the session-files and pub locations of api.conf, which match on longer prefixes |
chain with the admin_users/admin_groups policy |
webdav_auth.lua |
/s/dav/ |
BasicAuth.handle(true): Basic password must be the master or a Branca token; public read-only fallback (05) |
s3_auth.lua |
/s/s3/ |
chain with no group or user requirement, reading the token from X-Amz-Security-Token |
All chain calls share one token extraction path: the configured header
(Authorization by default, flattened by OAuth2 Proxy into
<token>,Bearer <access-token> — the first comma-separated part wins;
X-Amz-Security-Token for S3), then the OAuth2 session cookie s, then the
BasicAuth cookie basic. Typed prefixes Bearer, Branca, and Master
select the token kind inside the header. The chain order, JWT
introspection, and the authentication caches (150-second negative entries,
at most 10-second positive Branca entries) are specified in
10-security; this page only maps modules to locations.
Site browsing user interface¶
When no site-custom application replaces it, the proxy serves a
file-listing UI built from the fancyindex templates
(proxy/html/template/) and browser scripts (proxy/html/js/). The
configuration renderer minifies the scripts and inlines them: styles into
header.html, scripts into footer.html
(core/src/ecmwf/sites/core/misc/asset_utils.py). The listing uses
fancyindex with date_desc sorting, human-readable sizes, and the
parent-directory entry hidden (server.conf).
The scripts cooperate through two shared primitives: 0.js (named to load
first) defines the AppReady promises for DOM and site-information
readiness, and utilities.js publishes the site information (path, space,
name, user identity, default signed/session URL durations in days, which the
header template injects from platform configuration) and the Site API
endpoint builders. Several features render as custom elements
(<user-name>, <main-menu>, <file-toolbar>).
- Top menu and breadcrumbs render the Site branding, links, and current path; the username script shows the signed-in user from the identity headers that OAuth2 Proxy injects.
- The toolbar (
file_toolbar.js, a<file-toolbar>element) adds a search box and sort controls. Search is a client-side, case-insensitive substring filter on entry names that applies to both the table and the grid view; it never calls the server. The sort controls write the fancyindex query parameters in the URL:Cfor the column (Ddate,Nname,Ssize) andOfor the order (Ddescending,Aascending), then reload the page; with no parameters the listing uses the configured defaultdate_desc. - Edit Mode (Ctrl+E) is visible to site administrators only. It adds
delete buttons to entries, a directory-creation modal, and the upload
drop area. The state persists in
localStorage. - Dropzone upload accepts files and directories by drag-and-drop and
uploads them with
PUTto the Site API, using resumable chunks (HEADplusContent-Range, 4 MiB default chunk, 4 concurrent uploads). - The right-click menu generates signed and session URLs in the browser:
it calls
s/api/v2/signed-urls/ands/api/v2/session-urls/with a scope and expiry, and copies the result to the clipboard. Default durations come from platform configuration through the header template. - Thumbnails and grid layout render previews and an alternative layout; toasts report upload and action results; a retention banner is injected when the retention date has passed.
Headers, errors, and retention¶
- Server-level
add_headerdirectives —X-Site-Proxy-Serverwith the proxy identity line (<account>; <site version>; <namespace>;,custom-headers.conf, added withalways) andX-Robots-Tagwhen no-robots is on — reach every response except the WebDAV locations:webdav.confdeclares its ownadd_headerdirectives, and nginx does not inherit server-leveladd_headerinto a location that sets them. - CORS maps and headers are rendered only when CORS is enabled;
OPTIONSthen answers 204 preflights directly. When CORS is disabled, the response instead clears any upstream CORS headers and addsStrict-Transport-Security(two years, includeSubdomains, preload) andX-Permitted-Cross-Domain-Policies: none. trusted_domainscontributes the platform domains and their subdomains to the CSP frame-ancestor policy; CORSextra_domainsextend it only when CORS is enabled.headers.no_cacheaddsexpires -1for all content or the configured file extensions.headers.no_robotsaddsX-Robots-Tag: noindex.- Custom error entries use
CODE=URL_OR_PATH; root-relative targets receive the Site path prefix; each entry becomes anerror_pageto a named location that answers 302. Without custom handling and without CORS, the defaults redirect 403/404/405/410/500/502/503 tohttps://$host/hub/error/page/<code>/. - A request for
favicon.icounder the Site path or the admin UI redirects (301) to the platform-root favicon, which the Hub serves from the branding (favicon.conf, included bylocation.confandadmin.conf). - Retention: the retention module runs in every content authentication path.
It always sets
X-Site-Retention-Date-Expiredtotrueorfalse(result cached 150 seconds per worker). When the date has passed, it loads the brandedretention-banner.htmlfrom the files ConfigMap and nginxsub_filterinjects it before the closing</html>tag of responses. A missing or empty banner file logs an ERROR and injects nothing; an invalidRETENTION_DATElogs a WARNING and behaves like no date (lua/retention_date_notification.lua).
Behaviour requirements¶
| ID | Requirement | Implemented in | Verified by |
|---|---|---|---|
| PRX-01 | The proxy renders NGINX and listing-template files from the Site and platform configuration. | site/.../applications/proxy/app_config.py, proxy/nginx/, proxy/html/ |
site/tests/test_proxy_app_config.py |
| PRX-02 | Invalid values that would become NGINX settings are removed before configuration is rendered. | proxy/app_config.py (remove_unsafe_nginx_settings) |
site/tests/test_proxy_app_config.py |
| PRX-03 | Public and private Sites use different request maps for .htpasswd, signed URLs, session URLs, and anonymous methods. |
proxy/nginx/, proxy/lua/ |
Lua tests (public_auth.lua itself is not covered) |
| PRX-04 | API, admin, WebDAV, S3, health, and metrics requests use separate locations and authentication handlers. | proxy/nginx/*.conf, proxy/lua/*.lua |
site/tests/test_proxy_app_config.py, Lua tests (per-location wiring is not asserted) |
| PRX-05 | The default content backend serves /www/html, while custom applications select proxy or FastCGI integration. |
proxy/nginx/location.conf, proxy/nginx/*-app.conf |
— |
| PRX-06 | The health location allows the pod loopback address and trusted_ips; the metrics location allows trusted_ips only, evaluated against the real IP; an empty list denies all metrics requests. |
proxy/nginx/health.conf, metrics.conf |
site/tests/test_proxy_app_config.py (health render) |
| PRX-07 | CORS, CSP, cache, robots, custom error, and retention settings change only the generated locations that use them. | proxy/nginx/*.conf, proxy/app_config.py |
site/tests/test_proxy_app_config.py (CORS/CSP and unsafe settings; the X-Robots-Tag effect is not asserted) |
| PRX-08 | The listing UI adds administrator-only Edit Mode, dropzone uploads to the Site API, and in-browser signed and session URL generation. | proxy/html/js/, proxy/html/template/ |
— |
| PRX-09 | Authentication failures answer as JSON on the /s/api/vN, /s/_m2m, and /s/s3 paths, and with the HTML or redirect treatment elsewhere. |
proxy/nginx/nginx.conf (map $want_json), proxy/lua/auth_utils.lua |
— |
| PRX-10 | Content authentication paths always set X-Site-Retention-Date-Expired; an expired retention date injects the branded banner via sub_filter; a missing banner or an invalid RETENTION_DATE logs and degrades without injecting. |
proxy/lua/retention_date_notification.lua, proxy/nginx/server.conf, proxy/nginx/location.conf |
site/tests/test_proxy_lua_retention_date.py |
| PRX-11 | When the metrics integration is active, the proxy collects counters in the log phase and serves /s/metrics from the shared dictionary; no metrics code is rendered when it is inactive. |
proxy/lua/metrics.lua, proxy/nginx/nginx.conf, server.conf, metrics.conf |
site/tests/test_proxy_app_config.py (render wiring on and off; metrics.lua logic is not covered, see 21) |