Site S3¶

Path shortcuts: site/.../applications/ is site/src/ecmwf/sites/website/applications/; Lua and nginx paths are relative to site/.../applications/proxy/.

site-s3 is an optional S3-compatible application on port 6000 under <site-path>s/s3. Its filesystem backend is rooted at /www/html. The operator exports the s3proxy contract to the container: S3PROXY_ENDPOINT on http://0.0.0.0:6000, S3PROXY_AUTHORIZATION=none, S3PROXY_SERVICE_PATH=<site-path>s/s3, JCLOUDS_PROVIDER=filesystem, and the filesystem base directories JCLOUDS_FILESYSTEM_BASEDIR and JCLOUDS_FILESYSTEM_NIO2_BASEDIR (s3/app_config.py).

The client authenticates at the proxy: s3_auth.lua runs the authentication chain over the X-Amz-Security-Token request header with no group requirements, so any valid Site token passes. The proxy then clears both the Authorization and X-Amz-Security-Token headers before forwarding (nginx/s3.conf). The application therefore runs without any authentication material, and it receives no public route outside the Site proxy.

Use cases¶

Tools that already speak S3: dataset uploaders, s5cmd, rclone, the Amazon Web Services (AWS) CLI, and scripts with an S3 SDK.
Large datasets: objects stream, instead of the JSON listing of the Site API.
Distribution to readers: a reader token in the session-token slot gives read access without the WebDAV or API paths.

Example usage¶

The endpoint is the Site URL plus s/s3. Send a Site token with its type prefix in the session-token slot; curl uses the header directly:

curl -H "X-Amz-Security-Token: Master ${SITE_TOKEN}" \
  "https://<domain>/<space>/<site>/s/s3/"

The AWS CLI, s5cmd, and rclone read the same value from AWS_SESSION_TOKEN (Master <token> or Bearer <token>). A bare token fails with 403. Buckets are the top-level folders of the Site content (end-user WebDAV and S3 guide).

Request flow¶

flowchart LR
    C["S3 client (X-Amz-Security-Token)"] --> P["site-proxy s3_auth.lua"]
    P --> S["site-s3 :6000 (s3proxy)"]
    S --> V[("Site content /www/html")]