Skip to content

Site API design

The Site API is a Go (Gin) service (site/src/ecmwf/sites/website/ applications/api/) that operates on the site's storage volume. The Lua proxy exposes it under /<space>/<site>/s/api/ and enforces authentication. The OpenAPI specification in hub/src/ecmwf/sites/hub/static/specs/site-api-spec.yaml is the complete endpoint reference; hub/tests/test_api_spec_coverage.py verifies that it documents exactly the routes registered in api/main.go. For call examples, use the Site API cookbook.

Hub API and Site API boundary

The platform has two API surfaces with different responsibilities:

API Owns Typical caller
Hub API Sites, users, sharing, platform actions, upgrades, read-only state, and administration Hub UI, platform automation, sitesctl
Site API Files, directories, tokens, signed URLs, session URLs, checksums, and site health Site owners, CI jobs, content clients

The Hub API writes the Site resource. The operator then reconciles that resource into the site workload. The Site API does not create or update Site resources. The endpoint references are hub/src/ecmwf/sites/hub/static/specs/hub-api-spec.yaml and hub/src/ecmwf/sites/hub/static/specs/site-api-spec.yaml; see 12-hub-api for the Hub routes.

Hub request validation, authorization, Site actions, and administration are described in Hub API and administration.

Use the Hub API to discover or manage sites. Use the Site API for content operations. Both APIs use bearer authentication, but site requests also pass through the site proxy authentication chain described in 10-security.

Endpoints

The Go service listens on 127.0.0.1:PORT (default port 10000) with these route groups (api/main.go). Only the proxy in the same Pod can connect. The command /api healthcheck connects to that address and exits with 0 when the port is open; the liveness probe uses it.

Group Endpoints
Health and version GET /api/v2/health/, GET /api/v2/version/ (returns the image's IMAGE_TAG, or N/A when unset; v1 equivalents exist)
Files GET/HEAD/PUT/DELETE /api/v2/files/{path} (+ v1), OPTIONS /api/v2/files/{path} (v2)
Directories POST /api/v2/directories/{path} (+ v1; the v1 route runs the same handler)
Tokens (v2 only) POST /api/v2/token/, POST /api/v2/token/decode/, DELETE /api/v2/token/, GET/DELETE /api/v2/token/history/, POST /api/v2/token/history/backup/, POST /api/v2/token/history/restore/
Symlinks (v2 only) PUT/GET/DELETE /api/v2/symlinks/{path}
Signed/session URLs (v2 only) POST /api/v2/signed-urls/, .../verify/, POST /api/v2/session-urls/, .../verify/
Public (only when SITE_TYPE=public) GET/HEAD/OPTIONS /api/v2/public/files/{path}, exposed as .../s/api/v2/pub/...

The service registers no authorization of its own: the proxy's admin_auth module guards every /s/api/ route except the longer-prefix matches of api.conf — the session-files location, which runs session_files_auth (staff may use its read methods only), and the public files location, which carries no access module by design. The public routes exist only on public Sites. The service does start only with MASTER_TOKEN and SITE_PATH set; the token is the signing key for Branca tokens and signed URLs.

The v1 file routes are reduced legacy surfaces: v1 GET ignores limit, continuation_token, and archive (no pagination, no archives); v1 PUT accepts only multipart/form-data uploads and processes only the first file part; v1 finalize skips the pre-rename removal and directory sync that v2 performs. New clients must use v2.

CORS

The OPTIONS preflight (files and public files routes) answers 204 and, when an Origin is present, echoes it with Access-Control-Allow-Credentials: true, Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers, the requested method and headers (or the documented defaults), and Access-Control-Max-Age: 86400 (api/file_utils.go). Plain GET downloads, listings, and archives carry no CORS headers. HEAD on a file and every PUT response add the upload progress surface: Accept-Ranges, the echoed origin, and Access-Control-Expose-Headers for Upload-Offset, Content-Range, Range, and the digest headers — what the browser uploader reads.

Tests

Go tests run with go test ./... (make test). Beyond the units behind the requirements below, note the honest gaps: the health handler, the OPTIONS preflight handler, v1 listings, the symlink GET success path, the backup/restore and history handlers, and the shutdown drain order have no direct tests.

Behaviour requirements

ID Requirement Implemented in Verified by
API-01 A full upload becomes visible only after an atomic rename following an fsync of the temporary file. api/files.go api/v2_put_file_test.go, api/v1_put_file_test.go (visibility and temporary-file cleanup; the fsync-then-rename order itself is not asserted)
API-02 A resumable chunk whose start does not equal the current size is rejected with 409 and the authoritative Upload-Offset. api/files.go, api/file_utils.go api/v2_put_file_test.go
API-03 Upload buffering is bounded per upload and globally; a stalled disk causes 503 with Retry-After, never silent data loss. api/upload_buffer.go api/upload_buffer_test.go, api/v2_put_file_test.go (queue saturation)
API-04 File responses always carry a weak ETag; Repr-Digest appears only on explicit Want-Repr-Digest, never for zero-byte files or archives. api/checksum_utils.go api/checksum_utils_test.go, api/file_metadata_handlers_test.go, api/file_metadata_router_test.go
API-05 Continuation tokens are short-lived (3 min) server state; an unknown token returns an empty page, not an error. api/pagination_utils.go, cache_utils.go api/cache_utils_test.go, api/cache_collision_test.go (key-space separation; the unknown-token path is not asserted)
API-06 Writes that exceed the site quota fail with 507 before or during the write. api/file_utils.go (HasEnoughDiskSpace), files.go api/quota_test.go (space check; the upload pre-write gate answers 507 and leaves no file)
API-07 The public router is registered only when SITE_TYPE=public. api/main.go api/public_files_test.go (handler filtering; the registration gate is not asserted)
API-08 Bulk deletes above 500 entries or 100 MB return 202 and finish asynchronously. api/files.go, common.go api/files_test.go
API-09 Requested paths are jailed to the content root: clean + filepath.IsLocal at parse (400 on escape) and an EvalSymlinks check for existing paths, after up to two URL decodes. A missing path, a scan entry, a pattern-delete parent, and the parent of a symlink endpoint path must also resolve inside the root through existing and dangling symlinks. File operations on content go through an os.Root on the content root, so a symlink that changes after the checks cannot reach a file outside the root; absolute symlinks are not followed. api/models.go, api/file_utils.go, api/files.go, api/symlink.go, api/site_root.go api/file_utils_test.go, api/files_test.go (double-encoded traversal rejected; double-encoded space accepted), api/symlink_jail_test.go (symlinked parent, escaping parent, missing and dangling paths, scans, pattern delete), api/site_root_test.go (parent swapped to an outside link for each operation, absolute link, downloads)
API-10 Upload dispatch is header-driven: multipart by Content-Type prefix first, then a valid Content-Range, then a raw body (a malformed range is raw, not rejected). api/files.go, api/file_utils.go (parseContentRange) api/v2_put_file_test.go (raw and resumable paths; the malformed-header downgrade is not asserted)
API-11 On public Sites, the public router hides .htpasswd files, every tree under a directory containing one, and prunes both from listings and from cached continuation pages, without prefix over-match. The check also covers the resolved physical path, so a symlink whose target is protected is hidden under any name. api/public_files.go, api/file_utils.go (baseOrAncestorProtected, resolvedHtpasswdProtected, symlinkResolvesToProtected) api/public_files_test.go (seven filtering cases)
API-12 Signed-URL verify answers 204 or 403; session-URL verify answers the payload or 403 for expiry, a bad token, or a path outside the scope. api/sign_urls.go, api/session_urls.go, api/models.go (IsPathWithinScope) api/sign_urls_test.go, api/session_urls_test.go
API-13 Token creation requires duration and a valid identifier; revocation consults a startup-seeded Bloom filter before the database, and revoke-by-hash answers 404 for an unknown hash. api/token.go, api/models.go, api/token_revocation_bloom.go api/token_test.go, api/token_utils_test.go (Bloom init/fast path, hash-only revoke)
API-14 When the token-history database is unavailable, token decode fails closed: a loaded Bloom filter that does not contain the token hash still accepts the token without a database read, but a Bloom hit or a missing filter needs the database and the decode answers 500. The history, backup, and revoke endpoints answer 503. The API retries the database in the background every 30 s (StartTokenDatabaseRecovery) and rebuilds the revocation filter and the endpoints when it opens, so a transient start-up failure recovers on its own; a snapshot is written at startup and once per day (StartTokenHistoryBackup) as the source for POST /token/history/restore/. api/token_storage_sqlite.go, api/token.go, api/main.go api/token_utils_test.go (TestIsTokenRevokedFailsClosedWhenDatabaseUnavailable, TestIsTokenRevokedFailsClosedForABloomHitWhenDatabaseUnavailable, TestIsTokenRevokedAllowsABloomMissWithoutTheDatabase), api/token_recovery_test.go (the 503 responses of the history, backup, and revoke endpoints are not asserted)
API-15 Graceful shutdown drains HTTP traffic (15 s), then background deletes, then stops cleanup, the token-database recovery loop (StopTokenDatabaseRecovery), and the history-backup loop (StopTokenHistoryBackup), and drains history writers before closing the database. api/main.go — (only the server timeouts are tested, main_test.go)
API-16 Recursive scans (listings, archives, pattern deletes) do not enter symlinked directories, so a symlink loop cannot make a scan run without a limit; links are still listed as entries, and a symlink in the requested base path is followed. api/file_utils.go (filePathGlob) api/symlink_loop_test.go (two self-loop links, directory link listed but not entered, archive entries, linked base path)
API-17 The API binds only the loopback address, so only containers in the site Pod can connect. The exec liveness probe /api healthcheck checks the loopback port. api/main.go (listenAddress, healthcheck), site/.../api/app_config.py api/main_test.go (TestListenAddressIsLoopbackOnly, TestHealthcheck), site/tests/test_api_app_config.py
API-18 A resumable chunk whose range length is larger than the maximum int64 value is rejected with 416 and Upload-Offset before the target file is opened, so the length arithmetic cannot overflow. api/models.go (ContentRange.length), api/files.go api/v2_put_file_test.go (TestContentRangeLength, TestV2PutFile_RangeLengthOverflowReturns416)