Files, uploads, and integrity¶

Path jailing, listings and downloads, the upload protocols, integrity headers, and directory and symlink endpoints. The endpoint groups are on the Site API overview; the authentication that fronts them is in 10-security.

Path handling and jailing¶

The API serves {path} values jailed to the Site content root (/www/html, fixed):

• Gin URL-decodes the path once; if the result still contains only well-formed %XX escapes, the API decodes a second time (so a double encoded %252E%252E reaches the checks as ..; a bare % such as 100% done stays literal).
• Every parsed path is cleaned and checked with filepath.IsLocal; an escape answers 400 (ParseMiddleware aborts).
• For paths that already exist, a second check resolves symlinks with filepath.EvalSymlinks and rejects any resolution outside the root.
• For paths that do not exist yet (uploads, new directories, renames), the check resolves every symlink on the path that exists, also a dangling symlink at the path itself, because a write follows it. A path whose location on disk is outside the root, or that has more than 40 levels of links, answers 400.
• The router collapses duplicate slashes and fixes trailing slashes (RemoveExtraSlash, RedirectFixedPath), so encoded separators cannot smuggle a second root.
• The API does each file operation on the content through one os.Root that it opens on the content root (api/site_root.go). This applies to open, create, stat, remove, rename, directory creation, symlink operations, temporary upload files, downloads, archives, and checksums. The kernel resolves each path component below the root. Thus, a tenant that changes a directory to a symlink after the checks above cannot make the operation read, write, or delete a file outside the root. The operation fails.
• os.Root does not follow an absolute symlink, also when the absolute target is inside the root. A download, an archive, or a write through such a link fails. Use a relative link. The symlink endpoints always store relative links.
• Recursive scans and the size estimate of a recursive delete use paths and do not follow symlinked directories. The public .htpasswd protection check also resolves the physical path, so a symlink whose target sits under a protected tree is hidden too (baseOrAncestorProtected, resolvedHtpasswdProtected, symlinkResolvesToProtected).

(TestRootRelPath, TestResolveRootPath, TestResolveRootPathRejectsMissingPathsThatEscape, and TestDeleteFile_DoubleEncodedTraversal_Rejected cover the rules; TestDeleteFile_DoubleEncodedSpaces_PathFromClient_Success shows that an encoded space that is not a traversal still works.)

Listing and download¶

GET /files/{path} returns a directory listing (JSON) for a directory, or a file body for a file. File downloads set a Content-Disposition attachment header; a requested path that ends with index.html is served inline instead. ?list=1 on a file returns a single metadata entry instead of the body. HEAD on a directory answers 409 with X-Error: path is a directory; a zero-byte file answers Content-Range: bytes */0.

v2 listings return {files: [{type, path, stats: {size, mode, mtime}}], continuation_token}; v1 returns a flat path array. Paths are relative to the requested directory. Listing entries follow symlinks for the type and stat data (a link to a directory lists as d); a dangling link is skipped with a warning, and recursive scans do not descend symlinked directories. The scan passes doublestar.WithNoFollow() to the glob, so ** lists a link to a directory but does not enter it. A link loop such as a -> . therefore cannot make a recursive listing or archive run without a limit (api/symlink_loop_test.go). A symlink in the requested base path is still followed, so a client can list a linked directory directly. An archive contains only regular files, so a link to a directory is not in it. A scan skips every entry that resolves outside the root through a symlink. Listings, archives, and pattern deletes use the same scan, so they cannot read or delete files outside the root. A pattern delete also requires that the parent of each entry resolves inside the root; for a symlink entry, it deletes only the link.

Pagination caches the scan remainder in process memory for 3 minutes, keyed by the continuation token itself: the next token is the SHA-256 of the previous one (api/cache_utils.go, a random UUID on the first call), and the used token is dropped. The limit is re-applied on every page, so changing it mid-pagination keeps the token valid and only changes the page sizes. An expired or unknown token yields an empty listing with an empty token, not an error. The last page returns an empty token. The first call therefore pays for the scan.

GET /files/{dir}?archive=zip streams a fresh archive; combine with recursive for whole trees. Archives also work on a single file, contain files only (directory entries are excluded), and answer 404 with a hint about recursive when the scan is empty. Responses carry the archive MIME type, a Content-Disposition: attachment; filename=archive.<ext>, and X-Accel-Buffering: no, flushed in 256 KiB blocks. Streaming means archives carry no ETag and no Repr-Digest. A proxy shortcut exists for the listing UI: .../download-folder rewrites to the files endpoint with archive=zip and recursive=false; it rejects other methods with 405, applies the site type's access module first, and on public sites it uses the public router, so protected trees answer 404 like other public reads.

Uploads¶

PUT /files/{path} dispatches on headers, in this order:

• Content-Type: multipart/form-data... (header prefix, checked first) streams the parts named file as atomic uploads; every file part is stored (last write wins on repeated names); a malformed body or a missing file part answers 400.
• Otherwise, a valid Content-Range: bytes start-end/total header selects the resumable chunk path. In a valid header, start is 0 or more, end is not less than start, and total is * (unknown) or a positive integer that is larger than end.
• Anything else, including a malformed Content-Range, is a raw full upload body.

Full uploads write a *.upload.tmp file, fsync it, remove any target, rename it atomically, and sync the directory. Parent directories are created automatically. Writers racing on the same target path are serialized by a striped in-process lock. A PUT onto an existing directory answers 400 (the directory is not replaced). Status is 201 for a new file and 200 for a replacement. backup_if_present (optionally with rename_to, which names the backup target; default suffix YYYYMMDDHHMMSS) moves the old file aside first and returns 201; rename_to alone has no effect.

The resumable protocol is offset-strict:

sequenceDiagram
    participant C as Client
    participant A as Site API
    C->>A: HEAD /files/{path}
    A-->>C: Upload-Offset: 1024 (current size)
    C->>A: PUT with Content-Range: bytes 1024-2047/2048
    alt offset matches
        A-->>C: 200/201, Upload-Offset: 2048
    else offset mismatch
        A-->>C: 409 Conflict, Upload-Offset: actual size
        Note over C: resume from the returned offset
    end

A Content-Length that does not match the range length is a 400 (as is a short write). A range whose length is larger than the maximum int64 value (bytes 0-9223372036854775807/*) is a 416 with Upload-Offset; the API does not open the target file. The first chunk truncates the target, and every accepted chunk is fsynced for NFS durability. When total is known, the free-space pre-check sizes the remaining bytes, not the chunk.

Write buffering and backpressure¶

flowchart LR
    N["Network read"] --> Q["Per-upload chunk queue (UPLOAD_BUFFER_PER_UPLOAD_MB, default 8 MiB)"]
    Q --> G["Global token bucket (UPLOAD_BUFFER_GLOBAL_MB, default 64 MiB)"]
    G --> F["Background flusher: write + fsync"]
    F --> D[("Storage volume")]
    F -- "no flush for UPLOAD_STALL_TIMEOUT_SEC (30 s)" --> X["503 + Retry-After: UPLOAD_RETRY_AFTER_SEC (60)"]

Out-of-range values are ignored silently and the default applies (api/upload_buffer.go). When storage is slow, the queue and the global budget fill up, and Write blocks — backpressure slows the client instead of failing. When no chunk has reached disk for the idle timeout, the API returns 503 Storage system not responding, please retry. with Retry-After. Treat that response as retryable.

File integrity¶

• Every file GET/HEAD response carries a weak ETag of W/"<size hex>-<mtime nanos hex>". It costs no I/O.
• GET/HEAD with Want-Repr-Digest: sha-256 adds Repr-Digest: sha-256=:<base64>: per RFC 9530. Only sha-256 is supported, and the negotiation token must be the bare value (parameters are rejected; other algorithms in the list are skipped, not rejected). Zero-byte files omit the header. Digests are cached for 3 minutes, keyed by path + size + mtime, and warmed on upload. A single file delete invalidates its cache entry; bulk deletes do not (entries expire). On the v1 upload path the cache warm-up honours ENABLE_CHECKSUM_CACHE; the v2 path always warms.
• Archive and listing responses never carry these headers. Listings include no per-file digests.

Directories and symlinks¶

POST /directories/{path} creates the directory tree under the root (an existing directory answers with success; a file collision answers through the error map; the volume root itself is rejected).

PUT /api/v2/symlinks/{path} creates a symbolic link at {path} pointing to a target inside the volume. The target must exist. An existing link is replaced only with force (a query parameter; otherwise 409). The stored link is relative to its directory. GET returns the current target; DELETE removes the link and answers 400 when the path exists but is not a symlink. Targets and link paths outside the volume are rejected (api/symlink.go).

The parent directory of the link is resolved through symlinks before the operation, and it must be inside the root; otherwise all three endpoints answer 400. The link is created in that resolved directory, and the relative target is computed from it, not from the request path. Thus a link below a symlinked directory (for example a/l when a points to the root) points to the requested target. After creation, the API resolves the new link; if it resolves outside the root, the API removes it and answers 400 (api/symlink_jail_test.go). The endpoints create, read, and remove the link through the content root os.Root, so a parent that changes to a symlink after the checks causes an error and not a link outside the root (api/site_root_test.go).