Testing¶

This page inventories the implemented tests and maps them to the design requirements. The CI wiring around the test entrypoints is in 15-ci-cd.

Entry points¶

make test            # full Python + Go suite (needs lua on PATH)
make init-workspace  # uv venv + uv sync --all-packages --all-groups

make test runs, in order:

uv run --package sites-core     python -m unittest discover core/tests/
uv run --package sites-hub      python -m unittest discover hub/tests/
uv run --package sites-operator python -m unittest discover operator/tests/
uv run --package sites-site     python -m unittest discover site/tests/
( cd site/src/ecmwf/sites/website/applications/api && go test ./... )

Test layers¶

Layer	Location	How it runs	Covers
Python core	`core/tests/` (≈55 test modules)	`unittest discover`	Config schema/consistency, model + exporter, routing backends, integrations, jobs, forms and user lists, helm config rendering (`test_helm_configuration.py`)
Python hub	`hub/tests/` (24 modules)	`unittest discover`	Hub services and API (request validation, cross-site request forgery (CSRF), spec coverage against the OpenAPI files), routing rules (gateway middleware and the Hub Content Security Policy (CSP)), branding/status/template rendering, vendor assets (no CDN references, npm lockfile and allow-list, licences), notifications 503 gate, upgrade flows
Python operator	`operator/tests/`	`unittest discover`	kopf diff gating, create/update handlers, forced-upgrade annotation marker (`test_forced_upgrade.py`), logging policy (`test_logging_policy.py`), secret logging, event metadata
Python site	`site/tests/`	`unittest discover`	Per-application config, Dockerfile user/licence/checksum policy, proxy HTML utilities URLs, mirror script, Lua behaviour
Lua	`site/tests/test_proxy_lua_*.py`	Python tests that spawn `lua` 5.4 (`site/tests/lua_test_utils.py`), skip when `lua` is absent	`auth_chain`, `auth_utils`, `basic`, `branca`, `hex64`, `jwt`, `session_url`, `signed_url`, `retention_date`
Go	`site/src/.../applications/api/` (`main_test.go` + 21 modules in `api/`)	`go test ./...`	Uploads (v1/v2, resumable, buffer), checksums/ETag, pagination, public listing, signed/session URLs, tokens, symlinks, directories, quotas, cache keys
Node (workflow helpers and Hub frontend)	`.github/actions/check-develop-gate/gate.test.cjs`, `.github/actions/prepare-credentials/credentials.test.cjs`, `hub/frontend/tests/copy-vendor.test.mjs`	`node --test`	Publishing gate, credential resolution/masking, Hub vendor file copy (lockfile and allow-list checks, safe paths, licence files)
Shell (repository tooling)	`utilities/tests/*.test.sh`	run directly (`bash <file>`)	Docs build/publish, changelog notes, dependency audits, deploy-script branding, Dockerfile policy, Helm routing/secrets, local-dev post-renderer, proxy checksum updates
Python (workflow helpers)	`utilities/tests/test_*.py`	`python <file>`	Security scan wrapper, Security workflow structure, job summaries, dependency-audit fixer, dependency-updates helper

Two shell tests are manual only (no workflow references them): deploy-sites-branding.test.sh and local-dev-post-renderer.test.sh. Run them after touching the local deployment path (see 18-local-development).

The Security workflow runs the Node, shell, and helper-Python tests in its repository-tests job (see 15-ci-cd); CI runs make test.

Helper tests check script outputs, failure handling, credential handling, and workflow safety rules. The tests do not check prose, page counts, or runner versions — except the ubuntu-24.04 pin of the gate job, which gate.test.cjs asserts. Review documentation content for accuracy; use the strict documentation build to check rendering (15 — Documentation site).

Chart and configuration coverage¶

There is no helm/tests/ directory. Chart behaviour is checked from two places:

core/tests/test_helm_configuration.py renders chart values through the configuration schema.
utilities/tests/helm-routing-and-secrets.test.sh runs helm template and asserts routing resources and configurable Secret references.

Requirements coverage map¶

Prefixes are defined in each design document. This map shows where each family gets its automated evidence.

Prefix	Primary tests
OVR	`utilities/tests/changelog-notes.test.sh`, `make check-lockstep-version`, `test_security_workflow.py`
ARC	`operator/tests/`, `utilities/tests/helm-routing-and-secrets.test.sh`
DM	`core/tests/` (model, exporter), `site/tests/` (app configs), `hub/tests/test_sites_service.py`
APP	`site/tests/test_*app_config.py`, `site/tests/test_mirror_script.py`
BIA	`site/tests/test_*_app_config.py`, `site/tests/test_package_resources.py`
PRX	`site/tests/test_proxy_app_config.py`, `site/tests/test_proxy_lua_*.py`
CUS	`site/tests/test_custom_app_config.py`, `core/tests/`, `hub/tests/test_api_request_validation.py`, `hub/tests/test_sites_service.py`
MIR	`site/tests/test_mirror_app_config.py`, `site/tests/test_mirror_script.py`
HUB	`hub/tests/`, `core/tests/test_backups.py`
HUI	`hub/tests/` (request validation, CSRF, branding/status/template rendering, notifications 503, upgrade flows), `core/tests/test_add_edit_site_form.py`, `core/tests/test_form_fixes.py`
DAT	`core/tests/`, `site/src/.../api/` Go tests, `operator/tests/`
LIFE	`operator/tests/`
SEC	`site/tests/test_proxy_lua_*.py`, `api/public_files_test.go`, `core/tests/test_oidc_discovery.py`, `hub/tests/test_routing_rules.py`
API	`site/src/.../api/` Go tests, `api/upload_buffer_test.go`
RTG	`core/tests/test_routing.py`, `operator/tests/test_site_update.py`, `helm-routing-and-secrets.test.sh`
QBR	`core/tests/test_backups.py`, `test_retention_crawler_broadcast_channels.py`, `test_config_consistency.py`
CFG	`core/tests/` (config, branding), `hub/tests/test_branding_rendering.py`, `deploy-sites-branding.test.sh`
INT	`core/tests/` (`test_integration_status.py`, `test_notification_dispatcher.py`, `test_notification_crawler_.py`, `test_none_directory.py`, `test_ldap_.py`, `test_analytics.py`, `test_config_consistency.py`), `hub/tests/test_notifications_required.py`, `site/tests/test_proxy_app_config.py`
DEV	`deploy-sites-branding.test.sh`, `local-dev-post-renderer.test.sh`
OBS	`operator/tests/test_secret_logging.py`, `core/tests/test_storage_utils.py` (Nagios health strings), `site/tests/test_proxy_app_config.py`, `core/tests/test_helm_configuration.py`; the HTTP status-code contracts of the health endpoints have no automated test
CI	`test_security_workflow.py`, `test_security_scan.py`, `test_workflow_summaries.py`, `test_committed_tokens.py`, `gate.test.cjs`, `credentials.test.cjs`
TST	`CI` workflow (`make test` gate), `gate.test.cjs`, `test_security_scan.py`

Requirements marked — in a design document have no automated test today; they are the natural backlog for new tests.

Behaviour requirements¶

ID	Requirement	Implemented in	Verified by
TST-01	`make test` runs the four Python suites and the Go suite in one command, and fails if any test fails.	`Makefile`	CI workflow
TST-02	Lua logic is tested with the standalone `lua` 5.4 interpreter installed by the shared test action (the proxy image itself runs Lua inside OpenResty); missing `lua` skips those tests instead of silently passing them in the developer run.	`site/tests/lua_test_utils.py`	—
TST-03	CI publishes only after `make test` passed for the exact commit.	`.github/workflows/ci.yml`, `develop.yml`, `release.yml` gates	`gate.test.cjs`, `test_security_workflow.py`
TST-04	Advisory scans report findings without failing the build, while a scan that cannot run fails the job.	`utilities/release/security_scan.py`	`test_security_scan.py`