Local testing¶
How to exercise the automation on a workstation: Make targets, helper tests, dependency audits, and workflow runners. The audit helper internals also apply to CI; the workflow wiring is in Continuous integration.
Local testing¶
Most workflow behavior can be tested locally through the same Make targets that GitHub Actions calls:
make init-workspace
make hub-assets
make test
make fuzz-go FUZZTIME=10s
make dependency-audit
make dependency-update-check
make dependency-update
make check-lockstep-version
make --silent print-version
make publish-images IMAGE_TAG=develop
make publish-develop
make release
make release-notes
make build-docs
utilities/tests/changelog-notes.test.sh
bash utilities/tests/build-publish-docs.test.sh
bash utilities/tests/dependency-audit.test.sh
bash utilities/tests/update-proxy-checksums.test.sh
bash utilities/tests/dockerfile-policy.test.sh
python3 utilities/tests/test_dependency_audit_fix.py
python3 utilities/tests/test_dependency_updates.py
python3 utilities/tests/test_security_scan.py
python3 utilities/tests/test_security_workflow.py
python3 utilities/tests/test_workflow_summaries.py
python3 utilities/tests/test_committed_tokens.py
node --test hub/frontend/tests/copy-vendor.test.mjs
actionlint .github/workflows/*.yml
To run the secret-scan job locally, install gitleaks and run:
Set GITLEAKS to the path of the gitleaks binary if it is not on PATH.
The job also checks the root commits of all refs. To run this check locally, use:
The check needs the full history (git fetch --unshallow in a shallow clone). It reads the refs that your clone has. To include the pull request refs, run
git fetch origin '+refs/pull/*/head:refs/remotes/pull/*' first.
The .github/actionlint.yaml file lists the private self-hosted runner label
so that it is checked as a valid runner label.
make fuzz-go runs the Go fuzz targets of the Site API for FUZZTIME
each (default 30s, the same as the go-fuzz job). When a target finds a
failing input, Go writes it to
site/src/ecmwf/sites/website/applications/api/api/testdata/fuzz/<target>/.
After that, go test ./... runs the input again as a seed. Fix the bug, and
keep the input as a regression test.
The dependency audit shell test uses stubs and does not need network access.
It checks temporary paths with spaces, command order, and error handling.
The export stub creates the requirements file in an existing temporary
directory, as uv export --output-file does. The Python scan tests check
file cleanup after success, export failure, and audit failure.
Run the shell test with Bash 5, as used in CI, as well as the local default
Bash. Ensure that PATH selects the same Bash version for the test stubs.
macOS Bash 3.2 can ignore failed checks inside a case branch.
The dependency audit requires uv (including uvx), Python 3.13, Go with the
toolchain specified in site/src/ecmwf/sites/website/applications/api/go.mod,
and network access to download audit tools and query vulnerability data.
It does not require make init-workspace. The Python target uses
uv export --frozen --all-packages --no-hashes --no-emit-workspace and removes
its temporary requirements file on success or failure. Both targets call
utilities/release/security_scan.py. The Go target installs the pinned
govulncheck into a temporary directory and runs it from the API module
directory.
Vulnerability findings are advisory. The targets print a warning and return exit code 0. In GitHub Actions, the warning is an annotation and the result is added to the job summary. The targets return a nonzero exit code only when a scan cannot complete: for example, when an export or tool install fails, a tool returns an unexpected exit code, or a report is invalid. The Python export writes its requirements only to the temporary file, not to the terminal. Export errors and full scanner reports remain visible.
The Go helper runs the govulncheck binary directly. go run changes the
scanner exit code 3 (findings) to 1, so it cannot tell findings from errors.
The Go audit checks reachable vulnerabilities. Its output can also list
module or package advisories that the code does not call. The workflow scan
treats zizmor exit codes 11 to 14 as findings and other nonzero exit
codes as errors.
The Security workflow calls make dependency-audit-python,
make dependency-audit-go, and make dependency-audit-npm after their tool
setup steps. Use those targets to run one check locally, or
make dependency-audit to run all three in order,
including when Make uses parallel jobs. Tool versions and audit options
are shared with CI, but results can differ with the platform, toolchain,
or vulnerability data. These targets do not run dependency review, CodeQL,
or workflow lint.
The combined target stops if a scan cannot complete. If the Python check
fails to run, run make dependency-audit-go and make dependency-audit-npm
separately to get the other reports.
The npm helper runs npm audit --package-lock-only --json in hub/frontend.
It reads only package-lock.json and installs nothing. A report with
vulnerabilities is a finding. A report with an error object, a report that
is not JSON, or a nonzero exit code without vulnerabilities is an error.
make dependency-audit-fix does not change the npm lockfile; update it with
npm install --ignore-scripts <package>@<version> in hub/frontend.
make dependency-audit-fix can change uv.lock and the API's go.mod and
go.sum. For Python, it exports the frozen lockfile,
runs pip-audit with JSON output, and calls uv lock once with
--upgrade-package for each package that has a published fix. uv chooses the
newest version that the existing constraints allow, so related packages can
also change and a major version can change. The command does not edit
pyproject.toml files, use overrides, or run a general upgrade.
For Go, the helper reads go.mod and runs the pinned govulncheck with
JSON output and GOTOOLCHAIN set to the declared toolchain. Without a
toolchain directive, it uses the go directive. A newer local installation
must not hide findings in that declared version. The report is a stream
of JSON objects. The helper validates it and selects symbol-level findings
that fail the normal audit; module-only findings do not trigger updates.
The helper uses targeted go get module@version commands for the reported
fixed versions. It uses go mod edit to update the toolchain for a
standard-library fix, within the current Go release line only. Go maintains
the module files and can update related requirements or raise the minimum
go version. Commands use an explicit toolchain rather than silently
switching release lines. Module and toolchain downloads require network
access. The helper verifies Go updates with another structured scan.
Findings without a published fix, replaced modules, unsupported version
formats, module major-version changes, and Go release-line changes need
manual action. Automatic module updates support stable vMAJOR.MINOR.PATCH
versions, including the +incompatible suffix; prerelease and pseudo-version
findings need manual review. These cases and malformed reports, scanner failures, or
update failures produce errors. Docker image pins are not changed.
Both language fixes are attempted, and both final audits run even if a fix
step fails. The target returns nonzero if any fix or audit fails. A pinned
Python dependency that blocks a fix needs a manual change. The command
does not suppress findings or undo partial updates on failure. Review the
uv.lock, go.mod, and go.sum diffs, check that the API Docker build
image meets any new Go requirement, and run the affected tests before you
commit the result. The repository-tests job tests the helper
(utilities/release/dependency_audit_fix.py) without network access, and does not run
the command.
Run the Go fix integration tests with Go and network access:
These tests use temporary modules. They exercise a module fix and a Go toolchain patch fix, then run the audits and compile checks again. They do not update the repository's dependency files.
The dependency update targets (make dependency-update-check and
make dependency-update) are separate from the audits. They need uv, Go with
the declared toolchain, and npm, and network access to the package registries.
The check runs uv pip compile, uv lock --upgrade --dry-run,
go list -m -u -json all, and npm outdated --json, each bounded by
DEPENDENCY_COOLDOWN_DAYS (default 7), and writes nothing. Available updates
are not a failure; only a tool that cannot run is. The update applies what the
same bounded report lists: the exact == pins of the member pyproject.toml
files, uv.lock, the API go.mod and go.sum, and
hub/frontend/package.json with its lockfile. The Python part rewrites a pin
in place with uv add --frozen, so the diff of a pyproject.toml file changes
version digits only, and re-locks uv.lock so the pins and the lockfile agree.
The Go part covers the modules that the API go.mod requires, never changes the
go or toolchain directives, and lists a replaced module, a major-version
change, or a module without a usable release date for a manual change. The npm
part rewrites exact pins, so review hub/frontend/vendor-files.json when a
copied file name changes; the target runs make hub-assets last. The helper
(utilities/release/dependency_updates.py) needs no lockfile or module to
change, so the tests mock the tools and run without network access
(utilities/tests/test_dependency_updates.py). The tests also check the Makefile
targets and that no workflow calls them. Run make test and
make dependency-audit after an update, before you commit it.
If you want to exercise the GitHub workflow wiring itself, you can also use act. For example:
Publishing workflows require registry credentials and GitHub event payloads.
Use only a test environment and a payload that matches the workflow trigger.
The check-gate job in develop.yml reads workflow runs and the develop
branch from the GitHub API, so it needs GITHUB_TOKEN. To test only the gate
logic, run:
To test fork-PR gating behavior, pass an event payload that includes:
pull_request.head.repo.fork- the PR
labels