pymetkit.paramdb._db#

ParamDB: ECMWF parameter metadata lookup (shortname <-> paramid).

Python interface for manipulation of parameters and their metadata. uses metkit expansion engine to resolve shortname collisions when mars context is provided. Libmetkit is required.

Attributes#

__version__

_requests

_platformdirs

_HAVE_EXPAND

Exceptions#

AmbiguousParamError

Raised when a shortname (given the supplied context) maps to >1 paramid.

Classes#

ParamIDCandidate

One possible paramid for a shortname, plus the context that selects it.

ParamDB

Parameter database providing metadata lookup for ECMWF parameters.

Module Contents#

__version__#
_requests = None#
_platformdirs = None#
_HAVE_EXPAND = True#
class ParamIDCandidate#

One possible paramid for a shortname, plus the context that selects it.

param_id#

The candidate numeric parameter ID.

table#

GRIB parameter table the id encodes to (e.g. 128, 228).

origin#

WMO originating centre ids associated with this candidate — the full list (e.g. [0, 34, 98]), since several centres may share the id.

access#

Access categories (e.g. ["dissemination"]) — the full list.

mars_request_context#

The minimal set of MARS key/value pairs that, when passed as context= to ParamDB.shortname_to_param_id(), selects this candidate (e.g. {"class": "ai"}). Special values:

  • {} — this candidate is the default: an empty context={} resolves to it via the C++ expand layer.

  • None — no MARS context can select this candidate; use the hard filters instead (see hard_filter_selector).

hard_filter_selector#

The minimal table/origin/access kwargs that, when passed to ParamDB.shortname_to_param_id(), are proven to select exactly this candidate among all parameters sharing the short name. Special value:

  • None — no combination of the available hard filters uniquely identifies this candidate (e.g. two ids share the same table, origin and access). In that case there is no hard-filter selector to advertise, and the collision cannot be resolved by hard filters alone.

param_id: int#
table: int#
origin: list[int]#
access: list[str]#
mars_request_context: dict | None = None#
hard_filter_selector: dict | None = None#
exception AmbiguousParamError(shortname: str, candidates: list[ParamIDCandidate])#

Bases: KeyError

Raised when a shortname (given the supplied context) maps to >1 paramid.

shortname#

The shortname that could not be uniquely resolved.

candidates#

Every matching candidate, each carrying the context needed to select it, sorted by (table, origin, access, mars_request_context). The caller inspects these and re-calls with a narrowing context=.

Initialize self.  See help(type(self)) for accurate signature.
shortname#
candidates#
class ParamDB(mode: str = 'offline', cache_ttl: timedelta | None = None, cache_path: Path | str | None = None, yaml_path: Path | str | None = None)#

Parameter database providing metadata lookup for ECMWF parameters.

Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file).

When using mode="online" a local JSON cache is maintained so that repeated instantiations within the TTL window do not make a new HTTP request. The cache is stored under the OS user-cache directory (e.g. ~/.cache/pymetkit/ on Linux, ~/Library/Caches/pymetkit/ on macOS) using the fixed filename defined by _CACHE_FILENAME.

Shortname collision resolution#

Some short names (e.g. t, tp, u) are reused across different GRIB parameter tables and originating centres. Collisions are not silently guessed by shortname_to_param_id():

  • shortname_to_param_id() raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. The error’s .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id); for tp this is 228. Note this is not the canonical paramID as without mars context this cannot be determined.

  • shortname_to_longname() behaves the same way: it raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id).

To resolve a collision explicitly, narrow the lookup with a MARS context= (resolved via the C++ expand engine) or the table=/origin=/access= hard filters, both accepted by shortname_to_param_id() and shortname_to_longname().

Initialise the parameter database.

The underlying data is loaded lazily — no file I/O or network request is made until the first lookup method is called. This makes instantiation cheap and safe to do at import time or inside hot paths.

param mode:

Either "online" (fetch from the ECMWF API) or "offline" (load from a YAML file).

type mode:

str

param cache_ttl:

How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when mode="online". Pass timedelta(0) to disable caching entirely (always fetch).

type cache_ttl:

datetime.timedelta, optional

param cache_path:

Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires platformdirs). Only relevant when mode="online".

type cache_path:

Path or str, optional

param yaml_path:

Path to a custom YAML file to load instead of the bundled parameter_metadata.yaml. The file must be a YAML list where each entry contains at minimum an id (integer), a short name (shortname), and a long name (longname). Only the canonical key spellings are accepted. Only valid with mode="offline"; raises ValueError if combined with mode="online".

type yaml_path:

Path or str, optional

_API_URL = 'https://codes.ecmwf.int/parameter-database/api/v1/param/'#
_CACHE_FILENAME = 'paramdb_online_cache.json'#
_DEFAULT_CACHE_TTL#
_REQUEST_TIMEOUT = 30#
_context_notice_emitted = False#
_mode = 'offline'#
_effective_cache_ttl#
_cache_path = None#
_yaml_path = None#
_by_id: dict[int, dict]#
_by_shortname: dict[str, dict]#
_by_shortname_all: dict[str, list[dict]]#
_by_longname: dict[str, dict]#
_loaded: bool = False#
_ctx_cache: dict#
static _table_from_id(param_id: int) → int#

Decode the GRIB parameter table number from an encoded param ID.

The encoding scheme mirrors the C++ Param::paramId() logic:

  • IDs 1–999 → table 128 (classic ECMWF, table prefix suppressed)

  • IDs 1 000–999 999 → table * 1000 + param (e.g. 228228 → table 228)

  • IDs ≥ 1 000 000 → center * 1_000_000 + table * 1000 + param (e.g. 7001292 → center 7, table 1)

static _center_from_id(param_id: int) → int | None#

Decode the originating WMO center from an encoded param ID.

Returns None for IDs below 1 000 000 (ECMWF-local parameters). For IDs ≥ 1 000 000 the center is param_id // 1_000_000.

_context_resolved_ids(shortname: str, context: dict) → set[int]#

Resolve shortname + context to paramid(s) via the C++ engine.

Builds MarsRequest(param=shortname, **context), expands it (which runs TypeParam::pass2 and fills MARS defaults for unspecified keys), and reads back the resolved param value(s). Results are memoised on self._ctx_cache since each expand is a comparatively expensive (~ms) C++ round-trip and the minimal-context search repeats queries.

The C++ library reads its language files (params.yaml etc.) from ~metkit/share/metkit — set METKIT_HOME to this repository root to resolve against the in-repo share/metkit data.

Returns:

The set of resolved numeric ids (empty if the selection is invalid or resolves to no param value).

Return type:

set[int]

static _warn_context_unavailable() → None#

Notify that per-candidate MARS context computation is deferred.

The mars_request_context field of returned/raised candidates is currently always None. This is emitted once per process on the ambiguous path so callers know the context is not yet advertised (but that context= narrowing still works, and full context will come in a future release).

_make_candidate(entry: dict, siblings: list[dict], shortname: str, minimal: bool = True) → ParamIDCandidate#

Build a ParamIDCandidate from a metadata entry.

The candidate’s param_id, table, origin and access (the hard-filter metadata) are always populated. mars_request_context is currently left as None: per-candidate MARS context computation is deferred (see the “NOT YET IMPLEMENTED” section at the end of this class, _minimal_distinguishing_context(), dormant), so we no longer advertise which context selects each candidate. Callers can still pass context= to shortname_to_param_id() / shortname_to_param_id_candidates() to narrow the lookup.

The minimal parameter is retained for forward compatibility (it will re-enable the expand-oracle context computation once that ships) but is currently ignored.

_unique_hard_filter_selector(entry: dict, siblings: list[dict]) → dict | None#

Return the minimal hard-filter selector proven to select entry alone.

Builds table/origin/access selectors of increasing specificity and returns the first one that matches exactly this entry among siblings (all parameters sharing the short name). Returns None when no combination of the available hard filters uniquely identifies the entry — e.g. two ids sharing the same table, origin and access (such as short name ~ with ids 24 and 25). This ensures the API only ever advertises a selector that genuinely disambiguates, and otherwise honestly represents that no hard-filter selector exists.

static _candidate_sort_key(cand: ParamIDCandidate) → tuple#
static _normalise(raw: dict) → dict#

Return a normalised and validated parameter dict with canonical key names.

Uses ParameterEntry for field coercion and validation. Extra keys present in raw (e.g. "url", "units_id") are preserved in the returned dict so no information is discarded.

_index(entry: dict) → None#

Insert a normalised entry into the internal lookup dicts.

_ensure_loaded() → None#

Load parameter data on first access (lazy initialisation).

Subsequent calls are no-ops — once _loaded is True the method returns immediately without any I/O.

_load_online(cache_ttl: datetime.timedelta, cache_path: Path | str | None) → None#
_load_offline(yaml_path: Path | str | None = None) → None#
static _read_bundled_text(filename: str) → str | None#

Return the text of a bundled data file, or None if not found.

The resource is read while the importlib.resources context is active, so zip-safe (materialised-to-temp) resources are read before the temporary file is removed. Falls back to the dev-tree / editable-install filesystem locations.

Searches, in order:

  1. Via importlib.resources from the installed package (reliable in both regular installs and zip-safe wheels).

  2. Next to this module file (editable / development install layout).

  3. <repo_root>/share/metkit/ (development tree layout after the data files were moved out of the Python package directory).

classmethod _load_bundled_json() → list | None#

Load and parse bundled parameter_metadata.json.

Returns None if the JSON file is not found — caller should fall back to the YAML file.

classmethod _load_bundled_yaml() → list#

Load and parse bundled parameter_metadata.yaml.

Raises FileNotFoundError if the YAML file cannot be located.

_resolve_cache_dir(cache_path: Path | str | None) → Path | None#

Return the directory to use for the cache file, or None if unavailable.

_cache_file(cache_path: Path | str | None) → Path | None#

Return the full path to the cache file, or None if no cache dir is available.

_read_cache(cache_path: Path | str | None, cache_ttl: datetime.timedelta) → list | None#

Return the cached parameter list if it exists and is still fresh, otherwise return None.

_write_cache(params: list, cache_path: Path | str | None) → None#

Persist params to the cache file (best-effort; errors are silenced).

_resolve(identifier: int | str) → dict#

Resolve identifier (param_id, shortname, or longname) to a metadata dict.

shortname_to_longname(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → str#

Return the long name for shortname, given optional context.

Mirrors shortname_to_param_id(): ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/access filters, the behaviour depends on default:

  • default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup.

  • default=True — the long name of the canonical candidate is returned: the first in sorted order (lowest table, then origin/access, then lowest id).

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient.

  • default – When True, return the canonical (first-sorted) candidate’s long name instead of raising on ambiguity. Off by default.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved long name (or the canonical one when default=True and the lookup is ambiguous).

Return type:

str

Raises:

  • KeyError – If shortname is unknown, or no candidate survives the filters.

  • AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

longname_to_shortname(longname: str) → str#
shortname_to_param_id_candidates(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → list[ParamIDCandidate]#

Return all candidate paramids for shortname, each with its context.

The programmatic counterpart to AmbiguousParamError: it returns the candidate + context information as a normal value, so callers can inspect the options and then call shortname_to_param_id() with the narrowing context= (or table/origin/access) they want.

Two independent narrowing mechanisms are available and may be combined:

  • context — a dict of MARS keys resolved via the C++ expand engine (authoritative, cycle-correct). When the libmetkit is unavailable, the baked mars_request_context metadata is used as a fallback.

  • table / origin / access — direct hard filters on the candidate metadata, applied without constructing a MARS request.

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys used to pre-narrow the candidate set (e.g. {"class": "ai"}). When omitted, all candidates surviving the hard filters are returned.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

Every matching candidate, sorted by (table, origin, access, mars_request_context, param_id). Length 1 means the shortname (given any supplied context/filters) is unambiguous.

Return type:

list[ParamIDCandidate]

Raises:

KeyError – If shortname is unknown, or no candidate survives the filters.

_filter_shortname_entries(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → tuple[list[dict], list[dict]]#

Return (surviving_entries, all_siblings) for shortname.

Applies the table/origin/access hard filters and the MARS context filter (via the C++ expand engine). Does NOT compute minimal contexts — that expensive step is deferred to _make_candidate(). Raises KeyError if the shortname is unknown or no entry survives the filters.

shortname_to_param_id(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → int#

Return the single param ID for shortname, given optional context.

Ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/ access filters, the behaviour depends on default:

  • default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate (mars_request_context is currently None — see note below).

  • default=True — the canonical candidate is returned: the first in the sorted candidate order (lowest table / lowest id). For tp this is 228.

Note

Per-candidate MARS context computation is temporarily deferred, so every returned/raised ParamIDCandidate carries mars_request_context=None. Passing context= to narrow the lookup still works; only the advertised selecting context is unavailable for now.

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient — expand fills defaults for unspecified keys.

  • default – When True, return the canonical (first-sorted) candidate instead of raising on ambiguity. Off by default.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved paramid (or the canonical one when default=True and the lookup is ambiguous).

Return type:

int

Raises:

  • KeyError – If shortname is unknown, or no candidate survives the filters.

  • AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

param_id_to_shortname(param_id: int) → str#
static _param_context_from_cpp(param_id: int) → list[dict] | None#

Return MARS contexts for param_id from the C++ layer, if available.

Placeholder for the future metkit_param_context C API. That hook does not exist yet, so this always returns None to signal “not available”, causing param_id_to_context() to fall back to the precomputed YAML data. Once implemented, add it here and the public method will transparently prefer it.

param_id_to_context(param_id: int) → list[dict]#

Return the MARS-key contexts in which param_id is valid.

Each context is a dict of MARS keys (e.g. {"class": "ai", "stream": "enfo", "type": "cf", "levtype": "sfc"}) drawn from the authoritative params.yaml map. These are the raw contexts used to disambiguate shortname collisions.

Resolution order:

  1. The C++ layer (metkit_param_context) when available — see _param_context_from_cpp(). Not yet implemented.

  2. Fallback: the precomputed mars_request_context field baked into the bundled parameter metadata.

Parameters:

param_id – Numeric parameter id.

Returns:

Zero or more MARS context dicts. Empty if the id has no recorded context (e.g. not referenced in params.yaml).

Return type:

list[dict]

Raises:

KeyError – If param_id is not in the database.

longname_to_param_id(longname: str) → int#
param_id_to_longname(param_id: int) → str#
get_metadata(identifier: int | str) → dict#

Return the full metadata dictionary for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

get_units(identifier: int | str) → str#

Return the units string for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

Returns:

The units string, or "unknown" if not available.

Return type:

str

get_all_by_shortname(shortname: str) → list[dict]#

Return all parameter entries that share shortname.

Most short names map to exactly one param ID, but ~163 short names are reused across different GRIB parameter tables or originating centres. This method exposes every candidate so callers can inspect the collisions and choose the appropriate one.

Parameters:

shortname – ECMWF short name to look up.

Returns:

List of metadata dicts, sorted by ascending param ID. Each dict contains at minimum id, shortname, and longname.

Return type:

list[dict]

Raises:

KeyError – If shortname is not found in the database at all.

Examples

>>> db = ParamDB()
>>> entries = db.get_all_by_shortname("t")
>>> [(e["id"], e["longname"]) for e in entries]
[(130, 'Temperature'), (500014, 'Temperature')]
shortname_has_collisions(shortname: str) → bool#

Return True if shortname maps to more than one param ID.

Parameters:

shortname – ECMWF short name to check.

Raises:

KeyError – If shortname is not found in the database at all.

_minimal_distinguishing_context(entry: dict, siblings: list[dict], shortname: str) → dict | None#

Return the smallest MARS key-subset that selects entry over siblings.

Warning

DORMANT — not currently called from any public API. See the “NOT YET IMPLEMENTED” banner above this method.

Searches the key-subsets of entry’s baked mars_request_context dicts for the smallest subset that uniquely selects this id.

Uses expand as the authoritative oracle: a subset qualifies only if MarsRequest(param=shortname, **subset).expand() resolves to exactly this id. This guarantees the advertised context actually round-trips (bare {"class": "ai"} is rejected for tp because expand’s inherited defaults resolve it to 228, not 228228).

Returns:

  • {} — entry is the default: an empty context={} already resolves uniquely to it.

  • a non-empty dict — the minimal distinguishing MARS context.

  • None — no MARS context can uniquely select entry (residual ambiguity, or no baked context).

Return type:

dict | None

On this page
Show Source