pymetkit.paramdb

Submodules

• pymetkit.paramdb._db
• pymetkit.paramdb.generate_metadata
• pymetkit.paramdb.models

Attributes

_HAVE_EXPAND

Exceptions

AmbiguousParamError

Raised when a shortname (given the supplied context) maps to >1 paramid.

Classes

ParamDB	Parameter database providing metadata lookup for ECMWF parameters.
ParamIDCandidate	One possible paramid for a shortname, plus the context that selects it.
ParameterEntry	A single entry from the ECMWF parameter database.
MarsRequestContext	A MARS-key context that (partly) selects a parameter id.

Package Contents

class ParamDB(mode: str = 'offline', cache_ttl: timedelta | None = None, cache_path: Path | str | None = None, yaml_path: Path | str | None = None)

Parameter database providing metadata lookup for ECMWF parameters.

Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file).

When using mode="online" a local JSON cache is maintained so that repeated instantiations within the TTL window do not make a new HTTP request. The cache is stored under the OS user-cache directory (e.g. ~/.cache/pymetkit/ on Linux, ~/Library/Caches/pymetkit/ on macOS) using the fixed filename defined by _CACHE_FILENAME.

Shortname collision resolution

Some short names (e.g. t, tp, u) are reused across different GRIB parameter tables and originating centres. Collisions are not silently guessed by shortname_to_param_id():

• shortname_to_param_id() raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. The error’s .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id); for tp this is 228. Note this is not the canonical paramID as without mars context this cannot be determined.

• shortname_to_longname() behaves the same way: it raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id).

To resolve a collision explicitly, narrow the lookup with a MARS context= (resolved via the C++ expand engine) or the table=/origin=/access= hard filters, both accepted by shortname_to_param_id() and shortname_to_longname().

Initialise the parameter database.

The underlying data is loaded lazily — no file I/O or network request is made until the first lookup method is called. This makes instantiation cheap and safe to do at import time or inside hot paths.

param mode:

Either "online" (fetch from the ECMWF API) or "offline" (load from a YAML file).

type mode:

str

param cache_ttl:

How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when mode="online". Pass timedelta(0) to disable caching entirely (always fetch).

type cache_ttl:

datetime.timedelta, optional

param cache_path:

Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires platformdirs). Only relevant when mode="online".

type cache_path:

Path or str, optional

param yaml_path:

Path to a custom YAML file to load instead of the bundled parameter_metadata.yaml. The file must be a YAML list where each entry contains at minimum an id (integer), a short name (shortname), and a long name (longname). Only the canonical key spellings are accepted. Only valid with mode="offline"; raises ValueError if combined with mode="online".

type yaml_path:

Path or str, optional

_API_URL = 'https://codes.ecmwf.int/parameter-database/api/v1/param/'

_CACHE_FILENAME = 'paramdb_online_cache.json'

_DEFAULT_CACHE_TTL

_REQUEST_TIMEOUT = 30

_context_notice_emitted = False

_mode = 'offline'

_effective_cache_ttl

_cache_path = None

_yaml_path = None

_by_id: dict[int, dict]

_by_shortname: dict[str, dict]

_by_shortname_all: dict[str, list[dict]]

_by_longname: dict[str, dict]

_loaded: bool = False

_ctx_cache: dict

static _table_from_id(param_id: int) → int

Decode the GRIB parameter table number from an encoded param ID.

The encoding scheme mirrors the C++ Param::paramId() logic:

• IDs 1–999 → table 128 (classic ECMWF, table prefix suppressed)

• IDs 1 000–999 999 → table * 1000 + param (e.g. 228228 → table 228)

• IDs ≥ 1 000 000 → center * 1_000_000 + table * 1000 + param (e.g. 7001292 → center 7, table 1)

static _center_from_id(param_id: int) → int | None

Decode the originating WMO center from an encoded param ID.

Returns None for IDs below 1 000 000 (ECMWF-local parameters). For IDs ≥ 1 000 000 the center is param_id // 1_000_000.

_context_resolved_ids(shortname: str, context: dict) → set[int]

Resolve shortname + context to paramid(s) via the C++ engine.

Builds MarsRequest(param=shortname, **context), expands it (which runs TypeParam::pass2 and fills MARS defaults for unspecified keys), and reads back the resolved param value(s). Results are memoised on self._ctx_cache since each expand is a comparatively expensive (~ms) C++ round-trip and the minimal-context search repeats queries.

The C++ library reads its language files (params.yaml etc.) from ~metkit/share/metkit — set METKIT_HOME to this repository root to resolve against the in-repo share/metkit data.

Returns:

The set of resolved numeric ids (empty if the selection is invalid or resolves to no param value).

Return type:

set[int]

static _warn_context_unavailable() → None

Notify that per-candidate MARS context computation is deferred.

The mars_request_context field of returned/raised candidates is currently always None. This is emitted once per process on the ambiguous path so callers know the context is not yet advertised (but that context= narrowing still works, and full context will come in a future release).

_make_candidate(entry: dict, siblings: list[dict], shortname: str, minimal: bool = True) → ParamIDCandidate

Build a ParamIDCandidate from a metadata entry.

The candidate’s param_id, table, origin and access (the hard-filter metadata) are always populated. mars_request_context is currently left as None: per-candidate MARS context computation is deferred (see the “NOT YET IMPLEMENTED” section at the end of this class, _minimal_distinguishing_context(), dormant), so we no longer advertise which context selects each candidate. Callers can still pass context= to shortname_to_param_id() / shortname_to_param_id_candidates() to narrow the lookup.

The minimal parameter is retained for forward compatibility (it will re-enable the expand-oracle context computation once that ships) but is currently ignored.

_unique_hard_filter_selector(entry: dict, siblings: list[dict]) → dict | None

Return the minimal hard-filter selector proven to select entry alone.

Builds table/origin/access selectors of increasing specificity and returns the first one that matches exactly this entry among siblings (all parameters sharing the short name). Returns None when no combination of the available hard filters uniquely identifies the entry — e.g. two ids sharing the same table, origin and access (such as short name ~ with ids 24 and 25). This ensures the API only ever advertises a selector that genuinely disambiguates, and otherwise honestly represents that no hard-filter selector exists.

static _candidate_sort_key(cand: ParamIDCandidate) → tuple

static _normalise(raw: dict) → dict

Return a normalised and validated parameter dict with canonical key names.

Uses ParameterEntry for field coercion and validation. Extra keys present in raw (e.g. "url", "units_id") are preserved in the returned dict so no information is discarded.

_index(entry: dict) → None

Insert a normalised entry into the internal lookup dicts.

_ensure_loaded() → None

Load parameter data on first access (lazy initialisation).

Subsequent calls are no-ops — once _loaded is True the method returns immediately without any I/O.

_load_online(cache_ttl: datetime.timedelta, cache_path: Path | str | None) → None

_load_offline(yaml_path: Path | str | None = None) → None

static _read_bundled_text(filename: str) → str | None

Return the text of a bundled data file, or None if not found.

The resource is read while the importlib.resources context is active, so zip-safe (materialised-to-temp) resources are read before the temporary file is removed. Falls back to the dev-tree / editable-install filesystem locations.

Searches, in order:

1. Via importlib.resources from the installed package (reliable in both regular installs and zip-safe wheels).

2. Next to this module file (editable / development install layout).

3. <repo_root>/share/metkit/ (development tree layout after the data files were moved out of the Python package directory).

classmethod _load_bundled_json() → list | None

Load and parse bundled parameter_metadata.json.

Returns None if the JSON file is not found — caller should fall back to the YAML file.

classmethod _load_bundled_yaml() → list

Load and parse bundled parameter_metadata.yaml.

Raises FileNotFoundError if the YAML file cannot be located.

_resolve_cache_dir(cache_path: Path | str | None) → Path | None

Return the directory to use for the cache file, or None if unavailable.

_cache_file(cache_path: Path | str | None) → Path | None

Return the full path to the cache file, or None if no cache dir is available.

_read_cache(cache_path: Path | str | None, cache_ttl: datetime.timedelta) → list | None

Return the cached parameter list if it exists and is still fresh, otherwise return None.

_write_cache(params: list, cache_path: Path | str | None) → None

Persist params to the cache file (best-effort; errors are silenced).

_resolve(identifier: int | str) → dict

Resolve identifier (param_id, shortname, or longname) to a metadata dict.

shortname_to_longname(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → str

Return the long name for shortname, given optional context.

Mirrors shortname_to_param_id(): ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/access filters, the behaviour depends on default:

• default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup.

• default=True — the long name of the canonical candidate is returned: the first in sorted order (lowest table, then origin/access, then lowest id).

Parameters:

• shortname – ECMWF short name (e.g. "t", "tp").

• context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient.

• default – When True, return the canonical (first-sorted) candidate’s long name instead of raising on ambiguity. Off by default.

• table – Optional hard filter — GRIB parameter table number.

• origin – Optional hard filter — WMO originating centre id (membership).

• access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved long name (or the canonical one when default=True and the lookup is ambiguous).

Return type:

str

Raises:

• KeyError – If shortname is unknown, or no candidate survives the filters.

• AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

longname_to_shortname(longname: str) → str

shortname_to_param_id_candidates(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → list[ParamIDCandidate]

Return all candidate paramids for shortname, each with its context.

The programmatic counterpart to AmbiguousParamError: it returns the candidate + context information as a normal value, so callers can inspect the options and then call shortname_to_param_id() with the narrowing context= (or table/origin/access) they want.

Two independent narrowing mechanisms are available and may be combined:

• context — a dict of MARS keys resolved via the C++ expand engine (authoritative, cycle-correct). When the libmetkit is unavailable, the baked mars_request_context metadata is used as a fallback.

• table / origin / access — direct hard filters on the candidate metadata, applied without constructing a MARS request.

Parameters:

• shortname – ECMWF short name (e.g. "t", "tp").

• context – Optional dict of MARS keys used to pre-narrow the candidate set (e.g. {"class": "ai"}). When omitted, all candidates surviving the hard filters are returned.

• table – Optional hard filter — GRIB parameter table number.

• origin – Optional hard filter — WMO originating centre id (membership).

• access – Optional hard filter — access category string (membership).

Returns:

Every matching candidate, sorted by (table, origin, access, mars_request_context, param_id). Length 1 means the shortname (given any supplied context/filters) is unambiguous.

Return type:

list[ParamIDCandidate]

Raises:

KeyError – If shortname is unknown, or no candidate survives the filters.

_filter_shortname_entries(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → tuple[list[dict], list[dict]]

Return (surviving_entries, all_siblings) for shortname.

Applies the table/origin/access hard filters and the MARS context filter (via the C++ expand engine). Does NOT compute minimal contexts — that expensive step is deferred to _make_candidate(). Raises KeyError if the shortname is unknown or no entry survives the filters.

shortname_to_param_id(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → int

Return the single param ID for shortname, given optional context.

Ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/ access filters, the behaviour depends on default:

• default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate (mars_request_context is currently None — see note below).

• default=True — the canonical candidate is returned: the first in the sorted candidate order (lowest table / lowest id). For tp this is 228.

Note

Per-candidate MARS context computation is temporarily deferred, so every returned/raised ParamIDCandidate carries mars_request_context=None. Passing context= to narrow the lookup still works; only the advertised selecting context is unavailable for now.

Parameters:

• shortname – ECMWF short name (e.g. "t", "tp").

• context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient — expand fills defaults for unspecified keys.

• default – When True, return the canonical (first-sorted) candidate instead of raising on ambiguity. Off by default.

• table – Optional hard filter — GRIB parameter table number.

• origin – Optional hard filter — WMO originating centre id (membership).

• access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved paramid (or the canonical one when default=True and the lookup is ambiguous).

Return type:

int

Raises:

• KeyError – If shortname is unknown, or no candidate survives the filters.

• AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

param_id_to_shortname(param_id: int) → str

static _param_context_from_cpp(param_id: int) → list[dict] | None

Return MARS contexts for param_id from the C++ layer, if available.

Placeholder for the future metkit_param_context C API. That hook does not exist yet, so this always returns None to signal “not available”, causing param_id_to_context() to fall back to the precomputed YAML data. Once implemented, add it here and the public method will transparently prefer it.

param_id_to_context(param_id: int) → list[dict]

Return the MARS-key contexts in which param_id is valid.

Each context is a dict of MARS keys (e.g. {"class": "ai", "stream": "enfo", "type": "cf", "levtype": "sfc"}) drawn from the authoritative params.yaml map. These are the raw contexts used to disambiguate shortname collisions.

Resolution order:

1. The C++ layer (metkit_param_context) when available — see _param_context_from_cpp(). Not yet implemented.

2. Fallback: the precomputed mars_request_context field baked into the bundled parameter metadata.

Parameters:

param_id – Numeric parameter id.

Returns:

Zero or more MARS context dicts. Empty if the id has no recorded context (e.g. not referenced in params.yaml).

Return type:

list[dict]

Raises:

KeyError – If param_id is not in the database.

longname_to_param_id(longname: str) → int

param_id_to_longname(param_id: int) → str

get_metadata(identifier: int | str) → dict

Return the full metadata dictionary for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

get_units(identifier: int | str) → str

Return the units string for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

Returns:

The units string, or "unknown" if not available.

Return type:

str

get_all_by_shortname(shortname: str) → list[dict]

Return all parameter entries that share shortname.

Most short names map to exactly one param ID, but ~163 short names are reused across different GRIB parameter tables or originating centres. This method exposes every candidate so callers can inspect the collisions and choose the appropriate one.

Parameters:

shortname – ECMWF short name to look up.

Returns:

List of metadata dicts, sorted by ascending param ID. Each dict contains at minimum id, shortname, and longname.

Return type:

list[dict]

Raises:

KeyError – If shortname is not found in the database at all.

Examples

>>> db = ParamDB()
>>> entries = db.get_all_by_shortname("t")
>>> [(e["id"], e["longname"]) for e in entries]
[(130, 'Temperature'), (500014, 'Temperature')]

shortname_has_collisions(shortname: str) → bool

Return True if shortname maps to more than one param ID.

Parameters:

shortname – ECMWF short name to check.

Raises:

KeyError – If shortname is not found in the database at all.

_minimal_distinguishing_context(entry: dict, siblings: list[dict], shortname: str) → dict | None

Return the smallest MARS key-subset that selects entry over siblings.

Warning

DORMANT — not currently called from any public API. See the “NOT YET IMPLEMENTED” banner above this method.

Searches the key-subsets of entry’s baked mars_request_context dicts for the smallest subset that uniquely selects this id.

Uses expand as the authoritative oracle: a subset qualifies only if MarsRequest(param=shortname, **subset).expand() resolves to exactly this id. This guarantees the advertised context actually round-trips (bare {"class": "ai"} is rejected for tp because expand’s inherited defaults resolve it to 228, not 228228).

Returns:

• {} — entry is the default: an empty context={} already resolves uniquely to it.

• a non-empty dict — the minimal distinguishing MARS context.

• None — no MARS context can uniquely select entry (residual ambiguity, or no baked context).

Return type:

dict | None

class ParamIDCandidate

One possible paramid for a shortname, plus the context that selects it.

param_id

The candidate numeric parameter ID.

table

GRIB parameter table the id encodes to (e.g. 128, 228).

origin

WMO originating centre ids associated with this candidate — the full list (e.g. [0, 34, 98]), since several centres may share the id.

access

Access categories (e.g. ["dissemination"]) — the full list.

mars_request_context

The minimal set of MARS key/value pairs that, when passed as context= to ParamDB.shortname_to_param_id(), selects this candidate (e.g. {"class": "ai"}). Special values:

• {} — this candidate is the default: an empty context={} resolves to it via the C++ expand layer.

• None — no MARS context can select this candidate; use the hard filters instead (see hard_filter_selector).

hard_filter_selector

The minimal table/origin/access kwargs that, when passed to ParamDB.shortname_to_param_id(), are proven to select exactly this candidate among all parameters sharing the short name. Special value:

• None — no combination of the available hard filters uniquely identifies this candidate (e.g. two ids share the same table, origin and access). In that case there is no hard-filter selector to advertise, and the collision cannot be resolved by hard filters alone.

param_id: int

table: int

origin: list[int]

access: list[str]

mars_request_context: dict | None = None

hard_filter_selector: dict | None = None

exception AmbiguousParamError(shortname: str, candidates: list[ParamIDCandidate])

Bases: KeyError

Raised when a shortname (given the supplied context) maps to >1 paramid.

shortname

The shortname that could not be uniquely resolved.

candidates

Every matching candidate, each carrying the context needed to select it, sorted by (table, origin, access, mars_request_context). The caller inspects these and re-calls with a narrowing context=.

Initialize self.  See help(type(self)) for accurate signature.

shortname

candidates

_HAVE_EXPAND = True

class ParameterEntry(/, **data: Any)

Bases: pydantic.BaseModel

A single entry from the ECMWF parameter database.

This model accepts the canonical field names only (shortname / longname). Raw API alias spellings (e.g. shortName, name) are NOT accepted and must be normalised to canonical keys before validation.

The JSON schema emitted from this model (and bundled as parameter_entry_schema.json) mirrors this contract: it validates the canonical keys only.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

id: int = None

shortname: str = None

longname: str = None

units: str = None

table: int | None = None

mars_request_context: list[MarsRequestContext] = None

origin_ids: list[int] = None

access_ids: list[str] = None

classmethod coerce_id_to_int(v: Any) → int

classmethod normalise_shortname_key(v: Any) → str

classmethod normalise_longname_key(v: Any) → str

classmethod default_empty_units(v: Any) → str

classmethod coerce_table(v: Any) → int | None

classmethod coerce_origin_ids(v: Any) → list[int]

classmethod coerce_access_ids(v: Any) → list[str]

class MarsRequestContext(/, **data: Any)

Bases: pydantic.BaseModel

A MARS-key context that (partly) selects a parameter id.

This mirrors a single matcher rule from share/metkit/params.yaml. Its schema is kept separate from ParameterEntry so the context contract can evolve independently and so user-supplied context schemas can validate against it. All fields are optional because params.yaml rules do not always constrain every key (e.g. levtype is occasionally absent).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class_: str | None = None

stream: str | None = None

type: str | None = None

levtype: str | None = None