pymetkit#

Submodules#

Attributes#

`_requests`
`_platformdirs`
`ffi`
`lib`

Exceptions#

`AmbiguousParamError`	Raised when a shortname (given the supplied context) maps to >1 paramid.
`MetKitException`	Raised when MetKit library throws exception
`CFFIModuleLoadFailed`	Raised when the shared library fails to load
`AmbiguousParamError`	Raised when a shortname (given the supplied context) maps to >1 paramid.

Classes#

`ParameterEntry`	A single entry from the ECMWF parameter database.
`ParamIDCandidate`	One possible paramid for a shortname, plus the context that selects it.
`MarsRequest`	Create MetKit MarsRequest object. Parameters and values in
`PatchedLib`	Patch a CFFI library with error handling
`ParamDB`	Parameter database providing metadata lookup for ECMWF parameters.
`ParamDB`	Parameter database providing metadata lookup for ECMWF parameters.
`ParamIDCandidate`	One possible paramid for a shortname, plus the context that selects it.
`ParameterEntry`	A single entry from the ECMWF parameter database.
`MarsRequestContext`	A MARS-key context that (partly) selects a parameter id.

Functions#

`ffi_encode`(→ bytes)
`ffi_decode`(→ str)
`parse_mars_request`(→ list[MarsRequest])	Function for parsing mars request from file object or string.

Package Contents#

class ParameterEntry#

Bases: pydantic.BaseModel

A single entry from the ECMWF parameter database.

Accepts both the canonical field names produced by _normalise and the raw aliases that may appear in YAML or API responses.

model_config#

id: int#

shortname: str#

longname: str#

units: str#

table: int | None#

mars_request_context: list[MarsRequestContext]#

origin_ids: list[int]#

access_ids: list[str]#

classmethod coerce_id_to_int(v: Any) → int#

classmethod normalise_shortname_key(v: Any) → str#

classmethod normalise_longname_key(v: Any) → str#

classmethod default_empty_units(v: Any) → str#

classmethod coerce_table(v: Any) → int | None#

classmethod coerce_origin_ids(v: Any) → list[int]#

classmethod coerce_access_ids(v: Any) → list[str]#

classmethod _normalise_aliases(data: Any) → Any#: Accept legacy key spellings from raw YAML / API payloads.

_requests = None#

_platformdirs = None#

ffi#

ffi_encode(data) → bytes#

ffi_decode(data: cffi.FFI.CData) → str#

class ParamIDCandidate#

One possible paramid for a shortname, plus the context that selects it.

param_id#: The candidate numeric parameter ID.

table#: GRIB parameter table the id encodes to (e.g. 128, 228).

origin#: WMO originating centre ids associated with this candidate — the full list (e.g. [0, 34, 98]), since several centres may share the id.

access#: Access categories (e.g. ["dissemination"]) — the full list.

mars_request_context#: The minimal set of MARS key/value pairs that, when passed as context= to ParamDB.shortname_to_param_id(), selects this candidate (e.g. {"class": "ai"}). Empty if no baked context is recorded or if no key-subset uniquely distinguishes this candidate.

param_id: int#

table: int#

origin: list[int]#

access: list[str]#

mars_request_context: dict#

exception AmbiguousParamError(shortname: str, candidates: list[ParamIDCandidate])#

Bases: KeyError

Raised when a shortname (given the supplied context) maps to >1 paramid.

shortname#: The shortname that could not be uniquely resolved.

candidates#: Every matching candidate, each carrying the context needed to select it, sorted by (table, origin, access, mars_request_context). The caller inspects these and re-calls with a narrowing context=.

Initialize self. See help(type(self)) for accurate signature.

shortname#

candidates#

class MarsRequest(verb: str | None = None, **kwargs)#

Create MetKit MarsRequest object. Parameters and values in the request can be specified through kwargs, noting that reserved words in Python must be suffixed with “_” e.g. “class_”

__request#

ctype() → cffi.FFI.CData#

verb() → str#

expand(inherit: bool = True, strict: bool = False) → MarsRequest#

Return expanded request

Params#

inherit: bool, if True, populates expanded request with default values strict: bool, if True, raise error instead of warning for invalid values

rtype:: Request, resulting from expansion

resolved_param() → int#

Return the single resolved param value as an int.

Convenience accessor for the shortname -> paramid resolution path: after building a request with param=<shortname> plus context keys and calling expand(), the C++ TypeParam::pass2 rewrites param to the numeric id. This reads that id back.

Returns:

The single resolved paramid.

Return type:

int

Raises:

KeyError – If the request has no param key.
ValueError – If param resolved to zero or more than one value (i.e. the context did not collapse to a single id).

validate()#

Check if request is valid against MARS language definition. Does not inherit missing parameters.

Raises:: Exception if request is incompatible with MARS language definition –

keys() → Iterator[str]#

Get iterator over parameters in request

Return type:: Iterator over parameter names

num_values(param: str) → int#

Number of values for parameter

Params#

param: parameter name

rtype:: int

merge(other: MarsRequest) → MarsRequest#

Merge the values in another request with existing request and returns result as a new Request object. Does not modify inputs to merge. Both input requests must contain the same values and the resulting request object must be compatible with MARS language definition

Params#

other: Request, request to merge with self

rtype:: Request, containing the result of the merge
raises ValueError if parameters in the two requests do not match:
raises MetKitException if resulting request is not compatible with MARS language definition:

__iter__() → Iterator[tuple[str, list[str]]]#

__getitem__(param: str) → str | list[str]#

__contains__(param: str) → bool#

__setitem__(param: str, values: int | str | list[str])#

__eq__(other: MarsRequest) → bool#

parse_mars_request(file_or_str: IO | str, strict: bool = False) → list[MarsRequest]#

Function for parsing mars request from file object or string.

Params#

file_or_str: string or file-like object, containing mars request strict: bool, whether to raise error or warning when request is not compatible with MARS language definition. In the case of warning, when False, the incompatible parameters are unset from the request.

rtype:: list of Request

exception MetKitException#

Bases: RuntimeError

Raised when MetKit library throws exception

Initialize self. See help(type(self)) for accurate signature.

exception CFFIModuleLoadFailed#

Bases: ImportError

Raised when the shared library fails to load

Initialize self. See help(type(self)) for accurate signature.

class PatchedLib#

Patch a CFFI library with error handling

Finds the header file associated with the MetKit C API and parses it, loads the shared library, and patches the accessors with automatic python-C error handling.

__read_header()#

__check_error(fn, name: str)#: If calls into the MetKit library return errors, ensure that they get detected and reported by throwing an appropriate python exception.

Parameter database providing metadata lookup for ECMWF parameters.

Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file).

When using mode="online" a local JSON cache is maintained so that repeated instantiations within the TTL window do not make a new HTTP request. The cache is stored under the OS user-cache directory (e.g. ~/.cache/pymetkit/ on Linux, ~/Library/Caches/pymetkit/ on macOS) using the fixed filename defined by _CACHE_FILENAME.

Shortname collision resolution#

Some short names (e.g. t, tp, u) are reused across different GRIB parameter tables and originating centres. When no context is given the default resolution priority is:

Prefer parameters with "dissemination" in their access_ids.
Among those, prefer parameters whose origin_ids include an origin from _DEFAULT_ORIGIN_PREFERENCE (tried in order).
Fall back to the lowest param ID.

Pass table=, origin=, or access= to shortname_to_param_id() / shortname_to_longname() to override this behaviour explicitly.

Initialise the parameter database.

The underlying data is loaded lazily — no file I/O or network request is made until the first lookup method is called. This makes instantiation cheap and safe to do at import time or inside hot paths.

param mode:: Either "online" (fetch from the ECMWF API) or "offline" (load from a YAML file).
type mode:: str
param cache_ttl:: How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when mode="online". Pass timedelta(0) to disable caching entirely (always fetch).
type cache_ttl:: datetime.timedelta, optional
param cache_path:: Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires platformdirs). Only relevant when mode="online".
type cache_path:: Path or str, optional
param yaml_path:: Path to a custom YAML file to load instead of the bundled parameter_metadata.yaml. The file must be a YAML list where each entry contains at minimum an id (integer), a short name (shortname / shortName / short_name), and a long name (longname / longName / long_name / name). Only valid with mode="offline"; raises ValueError if combined with mode="online".
type yaml_path:: Path or str, optional

_API_URL = 'https://codes.ecmwf.int/parameter-database/api/v1/param/'#

_CACHE_FILENAME = 'paramdb_online_cache.json'#

_DEFAULT_CACHE_TTL#

_REQUEST_TIMEOUT = 30#

_DEFAULT_ORIGIN_PREFERENCE: list[int] = [98, 0]#

_mode = 'offline'#

_effective_cache_ttl#

_cache_path = None#

_yaml_path = None#

_by_id: dict[int, dict]#

_by_shortname: dict[str, dict]#

_by_shortname_all: dict[str, list[dict]]#

_by_longname: dict[str, dict]#

_loaded: bool = False#

_ctx_cache: dict#

static _table_from_id(param_id: int) → int#

Decode the GRIB parameter table number from an encoded param ID.

The encoding scheme mirrors the C++ Param::paramId() logic:

IDs 1–999 → table 128 (classic ECMWF, table prefix suppressed)
IDs 1 000–999 999 → table * 1000 + param (e.g. 228228 → table 228)
IDs ≥ 1 000 000 → center * 1_000_000 + table * 1000 + param (e.g. 7001292 → center 7, table 1)

static _center_from_id(param_id: int) → int | None#

Decode the originating WMO center from an encoded param ID.

Returns None for IDs below 1 000 000 (ECMWF-local parameters). For IDs ≥ 1 000 000 the center is param_id // 1_000_000.

_resolve_shortname_with_context(shortname: str, table: int | None = None, origin: int | None = None, access: str | None = None) → dict#

Return the best-matching entry for shortname given optional context.

Parameters:

shortname – The ECMWF short name to look up.
table – GRIB parameter table number (e.g. 128 for classic ECMWF, 140 for ocean waves, 228 for “Standard 2”). When provided, only candidates whose encoded param ID belongs to this table are considered.
origin – WMO originating centre ID (e.g. 98 for ECMWF, 0 for WMO, 7 for NCEP). When provided, only candidates whose origin_ids list includes this value are considered.
access – Access category string (e.g. "dissemination"). When provided, only candidates whose access_ids list includes this value are considered.

Returns:

The matched parameter metadata entry.

Return type:

dict

Raises:

KeyError – If shortname is not found, or if no candidate matches the supplied context.

_context_resolved_ids(shortname: str, context: dict) → set[int] | None#

Resolve shortname + context to paramid(s) via the C++ engine.

Builds MarsRequest(param=shortname, **context), expands it (which runs TypeParam::pass2 and fills MARS defaults for unspecified keys), and reads back the resolved param value(s). Results are memoised on self._ctx_cache since each expand is a comparatively expensive (~ms) C++ round-trip and the minimal-context search repeats queries.

The C++ library reads its language files (params.yaml etc.) from ~metkit/share/metkit — set METKIT_HOME to this repository root to resolve against the in-repo share/metkit data.

Returns:: The set of resolved numeric ids, or None if the MetKit C library is unavailable (caller should fall back to baked contexts).
Return type:: set[int] | None

static _entry_matches_context(entry: dict, context: dict) → bool#

Offline fallback: does any baked context of entry satisfy context?

Used only when the C++ library is unavailable. An entry matches when at least one of its mars_request_context dicts contains every key == value pair in context.

_minimal_distinguishing_context(entry: dict, siblings: list[dict], shortname: str) → dict#

Return the smallest MARS key-subset that selects entry over siblings.

Searches the key-subsets of entry’s baked mars_request_context dicts for the smallest subset that uniquely selects this id.

When the MetKit C library is available, expand is used as the authoritative oracle: a subset qualifies only if MarsRequest(param=shortname, **subset).expand() resolves to exactly this id. This guarantees the advertised context actually round-trips (bare {"class": "ai"} is rejected for tp because expand’s inherited defaults resolve it to 228, not 228228).

Offline (no library), it falls back to a set-membership heuristic against the baked contexts of the sibling candidates.

Returns {} if no subset uniquely distinguishes entry (residual ambiguity) or if entry has no baked context.

_make_candidate(entry: dict, siblings: list[dict], shortname: str, minimal: bool = True) → ParamIDCandidate#

Build a ParamIDCandidate from a metadata entry.

When minimal is False the (expensive, expand-oracle) minimal distinguishing context is skipped and left empty — used when the entry is already unambiguous so no disambiguating context is required.

static _candidate_sort_key(cand: ParamIDCandidate) → tuple#

static _normalise(raw: dict) → dict#

Return a normalised and validated parameter dict with canonical key names.

Uses ParameterEntry for field coercion and validation. Extra keys present in raw (e.g. "url", "units_id") are preserved in the returned dict so no information is discarded.

_index(entry: dict) → None#: Insert a normalised entry into the internal lookup dicts.

_ensure_loaded() → None#

Load parameter data on first access (lazy initialisation).

Subsequent calls are no-ops — once _loaded is True the method returns immediately without any I/O.

_load_online(cache_ttl: datetime.timedelta, cache_path: Path | str | None) → None#

_load_offline(yaml_path: Path | str | None = None) → None#

static _find_offline_yaml() → pathlib.Path#

Locate parameter_metadata.yaml, searching in order:

Via importlib.resources from the installed package (reliable in both regular installs and zip-safe wheels).
Next to this module file (editable / development install layout).
<repo_root>/share/metkit/ (development tree layout after the YAML files were moved out of the Python package directory).

static _find_offline_json() → Path | None#

Locate parameter_metadata.json (fast-load format).

Returns None if the JSON file is not found — caller should fall back to the YAML file. Searches the same locations as _find_offline_yaml().

_resolve_cache_dir(cache_path: Path | str | None) → Path | None#: Return the directory to use for the cache file, or None if unavailable.

_cache_file(cache_path: Path | str | None) → Path | None#: Return the full path to the cache file, or None if no cache dir is available.

_read_cache(cache_path: Path | str | None, cache_ttl: datetime.timedelta) → list | None#: Return the cached parameter list if it exists and is still fresh, otherwise return None.

_write_cache(params: list, cache_path: Path | str | None) → None#: Persist params to the cache file (best-effort; errors are silenced).

_resolve(identifier: int | str) → dict#: Resolve identifier (param_id, shortname, or longname) to a metadata dict.

shortname_to_longname(shortname: str, table: int | None = None, origin: int | None = None, access: str | None = None) → str#

Return the long name for shortname.

Parameters:

shortname – ECMWF short name (e.g. "t", "tp").
table – Optional GRIB parameter table number to disambiguate collisions (e.g. 128 for classic ECMWF, 140 for ocean waves).
origin – Optional WMO originating centre ID (e.g. 98 for ECMWF, 0 for WMO, 7 for NCEP).
access – Optional access category filter (e.g. "dissemination").

longname_to_shortname(longname: str) → str#

shortname_to_param_id_candidates(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → list[ParamIDCandidate]#

Return all candidate paramids for shortname, each with its context.

The programmatic counterpart to AmbiguousParamError: it returns the candidate + context information as a normal value, so callers can inspect the options and then call shortname_to_param_id() with the narrowing context= (or table/origin/access) they want.

Two independent narrowing mechanisms are available and may be combined:

context — a dict of MARS keys resolved via the C++ expand engine (authoritative, cycle-correct). When the MetKit C library is unavailable, the baked mars_request_context metadata is used as a fallback.
table / origin / access — direct hard filters on the candidate metadata, applied WITHOUT constructing a MARS request.

Parameters:

shortname – ECMWF short name (e.g. "t", "tp").
context – Optional dict of MARS keys used to pre-narrow the candidate set (e.g. {"class": "ai"}). When omitted, all candidates surviving the hard filters are returned.
table – Optional hard filter — GRIB parameter table number.
origin – Optional hard filter — WMO originating centre id (membership).
access – Optional hard filter — access category string (membership).

Returns:

Every matching candidate, sorted by (table, origin, access, mars_request_context, param_id). Length 1 means the shortname (given any supplied context/filters) is unambiguous.

Return type:

list[ParamIDCandidate]

Raises:

KeyError – If shortname is unknown, or no candidate survives the filters.

_filter_shortname_entries(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → tuple[list[dict], list[dict]]#

Return (surviving_entries, all_siblings) for shortname.

Applies the table/origin/access hard filters and the MARS context filter (via the C++ expand engine, or the baked-context offline fallback). Does NOT compute minimal contexts — that expensive step is deferred to _make_candidate(). Raises KeyError if the shortname is unknown or no entry survives the filters.

shortname_to_param_id(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → int#

Return the single param ID for shortname, given optional context.

Ambiguity is never resolved by guessing. When more than one candidate remains after applying context and the table/origin/ access filters, AmbiguousParamError is raised; its .candidates attribute lists every remaining id with the context needed to select it.

Parameters:

shortname – ECMWF short name (e.g. "t", "tp").
context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient — expand fills defaults for unspecified keys.
table – Optional hard filter — GRIB parameter table number.
origin – Optional hard filter — WMO originating centre id (membership).
access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved paramid.

Return type:

int

Raises:

KeyError – If shortname is unknown, or no candidate survives the filters.
AmbiguousParamError – If more than one candidate remains after applying context/filters.

param_id_to_shortname(param_id: int) → str#

static _param_context_from_cpp(param_id: int) → list[dict] | None#

Return MARS contexts for param_id from the C++ layer, if available.

Placeholder for the future metkit_param_context C API (outcomes §4). That hook does not exist yet, so this always returns None to signal “not available”, causing param_id_to_context() to fall back to the precomputed YAML data. Once the C API lands, implement it here and the public method will transparently prefer it.

param_id_to_context(param_id: int) → list[dict]#

Return the MARS-key contexts in which param_id is valid.

Each context is a dict of MARS keys (e.g. {"class": "ai", "stream": "enfo", "type": "cf", "levtype": "sfc"}) drawn from the authoritative params.yaml map. These are the raw contexts used to disambiguate shortname collisions.

Resolution order:

The C++ layer (metkit_param_context) when available — see _param_context_from_cpp(). Not yet implemented.
Fallback: the precomputed mars_request_context field baked into the bundled parameter metadata.

Parameters:: param_id – Numeric parameter id.
Returns:: Zero or more MARS context dicts. Empty if the id has no recorded context (e.g. not referenced in params.yaml).
Return type:: list[dict]
Raises:: KeyError – If param_id is not in the database.

longname_to_param_id(longname: str) → int#

param_id_to_longname(param_id: int) → str#

get_metadata(identifier: int | str) → dict#

Return the full metadata dictionary for a parameter.

Parameters:: identifier (int or str) – A param ID (int), shortname, or longname.

get_units(identifier: int | str) → str#

Return the units string for a parameter.

Parameters:: identifier (int or str) – A param ID (int), shortname, or longname.
Returns:: The units string, or "unknown" if not available.
Return type:: str

get_all_by_shortname(shortname: str) → list[dict]#

Return all parameter entries that share shortname.

Most short names map to exactly one param ID, but ~163 short names are reused across different GRIB parameter tables or originating centres. This method exposes every candidate so callers can inspect the collisions and choose the appropriate one.

Parameters:: shortname – ECMWF short name to look up.
Returns:: List of metadata dicts, sorted by ascending param ID. Each dict contains at minimum id, shortname, and longname.
Return type:: list[dict]
Raises:: KeyError – If shortname is not found in the database at all.

Examples

>>> db = ParamDB()
>>> entries = db.get_all_by_shortname("t")
>>> [(e["id"], e["longname"]) for e in entries]
[(130, 'Temperature'), (500014, 'Temperature')]

shortname_has_collisions(shortname: str) → bool#

Return True if shortname maps to more than one param ID.

Parameters:: shortname – ECMWF short name to check.
Raises:: KeyError – If shortname is not found in the database at all.

lib#

Parameter database providing metadata lookup for ECMWF parameters.

Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file).

Shortname collision resolution#

Some short names (e.g. t, tp, u) are reused across different GRIB parameter tables and originating centres. When no context is given the default resolution priority is:

Prefer parameters with "dissemination" in their access_ids.
Among those, prefer parameters whose origin_ids include an origin from _DEFAULT_ORIGIN_PREFERENCE (tried in order).
Fall back to the lowest param ID.

Pass table=, origin=, or access= to shortname_to_param_id() / shortname_to_longname() to override this behaviour explicitly.

Initialise the parameter database.

param mode:: Either "online" (fetch from the ECMWF API) or "offline" (load from a YAML file).
type mode:: str
param cache_ttl:: How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when mode="online". Pass timedelta(0) to disable caching entirely (always fetch).
type cache_ttl:: datetime.timedelta, optional
param cache_path:: Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires platformdirs). Only relevant when mode="online".
type cache_path:: Path or str, optional
param yaml_path:: Path to a custom YAML file to load instead of the bundled parameter_metadata.yaml. The file must be a YAML list where each entry contains at minimum an id (integer), a short name (shortname / shortName / short_name), and a long name (longname / longName / long_name / name). Only valid with mode="offline"; raises ValueError if combined with mode="online".
type yaml_path:: Path or str, optional

_API_URL = 'https://codes.ecmwf.int/parameter-database/api/v1/param/'#

_CACHE_FILENAME = 'paramdb_online_cache.json'#

_DEFAULT_CACHE_TTL#

_REQUEST_TIMEOUT = 30#

_DEFAULT_ORIGIN_PREFERENCE: list[int] = [98, 0]#

_mode = 'offline'#

_effective_cache_ttl#

_cache_path = None#

_yaml_path = None#

_by_id: dict[int, dict]#

_by_shortname: dict[str, dict]#

_by_shortname_all: dict[str, list[dict]]#

_by_longname: dict[str, dict]#

_loaded: bool = False#

_ctx_cache: dict#

static _table_from_id(param_id: int) → int#

Decode the GRIB parameter table number from an encoded param ID.

The encoding scheme mirrors the C++ Param::paramId() logic:

IDs 1–999 → table 128 (classic ECMWF, table prefix suppressed)
IDs 1 000–999 999 → table * 1000 + param (e.g. 228228 → table 228)
IDs ≥ 1 000 000 → center * 1_000_000 + table * 1000 + param (e.g. 7001292 → center 7, table 1)

static _center_from_id(param_id: int) → int | None#

Decode the originating WMO center from an encoded param ID.

Returns None for IDs below 1 000 000 (ECMWF-local parameters). For IDs ≥ 1 000 000 the center is param_id // 1_000_000.

_resolve_shortname_with_context(shortname: str, table: int | None = None, origin: int | None = None, access: str | None = None) → dict#

Return the best-matching entry for shortname given optional context.

Parameters:

shortname – The ECMWF short name to look up.
table – GRIB parameter table number (e.g. 128 for classic ECMWF, 140 for ocean waves, 228 for “Standard 2”). When provided, only candidates whose encoded param ID belongs to this table are considered.
origin – WMO originating centre ID (e.g. 98 for ECMWF, 0 for WMO, 7 for NCEP). When provided, only candidates whose origin_ids list includes this value are considered.
access – Access category string (e.g. "dissemination"). When provided, only candidates whose access_ids list includes this value are considered.

Returns:

The matched parameter metadata entry.

Return type:

dict

Raises:

KeyError – If shortname is not found, or if no candidate matches the supplied context.

_context_resolved_ids(shortname: str, context: dict) → set[int] | None#

Resolve shortname + context to paramid(s) via the C++ engine.

The C++ library reads its language files (params.yaml etc.) from ~metkit/share/metkit — set METKIT_HOME to this repository root to resolve against the in-repo share/metkit data.

Returns:: The set of resolved numeric ids, or None if the MetKit C library is unavailable (caller should fall back to baked contexts).
Return type:: set[int] | None