Parameter Database#

pymetkit.paramdb maps between ECMWF short names, long names and numeric parameter IDs. It is backed by a bundled parameter_metadata.json (with a parameter_metadata.yaml fallback loaded only when JSON is absent), or by the live ECMWF parameter database API in mode="online".

Usage#

from pymetkit.paramdb import ParamDB, AmbiguousParamError

db = ParamDB()                        # mode="offline" by default; loads lazily

db.shortname_to_param_id("msl")       # 151  — unambiguous
db.param_id_to_shortname(151)         # "msl"
db.shortname_to_longname("2t")        # "2 metre temperature"
db.get_units(167)                     # "K"

Ambiguous short names#

Some short names map to more than one parameter ID (e.g. tp → 228 and 228228). ParamDB never guesses — an ambiguous lookup raises by default:

try:
    db.shortname_to_param_id("tp")
except AmbiguousParamError as exc:
    for cand in exc.candidates:       # every ParamIDCandidate, sorted
        print(cand.param_id, cand.table)

Resolve the ambiguity in three ways:

# 1. Narrow with a MARS context (resolved via the C++ expand engine)
db.shortname_to_param_id("tp", context={"class": "od"})   # 228

# 2. Narrow with hard metadata filters (no MARS request constructed)
db.shortname_to_param_id("tp", table=128)                 # 228

# 3. Accept the canonical (first-sorted, lowest-table/id) candidate
db.shortname_to_param_id("tp", default=True)              # 228

API reference#

class ParamDB(mode: str = 'offline', cache_ttl: timedelta | None = None, cache_path: Path | str | None = None, yaml_path: Path | str | None = None)#

Parameter database providing metadata lookup for ECMWF parameters.

Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file).

When using mode="online" a local JSON cache is maintained so that repeated instantiations within the TTL window do not make a new HTTP request. The cache is stored under the OS user-cache directory (e.g. ~/.cache/pymetkit/ on Linux, ~/Library/Caches/pymetkit/ on macOS) using the fixed filename defined by _CACHE_FILENAME.

Shortname collision resolution#

Some short names (e.g. t, tp, u) are reused across different GRIB parameter tables and originating centres. Collisions are not silently guessed by shortname_to_param_id():

  • shortname_to_param_id() raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. The error’s .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id); for tp this is 228. Note this is not the canonical paramID as without mars context this cannot be determined.

  • shortname_to_longname() behaves the same way: it raises AmbiguousParamError when more than one candidate remains after applying any context and the table/origin/access hard filters. Pass default=True to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id).

To resolve a collision explicitly, narrow the lookup with a MARS context= (resolved via the C++ expand engine) or the table=/origin=/access= hard filters, both accepted by shortname_to_param_id() and shortname_to_longname().

Initialise the parameter database.

The underlying data is loaded lazily — no file I/O or network request is made until the first lookup method is called. This makes instantiation cheap and safe to do at import time or inside hot paths.

param mode:

Either "online" (fetch from the ECMWF API) or "offline" (load from a YAML file).

type mode:

str

param cache_ttl:

How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when mode="online". Pass timedelta(0) to disable caching entirely (always fetch).

type cache_ttl:

datetime.timedelta, optional

param cache_path:

Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires platformdirs). Only relevant when mode="online".

type cache_path:

Path or str, optional

param yaml_path:

Path to a custom YAML file to load instead of the bundled parameter_metadata.yaml. The file must be a YAML list where each entry contains at minimum an id (integer), a short name (shortname), and a long name (longname). Only the canonical key spellings are accepted. Only valid with mode="offline"; raises ValueError if combined with mode="online".

type yaml_path:

Path or str, optional

get_all_by_shortname(shortname: str) → list[dict]#

Return all parameter entries that share shortname.

Most short names map to exactly one param ID, but ~163 short names are reused across different GRIB parameter tables or originating centres. This method exposes every candidate so callers can inspect the collisions and choose the appropriate one.

Parameters:

shortname – ECMWF short name to look up.

Returns:

List of metadata dicts, sorted by ascending param ID. Each dict contains at minimum id, shortname, and longname.

Return type:

list[dict]

Raises:

KeyError – If shortname is not found in the database at all.

Examples

>>> db = ParamDB()
>>> entries = db.get_all_by_shortname("t")
>>> [(e["id"], e["longname"]) for e in entries]
[(130, 'Temperature'), (500014, 'Temperature')]
get_metadata(identifier: int | str) → dict#

Return the full metadata dictionary for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

get_units(identifier: int | str) → str#

Return the units string for a parameter.

Parameters:

identifier (int or str) – A param ID (int), shortname, or longname.

Returns:

The units string, or "unknown" if not available.

Return type:

str

longname_to_param_id(longname: str) → int#
longname_to_shortname(longname: str) → str#
param_id_to_context(param_id: int) → list[dict]#

Return the MARS-key contexts in which param_id is valid.

Each context is a dict of MARS keys (e.g. {"class": "ai", "stream": "enfo", "type": "cf", "levtype": "sfc"}) drawn from the authoritative params.yaml map. These are the raw contexts used to disambiguate shortname collisions.

Resolution order:

  1. The C++ layer (metkit_param_context) when available — see _param_context_from_cpp(). Not yet implemented.

  2. Fallback: the precomputed mars_request_context field baked into the bundled parameter metadata.

Parameters:

param_id – Numeric parameter id.

Returns:

Zero or more MARS context dicts. Empty if the id has no recorded context (e.g. not referenced in params.yaml).

Return type:

list[dict]

Raises:

KeyError – If param_id is not in the database.

param_id_to_longname(param_id: int) → str#
param_id_to_shortname(param_id: int) → str#
shortname_has_collisions(shortname: str) → bool#

Return True if shortname maps to more than one param ID.

Parameters:

shortname – ECMWF short name to check.

Raises:

KeyError – If shortname is not found in the database at all.

shortname_to_longname(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → str#

Return the long name for shortname, given optional context.

Mirrors shortname_to_param_id(): ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/access filters, the behaviour depends on default:

  • default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate so the caller can narrow the lookup.

  • default=True — the long name of the canonical candidate is returned: the first in sorted order (lowest table, then origin/access, then lowest id).

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient.

  • default – When True, return the canonical (first-sorted) candidate’s long name instead of raising on ambiguity. Off by default.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved long name (or the canonical one when default=True and the lookup is ambiguous).

Return type:

str

Raises:

  • KeyError – If shortname is unknown, or no candidate survives the filters.

  • AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

shortname_to_param_id(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) → int#

Return the single param ID for shortname, given optional context.

Ambiguity is not resolved by guessing. When more than one candidate remains after applying context and the table/origin/ access filters, the behaviour depends on default:

  • default=False (the default) — AmbiguousParamError is raised; its .candidates attribute lists every remaining ParamIDCandidate (mars_request_context is currently None — see note below).

  • default=True — the canonical candidate is returned: the first in the sorted candidate order (lowest table / lowest id). For tp this is 228.

Note

Per-candidate MARS context computation is temporarily deferred, so every returned/raised ParamIDCandidate carries mars_request_context=None. Passing context= to narrow the lookup still works; only the advertised selecting context is unavailable for now.

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys resolved via the C++ expand engine (e.g. {"class": "ai"}). Partial context is usually sufficient — expand fills defaults for unspecified keys.

  • default – When True, return the canonical (first-sorted) candidate instead of raising on ambiguity. Off by default.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

The uniquely resolved paramid (or the canonical one when default=True and the lookup is ambiguous).

Return type:

int

Raises:

  • KeyError – If shortname is unknown, or no candidate survives the filters.

  • AmbiguousParamError – If more than one candidate remains after applying context/filters and default=False.

shortname_to_param_id_candidates(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) → list[ParamIDCandidate]#

Return all candidate paramids for shortname, each with its context.

The programmatic counterpart to AmbiguousParamError: it returns the candidate + context information as a normal value, so callers can inspect the options and then call shortname_to_param_id() with the narrowing context= (or table/origin/access) they want.

Two independent narrowing mechanisms are available and may be combined:

  • context — a dict of MARS keys resolved via the C++ expand engine (authoritative, cycle-correct). When the libmetkit is unavailable, the baked mars_request_context metadata is used as a fallback.

  • table / origin / access — direct hard filters on the candidate metadata, applied without constructing a MARS request.

Parameters:

  • shortname – ECMWF short name (e.g. "t", "tp").

  • context – Optional dict of MARS keys used to pre-narrow the candidate set (e.g. {"class": "ai"}). When omitted, all candidates surviving the hard filters are returned.

  • table – Optional hard filter — GRIB parameter table number.

  • origin – Optional hard filter — WMO originating centre id (membership).

  • access – Optional hard filter — access category string (membership).

Returns:

Every matching candidate, sorted by (table, origin, access, mars_request_context, param_id). Length 1 means the shortname (given any supplied context/filters) is unambiguous.

Return type:

list[ParamIDCandidate]

Raises:

KeyError – If shortname is unknown, or no candidate survives the filters.

class ParamIDCandidate#

One possible paramid for a shortname, plus the context that selects it.

param_id#

The candidate numeric parameter ID.

table#

GRIB parameter table the id encodes to (e.g. 128, 228).

origin#

WMO originating centre ids associated with this candidate — the full list (e.g. [0, 34, 98]), since several centres may share the id.

access#

Access categories (e.g. ["dissemination"]) — the full list.

mars_request_context#

The minimal set of MARS key/value pairs that, when passed as context= to ParamDB.shortname_to_param_id(), selects this candidate (e.g. {"class": "ai"}). Special values:

  • {} — this candidate is the default: an empty context={} resolves to it via the C++ expand layer.

  • None — no MARS context can select this candidate; use the hard filters instead (see hard_filter_selector).

hard_filter_selector#

The minimal table/origin/access kwargs that, when passed to ParamDB.shortname_to_param_id(), are proven to select exactly this candidate among all parameters sharing the short name. Special value:

  • None — no combination of the available hard filters uniquely identifies this candidate (e.g. two ids share the same table, origin and access). In that case there is no hard-filter selector to advertise, and the collision cannot be resolved by hard filters alone.

access: list[str]#
hard_filter_selector: dict | None = None#
mars_request_context: dict | None = None#
origin: list[int]#
param_id: int#
table: int#
class ParameterEntry(/, **data: Any)#

A single entry from the ECMWF parameter database.

This model accepts the canonical field names only (shortname / longname). Raw API alias spellings (e.g. shortName, name) are NOT accepted and must be normalised to canonical keys before validation.

The JSON schema emitted from this model (and bundled as parameter_entry_schema.json) mirrors this contract: it validates the canonical keys only.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

access_ids: list[str] = None#
classmethod coerce_access_ids(v: Any) → list[str]#
classmethod coerce_id_to_int(v: Any) → int#
classmethod coerce_origin_ids(v: Any) → list[int]#
classmethod coerce_table(v: Any) → int | None#
classmethod default_empty_units(v: Any) → str#
id: int = None#
longname: str = None#
mars_request_context: list[MarsRequestContext] = None#
model_config#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod normalise_longname_key(v: Any) → str#
classmethod normalise_shortname_key(v: Any) → str#
origin_ids: list[int] = None#
shortname: str = None#
table: int | None = None#
units: str = None#
class MarsRequestContext(/, **data: Any)#

A MARS-key context that (partly) selects a parameter id.

This mirrors a single matcher rule from share/metkit/params.yaml. Its schema is kept separate from ParameterEntry so the context contract can evolve independently and so user-supplied context schemas can validate against it. All fields are optional because params.yaml rules do not always constrain every key (e.g. levtype is occasionally absent).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

class_: str | None = None#
levtype: str | None = None#
model_config#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

stream: str | None = None#
type: str | None = None#
exception AmbiguousParamError(shortname: str, candidates: list[ParamIDCandidate])#

Raised when a shortname (given the supplied context) maps to >1 paramid.

shortname#

The shortname that could not be uniquely resolved.

candidates#

Every matching candidate, each carrying the context needed to select it, sorted by (table, origin, access, mars_request_context). The caller inspects these and re-calls with a narrowing context=.

Initialize self.  See help(type(self)) for accurate signature.
candidates#
shortname#

Regenerating bundled metadata#

The bundled files under share/metkit/ are generated by fetching from the ECMWF parameter database API. Run the generator script when the upstream database changes:

python -m pymetkit.paramdb.generate_metadata

This requires network access and the requests and pyyaml packages. It writes the following files relative to the repository root:

File

Description

share/metkit/parameter_metadata.json

Compact JSON — preferred at runtime (~10–50× faster than YAML)

share/metkit/parameter_metadata.yaml

Human-readable YAML — fallback if JSON is absent

share/metkit/unit_metadata.yaml

Unit definitions

share/metkit/parameter_entry_schema.json

JSON Schema for ParameterEntry validation

share/metkit/mars_context_schema.json

JSON Schema for MarsRequestContext validation

The generator also enriches each entry with a table field (decoded from the param ID encoding) and mars_request_context (inverted from share/metkit/params.yaml) — both derived locally without additional network access.

Commit the updated files to keep the bundled metadata in sync with the upstream database.

On this page
Show Source