pymetkit.paramdb ================ .. py:module:: pymetkit.paramdb Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/pymetkit/paramdb/_db/index /autoapi/pymetkit/paramdb/generate_metadata/index /autoapi/pymetkit/paramdb/models/index Attributes ---------- .. autoapisummary:: pymetkit.paramdb._HAVE_EXPAND Exceptions ---------- .. autoapisummary:: pymetkit.paramdb.AmbiguousParamError Classes ------- .. autoapisummary:: pymetkit.paramdb.ParamDB pymetkit.paramdb.ParamIDCandidate pymetkit.paramdb.ParameterEntry pymetkit.paramdb.MarsRequestContext Package Contents ---------------- .. py:class:: ParamDB(mode: str = 'offline', cache_ttl: timedelta | None = None, cache_path: Path | str | None = None, yaml_path: Path | str | None = None) Parameter database providing metadata lookup for ECMWF parameters. Supports both online mode (fetching from the ECMWF parameter database API) and offline mode (loading from a bundled YAML file). When using ``mode="online"`` a local JSON cache is maintained so that repeated instantiations within the TTL window do not make a new HTTP request. The cache is stored under the OS user-cache directory (e.g. ``~/.cache/pymetkit/`` on Linux, ``~/Library/Caches/pymetkit/`` on macOS) using the fixed filename defined by ``_CACHE_FILENAME``. Shortname collision resolution -------------------------------- Some short names (e.g. ``t``, ``tp``, ``u``) are reused across different GRIB parameter tables and originating centres. Collisions are not silently guessed by :meth:`shortname_to_param_id`: * :meth:`shortname_to_param_id` raises :class:`AmbiguousParamError` when more than one candidate remains after applying any ``context`` and the ``table``/``origin``/``access`` hard filters. The error's ``.candidates`` attribute lists every remaining :class:`ParamIDCandidate` so the caller can narrow the lookup. Pass ``default=True`` to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id); for ``tp`` this is ``228``. Note this is not the canonical paramID as without mars context this cannot be determined. * :meth:`shortname_to_longname` behaves the same way: it raises :class:`AmbiguousParamError` when more than one candidate remains after applying any ``context`` and the ``table``/``origin``/``access`` hard filters. Pass ``default=True`` to instead return a single candidate — the first in sorted order (lowest table, then origin/access, then lowest id). To resolve a collision explicitly, narrow the lookup with a MARS ``context=`` (resolved via the C++ ``expand`` engine) or the ``table=``/``origin=``/``access=`` hard filters, both accepted by :meth:`shortname_to_param_id` and :meth:`shortname_to_longname`. Initialise the parameter database. The underlying data is loaded **lazily** — no file I/O or network request is made until the first lookup method is called. This makes instantiation cheap and safe to do at import time or inside hot paths. :param mode: Either ``"online"`` (fetch from the ECMWF API) or ``"offline"`` (load from a YAML file). :type mode: str :param cache_ttl: How long a previously fetched online result may be reused before a fresh HTTP request is made. Defaults to 1 hour. Only relevant when ``mode="online"``. Pass ``timedelta(0)`` to disable caching entirely (always fetch). :type cache_ttl: datetime.timedelta, optional :param cache_path: Directory in which to store the cache file. Defaults to the OS-appropriate user cache directory (requires ``platformdirs``). Only relevant when ``mode="online"``. :type cache_path: Path or str, optional :param yaml_path: Path to a custom YAML file to load instead of the bundled ``parameter_metadata.yaml``. The file must be a YAML list where each entry contains at minimum an ``id`` (integer), a short name (``shortname``), and a long name (``longname``). Only the canonical key spellings are accepted. Only valid with ``mode="offline"``; raises ``ValueError`` if combined with ``mode="online"``. :type yaml_path: Path or str, optional .. py:attribute:: _API_URL :value: 'https://codes.ecmwf.int/parameter-database/api/v1/param/' .. py:attribute:: _CACHE_FILENAME :value: 'paramdb_online_cache.json' .. py:attribute:: _DEFAULT_CACHE_TTL .. py:attribute:: _REQUEST_TIMEOUT :value: 30 .. py:attribute:: _context_notice_emitted :value: False .. py:attribute:: _mode :value: 'offline' .. py:attribute:: _effective_cache_ttl .. py:attribute:: _cache_path :value: None .. py:attribute:: _yaml_path :value: None .. py:attribute:: _by_id :type: dict[int, dict] .. py:attribute:: _by_shortname :type: dict[str, dict] .. py:attribute:: _by_shortname_all :type: dict[str, list[dict]] .. py:attribute:: _by_longname :type: dict[str, dict] .. py:attribute:: _loaded :type: bool :value: False .. py:attribute:: _ctx_cache :type: dict .. py:method:: _table_from_id(param_id: int) -> int :staticmethod: Decode the GRIB parameter table number from an encoded param ID. The encoding scheme mirrors the C++ ``Param::paramId()`` logic: * IDs 1–999 → table 128 (classic ECMWF, table prefix suppressed) * IDs 1 000–999 999 → ``table * 1000 + param`` (e.g. 228228 → table 228) * IDs ≥ 1 000 000 → ``center * 1_000_000 + table * 1000 + param`` (e.g. 7001292 → center 7, table 1) .. py:method:: _center_from_id(param_id: int) -> int | None :staticmethod: Decode the originating WMO center from an encoded param ID. Returns ``None`` for IDs below 1 000 000 (ECMWF-local parameters). For IDs ≥ 1 000 000 the center is ``param_id // 1_000_000``. .. py:method:: _context_resolved_ids(shortname: str, context: dict) -> set[int] Resolve *shortname* + *context* to paramid(s) via the C++ engine. Builds ``MarsRequest(param=shortname, **context)``, expands it (which runs ``TypeParam::pass2`` and fills MARS defaults for unspecified keys), and reads back the resolved ``param`` value(s). Results are memoised on ``self._ctx_cache`` since each ``expand`` is a comparatively expensive (~ms) C++ round-trip and the minimal-context search repeats queries. The C++ library reads its language files (``params.yaml`` etc.) from ``~metkit/share/metkit`` — set ``METKIT_HOME`` to this repository root to resolve against the in-repo ``share/metkit`` data. :returns: The set of resolved numeric ids (empty if the selection is invalid or resolves to no ``param`` value). :rtype: set[int] .. py:method:: _warn_context_unavailable() -> None :staticmethod: Notify that per-candidate MARS context computation is deferred. The ``mars_request_context`` field of returned/raised candidates is currently always ``None``. This is emitted once per process on the ambiguous path so callers know the context is not yet advertised (but that ``context=`` narrowing still works, and full context will come in a future release). .. py:method:: _make_candidate(entry: dict, siblings: list[dict], shortname: str, minimal: bool = True) -> ParamIDCandidate Build a :class:`ParamIDCandidate` from a metadata *entry*. The candidate's ``param_id``, ``table``, ``origin`` and ``access`` (the hard-filter metadata) are always populated. ``mars_request_context`` is currently left as ``None``: per-candidate MARS context computation is deferred (see the "NOT YET IMPLEMENTED" section at the end of this class, :meth:`_minimal_distinguishing_context`, dormant), so we no longer advertise which context selects each candidate. Callers can still pass ``context=`` to :meth:`shortname_to_param_id` / :meth:`shortname_to_param_id_candidates` to narrow the lookup. The *minimal* parameter is retained for forward compatibility (it will re-enable the expand-oracle context computation once that ships) but is currently ignored. .. py:method:: _unique_hard_filter_selector(entry: dict, siblings: list[dict]) -> dict | None Return the minimal hard-filter selector proven to select *entry* alone. Builds ``table``/``origin``/``access`` selectors of increasing specificity and returns the first one that matches **exactly** this entry among *siblings* (all parameters sharing the short name). Returns ``None`` when no combination of the available hard filters uniquely identifies the entry — e.g. two ids sharing the same table, origin and access (such as short name ``~`` with ids 24 and 25). This ensures the API only ever advertises a selector that genuinely disambiguates, and otherwise honestly represents that no hard-filter selector exists. .. py:method:: _candidate_sort_key(cand: ParamIDCandidate) -> tuple :staticmethod: .. py:method:: _normalise(raw: dict) -> dict :staticmethod: Return a normalised and validated parameter dict with canonical key names. Uses :class:`~pymetkit.models.ParameterEntry` for field coercion and validation. Extra keys present in *raw* (e.g. ``"url"``, ``"units_id"``) are preserved in the returned dict so no information is discarded. .. py:method:: _index(entry: dict) -> None Insert a normalised entry into the internal lookup dicts. .. py:method:: _ensure_loaded() -> None Load parameter data on first access (lazy initialisation). Subsequent calls are no-ops — once ``_loaded`` is ``True`` the method returns immediately without any I/O. .. py:method:: _load_online(cache_ttl: datetime.timedelta, cache_path: Path | str | None) -> None .. py:method:: _load_offline(yaml_path: Path | str | None = None) -> None .. py:method:: _read_bundled_text(filename: str) -> str | None :staticmethod: Return the text of a bundled data file, or ``None`` if not found. The resource is read **while** the ``importlib.resources`` context is active, so zip-safe (materialised-to-temp) resources are read before the temporary file is removed. Falls back to the dev-tree / editable-install filesystem locations. Searches, in order: 1. Via ``importlib.resources`` from the installed package (reliable in both regular installs and zip-safe wheels). 2. Next to this module file (editable / development install layout). 3. ``/share/metkit/`` (development tree layout after the data files were moved out of the Python package directory). .. py:method:: _load_bundled_json() -> list | None :classmethod: Load and parse bundled ``parameter_metadata.json``. Returns ``None`` if the JSON file is not found — caller should fall back to the YAML file. .. py:method:: _load_bundled_yaml() -> list :classmethod: Load and parse bundled ``parameter_metadata.yaml``. Raises ``FileNotFoundError`` if the YAML file cannot be located. .. py:method:: _resolve_cache_dir(cache_path: Path | str | None) -> Path | None Return the directory to use for the cache file, or None if unavailable. .. py:method:: _cache_file(cache_path: Path | str | None) -> Path | None Return the full path to the cache file, or None if no cache dir is available. .. py:method:: _read_cache(cache_path: Path | str | None, cache_ttl: datetime.timedelta) -> list | None Return the cached parameter list if it exists and is still fresh, otherwise return None. .. py:method:: _write_cache(params: list, cache_path: Path | str | None) -> None Persist *params* to the cache file (best-effort; errors are silenced). .. py:method:: _resolve(identifier: int | str) -> dict Resolve *identifier* (param_id, shortname, or longname) to a metadata dict. .. py:method:: shortname_to_longname(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) -> str Return the long name for *shortname*, given optional context. Mirrors :meth:`shortname_to_param_id`: ambiguity is not resolved by guessing. When more than one candidate remains after applying ``context`` and the ``table``/``origin``/``access`` filters, the behaviour depends on ``default``: * ``default=False`` (the default) — :class:`AmbiguousParamError` is raised; its ``.candidates`` attribute lists every remaining :class:`ParamIDCandidate` so the caller can narrow the lookup. * ``default=True`` — the long name of the canonical candidate is returned: the first in sorted order (lowest table, then origin/access, then lowest id). :param shortname: ECMWF short name (e.g. ``"t"``, ``"tp"``). :param context: Optional dict of MARS keys resolved via the C++ ``expand`` engine (e.g. ``{"class": "ai"}``). Partial context is usually sufficient. :param default: When ``True``, return the canonical (first-sorted) candidate's long name instead of raising on ambiguity. Off by default. :param table: Optional hard filter — GRIB parameter table number. :param origin: Optional hard filter — WMO originating centre id (membership). :param access: Optional hard filter — access category string (membership). :returns: The uniquely resolved long name (or the canonical one when ``default=True`` and the lookup is ambiguous). :rtype: str :raises KeyError: If *shortname* is unknown, or no candidate survives the filters. :raises AmbiguousParamError: If more than one candidate remains after applying context/filters and ``default=False``. .. py:method:: longname_to_shortname(longname: str) -> str .. py:method:: shortname_to_param_id_candidates(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) -> list[ParamIDCandidate] Return all candidate paramids for *shortname*, each with its context. The programmatic counterpart to :class:`AmbiguousParamError`: it returns the candidate + context information as a normal value, so callers can inspect the options and then call :meth:`shortname_to_param_id` with the narrowing ``context=`` (or ``table``/``origin``/``access``) they want. Two independent narrowing mechanisms are available and may be combined: * ``context`` — a dict of MARS keys resolved via the C++ ``expand`` engine (authoritative, cycle-correct). When the libmetkit is unavailable, the baked ``mars_request_context`` metadata is used as a fallback. * ``table`` / ``origin`` / ``access`` — direct hard filters on the candidate metadata, applied without constructing a MARS request. :param shortname: ECMWF short name (e.g. ``"t"``, ``"tp"``). :param context: Optional dict of MARS keys used to pre-narrow the candidate set (e.g. ``{"class": "ai"}``). When omitted, all candidates surviving the hard filters are returned. :param table: Optional hard filter — GRIB parameter table number. :param origin: Optional hard filter — WMO originating centre id (membership). :param access: Optional hard filter — access category string (membership). :returns: Every matching candidate, sorted by ``(table, origin, access, mars_request_context, param_id)``. Length 1 means the shortname (given any supplied context/filters) is unambiguous. :rtype: list[ParamIDCandidate] :raises KeyError: If *shortname* is unknown, or no candidate survives the filters. .. py:method:: _filter_shortname_entries(shortname: str, context: dict | None = None, *, table: int | None = None, origin: int | None = None, access: str | None = None) -> tuple[list[dict], list[dict]] Return ``(surviving_entries, all_siblings)`` for *shortname*. Applies the ``table``/``origin``/``access`` hard filters and the MARS ``context`` filter (via the C++ ``expand`` engine). Does NOT compute minimal contexts — that expensive step is deferred to :meth:`_make_candidate`. Raises ``KeyError`` if the shortname is unknown or no entry survives the filters. .. py:method:: shortname_to_param_id(shortname: str, context: dict | None = None, *, default: bool = False, table: int | None = None, origin: int | None = None, access: str | None = None) -> int Return the single param ID for *shortname*, given optional context. Ambiguity is not resolved by guessing. When more than one candidate remains after applying ``context`` and the ``table``/``origin``/ ``access`` filters, the behaviour depends on ``default``: * ``default=False`` (the default) — :class:`AmbiguousParamError` is raised; its ``.candidates`` attribute lists every remaining :class:`ParamIDCandidate` (``mars_request_context`` is currently ``None`` — see note below). * ``default=True`` — the canonical candidate is returned: the first in the sorted candidate order (lowest table / lowest id). For ``tp`` this is ``228``. .. note:: Per-candidate MARS context computation is temporarily deferred, so every returned/raised :class:`ParamIDCandidate` carries ``mars_request_context=None``. Passing ``context=`` to narrow the lookup still works; only the *advertised* selecting context is unavailable for now. :param shortname: ECMWF short name (e.g. ``"t"``, ``"tp"``). :param context: Optional dict of MARS keys resolved via the C++ ``expand`` engine (e.g. ``{"class": "ai"}``). Partial context is usually sufficient — ``expand`` fills defaults for unspecified keys. :param default: When ``True``, return the canonical (first-sorted) candidate instead of raising on ambiguity. Off by default. :param table: Optional hard filter — GRIB parameter table number. :param origin: Optional hard filter — WMO originating centre id (membership). :param access: Optional hard filter — access category string (membership). :returns: The uniquely resolved paramid (or the canonical one when ``default=True`` and the lookup is ambiguous). :rtype: int :raises KeyError: If *shortname* is unknown, or no candidate survives the filters. :raises AmbiguousParamError: If more than one candidate remains after applying context/filters and ``default=False``. .. py:method:: param_id_to_shortname(param_id: int) -> str .. py:method:: _param_context_from_cpp(param_id: int) -> list[dict] | None :staticmethod: Return MARS contexts for *param_id* from the C++ layer, if available. Placeholder for the future ``metkit_param_context`` C API. That hook does not exist yet, so this always returns ``None`` to signal "not available", causing :meth:`param_id_to_context` to fall back to the precomputed YAML data. Once implemented, add it here and the public method will transparently prefer it. .. py:method:: param_id_to_context(param_id: int) -> list[dict] Return the MARS-key contexts in which *param_id* is valid. Each context is a dict of MARS keys (e.g. ``{"class": "ai", "stream": "enfo", "type": "cf", "levtype": "sfc"}``) drawn from the authoritative ``params.yaml`` map. These are the raw contexts used to disambiguate shortname collisions. Resolution order: 1. The C++ layer (``metkit_param_context``) when available — see :meth:`_param_context_from_cpp`. Not yet implemented. 2. Fallback: the precomputed ``mars_request_context`` field baked into the bundled parameter metadata. :param param_id: Numeric parameter id. :returns: Zero or more MARS context dicts. Empty if the id has no recorded context (e.g. not referenced in ``params.yaml``). :rtype: list[dict] :raises KeyError: If *param_id* is not in the database. .. py:method:: longname_to_param_id(longname: str) -> int .. py:method:: param_id_to_longname(param_id: int) -> str .. py:method:: get_metadata(identifier: int | str) -> dict Return the full metadata dictionary for a parameter. :param identifier: A param ID (int), shortname, or longname. :type identifier: int or str .. py:method:: get_units(identifier: int | str) -> str Return the units string for a parameter. :param identifier: A param ID (int), shortname, or longname. :type identifier: int or str :returns: The units string, or ``"unknown"`` if not available. :rtype: str .. py:method:: get_all_by_shortname(shortname: str) -> list[dict] Return *all* parameter entries that share *shortname*. Most short names map to exactly one param ID, but ~163 short names are reused across different GRIB parameter tables or originating centres. This method exposes every candidate so callers can inspect the collisions and choose the appropriate one. :param shortname: ECMWF short name to look up. :returns: List of metadata dicts, sorted by ascending param ID. Each dict contains at minimum ``id``, ``shortname``, and ``longname``. :rtype: list[dict] :raises KeyError: If *shortname* is not found in the database at all. .. rubric:: Examples >>> db = ParamDB() >>> entries = db.get_all_by_shortname("t") >>> [(e["id"], e["longname"]) for e in entries] [(130, 'Temperature'), (500014, 'Temperature')] .. py:method:: shortname_has_collisions(shortname: str) -> bool Return ``True`` if *shortname* maps to more than one param ID. :param shortname: ECMWF short name to check. :raises KeyError: If *shortname* is not found in the database at all. .. py:method:: _minimal_distinguishing_context(entry: dict, siblings: list[dict], shortname: str) -> dict | None Return the smallest MARS key-subset that selects *entry* over siblings. .. warning:: DORMANT — not currently called from any public API. See the "NOT YET IMPLEMENTED" banner above this method. Searches the key-subsets of *entry*'s baked ``mars_request_context`` dicts for the smallest subset that uniquely selects this id. Uses ``expand`` as the authoritative oracle: a subset qualifies only if ``MarsRequest(param=shortname, **subset).expand()`` resolves to exactly this id. This guarantees the advertised context actually round-trips (bare ``{"class": "ai"}`` is rejected for ``tp`` because ``expand``'s inherited defaults resolve it to 228, not 228228). :returns: * ``{}`` — *entry* is the default: an empty ``context={}`` already resolves uniquely to it. * a non-empty dict — the minimal distinguishing MARS context. * ``None`` — no MARS context can uniquely select *entry* (residual ambiguity, or no baked context). :rtype: dict | None .. py:class:: ParamIDCandidate One possible paramid for a shortname, plus the context that selects it. .. attribute:: param_id The candidate numeric parameter ID. .. attribute:: table GRIB parameter table the id encodes to (e.g. ``128``, ``228``). .. attribute:: origin WMO originating centre ids associated with this candidate — the full list (e.g. ``[0, 34, 98]``), since several centres may share the id. .. attribute:: access Access categories (e.g. ``["dissemination"]``) — the full list. .. attribute:: mars_request_context The minimal set of MARS key/value pairs that, when passed as ``context=`` to :meth:`ParamDB.shortname_to_param_id`, selects this candidate (e.g. ``{"class": "ai"}``). Special values: * ``{}`` — this candidate is the **default**: an empty ``context={}`` resolves to it via the C++ ``expand`` layer. * ``None`` — no MARS context can select this candidate; use the hard filters instead (see :attr:`hard_filter_selector`). .. attribute:: hard_filter_selector The minimal ``table``/``origin``/``access`` kwargs that, when passed to :meth:`ParamDB.shortname_to_param_id`, are **proven to select exactly this candidate** among all parameters sharing the short name. Special value: * ``None`` — no combination of the available hard filters uniquely identifies this candidate (e.g. two ids share the same table, origin and access). In that case there is no hard-filter selector to advertise, and the collision cannot be resolved by hard filters alone. .. py:attribute:: param_id :type: int .. py:attribute:: table :type: int .. py:attribute:: origin :type: list[int] .. py:attribute:: access :type: list[str] .. py:attribute:: mars_request_context :type: dict | None :value: None .. py:attribute:: hard_filter_selector :type: dict | None :value: None .. py:exception:: AmbiguousParamError(shortname: str, candidates: list[ParamIDCandidate]) Bases: :py:obj:`KeyError` Raised when a shortname (given the supplied context) maps to >1 paramid. .. attribute:: shortname The shortname that could not be uniquely resolved. .. attribute:: candidates Every matching candidate, each carrying the context needed to select it, sorted by ``(table, origin, access, mars_request_context)``. The caller inspects these and re-calls with a narrowing ``context=``. .. attribute:: Initialize self. See help(type(self)) for accurate signature. .. py:attribute:: shortname .. py:attribute:: candidates .. py:data:: _HAVE_EXPAND :value: True .. py:class:: ParameterEntry(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` A single entry from the ECMWF parameter database. This model accepts the **canonical** field names only (``shortname`` / ``longname``). Raw API alias spellings (e.g. ``shortName``, ``name``) are NOT accepted and must be normalised to canonical keys before validation. The JSON schema emitted from this model (and bundled as ``parameter_entry_schema.json``) mirrors this contract: it validates the canonical keys only. Create a new model by parsing and validating input data from keyword arguments. Raises [`ValidationError`][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model. `self` is explicitly positional-only to allow `self` as a field name. .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: id :type: int :value: None .. py:attribute:: shortname :type: str :value: None .. py:attribute:: longname :type: str :value: None .. py:attribute:: units :type: str :value: None .. py:attribute:: table :type: int | None :value: None .. py:attribute:: mars_request_context :type: list[MarsRequestContext] :value: None .. py:attribute:: origin_ids :type: list[int] :value: None .. py:attribute:: access_ids :type: list[str] :value: None .. py:method:: coerce_id_to_int(v: Any) -> int :classmethod: .. py:method:: normalise_shortname_key(v: Any) -> str :classmethod: .. py:method:: normalise_longname_key(v: Any) -> str :classmethod: .. py:method:: default_empty_units(v: Any) -> str :classmethod: .. py:method:: coerce_table(v: Any) -> int | None :classmethod: .. py:method:: coerce_origin_ids(v: Any) -> list[int] :classmethod: .. py:method:: coerce_access_ids(v: Any) -> list[str] :classmethod: .. py:class:: MarsRequestContext(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` A MARS-key context that (partly) selects a parameter id. This mirrors a single matcher rule from ``share/metkit/params.yaml``. Its schema is kept **separate** from :class:`ParameterEntry` so the context contract can evolve independently and so user-supplied context schemas can validate against it. All fields are optional because ``params.yaml`` rules do not always constrain every key (e.g. ``levtype`` is occasionally absent). Create a new model by parsing and validating input data from keyword arguments. Raises [`ValidationError`][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model. `self` is explicitly positional-only to allow `self` as a field name. .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: class_ :type: str | None :value: None .. py:attribute:: stream :type: str | None :value: None .. py:attribute:: type :type: str | None :value: None .. py:attribute:: levtype :type: str | None :value: None