z3fdb ===== .. py:module:: z3fdb .. autoapi-nested-parse:: z3FDB enables to create views into FDB where the view is a Zarr array. Views are defined by one or more MARS requests. Each keyword in the MARS request with more than one value defines an 'Axis'. 'Axis' from MARS requests need to be mapped to 'Axis' in the Zarr array. This mapping can be a 1-1 or many-1 mapping, allowing to create a time based axis in the Zarr array that is composed from the 'date' and 'time' keyword when dealing with climate data. For example, the request ``"..., date=1970-01-1/to/2020-12-31, time=00/06/12/18, ..."`` spans two axis 'date' and 'time'. If you want to work on a unified time axis, then you can use the following :class:`AxisDefinition` to map accordingly: Example:: AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE) This defines an 'Axis' in the Zarr array that follows 'date' and 'time' from the MARS request, where the rightmost references 'Axis' ('time') is varying fastest. You can combine multiple MARS request into one view. This is useful if you want to access surface and pressure level data in one view. In this case you need to select on which 'Axis' of the Zarr array the requests extend each other. The remaining axis have to have the same cardinality. .. code-block:: python builder.add_part( { "type": "an", "class": "ea", "domain": "g", "expver": "0001", "stream": "oper", "date": ["2020-01-01", "2020-01-02"], "levtype": "sfc", "step": 0, "param": [165, 166], "time": "0/to/21/by/3", }, [ AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE), AxisDefinition(["param"], Chunking.SINGLE_VALUE) ], ExtractorType.Grib(), ) builder.add_part( { "type": "an", "class": "ea", "domain": "g", "expver": "0001", "stream": "oper", "date": ["2020-01-01", "2020-01-02"], "levtype": "pl", "step": 0, "param": [131, 132], "levelist": [50, 100], "time": "0/to/21/by/3", }, [ AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE), AxisDefinition(["param", "levelist"], Chunking.SINGLE_VALUE) ], ExtractorType.Grib(), ) builder.extend_on_axis(1) store = builder.build() The created Zarr array will always have the actual data points available as the final 'Axis'. .. code-block:: text arr[0][0][0][0] ^ ^ ^ ^ | | | Index in field -> Implicit | | Ensemble -> Created from an AxisDefinition | Step -> Created from an AxisDefinition Date -> Created from an AxisDefinition Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/z3fdb/custom_store_builder/index /autoapi/z3fdb/simple_store_builder/index /autoapi/z3fdb/z3fdb_error/index Exceptions ---------- .. autoapisummary:: z3fdb.Z3fdbError Classes ------- .. autoapisummary:: z3fdb.SimpleStoreBuilder z3fdb.CustomStoreBuilder Package Contents ---------------- .. py:class:: SimpleStoreBuilder(fdb_config_file: pathlib.Path | None = None) Builder to create a Zarr store with FDB backing. This builder will create a Zarr store with a Zarr Array at its root ("/") containing the data from your MARS request(s). It is exactly :class:`~z3fdb.CustomStoreBuilder` restricted to the root array, and delegates to it -- so the two cannot drift apart. :param fdb_config_file: Optional path to FDB config file. If not set normal FDB config file resolution is applied. .. py:attribute:: _builder .. py:method:: add_part(mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor: pychunked_data_view.ExtractorType.Grib | pychunked_data_view.ExtractorType.GribJump) -> None Add a MARS request to the view. :param mars_request: A dict mapping MARS keys to their values. Single values may be given as ``str``, ``int``, or ``float``; multi-valued keys may be given as a list. MARS range expressions (e.g. ``"2020-01-01/to/2020-01-04"``) must be passed as a plain string value. For example:: { "type": "an", "class": "ea", "domain": "g", "expver": "0001", "stream": "oper", "date": "2020-01-01/to/2020-01-04", "levtype": "sfc", "step": 0, "param": [167, 131, 132], "time": "0/to/21/by/3", } :type mars_request: MarsSelection :param axes: List of AxisDefinitions that describe how axis in the MARS request are mapped to axis in the Zarr array. :type axes: :obj:`list` of :obj:`AxisDefinition` :param extractor: Extractor configuration object. Use ``ExtractorType.Grib()`` for full-field GRIB extraction or ``ExtractorType.GribJump(...)`` for partial-field extraction. .. py:method:: fill_missing_value(value: float) -> None Set the fill value used for bitmap-masked grid points. Call :meth:`add_part` first: this configures the array, so there has to be one. :param value: Fill value written into array positions that carry a GRIB bitmap missing flag. Also used as the zarr array fill_value. :type value: float :raises ValueError: If no part has been added yet. .. py:method:: extend_on_axis(axis: int) -> None Defines the extension axis when multiple parts are added. Call :meth:`add_part` first: this configures the array, so there has to be one. :param axis: Index of the axis that is extended when multiple parts have been added. :type axis: int :raises ValueError: If no part has been added yet. .. py:method:: build() -> z3fdb._internal.zarr.FdbZarrStore Build the store from the registered parts. :returns: :class:`~z3fdb._internal.zarr.FdbZarrStore` ready to pass to ``zarr.open_array()``. :raises RuntimeError: If the view is misconfigured -- no parts, a missing extension axis, incompatible part shapes, or parts that disagree about the grid. These originate as ``eckit::UserError`` in the C++ layer. .. py:class:: CustomStoreBuilder(fdb_config_file: pathlib.Path | None = None) Builds a zarr store backed by FDB with an arbitrary group/array hierarchy. Use :meth:`add_part` to register one or more MARS request parts (each producing a virtual zarr array) at arbitrary nested paths, then call :meth:`build` to obtain a read-only :class:`FdbZarrStore` that zarr can open directly. :param fdb_config_file: Optional path to an FDB config file. ``None`` (default) lets FDB resolve its configuration from the environment. .. py:attribute:: _ROOT_KEY :value: '' .. py:attribute:: _config :value: None .. py:attribute:: _structure :type: dict[str, _VArray] .. py:attribute:: _root .. py:method:: _merge_vgroup(vgroup: _VGroup) -> None Insert *vgroup* into the virtual group tree rooted at self._root. .. py:method:: _parse_path(path: str) -> list[str] :staticmethod: Convert a zarr-style path string to a list of name segments. Leading and trailing slashes are stripped; multiple consecutive slashes are collapsed. An empty result (e.g. ``""`` or ``"/"``) raises :exc:`ValueError`. Examples:: "sfc/wind" -> ["sfc", "wind"] "/sfc/wind" -> ["sfc", "wind"] # leading slash accepted "t2m" -> ["t2m"] # top-level array "" -> ValueError "/" -> ValueError .. py:method:: _build_structure(path: list[str] | None) -> pychunked_data_view.ChunkedDataViewBuilder Return the ChunkedDataViewBuilder for *path*, creating it if needed. Pass ``None`` to obtain the builder for the root array. .. py:method:: add_part(path: str | None, mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor: pychunked_data_view.ExtractorType.Grib | pychunked_data_view.ExtractorType.GribJump) -> None Register a MARS request as a part of a virtual zarr array at *path*. Calling this method multiple times with the same *path* adds further parts to the same array (equivalent to :meth:`ChunkedDataViewBuilder.add_part` called repeatedly). :param path: Zarr-style path of the array in the hierarchy, e.g. ``"group_a/sub_group/my_array"`` or ``"t2m"`` for a top-level (no-group) array. A leading ``/`` is accepted and ignored. Pass ``None`` to place the array at the store root (accessible via ``zarr.open_array(store)``); this is mutually exclusive with any named path. :param mars_request: MARS request as a dict mapping keys to values. :param axes: Axis definitions describing how the request dimensions map to zarr array dimensions. :param extractor: Extractor configuration (``ExtractorType.Grib`` or ``ExtractorType.GribJump``). .. py:method:: _existing_array(path: list[str] | None) -> pychunked_data_view.ChunkedDataViewBuilder Return the builder for an array already registered at *path*. Unlike :meth:`_build_structure` this never creates one. :meth:`extend_on_axis` and :meth:`fill_missing_value` *configure* an existing array, so an unknown path is a mistake -- usually a typo -- rather than a request for a new empty array. Creating one silently would only surface much later, as "must add at least one part" from :meth:`build`. :param path: Path segments, or ``None`` for the root array. :returns: The builder registered at *path*. :rtype: ChunkedDataViewBuilder :raises ValueError: If no array is registered at *path*. .. py:method:: extend_on_axis(path: str | None, axis: int) -> None Declare the extension axis of the array at *path*. The array must already exist: call :meth:`add_part` for *path* first. :param path: Zarr-style path (same format as :meth:`add_part`). ``None`` refers to the root array. :param axis: Zero-based index of the axis to extend. :raises ValueError: If no array is registered at *path*. .. py:method:: fill_missing_value(path: str | None, value: float) -> None Set the fill value for the array at *path*. The array must already exist: call :meth:`add_part` for *path* first. :param path: Zarr-style path (same format as :meth:`add_part`). ``None`` refers to the root array. :param value: Value written into positions flagged as missing by the GRIB bitmap. Also becomes the zarr array's ``fill_value``. Defaults to NaN when not set. :raises ValueError: If no array is registered at *path*. .. py:method:: build() -> z3fdb._internal.zarr.FdbZarrStore Assemble all registered views into a read-only :class:`FdbZarrStore`. :returns: A zarr-compatible store that can be opened with ``zarr.open(store)`` (group hierarchy) or ``zarr.open_array(store)`` (when built from a single root array registered via ``path=None``). .. py:exception:: Z3fdbError Bases: :py:obj:`Exception` Base Exception of all Z3fdb related errors. Raised when an operation fails. Initialize self. See help(type(self)) for accurate signature.