z3fdb#

z3FDB enables to create views into FDB where the view is a Zarr array.

Views are defined by one or more MARS requests. Each keyword in the MARS request with more than one value defines an ‘Axis’. ‘Axis’ from MARS requests need to be mapped to ‘Axis’ in the Zarr array. This mapping can be a 1-1 or many-1 mapping, allowing to create a time based axis in the Zarr array that is composed from the ‘date’ and ‘time’ keyword when dealing with climate data.

For example, the request "..., date=1970-01-1/to/2020-12-31, time=00/06/12/18, ..." spans two axis ‘date’ and ‘time’. If you want to work on a unified time axis, then you can use the following AxisDefinition to map accordingly:

Example:

AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE)

This defines an ‘Axis’ in the Zarr array that follows ‘date’ and ‘time’ from the MARS request, where the rightmost references ‘Axis’ (‘time’) is varying fastest.

You can combine multiple MARS request into one view. This is useful if you want to access surface and pressure level data in one view. In this case you need to select on which ‘Axis’ of the Zarr array the requests extend each other. The remaining axis have to have the same cardinality.

builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "sfc",
        "step": 0,
        "param": [165, 166],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.Grib(),
)
builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "pl",
        "step": 0,
        "param": [131, 132],
        "levelist": [50, 100],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param", "levelist"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.Grib(),
)
builder.extend_on_axis(1)
store = builder.build()

The created Zarr array will always have the actual data points available as the final ‘Axis’.

arr[0][0][0][0]
    ^  ^  ^  ^
    |  |  | Index in field  -> Implicit
    |  | Ensemble           -> Created from an AxisDefinition
    | Step                  -> Created from an AxisDefinition
   Date                     -> Created from an AxisDefinition

Submodules#

Exceptions#

Z3fdbError

Base Exception of all Z3fdb related errors.

Classes#

SimpleStoreBuilder

Builder to create a Zarr store with FDB backing.

CustomStoreBuilder

Builds a zarr store backed by FDB with an arbitrary group/array hierarchy.

Package Contents#

class SimpleStoreBuilder(fdb_config_file: pathlib.Path | None = None)#

Builder to create a Zarr store with FDB backing.

This builder will create a Zarr store with a Zarr Array at its root (“/”) containing the data from your MARS request(s).

It is exactly CustomStoreBuilder restricted to the root array, and delegates to it – so the two cannot drift apart.

Parameters:

fdb_config_file – Optional path to FDB config file. If not set normal FDB config file resolution is applied.

_builder#
add_part(mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor: pychunked_data_view.ExtractorType.Grib | pychunked_data_view.ExtractorType.GribJump) → None#

Add a MARS request to the view.

Parameters:

  • mars_request (MarsSelection) –

    A dict mapping MARS keys to their values. Single values may be given as str, int, or float; multi-valued keys may be given as a list. MARS range expressions (e.g. "2020-01-01/to/2020-01-04") must be passed as a plain string value.

    For example:

    {
    "type": "an",
    "class": "ea",
    "domain": "g",
    "expver": "0001",
    "stream": "oper",
    "date": "2020-01-01/to/2020-01-04",
    "levtype": "sfc",
    "step": 0,
    "param": [167, 131, 132],
    "time": "0/to/21/by/3",
}

  • axes (list of AxisDefinition) – List of AxisDefinitions that describe how axis in the MARS request are mapped to axis in the Zarr array.

  • extractor – Extractor configuration object. Use ExtractorType.Grib() for full-field GRIB extraction or ExtractorType.GribJump(...) for partial-field extraction.

fill_missing_value(value: float) → None#

Set the fill value used for bitmap-masked grid points.

Call add_part() first: this configures the array, so there has to be one.

Parameters:

value (float) – Fill value written into array positions that carry a GRIB bitmap missing flag. Also used as the zarr array fill_value.

Raises:

ValueError – If no part has been added yet.

extend_on_axis(axis: int) → None#

Defines the extension axis when multiple parts are added.

Call add_part() first: this configures the array, so there has to be one.

Parameters:

axis (int) – Index of the axis that is extended when multiple parts have been added.

Raises:

ValueError – If no part has been added yet.

build() → z3fdb._internal.zarr.FdbZarrStore#

Build the store from the registered parts.

Returns:

FdbZarrStore ready to pass to zarr.open_array().

Raises:

RuntimeError – If the view is misconfigured – no parts, a missing extension axis, incompatible part shapes, or parts that disagree about the grid. These originate as eckit::UserError in the C++ layer.

class CustomStoreBuilder(fdb_config_file: pathlib.Path | None = None)#

Builds a zarr store backed by FDB with an arbitrary group/array hierarchy.

Use add_part() to register one or more MARS request parts (each producing a virtual zarr array) at arbitrary nested paths, then call build() to obtain a read-only FdbZarrStore that zarr can open directly.

Parameters:

fdb_config_file – Optional path to an FDB config file. None (default) lets FDB resolve its configuration from the environment.

_ROOT_KEY = ''#
_config = None#
_structure: dict[str, _VArray]#
_root#
_merge_vgroup(vgroup: _VGroup) → None#

Insert vgroup into the virtual group tree rooted at self._root.

static _parse_path(path: str) → list[str]#

Convert a zarr-style path string to a list of name segments.

Leading and trailing slashes are stripped; multiple consecutive slashes are collapsed. An empty result (e.g. "" or "/") raises ValueError.

Examples:

"sfc/wind"   -> ["sfc", "wind"]
"/sfc/wind"  -> ["sfc", "wind"]   # leading slash accepted
"t2m"        -> ["t2m"]           # top-level array
""           -> ValueError
"/"          -> ValueError
_build_structure(path: list[str] | None) → pychunked_data_view.ChunkedDataViewBuilder#

Return the ChunkedDataViewBuilder for path, creating it if needed.

Pass None to obtain the builder for the root array.

add_part(path: str | None, mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor: pychunked_data_view.ExtractorType.Grib | pychunked_data_view.ExtractorType.GribJump) → None#

Register a MARS request as a part of a virtual zarr array at path.

Calling this method multiple times with the same path adds further parts to the same array (equivalent to ChunkedDataViewBuilder.add_part() called repeatedly).

Parameters:

  • path – Zarr-style path of the array in the hierarchy, e.g. "group_a/sub_group/my_array" or "t2m" for a top-level (no-group) array. A leading / is accepted and ignored. Pass None to place the array at the store root (accessible via zarr.open_array(store)); this is mutually exclusive with any named path.

  • mars_request – MARS request as a dict mapping keys to values.

  • axes – Axis definitions describing how the request dimensions map to zarr array dimensions.

  • extractor – Extractor configuration (ExtractorType.Grib or ExtractorType.GribJump).

_existing_array(path: list[str] | None) → pychunked_data_view.ChunkedDataViewBuilder#

Return the builder for an array already registered at path.

Unlike _build_structure() this never creates one. extend_on_axis() and fill_missing_value() configure an existing array, so an unknown path is a mistake – usually a typo – rather than a request for a new empty array. Creating one silently would only surface much later, as “must add at least one part” from build().

Parameters:

path – Path segments, or None for the root array.

Returns:

The builder registered at path.

Return type:

ChunkedDataViewBuilder

Raises:

ValueError – If no array is registered at path.

extend_on_axis(path: str | None, axis: int) → None#

Declare the extension axis of the array at path.

The array must already exist: call add_part() for path first.

Parameters:

  • path – Zarr-style path (same format as add_part()). None refers to the root array.

  • axis – Zero-based index of the axis to extend.

Raises:

ValueError – If no array is registered at path.

fill_missing_value(path: str | None, value: float) → None#

Set the fill value for the array at path.

The array must already exist: call add_part() for path first.

Parameters:

  • path – Zarr-style path (same format as add_part()). None refers to the root array.

  • value – Value written into positions flagged as missing by the GRIB bitmap. Also becomes the zarr array’s fill_value. Defaults to NaN when not set.

Raises:

ValueError – If no array is registered at path.

build() → z3fdb._internal.zarr.FdbZarrStore#

Assemble all registered views into a read-only FdbZarrStore.

Returns:

A zarr-compatible store that can be opened with zarr.open(store) (group hierarchy) or zarr.open_array(store) (when built from a single root array registered via path=None).

exception Z3fdbError#

Bases: Exception

Base Exception of all Z3fdb related errors.

Raised when an operation fails.

Initialize self. See help(type(self)) for accurate signature.