API#

Overview #

z3FDB enables to create views into FDB where the view is a Zarr array.

Views are defined by one or more MARS requests. Each keyword in the MARS request with more than one value defines an ‘Axis’. ‘Axis’ from MARS requests need to be mapped to ‘Axis’ in the Zarr array. This mapping can be a 1-1 or many-1 mapping, allowing to create a time based axis in the Zarr array that is composed from the ‘date’ and ‘time’ keyword when dealing with climate data.

For example, the request "..., date=1970-01-1/to/2020-12-31, time=00/06/12/18, ..." spans two axis ‘date’ and ‘time’. If you want to work on a unified time axis, then you can use the following AxisDefinition to map accordingly:

Example:

AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE)

This defines an ‘Axis’ in the Zarr array that follows ‘date’ and ‘time’ from the MARS request, where the rightmost references ‘Axis’ (‘time’) is varying fastest.

You can combine multiple MARS request into one view. This is useful if you want to access surface and pressure level data in one view. In this case you need to select on which ‘Axis’ of the Zarr array the requests extend each other. The remaining axis have to have the same cardinality.

builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "sfc",
        "step": 0,
        "param": [165, 166],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.Grib(),
)
builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "pl",
        "step": 0,
        "param": [131, 132],
        "levelist": [50, 100],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param", "levelist"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.Grib(),
)
builder.extend_on_axis(1)
store = builder.build()

The created Zarr array will always have the actual data points available as the final ‘Axis’.

arr[0][0][0][0]
    ^  ^  ^  ^
    |  |  | Index in field  -> Implicit
    |  | Ensemble           -> Created from an AxisDefinition
    | Step                  -> Created from an AxisDefinition
   Date                     -> Created from an AxisDefinition

Exceptions #

z3fdb.Z3fdbError #

exception Z3fdbError#

Base Exception of all Z3fdb related errors.

Raised when an operation fails.

Initialize self. See help(type(self)) for accurate signature.

Extractor errors #

Raised by the extractor backends and re-exported from pychunked_data_view, so they can be caught by type:

GribExtractorError: A GRIB field could not be retrieved or decoded. For example, FDB returned no field for a sub-request, or a field’s size does not match the rest of the view.
GribJumpExtractorError: GribJump extraction failed. For example, a field location carried no usable file offset, or the request matched nothing.
MarsRequestFormattingError: A malformed MARS request string: a trailing comma, a missing comma between keys, or a misspelled key. Raised from build(); a subclass of RuntimeError.
InternalError: Something inside pychunked_data_view is inconsistent. You should not see this.

Note that other misconfiguration detected by the builder (unmapped axes, incompatible parts, an invalid chunk size, parts disagreeing about the grid) surfaces as a plain RuntimeError, since it originates as an eckit::UserError.

Build capability #

pychunked_data_view.has_gribjump_extractor: bool#: Whether this build compiled the GribJump extractor. GribJump can always be constructed, so this is the way to find out whether it can actually be used. See Optional: the GribJump Extractor.

Type aliases #

z3fdb.MarsSelection #

Classes #

z3fdb.SimpleStoreBuilder #

Creates a store whose root is the array. Equivalent to CustomStoreBuilder restricted to path=None, which is what it delegates to.

class SimpleStoreBuilder(fdb_config_file: pathlib.Path | None = None)#

Builder to create a Zarr store with FDB backing.

This builder will create a Zarr store with a Zarr Array at its root (“/”) containing the data from your MARS request(s).

It is exactly CustomStoreBuilder restricted to the root array, and delegates to it – so the two cannot drift apart.

Parameters:: fdb_config_file – Optional path to FDB config file. If not set normal FDB config file resolution is applied.

add_part(mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor: pychunked_data_view.ExtractorType.Grib | pychunked_data_view.ExtractorType.GribJump) → None#

Add a MARS request to the view.

Parameters:

mars_request (MarsSelection) –

A dict mapping MARS keys to their values. Single values may be given as str, int, or float; multi-valued keys may be given as a list. MARS range expressions (e.g. "2020-01-01/to/2020-01-04") must be passed as a plain string value.

For example:

{
    "type": "an",
    "class": "ea",
    "domain": "g",
    "expver": "0001",
    "stream": "oper",
    "date": "2020-01-01/to/2020-01-04",
    "levtype": "sfc",
    "step": 0,
    "param": [167, 131, 132],
    "time": "0/to/21/by/3",
}

axes (list of AxisDefinition) – List of AxisDefinitions that describe how axis in the MARS request are mapped to axis in the Zarr array.
extractor – Extractor configuration object. Use ExtractorType.Grib() for full-field GRIB extraction or ExtractorType.GribJump(...) for partial-field extraction.

build() → z3fdb._internal.zarr.FdbZarrStore#

Build the store from the registered parts.

Returns:: FdbZarrStore ready to pass to zarr.open_array().
Raises:: RuntimeError – If the view is misconfigured – no parts, a missing extension axis, incompatible part shapes, or parts that disagree about the grid. These originate as eckit::UserError in the C++ layer.

extend_on_axis(axis: int) → None#

Defines the extension axis when multiple parts are added.

Call add_part() first: this configures the array, so there has to be one.

Parameters:: axis (int) – Index of the axis that is extended when multiple parts have been added.
Raises:: ValueError – If no part has been added yet.

fill_missing_value(value: float) → None#

Set the fill value used for bitmap-masked grid points.

Call add_part() first: this configures the array, so there has to be one.

Parameters:: value (float) – Fill value written into array positions that carry a GRIB bitmap missing flag. Also used as the zarr array fill_value.
Raises:: ValueError – If no part has been added yet.

z3fdb.ChunkedDataView #

The read-only array returned by build() on the lower-level ChunkedDataViewBuilder. Zarr normally drives it for you.

Note

chunkShape() is deprecated in favour of chunk_shape(); it still works but emits a DeprecationWarning. Every other accessor on the class is already snake_case.

class ChunkedDataView(obj: chunked_data_view_bindings.ChunkedDataView)#

Python wrapper around the C++ ChunkedDataView.

Provides shape and chunk-count metadata, and per-chunk data access via at(). Instances are returned by ChunkedDataViewBuilder.build().

at(index: list[int] | tuple[int, ...]) → numpy.ndarray#

Return the values of the chunk at index.

Parameters:: index (list[int] | tuple[int, ...]) – Per-dimension chunk coordinates, including the implicit grid-point dimension.
Returns:: 1-D float32 array of chunk_shape() values, C-order.
Return type:: numpy.ndarray
Raises:: RuntimeError – If index is out of bounds or the FDB retrieval fails.

chunkShape() → tuple[int, ...]#

Deprecated alias of chunk_shape().

Kept so existing callers keep working; every other method on this class is snake_case.

chunk_shape() → tuple[int, ...]#

Return the per-dimension element count of one chunk.

Returns:: Number of elements along each dimension within a single chunk.
Return type:: tuple[int, …]

chunks() → tuple[int, ...]#

Return the per-dimension number of chunks.

Returns:: Number of chunks along each dimension.
Return type:: tuple[int, …]

fill_missing_value() → float#

Return the fill value used for bitmap-masked grid points.

Returns:: Value written into positions flagged as missing by the GRIB bitmap.
Return type:: float

shape() → tuple[int, ...]#

Return the total array shape in elements (not chunks).

Returns:: Total number of elements along each dimension.
Return type:: tuple[int, …]

z3fdb.CustomStoreBuilder #

Creates a store with an arbitrary group/array hierarchy: every method takes a zarr-style path naming the array it applies to, and path=None addresses a root array (mutually exclusive with any named path).

z3fdb.AxisDefinition #

See Dimension Mapping and Data Model for how axis definitions map MARS keywords to Zarr dimensions.

class AxisDefinition(keys: list[str], chunking: Chunking | Chunking, name: str | None = None)#

Maps one or more MARS keys to a single zarr array axis with a given chunking strategy.

Defines which MARS keys form an axis in the zarr array, and how it is chunked.

Parameters:

keys (list[str]) – MARS keys that form this axis.
chunking (Chunking | FixedSizeChunk) – How this axis shall be chunked.
name (str | None) – Zarr dimension name. Defaults to the keys joined by "_".

property chunking: Chunking | Chunking#

The chunking strategy for this axis.

Raises:: InternalError – If the underlying C++ chunking type is unrecognised.

property keys: list[str]#: The MARS keys that form this axis.

property name: str | None#: The zarr dimension name for this axis, or None to derive it from the keys.

Chunking #

z3fdb.Chunking #

class Chunking(*args, **kwds)#

Defines how an axis will be chunked.

WHOLE_AXIS#: The entire axis is a single chunk; accessing any value loads all values on that axis.

SINGLE_VALUE#: Each value along the axis is its own chunk.

FixedSizeChunk#: Groups every chunk_shape consecutive values into one chunk.

class FixedSizeChunk(chunk_shape: int)#

chunk_shape#

SINGLE_VALUE#

WHOLE_AXIS#

class pychunked_data_view.Chunking.FixedSizeChunk(chunk_shape)#

Specifies a custom chunk size along a single axis. This is a frozen dataclass nested inside Chunking.

chunk_shape: int#: Number of consecutive axis values grouped into each chunk. Must be a positive integer that divides the axis length exactly; otherwise build() raises an exception.

Example

# Chunk a 12-date axis into groups of 3 (gives 4 chunks)
AxisDefinition(["date"], Chunking.FixedSizeChunk(chunk_shape=3))

See Chunking for a full comparison of chunking modes and guidance on when to use each one.

Extractors #

ExtractorType is a namespace class, not an enum. Its nested classes carry per-extractor configuration. Pass an instance to add_part().

add_part stores a copy of the configuration, so one instance can be reused across as many parts and builders as you like, and the fdb_config a builder fills in for you is never written back into your object.

z3fdb.ExtractorType.Grib #

class pychunked_data_view.ExtractorType.Grib(*, fdb_config=None)#

Reads full GRIB fields from FDB and decodes them to float32 via eccodes. This is the default extractor for standard GRIB data.

Parameters:: fdb_config (pathlib.Path or None) – Path to an FDB configuration YAML file. None (default) uses the path passed to SimpleStoreBuilder.

Example

builder.add_part(mars_request, axes, ExtractorType.Grib())

# With an explicit FDB config
builder.add_part(mars_request, axes, ExtractorType.Grib(fdb_config=Path("/etc/fdb/config.yaml")))

z3fdb.ExtractorType.GribJump #

class pychunked_data_view.ExtractorType.GribJump(*, fdb_config=None, gribjump_config=None, field_chunking=None)#

Reads grid-point values from FDB using GribJump, a library that jumps directly to the values inside the GRIB message without performing a full decode.

Parameters:

fdb_config (pathlib.Path or None) – Path to an FDB configuration YAML file. None (default) uses the path passed to SimpleStoreBuilder.
gribjump_config (pathlib.Path or None) – Path to a GribJump configuration YAML file. None (default) reads the GRIBJUMP_CONFIG_FILE environment variable.
field_chunking (pychunked_data_view.Chunking.FixedSizeChunk or None) – How to sub-divide the implicit (grid-point) dimension into Zarr chunks. None (default) produces a single chunk covering the full field. Pass pychunked_data_view.Chunking.FixedSizeChunk to split the implicit axis into equal-sized pieces; the size must divide the grid exactly, as that dimension cannot be left ragged.

Example

# Full field: avoids eccodes decode
builder.add_part(mars_request, axes, ExtractorType.GribJump())

# Split the implicit grid-point axis into chunks of 1312
builder.add_part(mars_request, axes,
                 ExtractorType.GribJump(field_chunking=Chunking.FixedSizeChunk(1312)))