API#

Overview#

z3FDB enables to create views into FDB where the view is a Zarr array.

Views are defined by one or more MARS requests. Each keyword in the MARS request with more than one value defines an ‘Axis’. ‘Axis’ from MARS requests need to be mapped to ‘Axis’ in the Zarr array. This mapping can be a 1-1 or many-1 mapping, allowing to create a time based axis in the Zarr array that is composed from the ‘date’ and ‘time’ keyword when dealing with climate data.

For example, the request "..., date=1970-01-1/to/2020-12-31, time=00/06/12/18, ..." spans two axis ‘date’ and ‘time’. If you want to work on a unified time axis, then you can use the following AxisDefinition to map accordingly:

Example:

AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE)

This defines an ‘Axis’ in the Zarr array that follows ‘date’ and ‘time’ from the MARS request, where the rightmost references ‘Axis’ (‘time’) is varying fastest.

You can combine multiple MARS request into one view. This is useful if you want to access surface and pressure level data in one view. In this case you need to select on which ‘Axis’ of the Zarr array the requests extend each other. The remaining axis have to have the same cardinality.

builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "sfc",
        "step": 0,
        "param": [165, 166],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.GRIB,
)
builder.add_part(
    {
        "type": "an",
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "date": ["2020-01-01", "2020-01-02"],
        "levtype": "pl",
        "step": 0,
        "param": [131, 132],
        "levelist": [50, 100],
        "time": "0/to/21/by/3",
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),
        AxisDefinition(["param", "levelist"], Chunking.SINGLE_VALUE)
    ],
    ExtractorType.GRIB,
)
builder.extend_on_axis(1)
store = builder.build()

The created Zarr array will always have the actual data points available as the final ‘Axis’.

arr[0][0][0][0]
    ^  ^  ^  ^
    |  |  | Index in field  -> Implicit
    |  | Ensemble           -> Created from an AxisDefinition
    | Step                  -> Created from an AxisDefinition
   Date                     -> Created from an AxisDefinition

Exceptions#

z3fdb.Z3fdbError#

exception Z3fdbError#

Base Exception of all Z3fdb related errors.

Raised when an operation fails.

Initialize self. See help(type(self)) for accurate signature.

Type aliases#

z3fdb.MarsSelection#

Classes#

z3fdb.SimpleStoreBuilder#

class SimpleStoreBuilder(fdb_config_file: pathlib.Path | None = None)#

Builder to create a Zarr store with FDB backing.

This builder will create a Zarr store with a Zarr Array at its root (“/”) containing the data from your MARS request(s).

Parameters:

fdb_config_file – Optional path to FDB config file. If not set normal FDB config file resolution is applied.

add_part(mars_request: pychunked_data_view.MarsSelection, axes: list[pychunked_data_view.AxisDefinition], extractor_type: pychunked_data_view.ExtractorType) → None#

Add a MARS request to the view.

Parameters:

  • mars_request (MarsSelection) –

    A dict mapping MARS keys to their values. Single values may be given as str, int, or float; multi-valued keys may be given as a list. MARS range expressions (e.g. "2020-01-01/to/2020-01-04") must be passed as a plain string value.

    For example:

    {
    "type": "an",
    "class": "ea",
    "domain": "g",
    "expver": "0001",
    "stream": "oper",
    "date": "2020-01-01/to/2020-01-04",
    "levtype": "sfc",
    "step": 0,
    "param": [167, 131, 132],
    "time": "0/to/21/by/3",
}

  • axes (list of AxisDefinition) – List of AxisDefinitions that describe how axis in the MARS request are mapped to axis in the Zarr array.

  • extractor_type – Defines how to extract data from FDB. Currently only ExtractorType.GRIB is supported.

build() → zarr.abc.store.Store#

Build the store from the inputs.

Raises:

Z3fdbError if store cannot be created. –

extend_on_axis(axis: int) → None#

Defines the extension axis when multiple parts are added.

Parameters:

axis (int) – Index of the axis that is extendet when multiple parts have been added.

fill_value(value: float) → None#

Set the fill value used for missing / bitmap-masked grid points.

Parameters:

value (float) – Fill value written into array positions that carry a GRIB bitmap missing flag. Also used as the zarr array fill_value.

z3fdb.AxisDefinition#

See Dimension Mapping and Data Model for how axis definitions map MARS keywords to Zarr dimensions.

class AxisDefinition(keys: list[str], chunking: Chunking | Chunking, dim_name: str | None = None)#

Defines which axis from a MARS Request form an axis in the Zarr array.

Also defines if the data is to be chunked.

Parameters:

  • keys (list of str) – mars keys that form this axis.

  • chunking (Chunking) – Define how this axis shall be chunked.

  • dim_name (str | None) – Optional dimension name for xarray compatibility. If not provided, the name is auto-derived as the keys joined by “_” (e.g. ["date", "time"] → "date_time").

property chunking#
property dim_name: str#
property keys: list[str]#

Enums#

z3fdb.Chunking#

class Chunking(*args, **kwds)#

Defines how a axis will be chunked

NONE#

Axis will not be chunked, accessing any value from this axis will load all values.

SINGLE_VALUE#

Axis will be chunked. One chunk per value.

class IndividualChunk#
chunkShape: int#
NONE#
SINGLE_VALUE#
class pychunked_data_view.Chunking.IndividualChunk(chunkShape)#

Specifies a custom chunk size along a single axis. This is a frozen dataclass nested inside Chunking.

chunkShape: int#

Number of consecutive axis values grouped into each chunk. Must be a positive integer that divides the axis length exactly; otherwise build() raises an exception.

Example

# Chunk a 12-date axis into groups of 3 (gives 4 chunks)
AxisDefinition(["date"], Chunking.IndividualChunk(chunkShape=3))

See dimension_mapping:Chunking for a full comparison of chunking modes and guidance on when to use each one.

z3fdb.ExtractorType#

class ExtractorType(*args, **kwds)#

Suported data extractors.

Defines what storage format the caller expects to be stored in FDB.

GRIB#

Extract data from GRIB