Z3FDB

Version:

5.22.0

Introduction

Z3FDB enables FDB to function as a Zarr store without copying your GRIB data. You describe which data you want through MARS requests, map MARS keywords to Zarr dimensions, and Z3FDB handles the rest: chunk requests are translated into FDB retrieves on the fly and the returned GRIB data is decoded to float32 in memory.

Supported data

FDB can store arbitrary data, not just GRIB. At this point Z3FDB only supports data extraction from GRIB. It is the responsibility of the caller to ensure that only GRIB data is accessed through Z3FDB.

When to use Z3FDB

Z3FDB is a good fit when a 2–5× slowdown compared to on-disk Zarr is acceptable. That trade-off makes sense in two situations:

Prototyping with an existing FDB

You have GRIB data in FDB and want a Zarr interface without first writing everything to disk. Redefining the store is a matter of changing a few lines of Python — no data copy required. Z3FDB stores are cheap to create because data is only fetched when a chunk is accessed.

Very large datasets

You have more data in FDB than you could redundantly store as both GRIB and Zarr. Z3FDB gives you Zarr-compatible access with zero extra storage.

Installation

Z3FDB requires libfdb5 to be present at runtime.

During the CMake configure step, set:

-DENABLE_PYTHON_ZARR_INTERFACE=ON

After building with ninja or make, a Python project is created inside CMAKE_BINARY_DIR in a directory named staging…. Change into that directory and install normally:

pip install .

To verify the installation, run the test suite:

pytest <FDB_SRC_FOLDER>/tests/z3fdb

Next steps

New to Z3FDB? Start with the step-by-step tutorial: Getting Started.

For a full reference on how MARS keywords map to Zarr dimensions, chunking strategies, fill values, and multi-part views, see Dimension Mapping and Data Model.