Getting Started#

This guide walks you through building your first Z3FDB store from scratch. By the end you will have a working Zarr array backed by FDB GRIB data and understand how to index into it.

Prerequisites#

  • Z3FDB installed — see Z3FDB for build instructions.

  • An FDB instance containing GRIB data accessible from your environment.

  • zarr installed.

Your First Store#

The example below creates a 3-dimensional Zarr array from two dates, four time steps, and one surface parameter. Read it top to bottom — each step is explained immediately after the code block.

import zarr
from z3fdb import SimpleStoreBuilder, AxisDefinition, Chunking, ExtractorType

builder = SimpleStoreBuilder()

builder.add_part(
    {
        "class": "od",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "type": "fc",
        "date": ["2020-01-01", "2020-01-02"],   # 2 dates
        "time": ["0000", "0600", "1200", "1800"],  # 4 times
        "levtype": "sfc",
        "step": 0,
        "param": 167,
    },
    [
        AxisDefinition(["date"], Chunking.SINGLE_VALUE),  # → Dim 0, size 2
        AxisDefinition(["time"], Chunking.SINGLE_VALUE),  # → Dim 1, size 4
    ],
    ExtractorType.GRIB,
)

store = builder.build()
data = zarr.open_array(store, mode="r", zarr_format=3, use_consolidated=False)

What each piece does:

SimpleStoreBuilder

Collects MARS requests and axis definitions before assembling the store. Call add_part() once per MARS request, then build() to produce a Zarr store.

add_part(mars_request, axes, ExtractorType.GRIB)

Registers one MARS request. The axes list controls how MARS keywords become Zarr dimensions. ExtractorType.GRIB tells Z3FDB the data is encoded as GRIB.

AxisDefinition(keys, chunking)

Maps one or more MARS keywords to exactly one Zarr dimension. The position in the list sets the dimension index (0-based).

Chunking.SINGLE_VALUE

The most common chunking mode: each value along the axis is its own chunk. Reading data[i, j, :] triggers exactly one FDB retrieve. See Dimension Mapping and Data Model for other options.

Understanding the Array Shape#

After build(), data is a 3-dimensional array:

data.shape  ->  (2, 4, N)

Dim 0 — date:        2 entries  (2020-01-01, 2020-01-02)
Dim 1 — time:        4 entries  (0000, 0600, 1200, 1800)
Dim 2 — grid points: N float32 values decoded from the GRIB field  [implicit]

The implicit final dimension always holds the decoded grid-point values for one field. Its size N is determined by the GRIB grid.

Indexing#

Index the array with one integer per explicit dimension. The implicit dimension is typically retrieved as a slice:

# date index 0 (2020-01-01), time index 2 (1200)
field = data[0, 2, :]   # returns a 1-D NumPy array of N float32 values

# date index 1 (2020-01-02), time index 0 (0000)
field = data[1, 0, :]

No data is fetched from FDB until you index. Building the store is cheap — it validates your MARS request and pre-fetches layout metadata but does not retrieve field values.

Missing Data and Fill Values#

Some GRIB fields use a bitmap to mark individual grid points as missing (for example, a sea-surface temperature field where land points have no value). Z3FDB replaces those bitmap-masked points with a configurable fill value before returning data to you.

By default the fill value is NaN, which means missing points are immediately detectable with numpy.isnan():

import numpy as np

field = data[0, 2, :]
missing = np.isnan(field)

To use a different sentinel — for example -999.0 — call fill_value() on the builder before calling build():

builder = SimpleStoreBuilder()
builder.add_part(...)
builder.fill_value(-999.0)   # replaces missing points with -999.0
store = builder.build()

Warning

If the fill value you choose can also appear as a legitimate field value, there is no way to distinguish actual data from fill-value positions after retrieval. NaN is the default precisely because it cannot occur as a real float32 computation result, making it unambiguous. Only change the fill value if you are certain it cannot collide with your data.

Note

The fill value is written into the Zarr array metadata as well, so Zarr-aware tools and libraries see the same sentinel when they open the store.

Merging Keywords into One Dimension#

A common pattern is to merge date and time into a single datetime axis. Pass both keys to one AxisDefinition:

builder.add_part(
    {
        "class": "od",
        ...,
        "date": ["2020-01-01", "2020-01-02"],
        "time": ["0000", "0600", "1200", "1800"],
        ...,
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),  # -> Dim 0, size 8
    ],
    ExtractorType.GRIB,
)

This produces shape (8, N). The rightmost key varies fastest (C / NumPy row-major order): time cycles through all its values before date advances.

Index:  0           1           2           3           4           ...
date:   2020-01-01  2020-01-01  2020-01-01  2020-01-01  2020-01-02  ...
time:   0000        0600        1200        1800        0000        ...

index = date_i × num_times + time_j

Important

Keyword order matters. ["date", "time"] puts time as the fast axis. ["time", "date"] would put date as the fast axis. Always verify your indexing formula matches the order you declared.

Combining Surface and Pressure-Level Data#

When your analysis spans both surface and pressure-level parameters you need two MARS requests (the levtype keyword changes between them). Z3FDB lets you merge them into a single array by calling add_part twice and declaring which dimension grows across the two parts.

import zarr
from z3fdb import SimpleStoreBuilder, AxisDefinition, Chunking, ExtractorType

builder = SimpleStoreBuilder()

# Part 1 — surface parameters (levtype=sfc): 2 params
builder.add_part(
    {
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "type": "an",
        "step": 0,
        "date": ["2020-01-01", "2020-01-02"],
        "time": ["0000", "0600", "1200", "1800"],
        "levtype": "sfc",
        "param": [165, 166],               # 2 surface params
    },
    [
        AxisDefinition(["date", "time"], Chunking.SINGLE_VALUE),  # Dim 0 — 8 entries
        AxisDefinition(["param"],        Chunking.SINGLE_VALUE),  # Dim 1 — 2 entries
    ],
    ExtractorType.GRIB,
)

# Part 2 — pressure-level parameters (levtype=pl): 2 params × 3 levels = 6 entries
builder.add_part(
    {
        "class": "ea",
        "domain": "g",
        "expver": "0001",
        "stream": "oper",
        "type": "an",
        "step": 0,
        "date": ["2020-01-01", "2020-01-02"],
        "time": ["0000", "0600", "1200", "1800"],
        "levtype": "pl",
        "param": [131, 132],               # 2 pressure-level params
        "levelist": [500, 850, 1000],       # 3 levels
    },
    [
        AxisDefinition(["date", "time"],       Chunking.SINGLE_VALUE),  # Dim 0 — must match Part 1
        AxisDefinition(["param", "levelist"],  Chunking.SINGLE_VALUE),  # Dim 1 — 6 entries
    ],
    ExtractorType.GRIB,
)

# Dim 1 (param) grows: Part 1 contributes indices 0–1, Part 2 contributes 2–7
builder.extend_on_axis(1)

store = builder.build()
data = zarr.open_array(store, mode="r", zarr_format=3, use_consolidated=False)

The resulting array has shape (8, 8, N) — 8 datetime steps and 8 entries on the param dimension (2 sfc + 6 pl).

# Surface param 165, date=2020-01-01, time=1200 → dim0=2, dim1=0
field = data[2, 0, :]

# Pressure-level param 131 at 850 hPa, same time → dim0=2, dim1=3
# (pl part starts at offset 2; within pl part: param 131 → 0, level 850 → 1
#  → flat index 0*3 + 1 = 1, global index = 2 + 1 = 3)
field = data[2, 3, :]

See also

Dimension Mapping and Data Model for a full reference on axis mapping, all chunking strategies, fill values, and the complete multi-part data model.

Common Pitfalls#

MARS request ends with a comma

"...,param=167," — the trailing comma causes a parse error. Omit the comma on the last key–value pair.

Multi-valued keyword not covered by any AxisDefinition

Every MARS keyword that has more than one value must appear in exactly one AxisDefinition. Z3FDB raises an error at build time if you omit one.

Wrong array shape

If data.shape does not match what you expect, check the order of your AxisDefinition list — position in the list is the dimension index.

Large chunk memory use

Chunking.NONE on several axes can produce chunks of many gigabytes. Start with Chunking.SINGLE_VALUE on every axis and switch to NONE only when you know you always read that axis in full. See Dimension Mapping and Data Model for details.