GRIB and GribJump Extractors

Z3FDB ships two extractor backends. Both implement the same interface and both read the same data from the same FDB; the only difference is how the values leave a GRIB message. This page covers what each one does, how to choose, and the constraints each imposes on a view.

For the machinery around them, see Extractor Ownership and Lifetime. That covers ownership, the factory, and when the layout is established.

Choosing between them

	ExtractorType.Grib	ExtractorType.GribJump
How values are read	Full eccodes decode of the message	Jumps to the values inside the message
Grid-point axis	Always one chunk: the whole field	Optionally split, via field_chunking
Best for	Reading whole fields	Reading a sub-range of each field, repeatedly
Availability	Always	Opt-in build feature (see Availability)

The rule of thumb: if you read whole fields, Grib is the simpler choice and there is nothing to gain from GribJump. If you slice into the grid-point axis, GribJump avoids decoding the values you are going to discard. A time series at one location, or a region out of a global field, are the typical cases, and field_chunking is what lets zarr fetch only the piece you asked for.

Mixed-Extractor Custom Store builds one store containing both, which is a reasonable pattern when different consumers of the same data have different access patterns.

GribExtractor

Reads GRIB messages from FDB and decodes them to float32 via eccodes. The entire field is decoded on every chunk access, so the implicit grid-point dimension is always a single chunk covering the whole field.

Where a message carries a bitmap, the masked grid points are replaced with the view’s fill value (see fill_missing_value()).

GribJumpExtractor

Uses the GribJump library to read grid-point values without performing a full GRIB decode.

Field enumeration still goes through FDB: the extractor inspects FDB to learn which fields match the part’s sub-request and in what order, then asks GribJump for the values. The order matters, because it is what maps each field onto its slot in the chunk buffer.

Field chunking. field_chunking subdivides the implicit grid-point dimension into equally sized zarr chunks:

# 5248 grid points, split into 4 chunks of 1312
ExtractorType.GribJump(field_chunking=Chunking.FixedSizeChunk(chunk_shape=1312))

Each extractInto call derives the range it needs from the chunk it was handed; nothing is cached between calls, and there is no separate sub-range selection.

Missing values. GribJump returns a bitmask alongside the values, in which a set bit means the point is valid and a clear bit means it is missing. Masked points are written as the view’s fill value.

Configuration

fdb_config (both backends)

Path to an FDB configuration YAML. When left as None, the extractor inherits the path given to the store builder; if that is also unset, FDB resolves its own configuration from FDB5_CONFIG / FDB_HOME.

gribjump_config (GribJump only)

Path to a GribJump configuration YAML. When set, the binding exports it as GRIBJUMP_CONFIG_FILE before constructing the GribJump object; when unset, that environment variable is used as-is.

field_chunking (GribJump only)

Chunking of the implicit grid-point dimension. Defaults to a single chunk covering the whole field. That is the only setting which can be mixed with a Grib part, see below.

An extractor configuration object is copied when it is handed to add_part, so one object can be passed to as many parts and as many builders as you like, and the fdb_config a builder fills in never leaks back into your object.

Constraints

Three rules are enforced when the view is built. All three surface as RuntimeError in Python, carrying the message quoted below.

A field chunk size must divide the grid exactly. The grid-point dimension is the one dimension a zarr array cannot leave ragged, so 5248 points can be split into chunks of 1312 or 2624, but not 1000:

GribJumpExtractor: field chunk size 1000 does not evenly divide the window size 5248.

A size of zero is rejected earlier still, by Chunking.FixedSizeChunk itself.

Every part of a view must cover the same grid. The grid-point dimension is never the extension axis. Like every other non-extension axis, all parts have to agree on it:

ChunkedDataViewBuilder::build: part 1 has 2048 grid points but part 0 has 5248. The
grid-point dimension is never the extension axis, so every part must cover the same grid;
a view cannot have a ragged last dimension.

A GribJump part mixed with a Grib part must use whole-axis field chunking. Grib always returns the whole field, so its chunk on the grid-point axis is the full grid; a GribJump part using FixedSizeChunk would write a smaller block into a buffer laid out for a larger one:

ChunkedDataViewBuilder::build: part 1 splits the grid-point dimension into chunks of 1312
values but part 0 uses 5248. All parts must agree on the field chunking, so a GribJump part
mixed with a Grib part has to use the default WholeAxisChunking.

Availability

GribExtractor is always present. GribJumpExtractor is built only when fdb is configured with -DENABLE_ZARR_GRIBJUMP_EXTRACTOR=ON, which is off by default. See Installation.

ExtractorType.GribJump is importable and constructible in every build, so your code does not have to branch on how the wheel was compiled. Only building a view from it fails:

from pychunked_data_view import has_gribjump_extractor

if not has_gribjump_extractor:
    ...   # fall back to ExtractorType.Grib()

Concurrency

Each extractor serialises its own extractInto calls with a mutex: it drives a shared FDB handle (and, for GribJump, a shared GribJump object) while the Python layer has released the GIL, so concurrent chunk reads would otherwise race.

The unit of parallelism is therefore the part, not the chunk. Two threads reading chunks that are served by the same part will take turns; two threads reading chunks served by different parts proceed at the same time. Worth knowing when sizing a dask cluster against a single-part store.

See also

Extractor Ownership and Lifetime for the ownership model and the ExtractorDefinition factory, Chunk Access Pipeline for where extractInto sits in the chunk-access pipeline, and API for the full configuration reference.