Technical Insights

Note

This page describes Z3FDB’s internal mechanics. It is aimed at contributors and anyone debugging unexpected performance behaviour. If you are using Z3FDB for the first time, start with Getting Started instead.

This page covers internal mechanics that are useful when reasoning about performance, debugging unexpected results, or contributing to the library.

Part, Intersection, and Buffer Layout

When a chunk is accessed the library executes these steps for every Part:

1. Intersection — compute the overlap of the chunk bounding box and the Part bounding box. Parts with no overlap are skipped.

2. FDB retrieve — send a sub-request to FDB for exactly the fields inside the intersection.

3. Buffer fill — decode each returned field (GRIB → float32) and write it at its correct position in the flat chunk buffer (row-major / C order).

The examples below use the two-part view introduced in Dimension Mapping and Data Model: Part A (sfc, 2 params, axis1=[0,1]) and Part B (pl, 4 params, axis1=[2,5]) with 4 date×time values on axis 0.

WHOLE_AXIS chunking — whole-view example

With WHOLE_AXIS on all axes there is one chunk covering the entire array. Both parts are queried and each populates a disjoint region of the buffer.

Chunk (0,0) bounding box: axis0=[0,3], axis1=[0,5]

           0   1   2   3   4   5
         ┌───┬───┬───┬───┬───┬───┐
      0  │ A │ A │ B │ B │ B │ B │
         ├───┼───┼───┼───┼───┼───┤
      1  │ A │ A │ B │ B │ B │ B │
         ├───┼───┼───┼───┼───┼───┤
      2  │ A │ A │ B │ B │ B │ B │
         ├───┼───┼───┼───┼───┼───┤
      3  │ A │ A │ B │ B │ B │ B │
         └───┴───┴───┴───┴───┴───┘

Intersection A = axis0=[0,3], axis1=[0,1]   (left two columns)
Intersection B = axis0=[0,3], axis1=[2,5]   (right four columns)

The chunk buffer mirrors the index space exactly:

           0   1   2   3   4   5
         ┌───┬───┬───┬───┬───┬───┐
      0  │ A │ A │ B │ B │ B │ B │
         ├───┼───┼───┼───┼───┼───┤
      1  │ A │ A │ B │ B │ B │ B │   bufferExtent = [4, 6]
         ├───┼───┼───┼───┼───┼───┤
      2  │ A │ A │ B │ B │ B │ B │
         ├───┼───┼───┼───┼───┼───┤
      3  │ A │ A │ B │ B │ B │ B │
         └───┴───┴───┴───┴───┴───┘

Part A — partAxisOffset=[0,0], bufferOffset=[0,0]
          (intersection starts at A's local origin and at buffer corner)

Part B — partAxisOffset=[0,0], bufferOffset=[0,2]
          (intersection starts at B's local origin but at buffer column 2)

SINGLE_VALUE chunking — single-field example

With SINGLE_VALUE every chunk holds exactly one field. Chunk (1,2) covers one cell that falls inside Part B.

Chunk (1,2) bounding box: axis0=[1,1], axis1=[2,2]

           0   1   2   3   4   5
         ┌───┬───┬───┬───┬───┬───┐
      0  │   │   │   │   │   │   │
         ├───┼───┼───┼───┼───┼───┤
      1  │   │   │ ■ │   │   │   │   ← chunk (1,2)
         ├───┼───┼───┼───┼───┼───┤
      2  │   │   │   │   │   │   │
         ├───┼───┼───┼───┼───┼───┤
      3  │   │   │   │   │   │   │
         └───┴───┴───┴───┴───┴───┘

Intersection with Part B = axis0=[1,1], axis1=[2,2]

Part B — partAxisOffset=[1,0], bufferOffset=[0,0], bufferExtent=[1,1]

   axis 0:  partAxisOffset=1 because the intersection starts at
            date×time index 1 within Part B's local axis.
   axis 1:  partAxisOffset=0 because global param index 2 is the first
            value in Part B's local param axis (B starts at global 2).
   bufferOffset=[0,0] because the intersection coincides with the
            chunk's own lower corner.

FDB returns one field; axis.index(key) within Part B = [1, 0]:
   axis 0:  local = 1 − 1 = 0,   bufPos = 0 + 0 = 0
   axis 1:  local = 0 − 0 = 0,   bufPos = 0 + 0 = 0
   -> written to buffer slot (0, 0)

Extractor Ownership and Lifetime

Each ViewPart holds its Extractor via std::shared_ptr. This allows multiple parts to share one extractor instance (e.g. two parts reading from the same FDB store) without copying it. Concrete extractors are stateful and non-copyable — an FDB-backed extractor owns an open handle that must not be duplicated. Shared ownership also ties each extractor’s lifetime to the view: as long as the ChunkedDataViewImpl is alive, all its extractors remain alive.

Buffer Position Formula

For every field returned by FDB, the buffer slot along each axis i is:

local_i  = axis.index(key) − partAxisOffset[i]
bufPos_i = local_i + bufferOffset[i]

Where:

• axis.index(key) — zero-based position of the key within the Part’s local axis (0 to axisSize − 1).

• partAxisOffset[i] — start of the intersection within the Part’s local axis: intersection.lower[i] − partBoundingBox.lower[i].

• bufferOffset[i] — start of the intersection within the chunk buffer: intersection.lower[i] − chunkBoundingBox.lower[i].

• bufferExtent[i] — total size of the chunk buffer along axis i (= chunk size on that axis).

local_i is the zero-based position within the intersection; adding bufferOffset[i] shifts it to the correct slot in the chunk buffer. The flat buffer index is then the C-order (row-major) combination of all bufPos_i values, with each slot holding num_values consecutive floats.