Introduction

Tensogram is a binary message format for N-dimensional scientific tensors — the kind of data that appears in weather and climate forecasting, Earth observation, medical and microscopy imaging, genomics, particle physics, materials simulation, and machine-learning pipelines. It carries its own metadata, supports arbitrary tensor dimensions, and is fast to encode and decode.

What Tensogram gives you

• Self-describing messages. Every message carries the metadata needed to decode it — shape, dtype, encoding pipeline, application annotations — using CBOR. No external schema required.
• Any number of dimensions. A single message can carry multiple tensors, each with its own shape, dtype, and encoding. A 3-D spectrum, a 2-D field, and a 4-D ensemble tensor can coexist in one message.
• Vocabulary-agnostic. The library never interprets metadata keys. Application layers (MARS at ECMWF, CF in climate, BIDS in neuroimaging, your in-house taxonomy) own key names.
• Transport and file in one format. The same bytes that traverse a socket can be appended to a .tgm file; both support O(1) random access to any object.
• Interop with existing formats. Importers for GRIB and NetCDF let you bring existing data into Tensogram pipelines without a lossy re-modelling step.
• Partial range decode. Extract sub-tensor slices without decoding the whole object — useful for remote data at scale.

Tensogram is developed and maintained by ECMWF and is used in operational weather-forecasting workloads, but nothing in the format is weather-specific. The design targets the N-tensor-at-scale problem common to many scientific domains.

Crate Layout

The primary four Rust crates make up the default workspace build:

tensogram/
├── rust/
│   ├── tensogram        ← encode, decode, framing, file API,
│   │                            validation, remote object store
│   ├── tensogram-encodings   ← simple_packing, shuffle, compression
│   ├── tensogram-cli         ← `tensogram` command-line tool
│   └── tensogram-ffi         ← C FFI layer for C/C++ callers
├── python/
│   └── bindings/             ← Python bindings (PyO3 / maturin)
├── cpp/
│   └── include/              ← C++ wrapper header + C header

On top of those, the repository ships several opt-in crates — the tensogram-grib / tensogram-netcdf importers (exposed as the convert-grib / convert-netcdf CLI subcommands), the tensogram-wasm WebAssembly bindings, and the pure-Rust tensogram-szip / tensogram-sz3 / tensogram-sz3-sys compression crates — together with the separate Python packages tensogram-xarray (xarray backend) and tensogram-zarr (Zarr v3 store backend), and a tensogram-benchmarks crate. See plans/ARCHITECTURE.md for the full crate list and build recipes.

Most users interact with tensogram and the CLI. The encodings crate is used internally by the core but is also importable directly if you need to call the encoding functions outside of a full message.

Installation

Rust:

cargo add tensogram

Python:

pip install tensogram          # core
pip install tensogram[all]     # with xarray + zarr backends

CLI:

cargo install tensogram-cli

See the Quick Start for feature flags, optional dependencies, and detailed setup.

Quick Example

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;
use tensogram::{
    encode, decode, GlobalMetadata, DataObjectDescriptor,
    ByteOrder, Dtype, EncodeOptions, DecodeOptions,
};

// Describe what you're storing: a 100×200 grid of f32 values
let desc = DataObjectDescriptor {
    obj_type: "ntensor".to_string(),
    ndim: 2,
    shape: vec![100, 200],
    strides: vec![200, 1],
    dtype: Dtype::Float32,
    byte_order: ByteOrder::Big,
    encoding: "none".to_string(),
    filter: "none".to_string(),
    compression: "none".to_string(),
    params: BTreeMap::new(),
    hash: None,
};

let global_meta = GlobalMetadata {
    version: 2,
    ..Default::default()
};

// Your raw bytes (100 × 200 × 4 bytes = 80,000 bytes)
let data = vec![0u8; 100 * 200 * 4];

// Encode into a self-contained message
let message = encode(&global_meta, &[(&desc, &data)], &EncodeOptions::default()).unwrap();

// Decode it back
let (meta, objects) = decode(&message, &DecodeOptions::default()).unwrap();
assert_eq!(objects[0].0.shape, vec![100, 200]);
assert_eq!(objects[0].1, data);
}

The message bytes can be written to a file, sent over a socket, or stored in a database. The receiver does not need any external schema — everything is self-describing.

What is a Message?

A Tensogram message is a single, self-contained binary blob. It carries:

1. A Preamble – fixed-size header with magic bytes, version, flags, and total length
2. Optional header frames – metadata, index, and hash frames for fast random access
3. One or more data object frames – each containing a CBOR descriptor and the actual tensor bytes
4. Optional footer frames – metadata, index, and hash frames (used in streaming mode)
5. A Postamble – footer offset and terminator magic

Every message begins with the ASCII string TENSOGRM and ends with 39277777. This makes it trivial to find message boundaries even in a file containing hundreds of concatenated messages.

Structure at a Glance

block-beta
    columns 1
    A["PREAMBLE (24 bytes)\nTENSOGRM · version · flags · total_length"]
    B["Header Metadata Frame (optional)\nCBOR GlobalMetadata"]
    C["Header Index Frame (optional)\nobject count + offsets"]
    D["Header Hash Frame (optional)\nobject count + hash type + hashes"]
    E["Data Object Frame 0\nCBOR descriptor + payload bytes"]
    F["Data Object Frame 1 (if present)\nCBOR descriptor + payload bytes"]
    G["... (more data object frames)"]
    H["Footer Hash / Index / Metadata Frames (optional)"]
    I["POSTAMBLE (16 bytes)\nfirst_footer_offset · 39277777"]

Frame-Based Design

The v2 wire format is entirely frame-based. Every piece of data between the Preamble and Postamble is wrapped in a frame. Each frame starts with a 4-byte marker (FR + a uint16 frame type), a version, flags, and a length field. This uniform structure means a decoder can skip any frame it does not understand by jumping over its declared length.

Frame types:

Padding between frames is allowed (from ENDF to the next FR marker) for 64-bit memory alignment.

Why Header Frames?

When a message is encoded in a single buffer (the common case), the index and hash frames are placed in the header, right after the Preamble. A decoder reads the Preamble, then the metadata frame, then the index frame, and can immediately seek to any data object by offset. That is O(1) random access, which matters when a message carries many large tensors.

Streaming Support

When encoding in streaming mode, the producer may not know in advance how many data objects the message will contain. In this case:

• total_length in the Preamble is set to 0 (unknown)
• Index and hash frames are written in the footer instead of the header
• The Postamble’s first_footer_offset field points back to where the footer frames begin

A decoder reading a streamed message seeks to the end, reads the Postamble, then jumps to the footer frames to find the index. Both paths (header index and footer index) give O(1) access to any object.

Data Object Frames

Each data object is self-contained in its own frame. The frame carries:

• A CBOR descriptor (DataObjectDescriptor) describing the tensor shape, dtype, encoding pipeline, and optional hash
• The binary payload (the actual encoded tensor bytes)

The CBOR descriptor can appear before or after the payload within the frame. By default it is placed after the payload, since some encoding parameters (like hash values) are only known after the payload has been written. A flag in the frame header indicates the position.

Messages vs Files

A .tgm file is just a sequence of messages written one after another:

[message 1][message 2][message 3]...

There is no file-level index or header. The TensogramFile API scans the file once (lazily, on first access) and builds an in-memory list of (offset, length) pairs for each message. After that, reading any message is a seek + read – no scan needed.

To find message boundaries in a file:

1. Scan for TENSOGRM magic (8 bytes)
2. If total_length is non-zero, use it to advance to the next message
3. Otherwise, walk frames using their length fields until the next magic or EOF

Self-Description

Every message carries all the information needed to decode it:

• The dtype of every object (float32, int16, etc.)
• The shape and strides (dimensions and memory layout)
• The full encoding pipeline applied to the payload (encoding, filter, compression)
• The byte order of each object’s data
• Any application-level metadata (MARS keys, units, timestamps, etc.)

This means a decoder never needs an external schema. You can receive a Tensogram message on a new machine, years after it was encoded, and decode it correctly.

Edge Case: Zero-Object Messages

A message with no data object frames is valid. It contains only the Preamble, a metadata frame, and the Postamble. This is useful for sending pure metadata (e.g. a control message or an acknowledgement with provenance information) without any tensor payload.

#![allow(unused)]
fn main() {
let metadata = GlobalMetadata {
    version: 2,
    ..Default::default()
};
let msg = encode(&metadata, &[], &EncodeOptions::default()).unwrap();
}

Metadata

Metadata in Tensogram is stored as CBOR – Concise Binary Object Representation (RFC 8949). Think of it as a compact, binary version of JSON. It supports the same types (strings, integers, floats, booleans, arrays, maps), but is smaller and faster to parse.

Metadata Locations

In v2, metadata lives in two distinct places:

Each data object carries its own descriptor inline within its frame.

GlobalMetadata

The global metadata frame contains a GlobalMetadata struct with three named sections:

#![allow(unused)]
fn main() {
GlobalMetadata {
    version: 2,
    base: Vec::new(),              // one BTreeMap per data object (independent entries)
    reserved: BTreeMap::new(),     // library internals (_reserved_ in CBOR)
    extra: BTreeMap::new(),        // client-writable catch-all (_extra_ in CBOR)
}
}

In CBOR, this looks like (using ECMWF MARS keys as one concrete example vocabulary):

{
  "version": 2,
  "base": [
    {
      "mars": {
        "class": "od", "type": "fc",
        "date": "20260401", "time": "1200", "param": "2t"
      }
    }
  ],
  "_extra_": {
    "source": "ifs-cycle49r2"
  }
}

The same mechanism works for any application vocabulary. A neuroimaging pipeline might use a BIDS namespace:

{
  "version": 2,
  "base": [{
    "bids": { "subject": "sub-01", "session": "ses-01",
              "task": "rest", "run": 1 }
  }]
}

A materials-simulation pipeline might use a custom namespace:

{
  "version": 2,
  "base": [{
    "material": { "composition": "Fe3O4", "lattice": "cubic", "T_K": 300.0 }
  }]
}

The library does not know or care which vocabulary is used — it simply stores, serialises, and returns the keys you supply.

The version field is required (u16). The base array holds per-object metadata. _extra_ is a free-form catch-all – you can add any key using any CBOR value type. The library does not interpret or validate these keys. Your application layer assigns meaning.

Per-Object Metadata in base

The base section is a CBOR array of maps — one entry per data object. Each entry holds ALL structured metadata for that object independently. Entries are self-contained — there is no tracking of which keys are common across objects.

The encoder auto-populates _reserved_.tensor (with ndim, shape, strides, dtype) in each entry when you call encode() or StreamingEncoder::finish(). Application keys are preserved:

{
  "base": [
    {
      "mars": { "class": "od", "type": "fc", "param": "2t", "levtype": "sfc" },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float64" }
      }
    },
    {
      "mars": { "class": "od", "type": "fc", "param": "10u", "levtype": "sfc" },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float64" }
      }
    }
  ]
}

This lets readers discover the shape, type, and per-object metadata of every object by reading only the global metadata frame — without opening each data object frame.

No common/varying split: Every base[i] entry is self-contained. MARS keys shared across all objects (e.g. class, type) are simply repeated in each entry. If you need to extract commonalities (e.g. for display or merges), use the compute_common() utility in software after decoding.

DataObjectDescriptor

The params field of each DataObjectDescriptor is a BTreeMap<String, ciborium::Value> for encoding parameters only (e.g. reference_value, bits_per_value). These are flattened into the CBOR descriptor alongside the fixed tensor fields.

For example, a data object’s CBOR descriptor might look like:

{
  "type": "ntensor",
  "ndim": 2,
  "shape": [721, 1440],
  "strides": [1440, 1],
  "dtype": "float32",
  "byte_order": "big",
  "encoding": "simple_packing",
  "filter": "none",
  "compression": "szip",
  "reference_value": 230.5,
  "bits_per_value": 16,
  "hash": { "type": "xxh3", "value": "a1b2c3d4e5f6..." }
}

Here, reference_value and bits_per_value live in the params map. Application metadata such as MARS keys belongs in base[i]["mars"] in the global metadata.

Namespaced Keys

Convention: application-layer keys are grouped under a namespace key, so that multiple vocabularies can coexist in the same message. For example, ECMWF’s MARS vocabulary lives under "mars":

{
  "version": 2,
  "base": [
    {
      "mars": {
        "class": "od", "type": "fc",
        "param": "2t", "date": "20260401", "step": 6
      }
    }
  ]
}

Other pipelines use other namespaces — "cf" for CF conventions, "bids" for neuroimaging, "dicom" for medical imaging, or anything your application defines. This convention applies at both levels — global metadata and per-object params.

Filtering with the CLI

The -w flag on ls, dump, get, and copy uses dot-notation to filter messages on any namespace. The examples below use the MARS vocabulary, but the same syntax works with any application namespace (e.g. bids.subject, dicom.Modality, product.name):

# Only messages where mars.param equals "2t" or "10u"
tensogram ls data.tgm -w "mars.param=2t/10u"

# Exclude messages where mars.class equals "od"
tensogram ls data.tgm -w "mars.class!=od"

The / character separates OR values. Key lookup searches base[i] entries first (skipping _reserved_, first match across entries), then _extra_ for backwards compatibility.

Preceder Metadata Frames

In streaming mode, per-object metadata is normally only available in the footer metadata frame (written after all objects). A Preceder Metadata Frame (frame type 8) allows producers to send per-object metadata before the data object, without waiting for the footer.

A preceder carries a GlobalMetadata CBOR with a single-entry base array for the next data object:

{
  "version": 2,
  "base": [{"product": {"name": "temperature"}, "units": "K"}]
}

Merge rule: On decode, preceder keys override footer base[i] keys on conflict. Structural keys auto-populated by the encoder (in _reserved_.tensor: ndim, shape, strides, dtype) are preserved from the footer when absent from the preceder. The consumer sees a unified GlobalMetadata.base — the preceder/footer distinction is transparent.

Use StreamingEncoder::write_preceder() before write_object() to emit a preceder frame. Preceders are optional per-object: some objects may have them, others may not.

Value Type Rules

Keys must be text strings. Values must be JSON-compatible CBOR types: string, integer, float, boolean, null, array, or map. Byte strings, CBOR tags, undefined, and half-precision floats are not allowed. See Metadata Value Types for the full rules and rationale.

Deterministic Encoding

When Tensogram encodes metadata to CBOR, it sorts all map keys by their CBOR byte representation (RFC 8949 Section 4.2 canonical form). This guarantees that the same metadata always produces the same bytes, regardless of the order you inserted keys in your application code. This matters for hashing and reproducibility.

Edge case: Nested maps are also sorted recursively. Even metadata stored inside a CBOR map value (like the "mars" namespace) gets canonical ordering.

Objects and Dtypes

An object is one N-dimensional tensor inside a message. A message can carry multiple objects. In v2, each object is fully described by a single struct:

• A DataObjectDescriptor carrying tensor metadata, encoding pipeline, and integrity hash – all in one place
• The actual binary payload within the object’s frame

There is no separate “payload descriptor” array. The descriptor travels with the data inside the same frame.

DataObjectDescriptor

#![allow(unused)]
fn main() {
DataObjectDescriptor {
    // ── Tensor metadata ──
    obj_type: "ntensor",           // always "ntensor" for now
    ndim: 2,                       // number of dimensions
    shape: vec![100, 200],         // size of each dimension
    strides: vec![200, 1],         // elements to skip per dimension step
    dtype: Dtype::Float32,         // element type

    // ── Encoding pipeline ──
    byte_order: ByteOrder::Big,    // big or little endian
    encoding: "simple_packing",    // or "none"
    filter: "shuffle",             // or "none"
    compression: "szip",           // or "none", "zstd", "lz4", etc.

    // ── Flexible parameters (encoding only) ──
    params: BTreeMap::from([       // BTreeMap<String, ciborium::Value>
        ("reference_value".into(), ciborium::Value::Float(230.5)),
        ("bits_per_value".into(), ciborium::Value::Integer(16.into())),
    ]),

    // ── Integrity ──
    hash: Some(HashDescriptor {
        hash_type: "xxh3",
        value: "a1b2c3d4e5f6...",
    }),
}
}

The params map is flattened into the CBOR alongside the fixed fields, so the on-wire CBOR is a single flat map. This keeps things simple for decoders – no nested “encoding” or “tensor” sub-objects to navigate.

Each data object has its own descriptor, so different objects in the same message can use different encodings, byte orders, and hash algorithms.

Strides

Strides tell you how to navigate the memory layout. For a C-contiguous (row-major) array of shape [100, 200]:

• Advancing along axis 0 (rows) skips 200 elements
• Advancing along axis 1 (columns) skips 1 element

So strides = [200, 1]. For a Fortran-contiguous (column-major) array the strides would be reversed: [1, 100].

To compute C-contiguous strides from shape:

#![allow(unused)]
fn main() {
fn compute_strides(shape: &[u64]) -> Vec<u64> {
    let mut strides = vec![1u64; shape.len()];
    for i in (0..shape.len() - 1).rev() {
        strides[i] = strides[i + 1] * shape[i + 1];
    }
    strides
}
// shape [100, 200] → strides [200, 1]
// shape [4, 5, 6]  → strides [30, 6, 1]
}

Supported Data Types

Edge case: bitmask returns 0 from byte_width(). Callers that need the actual byte count must compute it from the element count: (num_elements + 7) / 8.

Multiple Objects in One Message

A message can carry several related tensors. Two concrete examples:

• A wave-spectrum message with the spectrum itself as a 3-tensor and a land/sea mask as a 2-tensor.
• A medical-imaging message with a 4-D time-series volume, a 3-D segmentation mask, and a 1-D array of acquisition timestamps.

block-beta
    columns 3
    A["Object 0\nSpectrum\nf32 · 721×1440×30\nencoding: simple_packing"]:2
    B["Object 1\nLand mask\nuint8 · 721×1440\nencoding: none"]:1

All objects live in the same message. Each object has its own DataObjectDescriptor embedded in its frame and its own entry in GlobalMetadata.base holding per-object application metadata. Different objects can use completely different encoding pipelines.

Edge case: The number of DataObjectDescriptor entries and the data slices passed to encode() must be equal. The encoder returns an error if they do not match.

The Encoding Pipeline

Every object payload passes through a three-stage pipeline on the way in (encoding) and out (decoding). The stages always run in the same order:

flowchart TD
    subgraph Encode["Encode Path"]
        direction TB
        A["Raw bytes"]
        B["Stage 1 — Encoding
        (lossy quantization)"]
        C["Stage 2 — Filter
        (byte shuffle)"]
        D["Stage 3 — Compression
        (szip / zstd / lz4 / blosc2 / zfp / sz3)"]
        A --> B --> C --> D
    end

    S[("Stored bytes")]

    subgraph Decode["Decode Path"]
        direction TB
        F["Stage 3 — Decompress"]
        G["Stage 2 — Unshuffle"]
        H["Stage 1 — Dequantize"]
        I["Raw bytes"]
        F --> G --> H --> I
    end

    D --> S --> F

    style A fill:#e8f5e9,stroke:#388e3c
    style S fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style I fill:#e8f5e9,stroke:#388e3c
    style Encode fill:#e3f2fd,stroke:#1565c0,color:#1565c0
    style Decode fill:#fce4ec,stroke:#c62828,color:#c62828

Each stage is independently configurable per object via fields in the DataObjectDescriptor. Set a stage to "none" to skip it. For callers with already-encoded payloads, a pipeline-bypass option exists via encode_pre_encoded (see Pre-encoded Payloads).

Stage 1: Encoding

Encoding transforms values to reduce the number of bits needed to represent them. The only supported encoding right now is simple_packing — a lossy quantisation that maps a bounded range of floating-point values onto N-bit integers. The bit layout matches GRIB 2 simple_packing so quantised payloads are interoperable with existing GRIB tooling.

Stage 2: Filter

Filters rearrange bytes to improve compression ratios. The shuffle filter reorders bytes by their significance level (all most-significant bytes first, then all second-most-significant bytes, etc.), which makes float data much more compressible because nearby values have similar high bytes.

Stage 3: Compression

Compression reduces the total byte count. Seven compressors are implemented:

See Compression for full details on each compressor, including parameters and random access support.

Note: ZFP and SZ3 operate directly on typed floating-point data. Use them with encoding: "none" and filter: "none" – they replace both encoding and compression.

Typical Combinations

How It Looks in Code

The entire pipeline is configured through the DataObjectDescriptor:

#![allow(unused)]
fn main() {
DataObjectDescriptor {
    obj_type: "ntensor".into(),
    ndim: 2,
    shape: vec![721, 1440],
    strides: vec![1440, 1],
    dtype: Dtype::Float32,
    byte_order: ByteOrder::Big,
    encoding: "simple_packing".into(),
    filter: "none".into(),
    compression: "szip".into(),
    params: BTreeMap::from([
        ("reference_value".into(), Value::Float(230.5)),
        ("bits_per_value".into(), Value::Integer(16.into())),
    ]),
    hash: None, // set automatically during encoding
}
}

All encoding parameters (reference_value, bits_per_value, szip_block_offsets, etc.) go into the params map. The encoder populates additional params during encoding (like block offsets for szip), and the decoder reads them back.

Integrity Hashing

After all three stages, the stored bytes can be hashed. The hash is stored in the DataObjectDescriptor’s hash field alongside the encoded bytes. On decode, if verify_hash: true is set, the hash is recomputed and compared.

Edge case: The hash covers the stored bytes (after encoding + filter + compression), not the original raw bytes. This means a hash mismatch always indicates storage or transmission corruption, not a quantization difference from lossy encoding.

Wire Format (v3)

This page describes the exact byte layout of a Tensogram v3 message — the format shipped in 0.17.0. You need this if you are implementing a reader in another language, debugging a corrupted file, or just want to understand what is happening under the hood. For the normative specification, see plans/WIRE_FORMAT.md.

All integer fields are big-endian (network byte order).

Overview

A Tensogram message is built from three sections: a header (preamble + optional frames), one or more data object frames, and a footer (optional frames + postamble).

┌────────────────────────────────────────────────────────────────────┐
│  PREAMBLE                  magic, version, flags, length  (24 B)   │
├────────────────────────────────────────────────────────────────────┤
│  HEADER METADATA FRAME     CBOR global metadata      (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  HEADER INDEX FRAME        CBOR object offsets       (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  HEADER HASH FRAME         CBOR object hashes        (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  PRECEDER METADATA FRAME   per-object metadata       (optional)    │
│  DATA OBJECT FRAME 0       header + payload + descriptor           │
│  PRECEDER METADATA FRAME   per-object metadata       (optional)    │
│  DATA OBJECT FRAME 1       ...                                     │
│  DATA OBJECT FRAME 2       (no preceder)                           │
│  ...                       (any number of objects)                 │
├────────────────────────────────────────────────────────────────────┤
│  FOOTER HASH FRAME         CBOR object hashes        (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  FOOTER INDEX FRAME        CBOR object offsets       (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  FOOTER METADATA FRAME     CBOR global metadata      (optional)    │
├────────────────────────────────────────────────────────────────────┤
│  POSTAMBLE   first_footer_offset, total_length, end_magic  (24 B)  │
└────────────────────────────────────────────────────────────────────┘

At least one metadata frame (header or footer) must be present — messages cannot exist without metadata. Index and hash frames are optional but highly encouraged. By default, the encoder places them in the header when writing to a buffer, or in the footer when streaming.

Frame ordering: The decoder enforces that frames appear in order: header frames, then data object frames, then footer frames. A header frame appearing after a data object frame, or a data object frame appearing after a footer frame, is rejected as malformed.

Preamble (24 bytes)

The preamble is the fixed-size start of every message.

Offset  Size    Field
──────  ──────  ─────────────────────────────────
0       8       Magic: "TENSOGRM" (ASCII)
8       2       Version (uint16 BE) — must be 3 in v3
10      2       Flags (uint16 BE)
12      4       Reserved (uint32 BE) — set to zero
16      8       Total length (uint64 BE)

Total length is the byte count of the entire message from the first byte of the preamble to the last byte of the postamble. A value of zero means the encoder is in streaming mode — the total length was not known when the preamble was written.

Version compatibility. v3 decoders reject any preamble whose version field is not exactly 3. Older v1/v2 messages must be re-encoded.

Preamble flags

The flags field is a bitmask indicating which optional frames are present and, new in v3, whether inline per-frame hash slots are populated:

Unused flag bits must be set to zero.

Frames

Every frame (header, footer, and data object) shares a common 16-byte frame header and ends with a type-specific footer whose last 12 bytes are always [hash u64][ENDF 4] (new in v3).

Frame header (16 bytes)

Offset  Size    Field
──────  ──────  ─────────────────────────────────
0       2       Start marker: "FR" (ASCII)
2       2       Frame type (uint16 BE)
4       2       Frame version (uint16 BE)
6       2       Reserved flags (uint16 BE)
8       8       Frame length — offset to end of frame (uint64 BE)

Frame versions are independent from the message version and from each other.

Frame common footer (12 bytes)

Every frame ends with this fixed-size tail:

Offset (from frame end)  Size    Field
───────────────────────  ──────  ─────────────────────────────────
-12                      8       hash (uint64 BE) — xxh3-64 digest of the frame body, or 0x0000000000000000 when HASHES_PRESENT = 0
-4                       4       End marker: "ENDF" (ASCII)

Data-object frames (type 9) have a larger 20-byte footer that adds an 8-byte cbor_offset field before the common tail.

Frame types

The body phase of a v3 message carries one or more data-object frames. In v3 only NTensorFrame (type 9) is defined; future types can slot in at fresh unused numbers without bumping the wire version.

Padding between frames

It is valid to have padding bytes between a frame’s ENDF marker and the next frame’s FR marker. This allows encoders to align frame starts to 8-byte (64-bit) boundaries for memory-mapped access.

Data Object Frames

A data object frame wraps one tensor’s payload together with its CBOR descriptor. v3 defines exactly one concrete data-object type, NTensorFrame (type 9). The descriptor can go either before or after the payload — flag bit 0 in the frame header controls this. The default is after, because when encoding the descriptor is sometimes only fully known once the payload has been written (e.g. after computing a hash or determining compressed size).

NTensorFrame (type 9) — v3 canonical layout

┌──────────────────────────────────────────────────────────────┐
│  FRAME HEADER       "FR" + type(9) + ver + flags + len (16 B)│
├──────────────────────────────────────────────────────────────┤
│  DATA PAYLOAD       raw or compressed bytes, NaN/Inf         │
│                     positions substituted with 0.0           │
├──────────────────────────────────────────────────────────────┤
│  mask_nan blob      OPTIONAL — compressed NaN position mask  │
├──────────────────────────────────────────────────────────────┤
│  mask_inf+ blob     OPTIONAL — compressed +Inf position mask │
├──────────────────────────────────────────────────────────────┤
│  mask_inf- blob     OPTIONAL — compressed -Inf position mask │
├──────────────────────────────────────────────────────────────┤
│  CBOR DESCRIPTOR    carries a top-level "masks" sub-map      │
│                     when any mask is present (see below)     │
├──────────────────────────────────────────────────────────────┤
│  cbor_offset (uint64 BE, 8 B)                                │
│  hash        (uint64 BE, 8 B)   xxh3-64 of body              │
│  "ENDF"      (4 B)                                           │
└──────────────────────────────────────────────────────────────┘

The data-object footer is 20 bytes: [cbor_offset u64] [hash u64][ENDF 4]. The cbor_offset field points at the CBOR descriptor’s start relative to the frame’s first byte. The inline hash slot carries the xxh3-64 of the frame body (everything between the 16-byte header and this 20-byte footer) when the message’s HASHES_PRESENT preamble flag is set; otherwise it is 0x0000000000000000.

Hash scope includes payload + masks + CBOR. It does NOT include the header, the cbor_offset field, the hash slot itself, or ENDF.

The CBOR descriptor fully describes the data object: its type, shape, strides, data type, byte order, encoding pipeline, and optional per-object metadata. See the CBOR Metadata page for the schema.

See NaN / Inf Handling for the mask encode / decode semantics and the documented lossy-reconstruction caveat.

Preceder Metadata Frame

A Preceder Metadata Frame (type 8) optionally appears immediately before a Data Object Frame. It carries per-object metadata for the following data object, using the same GlobalMetadata CBOR format but with a single-entry base array.

Use case: Streaming producers that do not know ahead of time when the message will end can emit per-object metadata early via preceders, rather than waiting for the footer.

Ordering rules:

• Must appear in the data objects phase (after headers, before footers).
• Must be followed by exactly one Data Object Frame.
• Two consecutive preceders without an intervening DataObject are invalid.
• A dangling preceder (not followed by a DataObject) is invalid.
• Preceders are optional per-object.

CBOR structure:

{
  "version": 2,
  "base": [{"mars": {"param": "2t"}, "units": "K"}]
}

Merge on decode: Preceder keys override footer base[i] keys on conflict. Footer-only keys (e.g., auto-populated _reserved_.tensor with ndim, shape, strides, dtype) are preserved. The consumer sees a unified GlobalMetadata.base — the preceder/footer distinction is transparent.

Postamble (16 bytes)

The postamble sits at the very end of every message.

Offset  Size    Field
──────  ──────  ─────────────────────────────────
0       8       first_footer_offset (uint64 BE)
8       8       End magic: "39277777" (ASCII)

first_footer_offset is the byte offset (from the start of the message) to the first footer frame. This is never zero:

• If footer frames exist, it points to the start of the first one (e.g., the Footer Hash Frame).
• If no footer frames exist, it points to the postamble itself.

This guarantee means a reader can always distinguish “no footer frames” from “footer at offset 0” without ambiguity.

The end magic 39277777 was chosen because it is unlikely to appear naturally in floating-point or integer data, making it useful as a corruption boundary detector.

Random Access Patterns

With a header index (most common)

When a message was written in non-streaming mode, the index is in the header. This is the fastest path — no seeking to the end required.

1. Read preamble (24 B) → check flags
2. Read header metadata frame → global context
3. Read header index frame → offsets[], lengths[]
4. Seek to offsets[N], read data object frame → decode

With a footer index only (streaming mode)

When a message was written in streaming mode, the encoder did not know the object count or offsets up front. The index lives in the footer.

1. Seek to end − 24, read postamble → first_footer_offset
2. Seek to first_footer_offset, scan footer frames → find index
3. Read footer index frame → offsets[], lengths[]
4. Seek to offsets[N], read data object frame → decode

Both paths give O(1) access to any data object by index. The object count is derived from offsets.len().

Scanning a Multi-Message File

Multiple messages can be concatenated into a single .tgm file. To find message boundaries:

1. Scan forward for the TENSOGRM magic (8 bytes).
2. Read total_length from the preamble.
   • If total_length is non-zero, advance by that many bytes to reach the next message.
   • If total_length is zero (streaming mode), use the header index frame length if present.
3. If neither total length nor header index is available, walk frame-by-frame — each frame header contains a length field — until the next TENSOGRM magic or EOF.
4. Verify the 39277777 end magic at the expected position to confirm message integrity.

flowchart TD
    A[Start of file] --> B{Find TENSOGRM?}
    B -- No --> Z[End of scan]
    B -- Yes --> C[Read total_length at +16]
    C --> D{total_length > 0?}
    D -- Yes --> E[Advance to offset + total_length]
    D -- No --> F[Walk frame-by-frame to next magic]
    E --> G[Verify 39277777 end magic]
    F --> G
    G -- Valid --> H[Record message]
    H --> B
    G -- Invalid --> I[Skip 1 byte, resume scan]
    I --> B

If the end magic does not match, the message is likely corrupt. The scanner skips one byte and resumes searching — this is the corruption recovery path.

A Note on CBOR

Frames that contain CBOR data (metadata, index, hash) use length-prefixed CBOR encoding — there are no explicit start/end markers within the CBOR stream itself. The CBOR decoder reads the first byte to determine the data type and item count, then consumes exactly that many bytes. The frame boundaries (FR…ENDF) provide the outer containment.

All CBOR maps use deterministic encoding with canonical key ordering (RFC 8949 section 4.2). See CBOR Metadata for details.

CBOR Metadata Schema

Tensogram v2 uses CBOR (Concise Binary Object Representation) for all structured metadata. There are four kinds of CBOR structures in a message, each living in its own frame:

1. GlobalMetadata — in header or footer metadata frames
2. DataObjectDescriptor — inside each data object frame
3. IndexFrame — in header or footer index frames
4. HashFrame — in header or footer hash frames

All CBOR maps use deterministic encoding with canonical key ordering per RFC 8949 section 4.2. Keys are sorted by the byte representation of their CBOR-encoded key, applied recursively to nested maps. This means the same metadata always produces the same bytes — important if you hash messages or compare them by digest.

GlobalMetadata

The global metadata frame contains a single CBOR map. The only required key is version; everything else is optional.

Each data object is self-describing via its own per-frame descriptor (see below). The base array provides per-object metadata at the message level so readers can discover object metadata from the global frame alone, without opening each data object frame.

The base Array

The base array is one entry per data object. Each entry is a CBOR map holding ALL structured metadata for that object. The encoder auto-populates _reserved_.tensor (containing ndim, shape, strides, dtype) in each entry. Application keys (e.g. "mars") are preserved:

{
  "base": [
    {
      "mars": { "class": "od", "stream": "oper", "param": "2t", "date": "20260404" },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float32" }
      }
    },
    {
      "mars": { "class": "od", "stream": "oper", "param": "10u", "date": "20260404" },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float32" }
      }
    }
  ]
}

Each entry corresponds to one data object in order. Entries are independent — there is no tracking of which keys are common across objects. If you need to extract commonalities (e.g. for display or merge operations), use the compute_common() utility in software after decoding.

Key difference from earlier versions: There is no common/payload split. Every base[i] entry is self-contained. MARS keys that are shared across all objects (e.g. class, stream, date) are simply repeated in each entry.

The _reserved_ Section

The _reserved_ section at the message level holds library-managed provenance information. Client code can read these values but must not write to _reserved_ — the encoder validates this and rejects messages where client code has written to it.

{
  "_reserved_": {
    "encoder": { "name": "tensogram", "version": "0.1.0" },
    "time": "2026-04-06T12:00:00Z",
    "uuid": "550e8400-e29b-41d4-a716-446655440000"
  }
}

Note: _reserved_.encoder.version is set to the library’s crate version at compile time via env!("CARGO_PKG_VERSION") — the value above reflects the tensogram version in use.

Within each base[i] entry, the encoder also auto-populates _reserved_.tensor:

{
  "_reserved_": {
    "tensor": {
      "ndim": 2,
      "shape": [721, 1440],
      "strides": [1440, 1],
      "dtype": "float32"
    }
  }
}

The _extra_ Section

The _extra_ section is a client-writable catch-all for ad-hoc message-level annotations:

{
  "_extra_": {
    "source": "ifs-cycle49r2",
    "experiment_tag": "alpha-run-003"
  }
}

Example GlobalMetadata

A complete example with two data objects (temperature and wind fields):

{
  "version": 2,
  "base": [
    {
      "mars": {
        "class": "od", "stream": "oper", "expver": "0001",
        "date": "20260404", "time": "0000", "step": "0",
        "levtype": "sfc", "grid": "regular_ll", "param": "2t"
      },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float32" }
      }
    },
    {
      "mars": {
        "class": "od", "stream": "oper", "expver": "0001",
        "date": "20260404", "time": "0000", "step": "0",
        "levtype": "sfc", "grid": "regular_ll", "param": "10u"
      },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float32" }
      }
    }
  ],
  "_reserved_": {
    "encoder": { "name": "tensogram", "version": "0.6.0" },
    "time": "2026-04-06T12:00:00Z",
    "uuid": "550e8400-e29b-41d4-a716-446655440000"
  },
  "_extra_": {
    "source": "ifs-cycle49r2"
  }
}

Each base[i] entry is fully self-contained. The only key that varies between the two entries above is param. All other MARS keys are repeated — this is by design. Commonalities can be computed in software via compute_common() when needed.

Optional: Full GRIB Namespace Keys

When the GRIB importer runs with preserve_all_keys (CLI: --all-keys), all non-mars ecCodes namespace keys are stored under a "grib" sub-object within each base[i] entry:

{
  "base": [
    {
      "mars": { "class": "od", "grid": "regular_ll", "param": "2t", "..." : "..." },
      "grib": {
        "geography": { "Ni": 1440, "Nj": 721, "gridType": "regular_ll" },
        "time":      { "dataDate": 20260404, "dataTime": 0 },
        "ls":        { "edition": 2, "centre": "ecmf", "packingType": "grid_ccsds" },
        "parameter":  { "paramId": 167, "shortName": "2t", "units": "K" },
        "statistics": { "max": 311.03, "min": 212.84, "avg": 277.6 }
      },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float64" }
      }
    }
  ]
}

The namespaces captured are: ls, geography, time, vertical, parameter, statistics. Keys may overlap between namespaces (e.g. gridType appears in both ls and geography); each namespace stores its own copy. Empty namespaces are omitted.

DataObjectDescriptor

Each data object frame contains its own CBOR descriptor. This descriptor fully describes how to decode the payload — its type, shape, encoding pipeline, and optional per-object metadata. It lives inside the data object frame (not in a central metadata block).

Example: Temperature Field Descriptor

Here is what a descriptor might look like for a global temperature field at 0.25-degree resolution, compressed with zstd:

{
  "type": "ntensor",
  "ndim": 2,
  "shape": [721, 1440],
  "strides": [1440, 1],
  "dtype": "float32",
  "byte_order": "little",
  "encoding": "simple_packing",
  "reference_value": 193.72,
  "binary_scale_factor": -16,
  "decimal_scale_factor": 0,
  "bits_per_value": 16,
  "filter": "none",
  "compression": "zstd",
  "zstd_level": 3,
  "hash": {
    "type": "xxh3",
    "value": "a1b2c3d4e5f60718"
  }
}

The params field in DataObjectDescriptor is for encoding parameters only (e.g. reference_value, bits_per_value). MARS keys and other application metadata are stored in the global metadata base[i]["mars"].

Encoding Parameters (simple_packing)

Filter Parameters (shuffle)

Compression Parameters

szip:

zstd:

lz4: No additional parameters required.

blosc2:

zfp:

sz3:

Hash Descriptor

The optional hash field records an integrity digest of the raw payload bytes.

NaN / Inf mask companion (masks)

When the object was encoded with allow_nan=true and/or allow_inf=true AND the payload actually contained at least one matching non-finite value, the descriptor carries a masks sub-map. Each kind (nan, inf+, inf-) is independently optional — only the kinds that appeared are present.

{
  ... standard DataObjectDescriptor fields ...,
  "masks": {
    "nan": {
      "method": "roaring",
      "offset": 40,
      "length": 12
    },
    "inf+": {
      "method": "rle",
      "offset": 52,
      "length": 3
    }
  }
}

Each entry:

Canonical key order for masks is the byte-lex sort inf+ < inf- < nan. The encoder writes mask blobs between the payload and the CBOR descriptor in the same canonical order. See NaN / Inf Handling for the encode / decode semantics.

IndexFrame

Index frames (header or footer) contain a CBOR map that lets readers jump directly to any data object without scanning.

Object count is derived from offsets.len(); lengths.len() must equal offsets.len() or the decoder emits a MetadataError.

Example IndexFrame

{
  "offsets": [256, 1048832, 2097408],
  "lengths": [1048576, 1048576, 524288]
}

The offsets array gives O(1) random access to any object — seek to offsets[i] and read lengths[i] bytes.

HashFrame

Hash frames (header or footer) mirror the per-object inline hash slots of each data-object frame’s footer (see wire-format.md §2.4), so readers can inspect the aggregate without walking every frame.

Object count is derived from hashes.len(). An unknown algorithm value triggers an UnknownHashAlgorithm warning at validate time; the inline slots remain the authoritative check.

Example HashFrame

{
  "algorithm": "xxh3",
  "hashes": [
    "a1b2c3d4e5f60718",
    "b2c3d4e5f6071829",
    "c3d4e5f60718293a"
  ]
}

Canonical Encoding

All CBOR maps are encoded with keys sorted by the byte representation of their CBOR-encoded key (RFC 8949 section 4.2). This sorting is applied recursively — nested maps are also sorted.

For short string keys (the common case), this is equivalent to sorting by the key string itself. For long keys or non-string keys, the CBOR byte encoding determines the order.

Why does this matter? If you hash an entire message or compare messages by digest, deterministic encoding ensures that logically identical messages produce identical bytes even if the keys were inserted in different order during construction.

Metadata Value Types

All Tensogram metadata — whether in GlobalMetadata, the base / _reserved_ / _extra_ sections, or per-object params — is stored as CBOR. This page describes which value types are valid, which are forbidden, and why.

Allowed Types

Use only the subset of CBOR types that have direct JSON equivalents:

Map keys must be text strings. Nested arrays and maps are allowed and encoded recursively.

Forbidden Types

The following CBOR types are not allowed in Tensogram metadata:

The base Section

The base section of GlobalMetadata is a CBOR array of maps — one entry per data object. Each entry holds ALL structured metadata for that object independently. The encoder auto-populates _reserved_.tensor (with ndim, shape, strides, dtype) in each entry when you call encode() or StreamingEncoder::finish(). Any other keys the application placed in a base entry before encoding (e.g. a per-object vocabulary namespace) are preserved. The example below uses the MARS vocabulary; any application namespace works the same way:

{
  "version": 2,
  "base": [
    {
      "mars": { "class": "od", "type": "fc", "grid": "O1280", "param": "2t", "levtype": "sfc" },
      "_reserved_": {
        "tensor": { "ndim": 2, "shape": [721, 1440], "strides": [1440, 1], "dtype": "float64" }
      }
    },
    {
      "mars": { "class": "od", "type": "fc", "grid": "O1280", "param": "lnsp", "levtype": "ml" },
      "_reserved_": {
        "tensor": { "ndim": 1, "shape": [137], "strides": [1], "dtype": "float64" }
      }
    }
  ]
}

Each entry is fully self-contained — all keys for that object appear in its entry. There is no separate “common” section for shared keys. If you need to extract commonalities (e.g. for display), use the compute_common() utility in software after decoding.

Note: base describes the collection of objects at the message level. Individual tensor encoding details (encoding pipeline, hash) remain in each object’s own DataObjectDescriptor. The DataObjectDescriptor.params field is reserved for encoding parameters only — it does not carry application metadata.

Practical Guidance

• Prefer integers for numeric identifiers (paramId, date, run_id).
• Use text strings for classification codes even if they happen to be numeric-looking — consistency with your chosen vocabulary is more important than type optimisation.
• Use nested maps for namespaced keys (e.g., "mars": {...}, "bids": {...}, "dicom": {...}).
• Keep individual values small. Avoid storing large arrays (e.g., grid coordinates) in metadata — they belong in data objects.

See Also

• CBOR Metadata Schema — field-level reference for all CBOR structures
• Metadata Concepts — how global and per-object metadata relate
• Vocabularies — example application-layer vocabularies used with Tensogram
• GRIB MARS Key Mapping — how GRIB keys are mapped during import

Data Types

The dtype field in an object descriptor names the element type of the tensor. It is stored as a lowercase text string in CBOR.

Type Table

*bitmask returns 0 from byte_width() — see the edge case note below.

Byte Order

The byte_order field in the payload descriptor specifies whether multi-byte elements are stored in big-endian ("big") or little-endian ("little") order. This applies to the stored payload bytes after encoding.

Single-byte types (int8, uint8, bitmask) are unaffected by byte order.

Bitmask Edge Case

Dtype::Bitmask is for packing boolean or categorical data sub-byte. The payload size is ceil(num_elements / 8) bytes. The byte_width() method returns 0 as a sentinel; callers that need the actual payload size must compute it:

#![allow(unused)]
fn main() {
let payload_bytes = if dtype == Dtype::Bitmask {
    (num_elements + 7) / 8
} else {
    num_elements * dtype.byte_width()
};
}

Choosing a dtype

Quick Start

This page walks you through encoding and decoding a real tensor — a 2D temperature field — in about 20 lines of Rust.

Installation

Rust

cargo add tensogram

Or add it to your Cargo.toml manually:

[dependencies]
tensogram = "0.15"

Optional features:

All compression codecs (szip, zstd, lz4, blosc2, zfp, sz3) and multi-threading are enabled by default.

cargo add tensogram --features mmap,async,remote

Python

pip install tensogram

With xarray and Zarr backends:

pip install tensogram[all]      # everything
pip install tensogram[xarray]   # xarray backend only
pip install tensogram[zarr]     # Zarr v3 store only

CLI

cargo install tensogram-cli

Encode a 2-D Float Field

This example encodes a 100×200 float32 grid — representative of many scientific 2-D fields (temperature, pressure, intensity, density, …).

use std::collections::BTreeMap;
use tensogram::{
    encode, decode, GlobalMetadata, DataObjectDescriptor,
    ByteOrder, Dtype, EncodeOptions, DecodeOptions,
};

fn main() {
    // 1. Make some synthetic data: 100×200 float32 grid
    //    In production, this would come from your model output, sensor,
    //    or upstream pipeline.
    let shape = vec![100u64, 200];
    let strides = vec![200u64, 1]; // C-contiguous (row-major)
    let num_elements = 100 * 200;
    let data: Vec<u8> = (0..num_elements)
        .flat_map(|i| (273.15f32 + (i as f32 / 100.0)).to_be_bytes())
        .collect();

    // 2. Describe the tensor
    let global = GlobalMetadata {
        version: 2,
        ..Default::default()
    };

    let desc = DataObjectDescriptor {
        obj_type: "ntensor".to_string(),
        ndim: 2,
        shape,
        strides,
        dtype: Dtype::Float32,
        byte_order: ByteOrder::Big,
        encoding: "none".to_string(),
        filter: "none".to_string(),
        compression: "none".to_string(),
        params: BTreeMap::new(),
        hash: None, // hash is added automatically by EncodeOptions::default()
    };

    // 3. Encode — produces a self-contained message
    let message = encode(&global, &[(&desc, &data)], &EncodeOptions::default()).unwrap();

    println!("Encoded {} bytes", message.len());

    // 4. Decode it back
    let (meta, objects) = decode(&message, &DecodeOptions::default()).unwrap();

    println!(
        "Decoded: {} objects, shape {:?}, dtype {}",
        objects.len(),
        objects[0].0.shape,
        objects[0].0.dtype,
    );
    assert_eq!(objects[0].1, data);
}

Add Application Metadata

Real messages need application-layer metadata so downstream tools know what the data represents. Per-object metadata goes into the base array — one entry per data object — and is organised under a namespace key so that multiple vocabularies can coexist.

The example below uses ECMWF’s MARS vocabulary for concreteness. The same mechanism works with any vocabulary: CF conventions ("cf"), BIDS ("bids"), DICOM ("dicom"), or your own ("product", "experiment", "device", …).

#![allow(unused)]
fn main() {
use ciborium::Value;

// Build a "mars" namespace for the object — one concrete vocabulary example.
// You can just as easily use "bids", "dicom", "product", or any custom name.
let mars_map = vec![
    (Value::Text("class".into()), Value::Text("od".into())),
    (Value::Text("date".into()),  Value::Text("20260401".into())),
    (Value::Text("step".into()),  Value::Integer(6.into())),
    (Value::Text("type".into()),  Value::Text("fc".into())),
    (Value::Text("param".into()), Value::Text("2t".into())),
];

let mut entry = BTreeMap::new();
entry.insert("mars".to_string(), Value::Map(mars_map));

let global = GlobalMetadata {
    version: 2,
    base: vec![entry], // one entry per data object
    ..Default::default()
};

let desc = DataObjectDescriptor {
    obj_type: "ntensor".to_string(),
    ndim: 2,
    shape: vec![100, 200],
    strides: vec![200, 1],
    dtype: Dtype::Float32,
    byte_order: ByteOrder::Big,
    encoding: "none".to_string(),
    filter: "none".to_string(),
    compression: "none".to_string(),
    params: BTreeMap::new(),
    hash: None,
};
}

What’s Next?

• Use simple_packing to reduce payload size by 4-8x
• Use the File API to append many messages to a .tgm file
• Use the CLI to inspect files without writing any code

Vocabularies

Tensogram is vocabulary-agnostic: the library never interprets metadata keys. The same message can carry any combination of application-defined namespaces alongside the auto-populated library-reserved keys. This page collects example vocabularies that have been (or could naturally be) used with Tensogram, so you can pick a convention that matches your domain — or invent your own.

How metadata is structured

A Tensogram message’s per-object metadata lives in base[i], a BTreeMap<String, ciborium::Value>. By convention, each application vocabulary sits under its own top-level namespace key so that multiple vocabularies can coexist without collision:

{
  "version": 2,
  "base": [{
    "mars":   { "class": "od", "param": "2t" },
    "cf":     { "standard_name": "air_temperature", "units": "K" },
    "custom": { "experiment": "run-042" }
  }]
}

All three namespaces above are valid, visible to tooling, and survive round-trip. The library never reads or validates their contents.

Example vocabularies

MARS (ECMWF, weather forecasting)

Used internally at ECMWF and by downstream consumers of ECMWF’s MARS archive. Keys describe the operational provenance of a forecast field: class, stream, type, parameter, level, date/time, step, etc.

{
  "mars": {
    "class": "od", "stream": "oper", "type": "fc",
    "date": "20260401", "time": "1200", "step": 6,
    "param": "2t", "levtype": "sfc"
  }
}

The GRIB importer (tensogram convert-grib) automatically populates this namespace from GRIB MARS keys. See MARS Key Mapping for the full key list.

CF Conventions (climate, ocean, atmospheric)

CF Conventions are the standard attribute vocabulary for climate and forecast data in NetCDF. The NetCDF importer (tensogram convert-netcdf --cf) lifts the CF allow-list into a "cf" sub-map. See NetCDF CF Metadata Mapping.

{
  "cf": {
    "standard_name": "air_temperature",
    "long_name": "2 metre temperature",
    "units": "K",
    "cell_methods": "time: mean"
  }
}

BIDS (neuroimaging)

The Brain Imaging Data Structure organises neuroimaging datasets with entity-level metadata. A natural fit for Tensogram messages carrying fMRI, dMRI, or EEG tensors.

{
  "bids": {
    "subject": "sub-01", "session": "ses-01",
    "task": "rest", "run": 1, "acq": "hires"
  }
}

DICOM (medical imaging)

DICOM tags are the standard descriptors for medical imaging studies. They can be mapped into a "dicom" namespace for use with Tensogram messages carrying imaging volumes, time-series, or segmentation masks.

{
  "dicom": {
    "Modality": "MR", "SeriesDescription": "T2_FLAIR",
    "SliceThickness": 1.0, "RepetitionTime": 8000
  }
}

Zarr attributes (generic)

Zarr v3 attribute maps are generic key-value stores. When using the Zarr backend (tensogram-zarr), group-level and array-level attributes are surfaced through _extra_ and per-array descriptor params.

Custom namespaces

For any domain that does not have an established vocabulary, or when a pipeline wants to carry bespoke fields alongside a standard namespace, invent your own:

{
  "experiment": {
    "id": "run-042",
    "operator": "alice",
    "hypothesis": "beam stability",
    "started_at": "2026-04-18T10:30:00Z"
  }
}

Suggested conventions for custom namespaces:

• Use a short, lowercase namespace key ("product", "instrument", "run", "experiment", "device").
• Group related fields under a single namespace rather than scattering them at the top level of base[i].
• Prefer ISO 8601 timestamps, SI units in units fields, and UTF-8 text for identifiers.
• Document your namespace schema somewhere versioned (a README, a JSON schema, a wiki page) so downstream consumers can interpret it consistently.

Multiple vocabularies in one message

You can freely mix vocabularies in the same base[i] entry — the library preserves all of them:

{
  "base": [{
    "mars":       { "param": "2t", "levtype": "sfc" },
    "cf":         { "standard_name": "air_temperature", "units": "K" },
    "provenance": { "pipeline_id": "pp-17", "stage": "post-process" }
  }]
}

This lets one team’s producers emit messages that are simultaneously interpretable by tools expecting MARS, CF-aware tooling, and an internal provenance tracker.

Looking up keys

The dotted-path helpers exposed by each binding vary. The CLI, the C FFI (tgm_metadata_get_string / _get_int / _get_float), the C++ wrapper (metadata::get_string / get_int / get_float), and the TypeScript package (getMetaKey) all accept a full dotted path. The Rust crate and the Python package do not expose a dotted-path helper at this time; use direct nested access instead.

TypeScript — dotted path

import { getMetaKey } from '@ecmwf/tensogram';

const param   = getMetaKey(meta, 'mars.param');
const subject = getMetaKey(meta, 'bids.subject');

CLI — dotted path

# Filter messages on a namespaced key
tensogram ls data.tgm -w "mars.param=2t/10u"
tensogram ls data.tgm -w "bids.subject=sub-01"

# Print specific keys
tensogram get -p "cf.standard_name,cf.units" data.tgm

Python — dict-style nested access

# Metadata.__getitem__ does a top-level search across base[i] (skipping
# _reserved_) and falls back to the message-level _extra_ map. The returned
# value is a plain Python dict, so the next lookup is standard dict access.
param   = meta["mars"]["param"]
subject = meta["bids"]["subject"]

# meta.base[i], meta.reserved, and meta.extra are also available directly
# if you want the raw per-object / reserved / extra dicts.
first_base = meta.base[0]

Rust — pattern-match on ciborium::Value

#![allow(unused)]
fn main() {
use ciborium::Value;
use tensogram::GlobalMetadata;

// `meta.base` is `Vec<BTreeMap<String, Value>>`. Find the namespace on
// the first-matching base entry, then pull a text field from the nested
// map. Falls back to `meta.extra` for message-level annotations.
fn get_text<'a>(meta: &'a GlobalMetadata,
                namespace: &str, field: &str) -> Option<&'a str> {
    let pull = |map: &'a [(Value, Value)]| -> Option<&'a str> {
        map.iter().find_map(|(k, v)| match (k, v) {
            (Value::Text(k), Value::Text(v)) if k == field => Some(v.as_str()),
            _ => None,
        })
    };
    for entry in &meta.base {
        if let Some(Value::Map(items)) = entry.get(namespace)
            && let Some(val) = pull(items)
        {
            return Some(val);
        }
    }
    if let Some(Value::Map(items)) = meta.extra.get(namespace) {
        return pull(items);
    }
    None
}

let param = get_text(&meta, "mars", "param");
}

Tensogram keeps the Rust surface small on purpose. If your pipeline needs dotted-path lookup in Rust, wrap the snippet above in a helper of your own, or call out to the CLI.

Lookup semantics (all bindings that support dotted paths)

First match across base[0], base[1], … (skipping _reserved_ within each entry), then fall back to the message-level _extra_ map. An explicit _extra_.key (or extra.key) prefix bypasses the base search.

See also

• Metadata concepts — how the base, _reserved_, and _extra_ sections fit together.
• CBOR Metadata Schema — field-level reference.
• Metadata Value Types — which CBOR types are allowed inside metadata.
• GRIB MARS Key Mapping — what the GRIB importer produces.
• NetCDF CF Metadata Mapping — what the NetCDF importer produces with --cf.

Jupyter Notebook Walk-through

The examples/jupyter/ directory carries a curated set of narrative notebooks that introduce Tensogram interactively, with live visualisations. Unlike the flat .py examples under examples/python/ — which are minimal reference snippets for copy-paste — the notebooks are for learning.

Every notebook is executed end-to-end on every PR by the notebooks CI job, so they cannot rot.

The five journeys

Running the notebooks locally

Option 1 — uv pip install (recommended)

# Build the Python bindings with GRIB and NetCDF support.
# Requires libeccodes + libnetcdf installed at the OS level.
uv venv .venv --python 3.13
source .venv/bin/activate
uv pip install maturin
cd python/bindings
maturin develop --features grib,netcdf
cd ../..

# Install notebook-only dependencies + the xarray backend.
uv pip install -e examples/jupyter

# Launch JupyterLab.
jupyter lab examples/jupyter/

Option 2 — conda env create

conda env create -f examples/jupyter/environment.yml
conda activate tensogram-jupyter
jupyter lab examples/jupyter/

Option 3 — Binder / Colab

Launch badges in the notebook directory’s README.md — zero local install.

OS-level dependencies

Notebooks 03 (GRIB) and 04 (NetCDF) need C libraries installed at the operating system level. They are not Python packages.

The official PyPI wheels (pip install tensogram) do not ship GRIB / NetCDF support: the manylinux_2_28 base image lacks the C libraries. If you try to call tensogram.convert_grib(...) on a wheel without the feature, you get a clean RuntimeError("tensogram was built without GRIB support...") that points you at this page.

To enable the feature, rebuild from source:

git clone https://github.com/ecmwf/tensogram
cd tensogram/python/bindings
maturin develop --features grib,netcdf

Running the notebooks in CI

The repository runs the notebooks end-to-end on every PR via a dedicated notebooks job. The gate is:

pytest --nbval-lax examples/jupyter/ -v

--nbval-lax executes every cell in every notebook and fails the build on any exception. Cell outputs are not compared — we commit the notebooks with empty outputs (enforced by the python/tests/test_jupyter_structure.py guard).

Output hygiene

Committed notebooks must have empty cell outputs. Install the nbstripout pre-commit hook once:

uv pip install nbstripout
nbstripout --install

With the hook installed, git commit automatically strips outputs.

Adding a new notebook

1. Copy an existing .ipynb as a template.
2. First cell must be a markdown license banner mentioning “ECMWF” or “Apache”.
3. Last cell must be a “Where to go next” markdown pointer.
4. If you import matplotlib, call matplotlib.use("Agg") before the first import matplotlib.pyplot.
5. Update EXPECTED_NOTEBOOKS in python/tests/test_jupyter_structure.py.
6. Link it from examples/jupyter/README.md and this guide page.
7. Run pytest --nbval-lax examples/jupyter/ locally before committing.

Encoding Data

This page covers the encode() function and EncodeOptions in detail.

Function Signature

#![allow(unused)]
fn main() {
pub fn encode(
    global_metadata: &GlobalMetadata,
    descriptors: &[(&DataObjectDescriptor, &[u8])],
    options: &EncodeOptions,
) -> Result<Vec<u8>>
}

• global_metadata — reference to message-level metadata (version, base entries, _extra_ fields)
• descriptors — a slice of (descriptor, data) pairs, one per object
• options — controls hash algorithm and compression backend selection (the emit_preceders field is reserved for future buffered-mode support; preceders are currently only emitted via StreamingEncoder::write_preceder)

Returns a complete, self-contained message as a Vec<u8>.

EncodeOptions

#![allow(unused)]
fn main() {
pub struct EncodeOptions {
    /// Hash algorithm to use. None disables hashing entirely.
    pub hash_algorithm: Option<HashAlgorithm>,
    /// Reserved — buffered `encode()` rejects `true`. Use
    /// `StreamingEncoder::write_preceder()` instead.
    pub emit_preceders: bool,
    /// Which backend to use for szip / zstd when both FFI and pure-Rust
    /// implementations are compiled in.
    pub compression_backend: CompressionBackend,
}

impl Default for EncodeOptions {
    fn default() -> Self {
        Self {
            hash_algorithm: Some(HashAlgorithm::Xxh3),
            emit_preceders: false,
            compression_backend: CompressionBackend::default(),
        }
    }
}
}

The default applies xxh3 hashing to every object payload. Use None to skip hashing:

#![allow(unused)]
fn main() {
let options = EncodeOptions {
    hash_algorithm: None,
    ..Default::default()
};
}

What Encode Does

For each object, in order:

1. Validate — checks that each pair has a descriptor and corresponding data
2. Run the encoding pipeline — applies encoding, filter, compression from the object’s DataObjectDescriptor
3. Hash — if hash_algorithm is set, computes and stores the hash in the descriptor
4. Serialize CBOR — encodes the GlobalMetadata and all DataObjectDescriptors to canonical CBOR
5. Frame — assembles preamble, header frames (metadata/index/hash), data object frames, and postamble

Encoding with Simple Packing

To use simple_packing, you need to compute the quantization parameters first, then put them in the DataObjectDescriptor:

#![allow(unused)]
fn main() {
use tensogram_encodings::simple_packing;
use ciborium::Value;

// Your original values as f64 (simple_packing always works on f64).
// source_data might be a temperature grid, pressure field, intensity
// image, or any other bounded-range scalar field.
let values: Vec<f64> = source_data.iter().map(|&x| x as f64).collect();

// Compute quantization parameters for 16 bits per value
let params = simple_packing::compute_params(&values, 16, 0)?;

// Put the parameters into the descriptor
let mut packing_params = BTreeMap::new();
packing_params.insert("reference_value".into(),
    Value::Float(params.reference_value));
packing_params.insert("binary_scale_factor".into(),
    Value::Integer((params.binary_scale_factor as i64).into()));
packing_params.insert("decimal_scale_factor".into(),
    Value::Integer((params.decimal_scale_factor as i64).into()));
packing_params.insert("bits_per_value".into(),
    Value::Integer((params.bits_per_value as i64).into()));

let desc = DataObjectDescriptor {
    obj_type: "ntensor".to_string(),
    ndim: 2,
    shape: vec![100, 200],
    strides: vec![200, 1],
    dtype: Dtype::Float64,
    byte_order: ByteOrder::Big,
    encoding: "simple_packing".to_string(),
    filter: "none".to_string(),
    compression: "none".to_string(),
    params: packing_params,
    hash: None,
};
}

Then encode as normal, passing the original raw bytes (as f64 bytes):

#![allow(unused)]
fn main() {
let raw: Vec<u8> = values.iter().flat_map(|v| v.to_ne_bytes()).collect();

let global = GlobalMetadata { version: 2, ..Default::default() };
let message = encode(&global, &[(&desc, &raw)], &EncodeOptions::default())?;
}

The encoder applies simple_packing internally. The payload stored in the message is the packed bits, not the original f64 bytes.

Encoding Multiple Objects

Pass multiple (descriptor, data) pairs:

#![allow(unused)]
fn main() {
let global = GlobalMetadata { version: 2, ..Default::default() };

let message = encode(
    &global,
    &[(&spectrum_desc, &spectrum_data), (&mask_desc, &land_mask_data)],
    &EncodeOptions::default(),
)?;
}

Each descriptor independently specifies its own encoding, compression, dtype, and byte order. The encoder processes each pair in sequence.

Error Conditions

Pre-Encoded Data API (Advanced)

When to use this API

The encode_pre_encoded API is for advanced callers whose data is already encoded by an external pipeline (e.g., a GPU kernel that emits packed bytes, or a streaming receiver passing payloads through). It bypasses Tensogram’s internal encoding pipeline and uses the supplied bytes verbatim.

Do NOT use this API for ordinary encoding. Use encode() instead.

⚠️ The bit-vs-byte trap

WARNING: When using compression="szip", the szip_block_offsets parameter contains bit offsets, not byte offsets. The first offset must be 0 and every offset must satisfy offset <= encoded_bytes_len * 8. This matches the libaec/szip wire format. See cbor-metadata.md for the format reference.

Getting this wrong is the #1 caller mistake. Tensogram validates the offsets structurally (monotonicity, bounds) but cannot detect a byte-instead-of-bit mistake until decode_range fails.

API surface

Rust

#![allow(unused)]
fn main() {
pub fn encode_pre_encoded(
    metadata: &GlobalMetadata,
    descriptors_and_data: &[(&DataObjectDescriptor, &[u8])],
    options: &EncodeOptions,
) -> Result<Vec<u8>, TensogramError>
}

Python

import tensogram

msg: bytes = tensogram.encode_pre_encoded(
    global_meta_dict={"version": 2},
    descriptors_and_data=[(descriptor_dict, raw_bytes)],
    hash="xxh3",
)

C

tgm_error tgm_encode_pre_encoded(
    const char *metadata_json,
    const uint8_t *const *data_ptrs,
    const size_t *data_lens,
    size_t num_objects,
    const char *hash_algo,
    tgm_bytes_t *out
);

C++

std::vector<std::uint8_t> tensogram::encode_pre_encoded(
    const std::string& metadata_json,
    const std::vector<std::pair<const std::uint8_t*, std::size_t>>& objects,
    const encode_options& opts = {}
);

Hash semantics

The library always recomputes the hash of the pre-encoded bytes using the algorithm specified in EncodeOptions.hash_algorithm (default xxh3). Any hash the caller stored on the descriptor is silently overwritten. This guarantees the wire format invariant descriptor.hash == hash_algo(bytes) always holds.

Provenance semantics

The encoded message is byte-format-indistinguishable from one produced by encode(). The decoder cannot tell which API produced it. The provenance fields _reserved_.encoder.name, _reserved_.time, and _reserved_.uuid are populated identically.

Self-consistency checks

Before encoding, the library validates:

1. Caller has not set EncodeOptions.emit_preceders (rejected).
2. Caller has not put _reserved_ in their metadata (rejected).
3. Each descriptor passes the standard validate_object checks.
4. If compression="szip" and szip_block_offsets is supplied:
   • It’s a CBOR Array of u64.
   • First offset is 0.
   • Strictly monotonically increasing.
   • All bit offsets <= bytes_len * 8.
5. If szip_block_offsets is supplied but compression != "szip", rejected.

These are structural checks only. The library does NOT trial-decode the bytes to verify they actually decode correctly.

Limitation: encoding=“none” size check

When encoding="none", the validate_object check enforces payload_len == shape_product * dtype_byte_width. This means you cannot pass compression-only payloads (e.g., zstd-compressed raw bytes) with encoding="none" because the compressed size will not match the expected raw size. Wrap such payloads in at least simple_packing or another encoding.

Worked example: simple_packing + szip with decode_range

#![allow(unused)]
fn main() {
use tensogram::{
    encode_pre_encoded, DataObjectDescriptor, EncodeOptions,
    GlobalMetadata, ByteOrder, Dtype,
};
use std::collections::BTreeMap;
use ciborium::Value;

// Pre-encoded bytes from a GPU kernel + szip block offsets in BITS
let pre_encoded_bytes: Vec<u8> = /* from GPU */;
let szip_offsets_bits: Vec<u64> = vec![0, 8192, 16384, /* ... */];

let mut params: BTreeMap<String, ciborium::Value> = BTreeMap::new();
params.insert("bits_per_value".into(), Value::Integer(24u64.into()));
params.insert("reference_value".into(), Value::Float(0.0));
params.insert("binary_scale_factor".into(), Value::Integer((-10i64).into()));
params.insert("decimal_scale_factor".into(), Value::Integer(0i64.into()));
params.insert("szip_rsi".into(), Value::Integer(128i64.into()));
params.insert("szip_block_size".into(), Value::Integer(16i64.into()));
params.insert("szip_flags".into(), Value::Integer(8i64.into()));
params.insert("szip_block_offsets".into(),
    Value::Array(szip_offsets_bits.into_iter()
        .map(|o| Value::Integer(o.into()))
        .collect()));

let desc = DataObjectDescriptor {
    obj_type: "ntensor".into(),
    ndim: 2,
    shape: vec![1024, 1024],
    strides: vec![1024, 1],
    dtype: Dtype::Float32,
    byte_order: ByteOrder::Big,
    encoding: "simple_packing".into(),
    filter: "none".into(),
    compression: "szip".into(),
    params,
    hash: None,
};

let msg = encode_pre_encoded(
    &GlobalMetadata::default(),
    &[(&desc, &pre_encoded_bytes)],
    &EncodeOptions::default(),
)?;

// decode_range works because szip_block_offsets is present.
}

How it works

flowchart TD
    subgraph pre["encode_pre_encoded path"]
        A[Caller bytes] --> B[validate_object]
        B --> C[validate_szip_block_offsets]
        C --> D[Recompute hash]
    end

    subgraph normal["encode path"]
        G[Caller bytes] --> H[Run encoding pipeline]
        H --> D
    end

    D --> E[Wrap in CBOR framing]
    E --> F[Wire message]

The pre-encoded path skips the pipeline entirely. The wire format is identical.

Byte order

When using encoding="none", the caller’s bytes are stored verbatim — the library does NOT validate or flip byte order on encode. The bytes must be in the byte order declared in the descriptor’s byte_order field.

For example, if byte_order="big" and encoding="none", the caller must provide big-endian bytes.

On decode, the library automatically converts to native byte order by default (native_byte_order=true). Callers can use from_ne_bytes() or data_as<T>() directly without worrying about which byte order was used on the wire. Set native_byte_order=false to get the raw wire-order bytes.

Streaming API

StreamingEncoder::write_object_pre_encoded() is the streaming counterpart of encode_pre_encoded(). It writes a single pre-encoded object to the stream. It can be interleaved freely with write_object() (normal encode) calls.

Rust

#![allow(unused)]
fn main() {
let mut enc = StreamingEncoder::new(output, &metadata, &options)?;
enc.write_object_pre_encoded(&descriptor, &pre_encoded_bytes)?;
enc.finish()?;
}

Python

enc = tensogram.StreamingEncoder({"version": 2})
enc.write_object_pre_encoded(descriptor_dict, raw_bytes)
msg = enc.finish()

C++

tensogram::streaming_encoder enc(path, metadata_json);
enc.write_object_pre_encoded(descriptor_json, data_ptr, data_len);
enc.finish();

Error reference

encode_pre_encoded can raise the following errors:

Strides convention

The library treats strides as opaque metadata — it only validates that strides.len() == shape.len(). The convention differs between language bindings:

• Rust tests use element strides (e.g., [1] for 1D, [5, 1] for shape [4, 5])
• C++ tests use byte strides (e.g., [4] for float32, [12, 4] for shape [2, 3] float32)

Both conventions work correctly since the library does not interpret stride values.

Cross-references

• Encoding — the normal encode() API
• Decoding — decode_range requirements for partial reads
• Compression — szip details
• CBOR metadata — wire format reference

Decoding Data

Tensogram provides four decode functions for different use cases. Choose the one that does the least work for your situation — they are all zero-copy on the metadata path.

The DecodedObject Type

Before diving in, it helps to know the common return type:

#![allow(unused)]
fn main() {
type DecodedObject = (DataObjectDescriptor, Vec<u8>);
}

A DecodedObject is a tuple of the object’s descriptor (shape, dtype, encoding info, etc.) and the decoded raw bytes. You will see this pattern throughout the decode API.

Four Decode Functions

decode — full message

#![allow(unused)]
fn main() {
pub fn decode(
    message: &[u8],
    options: &DecodeOptions,
) -> Result<(GlobalMetadata, Vec<(DataObjectDescriptor, Vec<u8>)>)>
}

Decodes all objects. Returns the global metadata and a vector of DecodedObject tuples — one per object, with raw bytes in the logical dtype after de-quantization.

#![allow(unused)]
fn main() {
let (meta, objects) = decode(&message, &DecodeOptions::default())?;

// Each element is (DataObjectDescriptor, Vec<u8>)
let (ref desc, ref data) = objects[0];
println!("shape: {:?}, dtype: {}, bytes: {}", desc.shape, desc.dtype, data.len());
}

decode_metadata — metadata only

#![allow(unused)]
fn main() {
pub fn decode_metadata(message: &[u8]) -> Result<GlobalMetadata>
}

Reads only the CBOR section. Does not touch any payload bytes. Use this for filtering and listing.

#![allow(unused)]
fn main() {
let meta = decode_metadata(&message)?;
println!("version: {}", meta.version);
}

decode_object — single object by index

#![allow(unused)]
fn main() {
pub fn decode_object(
    message: &[u8],
    index: usize,
    options: &DecodeOptions,
) -> Result<(GlobalMetadata, DataObjectDescriptor, Vec<u8>)>
}

Decodes one object without reading the others. Uses the binary header’s offset table to seek directly to the right payload. O(1) seek regardless of how many objects the message contains.

Returns the global metadata, the object’s descriptor, and the decoded bytes as a three-element tuple.

#![allow(unused)]
fn main() {
// Decode only the second object (index 1)
let (meta, descriptor, payload) = decode_object(&message, 1, &DecodeOptions::default())?;
println!("shape: {:?}, dtype: {}", descriptor.shape, descriptor.dtype);
}

Edge case: If index >= num_objects, returns TensogramError::Object("index out of range").

decode_range — partial sub-tensor

#![allow(unused)]
fn main() {
pub fn decode_range(
    message: &[u8],
    object_index: usize,
    ranges: &[(u64, u64)],  // (offset, count) in flattened element order
    options: &DecodeOptions,
) -> Result<(DataObjectDescriptor, Vec<Vec<u8>>)>
}

Decodes one or more contiguous slices of elements from an object. Each (offset, count) pair in ranges selects a span of elements along the flattened dimension; the function returns one byte vector per range by default. This split-result design avoids an unnecessary copy when the caller needs the ranges individually (e.g. to feed separate array slices).

Rust — split results (default)

#![allow(unused)]
fn main() {
// Two separate ranges from object 0
let (desc, parts) = decode_range(
    &message, 0,
    &[(100, 50), (300, 25)],
    &DecodeOptions::default(),
)?;
assert_eq!(parts.len(), 2);           // one Vec<u8> per range
println!("first  range bytes: {}", parts[0].len());
println!("second range bytes: {}", parts[1].len());
}

Rust — joined result

If you prefer a single contiguous buffer, flatten the results:

#![allow(unused)]
fn main() {
let joined: Vec<u8> = parts.into_iter().flatten().collect();
}

Python — split results (default, join=False)

import tensogram

parts = tensogram.decode_range(buf, object_index=0, ranges=[(100, 50), (300, 25)])
# parts is a list of numpy arrays, one per range
print(len(parts))        # 2
print(parts[0].shape)    # (50,)

Python — joined result (join=True)

arr = tensogram.decode_range(buf, object_index=0, ranges=[(100, 50), (300, 25)], join=True)
# arr is a single flat numpy array with all ranges concatenated
print(arr.shape)          # (75,)

N-dimensional slicing: The xarray backend maps N-dimensional slice notation (e.g. ds["temperature"].sel(lat=slice(10, 20), lon=slice(30, 40))) into the (offset, count) pairs that decode_range expects, so you rarely need to compute flattened offsets by hand when working through xarray.

Pre-encoded messages: Messages produced via encode_pre_encoded only support decode_range if the caller provided the necessary bit-precise szip_block_offsets (see Pre-encoded Payloads).

Edge case: decode_range works with all encoding+compression combinations that support random access: uncompressed data, simple_packing (bit extraction), szip (RSI block seeking), blosc2 (chunk access), and zfp fixed-rate mode. It returns an error for the shuffle filter (byte rearrangement breaks contiguous sample ranges) and for stream compressors (zstd, lz4, sz3) that don’t support partial decode.

DecodeOptions

#![allow(unused)]
fn main() {
pub struct DecodeOptions {
    /// If true, verify the hash of each decoded payload.
    pub verify_hash: bool,
    /// When true (the default), decoded payloads are converted to the
    /// caller's native byte order. Set to false to receive bytes in the
    /// message's declared wire byte order.
    pub native_byte_order: bool,
    /// Which backend to use for szip / zstd when both FFI and pure-Rust
    /// implementations are compiled in.
    pub compression_backend: CompressionBackend,
}

impl Default for DecodeOptions {
    fn default() -> Self {
        Self {
            verify_hash: false,
            native_byte_order: true,
            compression_backend: CompressionBackend::default(),
        }
    }
}
}

Native byte order (default)

By default, all decoded data is returned in the caller’s native byte order — the library handles any necessary byte-swapping automatically. You never need to check byte_order or call .byteswap():

#![allow(unused)]
fn main() {
let (_, objects) = decode(&message, &DecodeOptions::default())?;
let floats: Vec<f32> = objects[0].1
    .chunks_exact(4)
    .map(|c| f32::from_ne_bytes(c.try_into().unwrap()))
    .collect();
}

In Python, numpy arrays are always directly usable:

_, objects = tensogram.decode(msg)
arr = objects[0][1]   # numpy array — values are correct, no byteswap needed

This applies to all decode functions (decode, decode_object, decode_range), all encodings (none, simple_packing), all compression codecs, and all language bindings (Rust, Python, C, C++).

Wire byte order (opt-in)

Set native_byte_order: false to receive the raw bytes in the message’s declared wire byte order. This is useful for zero-copy forwarding or when you need the exact on-wire representation:

#![allow(unused)]
fn main() {
let opts = DecodeOptions { native_byte_order: false, ..Default::default() };
let (_, objects) = decode(&message, &opts)?;
// objects[0].1 is in the descriptor's declared byte_order (e.g. big-endian)
}

Hash verification

Hash verification is opt-in. Enable it when data integrity is critical:

#![allow(unused)]
fn main() {
let options = DecodeOptions { verify_hash: true, ..Default::default() };
let result = decode(&message, &options);
// Returns Err(TensogramError::HashMismatch { expected, actual }) if corrupted
}

Edge case: If the descriptor has no hash (i.e. the message was encoded with hash_algorithm: None), verify_hash: true silently skips verification for that object. No error is returned.

Working with the Decoded Bytes

Decoded bytes are in native byte order (with the default DecodeOptions). Cast them as native:

#![allow(unused)]
fn main() {
// float32 object → use from_ne_bytes
let floats: Vec<f32> = data
    .chunks_exact(4)
    .map(|c| f32::from_ne_bytes(c.try_into().unwrap()))
    .collect();
}

For simple_packing decoded data, the output is always f64 bytes (8 bytes per element), regardless of the original dtype stored in the descriptor:

#![allow(unused)]
fn main() {
// simple_packing always decodes to f64, in native byte order
let values: Vec<f64> = data
    .chunks_exact(8)
    .map(|c| f64::from_ne_bytes(c.try_into().unwrap()))
    .collect();
}

Scanning for Messages First

If you’re working with a buffer that might contain multiple messages (e.g. a .tgm file loaded into memory), scan it first to get message boundaries:

#![allow(unused)]
fn main() {
let offsets = scan(&big_buffer); // Vec<(usize, usize)> = (start, length)

for (start, len) in offsets {
    let msg = &big_buffer[start..start + len];
    let meta = decode_metadata(msg)?;
    println!("version: {}", meta.version);
}
}

The scan function is tolerant of corruption — it skips invalid regions and continues looking for the next valid TENSOGRM marker.

NaN / Inf handling

By default the Tensogram encoder rejects any NaN or ±Inf in float / complex payloads. The encode call fails with TensogramError::Encoding (C FFI: TgmError::Encoding; Python: EncodingError; TypeScript: EncodingError; C++: tensogram::encoding_error) and names the element index, dtype, and a hint that points at the opt-in flags described below.

This chapter walks through the three policies available on encode:

1. Reject (default) — any non-finite input fails the call. Use this when your pipeline guarantees finite values and any NaN / Inf is a bug you want to surface loudly.
2. Allow NaN — NaN values are substituted with 0.0 on the wire and their positions are recorded in a compressed bitmask stored alongside the payload. Decode restores canonical NaN at those positions by default.
3. Allow ±Inf — same as allow_nan but for +∞ and −∞ together (the flag covers both signs; two per-sign bitmasks are written when both kinds appear in the payload).

The mask companion is formally called the NTensorFrame — wire-format type 9, defined in plans/BITMASK_FRAME.md and the wire-format reference.

When to use which policy

Don’t pre-process to a sentinel value when allow_nan / allow_inf does the job — the bitmask is designed to compress aggressively (hybrid Roaring containers by default) and keeps the missing-data semantics visible to the decoder. Sentinel values throw that information away.

Cross-language opt-in

Rust

#![allow(unused)]
fn main() {
use tensogram::{encode, EncodeOptions, GlobalMetadata, DataObjectDescriptor};

let options = EncodeOptions {
    allow_nan: true,
    allow_inf: true,
    ..Default::default()
};
let msg = encode(&meta, &[(&desc, payload_bytes)], &options)?;
}

Python

import numpy as np
import tensogram

data = np.array([1.0, np.nan, 3.0], dtype=np.float64)
msg = tensogram.encode(
    {"version": 2},
    [(desc, data)],
    allow_nan=True,
)
decoded = tensogram.decode(msg)
# decoded.objects[0].data() → [1.0, nan, 3.0]

TypeScript

import { encode, decode } from '@ecmwf/tensogram';

const msg = encode(
    { version: 2 },
    [{ descriptor, data: new Float64Array([1, NaN, 3]) }],
    { allowNan: true },
);
const decoded = decode(msg);

C++

tensogram::encode_options opts;
opts.allow_nan = true;
auto msg = tensogram::encode(metadata_json, objects, opts);

CLI

$ tensogram --allow-nan reshuffle -o out.tgm input.tgm
$ TENSOGRAM_ALLOW_NAN=1 tensogram convert-netcdf data.nc -o data.tgm

Decode-side reconstruction

By default every decode path restores the canonical quiet-NaN / ±Inf bit pattern at every masked position. Opt out (e.g. to inspect the on-disk zero-substituted representation) by passing restore_non_finite=false:

# Get the 0.0-substituted payload without the NaN bits.
raw = tensogram.decode(msg, restore_non_finite=False)
# raw.objects[0].data() → [1.0, 0.0, 3.0]

The advanced decode_with_masks API (Rust + Python) returns both the zero-substituted payload AND the raw decompressed per-kind Vec<bool> masks, so callers can build custom missing-value representations without materialising canonical NaN bytes.

Lossy reconstruction — read this carefully

The masked encode path does not preserve the original NaN payload bits. On decode every masked NaN is restored with the canonical quiet-NaN pattern:

• f32::NAN bits = 0x7FC00000
• f64::NAN bits = 0x7FF8000000000000
• Float16 / bfloat16 use their dtype-native quiet-NaN patterns
• Complex64 / complex128 restore the canonical pattern to both real and imag components

Signalling NaNs, custom payload bits, and mixed real / imag kinds for complex dtypes are therefore flattened to the canonical form through a mask round-trip. If you need bit-exact NaN preservation, pre-encode your payload and use encode_pre_encoded to bypass the substitute-and-mask stage entirely. See plans/BITMASK_FRAME.md §7.1 for the full design rationale.

Mask compression methods

Six methods are available per-kind:

Small masks (uncompressed bit-packed byte count ≤ 128 by default) automatically fall back to none regardless of the requested method — compressing a few bytes costs more than it saves. Set small_mask_threshold_bytes = 0 to disable the auto-fallback.

Set per-kind methods via the matching options:

msg = tensogram.encode(
    meta, [(desc, data)],
    allow_nan=True, allow_inf=True,
    nan_mask_method='rle',
    pos_inf_mask_method='roaring',
    neg_inf_mask_method='roaring',
    small_mask_threshold_bytes=0,
)

Validation

tensogram validate --full cross-checks every NaN / ±Inf in the decoded payload against the frame’s mask companion: masked positions are expected and pass; any NaN / Inf at a non-masked position is reported as NanDetected / InfDetected (see the validator reference).

Files without a mask companion keep the pre-0.17 semantics — any non-finite value in the decoded output is an error.

Migration from pre-0.17

Prior to 0.17 the reject_nan / reject_inf opt-in flags upgraded the NaN check to be pipeline-independent. These flags are removed in 0.17 (breaking change). Rejection is now always on by default; opt in to masked substitution with the replacement flags:

See CHANGELOG.md for the full breaking-change list and upgrade notes.

Working with Files

The TensogramFile struct provides a high-level API for reading and writing .tgm files. It handles lazy scanning, buffered appending, and random access by message index.

Creating a File

#![allow(unused)]
fn main() {
use tensogram::{TensogramFile, EncodeOptions};

let mut file = TensogramFile::create("forecast.tgm")?;
}

This creates (or truncates) the file. No data is written yet.

Appending Messages

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;
use tensogram::{
    GlobalMetadata, DataObjectDescriptor, ByteOrder, Dtype, EncodeOptions,
};

let global = GlobalMetadata { version: 2, ..Default::default() };

let desc = DataObjectDescriptor {
    obj_type: "ntensor".to_string(),
    ndim: 2,
    shape: vec![100, 200],
    strides: vec![200, 1],
    dtype: Dtype::Float32,
    byte_order: ByteOrder::Big,
    encoding: "none".to_string(),
    filter: "none".to_string(),
    compression: "none".to_string(),
    params: BTreeMap::new(),
    hash: None,
};

    file.append(&global, &[(&desc, &data)], &EncodeOptions::default())?;
}

Each append encodes one message and appends it to the end of the file. You can call it as many times as you like — each message is independent and self-describing.

Typical pattern for writing a multi-message file (one message per parameter, run, subject, sample, experiment — whatever your pipeline produces):

#![allow(unused)]
fn main() {
let mut file = TensogramFile::create("output.tgm")?;

for key in ["2t", "10u", "10v", "msl"] {
    let (global, desc, data) = produce_field(key);
    file.append(&global, &[(&desc, &data)], &EncodeOptions::default())?;
}
}

Opening and Counting Messages

#![allow(unused)]
fn main() {
let mut file = TensogramFile::open("forecast.tgm")?;

// Streaming scan happens here (lazily, on first access)
let count = file.message_count()?;
println!("{} messages in file", count);
}

The first access triggers a streaming scan that reads preamble-sized chunks and seeks forward, so it never loads the entire file into memory. After that, every read_message call is a seek + read — no further scanning.

Reading Messages

#![allow(unused)]
fn main() {
use tensogram::{decode, DecodeOptions};

// Read raw bytes of message 3
let raw_bytes = file.read_message(3)?;

// Decode message 3
let (meta, objects) = decode(&raw_bytes, &DecodeOptions::default())?;

// Each element is (DataObjectDescriptor, Vec<u8>)
let (ref desc, ref data) = objects[0];
println!("shape: {:?}, dtype: {}", desc.shape, desc.dtype);
}

Both are O(1) after the initial scan: they seek to the stored offset and read length bytes.

Iterating Over All Messages

#![allow(unused)]
fn main() {
let mut file = TensogramFile::open("forecast.tgm")?;

for raw in file.iter()? {
    let raw = raw?;
    let meta = tensogram::decode_metadata(&raw)?;
    println!("version: {}", meta.version);
}
}

Memory note: For files with many large messages, prefer iterating by index with read_message(i) inside a loop to process one at a time.

Random Access by Index

One of Tensogram’s design goals is O(1) object access. After scanning, any message is reachable in constant time. Within a message, any object is reachable in constant time via the binary header’s offset table:

flowchart TD
    A["file.read_message(42)"]
    B["Message bytes"]
    C["Binary header"]
    D["Seek to payload 2"]
    E["Decode only object 2"]

    A -- "seek + read" --> B
    B --> C
    C -- "lookup offset for object 2" --> D
    D --> E

    style A fill:#388e3c,stroke:#2e7d32,color:#fff
    style E fill:#1565c0,stroke:#0d47a1,color:#fff

File Layout Diagram

forecast.tgm
├── [message 0] — TENSOGRM ... 39277777
├── [message 1] — TENSOGRM ... 39277777
├── [message 2] — TENSOGRM ... 39277777
│   ├── Preamble (24B)
│   ├── Header Metadata Frame (CBOR GlobalMetadata)
│   ├── Header Index Frame (CBOR offsets)
│   ├── Data Object Frame 0 (payload + CBOR descriptor)
│   └── Data Object Frame 1 (payload + CBOR descriptor)
│   └── Postamble (16B)
└── ...

No file-level header, no file-level index. All indexing is per-message, built in-memory at scan time.

Remote Access (optional)

Enable the remote feature to open .tgm files on S3, GCS, Azure, or HTTP with selective range-based reads:

[dependencies]
tensogram = { path = "...", features = ["remote"] }

#![allow(unused)]
fn main() {
use tensogram::{TensogramFile, DecodeOptions};

let mut file = TensogramFile::open_source("s3://bucket/forecast.tgm")?;

// Fetch only the second object from message 0 — no full download
let (meta, desc, data) = file.decode_object(0, 1, &DecodeOptions::default())?;
}

Supports header-indexed and footer-indexed files (read-only) from Rust, Python, xarray, and zarr. See the Remote Access guide for storage options, request budgets, and limitations.

Memory-Mapped I/O (optional)

Enable the mmap feature to use memory-mapped file access:

[dependencies]
tensogram = { path = "...", features = ["mmap"] }

#![allow(unused)]
fn main() {
let mut file = TensogramFile::open_mmap("forecast.tgm")?;

// Scan happens during open_mmap — no lazy scan needed
let count = file.message_count()?;

// Reads from the memory-mapped region (no additional seek)
let raw = file.read_message(0)?;
}

This is useful for large files where you want to avoid per-message seek + read overhead. The file is mapped read-only. All existing decode functions work unchanged.

Async I/O (optional)

Enable the async feature for tokio-based non-blocking file operations:

[dependencies]
tensogram = { path = "...", features = ["async"] }

#![allow(unused)]
fn main() {
let mut file = TensogramFile::open_async("forecast.tgm").await?;

// Read a message without blocking the async runtime
let raw = file.read_message_async(0).await?;

// Decode also runs on a blocking thread (safe for FFI codecs)
let (meta, objects) = file.decode_message_async(0, &opts).await?;
}

All CPU-intensive work (scanning, decoding, FFI calls to compression libraries) runs via tokio::task::spawn_blocking, so it won’t block the async runtime.

Edge Cases

Appending to an Existing File

TensogramFile::create truncates. To append to an existing file, use standard file I/O:

#![allow(unused)]
fn main() {
use std::io::Write;
let mut f = std::fs::OpenOptions::new().append(true).open("forecast.tgm")?;

let global = GlobalMetadata { version: 2, ..Default::default() };
let message = encode(&global, &[(&desc, &data)], &EncodeOptions::default())?;
f.write_all(&message)?;
}

Or open the file with TensogramFile::open and use append() — the append method always writes at the end regardless of how the file was opened.

Corrupted Messages

The scanner skips corrupted messages and continues. A message is considered corrupted if:

• The total_length field points to a location where 39277777 is not present
• The header is truncated

The scanner recovers by advancing one byte and searching for the next TENSOGRM.

Empty Files

message_count() returns 0 for an empty file. read_message(0) returns an error.

Remote Access

Enable the remote feature to open .tgm files on HTTP, S3, GCS, or Azure without downloading the whole file. Individual objects are fetched via targeted range requests.

[dependencies]
tensogram = { path = "...", features = ["remote"] }

Opening a Remote File

#![allow(unused)]
fn main() {
use tensogram::TensogramFile;

// Auto-detect: local path or remote URL
let mut file = TensogramFile::open_source("https://example.com/data.tgm")?;

// S3
let mut file = TensogramFile::open_source("s3://bucket/data.tgm")?;
}

open_source inspects the URL scheme and routes to the remote backend for s3://, s3a://, gs://, az://, azure://, http://, https://. Everything else is treated as a local path.

The Rust open() method is unchanged and always opens a local file. In Python, TensogramFile.open() auto-detects remote URLs.

You can also check whether a string is a remote URL without opening:

#![allow(unused)]
fn main() {
use tensogram::is_remote_url;

assert!(is_remote_url("s3://bucket/file.tgm"));
assert!(!is_remote_url("/local/path/file.tgm"));
}

Storage Options (Credentials, Region, etc.)

Pass an explicit options map for fine-grained control:

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;
use tensogram::TensogramFile;

let mut opts = BTreeMap::new();
opts.insert("aws_access_key_id".to_string(), "AKIA...".to_string());
opts.insert("aws_secret_access_key".to_string(), "...".to_string());
opts.insert("region".to_string(), "eu-west-1".to_string());

let mut file = TensogramFile::open_remote("s3://bucket/data.tgm", &opts)?;
}

When no options are passed, credentials are read from the environment (e.g. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, GOOGLE_APPLICATION_CREDENTIALS).

Python Usage

import tensogram

# Auto-detect remote URL
with tensogram.TensogramFile.open("s3://bucket/data.tgm") as f:
    meta = f.file_decode_metadata(0)
    result = f.file_decode_object(0, 0)
    data = result["data"]  # numpy array

# With explicit storage options
with tensogram.TensogramFile.open_remote(
    "s3://bucket/data.tgm",
    {"region": "eu-west-1"}
) as f:
    print(f.source())   # "s3://bucket/data.tgm"
    print(f.is_remote()) # True

xarray Usage

import xarray as xr

ds = xr.open_dataset(
    "s3://bucket/data.tgm",
    engine="tensogram",
    storage_options={"region": "eu-west-1"},
)

Supported Schemes

All backends are provided by the object_store crate.

Object-Level Access

Three methods provide selective access without downloading full messages:

#![allow(unused)]
fn main() {
use tensogram::DecodeOptions;

// Metadata only — triggers layout discovery on first call, then cached
let meta = file.decode_metadata(0)?;

// Descriptors — reads only the descriptor data needed for each object
let (meta, descriptors) = file.decode_descriptors(0)?;

// Single object by index — fetches only the target object frame
let (meta, desc, data) = file.decode_object(0, 2, &DecodeOptions::default())?;
}

These methods also work on local files, where they read the full message and decode the requested parts.

Request Budget

Header-indexed files (buffered writes)

Footer-indexed files (buffered with known total_length)

Streaming files (total_length=0)

Layout discovery is combined with message scanning for both header-indexed and footer-indexed messages — the library reads the preamble and layout in one GET (header-indexed) or two GETs (footer-indexed suffix read). message_count() uses a lean scan path (24 bytes per preamble). Streaming messages (total_length=0) must be the last message in a multi-message file.

How It Works (Header-Indexed Example)

sequenceDiagram
    participant App
    participant TensogramFile
    participant ObjectStore

    App->>TensogramFile: open_source("s3://bucket/file.tgm")
    TensogramFile->>ObjectStore: HEAD (get file size)
    TensogramFile->>ObjectStore: GET range 0..24 (preamble)
    Note right of TensogramFile: Discover message offsets

    App->>TensogramFile: decode_object(0, 2)
    TensogramFile->>ObjectStore: GET range 24..N (header chunk, up to 256KB)
    Note right of TensogramFile: First access: parse metadata + index, cache layout
    TensogramFile->>ObjectStore: GET range offset..offset+len (object frame 2)
    TensogramFile-->>App: (metadata, descriptor, decoded_bytes)

Checking if a File is Remote

#![allow(unused)]
fn main() {
use tensogram::TensogramFile;

let file = TensogramFile::open_source("s3://bucket/data.tgm")?;
assert!(file.is_remote());
println!("source: {}", file.source()); // "s3://bucket/data.tgm"
}

source() returns the original URL for remote files and the file path for local files.

Error Handling

Remote access can return different TensogramError variants depending on the failure:

All errors are returned as Result. The library avoids panics.

Shared Runtime

Remote I/O uses a process-wide shared tokio runtime (multi-thread, 2 workers) created on first use. All RemoteBackend instances share the same runtime, so TCP connection pools and DNS caches are reused across calls.

The sync bridge adapts to the calling context:

• Not in a tokio runtime (Python, CLI): the shared runtime’s handle drives the future directly — no extra thread creation.
• Inside a multi-thread tokio runtime (#[tokio::test], server handler): block_in_place tells tokio to spawn a replacement worker so the blocked thread doesn’t cause runtime starvation.
• Inside a current-thread tokio runtime: falls back to a scoped thread, since block_in_place is not supported on single-threaded runtimes.

Async API

The async feature enables async methods for decode, read, and metadata extraction. These work for both local and remote files:

#![allow(unused)]
fn main() {
use tensogram::{TensogramFile, DecodeOptions};

// Async decode methods (feature = "async")
let meta = file.decode_metadata_async(0).await?;
let (meta, descs) = file.decode_descriptors_async(0).await?;
let (meta, desc, data) = file.decode_object_async(0, 0, &DecodeOptions::default()).await?;
let msg = file.read_message_async(0).await?;
}

When both remote and async features are enabled, async open methods are also available:

#![allow(unused)]
fn main() {
// Async open (auto-detects local vs remote) — requires remote + async
let mut file = TensogramFile::open_source_async("s3://bucket/data.tgm").await?;

// Async open with explicit storage options
let mut file = TensogramFile::open_remote_async(
    "s3://bucket/data.tgm",
    &opts,
).await?;
}

For remote backends, async methods directly await object store operations, bypassing the sync bridge entirely. For local backends, they use spawn_blocking for file I/O.

[dependencies]
tensogram = { path = "...", features = ["remote", "async"] }

Range Reads

TensogramFile::decode_range() supports partial object decoding for both local and remote files. It takes an object index and a list of (offset, count) element ranges, returning only the requested elements without decoding the entire object.

For remote files, it fetches the full object frame (via indexed access) then runs the range decode pipeline on the raw payload. This is most beneficial with szip-compressed objects that have szip_block_offsets, where only the compressed blocks covering the requested range are decompressed.

#![allow(unused)]
fn main() {
// Rust: decode elements 100..200 from object 0
let ranges = vec![(100, 100)];
let (desc, parts) = file.decode_range(0, 0, &ranges, &DecodeOptions::default())?;
}

# Python: decode elements 100..200 from object 0
arr = file.file_decode_range(0, 0, [(100, 100)], join=True)

The xarray backend uses file_decode_range automatically when slicing remote arrays that support partial decode (uncompressed or szip-compressed objects without shuffle filters).

Descriptor-Only Reads

decode_descriptors() fetches only the CBOR descriptor from each data object frame, not the full payload. For large objects (hundreds of MB), this avoids downloading the entire frame just to extract a few hundred bytes of metadata.

For frames smaller than 64 KB, the full frame is read in a single request (fewer round-trips). For larger frames, the library reads only the frame header (16 bytes), footer (12 bytes), and the CBOR descriptor region.

Limitations

• Streaming messages must be last. In multi-message files, streaming-encoded messages (total_length=0) must be the last message. The remote scanner assumes the streaming message extends to the end of the file.
• Optimistic scan for buffered messages. Remote message scanning validates preamble magic and total_length plausibility but does not verify end-of-message markers for buffered messages. Streaming messages (total_length=0) do validate the END_MAGIC at EOF.
• Read-only. Remote writes are not supported.
• Header probe size. Layout discovery reads a single chunk of up to 256 KB from the header region. If the metadata or index frame does not fit in this chunk, decode_metadata() will error (it does not retry with a larger read).
• HTTP server requirements. The remote HTTP server must support HEAD requests (for file size) and Range request headers (for partial reads).
• read_message() and decode_message() download the full message even for remote files. Use decode_metadata(), decode_descriptors(), or decode_object() for selective access.
• Zarr remote reads are lazy per-chunk. The zarr store fetches only metadata at open time; individual chunks are decoded on first access. Local files still use eager decode for lower latency.
• Sequential async access. Async methods take &mut self, so a single file handle cannot serve concurrent async reads. Open separate handles for parallelism.

Iterators

Tensogram provides lazy iterator APIs for traversing messages and objects without loading everything into memory at once.

Hierarchy

graph TD
    F[File / Buffer] -->|messages| M1[Message 1]
    F -->|messages| M2[Message 2]
    F -->|messages| M3[Message N]
    M1 -->|objects| O1["(DataObjectDescriptor, Vec<u8>)"]
    M1 -->|objects| O2["(DataObjectDescriptor, Vec<u8>)"]
    O1 -->|access| D1["descriptor + data"]
    O2 -->|access| D2["descriptor + data"]

Rust API

Buffer message iterator

Iterate over messages in a &[u8] byte buffer. Zero-copy: yields slices pointing into the original buffer.

#![allow(unused)]
fn main() {
use tensogram::{messages, decode, DecodeOptions};

let buf: Vec<u8> = std::fs::read("multi.tgm")?;

for msg_bytes in messages(&buf) {
    let (meta, objects) = decode(msg_bytes, &DecodeOptions::default())?;
    println!("version={} objects={}", meta.version, objects.len());
}
}

The iterator calls scan() once on construction, then yields &[u8] slices in sequence. Garbage between valid messages is silently skipped.

MessageIter implements ExactSizeIterator, so .len() returns the remaining count at any point.

Object iterator

Iterate over the decoded objects (tensors) inside a single message. Each item is a (DataObjectDescriptor, Vec<u8>) tuple:

#![allow(unused)]
fn main() {
use tensogram::{objects, DecodeOptions};

for result in objects(&msg_bytes, DecodeOptions::default())? {
    let (descriptor, data) = result?;
    println!("shape={:?} dtype={} encoding={} bytes={}",
             descriptor.shape, descriptor.dtype, descriptor.encoding, data.len());
}
}

Each object is decoded through the full pipeline on demand — objects you don’t consume are never decoded.

For metadata-only access (no payload decode), use objects_metadata. This returns DataObjectDescriptors without decoding any payloads:

#![allow(unused)]
fn main() {
use tensogram::objects_metadata;

for desc in objects_metadata(&msg_bytes)? {
    println!("shape={:?} dtype={} byte_order={}", desc.shape, desc.dtype, desc.byte_order);
}
}

File iterator

Iterate over messages stored in a .tgm file with seek-based lazy I/O:

#![allow(unused)]
fn main() {
use tensogram::{TensogramFile, objects, DecodeOptions};

let mut file = TensogramFile::open("forecast.tgm")?;
for raw in file.iter()? {
    let raw = raw?;
    // Nested: iterate objects within this message
    for result in objects(&raw, DecodeOptions::default())? {
        let (desc, data) = result?;
        println!("{:?} {} {} bytes", desc.shape, desc.dtype, data.len());
    }
}
}

file.iter() scans the file once (if not already scanned), then returns a FileMessageIter that reads each message via seek + read. The iterator does not borrow the TensogramFile — it owns an open file handle and a copy of the message offsets.

C / C++ API

The C FFI uses an opaque-handle + next() pattern. Each iterator returns TGM_OK while items remain, and TGM_END_OF_ITER as an end sentinel.

Buffer iterator

tgm_buffer_iter_t *iter;
tgm_buffer_iter_create(buf, buf_len, &iter);

const uint8_t *msg_ptr;
size_t msg_len;
while (tgm_buffer_iter_next(iter, &msg_ptr, &msg_len) == TGM_OK) {
    // msg_ptr borrows from the original buffer
    tgm_message_t *msg;
    tgm_decode(msg_ptr, msg_len, 0, &msg);
    // ... use msg ...
    tgm_message_free(msg);
}
tgm_buffer_iter_free(iter);

Lifetime: the buffer must remain valid until tgm_buffer_iter_free.

File iterator

tgm_file_t *file;
tgm_file_open("data.tgm", &file);

tgm_file_iter_t *iter;
tgm_file_iter_create(file, &iter);

tgm_bytes_t raw;
while (tgm_file_iter_next(iter, &raw) == TGM_OK) {
    // raw.data is owned — free with tgm_bytes_free
    tgm_message_t *msg;
    tgm_decode(raw.data, raw.len, 0, &msg);
    // ... use msg ...
    tgm_message_free(msg);
    tgm_bytes_free(raw);
}
tgm_file_iter_free(iter);
tgm_file_close(file);

Object iterator

tgm_object_iter_t *iter;
tgm_object_iter_create(msg_ptr, msg_len, 0, &iter);

tgm_message_t *obj;
while (tgm_object_iter_next(iter, &obj) == TGM_OK) {
    uint64_t ndim = tgm_object_ndim(obj, 0);
    const uint64_t *shape = tgm_object_shape(obj, 0);
    // ... use shape, data ...
    tgm_message_free(obj);
}
tgm_object_iter_free(iter);

C++ API

The C++ wrapper (include/tensogram.hpp) provides RAII iterator classes that manage the underlying C handles automatically.

Buffer iterator

#include <tensogram.hpp>

auto buf = /* read file into std::vector<uint8_t> */;
tensogram::buffer_iterator iter(buf.data(), buf.size());

const std::uint8_t* msg_ptr;
std::size_t msg_len;
while (iter.next(msg_ptr, msg_len)) {
    auto msg = tensogram::decode(msg_ptr, msg_len);
    std::printf("version=%llu objects=%zu\n", msg.version(), msg.num_objects());
}

File iterator

auto f = tensogram::file::open("forecast.tgm");
tensogram::file_iterator iter(f);

std::vector<std::uint8_t> raw;
while (iter.next(raw)) {
    auto msg = tensogram::decode(raw.data(), raw.size());
    std::printf("objects=%zu\n", msg.num_objects());
}

Object iterator

tensogram::object_iterator iter(msg_ptr, msg_len);
tensogram::message obj = tensogram::decode(msg_ptr, msg_len); // placeholder for next()
while (iter.next(obj)) {
    auto o = obj.object(0);
    auto shape = o.shape();
    std::printf("dtype=%s shape=[%llu, %llu]\n",
                o.dtype_string().c_str(), shape[0], shape[1]);
}

Range-based for on message

auto msg = tensogram::decode(buf, len);
for (const auto& obj : msg) {
    std::printf("dtype=%s bytes=%zu\n",
                obj.dtype_string().c_str(), obj.data_size());
}

Python API

TensogramFile supports iteration, indexing, and slicing:

import tensogram

# Iterate all messages
with tensogram.TensogramFile.open("forecast.tgm") as f:
    for meta, objects in f:
        for desc, arr in objects:
            print(f"  shape={arr.shape}  dtype={desc.dtype}")

# Index and slice
with tensogram.TensogramFile.open("forecast.tgm") as f:
    meta, objects = f[0]        # first message
    meta, objects = f[-1]       # last message
    subset = f[10:20]           # range of messages
    every_5th = f[::5]          # strided access

# Buffer iteration
buf = open("data.tgm", "rb").read()
for meta, objects in tensogram.iter_messages(buf):
    desc, arr = objects[0]
    print(f"  shape={arr.shape}")

decode(), decode_message(), file iteration, and iter_messages() return Message namedtuples with .metadata and .objects fields. Tuple unpacking (meta, objects = msg) also works. TensogramFile supports len(f) and context manager (with).

Thread safety: iterators own independent file handles and buffer copies — no shared mutable state. Safe under free-threaded Python (PEP 703, no GIL).

Edge cases

Python API

Tensogram provides native Python bindings via PyO3. All tensor data crosses the boundary as NumPy arrays.

Installation

# From PyPI (once published)
pip install tensogram

# From source
pip install maturin numpy
cd python/bindings && maturin develop

Quick Start

import numpy as np
import tensogram

# Encode a 2D temperature field
temps = np.random.randn(100, 200).astype(np.float32) + 273.15
meta = {"version": 2}
desc = {"type": "ntensor", "shape": [100, 200], "dtype": "float32"}

msg = tensogram.encode(meta, [(desc, temps)])

# Decode it back
meta, objects = tensogram.decode(msg)
desc, array = objects[0]
print(array.shape)  # (100, 200)

Encoding

Basic encoding

tensogram.encode() takes metadata, a list of (descriptor, array) pairs, and returns wire-format bytes:

msg = tensogram.encode(
    {"version": 2},
    [({"type": "ntensor", "shape": [3], "dtype": "float32"}, np.array([1, 2, 3], dtype=np.float32))],
    hash="xxh3",  # default; use None to skip hashing
)

Descriptor keys

Every object in a message is described by a dict. The three required keys define what the tensor looks like; the optional keys control how it is stored on the wire.

Any additional keys (e.g. "reference_value", "bits_per_value") are stored in the descriptor’s .params dict and passed through to the encoding pipeline.

The encoding pipeline

Each object passes through a three-stage pipeline before it is stored. You control each stage via descriptor keys:

raw bytes → encoding → filter → compression → wire payload

Encoding transforms the data representation:

Filter rearranges bytes to improve compressibility:

Compression reduces the payload size:

Compression parameters are passed as extra descriptor keys. For example, zstd level:

desc = {
    "type": "ntensor", "shape": [1000], "dtype": "float32",
    "compression": "zstd", "zstd_level": 9,
}

For the full list of compressor parameters, see Compression.

Common pipeline combinations

# Lossless, fast decompression
desc = {"type": "ntensor", "shape": shape, "dtype": "float32",
        "compression": "lz4"}

# Lossless, best ratio (shuffle_element_size must match dtype byte width)
desc = {"type": "ntensor", "shape": shape, "dtype": "float32",
        "filter": "shuffle", "shuffle_element_size": 4, "compression": "zstd", "zstd_level": 12}

# Quantise a bounded-range float field to 16-bit packed ints, then compress
# (the same pipeline GRIB 2 uses for simple_packing + CCSDS).
# compute_packing_params expects a flat float64 array
values = data.astype(np.float64).ravel()
params = tensogram.compute_packing_params(values, bits_per_value=16, decimal_scale_factor=0)
desc = {"type": "ntensor", "shape": shape, "dtype": "float64",
        "encoding": "simple_packing", "compression": "zstd", **params}

# Lossy float compression with error bound (zfp operates on float64)
desc = {"type": "ntensor", "shape": shape, "dtype": "float64",
        "compression": "zfp", "zfp_mode": "fixed_accuracy", "zfp_tolerance": 0.01}

Invalid combinations: Some pipeline combinations are rejected at encode time — e.g. zfp + shuffle (ZFP operates on typed floats, not byte-shuffled data) or simple_packing + sz3 (both are encoding stages). See Compression — Invalid Combinations.

Multiple objects per message

A single message can contain multiple tensors, each with its own descriptor:

spectrum = np.random.randn(256).astype(np.float64)
mask = np.array([1, 0, 1, 1, 0], dtype=np.uint8)

msg = tensogram.encode(
    {"version": 2},
    [
        ({"type": "ntensor", "shape": [256], "dtype": "float64", "compression": "zstd"}, spectrum),
        ({"type": "ntensor", "shape": [5], "dtype": "uint8"}, mask),
    ],
)

Pre-encoded data

If you already have compressed/packed payloads (e.g. from another system), use tensogram.encode_pre_encoded() with the same interface. The library skips the encoding pipeline and writes the bytes as-is:

msg = tensogram.encode_pre_encoded(meta, [(desc, pre_compressed_bytes)])

See Pre-Encoded Data API for details.

Decoding

Full decode

meta, objects = tensogram.decode(msg)

Returns a Message namedtuple with .metadata and .objects. Tuple unpacking works directly.

By default, decoded arrays are in the caller’s native byte order — the library handles byte-swapping automatically. Pass native_byte_order=False to receive the raw wire byte order instead:

meta, objects = tensogram.decode(msg, native_byte_order=False)

Metadata

meta is a Metadata object:

meta.version     # int — always 2
meta.base        # list[dict] — per-object metadata (one entry per object)
meta.extra       # dict — message-level annotations (_extra_ in CBOR)
meta.reserved    # dict — library internals (_reserved_ in CBOR, read-only)
meta["key"]      # dict-style access (checks base entries, then extra)

To read metadata without decoding any payloads:

meta = tensogram.decode_metadata(msg)

To read metadata and descriptors (no payload decode):

meta, descriptors = tensogram.decode_descriptors(msg)
for desc in descriptors:
    print(desc.shape, desc.dtype, desc.compression)

Selective decode

Decode a single object without touching the others — O(1) seek via the binary header’s offset table:

meta, desc, array = tensogram.decode_object(msg, index=2)

Decode a sub-range of elements from one object (for compressors that support random access):

# Elements 100-149 and 300-324 from object 0
parts = tensogram.decode_range(msg, object_index=0, ranges=[(100, 50), (300, 25)])
# parts is a list of numpy arrays, one per range

# Or join into a single contiguous array
joined = tensogram.decode_range(msg, object_index=0, ranges=[(100, 50), (300, 25)], join=True)
# joined is a single flat numpy array of shape (75,)

decode_range works with uncompressed data, simple_packing, szip, blosc2, and zfp fixed-rate mode. It returns an error for stream compressors (zstd, lz4, sz3) and for the shuffle filter. See Decoding Data for details.

Scanning and iteration

To find message boundaries in a buffer without decoding:

offsets = tensogram.scan(buf)  # list of (offset, length) pairs

To iterate messages in a multi-message buffer:

for meta, objects in tensogram.iter_messages(buf):
    print(meta.version, len(objects))

Hash verification

meta, objects = tensogram.decode(msg, verify_hash=True)

Raises RuntimeError if any object’s payload hash doesn’t match. If the message was encoded without a hash (hash=None), verification is silently skipped.

File API

Writing

with tensogram.TensogramFile.create("forecast.tgm") as f:
    for step in range(24):
        data = model.run(step)
        desc = {"type": "ntensor", "shape": list(data.shape), "dtype": "float32",
                "compression": "zstd"}
        f.append({"version": 2, "base": [{"step": step}]}, [(desc, data)])

Each append encodes one message and writes it to the end of the file. Messages are independent and self-describing.

Reading

with tensogram.TensogramFile.open("forecast.tgm") as f:
    print(len(f))                    # message count

    meta, objects = f[0]             # index (supports negative indices)
    subset = f[1:10:2]              # slice → list[Message]

    for meta, objects in f:          # iterate all messages
        for desc, array in objects:
            print(desc.shape, array.dtype)

    raw = f.read_message(0)          # raw bytes for forwarding/caching

The first access triggers a streaming scan that records message offsets. After that, every read is an O(1) seek.

Streaming encoder

For building a message one object at a time in memory:

enc = tensogram.StreamingEncoder({"version": 2}, hash="xxh3")
for desc, data in objects:
    enc.write_object(desc, data)
msg = enc.finish()  # returns complete message as bytes

For pre-encoded payloads, use enc.write_object_pre_encoded(desc, raw_bytes).

Async API

AsyncTensogramFile provides the same operations as TensogramFile but as asyncio coroutines. A single handle supports truly concurrent operations with no per-handle mutex; internal caches are thread-safe.

Opening and decoding

import asyncio
import tensogram

async def main():
    f = await tensogram.AsyncTensogramFile.open("forecast.tgm")

    meta, objects = await f.decode_message(0)
    result = await f.file_decode_object(0, 0)
    print(result["data"].shape)

asyncio.run(main())

For remote files with credentials:

    f = await tensogram.AsyncTensogramFile.open_remote(
        "s3://bucket/data.tgm", {"region": "eu-west-1"}
    )

Concurrent decoding with asyncio.gather

Multiple decode calls run concurrently on a single handle:

    results = await asyncio.gather(
        f.file_decode_object(0, 0),
        f.file_decode_object(1, 0),
        f.file_decode_object(2, 0),
    )

Batch decoding from many messages at once

When you need the same data from many messages, for example reading how a value at one grid point changes over 300 time steps, individual requests are slow because each one is a separate HTTP round-trip.

file_decode_range_batch collects the requested element ranges across messages and fetches the underlying data in a batched HTTP call. file_decode_object_batch does the same for full frames:

    indices = list(range(300))
    row, col, grid = 100, 200, 528
    offset = row * grid + col

    values = await f.file_decode_range_batch(indices, 0, [(offset, 1)], join=True)

    frames = await f.file_decode_object_batch(indices, 0)

For even more speed, split the work into chunks and run them concurrently:

    chunks = [indices[i::16] for i in range(16)]
    batch_results = await asyncio.gather(
        *[f.file_decode_range_batch(chunk, 0, [(offset, 1)], join=True)
          for chunk in chunks]
    )

The sync TensogramFile also has file_decode_range_batch and file_decode_object_batch with the same signatures. Both batch methods require a remote backend; calling them on a local file raises OSError.

Layout prefetching

Before running many concurrent decodes on a remote file, prefetch the internal layout metadata to avoid repeated discovery requests:

    count = await f.message_count()
    await f.prefetch_layouts(list(range(count)))

Context manager and iteration

    async with await tensogram.AsyncTensogramFile.open("data.tgm") as f:
        await f.message_count()   # required before async for or len(f)
        async for meta, objects in f:
            print(objects[0][1].shape)

Async iteration works on remote files (sync iteration does not). await f.message_count() must be called once before using async for or len(f), to discover the message count without blocking the event loop.

Other methods

    count = await f.message_count()
    raw = await f.read_message(0)
    all_raw = await f.messages()
    print(f.is_remote(), f.source())

Note: len(f) requires a prior await f.message_count() call. Without it, len(f) raises RuntimeError.

When to use async vs sync

Validation

Two functions check whether messages and files are well-formed without consuming the data. See also the CLI reference.

report = tensogram.validate(msg)
file_report = tensogram.validate_file("data.tgm")

Levels

# Full validation with canonical CBOR key-order checking
report = tensogram.validate(msg, level="full", check_canonical=True)

Return values

validate() returns:

{
    "issues": [
        {
            "code": "hash_mismatch",   # stable snake_case string
            "level": "integrity",      # which validation level found it
            "severity": "error",       # "error" or "warning"
            "description": "...",      # human-readable message
            "object_index": 0,         # optional — which object
            "byte_offset": 1234,       # optional — position in buffer
        }
    ],
    "object_count": 1,
    "hash_verified": False,
}

validate_file() returns file-level issues plus per-message reports:

{
    "file_issues": [
        {"byte_offset": 100, "length": 19, "description": "trailing bytes after last message"}
    ],
    "messages": [
        {"issues": [], "object_count": 1, "hash_verified": True}
    ],
}

Interpreting results

report = tensogram.validate(msg)
if not report["issues"]:
    print(f"OK — {report['object_count']} objects, hash verified")
else:
    for issue in report["issues"]:
        print(f"[{issue['severity']}] {issue['code']}: {issue['description']}")

GRIB / NetCDF conversion

Three PyO3-bound helpers wrap tensogram-grib and tensogram-netcdf. They are always callable — when the Python wheel was built without the corresponding Cargo feature, each raises RuntimeError with a pointer to rebuild instructions.

You can probe availability at runtime:

import tensogram

if tensogram.__has_grib__:
    msgs = tensogram.convert_grib("forecast.grib2")

if tensogram.__has_netcdf__:
    msgs = tensogram.convert_netcdf("data.nc")

convert_grib(path, **options) -> list[bytes]

Convert a GRIB file (as many messages as it contains) to Tensogram wire format. Returns one bytes per output Tensogram message — join or write sequentially to produce a .tgm file.

msgs = tensogram.convert_grib(
    "forecast.grib2",
    grouping="merge_all",      # "merge_all" | "one_to_one"
    preserve_all_keys=False,   # lift every ecCodes namespace into base[i]["grib"]
    encoding="simple_packing", # "none" | "simple_packing"
    bits=16,                   # None -> defaults to 16; ignored for encoding="none"
    filter="none",             # "none" | "shuffle"
    compression="szip",        # "none" | "zstd" | "lz4" | "blosc2" | "szip"
    compression_level=None,    # applies to zstd / blosc2 (None = codec default)
    threads=0,                 # 0 = sequential; honours TENSOGRAM_THREADS env var
    hash="xxh3",               # "xxh3" | None
    # NaN / Inf handling — see docs/src/guide/nan-inf-handling.md
    allow_nan=False,           # False (default) rejects any NaN input
    allow_inf=False,           # False (default) rejects any ±Inf input
)
with open("forecast.tgm", "wb") as fh:
    for msg in msgs:
        fh.write(msg)

Pipeline defaults and edge cases:

• bits=None with encoding="simple_packing" defaults to 16 bits.
• bits outside 1..=64 silently falls back to encoding="none" and emits a warning to stderr. Validate your inputs before calling if fail-fast is important.
• Unknown compression / encoding names raise ValueError with the list of valid choices in the message.
• Unknown grouping / split_by / hash values raise ValueError.
• Missing input paths raise FileNotFoundError.
• Building the wheel without the grib / netcdf feature causes the corresponding function to raise RuntimeError at call time with rebuild instructions.

Requires libeccodes at the OS level and the wheel built with --features grib (maturin develop --features grib). Official PyPI wheels do not currently include the grib feature — see Jupyter Notebook Walk-through.

convert_grib_buffer(buf, **options) -> list[bytes]

In-memory variant of convert_grib. Accepts any Python bytes-like object (bytes, bytearray, memoryview, numpy.uint8[:]). Useful when the GRIB bytes come from a byte-range HTTP fetch, a cache, or any other in-memory source — no filesystem staging needed.

import requests

# Byte-range download of a single GRIB message from data.ecmwf.int.
resp = requests.get(
    "https://data.ecmwf.int/forecasts/.../...grib2",
    headers={"Range": "bytes=74573515-75234113"},
)
msgs = tensogram.convert_grib_buffer(
    resp.content,
    encoding="simple_packing",
    bits=16,
    compression="szip",
    # See [NaN / Inf Handling](nan-inf-handling.md) for the
    # `allow_nan` / `allow_inf` opt-in if your data contains
    # non-finite values.
)

convert_grib and convert_grib_buffer produce bit-identical decoded payloads for the same input. The encoded bytes may differ — each call stamps a fresh timestamp and UUID into _reserved_.

convert_netcdf(path, **options) -> list[bytes]

Convert a NetCDF-3 or NetCDF-4 file to Tensogram. Packed variables (scale_factor / add_offset) are automatically unpacked to float64.

msgs = tensogram.convert_netcdf(
    "data.nc",
    split_by="file",           # "file" | "variable" | "record"
    cf=False,                  # lift 16 CF attributes into base[i]["cf"]
    encoding="none",
    bits=None,
    filter="none",
    compression="zstd",
    compression_level=3,
    threads=0,
    hash="xxh3",
    # NaN / Inf handling — see docs/src/guide/nan-inf-handling.md
    allow_nan=False,           # False (default) rejects any NaN input
    allow_inf=False,           # False (default) rejects any ±Inf input
)

Note on NaN and --encoding simple_packing. Since 0.17 the importer hard-fails on NaN or Inf in a variable targeted for simple_packing (previous behaviour: stderr warning + fallback to encoding="none"). If your NetCDF has _FillValue / missing_value fields unpacked to NaN, either stick with the default encoding="none" or pre-process the values. See the NetCDF Importer error-handling reference for the full contract.

Requires libnetcdf + libhdf5 at the OS level and the wheel built with --features netcdf.

Error Handling

Supported dtypes

bfloat16 is returned as ml_dtypes.bfloat16 when ml_dtypes is installed; otherwise it falls back to np.uint16.

See Data Types for byte widths and wire-format details.

Examples

See examples/python/ for complete working examples:

For narrative walk-throughs with plots and explanations, see also examples/jupyter/*.ipynb — five journey notebooks covering quickstart/MARS, encoding pipeline fidelity, GRIB conversion, NetCDF conversion with xarray, and validation with multi-threaded encoding.

C++ API

Tensogram provides a header-only C++17 wrapper at cpp/include/tensogram.hpp. It delegates all work to the C FFI and adds RAII handle management, typed exceptions, and idiomatic C++ patterns.

Requirements

• C++17 compiler (GCC 7+, Clang 5+, MSVC 19.14+)
• Rust static library built via cargo build --release
• CMake 3.16+ (recommended)

Build

cargo build --release
cmake -S cpp -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Quick Start

#include <tensogram.hpp>

// Encode
std::string meta_json = R"({"version": 2, "descriptors": [...]})";
std::vector<float> data(100 * 200, 0.0f);
auto encoded = tensogram::encode(
    meta_json,
    {{reinterpret_cast<const uint8_t*>(data.data()), data.size() * sizeof(float)}});

// Decode
auto msg = tensogram::decode(encoded.data(), encoded.size());
auto obj = msg.object(0);
const float* values = obj.data_as<float>();

RAII Classes

All classes are move-only (copy deleted). Handles are released automatically when the object goes out of scope.

Error Handling

C error codes are mapped to a typed exception hierarchy:

try {
    auto msg = tensogram::decode(buf, len);
} catch (const tensogram::framing_error& e) {
    // Invalid message framing
} catch (const tensogram::hash_mismatch_error& e) {
    // Payload integrity check failed
} catch (const tensogram::error& e) {
    // Any Tensogram error (base class)
    std::cerr << e.what() << " (code=" << e.code() << ")\n";
}

Validation

Two free functions validate messages and files, returning JSON strings:

// Validate a single message buffer (default level)
auto report = tensogram::validate(buf, len);

// Full validation with canonical CBOR check
auto full_report = tensogram::validate(buf, len, "full", /*check_canonical=*/true);

// Validate a .tgm file
auto file_report = tensogram::validate_file("data.tgm");
auto file_full   = tensogram::validate_file("data.tgm", "full");

Validation levels: "quick", "default", "checksum", "full".

The returned JSON contains issues, object_count, and hash_verified for single messages, or file_issues and messages for files. Parse with your preferred JSON library.

An invalid level string or a missing file throws tensogram::invalid_arg_error or tensogram::io_error respectively. Validation issues (corrupted data, hash mismatches) are reported in the JSON — they do not throw.

Iterators

See Iterators for buffer, file, and object iterator usage.

Examples

See examples/cpp/ for complete working examples covering encode/decode, metadata, file API, simple packing, and iterators.

TypeScript API

Tensogram ships @ecmwf/tensogram, a TypeScript package that wraps the WebAssembly build with typed, idiomatic helpers. Use it in any modern browser or Node ≥ 20.

Status: Scope B is complete. Typed encode / decode / scan, dtype dispatch, metadata helpers, progressive streaming decode, and the TensogramFile file / URL helper are all available. Scope C follow-ups (validate wrapper, encodePreEncoded, first-class float16 / bfloat16 / complex* types, npm publish pipeline) are tracked in plans/TYPESCRIPT_WRAPPER.md.

Installation

The package is not yet published to npm. Build it locally:

# First, build the WebAssembly blob from the Rust source
cd typescript
npm install
npm run build:wasm   # runs wasm-pack build -t web -d typescript/wasm
npm run build        # runs wasm-pack + tsc

Or use the top-level Makefile:

make ts-build        # build WASM + tsc
make ts-test         # vitest
make ts-typecheck    # strict tsc --noEmit on src + tests

Quick start

import {
  init, encode, decode,
  type DataObjectDescriptor,
  type GlobalMetadata,
} from '@ecmwf/tensogram';

// One-time WASM initialisation (idempotent)
await init();

// ── Encode ────────────────────────────────────────────────────────────
const temps = new Float32Array(100 * 200);
for (let i = 0; i < temps.length; i++) temps[i] = 273.15 + i / 100;

const meta: GlobalMetadata = { version: 2 };
const descriptor: DataObjectDescriptor = {
  type: 'ntensor',
  ndim: 2,
  shape: [100, 200],
  strides: [200, 1],
  dtype: 'float32',
  byte_order: 'little',
  encoding: 'none',
  filter: 'none',
  compression: 'none',
};

const msg: Uint8Array = encode(meta, [{ descriptor, data: temps }]);

// ── Decode ────────────────────────────────────────────────────────────
const { metadata, objects } = decode(msg);
const arr = objects[0].data();  // Float32Array (inferred from dtype)
console.log(arr.length);        // 20000

API surface

init(opts?)

Loads and instantiates the WASM blob. Must be awaited before any other function is called. Safe to call multiple times — subsequent calls reuse the same instance.

await init();                                              // defaults
await init({ wasmInput: new URL('...', import.meta.url) });  // custom location

encode(metadata, objects, opts?)

Returns: Uint8Array containing the complete wire-format message.

decode(buf, opts?)

Returns: { metadata: GlobalMetadata, objects: DecodedObject[], close() }.

decodeMetadata(buf)

Returns only the metadata; does not touch any payload bytes.

decodeObject(buf, index, opts?)

O(1) seek to object index, decoding only that object.

scan(buf)

Returns Array<{ offset: number; length: number }> for each Tensogram message found in a (potentially multi-message) buffer. Garbage between messages is silently skipped.

DecodedObject / DecodedFrame

interface DecodedObject {
  readonly descriptor: DataObjectDescriptor;
  /** Copy into the JS heap.  Safe across WASM memory growth. */
  data(): TypedArray;
  /** Zero-copy view.  Invalidated if WASM memory grows. */
  dataView(): TypedArray;
  readonly byteLength: number;
}

interface DecodedFrame extends /* structurally */ DecodedObject {
  /** The matching `base[i]` entry from the containing message. */
  readonly baseEntry: BaseEntry | null;
  close(): void;
}

The returned array type is picked from descriptor.dtype:

getMetaKey(meta, path)

Dot-path lookup matching the Rust / Python / CLI first-match-across-base semantics: searches base[0], base[1], …, skipping the _reserved_ key in each, then falls back to _extra_.

getMetaKey(meta, 'mars.param')      // 'base[0].mars.param' first match
getMetaKey(meta, '_extra_.source')  // explicit _extra_ prefix

Returns undefined if the key is missing (never throws).

computeCommon(meta)

Mirror of tensogram::compute_common. Returns a Record<string, CborValue> of keys that are present with identical values in every entry of meta.base. Useful for display and merge operations.

Error classes

All errors thrown from this package are instances of the abstract TensogramError class. Eight concrete subclasses match the Rust TensogramError variants plus the TS-layer InvalidArgumentError and StreamingLimitError:

import {
  TensogramError,
  FramingError,
  MetadataError,
  EncodingError,
  CompressionError,
  ObjectError,
  IoError,
  RemoteError,
  HashMismatchError,
  InvalidArgumentError,
  StreamingLimitError,
} from '@ecmwf/tensogram';

try {
  decode(corruptBuffer);
} catch (err) {
  if (err instanceof FramingError) {
    console.error('bad wire format:', err.message);
  } else if (err instanceof HashMismatchError) {
    console.error('integrity failure:', err.expected, err.actual);
  } else {
    throw err;
  }
}

Memory model

• Safe-copy by default. object.data() / frame.data() always allocate a new TypedArray on the JS heap. It remains valid even after the underlying DecodedMessage / DecodedFrame is freed or WASM memory grows.
• Zero-copy opt-in. object.dataView() / frame.dataView() return a view directly into WASM linear memory. It is invalidated the next time any WASM call grows linear memory — which can happen on the next encode() / decode(). Read the view immediately or copy it.
• Explicit cleanup. DecodedMessage, DecodedFrame, and TensogramFile all expose .close() to release WASM-side memory. A FinalizationRegistry also calls .free() on the underlying WASM handle when the wrapper is garbage-collected, but explicit .close() is strongly recommended for deterministic cleanup.

Streaming decode

Use decodeStream(readable, opts?) to progressively decode a ReadableStream<Uint8Array>. Works against any stream source — fetch().body, a Node Readable.toWeb(), a Blob.stream(), or a hand-rolled ReadableStream.

import { decodeStream } from '@ecmwf/tensogram';

const res = await fetch('/data.tgm');
for await (const frame of decodeStream(res.body!)) {
  render(frame.descriptor.shape, frame.data());
  frame.close();
}

Options:

Key behaviours:

• Chunk-boundary tolerant. A message can be split across any number of chunks. The decoder accumulates until a complete message is seen, then emits every object as a separate frame.
• Corruption resilient. A single bad message is skipped; the iterator keeps going with subsequent messages. Pass onError to observe the skips.
• Early break is safe. Breaking out of the for await loop runs the generator’s finally block, which releases the stream reader and frees the decoder.
• AbortSignal cancels cleanly. Firing the signal cancels the underlying reader; the generator throws whatever error the signal carries.

File API

TensogramFile gives you random-access reads over a .tgm file, whether it lives on the local file system, behind an HTTPS URL, or already in memory.

import { TensogramFile } from '@ecmwf/tensogram';

// Node: from the local file system
const file = await TensogramFile.open('/data/input.tgm');

// Browser or Node: over HTTPS
const file = await TensogramFile.fromUrl('https://example.com/input.tgm');

// Any runtime: from pre-loaded bytes
const file = TensogramFile.fromBytes(uint8ArrayFromSomewhere);

All three factories produce an identical object:

interface TensogramFile extends AsyncIterable<DecodedMessage> {
  readonly messageCount: number;
  readonly byteLength: number;
  readonly source: 'local' | 'remote' | 'buffer';

  message(index: number, opts?: DecodeOptions): Promise<DecodedMessage>;
  messageMetadata(index: number): Promise<GlobalMetadata>;
  rawMessage(index: number): Uint8Array;

  [Symbol.asyncIterator](): AsyncIterator<DecodedMessage>;
  close(): void;
}

Usage:

const file = await TensogramFile.open('/data/input.tgm');
try {
  console.log(`${file.messageCount} messages, ${file.byteLength} bytes`);

  // Random access
  const first = await file.message(0);
  console.log(first.objects[0].descriptor.shape);
  first.close();

  // Async iteration
  for await (const msg of file) {
    // ...
    msg.close();
  }
} finally {
  file.close();
}

TensogramFile.open(path, opts?) (Node only)

Loads the file via node:fs/promises. The node:fs/promises import is dynamic so browser bundlers can tree-shake this code path.

TensogramFile.fromUrl(url, opts?) (any fetch-capable runtime)

Downloads the file over HTTPS using the ambient globalThis.fetch.

TensogramFile.fromBytes(bytes)

Wraps an already-loaded Uint8Array. The buffer is defensively copied, so later mutation of the caller’s buffer is invisible to the TensogramFile.

Range-based lazy access

Since Scope C, TensogramFile.fromUrl automatically probes the server for HTTP Range support. When the HEAD response advertises Accept-Ranges: bytes and a finite Content-Length, the file switches to a lazy backend:

• The initial open issues a small HEAD + one 24-byte Range read per message preamble to build the boundary index. No payload data is downloaded.
• rawMessage(i) / message(i) fetch just the requested message’s bytes via a Range: bytes=offset-(offset+length-1) GET.
• A small LRU caches recently-fetched message bytes so repeat reads are free.

When the server omits Accept-Ranges, returns non-200 on HEAD, or the file uses streaming-mode messages (total_length=0 — the writer did not know the final length up front), the open falls back to a single eager GET. Behaviour is indistinguishable to callers except in memory use and timing.

Browser callers using fromUrl directly need CORS to expose the Accept-Ranges, Content-Range, and Content-Length headers.

Append (Node local file system)

TensogramFile#append(meta, objects, opts?) encodes the new message in-memory, appends it to the on-disk file, refreshes the position index, and makes the new message reachable via message(i) on the same handle. Only supported when the file was opened via TensogramFile.open(path) — fromBytes- and fromUrl-backed files throw InvalidArgumentError, matching the contract in the other language bindings.

const file = await TensogramFile.open('/data/forecast.tgm');
try {
  await file.append({ version: 2 }, [{ descriptor, data }]);
  console.log(`now has ${file.messageCount} messages`);
} finally {
  file.close();
}

Scope-C API additions

Scope C brought the TypeScript wrapper to full API parity with Rust / Python / FFI / C++. The surface additions are:

Streaming StreamingEncoder (no full-message buffering)

For browser uploads, WebSocket pushes, or any sink that needs bytes as soon as they are produced, pass an onBytes callback to the StreamingEncoder constructor:

const enc = new StreamingEncoder({ version: 2 }, {
  onBytes: (chunk) => uploadSocket.send(chunk),   // e.g. WebSocket.send
});
enc.writeObject(descriptor, new Float32Array([1, 2, 3]));
enc.finish();    // flushes footer; returns empty Uint8Array in streaming mode
enc.close();

Semantics:

• The callback is invoked during construction (preamble + header metadata frame), during each writeObject / writeObjectPreEncoded (one data-object frame’s bytes, potentially across multiple invocations), and during finish() (footer frames + postamble).
• Concatenating every chunk the callback sees (in order) yields a message byte-for-byte identical to what buffered mode would return. Tested via round-trip with decode().
• The callback must be synchronous — Promise return values are silently discarded because the Rust/WASM writer contract is synchronous. Buffer internally first if you need async work.
• Each chunk is JS-owned and fresh per invocation. Copy (new Uint8Array(chunk) or chunk.slice()) if you need to keep it past the next writeObject — the underlying ArrayBuffer is invalidated when WASM memory grows.
• If the callback throws, the exception surfaces as an IoError on the next writeObject / finish. The encoder state is undefined after an error — call close() and start over.
• enc.streaming (getter) reports whether an onBytes sink was supplied — useful for code that needs to branch on mode.

Parity note: the Rust core StreamingEncoder<W: Write> has always supported arbitrary sinks; the WASM/TS surface now exposes this capability to JS code. Python / FFI / C++ bindings remain buffered-only; extending them would follow the same JsCallbackWriter pattern with a language-specific sink abstraction and is tracked in plans/TYPESCRIPT_WRAPPER.md.

First-class half-precision and complex dtypes

Scope C also upgraded the dtype dispatch in {@link typedArrayFor}. obj.data() now returns a first-class view for dtypes JS does not have a native TypedArray for:

All three classes expose .bits / .data for zero-copy access to the underlying raw storage if you need it.

const m = decode(buf);
const f16 = m.objects[0].data();           // Float16Array or polyfill
const asFloat32 = f16.toFloat32Array();    // widened copy
const bits = f16.bits;                      // raw binary16

const cplx = m.objects[1].data() as ComplexArray;
for (let i = 0; i < cplx.length; i++) {
  console.log(cplx.real(i), cplx.imag(i));
}

The polyfill is used automatically when the host runtime does not ship globalThis.Float16Array. hasNativeFloat16Array() and getFloat16ArrayCtor() expose the detection machinery for callers that want direct control.

Breaking change from Scope B: Before Scope C, obj.data() on float16 / bfloat16 returned a raw Uint16Array of bits, and complex dtypes returned an interleaved Float32Array / Float64Array. Consumers that relied on that shape can reach the same bytes via .bits (for f16/bf16) or .data (for complex).

The low-level bit-conversion helpers (halfBitsToFloat, floatToHalfBits, bfloat16BitsToFloat, floatToBfloat16Bits) and the isComplexDtype type-guard are internal and are not re-exported from @ecmwf/tensogram. Callers that need bit-level manipulation should grab the raw storage from a view’s .bits / .data accessor and do the conversion themselves, or import directly from @ecmwf/tensogram/float16, …/bfloat16, …/complex with the understanding that these module paths are not part of the stable API.

Examples

See examples/typescript/ in the repository for runnable scripts:

• 01_encode_decode.ts — basic round-trip
• 02_mars_metadata.ts — per-object metadata using the MARS vocabulary
• 02b_generic_metadata.ts — per-object metadata using a generic application namespace
• 03_multi_object.ts — multiple dtypes in one message
• 04_decode_range.ts — partial sub-tensor decode
• 05_streaming_fetch.ts — progressive decode over a ReadableStream
• 06_file_api.ts — TensogramFile over Node fs, fetch, and in-memory bytes
• 07_hash_and_errors.ts — hash verification and typed errors
• 08_validate.ts — validate(buf) + validateFile(path)
• 11_encode_pre_encoded.ts — wrap already-encoded bytes
• 12_streaming_encoder.ts — frame-at-a-time encoder with preceders
• 13_range_access.ts — lazy TensogramFile.fromUrl over HTTP Range
• 14_streaming_callback.ts — StreamingEncoder with onBytes callback sink

Run them with:

cd examples/typescript
npm install
npx tsx 01_encode_decode.ts     # or any other file

Design notes

See plans/TYPESCRIPT_WRAPPER.md for the full design document covering architecture, phases, test strategy, memory model, and open follow-ups.

Cross-language parity

This TypeScript package decodes the same golden .tgm files used by the Rust, Python, and C++ test suites. The committed files at rust/tensogram/tests/golden/*.tgm are decoded by each language’s test runner; any drift in wire-format semantics fails all four suites.

Specifically, typescript/tests/golden.test.ts decodes:

• simple_f32.tgm — single-object Float32 round-trip
• multi_object.tgm — mixed-dtype message (f32 / i64 / u8)
• mars_metadata.tgm — MARS keys under base[0].mars
• multi_message.tgm — two concatenated messages (via scan())
• hash_xxh3.tgm — verifyHash success + tamper detection

typescript/tests/property.test.ts and the Scope-C dtype suites add fast-check property tests pinning:

• mapTensogramError never throws for any finite-string input and always returns a TensogramError subclass;
• encode → decode is bit-exact for random Float32 shapes across random application metadata;
• decode on random byte input either succeeds with a structurally valid message or throws a typed TensogramError — never panics;
• float32 → float16 → float32 round-trip stays within half-precision ulp for any random value in a reasonable magnitude band;
• float32 → bfloat16 → float32 round-trip stays within bfloat16 ulp;
• complex64 encode → decode preserves real(i) / imag(i) byte-for-byte across random shapes and values.

The CI typescript job rebuilds and runs every TS test on every PR.

Tensoscope

Tensoscope is an interactive web viewer for .tgm files. It runs entirely in the browser — no server-side component — by decoding data via the @ecmwf/tensogram WebAssembly package.

Quick start

Build the WASM package first, then start the dev server:

cd typescript && make ts-build
cd tensoscope && npm install && npm run dev

Open http://localhost:5173 in your browser, then drag-and-drop a .tgm file onto the page or paste a URL into the file open dialog.

Loading a file

Two modes are supported:

• Local file — drag the .tgm file onto the drop zone, or click Open file.
• Remote URL — paste an HTTP/HTTPS URL. The file is fetched in full before scanning. (HTTP Range support for lazy loading is planned.)

Once loaded, Tensoscope scans all messages and builds a field index without decoding any payloads.

Field browser

The left sidebar lists every decodable field in the file. Each entry shows:

• Variable name (resolved from mars.param, name, or param metadata keys)
• Shape and dtype

Click a field to decode it and render it on the map.

Map view

Fields with two spatial dimensions (latitude × longitude) are rendered as a coloured overlay on an interactive map. Regridding from the unstructured source grid onto the display pixel grid runs in a web worker so the UI stays responsive while large arrays are processed.

Projections

Switch between flat (Mercator, powered by MapLibre GL JS) and globe (3D sphere, powered by CesiumJS with OpenStreetMap base tiles) using the projection picker in the bottom-left of the map. Camera position is preserved when switching between the two renderers.

Render modes

A Heatmap / Contours toggle in the top-left of the map switches between two rendering styles:

• Heatmap — smooth continuous gradient from the active colour scale. Pixel colours are interpolated linearly across the data range.
• Contours — filled colour bands (like matplotlib.contourf). The data range is divided into N discrete bands where N is the number of colour steps in the active palette (default 10 for continuous palettes; stop count for custom palettes). Each band is rendered with a single solid colour.

Colour scale

The colour bar at the bottom of the map shows the current field range. Use the colour scale controls to:

• Change the colour map (perceptually uniform maps from d3-scale-chromatic)
• Lock or reset the min/max range

Animation

For files with a time or step dimension, the step slider appears below the map. Use play/pause to animate through steps at a fixed frame rate.

Docker deployment

cd tensoscope
make build          # build the container image
make run            # serve at http://localhost:8000
BASE_PATH=/scope make run   # serve under a subpath

The image uses nginx and accepts a BASE_PATH environment variable for subpath deployments behind a reverse proxy.

Known limitations

• Only lat/lon grids are currently regridded; polar stereographic and other projections are not yet handled.
• 3D fields (pressure levels) cannot yet be sliced via the level selector (the UI component exists but is not yet wired up).
• HTTP Range-based lazy loading is not yet implemented; the full file is fetched before any field can be displayed.

xarray Integration

The tensogram-xarray package provides a read-only xarray backend engine for .tgm files. Once installed, you can open tensogram data with:

import xarray as xr
ds = xr.open_dataset("data.tgm", engine="tensogram")

This chapter explains the conversion philosophy, the mapping rules, and walks through progressively complex examples so you know exactly what to expect – and what to provide – when loading tensogram data into xarray.

Philosophy: Why Mapping is Needed

Tensogram and xarray have fundamentally different data models:

Tensogram is vocabulary-agnostic by design. The library never interprets metadata keys – it does not know what "mars.param", "bids.subject", or "product.name" means. xarray, on the other hand, requires named dimensions and coordinate arrays to enable its powerful label-based indexing and alignment.

The tensogram-xarray backend bridges this gap. It applies a set of rules to translate tensogram structure into xarray structure, and lets you override those rules when the defaults are not enough.

flowchart LR
    A["Tensogram Message"] --> B["tensogram-xarray"]
    B --> C["xr.Dataset"]
    D["User Mapping<br/>(optional)"] -.-> B
    E["Coordinate<br/>Auto-Detection"] -.-> B

The Mapping Pipeline

When you call xr.open_dataset("file.tgm", engine="tensogram"):

1. Read metadata – only the CBOR metadata is parsed (no payload decode).
2. Detect coordinates – data objects whose name or param matches a known coordinate name (latitude, longitude, time, …) become coordinate arrays.
3. Name dimensions – if you provided dim_names, those are used. Otherwise, axes matching a detected coordinate use that coordinate’s name; remaining axes become dim_0, dim_1, …
4. Name variables – if you provided variable_key, the value at that metadata path becomes the variable name. Otherwise object_0, object_1, …
5. Wrap data lazily – each tensor is backed by a BackendArray that decodes on demand. No payload bytes are read until you access .values.

Example 1: Simplest Case – Single Object, No Metadata

Creating the file:

import numpy as np
import tensogram

data = np.arange(60, dtype=np.float32).reshape(6, 10)
meta = {"version": 2}
desc = {"type": "ntensor", "shape": [6, 10], "dtype": "float32",
        "byte_order": "little", "encoding": "none",
        "filter": "none", "compression": "none"}

with tensogram.TensogramFile.create("simple.tgm") as f:
    f.append(meta, [(desc, data)])

Opening in xarray:

>>> import xarray as xr
>>> ds = xr.open_dataset("simple.tgm", engine="tensogram")
>>> ds
<xarray.Dataset>
Dimensions:   (dim_0: 6, dim_1: 10)
Dimensions without coordinates: dim_0, dim_1
Data variables:
    object_0  (dim_0, dim_1) float32 ...
Attributes:
    tensogram_version:  2

The data object became a variable named object_0. Dimensions are auto-generated as dim_0, dim_1. No coordinates – tensogram has no information to generate them.

Adding dimension names:

>>> ds = xr.open_dataset("simple.tgm", engine="tensogram",
...                      dim_names=["latitude", "longitude"])
>>> ds["object_0"].dims
('latitude', 'longitude')

Example 2: Single Object with Coordinate Objects

When coordinate arrays are stored as separate data objects in the same message, the backend auto-detects them by name.

Creating the file:

lat = np.linspace(-90, 90, 5, dtype=np.float64)
lon = np.linspace(0, 360, 8, endpoint=False, dtype=np.float64)
temp = np.random.default_rng(42).random((5, 8)).astype(np.float32)

meta = {"version": 2, "base": [
    {"name": "latitude"},
    {"name": "longitude"},
    {"name": "temperature"},
]}

with tensogram.TensogramFile.create("with_coords.tgm") as f:
    f.append(meta, [
        ({"type": "ntensor", "shape": [5], "dtype": "float64", ...}, lat),
        ({"type": "ntensor", "shape": [8], "dtype": "float64", ...}, lon),
        ({"type": "ntensor", "shape": [5, 8], "dtype": "float32", ...}, temp),
    ])

Opening in xarray:

>>> ds = xr.open_dataset("with_coords.tgm", engine="tensogram")
>>> ds
<xarray.Dataset>
Dimensions:      (latitude: 5, longitude: 8)
Coordinates:
  * latitude     (latitude) float64 -90.0 -45.0 0.0 45.0 90.0
  * longitude    (longitude) float64 0.0 45.0 90.0 135.0 180.0 225.0 270.0 315.0
Data variables:
    temperature  (latitude, longitude) float32 ...
Attributes:
    tensogram_version:  2

How it works:

• Objects with name: "latitude" and name: "longitude" match known coordinate names (case-insensitive).
• They become coordinate arrays on the Dataset.
• The temperature object’s shape (5, 8) matches the sizes of latitude (5) and longitude (8), so its dimensions are automatically resolved to ("latitude", "longitude").

Known Coordinate Names

The following names are recognized (case-insensitive):

If no matching coordinate objects are found and no dim_names are provided, dimensions remain generic (dim_0, dim_1, …).

Example 3: Multi-Object with variable_key

When a message contains multiple data objects, each with per-object metadata identifying the parameter, you can use variable_key to name the variables.

Creating the file:

t2m = np.ones((3, 4), dtype=np.float32) * 273.15
u10 = np.ones((3, 4), dtype=np.float32) * 5.0

meta = {"version": 2,
    "base": [
        {"mars": {"class": "od", "date": "20260401", "type": "fc", "param": "2t", "levtype": "sfc"}},
        {"mars": {"class": "od", "date": "20260401", "type": "fc", "param": "10u", "levtype": "sfc"}},
    ],
}

with tensogram.TensogramFile.create("mars.tgm") as f:
    f.append(meta, [
        ({"type": "ntensor", "shape": [3, 4], "dtype": "float32", ...}, t2m),
        ({"type": "ntensor", "shape": [3, 4], "dtype": "float32", ...}, u10),
    ])

Without variable_key:

>>> ds = xr.open_dataset("mars.tgm", engine="tensogram")
>>> list(ds.data_vars)
['object_0', 'object_1']

With variable_key:

>>> ds = xr.open_dataset("mars.tgm", engine="tensogram",
...                      variable_key="mars.param")
>>> list(ds.data_vars)
['2t', '10u']
>>> ds.attrs
{'tensogram_version': 2}

The variable_key supports dotted paths: "mars.param" navigates into the nested mars dict within each object’s metadata.

Example 4: Multi-Message File with Auto-Merge

When a .tgm file contains many messages (one object each) that differ only in metadata, open_datasets() can stack them along outer dimensions.

Creating the file:

import tensogram_xarray

rng = np.random.default_rng(99)
with tensogram.TensogramFile.create("multi.tgm") as f:
    for param in ["2t", "10u"]:
        for date in ["20260401", "20260402"]:
            data = rng.random((3, 4), dtype=np.float32).astype(np.float32)
            meta = {"version": 2,
                    "base": [{"mars": {"param": param, "date": date}}]}
            desc = {"type": "ntensor", "shape": [3, 4], "dtype": "float32",
                    "byte_order": "little", "encoding": "none",
                    "filter": "none", "compression": "none"}
            f.append(meta, [(desc, data)])

Opening with open_datasets():

>>> datasets = tensogram_xarray.open_datasets(
...     "multi.tgm", variable_key="mars.param"
... )
>>> len(datasets)
1
>>> ds = datasets[0]
>>> list(ds.data_vars)
['2t', '10u']

What happened:

1. The scanner read metadata from all 4 messages (no payload decode).
2. Objects were grouped by structure: all have shape (3, 4) and float32.
3. variable_key="mars.param" split by parameter: 2t (2 objects) and 10u (2 objects).
4. Within each sub-group, mars.date varies across ["20260401", "20260402"], so it became an outer dimension.
5. Each variable has shape (2, 3, 4) with a mars.date coordinate.

Example 5: Heterogeneous File – Auto-Split

When a file contains objects of different shapes or dtypes, they cannot be merged into a single Dataset. open_datasets() automatically splits them into compatible groups.

Creating the file:

with tensogram.TensogramFile.create("hetero.tgm") as f:
    # Message 0: 2D float32 temperature field
    f.append({"version": 2, "base": [{"name": "temp"}]},
             [({"type": "ntensor", "shape": [3, 4], "dtype": "float32", ...},
               np.ones((3, 4), dtype=np.float32))])

    # Message 1: 2D float32 wind field (same shape -- compatible)
    f.append({"version": 2, "base": [{"name": "wind"}]},
             [({"type": "ntensor", "shape": [3, 4], "dtype": "float32", ...},
               np.ones((3, 4), dtype=np.float32) * 2)])

    # Message 2: 1D int32 counts (different shape AND dtype -- incompatible)
    f.append({"version": 2, "base": [{"name": "counts"}]},
             [({"type": "ntensor", "shape": [5], "dtype": "int32", ...},
               np.array([1, 2, 3, 4, 5], dtype=np.int32))])

Opening:

>>> datasets = tensogram_xarray.open_datasets("hetero.tgm")
>>> len(datasets)
2
>>> datasets[0]  # The (3, 4) float32 group
<xarray.Dataset>
Dimensions:  (dim_0: 3, dim_1: 4)
Data variables:
    temp     (dim_0, dim_1) float32 ...
    wind     (dim_0, dim_1) float32 ...

>>> datasets[1]  # The (5,) int32 group
<xarray.Dataset>
Dimensions:  (dim_0: 5)
Data variables:
    counts   (dim_0) int32 ...

Objects that share (shape, dtype) are grouped together. Incompatible objects go to separate Datasets.

Example 6: Providing Full User Mapping

For complete control, pass all mapping parameters:

ds = xr.open_dataset(
    "forecast.tgm",
    engine="tensogram",
    dim_names=["latitude", "longitude"],
    variable_key="mars.param",
    message_index=0,         # which message in a multi-message file
    verify_hash=True,        # verify xxh3 integrity on decode
)

For multi-message files, use tensogram_xarray.open_datasets() directly:

import tensogram_xarray

datasets = tensogram_xarray.open_datasets(
    "forecast.tgm",
    dim_names=["latitude", "longitude"],
    variable_key="mars.param",
    verify_hash=True,
)

Example 7: Lazy Loading with Dask

Data is always loaded lazily. Opening a file only reads metadata – the tensor payloads are decoded on first access. This enables working with larger-than-memory files via dask.

# Open with dask chunking
ds = xr.open_dataset("large.tgm", engine="tensogram", chunks={})
print(ds["object_0"])
# <xarray.DataArray 'object_0' (dim_0: 10000, dim_1: 10000)>
# dask.array<...>

# Compute a mean without loading the full array
mean = ds["object_0"].mean().compute()

See also: Dask Integration for a complete walkthrough with distributed computation, performance tuning, and a runnable 4-D tensor example.

When Partial Reads Are Used

The backend inspects each data object’s encoding pipeline to determine whether partial reads via decode_range(join=False) are available:

When partial reads are available, slicing a lazy array decodes only the requested region:

ds = xr.open_dataset("szip_data.tgm", engine="tensogram")
# Only the bytes for rows 100-110 are decompressed:
subset = ds["object_0"][100:110, :].values

When partial reads are not available (stream compressors or shuffle filter), the full object is decoded and then sliced in memory. This is transparent to the user – the API is identical.

N-Dimensional Slice Mapping

When you slice a lazy xarray variable backed by tensogram, the backend must convert an N-dimensional slice into flat element ranges that decode_range() understands. Here is how the decomposition works:

1. Find the split point – scan the slice dimensions from innermost to outermost and find the first (innermost) dimension whose slice does not cover the full axis. All dimensions inner to this point are contiguous in memory and form a single block per outer-index combination.

2. Compute the contiguous block size – multiply the lengths of all dimensions inner to (and including) the split dimension’s slice width. This gives the number of elements in each flat range.

3. Generate one range per outer-index combination – iterate over the Cartesian product of sliced indices in all dimensions outer to the split point. Each combination produces one (offset, count) pair.

4. Merge adjacent ranges – if two consecutive ranges abut in the flat layout (i.e. offset_i + count_i == offset_{i+1}), they are merged into a single wider range to reduce I/O calls.

Concrete example: an array of shape (100, 200) sliced as [10:20, 50:100]:

• The innermost dimension (axis 1) has slice 50:100 (width 50), which does not cover the full axis (200), so it is the split point.
• Contiguous block size = 50 elements (just the inner slice width).
• Outer indices: axis 0 slice 10:20 gives indices [10, 11, ..., 19] – 10 combinations.
• This produces 10 flat ranges, each of 50 elements: (10*200+50, 50), (11*200+50, 50), …, (19*200+50, 50).
• None are adjacent (gap of 150 elements between each), so no merging occurs.

If the slice were [10:20, :] instead (full inner axis), the split point moves to axis 0 and the 10 individual ranges of 200 elements each are adjacent in memory – they merge into a single range (10*200, 10*200).

flowchart TD
    A["N-D slice<br/>arr[10:20, 50:100]"] --> B["Find split point<br/>axis 1 (not full)"]
    B --> C["Block size = 50"]
    C --> D["Outer indices:<br/>axis 0 → [10..19]"]
    D --> E["10 ranges of 50 elements"]
    E --> F["Merge adjacent?<br/>No — gap of 150"]
    F --> G["decode_range()<br/>10 × (offset, 50)"]

Range Threshold Heuristic

Even when partial reads are technically available, reading many small ranges can be slower than decoding the entire array – especially for compressed data where decompression has fixed overhead per block.

The backend uses a ratio-based heuristic controlled by the range_threshold parameter (default 0.5):

Rule: partial reads are used only when the total number of requested elements is less than range_threshold × total_elements.

With the default of 0.5, if you request more than 50% of the array, the backend falls back to a full decode and slices in memory. Lower values make the backend more aggressive about using partial reads; higher values make it prefer full decodes.

# More aggressive partial reads (use when each range is cheap, e.g. uncompressed)
ds = xr.open_dataset("file.tgm", engine="tensogram", range_threshold=0.3)

# Almost always full decode (use when decode overhead is very low)
ds = xr.open_dataset("file.tgm", engine="tensogram", range_threshold=0.9)

Installation

uv venv .venv && source .venv/bin/activate   # if not already in a virtualenv
uv pip install tensogram-xarray

This pulls in tensogram and xarray as dependencies. The xarray backend is registered automatically via entry points – no extra configuration needed.

>>> import xarray as xr
>>> "tensogram" in xr.backends.list_engines()
True

For dask support:

source .venv/bin/activate   # if not already in the virtualenv
uv pip install "tensogram-xarray[dask]"

Error Handling

The backend reports errors with enough context for diagnosis. Common error scenarios and their messages:

Hash Verification and Partial Reads

When verify_hash=True is passed, xxh3 hash verification is performed on full object reads (decode_object) only. Partial reads via decode_range() intentionally skip hash verification because:

• Partial reads decode only a subset of the payload, so the full-object hash cannot be validated.
• The purpose of partial reads is to minimise I/O; verifying the hash would require reading the entire payload, defeating the optimisation.

This means that for lazily-loaded arrays, hash verification happens when a slice triggers a full-object decode (i.e. when the requested fraction exceeds range_threshold), but not when partial decode_range() is used.

Logging

The backend uses Python’s standard logging module. To see partial-read fallback diagnostics:

import logging
logging.getLogger("tensogram_xarray").setLevel(logging.DEBUG)

To see merge data-loss warnings (enabled by default at WARNING level):

import logging
logging.basicConfig(level=logging.WARNING)

Dask Integration

Tensogram supports Dask natively through its xarray backend. When you open a .tgm file with chunks={}, xarray wraps every tensor variable in a dask.array.Array. No data is read from disk until you call .compute() or .values.

import xarray as xr
ds = xr.open_dataset("forecast.tgm", engine="tensogram", chunks={})
# ds["temperature"].data is now a dask.array -- zero I/O so far
mean = ds["temperature"].mean().compute()  # data decoded here

This chapter explains how the integration works, walks through a complete example with distributed computation, and covers the performance knobs you can tune.

How It Works

The tensogram xarray backend implements BackendArray, xarray’s lazy-loading protocol. When dask requests a chunk, the backend:

1. Opens the .tgm file and reads the raw message bytes.
2. For small slices on compressors that support random access (none, szip, blosc2, zfp fixed-rate): maps the N-D slice to flat byte ranges and decodes only those ranges via decode_range().
3. For large slices or stream compressors: falls back to full decode_object() and slices in memory.

The BackendArray stores only the file path (no open handles), making it pickle-safe for dask multiprocessing and distributed execution.

flowchart LR
    A["xr.open_dataset<br/>chunks={}"] --> B["BackendArray<br/>(lazy, pickle-safe)"]
    B --> C["dask.array.Array"]
    C -->|".compute()"| D["decode_range()<br/>or decode_object()"]
    D --> E["numpy.ndarray"]

Chunking Strategies

For tensogram files, chunks={} is usually the right choice because each data object is already a self-contained tensor. Finer chunking adds overhead from repeated file opens.

Complete Example: Distributed Statistics over 4-D Tensors

This walkthrough corresponds to examples/python/09_dask_distributed.py. It creates 4 .tgm files representing a 4-D temperature field (time x level x latitude x longitude), then computes statistics entirely through dask’s lazy execution.

Step 1: Create the Data Files

Each file contains 10 data objects (one per pressure level) plus latitude and longitude coordinate arrays:

import numpy as np
import tensogram

def _desc(shape, dtype="float32", **extra):
    return {
        "type": "ntensor", "shape": list(shape), "dtype": dtype,
        "byte_order": "little", "encoding": "none",
        "filter": "none", "compression": "none", **extra,
    }

LEVEL_VALUES = [1000, 925, 850, 700, 500, 400, 300, 200, 100, 50]
NLAT, NLON = 36, 72

with tensogram.TensogramFile.create("temperature_20260401.tgm") as f:
    lat = np.linspace(-87.5, 87.5, NLAT, dtype=np.float64)
    lon = np.linspace(0, 355, NLON, dtype=np.float64)

    objects = [
        (_desc([NLAT], dtype="float64", name="latitude"), lat),
        (_desc([NLON], dtype="float64", name="longitude"), lon),
    ]
    for level_hpa in LEVEL_VALUES:
        field = np.random.default_rng(42).random((NLAT, NLON)).astype(np.float32)
        desc = _desc([NLAT, NLON], name=f"temperature_{level_hpa}hPa")
        objects.append((desc, field))

    f.append({"version": 2}, objects)

Step 2: Open with Dask Lazy Loading

The critical parameters are engine="tensogram" and chunks={}:

import xarray as xr
import tensogram_xarray  # registers the engine

ds = xr.open_dataset(
    "temperature_20260401.tgm",
    engine="tensogram",
    variable_key="name",  # name variables from descriptor "name" field
    chunks={},             # enable dask lazy loading
)

At this point:

• No tensor data has been decoded. Only CBOR metadata was read.
• Each variable is a dask.array.Array:

>>> type(ds["temperature_1000hPa"].data)
<class 'dask.array.core.Array'>

>>> ds["temperature_1000hPa"].shape
(36, 72)

>>> ds["temperature_1000hPa"].chunks
((36,), (72,))

Step 3: Build a 4-D Tensor from Multiple Files

Stack variables across levels within each file, then stack files across time:

import dask
import dask.array as da

# Open all 4 files
paths = ["temperature_20260401.tgm", "temperature_20260402.tgm",
         "temperature_20260403.tgm", "temperature_20260404.tgm"]

datasets = [
    xr.open_dataset(p, engine="tensogram", variable_key="name", chunks={})
    for p in paths
]

# Stack levels within each file, then stack across time
# Build in LEVEL_VALUES order (not alphabetical) so axis matches labels
temp_vars = [f"temperature_{lev}hPa" for lev in LEVEL_VALUES]

all_timesteps = []
for ds in datasets:
    level_arrays = [ds[v].data for v in temp_vars]
    all_timesteps.append(da.stack(level_arrays, axis=0))

full_4d = da.stack(all_timesteps, axis=0)
# Shape: (4, 10, 36, 72) -- (time, level, lat, lon)
# Still lazy -- zero I/O

Step 4: Compute Statistics with Dask

Schedule multiple computations, then execute them in a single dask.compute() call:

# Schedule (lazy -- no computation yet)
global_mean = full_4d.mean()
global_std  = full_4d.std()
global_min  = full_4d.min()
global_max  = full_4d.max()

# Execute all at once (data decoded from .tgm files here)
mean_val, std_val, min_val, max_val = dask.compute(
    global_mean, global_std, global_min, global_max
)

print(f"Mean: {mean_val:.2f} K")
print(f"Std:  {std_val:.2f} K")
print(f"Min:  {min_val:.2f} K")
print(f"Max:  {max_val:.2f} K")

Step 5: Selective Lazy Loading

Only the data you touch is decoded. Slicing the 4-D array triggers decoding of just the relevant chunks:

# Single point: backend uses decode_range() for the tiny slice
# (1 element out of 2592 = 0.04%, well below the 50% threshold)
point = full_4d[0, 0, 18, 0].compute()

# One pressure level across all times: touches 4 backing arrays
level_400 = full_4d[:, 5, :, :].mean().compute()

# Equatorial band: partial range decode for the selected rows
equatorial = full_4d[0, 0, 9:27, :].mean().compute()

Performance Tuning

The range_threshold Parameter

When dask requests a slice, the backend decides between partial decode (decode_range()) and full decode (decode_object()) based on the fraction of requested elements:

Rule: partial reads are used when requested_elements / total_elements <= range_threshold

# More aggressive partial reads
ds = xr.open_dataset("file.tgm", engine="tensogram",
                     chunks={}, range_threshold=0.3)

# Almost always full decode
ds = xr.open_dataset("file.tgm", engine="tensogram",
                     chunks={}, range_threshold=0.9)

Which Compressors Support Partial Reads?

The shuffle filter also disables partial reads (byte rearrangement breaks contiguous ranges). The fallback is always transparent: the full object is decoded and sliced in memory.

Dask Scheduler Choice

Tensogram’s backend is thread-safe (uses a threading.Lock per array). All three dask schedulers work:

# Synchronous (debugging)
dask.config.set(scheduler="synchronous")

# Threaded (default, good for I/O-bound work)
dask.config.set(scheduler="threads")

# Multiprocessing (BackendArray is pickle-safe)
dask.config.set(scheduler="processes")

For large-scale work, dask.distributed also works because the BackendArray stores only the file path (no unpicklable state).

Thread Safety

The TensogramBackendArray uses a per-array threading.Lock to serialise file I/O. This means:

• Multiple dask tasks can read different variables concurrently.
• Reads to the same variable are serialised (no concurrent file opens for the same array).
• The lock is excluded from pickle state and recreated on deserialise.

Installation

For dask support, install the optional dependency:

uv venv .venv && source .venv/bin/activate   # if not already in a virtualenv
uv pip install "tensogram-xarray[dask]"

This pulls in dask[array] alongside tensogram and xarray.

Debugging

Enable debug logging to see when partial reads are used vs full decodes:

import logging
logging.getLogger("tensogram_xarray").setLevel(logging.DEBUG)

You will see messages like:

DEBUG:tensogram_xarray.array:decode_range failed for forecast.tgm msg=0 obj=2,
    falling back to full decode: RangeNotSupported

This is expected for stream compressors and is not an error.

Error Handling

When Errors Are Raised

Key design point: errors in metadata (file not found, bad index, wrong dim_names) surface immediately at open_dataset() time. Errors in data decoding surface at .compute() time because payloads are lazy-loaded.

Partial Read Fallback

When decode_range() fails (e.g. unsupported compressor for partial reads), the backend catches the error and falls back to full decode_object():

except (ValueError, RuntimeError, OSError) as exc:
    logger.debug("decode_range failed ... falling back to full decode: %s", exc)

This fallback is transparent — the user gets correct data regardless. Enable DEBUG logging to see when fallbacks occur.

Dask Worker Errors

File paths are automatically resolved to absolute paths when the dataset is opened. This prevents “file not found” errors when dask sends work to processes with a different working directory.

If a dask worker encounters a decode error, it propagates through dask’s error handling. The traceback will show the tensogram error with file path, message index, and object index for diagnosis.

Edge Cases

Ambiguous Dimension Matching

When coordinate arrays have the same size (e.g. both latitude and longitude have 360 elements), the backend cannot distinguish them by shape alone. The first match gets the coordinate name; the second falls back to a generic dim_N.

Workaround: pass explicit dim_names to disambiguate:

ds = xr.open_dataset("file.tgm", engine="tensogram",
                     dim_names=["latitude", "longitude"], chunks={})

Stacking Files with Different Variables

When stacking multiple .tgm files into a single dask array, verify that every dataset contains the expected variables before stacking:

temp_vars = [f"temperature_{lev}hPa" for lev in LEVEL_VALUES]
for i, ds in enumerate(datasets):
    missing = [v for v in temp_vars if v not in ds.data_vars]
    if missing:
        raise KeyError(f"Dataset {i} missing: {missing}")

Otherwise da.stack() will fail with a confusing KeyError from a deep dask callback.

Zero-Object Messages

A .tgm file containing only metadata frames (no data objects) returns an empty xr.Dataset with no variables. This is valid and does not raise an error.

Scalar (0-D) Tensors

Data objects with shape=() (zero dimensions) are supported. They become scalar xr.Variable objects in the dataset.

Hash Verification with Partial Reads

When verify_hash=True is set, hash verification only runs on full object reads (via decode_object()). Partial reads via decode_range() skip verification because only a subset of the payload is decoded. This means:

• Large slices (above range_threshold) trigger full decode with hash verification.
• Small slices use decode_range() without hash verification.

This is by design. If you need guaranteed hash verification on every access, set range_threshold=0.0 to force full decodes.

Zarr v3 Backend

The tensogram-zarr package implements a Zarr v3 Store backed by .tgm files. This lets you read and write Tensogram data through the standard Zarr Python API.

Installation

uv venv .venv && source .venv/bin/activate   # if not already in a virtualenv
uv pip install tensogram-zarr

Requires zarr >= 3.0, tensogram, and numpy.

Reading a .tgm file through Zarr

import zarr
from tensogram_zarr import TensogramStore

# Open existing .tgm file as a read-only Zarr store
store = TensogramStore.open_tgm("data.tgm")
root = zarr.open_group(store=store, mode="r")

# Browse available arrays
for name, arr in root.members():
    print(f"{name}: shape={arr.shape}, dtype={arr.dtype}")

# Read an array (decoded eagerly at store open, served from memory)
temperature = root["2t"][:]
print(temperature.shape, temperature.mean())

# Access group-level metadata (from GlobalMetadata _extra_)
# The example below shows a MARS namespace; the attributes dict reflects
# whatever namespaces the producer put in the message's GlobalMetadata.
print(root.attrs["mars"])  # {'class': 'od', 'type': 'fc', ...}

How the mapping works

Each .tgm message maps to a Zarr group:

zarr.json                     # root group ← GlobalMetadata
temperature/zarr.json         # array metadata ← DataObjectDescriptor
temperature/c/0/0             # chunk data ← decoded object payload
pressure/zarr.json            # another array
pressure/c/0/0                # its chunk data

graph LR
    TGM[".tgm file"] --> GM["GlobalMetadata"]
    TGM --> OBJ1["Object 0: temperature"]
    TGM --> OBJ2["Object 1: pressure"]
    
    GM --> GZJ["zarr.json (group)"]
    OBJ1 --> AZJ1["temperature/zarr.json"]
    OBJ1 --> CHK1["temperature/c/0/0"]
    OBJ2 --> AZJ2["pressure/zarr.json"]
    OBJ2 --> CHK2["pressure/c/0/0"]

Key design decisions:

• Each TGM data object becomes one Zarr array with a single chunk (chunk shape = array shape)
• Variable names are resolved from metadata via a default lookup path (name, mars.param, param, mars.shortName, shortName), or a custom dot-path you supply
• TGM encoding metadata is preserved in Zarr array attributes under _tensogram_* keys
• Duplicate variable names get a numeric suffix (field, field_1)

Variable naming

By default, the store tries these metadata paths to name arrays:

1. name
2. mars.param
3. param
4. mars.shortName
5. shortName
6. Falls back to object_<index>

You can override with any dot-path, including non-MARS vocabularies:

# Weather pipeline using MARS
store = TensogramStore.open_tgm("weather.tgm", variable_key="mars.param")

# Neuroimaging pipeline using BIDS
store = TensogramStore.open_tgm("scans.tgm", variable_key="bids.task")

# Custom vocabulary
store = TensogramStore.open_tgm("data.tgm", variable_key="product.name")

Multi-message files

By default the store reads message 0. Select a different message with message_index:

store = TensogramStore.open_tgm("multi.tgm", message_index=2)

Writing a .tgm file through Zarr

import numpy as np
import zarr
from tensogram_zarr import TensogramStore

store = TensogramStore("output.tgm", mode="w")
root = zarr.open_group(store=store, mode="w")

# Create arrays — data is buffered in memory
root.create_array("temperature", data=np.random.rand(100, 200).astype(np.float32))
root.create_array("pressure", data=np.array([1000, 925, 850, 700], dtype=np.float64))

# Close flushes to .tgm
store.close()

The write path assembles all arrays into a single TGM message when the store is closed.

Context manager

with TensogramStore("data.tgm", mode="r") as store:
    root = zarr.open_group(store=store, mode="r")
    data = root["temperature"][:]
# Store automatically closed

Supported data types

Byte range support

The store supports Zarr’s ByteRequest types for efficient partial reads:

• RangeByteRequest(start, end) — read a byte range
• OffsetByteRequest(offset) — read from offset to end
• SuffixByteRequest(suffix) — read last N bytes

Comparison with tensogram-xarray

Use tensogram-zarr when you need direct Zarr API access or write support. Use tensogram-xarray when you want automatic coordinate detection and multi-message merging.

Edge cases and limitations

Variable name sanitization

If a metadata value used as a variable name contains / or \, those characters are replaced with _ to prevent spurious directory nesting in the virtual key space. Empty names become _.

mars.param = "temperature/surface"  →  variable name "temperature_surface"

Duplicate variable names

When multiple objects resolve to the same name, suffixes are appended: field, field_1, field_2, etc.

Zero-object messages

A message with no data objects is valid (metadata-only). The store produces a root group with attributes but no arrays.

Single chunk per array

Each TGM data object maps to a Zarr array with chunk_shape == array_shape (one chunk). There is no sub-chunking; partial reads within the array are handled by Zarr’s byte-range support against the single chunk. If a Zarr writer attempts to store multiple chunks for the same variable, a ValueError is raised — TensogramStore does not silently drop extra chunks.

Out-of-range message index

If message_index exceeds the number of messages in the file, an IndexError is raised. Negative indices are rejected with ValueError.

bfloat16 dtype

bfloat16 maps to Zarr data type "bfloat16" but is stored as raw 2-byte values (<V2 numpy dtype) since numpy has no native bfloat16 type. Use ml_dtypes.bfloat16 for interpretation.

Byte order handling

The read path normalises all chunk data to little-endian (matching the Zarr bytes codec default). The write path respects byte_order from the Zarr codecs metadata — if a big-endian bytes codec is specified, the data is byte-swapped before encoding to TGM.

JSON serialization (RFC 8259)

serialize_zarr_json() converts non-finite float values to their Zarr v3 string sentinels ("NaN", "Infinity", "-Infinity") so the output is valid RFC 8259 JSON.

Write path byte-count validation

When flushing to .tgm, the store validates that chunk byte count matches product(shape) * dtype_size. A mismatch raises ValueError with the expected and actual counts.

close() exception safety

If _flush_to_tgm() fails during close(), the store is still marked as closed (_is_open = False). The exception propagates normally — partial writes do not corrupt the file since TGM messages are written atomically.

When used as a context manager and an exception is already in flight, flush errors are logged at WARNING level instead of replacing the original exception.

Error handling

All errors surface with enough context for debugging:

Errors from the underlying Rust tensogram library are wrapped with Python-level context so users see which file, message, and variable caused the problem.

anemoi-inference Integration

The tensogram-anemoi package provides a plug-and-play output for anemoi-inference, the ECMWF framework for running AI-based weather forecast models. Once installed, anemoi-inference automatically discovers the plugin via Python entry points — no code changes to anemoi-inference are required.

Installation

pip install tensogram-anemoi

Or from source:

pip install -e python/tensogram-anemoi/

Usage

In an anemoi-inference run config, specify tensogram as the output:

output:
  tensogram:
    path: forecast.tgm

All forecast steps are written to a single .tgm file as they are produced. Remote destinations (S3, GCS, Azure, …) are supported via fsspec:

output:
  tensogram:
    path: s3://my-bucket/forecast.tgm
    storage_options:
      key: ...
      secret: ...

Configuration options

All options after path must be supplied as keyword arguments.

Pressure-level stacking

When stack_pressure_levels=True, all fields sharing the same GRIB param are merged into a single 2-D object of shape (n_grid, n_levels), sorted by level ascending. The "mars" namespace carries "levelist": [500, 850, ...] instead of a scalar "level" (following standard MARS convention). Non-pressure-level fields are always written as individual 1-D objects.

output:
  tensogram:
    path: forecast.tgm
    stack_pressure_levels: true

Simple packing

For compact storage, use simple_packing with a bits value:

output:
  tensogram:
    path: forecast.tgm
    encoding: simple_packing
    bits: 16
    compression: zstd

Coordinate arrays (lat/lon) are never lossy-encoded; only field arrays are packed.

Metadata reference

Each .tgm file produced by tensogram-anemoi contains one message per forecast step. This section documents exactly what is stored in each message and how to read it with the raw tensogram Python API.

Opening a file

import tensogram

tgm = tensogram.TensogramFile.open("forecast.tgm")
print(len(tgm), "steps")

meta, objects = tgm[0]   # first step

meta is the decoded message metadata. objects is a list of (descriptor, array) pairs, one entry per object in the message.

Object layout

Every message has the following fixed layout:

meta, objects = tgm[0]

lat_desc, lat_arr = objects[0]   # latitudes
lon_desc, lon_arr = objects[1]   # longitudes
fld_desc, fld_arr = objects[2]   # first field

The coordinate names "grid_latitude" and "grid_longitude" are intentionally distinct from the standard "latitude" / "longitude" names so that all objects in a message share a single flat grid dimension rather than each coordinate spawning its own dimension.

base[i] — per-object metadata

Each object has a corresponding entry in meta.base:

for i, entry in enumerate(meta.base):
    print(i, entry)

Every entry contains:

"anemoi" namespace

For coordinates, "variable" is "latitude" or "longitude" (the canonical name, not the "grid_*" name stored in "name"):

assert meta.base[0]["name"] == "grid_latitude"
assert meta.base[0]["anemoi"]["variable"] == "latitude"

assert meta.base[1]["name"] == "grid_longitude"
assert meta.base[1]["anemoi"]["variable"] == "longitude"

For fields, "variable" is the internal anemoi-inference name (e.g. "t500" for 500 hPa temperature, "2t" for 2 m temperature):

assert meta.base[2]["anemoi"]["variable"] == "2t"

"mars" namespace

Coordinate objects carry no "mars" key. Every field object carries a "mars" dict combining keys from the anemoi-inference checkpoint with the temporal keys derived from the forecast state:

Temporal keys (present on every field object):

Checkpoint keys (present when available in the model checkpoint):

Reading field metadata:

meta, objects = tgm[0]

# Surface field (e.g. 2 m temperature)
entry = meta.base[2]
print(entry["name"])                    # "2t"
print(entry["anemoi"]["variable"])      # "2t"
print(entry["mars"]["param"])           # "2t"
print(entry["mars"]["date"])            # "20240101"
print(entry["mars"]["time"])            # "0000"
print(entry["mars"]["step"])            # 6

# Pressure-level field (unstacked)
entry = meta.base[3]
print(entry["mars"]["param"])           # "t"
print(entry["mars"]["levtype"])         # "pl"
print(entry["mars"]["level"])           # 500

With stack_pressure_levels=True, the pressure-level group has "levelist" instead of "level", and the array is 2-D:

entry = meta.base[2]                    # stacked t group
print(entry["mars"]["levelist"])        # [500, 850, 1000]
print(entry["mars"]["param"])           # "t"

desc, arr = objects[2]
print(arr.shape)                        # (n_grid, 3)  — columns sorted by level

meta.extra — message-level metadata

meta.extra carries metadata that applies to the whole message rather than individual objects.

"dim_names" — axis-size hints

dim_names = meta.extra["dim_names"]
# e.g. {"21600": "values"}
# or   {"21600": "values", "3": "level"}  (with stack_pressure_levels=True)

dim_names maps the string representation of an axis length to a semantic name. It exists to allow downstream tools to assign meaningful axis names without requiring any anemoi-specific knowledge. The grid axis is always labelled "values"; when pressure-level stacking is enabled, each unique level-axis size is labelled "level".

Object descriptors

Each (descriptor, array) pair returned by objects[i] gives low-level encoding detail:

desc, arr = objects[2]

print(desc.dtype)        # "float32" or "float64"
print(desc.shape)        # [n_grid] for flat, [n_grid, n_levels] for stacked
print(desc.encoding)     # "none" or "simple_packing"
print(desc.compression)  # "zstd", "lz4", etc.

Coordinate arrays are always float64 regardless of the dtype setting. Field arrays use the configured dtype ("float32" by default), promoted to float64 automatically when encoding="simple_packing".

Full inspection example

import tensogram

tgm = tensogram.TensogramFile.open("forecast.tgm")

for step_idx, (meta, objects) in enumerate(tgm):
    print(f"\n--- step {step_idx} ---")

    # Dimension hints
    print("dim_names:", meta.extra.get("dim_names", {}))

    for i, entry in enumerate(meta.base):
        desc, arr = objects[i]
        anemoi = entry.get("anemoi", {})
        mars = entry.get("mars", {})

        print(
            f"  [{i}] name={entry['name']!r:20s}"
            f"  variable={anemoi.get('variable')!r:10s}"
            f"  shape={arr.shape}"
            f"  dtype={desc.dtype}"
            + (f"  step={mars.get('step')}" if mars else "")
        )

Example output for a single step with surface fields and stacked pressure levels:

--- step 0 ---
dim_names: {'21600': 'values', '3': 'level'}
  [0] name='grid_latitude'    variable='latitude'   shape=(21600,)  dtype=float64
  [1] name='grid_longitude'   variable='longitude'  shape=(21600,)  dtype=float64
  [2] name='2t'               variable='2t'         shape=(21600,)  dtype=float32  step=6
  [3] name='t'                variable='t'          shape=(21600, 3)  dtype=float32  step=6
  [4] name='u'                variable='u'          shape=(21600, 3)  dtype=float32  step=6

Free-Threaded Python

Tensogram supports free-threaded Python (CPython 3.13t / 3.14t), which removes the Global Interpreter Lock (GIL) and allows true multi-threaded parallelism from Python.

What This Means

On standard CPython, the GIL serializes access to the interpreter — only one thread runs Python code at a time. Tensogram already releases the GIL during Rust computation (py.detach()), which helps, but the GIL is still re-acquired for numpy array construction and Python object creation.

On free-threaded CPython (3.13t / 3.14t), there is no GIL at all. Multiple threads can call tensogram.encode() and tensogram.decode() in true parallel. Use the included benchmark (rust/benchmarks/python/bench_threading.py) to measure scaling on your hardware.

Building for Free-Threaded Python

Install a free-threaded Python build:

# uv (recommended)
uv python install cpython-3.14+freethreaded

# Or via pyenv
pyenv install 3.14t

Build tensogram:

uv venv .venv --python python3.14t
source .venv/bin/activate
uv pip install maturin "numpy>=2.1"
cd python/bindings && maturin develop --release

Verify the GIL is disabled:

import sys
print(sys._is_gil_enabled())  # False

Thread-Safe API

All tensogram read operations are safe to call from multiple threads simultaneously:

import threading
import numpy as np
import tensogram

data = np.random.randn(1_000_000).astype(np.float32)
meta = {"version": 2, "base": [{}]}
desc = {"type": "ntensor", "shape": [1_000_000], "dtype": "float32"}
msg = tensogram.encode(meta, [(desc, data)])

def decode_worker():
    for _ in range(100):
        result = tensogram.decode(msg)

threads = [threading.Thread(target=decode_worker) for _ in range(8)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Each thread can independently:

• Encode and decode messages
• Scan buffers
• Validate messages and files
• Read from TensogramFile instances (same handle or separate handles)
• Use StreamingEncoder (separate instances per thread)

TensogramFile Thread Safety

All read methods on TensogramFile (decode_message, read_message, decode_metadata, decode_descriptors, decode_object, decode_range, __getitem__, __len__, __iter__) use &self and support concurrent access from multiple threads on the same handle:

f = tensogram.TensogramFile.open("data.tgm")

def worker(thread_id):
    # Multiple threads can read from the same handle concurrently
    msg = f.decode_message(thread_id % len(f))

threads = [threading.Thread(target=worker, args=(i,)) for i in range(8)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Only append() requires exclusive access — calling it while other threads are reading will raise RuntimeError (PyO3 runtime borrow check).

Benchmark Results

Measured on Linux x86_64 (20 cores), NumPy 2.4.4, release build. Same-version paired comparisons to isolate the GIL effect.

All scaling below comes from Python-level threading (threading.Thread). Each call into Rust is single-threaded — there is no rayon or internal parallelism within a single encode/decode. The speedups reflect multiple Python threads entering Rust concurrently via py.detach(). A future Rust-level parallel pipeline would multiply on top of these numbers.

Headline: Decode Throughput (1M float32, no codec)

Headline: Encode Throughput (1M float32, no codec)

Small Messages (16K float32, no codec)

Other Operations (1M float32)

Scan (message boundary detection — ~0.2µs/call, GIL overhead dominates):

Validate (full message validation — CPU-bound, scales well on both):

Decode-range (sub-array extraction, 2x1K slices from 1M):

Iter-messages (3 messages, 100K f32 each):

Key Takeaways

Methodology: 5 runs per configuration, median reported. 200–500 warmup iterations for fast operations.

• Validate scales near-linearly on both GIL and free-threaded — 8.9x (GIL) and 10.5x (free-threaded) at 16 threads. This is the most CPU-bound operation and benefits fully from py.detach() regardless of GIL.
• Free-threaded decode scales to 4.7x at 8 threads for the headline workload (1M f32, no codec). GIL-enabled stays near 1.0x because numpy array construction dominates and serializes under the GIL.
• GIL-enabled decode-range plateaus at ~1.7x — py.detach() allows 2 threads of overlap but the lightweight result construction can’t overlap further. Free-threaded reaches 11.8x at 16 threads.
• Scan shows dramatic free-threaded scaling — free-threaded reaches 15.5x at 16 threads. GIL-enabled scales to 2.0x at 4 threads but drops back at higher thread counts due to contention.
• Small messages (16K) reach 13.0x at 16 threads on free-threaded (3.14t) vs 1.2x on GIL-enabled.
• iter_messages scales to 4.7x at 8 threads on free-threaded, then drops due to contention. GIL-enabled stays flat (~1.0x).
• Single-thread trade-off — free-threaded single-thread performance varies by workload: decode is within ~5% of GIL-enabled (396 vs 408 op/s on 3.14), encode varies by version (3.14t is 18% faster than 3.14, while 3.13t is 6% slower than 3.13). Validate is ~20% slower (4,347 vs 5,457 op/s) and scan ~4x slower due to reference counting overhead on returned Python objects — both recover by 2 threads.

These numbers are machine-specific. Run the benchmark on your hardware:

python rust/benchmarks/python/bench_threading.py              # full suite
python rust/benchmarks/python/bench_threading.py --headline   # quick comparison
python rust/benchmarks/python/bench_threading.py --quick      # CI smoke test

Reference Comparison: Tensogram (Python) vs ecCodes (C)

This section measures Tensogram’s Python throughput against ecCodes’ native C performance on the same pipeline — 10 million float64 values (80 MiB), 24-bit simple packing + szip compression — as a concrete reference point. The pipeline is common in operational weather forecasting and is representative of scientific-quantisation workloads more broadly.

What we measured

Both sides are measured end-to-end: from a float64 array to serialized compressed bytes (encode), and back to a float64 array (decode). Both include metadata serialization, framing, and integrity overhead — not just the raw packing step.

ecCodes (C, single-threaded): The Rust benchmark (rust/benchmarks/src/bin/grib_comparison.rs) calls ecCodes’ C library directly via FFI. Encode: allocate a GRIB handle, configure the grid (10M regular lat/lon), set packing type to CCSDS at 24 bits, write the values array, serialize to GRIB bytes. Decode: load the GRIB message from bytes, extract the values array. No Python involved. Median of 10 iterations, 3 warmup.

Tensogram (Python, multi-threaded): The same 10M float64 values, same 24-bit quantization, same szip compression. Encode: pass a numpy array + CBOR metadata dict to tensogram.encode(), which crosses the PyO3 boundary, quantizes, compresses, frames, computes the integrity hash, and returns Python bytes. Decode: pass bytes to tensogram.decode(), which deframes, decompresses, dequantizes, and returns a numpy array. Each Python thread makes independent encode/decode calls. The GIL is released during the Rust computation.

Why scaling depends on the codec

Threading helps most when the Rust computation (compression, quantization) is the dominant cost. With simple packing + szip, each encode/decode spends ~170 ms in Rust and ~20 ms in Python/numpy — so ~89% of the time runs with the GIL released and threads scale well. Without compression, the Rust work is trivial (~1 ms) and the Python overhead limits parallelism.

The tables above measure uncompressed data to isolate the threading mechanism. The results below use the production pipeline (24-bit packing + szip) and show what real workloads achieve.

Results

ecCodes CCSDS (Rust FFI, single-threaded): 870 MB/s encode, 531 MB/s decode.

Tensogram from Python (free-threaded 3.14t, 5-run median, 10M float64 24-bit packing+szip):

Decode:

Encode:

Single-threaded Tensogram from Python is slower than ecCodes from C (the PyO3 boundary costs ~10-15% on decode, ~50% on encode due to numpy data extraction for 80 MiB). But at 2 threads, decode already surpasses ecCodes. At 4 threads, both encode and decode exceed ecCodes. At 8 threads, decode reaches 4.9x ecCodes throughput — from Python.

Requirements

• Python >= 3.13t for free-threaded mode (3.12/3.13 GIL-enabled also works)
• NumPy >= 2.1 (free-threaded support)
• maturin >= 1.8 (free-threaded wheel building)

Known Limitations

Inherent:

• Shared mutable numpy arrays across threads can cause data races (same as any Python threading)
• xarray and zarr backends have their own threading models (dask, zarr locking)

By design:

• TensogramFile read methods (decode_message, read_message, __getitem__, etc.) support concurrent access from multiple threads on the same handle. Only append() requires exclusive access.
• bytes inputs to decode/scan/validate are zero-copy across the GIL release. bytearray inputs are copied once internally by PyO3.
• iter_messages / PyBufferIter own a full buffer copy (the buffer must outlive iteration).

Multi-Threaded Coding Pipeline

Since v0.13.0 Tensogram exposes a caller-controlled thread budget that spreads encoding and decoding work across a scoped pool of workers. The feature is off by default — existing code paths produce byte-identical output to previous releases until the caller opts in.

This page covers:

• The threads option
• Cross-language parity
• Axis-A vs axis-B dispatch
• Determinism contract
• Environment variable override
• Interaction with free-threaded Python
• Benchmarks and tuning

The threads option

All four bindings expose a threads: u32 option on encode and decode entry points:

#![allow(unused)]
fn main() {
use tensogram::{encode, decode, EncodeOptions, DecodeOptions};

// Encode with a 4-thread pool:
let msg = encode(&meta, &descriptors, &EncodeOptions {
    threads: 4,
    ..Default::default()
})?;

// Decode with an 8-thread pool:
let (meta, objs) = decode(&msg, &DecodeOptions {
    threads: 8,
    ..Default::default()
})?;
}

import tensogram

msg = tensogram.encode(meta, descriptors, threads=4)
decoded = tensogram.decode(msg, threads=8)

tensogram::encode_options enc{};
enc.threads = 4;
auto bytes = tensogram::encode(meta_json, objects, enc);

tensogram::decode_options dec{};
dec.threads = 8;
auto msg = tensogram::decode(buf, len, dec);

tgm_encode(meta_json, data_ptrs, data_lens, num_objects,
           "xxh3", /* threads= */ 4, &out);
tgm_decode(buf, len, /* verify_hash */ 0, /* native_byte_order */ 1,
           /* threads= */ 8, &msg);

tensogram --threads 8 merge -o merged.tgm a.tgm b.tgm
TENSOGRAM_THREADS=4 tensogram split -o 'part_[index].tgm' input.tgm

Value semantics

Cross-language parity

Every language binding exposes the same threads option on every encode/decode entry point that does CPU work. Metadata-only commands (scan, describe, list) never accept it because they never decode payloads.

Legend: ✅ = full support, ⚠ = flag accepted but currently a no-op (tracked in IDEAS), — = not applicable at this layer.

Threshold behaviour

For very small payloads the pool-build cost (~10–100 µs) outweighs any parallelism gain. The library transparently skips the pool when the total payload bytes are below a threshold (default 64 KiB). The threshold is tunable:

#![allow(unused)]
fn main() {
EncodeOptions {
    threads: 8,
    parallel_threshold_bytes: Some(0),       // always parallel
    // parallel_threshold_bytes: Some(usize::MAX), // never parallel
    ..Default::default()
}
}

Axis-A vs axis-B dispatch

The threads budget is spent along one of two axes:

• Axis A — across objects. When a message carries multiple data objects and none of them uses an axis-B-friendly codec, rayon par_iter() runs the encode/decode pipeline for each object on a worker in parallel. Output order is preserved exactly.

• Axis B — inside one codec. When any stage is axis-B-friendly (simple_packing encoding, shuffle filter, blosc2 or zstd compression), the budget flows into the codec’s internal parallelism:

  Stage	How it uses the budget
  simple_packing encode/decode	Chunked par_iter with byte-aligned chunk sizes — output bytes remain identical.
  shuffle / unshuffle	Parallelise the outer byte_idx loop (shuffle) or output-chunk scatter (unshuffle).
  blosc2	CParams::nthreads / DParams::nthreads — decompress path stays single-threaded in v0.13.0.
  zstd FFI	NbWorkers libzstd parameter on compress; decompress is inherently sequential.

Policy

Tensogram messages tend to carry a small number of very large objects, so the library prefers axis B when any codec can use it:

This decision happens once per encode/decode call based on the descriptors. Nothing is configurable beyond threads and parallel_threshold_bytes — the policy is deterministic.

Determinism contract

v0.13.0 makes two different promises depending on which codecs you use.

Transparent codecs — byte-identical across thread counts

These stages produce the same encoded bytes regardless of threads:

• encoding = "none"
• encoding = "simple_packing" (at any bits-per-value)
• filter = "none"
• filter = "shuffle"
• compression ∈ {none, lz4, szip, zfp, sz3}

Encoded payload bytes are bit-exact identical for threads ∈ {0, 1, 2, 4, 8, 16, ...}. This is exercised by the rust/tensogram/tests/threads_determinism.rs integration suite.

Opaque codecs — lossless round-trip, may differ

compression ∈ {blosc2, zstd} hand off work to third-party C libraries. When their internal thread pool is asked to run in parallel, blocks land in the output frame in worker completion order. The compressed bytes may therefore differ from the sequential path — but every variant round-trips losslessly:

• Encode with threads=8, decode with threads=0 → same decoded values as a pure sequential round-trip.
• Golden files (produced with threads=0) are still byte-for-byte stable across releases because the default path is unchanged.

Why this matters

Determinism across thread counts is the core property that lets Tensogram users turn threads on in production without worrying about cache keys, deduplication hashes, or reproducible builds breaking. The invariant is tested at every layer — Rust, Python, C FFI, C++ wrapper — with a sweep over {0, 1, 2, 4, 8}.

Interaction with integrity hashing

The xxh3-64 integrity hash attached to every data object (EncodeOptions.hash_algorithm = Some(Xxh3), on by default) is a pure function of the final encoded bytes. Hashing runs in the calling thread after any intra-codec parallelism has joined; each object owns its own Xxh3Default hasher on the stack and the hasher is never shared across threads.

As a consequence the hash follows the same contract as the encoded bytes:

For opaque codecs the hash is still internally consistent — descriptor.hash == xxh3_64(encoded_payload) always holds for the bytes that were actually written — it just may not match a hash computed at a different thread count. verify_hash on decode always succeeds regardless of the threads value used at encode time.

Since the hash is folded into the codec output in lockstep (see plans/DONE.md → Hash-while-encoding), turning on threads has no additional hash-computation cost beyond what threading already does to the encoded bytes themselves.

Environment variable override

TENSOGRAM_THREADS is consulted only when the caller-provided threads is 0. This matches the existing TENSOGRAM_COMPRESSION_BACKEND pattern:

# One-shot invocation — every library call inherits the budget.
TENSOGRAM_THREADS=4 python my_pipeline.py

# Explicit option still wins.
tensogram.encode(meta, descs, threads=0)   # sequential (env honoured)
tensogram.encode(meta, descs, threads=1)   # single-threaded (env ignored)
tensogram.encode(meta, descs, threads=16)  # 16 workers (env ignored)

The env var is parsed once per process (OnceLock), so changing it mid-run has no effect.

Interaction with free-threaded Python

threads is orthogonal to Python threading. For CPython 3.13+ built with --disable-gil, you can combine:

• Python threads — run multiple Tensogram calls concurrently.
• Tensogram threads — each call uses rayon internally.

The PyO3 bindings always release the GIL around encode/decode, so the two dimensions compose cleanly. Be careful about total thread count: N Python threads × M Tensogram threads creates N×M workers. The safest starting point is one dimension at a time.

Benchmarks and tuning

The threads-scaling benchmark measures encode/decode throughput for 7 representative codec combinations across a sweep of thread counts:

cargo build --release -p tensogram-benchmarks
./target/release/threads-scaling \
    --num-points 16000000 \
    --iterations 5 \
    --warmup 2 \
    --threads 0,1,2,4,8,16

Output columns (per case × thread count):

• enc (ms), dec (ms) — median wall time over iterations.
• enc MB/s, dec MB/s — throughput based on the original byte size.
• ratio — compressed size as a percentage of original.
• size (MiB) — compressed size.
• enc x, dec x — speedup relative to the threads=0 baseline.

See the Benchmark Results page for numbers on a reference machine.

Tuning recommendations

1. Start with threads=0. The default is deterministic, well tested, and fast for small-to-medium payloads.
2. Turn it on globally via env. TENSOGRAM_THREADS=$(nproc) is a reasonable starting point for CPU-bound data-movement pipelines. Leave the in-process tensogram calls as threads=0 unless you need finer control per call.
3. Measure before tuning. On small payloads the threshold keeps you safe, but the sweet spot for large tensors varies by codec. For simple_packing + szip, 2–4 threads already reaches diminishing returns; for blosc2 it can scale further.
4. Do not stack Python threads × Tensogram threads unless you know the total fits your CPU budget. Over-subscription destroys throughput.

Benchmarks

Tensogram ships with a benchmark suite that measures all encoding and compression combinations on synthetic data. It produces tabular comparisons of speed, compressed size, and decode fidelity. The benchmarks can be re-run at any time to measure the effect of changes.

Codec Matrix Benchmark

Tests all valid encoder × compressor × bit-width combinations on 16 million synthetic float64 values.

Quick start

cargo run --release -p tensogram-benchmarks --bin codec-matrix

Override parameters with CLI flags:

cargo run --release -p tensogram-benchmarks --bin codec-matrix -- \
    --num-points 16000000 \
    --iterations 10 \
    --warmup 3 \
    --seed 42

Combinations measured

For actual results, see Benchmark Results.

How to read the results

The results page splits each benchmark into a performance table (timing, throughput, compressed size) and a fidelity table (error norms for lossy codecs).