TypeScript API

Tensogram ships @ecmwf.int/tensogram, a TypeScript package that wraps the WebAssembly build with typed, idiomatic helpers. Use it in any modern browser or Node ≥ 20.

The package exposes typed encode / decode / scan, dtype-aware payload views, metadata helpers, progressive streaming decode, the TensogramFile file / URL helper, the validate wrapper, encodePreEncoded, and first-class float16 / bfloat16 / complex* view classes.

Installation

The package is not yet published to npm. Build it locally:

# First, build the WebAssembly blob from the Rust source
cd typescript
npm install
npm run build:wasm   # runs wasm-pack build -t web -d typescript/wasm
npm run build        # runs wasm-pack + tsc

Or use the top-level Makefile:

make ts-build        # build WASM + tsc
make ts-test         # vitest
make ts-typecheck    # strict tsc --noEmit on src + tests

Quick start

import {
  init, encode, decode,
  type DataObjectDescriptor,
  type GlobalMetadata,
} from '@ecmwf.int/tensogram';

// One-time WASM initialisation (idempotent)
await init();

// ── Encode ────────────────────────────────────────────────────────────
const temps = new Float32Array(100 * 200);
for (let i = 0; i < temps.length; i++) temps[i] = 273.15 + i / 100;

const meta: GlobalMetadata = { version: 3 };
const descriptor: DataObjectDescriptor = {
  type: 'ntensor',
  ndim: 2,
  shape: [100, 200],
  strides: [200, 1],
  dtype: 'float32',
  byte_order: 'little',
  encoding: 'none',
  filter: 'none',
  compression: 'none',
};

const msg: Uint8Array = encode(meta, [{ descriptor, data: temps }]);

// ── Decode ────────────────────────────────────────────────────────────
const { metadata, objects } = decode(msg);
const arr = objects[0].data();  // Float32Array (inferred from dtype)
console.log(arr.length);        // 20000

API surface

init(opts?)

Loads and instantiates the WASM blob. Must be awaited before any other function is called. Safe to call multiple times — subsequent calls reuse the same instance.

await init();                                              // defaults
await init({ wasmInput: new URL('...', import.meta.url) });  // custom location

encode(metadata, objects, opts?)

Returns: Uint8Array containing the complete wire-format message.

decode(buf, opts?)

Returns: { metadata: GlobalMetadata, objects: DecodedObject[], close() }.

decodeMetadata(buf)

Returns only the metadata; does not touch any payload bytes.

decodeObject(buf, index, opts?)

O(1) seek to object index, decoding only that object.

scan(buf)

Returns Array<{ offset: number; length: number }> for each Tensogram message found in a (potentially multi-message) buffer. Garbage between messages is silently skipped.

DecodedObject / DecodedFrame

interface DecodedObject {
  readonly descriptor: DataObjectDescriptor;
  /** Copy into the JS heap.  Safe across WASM memory growth. */
  data(): TypedArray;
  /** Zero-copy view.  Invalidated if WASM memory grows. */
  dataView(): TypedArray;
  readonly byteLength: number;
}

interface DecodedFrame extends /* structurally */ DecodedObject {
  /** The matching `base[i]` entry from the containing message. */
  readonly baseEntry: BaseEntry | null;
  close(): void;
}

The returned array type is picked from descriptor.dtype:

getMetaKey(meta, path)

Dot-path lookup matching the Rust / Python / CLI first-match-across-base semantics: searches base[0], base[1], …, skipping the _reserved_ key in each, then falls back to _extra_.

getMetaKey(meta, 'mars.param')      // 'base[0].mars.param' first match
getMetaKey(meta, '_extra_.source')  // explicit _extra_ prefix

Returns undefined if the key is missing (never throws).

computeCommon(meta)

Mirror of tensogram::compute_common. Returns a Record<string, CborValue> of keys that are present with identical values in every entry of meta.base. Useful for display and merge operations.

Error classes

All errors thrown from this package are instances of the abstract TensogramError class. Eight concrete subclasses match the Rust TensogramError variants plus the TS-layer InvalidArgumentError and StreamingLimitError:

import {
  TensogramError,
  FramingError,
  MetadataError,
  EncodingError,
  CompressionError,
  ObjectError,
  IoError,
  RemoteError,
  HashMismatchError,
  InvalidArgumentError,
  StreamingLimitError,
} from '@ecmwf.int/tensogram';

try {
  decode(corruptBuffer);
} catch (err) {
  if (err instanceof FramingError) {
    console.error('bad wire format:', err.message);
  } else if (err instanceof HashMismatchError) {
    console.error('integrity failure:', err.expected, err.actual);
  } else {
    throw err;
  }
}

Memory model

• Safe-copy by default. object.data() / frame.data() always allocate a new TypedArray on the JS heap. It remains valid even after the underlying DecodedMessage / DecodedFrame is freed or WASM memory grows.
• Zero-copy opt-in. object.dataView() / frame.dataView() return a view directly into WASM linear memory. It is invalidated the next time any WASM call grows linear memory — which can happen on the next encode() / decode(). Read the view immediately or copy it.
• Explicit cleanup. DecodedMessage, DecodedFrame, and TensogramFile all expose .close() to release WASM-side memory. A FinalizationRegistry also calls .free() on the underlying WASM handle when the wrapper is garbage-collected, but explicit .close() is strongly recommended for deterministic cleanup.

Streaming decode

Use decodeStream(readable, opts?) to progressively decode a ReadableStream<Uint8Array>. Works against any stream source — fetch().body, a Node Readable.toWeb(), a Blob.stream(), or a hand-rolled ReadableStream.

import { decodeStream } from '@ecmwf.int/tensogram';

const res = await fetch('/data.tgm');
for await (const frame of decodeStream(res.body!)) {
  render(frame.descriptor.shape, frame.data());
  frame.close();
}

Options:

Key behaviours:

• Chunk-boundary tolerant. A message can be split across any number of chunks. The decoder accumulates until a complete message is seen, then emits every object as a separate frame.
• Corruption resilient. A single bad message is skipped; the iterator keeps going with subsequent messages. Pass onError to observe the skips.
• Early break is safe. Breaking out of the for await loop runs the generator’s finally block, which releases the stream reader and frees the decoder.
• AbortSignal cancels cleanly. Firing the signal cancels the underlying reader; the generator throws whatever error the signal carries.

File API

TensogramFile gives you random-access reads over a .tgm file, whether it lives on the local file system, behind an HTTPS URL, or already in memory.

import { TensogramFile } from '@ecmwf.int/tensogram';

// Node: from the local file system
const file = await TensogramFile.open('/data/input.tgm');

// Browser or Node: over HTTPS
const file = await TensogramFile.fromUrl('https://example.com/input.tgm');

// Any runtime: from pre-loaded bytes
const file = TensogramFile.fromBytes(uint8ArrayFromSomewhere);

All three factories produce an identical object:

interface TensogramFile extends AsyncIterable<DecodedMessage> {
  readonly messageCount: number;
  readonly byteLength: number;
  readonly source: 'local' | 'remote' | 'buffer';

  // Whole-message access (unchanged since Scope B).
  message(index: number, opts?: DecodeOptions): Promise<DecodedMessage>;
  messageMetadata(index: number): Promise<GlobalMetadata>;
  rawMessage(index: number): Promise<Uint8Array>;

  // Layout-aware per-object access (new — cheap over HTTP Range).
  messageDescriptors(index: number): Promise<{
    metadata: GlobalMetadata;
    descriptors: DataObjectDescriptor[];
  }>;
  messageObject(msgIndex: number, objectIndex: number,
    opts?: DecodeOptions): Promise<DecodedMessage>;
  messageObjectRange(msgIndex: number, objectIndex: number,
    ranges: readonly RangePair[],
    opts?: DecodeRangeOptions): Promise<DecodeRangeResult>;

  // Bounded-concurrency fan-out.
  messageObjectBatch(msgIndices: readonly number[], objectIndex: number,
    opts?: DecodeOptions & { concurrency?: number }): Promise<DecodedMessage[]>;
  messageObjectRangeBatch(msgIndices: readonly number[], objectIndex: number,
    ranges: readonly RangePair[],
    opts?: DecodeRangeOptions & { concurrency?: number }): Promise<DecodeRangeResult[]>;
  prefetchLayouts(msgIndices: readonly number[],
    opts?: { concurrency?: number }): Promise<void>;

  [Symbol.asyncIterator](): AsyncIterator<DecodedMessage>;
  close(): void;
}

Numeric limit. All TensogramFile file positions are JavaScript number values, capped at Number.MAX_SAFE_INTEGER (2⁵³ − 1, ≈ 9 PB). Files larger than that must be processed via the Rust or Python bindings; the TS wrapper throws InvalidArgumentError when a WASM-returned u64 exceeds the safe range.

Usage:

const file = await TensogramFile.open('/data/input.tgm');
try {
  console.log(`${file.messageCount} messages, ${file.byteLength} bytes`);

  // Random access
  const first = await file.message(0);
  console.log(first.objects[0].descriptor.shape);
  first.close();

  // Async iteration
  for await (const msg of file) {
    // ...
    msg.close();
  }
} finally {
  file.close();
}

TensogramFile.open(path, opts?) (Node only)

Loads the file via node:fs/promises. The node:fs/promises import is dynamic so browser bundlers can tree-shake this code path.

TensogramFile.fromUrl(url, opts?) (any fetch-capable runtime)

Downloads the file over HTTPS using the ambient globalThis.fetch.

TensogramFile.fromBytes(bytes)

Wraps an already-loaded Uint8Array. The buffer is defensively copied, so later mutation of the caller’s buffer is invisible to the TensogramFile.

Range-based lazy access

TensogramFile.fromUrl automatically probes the server for HTTP Range support. When the HEAD response advertises Accept-Ranges: bytes and a finite Content-Length, the file switches to a lazy backend:

• The initial open issues a small HEAD + one 24-byte Range read per message preamble to build the boundary index. No payload data is downloaded.
• rawMessage(i) / message(i) fetch the full message via a Range: bytes=offset-(offset+length-1) GET and cache it in a 32-entry LRU.
• messageMetadata(i) fetches at most a 256 KB header chunk (or 256 KB footer suffix for footer-indexed messages) and caches the decoded GlobalMetadata in a per-message MessageLayout entry. Subsequent metadata reads are free.
• messageDescriptors(i) uses the cached index frame and the descriptor-prefix optimisation (header + footer + CBOR region for large frames, full frame for small ones) so a 10-object message with 100 MB frames fetches only a few KB per descriptor.
• messageObject(i, j) and messageObjectRange(i, j, ranges) each issue exactly one Range GET for the target object’s frame bytes.
• messageObjectBatch, messageObjectRangeBatch, and prefetchLayouts fan out with bounded concurrency (default 6, configurable via FromUrlOptions.concurrency or per-call opts.concurrency).

When the server omits Accept-Ranges, returns non-200 on HEAD, or the file uses streaming-mode messages (total_length=0 — the writer did not know the final length up front), the open falls back to a single eager GET. Behaviour is indistinguishable to callers except in memory use and timing.

Browser callers using fromUrl directly need CORS to expose the Accept-Ranges, Content-Range, and Content-Length headers.

Bidirectional scan

fromUrl runs a pipelined bidirectional walker by default. Each iteration of lazyScanMessages fetches the next forward preamble, the next backward postamble, AND the previous iteration’s candidate-preamble validation in one parallel Promise.allSettled round. That overlap collapses the per-round critical path from 2 RTTs to 1 RTT, so on real-network workloads the open-time scan is roughly half the wall-clock of forward-only.

import { init, TensogramFile } from '@ecmwf.int/tensogram';

await init();

// Default: bidirectional pipelined walker.
const file = await TensogramFile.fromUrl('https://example.com/data.tgm');
try {
  // ... use file ...
} finally {
  file.close();
}

// Force forward-only.
const fwdOnly = await TensogramFile.fromUrl('https://example.com/data.tgm', {
  bidirectional: false,
});
try {
  // ... use fwdOnly ...
} finally {
  fwdOnly.close();
}

Forward-only and bidirectional opens produce identical messageLayouts on well-formed files; the parity harness asserts this on every fixture. Set bidirectional: false when an adversarial server might serve disagreeing forward and backward reads, or when you specifically want serial Range fetches.

concurrency: 1 is illegal alongside the bidirectional default because the paired round needs two parallel fetches to be useful. The fromUrl promise rejects with InvalidArgumentError before any HTTP probe in that case. To use concurrency: 1, pair it with bidirectional: false explicitly.

Wire-format compatibility: existing .tgm files gain the speedup automatically — the walker is reader-side only, with no migration, no re-encoding, and no wire-format bump.

debug: true emits console.debug events on every state transition — tensogram:scan:mode, tensogram:scan:fallback, tensogram:scan:fwd-terminated, tensogram:scan:gap-closed, tensogram:scan:hop, tensogram:scan:footer-eager — same vocabulary as the Rust tracing events at target = "tensogram::remote_scan".

Eager footer-indexed backward discovery

When the bidirectional walker discovers a footer-indexed message via its postamble, the dispatcher folds an eager footer-region fetch into the same paired round as the candidate-preamble validation. The parsed footer’s metadata and index frames land in the cached layout inline, so a subsequent messageMetadata(idx) / messageDescriptors(idx) short-circuits without issuing a separate footer-region GET.

The fetch is best-effort: if the footer Range request fails or the chunk fails to parse, the layout still commits via the validated preamble alone, and the lazy populate path picks up footer discovery on first metadata access. The footer fetch fires only after the just-validated preamble’s flags carry both FOOTER_METADATA and FOOTER_INDEX, so header-indexed messages (whose footer region typically holds only a hash frame) skip the fetch entirely instead of paying a discarded round trip.

Behaviour is symmetric across the Rust sync / async dispatchers and the TypeScript walker — the same wire-format outcome enum (BackwardOutcome.NeedPreambleValidation) carries the postamble’s first_footer_offset so both sides decide identically.

AWS-signed (S3-compatible) access

For authenticated reads against S3 or any S3-compatible HTTPS endpoint, wrap fetch with createAwsSigV4Fetch and pass it through FromUrlOptions.fetch:

import { createAwsSigV4Fetch, TensogramFile } from '@ecmwf.int/tensogram';

const signedFetch = createAwsSigV4Fetch({
  accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  region: 'eu-west-1',
  // Optional: `sessionToken` for STS credentials; `service` for non-S3
  // S3-compatible endpoints (default 's3').
});

const file = await TensogramFile.fromUrl(
  'https://my-bucket.s3.eu-west-1.amazonaws.com/data.tgm',
  { fetch: signedFetch },
);

The pure signer signAwsV4Request is also exported for callers that want to manage the request lifecycle themselves. Both are covered by byte-for-byte AWS sig-v4-test-suite parity tests, including query-string canonicalisation, header value trim, session-token handling, and pre-encoded path round-tripping.

Azure Blob and Google Cloud Storage are not yet supported by a bundled helper — generate a presigned URL in your control plane and pass it as a plain HTTPS URL instead.

Append (Node local file system)

TensogramFile#append(meta, objects, opts?) encodes the new message in-memory, appends it to the on-disk file, refreshes the position index, and makes the new message reachable via message(i) on the same handle. Only supported when the file was opened via TensogramFile.open(path) — fromBytes- and fromUrl-backed files throw InvalidArgumentError, matching the contract in the other language bindings.

const file = await TensogramFile.open('/data/forecast.tgm');
try {
  await file.append({ version: 3 }, [{ descriptor, data }]);
  console.log(`now has ${file.messageCount} messages`);
} finally {
  file.close();
}

Scope-C API additions

Scope C brought the TypeScript wrapper to full API parity with Rust / Python / FFI / C++. The surface additions are:

Streaming StreamingEncoder (no full-message buffering)

For browser uploads, WebSocket pushes, or any sink that needs bytes as soon as they are produced, pass an onBytes callback to the StreamingEncoder constructor:

const enc = new StreamingEncoder({ version: 3 }, {
  onBytes: (chunk) => uploadSocket.send(chunk),   // e.g. WebSocket.send
});
enc.writeObject(descriptor, new Float32Array([1, 2, 3]));
enc.finish();    // flushes footer; returns empty Uint8Array in streaming mode
enc.close();

Semantics:

• The callback is invoked during construction (preamble + header metadata frame), during each writeObject / writeObjectPreEncoded (one data-object frame’s bytes, potentially across multiple invocations), and during finish() (footer frames + postamble).
• Concatenating every chunk the callback sees (in order) yields a message byte-for-byte identical to what buffered mode would return. Tested via round-trip with decode().
• The callback must be synchronous — Promise return values are silently discarded because the Rust/WASM writer contract is synchronous. Buffer internally first if you need async work.
• Each chunk is JS-owned and fresh per invocation. Copy (new Uint8Array(chunk) or chunk.slice()) if you need to keep it past the next writeObject — the underlying ArrayBuffer is invalidated when WASM memory grows.
• If the callback throws, the exception surfaces as an IoError on the next writeObject / finish. The encoder state is undefined after an error — call close() and start over.
• enc.streaming (getter) reports whether an onBytes sink was supplied — useful for code that needs to branch on mode.

Parity note: the Rust core StreamingEncoder<W: Write> has always supported arbitrary sinks; the WASM/TS surface now exposes this capability to JS code. Python / FFI / C++ bindings remain buffered-only; extending them would follow the same JsCallbackWriter pattern with a language-specific sink abstraction.

First-class half-precision and complex dtypes

Scope C also upgraded the dtype dispatch in {@link typedArrayFor}. obj.data() now returns a first-class view for dtypes JS does not have a native TypedArray for:

All three classes expose .bits / .data for zero-copy access to the underlying raw storage if you need it.

const m = decode(buf);
const f16 = m.objects[0].data();           // Float16Array or polyfill
const asFloat32 = f16.toFloat32Array();    // widened copy
const bits = f16.bits;                      // raw binary16

const cplx = m.objects[1].data() as ComplexArray;
for (let i = 0; i < cplx.length; i++) {
  console.log(cplx.real(i), cplx.imag(i));
}

The polyfill is used automatically when the host runtime does not ship globalThis.Float16Array. hasNativeFloat16Array() and getFloat16ArrayCtor() expose the detection machinery for callers that want direct control.

Breaking change from Scope B: Before Scope C, obj.data() on float16 / bfloat16 returned a raw Uint16Array of bits, and complex dtypes returned an interleaved Float32Array / Float64Array. Consumers that relied on that shape can reach the same bytes via .bits (for f16/bf16) or .data (for complex).

The low-level bit-conversion helpers (halfBitsToFloat, floatToHalfBits, bfloat16BitsToFloat, floatToBfloat16Bits) and the isComplexDtype type-guard are internal and are not re-exported from @ecmwf.int/tensogram. Callers that need bit-level manipulation should grab the raw storage from a view’s .bits / .data accessor and do the conversion themselves, or import directly from @ecmwf.int/tensogram/float16, …/bfloat16, …/complex with the understanding that these module paths are not part of the stable API.

Examples

See examples/typescript/ in the repository for runnable scripts:

• 01_encode_decode.ts — basic round-trip
• 02_mars_metadata.ts — per-object metadata using the MARS vocabulary
• 02b_generic_metadata.ts — per-object metadata using a generic application namespace
• 03_multi_object.ts — multiple dtypes in one message
• 04_decode_range.ts — partial sub-tensor decode
• 05_streaming_fetch.ts — progressive decode over a ReadableStream
• 06_file_api.ts — TensogramFile over Node fs, fetch, and in-memory bytes
• 07_hash_and_errors.ts — hash verification and typed errors
• 08_validate.ts — validate(buf) + validateFile(path)
• 11_encode_pre_encoded.ts — wrap already-encoded bytes
• 12_streaming_encoder.ts — frame-at-a-time encoder with preceders
• 13_range_access.ts — lazy TensogramFile.fromUrl over HTTP Range
• 14_streaming_callback.ts — StreamingEncoder with onBytes callback sink

Run them with:

cd examples/typescript
npm install
npx tsx 01_encode_decode.ts     # or any other file

Cross-language parity

This TypeScript package decodes the same golden .tgm files used by the Rust, Python, and C++ test suites. The committed files at rust/tensogram/tests/golden/*.tgm are decoded by each language’s test runner; any drift in wire-format semantics fails all four suites.

Specifically, typescript/tests/golden.test.ts decodes:

• simple_f32.tgm — single-object Float32 round-trip
• multi_object.tgm — mixed-dtype message (f32 / i64 / u8)
• mars_metadata.tgm — MARS keys under base[0].mars
• multi_message.tgm — two concatenated messages (via scan())
• hash_xxh3.tgm — verifyHash success + tamper detection

typescript/tests/property.test.ts and the Scope-C dtype suites add fast-check property tests pinning:

• mapTensogramError never throws for any finite-string input and always returns a TensogramError subclass;
• encode → decode is bit-exact for random Float32 shapes across random application metadata;
• decode on random byte input either succeeds with a structurally valid message or throws a typed TensogramError — never panics;
• float32 → float16 → float32 round-trip stays within half-precision ulp for any random value in a reasonable magnitude band;
• float32 → bfloat16 → float32 round-trip stays within bfloat16 ulp;
• complex64 encode → decode preserves real(i) / imag(i) byte-for-byte across random shapes and values.

The CI typescript job rebuilds and runs every TS test on every PR.