Compression

Compression is the third stage of the encoding pipeline. It reduces the total byte count of the already-encoded and filtered payload.

Supported Compressors

The Compressor Trait

All compressors implement a common interface with three operations:

#![allow(unused)]
fn main() {
pub trait Compressor {
    fn compress(&self, data: &[u8]) -> Result<CompressResult, CompressionError>;
    fn decompress(&self, data: &[u8], expected_size: usize) -> Result<Vec<u8>, CompressionError>;
    fn decompress_range(
        &self,
        data: &[u8],
        block_offsets: &[u64],
        byte_pos: usize,
        byte_size: usize,
    ) -> Result<Vec<u8>, CompressionError>;
}
}

decompress_range enables partial decode without decompressing the entire payload. Compressors that don’t support it return CompressionError::RangeNotSupported.

Lossless Compressors

Szip (libaec)

Szip implements CCSDS 121.0-B-3, a lossless compressor designed for scientific data. It works on integer data and exploits the block structure of packed values.

Random access: Szip records RSI (Reference Sample Interval) block boundaries during encoding. These offsets are stored in metadata as szip_block_offsets, enabling seek-to-block partial decode via decompress_range. When using encode_pre_encoded, the caller must provide these bit-precise block offsets themselves to enable random access (see Pre-encoded Payloads).

Important: libaec encodes integers only. For floating-point data, use either:

• simple_packing → szip (lossy quantization to integers, then compress)
• shuffle → szip (byte rearrangement, then compress as uint8)

Zstd (Zstandard)

General-purpose lossless compression with excellent ratio/speed tradeoff. Widely used and well-optimized.

No random access — decode_range is not supported with zstd.

LZ4

Fastest decompression of any compressor in the library. Slightly lower compression ratio than Zstd, but 3-5x faster to decompress.

No configurable parameters. No random access.

Blosc2

A meta-compressor that splits data into independently-compressed chunks, then stores them in a frame. Supports multiple internal codecs.

Random access: Because each chunk is independent, Blosc2 can decompress only the chunks covering the requested byte range. decompress_range works by mapping byte offsets to chunk indices.

blosc2_typesize is automatically computed from the preceding pipeline stage: dtype byte width for unencoded data, 1 for shuffled bytes, or packed byte width for simple_packing output.

Lossy Compressors

ZFP

Purpose-built compression for floating-point arrays. ZFP compresses data in blocks of 4 elements (1D) and supports three modes:

Random access: In fixed-rate mode, every block compresses to exactly the same number of bits. This means the byte offset of any block is computable from its index, enabling decompress_range without stored block offsets.

Important: ZFP operates directly on floating-point data. Use encoding: "none" and filter: "none" — ZFP replaces both encoding and compression.

SZ3

Error-bounded lossy compression for scientific data. SZ3 uses prediction-based methods (interpolation, Lorenzo, regression) to achieve high compression ratios within strict error bounds.

Error bound modes:

• abs — Absolute error: |original - decompressed| <= bound for every element
• rel — Relative error: |original - decompressed| / value_range <= bound
• psnr — Peak signal-to-noise ratio lower bound

No random access — decode_range is not supported with SZ3.

Important: Like ZFP, SZ3 operates on floating-point data. Use encoding: "none" and filter: "none".

Choosing a Compressor

flowchart TD
    A{"Data type?"}
    A -->|"Integer / packed"| B{"Need random access?"}
    A -->|"Float, lossy OK"| C{"Need random access?"}
    A -->|"Float, lossless"| D{"Speed priority?"}

    B -->|Yes| E["szip"]
    B -->|No| F{"Speed or ratio?"}
    F -->|Speed| G["lz4"]
    F -->|Ratio| H["zstd"]

    C -->|Yes| I["zfp (fixed_rate)"]
    C -->|No| J{"Error bound type?"}
    J -->|"Bits/precision"| K["zfp"]
    J -->|"Absolute/relative"| L["sz3"]

    D -->|"Fastest decompress"| M["lz4"]
    D -->|"Best ratio"| N["blosc2 or zstd"]
    D -->|"Need random access"| O["blosc2"]

    style E fill:#388e3c,stroke:#2e7d32,color:#fff
    style I fill:#388e3c,stroke:#2e7d32,color:#fff
    style O fill:#388e3c,stroke:#2e7d32,color:#fff

Invalid Combinations

Some pipeline combinations are rejected at configuration time: