fdb compare#

Compares data of two different FDBs or different data version within one FDB.
The comparison is accomplished in multiple steps. First all the index keys (typically MARS identifiers) are compared.
Second, if all keys match, the messages itself are compared by header and data section.
The message comparison supports different methods, by default all keys are compared.
The data section can be compared given a numeric tolerance.

Usage#

Compares two entire FDBs for MARS id equality:

% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> [options...]

Compare different data in the default (production) FDB:

% fdb-compare --request1="..." --request2="..." [options...]

Options#

--config1=string

Path to a FDB config. If not passed, FDB5 specific environment variables like FDB5_CONFIG_FILE will be evaluated. If --config1 is not provided, --request1 must be provided.

--config2=string

Optional: Path to a second different FDB config. Default: Same as --config1 (if passed), otherwise FDB5 specific environment variables are evaluated.

--request1=string

Optional: Specialized mars keys to request data from FDB1 or both FDBs e.g. --request1="class=rd,expver=1234". Allows comparing different MARS subtrees if --request2 is specified with different arguments.

--request2=string

Optional: Specialized mars keys to request different data from the second FDB, e.g. --request2="class=od,expver=abcd". Allows comparing different MARS subtrees. Only valid if --request1 has been specified.

--scope=string

Optional - Values: [mars (default) / header-only / all] The FDBs can be compared in different scopes: 1) mars Mars Metadata only (default), 2) header-only includes Mars Key comparison and the comparison of the data headers (e.g. grib headers). Note: with grib-comparison-type=bit-identical, the full message is still compared byte-for-byte; header-only filtering only applies to grib-keys comparison. 3) all includes Mars key and data header comparison but also the data sections up to a defined floating point tolerance.

--grib-comparison-type=string

Optional - Values: [hash-keys|grib-keys(default)|bit-identical] Comparing two Grib messages can be done via either (grib-comparison-type=hash-keys) in a bit-identical way with hash-keys for each sections, via grib keys in the message (grib-comparison-type=grib-keys (default)) or by directly comparing bit-identical memory segments (grib-comparison-type=bit-identical).

--fp-tolerance=real

Optional: Floating point tolerance for comparison (default=10.0*EPSILON).

--mars-keys-ignore=string

Optional: Format: Key1=Value1,Key2=Value2,...KeyN=ValueN All Messages that contain any of the defined key value pairs will be omitted.

--grib-keys-select=string

Optional: Format: Key1,Key2,Key3...KeyN Only the specified grib keys will be compared. Only effective with grib-comparison-type=grib-keys.

--grib-keys-ignore=string

Optional: Format: Key1,Key2,Key3...KeyN The specified key words will be ignored. Only effective with grib-comparison-type=grib-keys.

--verbose

Optional: Print additional output, including a progress bar for grib message comparisons

Examples#

Compares two entire FDBs for equality. Using the --scope=all option, not only the headers but also the data is compared:

% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> --scope=all

Compares a specific part (class=rd,expver=1234) of two different FDBs for equality:

% fdb-compare --config1=<path/to/config1.yaml> --config2=<path/to/config2.yaml> --request1="class=rd,expver=1234"

Compares different data versions within one FDB using --request1 and --request2. In that case the only usable --grib-comparison-type is grib-keys (the default option):

% fdb-compare --config1=<path/to/config1.yaml> --request1="class=od,expver=abcd" --request2="class=rd,expver=1234"  --grib-comparison-type="grib-keys" | tee out

Compares different data versions within the production FDB if executed on ATOS (the FDB5 config will be inferred from the environment):

% fdb-compare --request1="class=od,expver=abcd" --request2="class=rd,expver=1234"  --grib-comparison-type="grib-keys" | tee out

Exit Codes#

The tool exits with code 0 on success (all compared entries match) and non-zero on failure (any mismatch detected or an error occurred). This makes it suitable for use in scripts and CI pipelines.

Notes#

When comparing within a single FDB, using --config1 is enough. Minimum arguments are either --config1 or --request1. This is explicitly checked, otherwise the default behaviour would result in a comparison of the default FDB (prodfdb on ATOS) with itself.