API Reference

C++ API

class FDB

A handle to a general FDB.

FDB and its methods are threadsafe. However the caller needs to be aware that flush acts on all archive calls, including arcived messages from other threads. I.e. a call to flush will persist all archived messages regardles from which thread the message has been archived. In case the caller wants a finer control it is advised to instanciate one FDB object per thread to ensure only messages are flushed that have been archived on the same FDB object.

Subclassed by fdb_handle_t

Public Functions

FDB(const Config &config = Config().expandConfig())

~FDB()

FDB(const FDB&) = delete

FDB &operator=(const FDB&) = delete

FDB(FDB&&)

FDB &operator=(FDB&&)

void archive(eckit::message::Message msg)

Archive a eckit::message::Message.

Due to the message being self describing no key needs to be supplied. Any callback set with registerArchiveCallback will be invoked.

Parameters:

handle – eckit::message::Message to data to archive

void archive(eckit::DataHandle &handle)

Archives a stream of one or more messages.

Reads messages from the eckit::DatAaHandle and calls archive() on the corresponding messages. Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

handle – eckit::DataHandle reference data to archive

void archive(const void *data, size_t length)

Archive binary data to a FDB.

Internally creates a DataHandle and calls archive(). Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

• data – Pointer to the binary data to archive

• length – Size of the data to archive with the given

void archive(const metkit::mars::MarsRequest &request, eckit::DataHandle &handle)

Archives data from Datahandle and ensures all keys exactly match the provided MarsRequest.

Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

• request – a mars request

• handle – a data handle pointing to the data

Throws:

• eckit::UserError – if there are more keys in the MarsRequest then in the messages.

• eckit::UserError – if message key not present in MarsRequest.

void archive(const Key &key, const void *data, size_t length)

Archive a binary blob into FDB.

Any callback set with registerArchiveCallback will be invoked.

Note

No constistency checks are applied. The caller needs to ensure the provided key matches metadata present in data.

Parameters:

• key – Key used for indexing and archiving the data

• data – Pointer to the binary blob to archive

• length – Size in bytes of the binary blob to archive

void reindex(const Key &key, const FieldLocation &location)

Generate an new index entry for an existing field location.

Can be used to reindex existing data into a new catalogue (see fdb-reindex tool).

Parameters:

• key – Key used to index the data.

• location – Location of existing data in an FDB store.

void flush()

Flush all buffers and closes all data handles into a consistent DB state

Note

always safe to call

eckit::DataHandle *read(const eckit::URI &uri)

Read binary data from an URI.

Parameters:

uri – eckit uri to the data source

Returns:

DataHandle for reading the requested data from

eckit::DataHandle *read(const std::vector<eckit::URI> &uris, bool inStorageOrder = false)

Read binary data from an list of URI.

Parameters:

• vector – of uris eckit uris to the data source

• inStorageOrder – if set data will be returned in the order it is stored. If unset data will be returned in the order it was requested.

Returns:

DataHandle for reading the requested data

eckit::DataHandle *read(ListIterator &it, bool inStorageOrder = false)

Read binary from a ListIterator.

Parameters:

• uris – a list iterator which resembles a set of fields which should be read

• inStorageOrder – if set data will be returned in the order it is stored. If unset data will be returned in the order it was requested.

Returns:

DataHandle for reading the requested data from

eckit::DataHandle *retrieve(const metkit::mars::MarsRequest &request)

Retrieve data which is specified by a MARS request.

Parameters:

request – MarsRequest which describes the data which should be retrieved

Returns:

DataHandle for reading the requested data from

ListIterator inspect(const metkit::mars::MarsRequest &request)

ListIterator list(const FDBToolRequest &request, ListMode mode, int level = 3)

List data present at the archive and which can be retrieved.

Parameters:

• request – FDBToolRequest stating which data should be queried

• mode – select how duplicates should be handled in the returned iterator

• level – maximum level the visitor should respect

Returns:

ListIterator for iterating over the set of found items

ListIterator list(const FDBToolRequest &request, bool deduplicate = false, int level = 3)

Backwards-compatible overload using the previous deduplicate flag.

DumpIterator dump(const FDBToolRequest &request, bool simple = false)

Dump the structural content of the FDB

In particular, in the TOC formulation, enumerate the different entries in the Table of Contents (including INIT and CLEAR entries). The dump will include information identifying the data files that are referenced, and the “Axes” which describe the maximum possible extent of the data that is contained in the database.

Parameters:

• request –

• simple –

Returns:

DumpIterator for iterating over the set of found items

StatusIterator status(const FDBToolRequest &request)

WipeIterator wipe(const FDBToolRequest &request, bool doit = false, bool porcelain = false, bool unsafeWipeAll = false)

Wipe data from the database.

Deletes FDB databases and the data therein contained. Uses the passed request to identify the database to delete. This is equivalent to a UNIX rm command. This tool deletes either whole databases, or whole indexes within databases

Parameters:

• request – FDBToolRequest stating which data should be queried

• doit – flag for committing to the wipe (default is dry-run)

• porcelain – flag print only a list of files to be deleted / that are deleted

• unsafeWipeAll – flag for omitting all security checks and force a wipe

Returns:

WipeIterator for iterating over the set of wiped items

MoveIterator move(const FDBToolRequest &request, const eckit::URI &dest)

Move content of one FDB database.

This locks the source database, make it possible to create a second database in another root, duplicates all data. Source data are not automatically removed.

Parameters:

• request – a fdb tool request for the data which should be move

• dest – destination uri to which the data should be moved

Returns:

MoveIterator for iterating over the set of found items

PurgeIterator purge(const FDBToolRequest &request, bool doit = false, bool porcelain = false)

Remove duplicate data from the database.

Purge duplicate entries from the database and remove the associated data if the data is owned and not adopted. Data in the FDB5 is immutable. It is masked, but not removed, when overwritten with new data using the same key. Masked data can no longer be accessed. Indexes and data files that only contains masked data may be removed. If an index refers to data that is not owned by the FDB (in particular data which has been adopted from an existing FDB5), this data will not be removed.

Parameters:

• request – a fdb tool request for the data which should be purged

• doit – bool if true the purge is triggered, otherwise a dry-run is executed

• porcelain – bool for printing only those files which are deleted

Returns:

PurgeIterator for iterating over the set of found items

StatsIterator stats(const FDBToolRequest &request)

Prints information about FDB databases, aggregating the information over all the databases visited into a final summary.

Parameters:

request – FDB tool request for which the stats should be shown

Returns:

StatsIterator for iterating over the set of found items

ControlIterator control(const FDBToolRequest &request, ControlAction action, ControlIdentifiers identifiers)

Parameters:

• request – FDB tool request

• action – control action

• identifiers – identifiers

Returns:

ControlIterator for iterating over the set of found items

IndexAxis axes(const FDBToolRequest &request, int level = 3)

Parameters:

• request – FDB tool request

• level – maximum level the axis visitor should respect

Returns:

IndexAxis

AxesIterator axesIterator(const FDBToolRequest &request, int level = 3)

Parameters:

• request – FDB tool request

• level – maximum level the axis visitor should respect

Returns:

AxisIterator

bool enabled(const ControlIdentifier &controlIdentifier) const

Check whether a specific control identifier is enabled

Parameters:

controlIdentifier – a given control identifier

Returns:

bool true or false, depending on the internal status of the FDB

bool dirty() const

Return whether a flush of the FDB is needed

Returns:

true if an archive has happened and a flush is needed

void registerArchiveCallback(ArchiveCallback callback)

Register an archive callback.

Parameters:

callback – an archive callback which should be triggered during archive

void registerFlushCallback(FlushCallback callback)

Register a flush callback.

Parameters:

callback – an flush callback which should be triggered during flushing

FDBStats stats() const

const std::string &name() const

Type of FDB, local or remote

Returns:

name of FDB type used, ‘local’ or ‘remote’

const Config &config() const

Read-only access to this FDBs configuration.

Returns:

configuration

Friends

friend struct remote::WipeHelper

C API

Initialisation

int fdb_initialise()

Initialises API, must be called before any other function

Note

This is only required if being used from a context where eckit::Main() is not otherwise initialised.

Returns:

Return code (FdbErrorValues)

Version Accessors

int fdb_version(const char **version)

Retrieves the release version of the library in human-readable format, e.g. 5.10.8

Parameters:

version – Return variable for release version. Returned pointer valid throughout program lifetime.

Returns:

Return code (FdbErrorValues)

int fdb_vcs_version(const char **sha1)

Retrieves version control checksum of the latest change, e.g. a88011c007a0db48a5d16e296934a197eac2050a

Parameters:

version – Return variable for version control checksum. Returned pointer valid throughout program lifetime.

Returns:

Return code (FdbErrorValues)

Error Handling

enum FdbErrorValues

Return codes

Values:

enumerator FDB_SUCCESS

enumerator FDB_ERROR_GENERAL_EXCEPTION

enumerator FDB_ERROR_UNKNOWN_EXCEPTION

enumerator FDB_ITERATION_COMPLETE

typedef void (*fdb_failure_handler_t)(void *context, int error_code)

Error handler callback function signature

Param context:

Error handler context

Param error_code:

Error code (FdbErrorValues)

const char *fdb_error_string(int err)

Returns a human-readable error message for the last error given an error code

Parameters:

err – Error code (FdbErrorValues)

Returns:

Error message

int fdb_set_failure_handler(fdb_failure_handler_t handler, void *context)

Sets an error handler which will be called on error with the supplied context and an error code

Parameters:

• handler – Error handler function

• context – Error handler context

Key

typedef struct fdb_key_t fdb_key_t

Opaque type for the Key object. Holds the metadata of a Key.

int fdb_new_key(fdb_key_t **key)

Creates a Key instance.

ancillary functions for creating/destroying

FDB objects

Parameters:

key – Key instance. Returned instance must be deleted using fdb_delete_key.

Returns:

Return code (FdbErrorValues)

int fdb_key_add(fdb_key_t *key, const char *param, const char *value)

Adds a metadata pair to a Key

Parameters:

• key – Key instance

• param – Metadata name

• value – Metadata value

Returns:

Return code (FdbErrorValues)

int fdb_delete_key(fdb_key_t *key)

Deallocates Key object and associated resources.

Parameters:

key – Key instance

Returns:

Return code (FdbErrorValues)

Request

typedef struct fdb_request_t fdb_request_t

Opaque type for the Request object. Holds the metadata of a Request.

int fdb_new_request(fdb_request_t **req)

Creates a Request instance.

Parameters:

req – Request instance. Returned instance must be deleted using fdb_delete_request.

Returns:

Return code (FdbErrorValues)

int fdb_request_add(fdb_request_t *req, const char *param, const char *values[], int numValues)

Adds a metadata definition to a Request

Parameters:

• req – Request instance

• param – Metadata name

• values – Metadata values

• numValues – number of metadata values

Returns:

Return code (FdbErrorValues)

int fdb_request_get(fdb_request_t *req, const char *param, char **values[], size_t *numValues)

Get the Metadata values associated to a Request metadata

Parameters:

• req – Request instance

• param – Metadata name

• values – Metadata values

• numValues – number of metadata values

Returns:

Return code (FdbErrorValues)

int fdb_expand_request(fdb_request_t *req)

Expand a Request

Parameters:

req – Request instance

Returns:

Return code (FdbErrorValues)

int fdb_delete_request(fdb_request_t *req)

Deallocates Request object and associated resources.

Parameters:

req – Request instance

Returns:

Return code (FdbErrorValues)

SplitKey

typedef struct fdb_split_key_t fdb_split_key_t

Opaque type for the SplitKey object. Holds the Keys associated with a ListElement.

int fdb_new_splitkey(fdb_split_key_t **key)

Creates a SplitKey instance.

Parameters:

key – SplitKey instance. Returned instance must be deleted using fdb_delete_splitkey.

Returns:

Return code (FdbErrorValues)

int fdb_splitkey_next_metadata(fdb_split_key_t *it, const char **key, const char **value, size_t *level)

Returns the next set of metadata in a SplitKey object. For a given ListElement, the SplitKey represents the Keys associated with each level of the FDB index. Supports multiple fdb_split_key_t iterating over the same key.

Parameters:

• it – SplitKey instance

• key – Key metadata name

• value – Key metadata value

• level – level in the iondex of the current Key

Returns:

Return code (FdbErrorValues)

int fdb_delete_splitkey(fdb_split_key_t *key)

Deallocates SplitKey object and associated resources.

Parameters:

key – SplitKey instance

Returns:

Return code (FdbErrorValues)

ListIterator

typedef struct fdb_listiterator_t fdb_listiterator_t

Opaque type for the ListIterator object. Holds the fdb listing output.

int fdb_listiterator_next(fdb_listiterator_t *it)

Moves to the next ListElement in a ListIterator object.

Parameters:

it – ListIterator instance

Returns:

Return code (FdbErrorValues)

int fdb_listiterator_attrs(fdb_listiterator_t *it, const char **uri, size_t *off, size_t *len)

Returns the attribute of the current ListElement in a ListIterator object.

Parameters:

• it – ListIterator instance

• uri – URI describing the resource (i.e. file path) storing the ListElement data (i.e. GRIB message)

• off – Offset within the resource referred by #uri

• len – Length in bytes of the ListElement data

Returns:

Return code (FdbErrorValues)

int fdb_listiterator_splitkey(fdb_listiterator_t *it, fdb_split_key_t *key)

Lazy extraction of the key of a list element, key metadata can be retrieved with fdb_splitkey_next_metadata.

Parameters:

it – SplitKey instance (must be already initialised by fdb_new_splitkey)

Returns:

Return code (FdbErrorValues)

int fdb_delete_listiterator(fdb_listiterator_t *it)

Deallocates ListIterator object and associated resources.

Parameters:

it – ListIterator instance

Returns:

Return code (FdbErrorValues)

DataReader

typedef struct fdb_datareader_t fdb_datareader_t

Opaque type for the DataReader object. Provides access to the binary data returned by a FDB retrieval.

int fdb_new_datareader(fdb_datareader_t **dr)

Creates a DataReader instance.

Parameters:

dr – DataReader instance. Returned instance must be deleted using fdb_delete_datareader.

Returns:

Return code (FdbErrorValues)

int fdb_datareader_open(fdb_datareader_t *dr, long *size)

Open a DataReader for data reading.

Parameters:

• dr – DataReader instance

• size – size of the opened DataReader

Returns:

Return code (FdbErrorValues)

int fdb_datareader_close(fdb_datareader_t *dr)

Close a DataReader.

Parameters:

dr – DataReader instance

Returns:

Return code (FdbErrorValues)

int fdb_datareader_tell(fdb_datareader_t *dr, long *pos)

Returns the current read position in a DataReader.

Parameters:

• dr – DataReader instance

• pos – Read position (byte offset from the start of the DataReader)

Returns:

Return code (FdbErrorValues)

int fdb_datareader_seek(fdb_datareader_t *dr, long pos)

Sets a new read position in a DataReader.

Parameters:

• dr – DataReader instance

• pos – New read position (byte offset from the start of the DataReader)

Returns:

Return code (FdbErrorValues)

int fdb_datareader_skip(fdb_datareader_t *dr, long count)

Move forward the read position in a DataReader.

Parameters:

• dr – DataReader instance

• count – Offset w.r.t. the current read position retured by fdb_datareader_tell

Returns:

Return code (FdbErrorValues)

int fdb_datareader_size(fdb_datareader_t *dr, long *size)

Return size of internal datahandle in bytes.

Parameters:

• dr – DataReader instance

• size – Size of the DataReader

Returns:

Return code (FdbErrorValues)

int fdb_datareader_read(fdb_datareader_t *dr, void *buf, long count, long *read)

Read binary data from a DataReader to a given memory buffer.

Parameters:

• dr – DataReader instance

• buf – Pointer of the target memory buffer.

• count – Max size of the data to read

• read – Actual size of the data read from the DataReader into the memory buffer

Returns:

Return code (FdbErrorValues)

int fdb_delete_datareader(fdb_datareader_t *dr)

Deallocates DataReader object and associated resources.

Parameters:

key – DataReader instance

Returns:

Return code (FdbErrorValues)

FDB API

typedef struct fdb_handle_t fdb_handle_t

Opaque type for the FDB object.

int fdb_new_handle(fdb_handle_t **fdb)

Creates a FDB instance.

Parameters:

fdb – FDB instance. Returned instance must be deleted using fdb_delete_handle.

Returns:

Return code (FdbErrorValues)

int fdb_new_handle_from_yaml(fdb_handle_t **fdb, const char *system_config, const char *user_config)

Creates a FDB instance from a YAML configuration.

Parameters:

• fdb – FDB instance. Returned instance must be deleted using fdb_delete_handle.

• system_config – Override the system config with this YAML string

• user_config – Supply user level config with this YAML string

Returns:

Return code (FdbErrorValues)

int fdb_archive(fdb_handle_t *fdb, fdb_key_t *key, const char *data, size_t length)

Archives binary data to a FDB instance.

Warning

this is a low-level API. The provided key and the corresponding data are not checked for consistency

Parameters:

• fdb – FDB instance.

• key – Key used for indexing and archiving the data

• data – Pointer to the binary data to archive

• length – Size of the data to archive with the given #key

Returns:

Return code (FdbErrorValues)

int fdb_archive_multiple(fdb_handle_t *fdb, fdb_request_t *req, const char *data, size_t length)

Archives multiple messages to a FDB instance.

Parameters:

• fdb – FDB instance.

• req – If Request #req is not nullptr, the number of messages and their metadata are checked against the provided request

• data – Pointer to the binary data to archive. Metadata are extracted from data headers

• length – Size of the data to archive

Returns:

Return code (FdbErrorValues)

int fdb_list(fdb_handle_t *fdb, const fdb_request_t *req, fdb_listiterator_t **it, bool duplicates, int depth)

List all available data whose metadata matches a given user request.

Parameters:

• fdb – FDB instance.

• req – User Request

• it – ListIterator than can be used to retrieve metadata and attributes of all ListElement matching the user Request #req

• duplicates – Boolean flag used to specify if duplicated ListElements are to be reported or not.

Returns:

Return code (FdbErrorValues)

int fdb_retrieve(fdb_handle_t *fdb, fdb_request_t *req, fdb_datareader_t *dr)

Return all available data whose metadata matches a given user request.

Parameters:

• fdb – FDB instance.

• req – User Request. Metadata of retrieved data must match with the user Request

• dr – DataReader than can be used to read extracted data

Returns:

Return code (FdbErrorValues)

int fdb_flush(fdb_handle_t *fdb)

Force flushing of all write operations

Parameters:

key – FDB instance

Returns:

Return code (FdbErrorValues)

int fdb_delete_handle(fdb_handle_t *fdb)

Deallocates FDB object and associated resources.

Parameters:

key – FDB instance

Returns:

Return code (FdbErrorValues)

Wipe

typedef struct fdb_wipe_element_t fdb_wipe_element_t

typedef struct fdb_wipe_iterator_t fdb_wipe_iterator_t

int fdb_wipe(fdb_handle_t *fdb, fdb_request_t *req, bool doit, bool porcelain, bool unsafeWipeAll, fdb_wipe_iterator_t **it)

Initiates a wipe operation on the FDB. This identifies data matching the given request and, optionally, deletes it.

Parameters:

• fdb – FDB instance.

• req – Request specifying which data should be considered for wiping.

• doit – If true, matching data will be deleted. If false, only a dry-run is performed.

• porcelain – If true, output is formatted for machine parsing.

• unsafeWipeAll – If true, unrecognised data will also be deleted.

• it – [out] Iterator instance used to step through affected elements. Must be deleted using fdb_delete_wipe_iterator.

Returns:

Return code (FdbErrorValues)

int fdb_wipe_iterator_next(fdb_wipe_iterator_t *it, fdb_wipe_element_t **element)

Moves to the next element in a wipe iterator.

Parameters:

• it – WipeIterator instance.

• element – [out] Pointer to the next fdb_wipe_element_t. Must be deleted using fdb_delete_wipe_element.

Returns:

Return code (FdbErrorValues)

int fdb_delete_wipe_iterator(fdb_wipe_iterator_t *it)

Deallocates WipeIterator object.

Parameters:

it – WipeIterator instance

Returns:

Return code (FdbErrorValues)

int fdb_delete_wipe_element(fdb_wipe_element_t *element)

Deallocates WipeElement object.

Parameters:

element – WipeElement instance

Returns:

Return code (FdbErrorValues)

int fdb_wipe_element_string(fdb_wipe_element_t *element, const char **str)

Returns a string representation of a WipeElement.

Parameters:

• element – WipeElement instance

• str – String describing the element. Pointer valid until next() call or object deletion.

Returns:

Return code (FdbErrorValues)

Purge

typedef struct fdb_purge_element_t fdb_purge_element_t

typedef struct fdb_purge_iterator_t fdb_purge_iterator_t

int fdb_purge(fdb_handle_t *fdb, fdb_request_t *req, bool doit, bool porcelain, fdb_purge_iterator_t **it)

Initiates a purge operation on the FDB. This identifies duplicate data matching the given request and, optionally, deletes it.

Parameters:

• fdb – FDB instance.

• req – Request specifying which data should be considered for purging.

• doit – If true, matching data will be deleted. If false, only a dry-run is performed.

• porcelain – If true, output is formatted for machine parsing.

• it – [out] Iterator instance used to step through affected elements. Must be deleted using fdb_delete_purge_iterator.

Returns:

Return code (FdbErrorValues)

int fdb_purge_iterator_next(fdb_purge_iterator_t *it, fdb_purge_element_t **element)

Moves to the next element in a purge iterator.

Parameters:

• it – PurgeIterator instance.

• element – [out] Pointer to the next fdb_purge_element_t. Must be deleted using fdb_delete_purge_element.

Returns:

Return code (FdbErrorValues)

int fdb_delete_purge_iterator(fdb_purge_iterator_t *it)

Deallocates PurgeIterator object.

Parameters:

it – PurgeIterator instance

Returns:

Return code (FdbErrorValues)

int fdb_delete_purge_element(fdb_purge_element_t *element)

Deallocates PurgeElement object.

Parameters:

element – PurgeElement instance

Returns:

Return code (FdbErrorValues)

int fdb_purge_element_string(fdb_purge_element_t *element, const char **str)

Returns a string representation of a PurgeElement.

Parameters:

• element – PurgeElement instance

• str – String describing the element. Pointer valid until next() call or object deletion.

Returns:

Return code (FdbErrorValues)