Multithreading#

Thread safety in PyFDB depends on two independent factors that must both be considered:

Whether threads share one FDB instance or each own a separate one.
Whether the FDB configuration targets a local or a remote store.

The combination gives four distinct cases with very different safety profiles.

Note

For a local config, giving each thread its own FDB instance is fully safe and requires no extra care. This is the recommended pattern. For a remote config, read rule 2 below before adopting this approach.

Quick reference#

Per-thread FDB instances (each thread creates and owns its own FDB):

Operation	Per-thread, local config	Per-thread, remote config
`archive()`	Safe	Safe
`flush()`	Safe	Safe
`dirty()`	Safe	Safe
`retrieve()` / `inspect()`	Safe	Safe^†
`list()` / `axes()` / `stats()` / `wipe()`	Safe	Safe^†
Reading a `DataHandle` on a different thread	Not supported	Not supported

^† Safe when each thread uses its own instance sequentially. See rule 2.

Shared FDB instance (one FDB used by multiple threads simultaneously):

Operation	Shared instance, local config	Shared instance, remote config
`archive()`	Not safe	Not safe
`flush()`	Not safe	Not safe
`dirty()`	Not safe	Not safe
`retrieve()` / `inspect()`	Not safe	Not safe
`list()` / `axes()` / `stats()` / `wipe()`	Not safe	Not safe
Reading a `DataHandle` on a different thread	Not supported	Not supported

Best-practice guide#

Per-thread FDB, local config — fully safe, no extra care needed.

Each thread creates and owns its own FDB object. All internal state is private to that instance and no shared mutable state exists between threads on the local path.
```
import threading
import pyfdb

def worker(selection):
    fdb = pyfdb.FDB()          # local config; one instance per thread
    for element in fdb.list(selection):
        ...

threads = [threading.Thread(target=worker, args=(sel,)) for sel in selections]
for t in threads:
    t.start()
for t in threads:
    t.join()
```
If multiple per-thread instances all point at the same FDB root directory, they coordinate access to the shared on-disk catalogue through file-level locking — the same mechanism used for multi-process access. This is safe but may limit write throughput when many threads archive to the same location simultaneously.
Per-thread FDB, remote config — safe when calls are sequential per instance.

Each remote FDB instance maintains an internal background connection thread. This background thread and the application thread that owns the instance share some internal state. In the current implementation that sharing is not fully synchronised, so issuing concurrent calls on the same instance from multiple application threads is unsafe.

With a per-thread pattern each application thread is the sole caller on its own instance, so this internal overlap is not a concern in practice.
```
import pyfdb

def worker(selection):
    fdb = pyfdb.FDB(remote_config)   # remote config; one instance per thread
    # Make calls sequentially on this instance — do not share it.
    elements = list(fdb.list(selection))
    ...
```
Shared FDB, local config — serialise every call.

When one FDB instance is used by multiple threads at the same time, all operations are currently unsafe due to unsynchronised internal state. This affects every method: write-side operations (archive(), flush(), dirty()), read-side operations (retrieve(), inspect()), and query operations (list(), axes(), stats(), wipe()).

Wrap every call in a single shared lock:
```
import threading
import pyfdb

fdb = pyfdb.FDB()          # local config, shared across threads
fdb_lock = threading.Lock()

def writer(key, data):
    with fdb_lock:
        fdb.archive(key, data)
        fdb.flush()

def reader(selection):
    with fdb_lock:
        return list(fdb.list(selection))
```
Shared FDB, remote config — serialise every call.

All the same shared-local restrictions apply, and the unsynchronised background connection thread makes the situation strictly worse: concurrent calls from multiple application threads compound the internal contention. A single lock covering every call is required:
```
import threading
import pyfdb

fdb = pyfdb.FDB(remote_config)    # remote config, shared across threads
fdb_lock = threading.Lock()

def worker(selection):
    with fdb_lock:
        elements = list(fdb.list(selection))
    ...
```

Never pass a DataHandle between threads.

A DataHandle returned by retrieve() must be opened, read, and closed on the thread that created it. Handing it off to another thread is not supported.

def worker(selection):
    fdb = pyfdb.FDB()
    with fdb.retrieve(selection) as data_handle:
        payload = data_handle.readall()
    # Safe: the handle never leaves this thread.

Why per-thread FDB behaves differently for local and remote#

For a local config, all currently known synchronisation gaps are in state that belongs exclusively to a single FDB instance. Give each thread its own instance and there is nothing to race on. This is a clean separation: no shared mutable state, no locks needed.

For a remote config the picture is different. Every remote FDB instance opens an internal background thread to receive responses from the server. That background thread and the calling application thread share internal bookkeeping about which requests are in flight. This shared bookkeeping is not yet fully synchronised, which means even a single instance used by a single application thread is technically operating with two threads (the application thread and its background receiver) touching the same data. Per-thread FDB prevents application threads from sharing state, but cannot eliminate the instance’s own internal concurrency.

Current limitations and outlook#

The known synchronisation gaps are tracked and will be addressed in upcoming releases. Once the fixes land, the restrictions on shared instances for local configs are expected to be lifted and the remote config story simplified.

In the meantime:

For a local config, a per-thread FDB is the zero-friction solution.
For a remote config, a per-thread FDB with sequential per-instance calls is the safest readily available approach.
When sharing is unavoidable, a single threading.Lock() guarding every call is both correct and straightforward to add.

For further technical detail see tests/pyfdb/integration/test_threading.py.