Reader pool¶
ReaderProcessPool decodes MEF3 windows in parallel across worker processes,
each with its own mef3io MefReader
session.
mef3io_server.server.reader_pool ¶
Multi-process reader pool for parallel MEF3 decode.
Decode runs in separate worker processes, each with its own MefReader
session, so decodes never contend on shared session state. (The pool predates
mef3io: the legacy pymef backend was GIL-bound, so threads could not
decode in parallel; mef3io releases the GIL during reads, but per-process
sessions remain the isolation model here.) FileManager runs two disjoint
instances of this pool -- a foreground lane for interactive/cold reads and a
prefetch lane for background look-ahead -- so prefetch can never occupy a worker
that a foreground read needs.
A window is split by channel across workers (MEF3 stores channels separately,
so this is embarrassingly parallel). Each worker process lazily opens and reuses
its own MefReader per file (one mef3io session per process). Results are
returned as float32 arrays.
ReaderProcessPool ¶
ReaderProcessPool(max_workers=4)
A pool of worker processes that decode MEF3 windows in parallel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_workers
|
int
|
Number of worker processes. |
4
|
submit_read_tiles ¶
submit_read_tiles(actual_path, channel, block_indices, fs, channel_start_uutc, samples_per_tile)
Submit a decode of the given tiles for one channel; returns a Future.
The Future resolves to {block_index: float32 tile}. Used both for
foreground cold reads (results gathered synchronously) and for background
prefetch (via Future.add_done_callback).
warmup ¶
warmup(actual_path, channel)
Pre-spawn workers and open a reader in each (amortizes first-call cost).
read_window ¶
read_window(actual_path, channels, t_start, t_end, n_splits=None)
Read channels over [t_start, t_end), split across workers by channel.
Returns:
| Type | Description |
|---|---|
ndarray
|
|
prefetch_chunk ¶
prefetch_chunk(chunk_key, actual_path, channels, t_start, t_end)
Schedule a background prefetch of one time-chunk (all given channels).
Idempotent per chunk_key -- scheduling the same key twice is a no-op.
take_chunk ¶
take_chunk(chunk_key)
Return a prefetched chunk (blocking until ready) or None if unknown.