File manager¶

FileManager manages the state and operations for multiple MEF files in a thread-safe manner, backing the channels-and-time access path with a shared tile cache, parallel decode, and window prefetch.

mef3io_server.server.file_manager ¶

FileManager ¶

FileManager(max_workers=4, tile_duration_s=60, tile_cache_bytes=512 * 1024 * 1024, use_process_pool=True, reader_processes=None, prefetch_processes=None, min_parallel_tiles=2, prefetch_ahead_windows=1, prefetch_behind_windows=1, cache_ttl_s=1800)

Manages the state and operations for multiple MEF files in a thread-safe manner.

Every data access is oriented purely in channels and time: clients read arbitrary channels over arbitrary [start_uutc, end_uutc) windows via :meth:read_signal_range, served from a shared per-channel tile cache with parallel decode in worker processes and configurable window prefetch.

Thread safety is ensured via a lock for all state-changing operations. Each file is managed independently.

Initialize the FileManager.

Parameters:

Name	Type	Description	Default
`max_workers`	`int`	Maximum number of background threads for the thread-based prefetch fallback (used when the process pool is off).	`4`
`tile_duration_s`	`float`	Time-tile length in seconds for the timestamp-based (`read_signal_range`) access path.	`60`
`tile_cache_bytes`	`int`	Global byte budget for the tile cache (float32 tiles).	`512 * 1024 * 1024`
`use_process_pool`	`bool`	Decode cold reads / prefetch in worker processes for parallel MEF3 decode (one mef3io session per worker). When `False`, falls back to the in-process thread path.	`True`
`reader_processes`	`int or None`	Total decode worker processes. `None` auto-selects `cpu_count - 1`.	`None`
`prefetch_processes`	`int or None`	Of the total, how many form the background prefetch lane. `None` uses half. The remainder (always >= 1) is the reserved foreground lane so background prefetch can never starve an interactive read.	`None`
`min_parallel_tiles`	`int`	Only fan a cold read out to the process pool when at least this many tiles are missing; smaller reads stay in-process (IPC is not worth it).	`2`
`prefetch_ahead_windows`	`int`	How many full windows ahead of the requested window to prefetch (paging forward).	`1`
`prefetch_behind_windows`	`int`	How many full windows behind the requested window to prefetch (paging backward).	`1`
`cache_ttl_s`	`float or None`	Discard tiles not accessed within this many seconds. `None`/`0` disables idle expiry.	`1800`

open_file ¶

open_file(file_path)

Opens a MEF file and initializes its state.

Parameters:

Name	Type	Description	Default
`file_path`	`str`	Path to the MEF file.	required

Returns:

Type	Description
`FileInfoResponse`	Protobuf response with file info and open status.

read_signal_range ¶

read_signal_range(file_path, channels, start_uutc, end_uutc, prefetch=True)

Read arbitrary channels over an arbitrary [start_uutc, end_uutc) window.

Data is served from the per-channel tile cache, reading only the tiles that are missing and slicing the result sample-exact to the requested window. Neighboring tiles are prefetched in the background for smooth navigation.

Parameters:

Name	Type	Description	Default
`file_path`	`str`	Path to an open MEF file.	required
`channels`	`list[str] or None`	Channels to read; `None`/empty means all channels in the file.	required
`start_uutc`	`int`	Inclusive window start in microseconds (uUTC).	required
`end_uutc`	`int`	Exclusive window end in microseconds (uUTC).	required
`prefetch`	`bool`	Whether to prefetch neighboring tiles.	`True`

Returns:

Type	Description
`dict`	A dict with keys `array` (`np.ndarray` of shape `(n_channels, n_samples)`, `float32`), `channel_names`, `fs`, `start_uutc` and `end_uutc`.

Raises:

Type	Description
`ValueError`	If the file is not open, channels are invalid, the requested channels do not share a sampling rate, or the window is empty.

stream_signal_range ¶

stream_signal_range(file_path, channel_names, start_uutc, end_uutc)

Yield a [start_uutc, end_uutc) read as ~2.5MB streamed SignalChunks.

Wraps :meth:read_signal_range for the gRPC GetSignalRange RPC. On any error a single SignalChunk carrying error_message is yielded.

shutdown ¶

shutdown()

Shuts down the prefetch thread pool, process pools, TTL sweeper, and closes any still-open mef3io reader sessions.

close_file ¶

close_file(file_path)

Closes an open MEF file and cleans up its resources.

Parameters:

Name	Type	Description	Default
`file_path`	`str`	Path to the MEF file.	required

Returns:

Type	Description
`FileInfoResponse`	Protobuf response indicating the file is closed.

get_file_info ¶

get_file_info(file_path)

Gets information about an open MEF file.

Parameters:

Name	Type	Description	Default
`file_path`	`str`	Path to the MEF file.	required

Returns:

Type	Description
`FileInfoResponse`	Protobuf response with file info and open status.

list_open_files ¶

list_open_files()

Lists all currently open MEF files.

Returns:

Type	Description
`list`	List of file paths for open files.

is_running_in_docker ¶

is_running_in_docker()

Detect if running inside a Docker container.

get_actual_file_path ¶

get_actual_file_path(file_path)

Map the user-supplied absolute file path to the correct on-disk path. If running in Docker, prepend /host_root to absolute paths.