Skip to content

File manager

FileManager manages the state and operations for multiple MEF files in a thread-safe manner, backing the channels-and-time access path with a shared tile cache, parallel decode, and window prefetch.

mef3io_server.server.file_manager

FileManager

FileManager(max_workers=4, tile_duration_s=60, tile_cache_bytes=512 * 1024 * 1024, use_process_pool=True, reader_processes=None, prefetch_processes=None, min_parallel_tiles=2, prefetch_ahead_windows=1, prefetch_behind_windows=1, cache_ttl_s=1800, session_ttl_s=1800, max_sessions=1000)

Manages the state and operations for multiple MEF files in a thread-safe manner.

Every data access is oriented purely in channels and time: clients read arbitrary channels over arbitrary [start_uutc, end_uutc) windows via :meth:read_signal_range, served from a shared per-channel tile cache with parallel decode in worker processes and configurable window prefetch.

Thread safety is ensured via a lock for all state-changing operations. Each file is managed independently.

Sessions. A session is one client using one file, keyed by (client_id, file_path). The client id is a name tag the client library sends with every call (not authentication). Opening the same file again from the same client refreshes its session instead of creating another, so a tool that re-opens on every page view still holds one. Everything heavy is shared across sessions: one reader per file, and the tile cache, which is keyed by content (file, channel, tile), never by client. A file's reader closes when its last session closes or expires.

  • client_id="" is the default session of clients that send no id (older clients): exactly the behaviour before sessions existed, never expired.
  • Reading needs the file open by anyone; a client that has its own session refreshes it on every read (and, from step 2 on, uses its own settings).
  • Client sessions idle longer than session_ttl_s expire; at most max_sessions exist (the least recently used is closed first).

Initialize the FileManager.

Parameters:

Name Type Description Default
max_workers int

Maximum number of background threads for the thread-based prefetch fallback (used when the process pool is off).

4
tile_duration_s float

Time-tile length in seconds for the timestamp-based (read_signal_range) access path.

60
tile_cache_bytes int

Global byte budget for the tile cache (float32 tiles).

512 * 1024 * 1024
use_process_pool bool

Decode cold reads / prefetch in worker processes for parallel MEF3 decode (one mef3io session per worker). When False, falls back to the in-process thread path.

True
reader_processes int or None

Total decode worker processes. None auto-selects cpu_count - 1.

None
prefetch_processes int or None

Of the total, how many form the background prefetch lane. None uses half. The remainder (always >= 1) is the reserved foreground lane so background prefetch can never starve an interactive read.

None
min_parallel_tiles int

Only fan a cold read out to the process pool when at least this many tiles are missing; smaller reads stay in-process (IPC is not worth it).

2
prefetch_ahead_windows int

How many full windows ahead of the requested window to prefetch (paging forward).

1
prefetch_behind_windows int

How many full windows behind the requested window to prefetch (paging backward).

1
session_ttl_s float or None

Close a client's session after this many idle seconds (sessions of clients that send no id never expire).

1800
max_sessions int

Most client sessions at once; opening one more closes the least recently used.

1000
cache_ttl_s float or None

Discard tiles not accessed within this many seconds. None/0 disables idle expiry.

1800

open_file

open_file(file_path, client_id='', cache_fs=None, allow_mixed_rates=False)

Opens a MEF file for one client and returns its info.

Parameters:

Name Type Description Default
file_path str

Path to the MEF file.

required
client_id str

The calling client's id; "" for clients that send none (the behaviour before sessions existed).

''
cache_fs float or None

Serve this session's reads at this rate (order-preserving min-max of the original), cached and prefetched at it. None: each channel's own rate. A re-open replaces the session's settings.

None
allow_mixed_rates bool

With cache_fs: a channel recorded below it is served at its own rate. Without it, such a channel makes the open fail, saying which channels and why.

False

Returns:

Type Description
FileInfoResponse

Protobuf response with file info and open status.

expire_sessions

expire_sessions(now=None)

Close client sessions idle longer than session_ttl_s; return how many.

session_count

session_count()

How many sessions exist (clients' and the default ones).

read_signal_range

read_signal_range(file_path, channels, start_uutc, end_uutc, prefetch=True, client_id='', n_points=None)

Read channels over [start_uutc, end_uutc); see :meth:_read_level.

n_points (optional): return exactly that many points per channel, for example one per pixel, chosen by MinMaxLTTB (reduce.minmax_lttb): each point is a real sample, its bucket's minimum or maximum, picked to keep the line's shape, and each channel's overall max and min are always included. The source is the cheapest that still has detail to spare: the session's cached rate when it holds at least 2 * n_points samples in the window, else the original rate (zoomed in). When the original holds n_points samples or fewer it is returned unchanged: n_points never upsamples. The points are evenly spaced, so fs in the result is n_points / window.

stream_signal_range

stream_signal_range(file_path, channel_names, start_uutc, end_uutc, client_id='', n_points=None)

Yield a [start_uutc, end_uutc) read as ~2.5MB streamed SignalChunks.

Wraps :meth:read_signal_range for the gRPC GetSignalRange RPC. On any error a single SignalChunk carrying error_message is yielded.

shutdown

shutdown()

Shuts down the prefetch thread pool, process pools, TTL sweeper, and closes any still-open mef3io reader sessions.

close_file

close_file(file_path, client_id='')

Close one client's session on a file.

The file's reader (and its cached tiles) are released only when no other session uses the file. For clients that send no id this is the behaviour before sessions existed.

Parameters:

Name Type Description Default
file_path str

Path to the MEF file.

required
client_id str

The calling client's id ("": none sent).

''

Returns:

Type Description
FileInfoResponse

file_opened says whether the file is still open (for other sessions).

get_file_info

get_file_info(file_path, client_id='')

Gets information about an open MEF file.

Parameters:

Name Type Description Default
file_path str

Path to the MEF file.

required

Returns:

Type Description
FileInfoResponse

Protobuf response with file info and open status.

list_open_files

list_open_files()

Lists all currently open MEF files.

Returns:

Type Description
list

List of file paths for open files.

is_running_in_docker

is_running_in_docker()

Detect if running inside a Docker container.

get_actual_file_path

get_actual_file_path(file_path)

Map the user-supplied absolute file path to the correct on-disk path. If running in Docker, prepend /host_root to absolute paths.