File manager¶
FileManager manages the state and operations for multiple MEF files in a
thread-safe manner, backing the channels-and-time access path with a shared
tile cache, parallel decode, and window prefetch.
mef3io_server.server.file_manager ¶
FileManager ¶
FileManager(max_workers=4, tile_duration_s=60, tile_cache_bytes=512 * 1024 * 1024, use_process_pool=True, reader_processes=None, prefetch_processes=None, min_parallel_tiles=2, prefetch_ahead_windows=1, prefetch_behind_windows=1, cache_ttl_s=1800, session_ttl_s=1800, max_sessions=1000)
Manages the state and operations for multiple MEF files in a thread-safe manner.
Every data access is oriented purely in channels and time: clients read
arbitrary channels over arbitrary [start_uutc, end_uutc) windows via
:meth:read_signal_range, served from a shared per-channel tile cache with
parallel decode in worker processes and configurable window prefetch.
Thread safety is ensured via a lock for all state-changing operations. Each file is managed independently.
Sessions. A session is one client using one file, keyed by
(client_id, file_path). The client id is a name tag the client library
sends with every call (not authentication). Opening the same file again from
the same client refreshes its session instead of creating another, so a tool
that re-opens on every page view still holds one. Everything heavy is shared
across sessions: one reader per file, and the tile cache, which is keyed by
content (file, channel, tile), never by client. A file's reader closes when
its last session closes or expires.
client_id=""is the default session of clients that send no id (older clients): exactly the behaviour before sessions existed, never expired.- Reading needs the file open by anyone; a client that has its own session refreshes it on every read (and, from step 2 on, uses its own settings).
- Client sessions idle longer than
session_ttl_sexpire; at mostmax_sessionsexist (the least recently used is closed first).
Initialize the FileManager.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_workers
|
int
|
Maximum number of background threads for the thread-based prefetch fallback (used when the process pool is off). |
4
|
tile_duration_s
|
float
|
Time-tile length in seconds for the timestamp-based
( |
60
|
tile_cache_bytes
|
int
|
Global byte budget for the tile cache (float32 tiles). |
512 * 1024 * 1024
|
use_process_pool
|
bool
|
Decode cold reads / prefetch in worker
processes for parallel MEF3 decode (one mef3io session per
worker). When |
True
|
reader_processes
|
int or None
|
Total decode worker processes. |
None
|
prefetch_processes
|
int or None
|
Of the total, how many form the
background prefetch lane. |
None
|
min_parallel_tiles
|
int
|
Only fan a cold read out to the process pool when at least this many tiles are missing; smaller reads stay in-process (IPC is not worth it). |
2
|
prefetch_ahead_windows
|
int
|
How many full windows ahead of the requested window to prefetch (paging forward). |
1
|
prefetch_behind_windows
|
int
|
How many full windows behind the requested window to prefetch (paging backward). |
1
|
session_ttl_s
|
float or None
|
Close a client's session after this many idle seconds (sessions of clients that send no id never expire). |
1800
|
max_sessions
|
int
|
Most client sessions at once; opening one more closes the least recently used. |
1000
|
cache_ttl_s
|
float or None
|
Discard tiles not accessed within this many
seconds. |
1800
|
open_file ¶
open_file(file_path, client_id='', cache_fs=None, allow_mixed_rates=False)
Opens a MEF file for one client and returns its info.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str
|
Path to the MEF file. |
required |
client_id
|
str
|
The calling client's id; |
''
|
cache_fs
|
float or None
|
Serve this session's reads at this rate
(order-preserving min-max of the original), cached and prefetched
at it. |
None
|
allow_mixed_rates
|
bool
|
With |
False
|
Returns:
| Type | Description |
|---|---|
FileInfoResponse
|
Protobuf response with file info and open status. |
expire_sessions ¶
expire_sessions(now=None)
Close client sessions idle longer than session_ttl_s; return how many.
read_signal_range ¶
read_signal_range(file_path, channels, start_uutc, end_uutc, prefetch=True, client_id='', n_points=None)
Read channels over [start_uutc, end_uutc); see :meth:_read_level.
n_points (optional): return exactly that many points per channel,
for example one per pixel, chosen by MinMaxLTTB (reduce.minmax_lttb):
each point is a real sample, its bucket's minimum or maximum, picked to
keep the line's shape, and each channel's overall max and min are
always included. The source is the cheapest that still has
detail to spare: the session's cached rate when it holds at least
2 * n_points samples in the window, else the original rate (zoomed
in). When the original holds n_points samples or fewer it is
returned unchanged: n_points never upsamples. The points are evenly
spaced, so fs in the result is n_points / window.
stream_signal_range ¶
stream_signal_range(file_path, channel_names, start_uutc, end_uutc, client_id='', n_points=None)
Yield a [start_uutc, end_uutc) read as ~2.5MB streamed SignalChunks.
Wraps :meth:read_signal_range for the gRPC GetSignalRange RPC. On any
error a single SignalChunk carrying error_message is yielded.
shutdown ¶
shutdown()
Shuts down the prefetch thread pool, process pools, TTL sweeper, and closes any still-open mef3io reader sessions.
close_file ¶
close_file(file_path, client_id='')
Close one client's session on a file.
The file's reader (and its cached tiles) are released only when no other session uses the file. For clients that send no id this is the behaviour before sessions existed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str
|
Path to the MEF file. |
required |
client_id
|
str
|
The calling client's id ( |
''
|
Returns:
| Type | Description |
|---|---|
FileInfoResponse
|
|
get_file_info ¶
get_file_info(file_path, client_id='')
Gets information about an open MEF file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str
|
Path to the MEF file. |
required |
Returns:
| Type | Description |
|---|---|
FileInfoResponse
|
Protobuf response with file info and open status. |
list_open_files ¶
list_open_files()
Lists all currently open MEF files.
Returns:
| Type | Description |
|---|---|
list
|
List of file paths for open files. |
get_actual_file_path ¶
get_actual_file_path(file_path)
Map the user-supplied absolute file path to the correct on-disk path. If running in Docker, prepend /host_root to absolute paths.