Python API reference¶
Auto-generated from the source docstrings. For a task-oriented walkthrough see the Python guide.
The native API:
mef3io.Reader ¶
Reader(path: str, password: str = '', backend: str = 'cpp', n_threads: int = 0, cache=None, warn_declarations: bool = True, strict: bool = True, *, _defer_warnings: bool = False)
Read-only interface to a MEF 3.0 session.
Times throughout are uUTC (microseconds since the Unix epoch). Windowed reads fetch only the bytes they need, so they stay cheap on huge sessions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
Path to the |
required |
password
|
str
|
Password for encrypted sessions. A level-2 password unlocks everything; a level-1 password reads the signal and technical metadata but leaves subject metadata locked. Empty for unencrypted sessions. MEF encryption does not cover the samples themselves: refusing an encrypted session without a password is mef3io's policy, not protection (see the encryption model in the docs). |
''
|
backend
|
('cpp', 'pure')
|
Which implementation to use. Defaults to the C++ backend; the pure backend is not yet implemented. |
"cpp"
|
n_threads
|
int
|
How many threads this Reader may use for everything it does —
opening, file access, decoding, decrypting. |
0
|
cache
|
str or None
|
Opt-in warm-start cache for channel metadata. |
None
|
warn_declarations
|
bool
|
Emit a :class: |
True
|
Notes
The open itself (scanning the tree, parsing every segment's metadata) runs in C++ with the GIL released, as reads do, so opening a large session on a worker thread does not freeze other Python threads (a GUI's event loop, for instance).
Examples:
>>> with mef3io.Reader("session.mefd") as r:
... x = r.read(r.channels[0], t0, t1) # float64, NaN in gaps
problems
property
¶
problems: list
What a strict=False Reader skipped instead of raising.
A list of {"channel", "segment_number", "segment", "reason",
"kind"} dicts, empty when everything loaded. Always empty in strict
mode, where the same failure raises instead. kind is:
"segment"— a segment that could not be read; its time span reads back as NaN. Found when the session is opened."records"— a record (.rdat) file that could not be placed (nothing of it is read), or, when the file itself opened, ONE damaged record in it (CRC failure, unusable header, truncated tail;reasonnames the record and its byte offset). Every other record of that file is still returned, resynchronised through the.ridxoffsets where one exists. Found by :meth:records(so the list can grow after the open); the samples are unaffected.segmentis the file's path, andsegment_numberis -1 for a session- or channel-level file (channelis""at session level). Nothing is repaired or written.
Segments that fail during a read (a damaged index or data extent) are
added when the read meets them. A multi-channel :meth:read reads its
channels concurrently, so the entries it adds for different channels
can come in any order (one channel's stay in segment order).
Check it before trusting a read from a lenient Reader: a skipped segment's samples come back as NaN, exactly like a genuine gap, so the array alone cannot tell you data is missing.
declaration_issues
property
¶
declaration_issues: list
Section-2 size declarations this session leaves unset.
The structured form of the :class:SessionDeclarationWarning raised on
open: a list of {"channel", "segment", "field"} dicts, empty when
the session declares everything. Free — computed at open from metadata
that was already parsed. It sees only what is missing; use
:class:mef3io.Validator to find what is merely wrong.
n_threads
property
writable
¶
n_threads: int
How many threads this Reader may use (0 = automatic).
Settable at any time: the new count applies from the next call (calls already running keep theirs). The open used the value given to the constructor. One count covers everything the Reader does — opening, file access, decoding, decrypting; see the threads guide. Results never depend on it.
channels
property
¶
channels: list[str]
Channel names in the session, sorted.
Returns:
| Type | Description |
|---|---|
list of str
|
|
metadata
property
¶
metadata
Session subject/acquisition metadata as a :class:mef3io.Metadata.
Built from the first channel (metadata is session-wide). Subject fields are empty unless the reader was opened with a level-2 password.
Returns:
| Type | Description |
|---|---|
Metadata
|
|
close ¶
close() -> None
Release the backend and its file handles.
After this the reader is CLOSED: a further read raises rather than quietly reopening the session. A caller who closed to release handles — before archiving the directory, or to stay under an fd limit — got them back without notice, which is the opposite of what they asked for. Closing twice is harmless.
info ¶
info(channel: str) -> dict
Channel metadata.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Keys: |
read ¶
read(channel, t0: Optional[int] = None, t1: Optional[int] = None, *, dtype='float64', cancel: Optional['CancelToken'] = None, progress: Optional[Callable[[int, int], object]] = None) -> np.ndarray
Read samples in units on the uniform sampling grid.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str or list of str
|
A channel name, or a list (or tuple) of names for a
MULTI-CHANNEL read: one call returns a 2-D array with one row per
name, in the order given (duplicates allowed). Every row equals
:meth: |
required |
t0
|
int or float
|
Half-open time window |
None
|
t1
|
int or float
|
Half-open time window |
None
|
dtype
|
('float64', 'float32')
|
The output type (any spelling |
"float64"
|
cancel
|
CancelToken
|
Stop the read from another thread: it is checked before each
segment is fetched and before each block is decoded, and a
cancelled read raises :class: |
None
|
progress
|
callable
|
|
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
For a name: a 1-D array on the sampling grid; for a list: a 2-D
array of shape |
Raises:
| Type | Description |
|---|---|
ValueError
|
An empty list, or channels of different sampling rates. |
IndexError
|
An unknown channel (for a list: checked before anything is read). |
FormatError
|
A sample in the window comes from a segment whose factor is
invalid: NaN or infinite. Nothing is decoded; :meth: |
See Also
read_raw : the unscaled int32 form with an explicit validity mask (one channel per call).
read_raw ¶
read_raw(channel: str, t0: Optional[int] = None, t1: Optional[int] = None, *, cancel: Optional['CancelToken'] = None, progress: Optional[Callable[[int, int], object]] = None) -> dict
Read the stored int32 counts with an explicit validity mask.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name. |
required |
t0
|
int or float
|
Half-open |
None
|
t1
|
int or float
|
Half-open |
None
|
cancel
|
optional
|
As in :meth: |
None
|
progress
|
optional
|
As in :meth: |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Keys: |
segments ¶
segments(channel: str) -> list[dict]
Per-segment map of a channel — what data is where.
Read from metadata only (nothing is decoded), so it is cheap even for
huge, gap-riddled sessions. Use it to locate data across large
recording gaps, then :meth:toc for the block-level view within a
segment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name. |
required |
Returns:
| Type | Description |
|---|---|
list of dict
|
One dict per segment (sorted by segment number) with keys
|
block_envelope ¶
block_envelope(channel, t0: Optional[int] = None, t1: Optional[int] = None) -> dict
Minimum and maximum of every data block, from the block index alone.
A fast envelope for viewers: only each segment's block index is read (about 2% of the data), no signal is decoded, and segments outside the window are skipped. A whole day of a channel recorded in 10-second blocks is 8,640 rows.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str or list of str
|
Channel name, or a list (or tuple) of names: then the result is a dict keyed by channel name (in the order given, duplicates once), each value the dict one name returns. The indices of all listed channels are read concurrently on this Reader's threads; still index only. Any mix of sampling rates is fine here. |
required |
t0
|
int
|
Window in uUTC, |
None
|
t1
|
int
|
Window in uUTC, |
None
|
Returns:
| Type | Description |
|---|---|
dict of numpy.ndarray
|
One row per block, in time order: |
Notes
The resolution is the block length: mef3io writes 10-second blocks by
default (1-second blocks at 5 kHz and above), Writer(block_length=)
sets it, and other writers choose their own. Block times come from the
index; :meth:read places samples by each block's own header, which
agrees on every consistent file (see :meth:read_raw for blocks
whose two copies disagree).
Raises:
| Type | Description |
|---|---|
FormatError
|
A block that :meth: |
toc ¶
toc(channel: str) -> list[dict]
Block-level table of contents, for seeking and viewers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name. |
required |
Returns:
| Type | Description |
|---|---|
list of dict
|
One dict per RED block with |
records ¶
records(channel: Optional[str] = None, segment: Optional[int] = None, level: Optional[str] = None) -> list[dict]
Read records (annotations) at the session, channel or segment level.
MEF 3.0 keeps record files at three levels: the session
(<session>.mefd), each channel (<ch>.timd) and each segment
(<ch>-NNNNNN.segd). Every returned record says where it came from.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str or None
|
|
None
|
segment
|
int or None
|
A segment number (requires |
None
|
level
|
(None, 'session', 'channel', 'segment', 'all')
|
Which levels to return. |
None
|
Returns:
| Type | Description |
|---|---|
list of dict
|
One dict per record, grouped session -> channel -> its segments.
Every dict has |
Raises:
| Type | Description |
|---|---|
FormatError
|
A record file cannot be read, or the segments a channel or the
session spans disagree on the recording-time offset AND its file
stores offset-dependent (negated) times, so they cannot be placed.
Only when strict: a lenient Reader skips that file, lists it in
:attr: |
ValueError
|
A |
IndexError
|
Unknown channel or segment number. |
mef3io.MultiReader ¶
MultiReader(paths, password: str = '', backend: str = 'cpp', n_threads: int = 0, cache=None, warn_declarations: bool = True, strict: bool = True)
Several MEF 3.0 recordings read as one, without merging them on disk.
A drop-in for :class:mef3io.Reader: the same methods and properties,
with the same arguments, return types, dict keys and dtypes. The only
differences are the first argument (a sequence of sessions) and that
n_threads is the total for the whole MultiReader, shared between the
recordings it opens or reads at once (never multiplied by them).
What a read returns is exactly what :class:Reader returns for the same
window of the same recordings merged with :func:mef3io.merge_sessions:
the same length, the same values and NaN in the same places. Samples are
placed on the same sampling grid by the same rule (each recording is read
on the window's own grid, so nothing is re-rounded), and where two
recordings touch within half a sample, the block that starts later wins,
as it does in a merged session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
paths
|
sequence of str or os.PathLike
|
The sessions, each anything :class: |
required |
password
|
str
|
The password of every recording. |
''
|
backend
|
optional
|
Passed to every :class: |
'cpp'
|
warn_declarations
|
optional
|
Passed to every :class: |
'cpp'
|
strict
|
optional
|
Passed to every :class: |
'cpp'
|
n_threads
|
int
|
How many threads this MultiReader may use IN TOTAL, for everything it
does ( |
0
|
cache
|
(None, 'auto' or sequence)
|
|
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A channel has different sampling rates in different recordings, or
two recordings hold samples of the same channel at the same time
(compared on the recorded blocks; overlaps up to half a sample are
recorder rounding and allowed, the rule of
:func: |
Notes
How values are combined:
- :attr:
channels: every channel of any recording, sorted. A channel a recording does not have reads as a gap over that recording's time. - :meth:
info:start_timethe earliest,end_timethe latest,number_of_samplesandn_segmentsthe sums,section3_availableonly if every recording has it. The patient fields (subject_*,recording_location,gmt_offset) andsession_descriptionare combined over EVERY recording, also those without the channel, so :meth:infoand :attr:metadataalways agree. Every other field is the value the recordings holding the channel agree on. A field they disagree on isNone(a :class:MultiReaderDifferenceWarningnames such fields once, on open; notunits_conversion_factor, which every read handles per segment). A session description equal to its own recording's name (what mef3io and the legacy stack write by default) counts as unset: day files named by date do not "disagree" on it, and when every recording has only the default it isNone(one recording keeps its own). :func:merge_sessionskeeps the FIRST source's description, so for this one fieldmetadatais not what a merged session gives. - :attr:
metadata: a :class:mef3io.Metadatawith those same combined patient fields and session description, the rest from :meth:infoof the first channel (as :class:Readerdoes). Each recording's own values: :attr:metadata_by_source. - :meth:
segments: every recording's segments in time order, numbered from 0 like the segments of a merged session;pathis the segment's real folder.start_samplecounts the samples stored before it. Unlike a merged session, a segment is never split around another recording's data, so recordings that take turns can give fewer segments than :func:merge_sessionswould. - :meth:
tocand :meth:block_envelope: every block in time order;start_sampleas above. - :meth:
records: session- and channel-level records merged in time order, then each segment's own;segmentuses the numbering of :meth:segments(records(channel, segment=k)too). - :attr:
problems, :attr:declaration_issues: every recording's, in time order; a declaration issue'ssegmentuses the numbering of :meth:segments, a problem keeps the number of the recording's own folder (an unreadable segment is in no segment list) and its path. -
Warnings: each recording's own (declarations, unreadable segments, ...) are emitted once, on open, with
w.message.pathnaming that recording, plus one :class:MultiReaderDifferenceWarning. -
Recordings (or segments) that store different units-conversion factors for one channel: as in a :class:
Reader, :meth:readscales each sample by its own segment's factor, :meth:read_rawlists where each factor applies (units_conversion_factors), andinforeportsNone.
Where it cannot equal a merged session:
- A :meth:
readreads each recording it touches from the window's start, so the window's own grid applies with no re-rounding: a recording late in a long window allocates int32 counts and a validity byte for the part of the window before it as well: about 5 bytes per sample per recording touched, plus 5 for the combined counts, beside the float64 result. progress(done, total):totalcan grow while the recordings start reading (each part knows its size only when it starts);donenever goes back.
Calls on one MultiReader may overlap across threads, as on a
:class:Reader (with the same limits; see the Python guide).
Examples:
>>> days = sorted(Path("/data/patient7").glob("*.mefd"))
>>> with mef3io.MultiReader(days) as r:
... x = r.read("Fp1", t0, t0 + 3_600_000_000) # across midnight
paths
property
¶
paths: tuple
The recordings, in time order (as given, as str). Not on :class:Reader.
metadata_by_source
property
¶
metadata_by_source: dict
Each recording's own :attr:Reader.metadata, keyed by its path as given, in
time order. Not on :class:Reader.
problems
property
¶
problems: list
Every recording's :attr:Reader.problems, in time order.
Same dicts as :attr:Reader.problems; segment holds the path of
the file, so it says which recording. segment_number is the
number of that recording's own segment folder (an unreadable segment
is not in :meth:segments).
declaration_issues
property
¶
declaration_issues: list
Every recording's :attr:Reader.declaration_issues, in time order.
segment is the number in :meth:segments; the warning emitted on
open (one per recording) carries each recording's own numbers.
channels
property
¶
channels: list[str]
Channel names of every recording, sorted.
Returns:
| Type | Description |
|---|---|
list of str
|
|
metadata
property
¶
metadata
Session metadata as a :class:mef3io.Metadata, combined over every recording.
The subject fields and the session description: the value every
recording holds, or None where they differ (a description equal
to its own recording's name is the default and does not count; see
the class notes). The other acquisition
fields come from :meth:info of the first channel, as on
:class:Reader (so None where the recordings holding it differ).
Subject fields are empty unless every recording was opened with
level-2 access.
n_threads
property
writable
¶
n_threads: int
How many threads this MultiReader may use in total (0 = automatic).
Settable at any time: re-split between the recordings from the next call (calls already running keep theirs). See the threads guide.
info ¶
info(channel: str) -> dict
Channel metadata, combined over the recordings that hold the channel.
Same keys as :meth:Reader.info. start_time is the earliest,
end_time the latest, number_of_samples and n_segments the
sums, section3_available true only if every recording has it. The
patient fields and session_description are combined over every
recording (as :attr:metadata); any other field is the value the
recordings holding the channel agree on. A field they disagree on is
None. See the class notes for the session-description rule.
read ¶
read(channel, t0: Optional[int] = None, t1: Optional[int] = None, *, dtype='float64', cancel: Optional['CancelToken'] = None, progress: Optional[Callable[[int, int], object]] = None) -> np.ndarray
Read samples in units on the uniform sampling grid, across recordings.
As :meth:Reader.read, and equal to it on the same recordings merged
with :func:mef3io.merge_sessions. Time a recording does not cover
(or a channel it does not have) is NaN. Each recording's samples are
scaled by its own units-conversion factor. The read shares
:attr:n_threads between the recordings it touches; cancel stops
every recording's read; progress reports the sum over them.
channel may be a list of names, with dtype= and the rules of
:meth:Reader.read (one rate, a 2-D array, the same errors and the
same defaults). Here it reads the channels one after the other, each
with the whole budget, so it returns what a Reader of the merged
session returns but without the Reader's concurrent multi-channel
speed-up. progress is one sequence with one constant total for the
whole call, counted in units of channels (total = 2**20 per listed
channel, each channel's share advancing with its own decoded bytes),
not in bytes as for one channel.
read_raw ¶
read_raw(channel: str, t0: Optional[int] = None, t1: Optional[int] = None, *, cancel: Optional['CancelToken'] = None, progress: Optional[Callable[[int, int], object]] = None) -> dict
Read the stored int32 counts with a validity mask, across recordings.
As :meth:Reader.read_raw (same keys and dtypes); the counts are
placed as :meth:read places them. block_copy_mismatches holds
every recording's, in time order (each segment is a path, so it
says which recording).
units_conversion_factors says which factor applies where, over
the combined window, exactly as a :class:Reader of the merged
session gives it.
segments ¶
segments(channel: str) -> list[dict]
Every recording's segments of a channel, in time order.
As :meth:Reader.segments (same keys). segment numbers them from
0 across the recordings, as a merged session does; path is the
segment's own folder (so it says which recording); start_sample
counts the samples stored before it.
block_envelope ¶
block_envelope(channel, t0: Optional[int] = None, t1: Optional[int] = None) -> dict
Minimum and maximum of every data block, across recordings.
As :meth:Reader.block_envelope (same keys and dtypes): one row per
block, every recording's in time order. A list of names gives a dict
keyed by channel, as a Reader does.
toc ¶
toc(channel: str) -> list[dict]
Block-level table of contents, across recordings.
As :meth:Reader.toc (same keys): every block in time order;
start_sample counts the samples stored before it.
records ¶
records(channel: Optional[str] = None, segment: Optional[int] = None, level: Optional[str] = None) -> list[dict]
Records (annotations) of every recording, as :meth:Reader.records.
Same arguments and dicts. Session- and channel-level records are
merged in time order; segment-level ones follow their segment, whose
segment is the number in :meth:segments (also for the
segment argument). The grouping is Reader's: session, then each
channel's own, then its segments'.
mef3io.MultiReaderDifferenceWarning ¶
MultiReaderDifferenceWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
The recordings a :class:MultiReader combines disagree on a value.
Raised once when the MultiReader is opened. A value the recordings agree
on reads as that value; a value they disagree on (a subject field, the
units, a filter setting, a session description, ...) reads as None
in :meth:MultiReader.info and :attr:MultiReader.metadata, never as one
recording's value. Different patient names or IDs can mean the
recordings are not the same patient's: check before analysing them
together.
The message names the differing FIELDS only, never their values or a
path. w.message.details is the tuple of field names;
w.message.path is the first recording (in time order). Each
recording's own values are in :attr:MultiReader.metadata_by_source, or
open it with :class:Reader.
mef3io.Writer ¶
Writer(path: str, overwrite: bool = False, password1: str = '', password2: str = '', units: Optional[str] = None, block_length: Optional[int] = None, n_threads: int = 0, metadata=None, wait_for_disk: bool = False, flush: str = 'each')
Write a MEF 3.0 session.
Times throughout are uUTC (microseconds since the Unix epoch). The
first write to a channel creates segment 0; later writes append in-segment
(extending the existing files) unless new_segment=True.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
Path to the |
required |
overwrite
|
bool
|
|
False
|
password1
|
str
|
Level-1 and level-2 passwords. MEF has no "level-1 only" files, so to encrypt a session pass both; leave both empty for no encryption. |
''
|
password2
|
str
|
Level-1 and level-2 passwords. MEF has no "level-1 only" files, so to encrypt a session pass both; leave both empty for no encryption. |
''
|
units
|
str or None
|
Physical units label stored in metadata (e.g. |
None
|
block_length
|
int or None
|
RED block size in samples. |
None
|
flush
|
('each', 'close')
|
|
"each"
|
n_threads
|
int
|
How many threads this Writer may use (RED encoding): |
0
|
metadata
|
Metadata or dict
|
Session-wide subject/acquisition metadata written to every channel.
Also settable later via :meth: |
None
|
wait_for_disk
|
bool
|
|
False
|
Notes
When is my data on disk? By default :meth:write returns once its
data is handed to the operating system. Only each segment's .tmet is
flushed before it is published (a short .tmet would make the whole
session fail to open). :meth:flush and :meth:close (also leaving a
with block) return only when everything written so far is on disk,
in one combined wait. With wait_for_disk=True every write returns
only when its own data is on disk. A power cut before that can leave the
last append of a segment torn; :func:mef3io.recover_session with
apply=True restores a valid session holding every sample whose bytes
reached the disk. Earlier appends are never at risk: nothing rewrites
them. A write raises if a flush it waits for fails. The durability=
argument of 1.x was removed in 2.0 (passing it raises TypeError).
Examples:
>>> with mef3io.Writer("session.mefd", overwrite=True, units="uV") as w:
... w.write("ch1", data, start_uutc, fs=256.0) # NaN marks gaps
n_threads
property
writable
¶
n_threads: int
How many threads this Writer may use (0 = automatic).
Settable at any time; applies from the next write. See the threads guide.
unreadable_metadata
property
¶
unreadable_metadata: list[str]
Segment metadata this writer could not use, as "path: reason".
.tmet files that could not be read while choosing the
recording-time offset, and empty segment directories ignored on open
(what a crashed write can leave behind). In a session at offset 0 an
unreadable file is skipped so it does not block unrelated writes (with
an :class:UnreadableSegmentWarning); where the offset is non-zero,
or nothing else is readable, the write is refused instead.
set_metadata ¶
set_metadata(metadata) -> None
Set session-wide subject/acquisition metadata (a
:class:mef3io.Metadata, or a flat dict of fields). Call before
writing; applies to every channel.
flush ¶
flush() -> None
Return only when everything this Writer has written so far is on
disk: every file it created or extended since the last flush, in ONE
combined wait (not one per channel). Raises OSError/RuntimeError
when the system reports the flush failed, and keeps raising it on every
later call (a retry would otherwise claim data that is gone).
:meth:close does the same.
See When is my data on disk?
<https://bnelair.github.io/mef3io/long_recordings/#when-is-my-data-on-disk>_.
close ¶
close() -> None
Flush (see :meth:flush), then release the writer. Called
automatically on context-manager exit. Returns only when everything
written is on disk.
write ¶
write(channel: str, data: ndarray, start_uutc: int, fs: float, precision: 'int | str' = -1, new_segment: bool = False) -> dict
Write float data; NaN runs become discontinuity gaps.
Values are quantized to int32 counts as round(data * 10**precision)
with the conversion factor 10**-precision kept in metadata. NaN
samples are not stored — they read back as NaN.
precision="step" (opt-in) stores the data as whole counts of its
own quantization step instead — e.g. an amplifier's 0.195 uV per
count — which gives much smaller files for data that came from an
ADC. The step is estimated from a bounded sample of the data (fast
on any length) and every value is checked while it is converted: a
value stored this way reads back equal to what was written (exactly,
when the data was counts * step in float64; otherwise within
float64 rounding, 64 eps relative). Data that is not on one step
(filtered, resampled) falls back to the default path with a
:class:StepFallbackWarning, as does data showing too few distinct
values to establish a step (a DC level, a square wave). A channel
keeps the conversion factor of its first segment: on a channel that
already has data, "step" stores data on that factor exactly, and
REFUSES data off it (WriteConflictError, nothing written) rather
than round it. Appending with the default precision to a channel
whose factor is not a power of ten (a step, or write_int32) is
refused too: keep passing "step".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name (created on first write). |
required |
data
|
array_like
|
Any numeric input; coerced to float64 (lists, int arrays,
|
required |
start_uutc
|
int or float
|
Timestamp of the first sample, uUTC. |
required |
fs
|
float
|
Sampling frequency in Hz. |
required |
precision
|
int or 'step'
|
Decimal precision for quantization. |
-1
|
new_segment
|
bool
|
Force a fresh segment instead of appending in-segment. |
False
|
Returns:
| Type | Description |
|---|---|
dict
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
WriteConflictError
|
On an append conflict (fs / conversion-factor mismatch, data
starting before the segment's end, a default-precision append
to a factor that is no power of ten, or |
write_int32 ¶
write_int32(channel: str, data: ndarray, ufact: float, start_uutc: int, fs: float, valid: Optional[ndarray] = None, new_segment: bool = False) -> dict
Write integer counts verbatim with a conversion factor (bit-exact).
The primitive path: counts are stored exactly as given, with ufact
(e.g. an amplifier's volts-per-bit) in metadata. Physical units on read
are counts * ufact.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channel
|
str
|
Channel name (created on first write). |
required |
data
|
array_like of int
|
Integer counts (any integer width). Stored bit-exact. |
required |
ufact
|
float
|
Conversion factor from counts to physical units. |
required |
start_uutc
|
int or float
|
Timestamp of the first sample, uUTC. |
required |
fs
|
float
|
Sampling frequency in Hz. |
required |
valid
|
array_like or None
|
Same length as |
None
|
new_segment
|
bool
|
Force a fresh segment instead of appending in-segment. |
False
|
Returns:
| Type | Description |
|---|---|
dict
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If any value is outside the int32 range (it would wrap), if
|
RuntimeError
|
On an append conflict. |
write_int32_many ¶
write_int32_many(channels, data: ndarray, ufact, start_uutc: int, fs: float, new_segment: bool = False) -> list
Write the same run of integer counts for many channels in one call.
Byte-identical to calling :meth:write_int32 for each channel in the
given order (fresh segments differ only in their random UUIDs), but
the channels are encoded and written in parallel on this Writer's
n_threads (one budget for channels and blocks together), and the
.tmet barriers of all channels are ONE combined wait instead of
one per channel. See When is my data on disk?
<https://bnelair.github.io/mef3io/long_recordings/#when-is-my-data-on-disk>_.
Every refusal, including an append that conflicts with what is on
disk (rate, factor, start before the segment's end), is decided before
any byte is written. A failure while writing or flushing rolls
every channel of the call back and raises the underlying error
(nothing was written); if a rollback itself does not complete,
:class:RollbackIncompleteError names the channels
(incomplete_channels) whose files may be torn. A failure while
publishing raises :class:PartialWriteError, whose
written_channels lists the channels that are on disk (the rest
were rolled back; incomplete_channels as above).
Memory: besides the data, each channel being written holds a copy of
its samples and its encoded blocks (freed when it is written), and
only about n_threads channels run at once; write_many
also holds the converted counts and mask of every channel (about 5/8
of the input), so write long
recordings in chunks (minutes per call).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channels
|
sequence of str
|
Distinct channel names. |
required |
data
|
array_like of int, shape (n_channels, n_samples)
|
Counts, stored bit-exact (no gap mask: split the run, or use
:meth: |
required |
ufact
|
float or sequence of float
|
One conversion factor, or one per channel. |
required |
start_uutc
|
int or float
|
Timestamp of the first sample, uUTC. |
required |
fs
|
float
|
Sampling frequency in Hz. |
required |
new_segment
|
bool
|
Force a fresh segment for every channel instead of appending. |
False
|
Returns:
| Type | Description |
|---|---|
list of dict
|
One summary per channel, in order (as :meth: |
Raises:
| Type | Description |
|---|---|
ValueError
|
Duplicate channels, a shape other than |
TypeError
|
If |
PartialWriteError
|
Some channels were published before a failure; see above. |
write_many ¶
write_many(channels, data: ndarray, start_uutc: int, fs: float, precision: 'int | str' = -1, new_segment: bool = False) -> list
Write the same run of float data for many channels in one call.
The float counterpart of :meth:write_int32_many, with the semantics
of :meth:write per channel: NaN runs become discontinuity gaps, and
the precision (-1 = inferred) is decided per channel. Parallel
across channels, one combined barrier, all-or-nothing rollback; see
:meth:write_int32_many.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
channels
|
sequence of str
|
Distinct channel names. |
required |
data
|
(array_like, shape(n_channels, n_samples))
|
Coerced to float64. |
required |
start_uutc
|
int
|
As :meth: |
required |
fs
|
int
|
As :meth: |
required |
precision
|
int
|
As :meth: |
required |
new_segment
|
int
|
As :meth: |
required |
Returns:
| Type | Description |
|---|---|
list of dict
|
One summary per channel, in order (as :meth: |
write_annotations ¶
write_annotations(annotations, channel: Optional[str] = None) -> None
Write records (annotations).
Always appends to the records already stored at the given level
(the files are created the first time); it never replaces or deletes a
stored record. The new records are added in place, so the cost does not
grow with the number already stored. A record earlier than the stored
ones is appended as well; :meth:mef3io.Reader.records returns them
sorted by time. In encrypted sessions record bodies are level-2
encrypted. A file left torn by a power cut is refused until
:func:mef3io.recover_session has repaired it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
annotations
|
iterable of dict or pandas.DataFrame
|
Each record needs |
required |
channel
|
str or None
|
Channel name for channel-level records, or |
None
|
mef3io.archive_session ¶
archive_session(session_path, tar_path: Optional[str] = None, overwrite: bool = False, *, cancel: Optional[CancelToken] = None, progress: Optional[Callable[[int, int], object]] = None) -> str
Pack a session directory into a single uncompressed tar archive.
The archive (conventionally name.mefd.tar) is a plain ustar file:
:class:Reader opens it directly — no extraction — and any tar tool
(tar -xf) reproduces the original directory. Because it is
uncompressed, windowed reads still fetch only the byte ranges they need.
The source directory is left untouched; output is deterministic, so
archiving the same session twice yields identical bytes. Tar sessions are
read-only: :class:Writer refuses .tar paths.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session_path
|
str or path - like
|
The |
required |
tar_path
|
str
|
Target archive path; must end |
None
|
overwrite
|
bool
|
Replace an existing target archive. Default |
False
|
cancel
|
CancelToken
|
Stop the archive from another thread; checked between 1 MB chunks.
The archive is built in a temporary file and renamed into place only
when complete, so a cancelled call raises :class: |
None
|
progress
|
callable
|
|
None
|
Returns:
| Type | Description |
|---|---|
str
|
Path of the created archive. |
Examples:
>>> tar = mef3io.archive_session("session.mefd")
>>> with mef3io.Reader(tar) as r:
... x = r.read(r.channels[0])
mef3io.extract_session ¶
extract_session(tar_path, dest_dir: Optional[str] = None, overwrite: bool = False, *, cancel: Optional[CancelToken] = None, progress: Optional[Callable[[int, int], object]] = None) -> str
Unpack a session archive back into a .mefd directory.
The inverse of :func:archive_session — after extraction the directory is
a normal writable session again. The session root inside the archive is
stripped, so dest_dir becomes the session directory itself; archives
from foreign tar tools work too. The session is unpacked into a temporary
sibling directory and moved into place only when complete, so a failed or
cancelled extraction never leaves a half-written directory behind — and
with overwrite=True the existing directory survives until the new one
is whole.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tar_path
|
str or path - like
|
The session archive to unpack; must end |
required |
dest_dir
|
str
|
Target directory; must end |
None
|
overwrite
|
bool
|
Replace an existing target directory. Default |
False
|
cancel
|
CancelToken
|
Stop the extraction from another thread; checked between 1 MB chunks.
Raises :class: |
None
|
progress
|
callable
|
|
None
|
Returns:
| Type | Description |
|---|---|
str
|
Path of the extracted session directory. |
Examples:
>>> session = mef3io.extract_session("session.mefd.tar")
>>> with mef3io.Writer(session) as w: # writable again
... w.write("ch1", more_data, t, fs=256.0)
mef3io.password_interop_issue ¶
password_interop_issue(password: str) -> 'str | None'
Why a password will not behave as expected in other MEF readers, or None.
The same rule :class:Writer (and :func:mef3io.anonymize_session) use
for :class:PasswordInteropWarning, so an application can warn before it
writes. A password has an issue when it has:
- more than 15 characters: meflib and pymef refuse it, so only mef3io opens the session;
- more than 15 bytes in UTF-8: pymef (and so mef_tools) copies it into a 16-byte buffer;
- a character above U+00FF: MEF keeps only the low byte of each character's code point, so a different password with the same low bytes also opens the session.
mef3io itself writes and reads such a password correctly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
password
|
str
|
|
required |
Returns:
| Type | Description |
|---|---|
str or None
|
The reason in plain words (it never repeats the password), or None when the password interoperates. |
Examples:
>>> mef3io.password_interop_issue("secret")
>>> mef3io.password_interop_issue("a" * 20)
'it has 20 characters; meflib and pymef refuse passwords longer than 15 characters, ...'
Inspecting a session¶
See the Python guide.
mef3io.inspect ¶
inspect(path, password: str = '', *, include_values: bool = False, read_index: bool = False, strict: bool = True, cancel: Optional[CancelToken] = None, progress: Optional[Callable[[int, int], object]] = None) -> SessionSummary
Describe a session without reading any sample data.
Reads each segment's metadata (.tmet, the same bytes a
:class:mef3io.Reader reads when it opens), the record files, and — only
with read_index=True — the block indices. It never reads a
.tdat: those are measured by size only, so the cost does not grow
with the amount of data recorded.
Available in Python and C++ (mef3io::inspect_session) only; there is
no MATLAB or C ABI equivalent.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str or PathLike
|
A |
required |
password
|
str
|
Opens encrypted sections. Without it an encrypted session is still
described: what stays locked is reported ( |
''
|
include_values
|
bool
|
Return identity VALUES: subject name/ID/location, descriptions,
|
False
|
read_index
|
bool
|
Also read every segment's block index (about 2% of the data) to find gaps inside segments. Off by default. |
False
|
strict
|
bool
|
As everywhere in mef3io, a segment or record file that cannot be read
raises. |
True
|
cancel
|
CancelToken
|
Checked before each segment and each record file. Once set, the call
raises :class: |
None
|
progress
|
callable
|
|
None
|
Returns:
| Type | Description |
|---|---|
SessionSummary
|
|
Raises:
| Type | Description |
|---|---|
IoError
|
|
PasswordError
|
A password was given and is wrong. |
MefError
|
(strict only) A segment or record file cannot be read. |
Cancelled
|
|
Examples:
>>> s = mef3io.inspect("session.mefd")
>>> s.first_time, s.last_time # samples AND annotations
>>> s.identity_field("subject_id").present
>>> [w.code for w in s.anonymize_blockers]
mef3io.SessionSummary
dataclass
¶
SessionSummary(name: str, path: str, layout: str, total_bytes: int, archive_bytes: Optional[int], index_read: bool, offset_assumed_zero: bool, legacy_password_mapping: bool, encryption: dict, time_offset: dict, gmt_offset: Optional[int], dst_start_time: Optional[int], dst_end_time: Optional[int], start_time: Optional[int], end_time: Optional[int], records_start_time: Optional[int], records_end_time: Optional[int], first_time: Optional[int], last_time: Optional[int], channels: list, record_only_channels: list, video_channels: list, incomplete_channels: list, records: list, identity: list, warnings: list, problems: list, declaration_issues: list)
Everything about a session that does not need its samples.
Returned by :func:mef3io.inspect. All times are absolute uUTC (UTC), or
None where they cannot be known without guessing.
Attributes:
| Name | Type | Description |
|---|---|---|
name, path, layout |
The session's name, the path given, and |
|
total_bytes, archive_bytes |
Sizes of every MEF file ( |
|
encryption |
dict
|
|
time_offset |
dict
|
meflib's recording-time offset (RTO). |
gmt_offset, dst_start_time, dst_end_time |
Section 3 time-zone fields ( |
|
start_time, end_time |
First / last sample over all channels. |
|
records_start_time, records_end_time |
Earliest / latest time any record mentions. |
|
first_time, last_time |
Samples and records together: the span a time shift must cover. |
|
channels |
list of ChannelSummary
|
|
record_only_channels |
list of str
|
Channels with records but no data yet. |
video_channels |
list of str
|
|
incomplete_channels |
list of str
|
Channel folders without one complete segment (a segment needs its
|
records |
list of RecordSummary
|
Session, then channel and segment records. |
identity |
list of IdentityField
|
Subject name/ID/location (section 3), descriptions (section 2),
|
warnings |
list of SummaryWarning
|
|
problems |
list of dict
|
What a |
declaration_issues |
list of dict
|
Section-2 size declarations left unset (as
:attr: |
index_read |
bool
|
Whether the block indices were read ( |
legacy_password_mapping |
bool
|
The password opened some metadata or record file only through the
password-to-key mapping of mef3io 1.2.0 and earlier (issue #84): the
session was most likely encrypted by one of those versions with a
non-ASCII password. The data is read correctly, but meflib and pymef
cannot open those files with this password. A |
offset_assumed_zero |
bool
|
Some time here assumes recording-time offset 0 because a section 3
is locked (level-1 password, or none). Only a time stored negated
(relative to the offset) needs the assumption; absolute stored times
do not set it. The same flag is on every
channel, segment and record it concerns. mef3io and pymef always
write offset 0, so such times are right for almost every file; under
meflib's deprecated date masking they are the masked times, which is
what meflib meant a level-1 user to see. They are never called
verified, and an |
channel ¶
channel(name: str) -> ChannelSummary
The :class:ChannelSummary called name (KeyError if none).
identity_field ¶
identity_field(name: str) -> IdentityField
The :class:IdentityField called name (KeyError if none).
to_dict ¶
to_dict() -> dict
Plain nested dicts/lists, ready for json.dumps. Gap spans
become 2-element lists.
mef3io.ChannelSummary
dataclass
¶
ChannelSummary(name: str, sampling_frequency: Optional[float], units_conversion_factor: Optional[float], units_description: Optional[str], channel_description: Optional[str], start_time: Optional[int], end_time: Optional[int], number_of_samples: Optional[int], gaps: list, gaps_complete: bool, gaps_certain: bool, offset_assumed_zero: bool, bytes: int, records: int, segments: list)
One channel: signal metadata, its segment map and its gaps.
gaps are (start, end) uUTC spans with no samples. Without
read_index=True they are only the gaps BETWEEN segments and
gaps_complete is False: a gap inside a segment is found only in the
block index (whose copy of each block's time and count is used; the
authoritative RED block header lives in the .tdat, which is never
read). gaps_certain is False when a listed gap may not be one,
because a segment that could not be read or placed may hold data there,
or because a segment whose times assume offset 0 sits next to one whose
non-zero offset was read. number_of_samples counts stored samples (gaps excluded).
bytes is every file of the channel; records counts its channel-
and segment-level records.
sampling_frequency, units_conversion_factor and
units_description come from the first segment whose metadata section 2
is readable, and are None when every section 2 is locked (encrypted,
no level-1 password). Segments CAN differ (each has its own section 2);
only the first readable one is reported here, except that
units_conversion_factor is None when the readable segments store
different factors (each segment's is in segments). channel_description is
section 2's free-text description from the same segment; every distinct
value across segments and channels is in the identity table
(identity_field("channel_description").values, with
include_values=True). It can identify the subject, so, like
every identity value, it is returned only with include_values=True:
otherwise it is None, and so it is when section 2 is locked. Its
presence is always reported in the identity table
(identity_field("channel_description")).
mef3io.SegmentSummary
dataclass
¶
SegmentSummary(segment: int, path: str, readable: bool, problem: Optional[str], section2_available: bool, section3_available: bool, start_time: Optional[int], end_time: Optional[int], start_sample: Optional[int], number_of_samples: Optional[int], number_of_blocks: Optional[int], number_of_discontinuities: Optional[int], units_conversion_factor: Optional[float], time_offset: str, recording_time_offset: Optional[int], offset_assumed_zero: bool, tmet_bytes: int, tidx_bytes: int, tdat_bytes: int, record_bytes: int, has_records: bool, provenance: Optional[dict], unset_declarations: list)
One segment, from its metadata only.
Times are absolute uUTC (UTC). With section 3 locked (level-1 password,
or none) a time stored relative to the offset is resolved ASSUMING
offset 0, as :class:mef3io.Reader does, and offset_assumed_zero is
True: not verified (absolute stored times need no assumption). They are
None only for a segment that could not be read. time_offset is
"none", "applied" (meflib's offset is set), "locked" (section
3 encrypted and not opened) or "unknown" (the segment is unreadable);
recording_time_offset holds its value only with
include_values=True (the value is itself identifying: it is the local
midnight of the recording day). number_of_discontinuities is the
DECLARED count (legacy writers leave 0 there even with gaps).
units_conversion_factor is the segment's own, as stored (0.0 is
MEF's "no entry"; None when section 2 is locked). File sizes
are in bytes; the .tdat is measured, never read. provenance is the
dict described in :meth:mef3io.Reader.segments, or None.
mef3io.RecordSummary
dataclass
¶
RecordSummary(type: str, level: str, channel: Optional[str], segment: Optional[int], time: Optional[int], times_known: bool, offset_assumed_zero: bool, duration: Optional[int], first_time: Optional[int], last_time: Optional[int], text_bytes: int, body_locked: bool, fields: Optional[dict])
One record (annotation), at any level.
time is the header time (absolute uUTC). first_time /
last_time span every time the record mentions — the header, the body
times (Seiz onsets/offsets, Epoc timestamp/end, Curs trace timestamp) and
time + duration — which is what a time range must cover. They are
None when the file's times depend on an offset that is disputed or
unknown (an unreadable segment in scope); times_known is then
False, and the times inside fields are None too — never a
placeholder. offset_assumed_zero is True when they were placed
with an assumed offset 0 because a section 3 in their scope is locked
(as the Reader does): not verified. text_bytes is the length of its free text; the text itself
is in fields only with include_values=True (the full record, as
:meth:mef3io.Reader.records returns it). body_locked means the body
is encrypted and the password does not open it.
mef3io.IdentityField
dataclass
¶
IdentityField(name: str, location: str, occurrences: int, locked: int, max_length: int, distinct: int, values: list, present: Optional[bool])
Whether one identifying field holds anything, and where.
occurrences is the number of places (files, or records for
record_text) where it is non-empty, locked the number of places
it could not be read (encrypted, or a file that failed in lenient mode),
max_length the longest value in bytes and distinct the number of
different values. values lists them only with
include_values=True; otherwise it is empty. present is True
if it is set anywhere, False if it is set nowhere, and None if it
may be set somewhere that could not be read.
mef3io.SummaryWarning
dataclass
¶
SummaryWarning(code: str, message: str, blocks_anonymize: bool)
Something a caller should know before using the session.
blocks_anonymize marks the conditions anonymize_session refuses:
video channels, an ambiguous or disputed time
convention, unresolved times, locked (encrypted) parts, unreadable files.
Ordering and naming sessions¶
mef3io.time_range ¶
time_range(path, password: str = '', *, cancel: Optional[CancelToken] = None) -> TimeRange
The first and last sample time of a session, reading as little as possible.
Only the first and last complete segment's .tmet of each channel is
read (about 16 KB each), never a block index or data file. Only those
segments' files are checked for existence, the others are not touched, so it
suits ordering many sessions by time. Segments are assumed numbered in
time order, as every known writer numbers them: a middle segment earlier than the
first and last is not seen. Works on .mefd folders and
.mefd.tar archives. An encrypted session can be ordered without its
password (offset_known is then False); a wrong password raises
:class:PasswordError.
mef3io.naming.TimeRange
dataclass
¶
TimeRange(start_uutc: Optional[int], end_uutc: Optional[int], offset_known: bool, channels: int, files_read: int)
A session's first and last sample time.
Attributes:
| Name | Type | Description |
|---|---|---|
start_uutc, end_uutc |
int or None
|
uUTC; |
offset_known |
bool
|
|
channels, files_read |
int
|
Channels with data looked at, and metadata files read (two per channel at most). |
mef3io.name_drift ¶
name_drift(path, *, cancel: Optional[CancelToken] = None) -> NameDrift
Where a session's name is stored and whether those places agree. Writes nothing.
Reads the 1 KB header of every file (and lists the folders).
mef3io.naming.NameDrift
dataclass
¶
NameDrift(folder: str, record_files: Tuple[str, ...], stored_names: Dict[str, int], files_read: int, consistent: bool)
Where a session's name is stored, and whether the places agree.
Attributes:
| Name | Type | Description |
|---|---|---|
folder |
str
|
The folder's (or archive's) name without |
record_files |
tuple of str
|
The names of the session-level annotation files ( |
stored_names |
dict of str to int
|
Each distinct name stored in the files' headers, with how many files hold it. |
files_read |
int
|
|
consistent |
bool
|
All the places name the session the same. A folder renamed by
another tool leaves this |
mef3io.rename_archive ¶
rename_archive(path, new_name: str) -> ArchiveRename
Rename a .mefd.tar archive to <new_name>.mefd.tar in the same folder.
The rename never replaces an existing file: where the file system cannot
guarantee that, it is refused rather than done unsafely. Also refused: a
name over 246 bytes of UTF-8 (never truncated); a name that is not a
portable file name (a folder, a control character, :*?"<>|, a
Windows device name such as CON, a trailing dot or space); an archive
being written (.tmp or .partial beside it). A symlink is renamed
itself, not the file it points to.
Undo it with rename_archive(new_path, old_name).
mef3io.naming.ArchiveRename
dataclass
¶
ArchiveRename(old_path: str, new_path: str, inside_keeps_old_name: bool)
Result of :func:rename_archive. inside_keeps_old_name is always
True: the archive's internal folder, member names and stored names
still carry the old name (extract, rename and archive again to change
them).
mef3io.rename_session ¶
rename_session(path, new_name: str, *, password: str = '', backup_dir=None, descriptions: Optional[Sequence[str]] = None, _crash_at: int = -1, _fail_at: int = -1) -> RenameReport
Rename a .mefd session in every place its name lives.
A session's name is stored in four places, and other MEF software reads all of them: renaming only the folder makes it silently miss the session's annotations. This changes, together:
- the folder,
<old>.mefd-><new_name>.mefd; - the session's annotation files,
<old>.rdat/.ridx; - the name stored in every file's header;
session_descriptionin each channel's metadata, only where it is exactly the old name (a description that merely mentions it is kept).
No sample, block index or time changes, and every renamed file is
stamped rename in its provenance region.
A crash cannot leave it half-done. The old headers and a journal of
the planned steps go to backup_dir first, then a marker into the
session; mef3io refuses to write to a marked session. If the operation is
interrupted, the next rename_session / fix_name_drift call on the
session (with the same backup_dir) rolls it back first. The journal
and the backups are deleted when the operation completes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str or PathLike
|
The |
required |
new_name
|
str
|
Without |
required |
password
|
str
|
Needed when the session is encrypted: the descriptions are in the level-1 encrypted part. |
''
|
backup_dir
|
str or PathLike
|
Where the journal and old headers are kept while it runs. Default:
|
None
|
descriptions
|
list of str
|
Only change the descriptions in these |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A name that is not allowed, the same name, or a |
IoError
|
The target exists, a header is damaged (validate and repair first), or the file system cannot rename without the risk of replacing a file. |
WriteConflictError
|
The session is being written, carries the marker of an operation
whose journal is not in |
PasswordError
|
The session is encrypted and |
Notes
Undo with report.undo(), which renames back and reverts only the
descriptions this call changed. On a macOS HFS+ volume (which stores
names decomposed) a long accented name can be refused by the file
system at the last step; the operation is then rolled back.
mef3io.fix_name_drift ¶
fix_name_drift(path, target: Optional[str] = None, *, password: str = '', backup_dir=None, descriptions: Optional[Sequence[str]] = None, _crash_at: int = -1, _fail_at: int = -1) -> RenameReport
Bring a session's stored names in line with its folder.
For a session whose folder another tool renamed (:func:name_drift
shows it). Sets the name in every file's header, the session's
annotation files and the descriptions equal to the old name to
target (default: the folder's name). The folder itself is never
renamed. Same safety as :func:rename_session (journal, marker,
rollback, no-replace renames, provenance rename).
Refused when the old name is ambiguous: the files store different names,
or there are several session-level annotation files. Undo with
report.undo().
mef3io.naming.RenameReport
dataclass
¶
RenameReport(old_path: str, new_path: str, old_name: str, new_name: str, folder_renamed: bool, files_changed: Tuple[str, ...], descriptions_changed: Tuple[str, ...], records_renamed: Tuple[Tuple[str, str], ...], files: Dict[str, Tuple[int, int]], rolled_back: str, journals_discarded: int, operation: str, notes: Tuple[str, ...] = ())
Result of :func:rename_session or :func:fix_name_drift.
Attributes:
| Name | Type | Description |
|---|---|---|
old_path, new_path |
str
|
The session folder before and after (equal for |
old_name, new_name |
str
|
The stored session name before and after. |
folder_renamed |
bool
|
|
files_changed |
tuple of str
|
Session-relative paths of the files whose header changed. |
descriptions_changed |
tuple of str
|
Session-relative paths of the |
records_renamed |
tuple of (str, str)
|
Session-level annotation files renamed, as (old, new) file names. |
files |
dict of str to (int, int)
|
Each changed file's size and modification time (ns) right after the operation, so a caller can tell whether the session was written to since (an undo is still exact after an append). |
rolled_back |
str
|
The id of an earlier, interrupted operation that was rolled back
first, or |
journals_discarded |
int
|
Journals of operations that never started or had finished, removed. |
notes |
tuple of str
|
Things to know, e.g. a marker that could not be removed after the operation completed. |
operation |
str
|
|
changed
property
¶
changed: bool
Whether anything was changed (a session already named so is left alone).
undo ¶
undo(*, password: str = '', backup_dir=None) -> 'RenameReport'
Reverse this operation, logically: rename back to the old name and revert exactly the descriptions this operation changed.
Never a byte restore, so samples, times and annotations written since
(an append, say) are kept. It is a full operation with every check of
the original (marker, journal, no-replace renames, the password of an
encrypted session). For fix_name_drift the folder is left as it is.
mef3io.operation_marker ¶
operation_marker(path) -> Optional[Dict[str, str]]
The marker of an unfinished in-place operation, or None.
A dict with tool (e.g. "mef3io 1.5.0"), operation
("rename", "fix-name-drift") and id. It names no session,
path or person.
mef3io.clear_operation_marker ¶
clear_operation_marker(path) -> None
Remove the marker of an operation whose journal is lost.
Only for a session whose rename was interrupted and whose backup folder
is gone (another machine, a deleted folder). Refused unless
:func:name_drift shows the name consistent, so a session the operation
left half-changed keeps its marker. When the journal exists, run the
operation again instead: it rolls back.
mef3io.SessionOperationWarning ¶
SessionOperationWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
The session is in the middle of an in-place operation (a rename, #72).
The operation was interrupted, or is still running in another process:
its marker is in the session folder. Until it is finished or rolled back
the session's name can be half-changed, so other MEF software may miss
its annotations, and mef3io refuses to write to it. Reads are not
affected. Run the same operation again (with the same backup folder) to
roll it back; details holds the marker (tool, operation, id).
De-identified copies¶
See De-identification for the workflow. Python and C++ only (no MATLAB, no C ABI).
mef3io.anonymize_session ¶
anonymize_session(src, dst, shift_us: Optional[int] = None, *, pseudonym: Optional[str] = None, clear: Iterable[str] = _CLEAR_CHOICES, text: Union[str, TextCallback] = 'drop', rename: bool = True, password: str = '', output_password: Optional[Sequence[str]] = None, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> AnonymizeReport
Write a de-identified copy of a session: identity removed, times kept or shifted.
The source is never modified. The copy is built next to dst in
<dst>.partial/, checked with :func:verify, and only then moved to
dst. A failure, a refusal or a cancel leaves nothing behind.
The copy pass reads the source once and writes the copy once, streaming
data blocks; :func:verify then reads the source's data and the copy
again. Data blocks are never decoded or re-encoded: only their start time
(and, for encrypted blocks, the 16 encrypted statistics bytes) changes,
and the regions other programs may fill are cleared, so the samples are
bit-identical.
Times. By default (shift_us=None) every time is KEPT exactly as
the source stores it, and so is its recording-time offset: the copy reads
back the same times (and dates) as the source. The copy still shows
when the recording was made. Pass shift_us (see
:func:reference_shift) to hide the dates: every time then moves by
shift_us, each stored in the sign convention it came in:
universal-header start/end in every file, block-index entries, data-block
headers, record headers, record-index entries and the times inside
Seiz, Epoc and Curs records; the copy's
recording_time_offset is 0. In both modes GMT_offset = 0 and DST
start/end = "no entry" (they can say where the recording was made, and
no time mef3io returns depends on them). Identity: subject names,
ID and recording location are cleared (clear=); the session name is
replaced in every header, in the session-level record files' names and in
the descriptions; anonymized_name is set to pseudonym; every UUID
is regenerated; other applications' private header regions are cleared;
every file is written fresh. Each file is stamped with provenance
operation 6, anonymize.
What is refused (nothing is written): a destination that exists or
lies inside the source; a session inspect marks as blocking (video
channels, unreadable or incomplete segments, parts the password does not
open); damaged data (a failing block or record CRC is never re-sealed).
When shifting, also: a disputed or ambiguous recording-time offset, a
stored time at or before a non-zero recording-time offset, an
Epoc/Curs id_number holding a time of the recording, and a
shift that is zero, smaller in magnitude than the session's time span plus
one day, or that would take any time to 0 or below.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
str or path - like
|
The session: a |
required |
dst
|
str or path - like
|
The copy, a new |
required |
shift_us
|
int
|
|
None
|
pseudonym
|
str
|
Written to |
None
|
clear
|
iterable of {"subject", "location", "descriptions"}
|
What to clear. Default: all three. Leaving |
_CLEAR_CHOICES
|
text
|
('drop', 'keep')
|
Record free text (Note/SyLg/EDFA text, Seiz markers and annotation,
Epoc type and text, Curs name, CSti and ESti strings). |
"drop"
|
rename
|
bool
|
|
True
|
password
|
str
|
Opens an encrypted source. It must grant level-2 (full) access: section 3 holds the recording-time offset and the subject fields. |
''
|
output_password
|
(str, str)
|
|
None
|
channels
|
sequence of str
|
Copy only these channels. Mutually exclusive with |
None
|
exclude_channels
|
sequence of str
|
Leave these channels out. Nothing of an excluded channel stays in the
copy: no folder, no records, no entry in a |
None
|
time_range
|
(int, int)
|
Cut: keep only the samples whose time lies in |
None
|
exact
|
bool
|
With |
True
|
records
|
str
|
|
'keep'
|
progress
|
callable
|
|
None
|
cancel
|
CancelToken
|
Stop the copy from another thread; :class: |
None
|
Returns:
| Type | Description |
|---|---|
AnonymizeReport
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A bad argument: a given shift (zero, below the floor, into times <= 0), an
unknown channel, both |
WriteConflictError
|
The session cannot be copied without risking a misplaced or leaked date, or the copy failed verification. |
PasswordError
|
The source is encrypted and the password does not grant level 2. |
IoError
|
|
(FormatError, CrcError)
|
Damaged source data (never re-sealed into the copy). |
Every refusal above carries ``.code``, a stable string from
|
|
:data:`REFUSAL_CODES` (e.g. ``"dst-exists"``, ``"shift-below-floor"``,
|
|
``"name-contains-identity"``), and ``.details``, a dict with
|
|
``channel``, ``field`` and ``inspect_code`` (each ``None`` when it does
|
|
not apply). Match on ``code``, never on the message. :func:`check` returns
|
|
the same refusals without writing. Errors that are not refusals (a
|
|
cancel, an exception raised by a callback, an internal error) carry no
|
|
code. The refusals :func:`check` can report are all raised before
|
|
anything is written; those marked ``"copy"`` in :data:`REFUSAL_CODES`
|
|
come from the copy pass, which removes the partial copy.
|
|
Examples:
Keep the times (the copy shows the real dates):
>>> rep = mef3io.anonymize_session("patient.mefd", "S001.mefd")
>>> rep.shift_us is None
True
Hide the dates by moving the recording to another day:
>>> shift = mef3io.anonymize.reference_shift(start_us, "2000-01-03", tz="Europe/Prague")
>>> rep = mef3io.anonymize_session("patient.mefd", "S002.mefd", shift)
>>> rep.verification.ok
True
mef3io.anonymize.check ¶
check(src, dst, shift_us: Optional[int] = None, *, pseudonym: Optional[str] = None, clear: Iterable[str] = _CLEAR_CHOICES, text: Union[str, TextCallback] = 'drop', rename: bool = True, password: str = '', output_password: Optional[Sequence[str]] = None, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', cancel: Optional[CancelToken] = None) -> CheckReport
Every check :func:anonymize_session makes before it writes, without writing.
Takes the same arguments as :func:anonymize_session (no progress)
and runs the same checks, through the same code: one pre-flight function
in the C++ core serves both, so the two cannot disagree. Instead of
raising the first refusal it returns all of them, each with its stable
code (:data:REFUSAL_CODES).
It writes nothing (no dst, no <dst>.partial) and reads only
metadata: what :func:mef3io.inspect reads, plus each kept segment's
1 KB metadata header and the kept channels' record files. It never reads
the data (.tdat) files. The GIL is released while it runs.
shift_us decides WHICH copy is checked, exactly as for
:func:anonymize_session: None (default) the copy that keeps every
time, an int the shifted copy. The checks that concern only a shift
(marked below) are made only with one.
What it checks, in order: the options (shift zero, both channels and
exclude_channels, output passwords); the destination (exists, a
leftover <dst>.partial, not .mefd, inside the source); the source
(opens, password grants level 2, nothing :func:mef3io.inspect marks as
blocking, including the recording-time-offset refusals); the channel
selection; the shift* (floor, times above 0); the copy's name and
pseudonym (identity, length, rename=False); segment numbering; record
files (readable, index matches, an Epoc/Curs id_number
holding a time inside the recording). ( only with a shift.)
What it cannot decide (codes marked "copy" in :data:REFUSAL_CODES):
a data block failing its CRC or running past its file (block-crc,
data-damaged), and the final verification of the copy
(verify-failed), which needs the copy. With a shift, every time in
the metadata headers and the record files is put through the copy's own
shifting rule up front (time-before-rto, shift-not-positive,
time-overflow); only a time in a block index or data file header, or
a block's own time, is met by the copy pass alone, with the same code.
No refusal message, here or raised, holds one of the source's times.
Available in Python and C++ (mef3io::check_anonymize); not in MATLAB.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
As for :func: |
required | |
dst
|
As for :func: |
required | |
shift_us
|
int
|
The shift of the copy to check; |
None
|
pseudonym
|
Optional[str]
|
As for :func: |
None
|
clear
|
Optional[str]
|
As for :func: |
None
|
text
|
Optional[str]
|
As for :func: |
None
|
rename
|
Optional[str]
|
As for :func: |
None
|
password
|
Optional[str]
|
As for :func: |
None
|
output_password
|
Optional[str]
|
As for :func: |
None
|
channels
|
Optional[str]
|
As for :func: |
None
|
exclude_channels
|
Optional[str]
|
As for :func: |
None
|
time_range
|
Optional[str]
|
As for :func: |
None
|
exact
|
Optional[str]
|
As for :func: |
None
|
cancel
|
CancelToken
|
Stop the check from another thread; :class: |
None
|
Returns:
| Type | Description |
|---|---|
CheckReport
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
A channel list that is not made of strings (code |
Cancelled
|
|
Examples:
>>> rep = mef3io.anonymize.check("patient.mefd", "S001.mefd", shift, pseudonym="S001")
>>> rep.ok, rep.codes
(False, ('pseudonym-contains-identity',))
mef3io.anonymize.verify ¶
verify(src, dst, shift_us: Optional[int] = None, *, password: str = '', output_password: Union[None, str, Sequence[str]] = None, clear: Iterable[str] = _CLEAR_CHOICES, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> VerifyReport
Check a de-identified copy against its source. Reads both; writes nothing.
:func:anonymize_session runs this on every copy before publishing it;
call it yourself to re-check a copy later or one made elsewhere.
It checks that:
- the validator reports nothing on the copy the source did not already have;
- the copy holds a subset of the source's channels, segment for segment;
- every data block's compressed payload and its 256-byte statistics table (compared decrypted) are byte-identical to the source's, so the samples are too, with nothing decoded;
- with
shift_us=None(a copy that keeps the times) every time — universal-header start/end, block-index entries, data-block headers, record headers, record index, the times inside records — equals the source's as stored (so also as read), and so does each segment's recording-time offset; with a shift, every time differs from the source's by exactlyshift_us, in the same sign convention, and the offset is 0. In both, GMT offset 0 and DST "no entry"; - the records match in count, order and type (with
records="drop": the copy holds none); - a field-wise leak scan finds nothing: with a shift, every time field (universal headers, section 3, block index, data-block headers, record headers, record index, Seiz/Epoc/Curs body times) is checked against the source's recording range, plain or negated, widened by a day; every text field (session and anonymized names, descriptions, subject fields, record text) is checked for the names the copy must not carry; no source UUID is reused. Sizes, counts and sample values are never read as times, and the compressed data is compared rather than scanned, so the scan has no false positives from random bytes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
str or path - like
|
The source session and the copy. |
required |
dst
|
str or path - like
|
The source session and the copy. |
required |
shift_us
|
int
|
The shift the copy was made with; |
None
|
password
|
str
|
Opens an encrypted source (level 2). |
''
|
output_password
|
str or (str, str)
|
Opens an encrypted copy: its level-2 password, or the pair. |
None
|
clear
|
iterable of str
|
What the copy was asked to clear (as for :func: |
_CLEAR_CHOICES
|
time_range
|
optional
|
The cut the copy was made with (as for :func: |
None
|
exact
|
optional
|
The cut the copy was made with (as for :func: |
None
|
progress
|
optional
|
As for :func: |
None
|
cancel
|
optional
|
As for :func: |
None
|
Returns:
| Type | Description |
|---|---|
VerifyReport
|
|
mef3io.anonymize.reference_shift ¶
reference_shift(reference_start_us: int, new_date, *, tz: Optional[str] = None, keep_weekday: bool = False) -> int
The shift that moves a reference recording to new_date.
D is a whole number of days, measured on the clock being kept:
tzgiven (keep the LOCAL time of day):D = new_date - (the reference's local start date in tz)and the shift isD + off_ref, whereoff_refis the reference start's UTC offset intzat that instant. The copy's UTC clock then reads the original local time of day.tzomitted (keep the UTC time of day):D = new_date - (the reference's UTC start date)and the shift isD.
keep_weekday rounds D to the nearest multiple of 7 days, so the
copy falls on the same weekday (off_ref is never rounded).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference_start_us
|
int
|
The reference recording's start, uUTC. |
required |
new_date
|
date or str
|
The date the reference should start on, e.g. |
required |
tz
|
str
|
An IANA time zone (e.g. |
None
|
keep_weekday
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
int
|
The shift, in microseconds. |
mef3io.anonymize.redact_text ¶
redact_text(src, dst, texts: Sequence[str], *, pseudonym: Optional[str] = None, clear: Iterable[str] = _CLEAR_CHOICES, rename: bool = True, password: str = '', channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, cancel: Optional[CancelToken] = None) -> List[str]
The text :func:anonymize_session would keep, for each of texts.
Runs the exact removal the copy applies to text it keeps (record text with
text="keep", and the descriptions when "descriptions" is not in
clear), for the same source, destination name and options. An
application can then compare a copy's text with what mef3io itself would
have left, without copying mef3io's rules. It writes nothing, reads only
metadata (what :func:check reads), and never logs or stores its inputs.
It is not a safety check on the result: :func:check refuses text the
removal cannot take out (code text-unremovable), and :func:verify
checks every copy.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
As for :func: |
required | |
dst
|
As for :func: |
required | |
texts
|
sequence of str
|
The texts to redact. Text that is not valid UTF-8 can be passed with
|
required |
pseudonym
|
Optional[str]
|
As for :func: |
None
|
clear
|
Optional[str]
|
As for :func: |
None
|
rename
|
Optional[str]
|
As for :func: |
None
|
password
|
Optional[str]
|
As for :func: |
None
|
channels
|
Optional[str]
|
As for :func: |
None
|
exclude_channels
|
Optional[str]
|
As for :func: |
None
|
Returns:
| Type | Description |
|---|---|
list of str
|
One redacted text per input, in order. |
Raises:
| Type | Description |
|---|---|
What the copy itself would refuse, raised as :func:`anonymize_session`
|
|
raises it (its ``.code``): a source that cannot be read or is locked
|
|
(``source-missing``, ``password-required``, ``password-level1-only``),
|
|
an unknown channel, a destination, pseudonym or kept channel name holding
|
|
the patient's identity (``channel-name-identity``), and so on. Only refusals that do not decide the redaction are
|
|
ignored: the destination already existing (``dst-exists``,
|
|
``partial-exists``) and text the removal cannot take out
|
|
(``text-unremovable``, which is what :func:`check` reports).
|
|
mef3io.AnonymizeReport
dataclass
¶
AnonymizeReport(destination: str, session_name: str, pseudonym: str, shift_us: Optional[int], source_recording_time_offset: int, channels_kept: Tuple[str, ...], channels_excluded: Tuple[str, ...], channels_without_samples: Tuple[str, ...], segments: int, blocks: int, blocks_reencrypted: int, blocks_decrypted: int, records: int, record_text_fields_changed: int, record_bodies_cleared: int, seizure_channel_entries_dropped: int, blocks_trimmed: int, records_outside_window: int, bytes_written: int, output_encrypted: bool, not_copied: Tuple[str, ...], notes: Tuple[str, ...], verification: VerifyReport, source_start_us: Optional[int] = None, source_end_us: Optional[int] = None, source_channel_times: Dict[str, Tuple[Optional[int], Optional[int]]] = dict(), records_dropped: int = 0)
Result of :func:anonymize_session.
Attributes:
| Name | Type | Description |
|---|---|---|
destination |
str
|
The copy. |
session_name, pseudonym |
str
|
The copy's session name (its folder's stem) and the value written to
every |
shift_us |
int or None
|
The shift applied to every time, in microseconds; |
source_recording_time_offset |
int
|
The source's recording-time offset (meflib's deprecated date
masking). A shifted copy has 0; a copy that keeps the times keeps it
(each segment its own). 0 when the source's segments disagree on it:
there is no single one ( |
channels_kept, channels_excluded |
tuple of str
|
|
channels_without_samples |
tuple of str
|
A cut: channels left out because they have no sample in the window
(also in |
segments, blocks, records |
int
|
What was copied. |
blocks_reencrypted, blocks_decrypted |
int
|
Data blocks the source had encrypted, written encrypted with the new passwords, or decrypted because the copy has none. |
record_text_fields_changed |
int
|
Record text fields the |
record_bodies_cleared |
int
|
Record bodies of an unknown type or version, not carried over because they may hold text or dates that cannot be found. |
seizure_channel_entries_dropped |
int
|
Per-channel entries of |
blocks_trimmed, records_outside_window |
int
|
A cut ( |
records_dropped |
int
|
|
bytes_written |
int
|
|
output_encrypted |
bool
|
|
not_copied |
tuple of str
|
Files in the source that are not part of the MEF layout and were left out (source-relative paths). |
notes |
tuple of str
|
|
verification |
VerifyReport
|
The check the copy passed before it was published. |
source_start_us, source_end_us |
int or None
|
The SOURCE's first and last sample time, uUTC (None without samples). |
source_channel_times |
dict of str to (int or None, int or None)
|
The same for each kept channel (None for a channel with records only). |
Notes
source_start_us, source_end_us and source_channel_times are
the re-identification key: with shift_us they give the real dates
back (a copy that keeps the times shows them anyway, but they are handled
the same way). They are returned to the caller, in memory, and nowhere else:
mef3io never writes them into the copy (the copy's leak scan refuses it
if any appears), never logs them and never puts them in a warning or a
message. They are left out of repr(), :meth:summary, :meth:to_dict
(unless include_original_times=True) and pickling: a pickled
report arrives without them unless it was marked with
:meth:with_original_times_in_pickle (for example to pass it to a helper
process you control; never to a cache). copy.copy and
copy.deepcopy keep them. dataclasses.asdict(report) INCLUDES
them: prefer :meth:to_dict. Store them as you would the original
recording.
with_original_times_in_pickle ¶
with_original_times_in_pickle() -> 'AnonymizeReport'
A copy of this report whose pickle KEEPS the original times.
By default a pickled report drops them (the re-identification key). Use this only to hand the report to a process you control, never for a cache or a file.
to_dict ¶
to_dict(*, include_original_times: bool = False) -> dict
Plain dict (JSON-serialisable).
The source's original times (the re-identification key) are left out
unless include_original_times=True, so that logging or sharing
the dict cannot leak them by accident.
mef3io.anonymize.CheckReport
dataclass
¶
CheckReport(ok: bool, refusals: Tuple[Refusal, ...], notes: Tuple[str, ...], session_name: str, pseudonym: str, channels_kept: Tuple[str, ...], channels_excluded: Tuple[str, ...], channels_without_samples: Tuple[str, ...] = ())
Result of a pre-flight check (anonymize.check, transform.check).
Attributes:
| Name | Type | Description |
|---|---|---|
ok |
bool
|
True when the copy would start. It can still be refused by what only
the copy pass meets (codes marked |
refusals |
tuple of Refusal
|
Every refusal found, in the order the copy meets them (it raises the first). Checks that depend on one that failed are skipped: nothing about the source is checked when it cannot be opened. |
notes |
tuple of str
|
What would be noted up front: a source that uses meflib's recording-time offset (anonymize), an output password other MEF readers will not handle as expected. |
session_name |
str
|
The copy's session name ("" when |
pseudonym |
str
|
Anonymize only: the copy's pseudonym ("" for a transform). |
channels_kept, channels_excluded |
tuple of str
|
The channel selection (empty when the source could not be read). |
channels_without_samples |
tuple of str
|
A cut: kept channels with no sample in the window, which the copy
leaves out (also in |
mef3io.anonymize.Refusal
dataclass
¶
Refusal(code: str, message: str, error: str, channel: Optional[str] = None, field: Optional[str] = None, inspect_code: Optional[str] = None)
One reason a copy would be refused.
Attributes:
| Name | Type | Description |
|---|---|---|
code |
str
|
Stable identifier ( |
message |
str
|
Plain words, as the raised exception carries them. It may name the source's or the destination's path. |
error |
str
|
The exception type the copy raises for it: |
channel |
str or None
|
The channel concerned, if any. |
field |
str or None
|
The argument or field concerned ( |
inspect_code |
str or None
|
For a condition :func: |
mef3io.anonymize.REFUSAL_CODES
module-attribute
¶
REFUSAL_CODES: Dict[str, Tuple[str, str]] = {**COPY_REFUSAL_CODES, 'clear-unknown': ('ValueError', 'check'), 'text-invalid': ('ValueError', 'check'), 'shift-zero': ('ValueError', 'check'), 'text-callback-missing': ('ValueError', 'defensive'), 'rto-disputed': ('WriteConflictError', 'check'), 'rto-unreadable': ('WriteConflictError', 'check'), 'time-before-rto': ('WriteConflictError', 'check'), 'shift-below-floor': ('ValueError', 'check'), 'shift-not-positive': ('ValueError', 'check'), 'name-contains-identity': ('ValueError', 'check'), 'pseudonym-contains-identity': ('ValueError', 'check'), 'id-number-time': ('WriteConflictError', 'check'), 'time-overflow': ('FormatError', 'defensive'), 'text-unremovable': ('WriteConflictError', 'check'), 'channel-name-identity': ('WriteConflictError', 'check')}
mef3io.VerifyReport
dataclass
¶
VerifyReport(ok: bool, problems: Tuple[VerifyIssue, ...], leaks: Tuple[VerifyIssue, ...], notes: Tuple[VerifyIssue, ...], inherited_findings: Tuple[str, ...], channels: Tuple[str, ...], excluded_channels: Tuple[str, ...], segments: int, blocks: int, records: int, times_checked: int, text_fields_checked: int)
Result of a copy's verification (anonymize.verify, transform.verify).
Attributes:
| Name | Type | Description |
|---|---|---|
ok |
bool
|
True when nothing is wrong: no problem and no leak. |
problems |
tuple of VerifyIssue
|
Everything wrong with the copy, leaks included. |
leaks |
tuple of VerifyIssue
|
Anonymize only: the problems found by the leak scan (a time field holding a value in the source's recording range, plain or negated; a name that should be gone; a source UUID; an uncleared field). |
notes |
tuple of VerifyIssue
|
Anonymize only: overlaps reported but not refused, worth showing the user (an identity value inside a kept channel's name as part of a longer token or shorter than 3 characters, inside the copy's own name, or a number of fewer than 4 digits inside a longer number in kept text). |
inherited_findings |
tuple of str
|
Validator findings the source already had; a copy carries them over
unchanged and they do not count against |
channels, excluded_channels |
tuple of str
|
Channels in the copy, and the source's channels it leaves out. |
segments, blocks, records, times_checked, text_fields_checked |
int
|
How much was compared. |
mef3io.anonymize.VerifyIssue
dataclass
¶
VerifyIssue(location: str, field: str, message: str)
One problem a verification found.
message names what is wrong in plain words. It never repeats an
identifying string (a name it found is described, not quoted), so a
report can be logged or shared.
Modified copies¶
See Modified copies for what changes and what is copied as
stored. time_range= (a cut) is described there too. The report, check and refusal types are shared with the
de-identified copies above. Python and C++ only (no MATLAB, no C ABI).
mef3io.transform_session ¶
transform_session(src, dst, *, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', rename: bool = True, password: str = '', progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> TransformReport
Write a modified copy of a session: a choice of channels, passwords, name and details.
The source is never modified. The copy is built next to dst in
<dst>.partial/, checked with :func:verify, and only then moved to
dst. A failure, a refusal or a cancel leaves nothing behind.
What stays as stored. Every time: universal headers, block index,
data blocks, records, the recording-time offset and the time-zone
fields, so no date is moved or revealed. Every sample: data blocks are
copied without being decoded, only the 16 encrypted statistics bytes of
an encrypted block are re-keyed. The regions other programs fill in MEF
headers. Record bodies (except the Seiz entries of left-out
channels).
What changes. Only what you ask for: the channels (channels or
exclude_channels), the passwords (output_password; the source's
are never carried over), the session name (the stem of dst, in every
header and in the session's own annotation files) and the metadata
fields you set. Every file gets a new file UUID, and its provenance
records the transform (operation 8) while keeping the link to the source
file it came from.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
str or path - like
|
The session: a |
required |
dst
|
str or path - like
|
The copy, a new |
required |
channels
|
sequence of str
|
Copy only these channels. Mutually exclusive with |
None
|
exclude_channels
|
sequence of str
|
Leave these channels out: no folder, no records, no entry in a
|
None
|
output_password
|
(str, str)
|
|
None
|
metadata
|
mapping
|
Descriptive fields to set in the copy (:data: |
None
|
time_range
|
(int, int)
|
Cut (#103): keep only the samples whose time lies in |
None
|
exact
|
bool
|
With |
True
|
records
|
str
|
|
'keep'
|
rename
|
bool
|
|
True
|
password
|
str
|
Opens an encrypted source. It must grant level-2 (full) access. |
''
|
progress
|
callable
|
|
None
|
cancel
|
CancelToken
|
Stop the copy from another thread; :class: |
None
|
Returns:
| Type | Description |
|---|---|
TransformReport
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A bad argument: an unknown channel, both |
TypeError
|
A channel list or a metadata value of the wrong type. |
WriteConflictError
|
Parts of the session cannot be copied whole (video channels, unreadable or incomplete segments or annotation files), or the copy failed verification. |
PasswordError
|
The source is encrypted and the password does not grant level 2. |
IoError
|
|
(FormatError, CrcError)
|
Damaged source data (never re-sealed into the copy). |
Every refusal carries ``.code`` (:data:`REFUSAL_CODES`) and ``.details``
|
|
(``channel``, ``field``, ``inspect_code``), as for
|
|
:func:`mef3io.anonymize_session`. :func:`check` returns the same refusals
|
|
without writing.
|
|
Examples:
>>> rep = mef3io.transform_session(
... "patient.mefd", "patient_eeg.mefd", exclude_channels=["ECG"],
... metadata={"channel_descriptions": {"Fp1": "frontal pole, left"}})
>>> rep.verification.ok
True
mef3io.transform.check ¶
check(src, dst, *, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', rename: bool = True, password: str = '', cancel: Optional[CancelToken] = None) -> CheckReport
Every check :func:transform_session makes before it writes, without writing.
Takes the same arguments (no progress) and runs the same checks,
through the same pre-flight code in the C++ core, so the two cannot
disagree. Instead of raising the first refusal it returns all of them,
each with its stable code (:data:REFUSAL_CODES). It writes nothing
and reads only metadata (never a .tdat). What it cannot decide: a
data block failing its CRC (block-crc, data-damaged) and the
final verification (verify-failed).
Returns:
| Type | Description |
|---|---|
CheckReport
|
|
mef3io.transform.verify ¶
verify(src, dst, *, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, time_range: Optional[Tuple[int, int]] = None, exact: bool = True, records: str = 'keep', password: str = '', progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> VerifyReport
Check a transformed copy against its source. Reads both; writes nothing.
:func:transform_session runs this on every copy before publishing it.
Pass the metadata (and output_password) the copy was made with.
The channels are read from the copy; channels and
exclude_channels are accepted for symmetry and not needed.
It checks that the validator reports nothing new; the copy holds a subset of the source's channels, segment for segment; every data block's payload and statistics (compared decrypted) are byte-identical; every time equals the source's; every other byte of every header, metadata section and record body equals the source's, except the names (when renamed), UUIDs, password fields, encryption flags, provenance and CRCs, and the metadata fields, which hold exactly the requested edits.
Returns:
| Type | Description |
|---|---|
VerifyReport
|
|
mef3io.TransformReport
dataclass
¶
TransformReport(destination: str, session_name: str, channels_kept: Tuple[str, ...], channels_excluded: Tuple[str, ...], channels_without_samples: Tuple[str, ...], segments: int, blocks: int, blocks_reencrypted: int, blocks_decrypted: int, records: int, seizure_channel_entries_dropped: int, blocks_trimmed: int, records_outside_window: int, metadata_fields_changed: int, bytes_written: int, output_encrypted: bool, not_copied: Tuple[str, ...], notes: Tuple[str, ...], verification: VerifyReport, records_dropped: int = 0)
Result of :func:transform_session.
Attributes:
| Name | Type | Description |
|---|---|---|
destination |
str
|
The copy. |
session_name |
str
|
The copy's session name (its folder's stem). |
channels_kept, channels_excluded |
tuple of str
|
|
channels_without_samples |
tuple of str
|
A cut: channels left out because they have no sample in the window
(also in |
segments, blocks, records |
int
|
What was copied. |
blocks_reencrypted, blocks_decrypted |
int
|
Data blocks the source had encrypted, written encrypted with the new passwords, or decrypted because the copy has none. |
seizure_channel_entries_dropped |
int
|
Per-channel entries of |
blocks_trimmed, records_outside_window |
int
|
A cut ( |
records_dropped |
int
|
|
metadata_fields_changed |
int
|
Metadata fields whose stored value changed (one per field per segment). |
bytes_written |
int
|
|
output_encrypted |
bool
|
|
not_copied |
tuple of str
|
Files in the source that are not part of the MEF layout and were left out (source-relative paths). |
notes |
tuple of str
|
|
verification |
VerifyReport
|
The check the copy passed before it was published. |
mef3io.transform.REFUSAL_CODES
module-attribute
¶
REFUSAL_CODES: Dict[str, Tuple[str, str]] = {**COPY_REFUSAL_CODES, 'metadata-unknown': ('ValueError', 'check'), 'metadata-too-long': ('ValueError', 'check'), 'metadata-invalid': ('ValueError', 'check'), 'metadata-channel-unknown': ('ValueError', 'check')}
mef3io.transform.METADATA_FIELDS
module-attribute
¶
METADATA_FIELDS: Dict[str, int] = {'subject_name_1': 128, 'subject_name_2': 128, 'subject_id': 128, 'recording_location': 512, 'session_description': 2048, 'channel_descriptions': 2048, 'reference_descriptions': 2048}
Merged sessions¶
See Merging recordings. Python and C++ only (no MATLAB, no C ABI).
mef3io.merge_sessions ¶
merge_sessions(sources: Sequence, dst, *, passwords: Optional[Sequence[str]] = None, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, records: str = 'keep', time_ranges: Optional[Sequence[Optional[Tuple[int, int]]]] = None, exact: bool = True, anonymize: Union[None, bool, Mapping] = None, n_threads: int = 0, progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> MergeReport
Merge two or more recordings into ONE new session.
One call can do the whole pipeline: leave channels out, trim each
recording to its own time range, make the result anonymous, leave the
annotations out, and write ONE new session. The copy pass reads each
source byte once and writes it once (several channels at a time, see
n_threads); :func:verify then reads the sources and the result
again before anything is published.
The sources are never modified. The result is built next to dst in
<dst>.partial/, checked with :func:verify against every source, and
only then moved to dst. A failure, a refusal or a cancel leaves
nothing behind.
What is refused before anything is written (:func:check lists
every one):
merge-overlap: two sources hold samples of the same channel at the same moment. This is decided on the RECORDED data (from the block indexes), so recordings that take turns merge: A records and stops, B records and stops, A continues. A source's own slightly overlapping blocks (recorder jitter) never count.merge-rate-differs: a channel has different sampling rates.merge-rto-differs: the sources use different recording-time offsets.merge-identity-differs: the sources' subject fields (names, ID, location) differ, which may mean different patients. Set the fields for the result withmetadatato merge anyway. Withanonymizethe fields it clears (clear=, default all) may differ: the result carries none of them.
How the result is laid out. Each source segment becomes a segment of the result. A segment that has another source's data in one of its gaps is split there (at a block boundary, nothing decoded), so that the result's segments never overlap in time. They are ordered by time and numbered from 0 in each channel. Channels held by only some sources are kept. Annotations at the session and channel level are merged in time order; a segment's own annotations stay with it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sources
|
sequence of str or path-like
|
Two or more sessions ( |
required |
dst
|
str or path - like
|
The result, a new |
required |
passwords
|
sequence of str
|
The password opening each source, in order (level 2 for encrypted ones). Omit for unencrypted sources. |
None
|
channels
|
sequence of str
|
Which channels the result holds, over all sources. |
None
|
exclude_channels
|
sequence of str
|
Which channels the result holds, over all sources. |
None
|
output_password
|
(str, str)
|
Encrypt the result with these two passwords; |
None
|
records
|
str
|
|
'keep'
|
metadata
|
mapping
|
Descriptive fields for the result, as for
:func: |
None
|
time_ranges
|
sequence
|
One entry per source, in order: |
None
|
exact
|
bool
|
With |
True
|
anonymize
|
True or mapping
|
Make the result anonymous in the same pass, with
:func: |
None
|
n_threads
|
int
|
Threads for the whole operation (:doc: |
0
|
progress
|
optional
|
As for :func: |
None
|
cancel
|
optional
|
As for :func: |
None
|
Returns:
| Type | Description |
|---|---|
MergeReport
|
|
mef3io.merge.check ¶
check(sources: Sequence, dst, *, passwords: Optional[Sequence[str]] = None, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, records: str = 'keep', time_ranges: Optional[Sequence[Optional[Tuple[int, int]]]] = None, exact: bool = True, anonymize: Union[None, bool, Mapping] = None, n_threads: int = 0, cancel: Optional[CancelToken] = None) -> CheckReport
Every check :func:merge_sessions makes before it writes, without writing.
Reads only metadata and the block indexes of segments that overlap in
time across sources (with time_ranges, of the blocks around each
range); never a .tdat. Takes the arguments of :func:merge_sessions.
mef3io.merge.verify ¶
verify(sources: Sequence, dst, *, passwords: Optional[Sequence[str]] = None, channels: Optional[Sequence[str]] = None, exclude_channels: Optional[Sequence[str]] = None, output_password: Optional[Sequence[str]] = None, metadata: Optional[Mapping] = None, records: str = 'keep', time_ranges: Optional[Sequence[Optional[Tuple[int, int]]]] = None, exact: bool = True, anonymize: Union[None, bool, Mapping] = None, n_threads: int = 0, progress: Optional[Callable[[int, int], object]] = None, cancel: Optional[CancelToken] = None) -> VerifyReport
Check a merged session against its sources. Reads all; writes nothing.
Pass the arguments the result was made with. Every segment of the result
is checked against the piece of the source segment it came from (blocks
byte-identical, metadata as the source's plus the edits); annotations
against the sources' merged in time order; segments must not overlap; and
the validator must report nothing on the result. With anonymize every
time must be exactly the shift away and no text field or path may hold
the identity of ANY source. time_ranges and anonymize are the
ones the result was made with.
mef3io.MergeReport
dataclass
¶
MergeReport(destination: str, session_name: str, sources: Tuple[str, ...], channels_kept: Tuple[str, ...], channels_excluded: Tuple[str, ...], segments: int, segments_split: int, blocks: int, blocks_reencrypted: int, blocks_decrypted: int, records: int, seizure_channel_entries_dropped: int, metadata_fields_changed: int, bytes_written: int, output_encrypted: bool, not_copied: Tuple[str, ...], notes: Tuple[str, ...], verification: VerifyReport, channels_without_samples: Tuple[str, ...] = (), blocks_trimmed: int = 0, records_outside_window: int = 0, pseudonym: str = '', shift_us: Optional[int] = None, source_recording_time_offset: int = 0, record_text_fields_changed: int = 0, record_bodies_cleared: int = 0, source_count: int = 0, source_start_us: Optional[int] = None, source_end_us: Optional[int] = None, source_channel_times: Dict[str, Tuple[Optional[int], Optional[int]]] = dict(), records_dropped: int = 0, units_mixed: Dict[str, Tuple[str, ...]] = dict())
Result of :func:merge_sessions.
Attributes:
| Name | Type | Description |
|---|---|---|
destination, session_name |
str
|
|
sources |
tuple of str
|
As given. ANONYMIZED result: memory only, like the re-identification key
below (the paths name the patient): empty in :meth: |
channels_kept, channels_excluded |
tuple of str
|
Over all sources. |
channels_without_samples |
tuple of str
|
|
segments |
int
|
Segments of the result. |
segments_split |
int
|
Source segments split in two or more pieces because another source's data lies in one of their gaps. |
blocks, records, blocks_reencrypted, blocks_decrypted |
int
|
|
blocks_trimmed, records_outside_window |
int
|
|
records_dropped |
int
|
|
seizure_channel_entries_dropped, metadata_fields_changed |
int
|
|
bytes_written |
int
|
|
output_encrypted |
bool
|
|
not_copied, notes |
tuple of str
|
|
verification |
VerifyReport
|
The check the result passed before it was published. |
pseudonym |
str
|
|
shift_us |
int or None
|
|
source_recording_time_offset |
int
|
|
record_text_fields_changed, record_bodies_cleared |
int
|
|
source_start_us, source_end_us |
int or None
|
|
source_channel_times |
dict of str to (int or None, int or None)
|
|
with_original_times_in_pickle ¶
with_original_times_in_pickle() -> 'MergeReport'
A copy of this report whose pickle KEEPS the original times (see
:meth:mef3io.anonymize.AnonymizeReport.with_original_times_in_pickle).
to_dict ¶
to_dict(*, include_original_times: bool = False) -> dict
Plain dict (JSON-serialisable). The sources' original times (the
re-identification key) are left out unless include_original_times.
mef3io.merge.REFUSAL_CODES
module-attribute
¶
REFUSAL_CODES: Dict[str, Tuple[str, str]] = {**_TRANSFORM_CODES, **_ANONYMIZE_CODES, 'merge-too-few': ('ValueError', 'check'), 'merge-passwords-count': ('ValueError', 'check'), 'merge-time-range': ('ValueError', 'check'), 'merge-time-ranges-count': ('ValueError', 'check'), 'merge-anonymize-metadata': ('ValueError', 'check'), 'merge-threads-invalid': ('ValueError', 'check'), 'merge-rename': ('ValueError', 'check'), 'merge-rto-differs': ('WriteConflictError', 'check'), 'merge-rate-differs': ('WriteConflictError', 'check'), 'merge-identity-differs': ('WriteConflictError', 'check'), 'merge-overlap': ('WriteConflictError', 'check')}
Cancelling and progress¶
See the Python guide.
mef3io.CancelToken ¶
CancelToken()
A flag that stops a running read or long operation.
Pass it as cancel= to :meth:Reader.read, :meth:Reader.read_raw,
:func:archive_session, :func:extract_session, :func:inspect,
:meth:Validator.validate, :func:validate_session,
:func:repair_session or :func:recover_session, and call
:meth:cancel from any thread. The operation notices between units of
work (a decoded block, a 1 MB copy chunk, a segment) and raises
:class:Cancelled.
A token is one-shot: once cancelled it stays cancelled, so create a new one per request. One token may be shared by several calls to stop them together (for example every channel of one viewer request).
Cancellation and progress are available in Python and C++ only. MATLAB
has neither: a MEX call blocks MATLAB's only interpreter thread, so
nothing could call :meth:cancel while it runs.
Examples:
>>> token = mef3io.CancelToken()
>>> threading.Timer(0.5, token.cancel).start()
>>> try:
... x = reader.read("ch1", cancel=token)
... except mef3io.Cancelled:
... pass # a newer request superseded this one
raise_if_cancelled ¶
raise_if_cancelled() -> None
Raise :class:Cancelled if the token is set.
For a caller's own loops between mef3io calls — for example reading many channels for one request — so they stop at the same points the library does.
mef3io.Cancelled ¶
Bases: Exception
An operation was stopped through its :class:CancelToken.
Validation and repair¶
See the validation guide for what each check means.
mef3io.Validator ¶
Validator(path: str, password: str = '', channels: Sequence[str] | None = None, segments: Sequence[int] | None = None, exact_difference_bytes: bool = True)
Check one MEF 3.0 session. Read-only: it cannot write a byte.
There is no repair method here by design — see :func:repair_session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
A |
required |
password
|
str
|
Needed to read section 2 of an encrypted session. |
''
|
channels
|
sequence of str
|
Restrict to these channels. Default: every channel. |
None
|
segments
|
sequence of int
|
Restrict to these segment numbers. Default: every segment. |
None
|
exact_difference_bytes
|
bool
|
Read every RED block header to learn the real
|
True
|
available_checks
staticmethod
¶
available_checks() -> list[Check]
Every check in the registry, in the order they run.
describe_check
staticmethod
¶
describe_check(check_id: str) -> Check | None
One check by id, or None if unknown.
validate ¶
validate(check_ids: Iterable[str] | None = None, *, cancel: CancelToken | None = None, progress: Callable[[int, int], object] | None = None) -> Report
Run every check (or only check_ids) and return the full report.
Never modifies the session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
check_ids
|
iterable of str
|
Run only these checks (the integrity checks always run). |
None
|
cancel
|
CancelToken
|
Stop the pass from another thread. Checked per segment and, with
|
None
|
progress
|
callable
|
|
None
|
mef3io.Report
dataclass
¶
Report(findings: tuple[Finding, ...] = (), skipped: tuple[SkippedSegment, ...] = (), segments_checked: int = 0, segments_repaired: int = 0, checks_run: tuple[str, ...] = (), checks_repaired: tuple[str, ...] = (), path: str = '', channels: tuple[str, ...] = (), segments: tuple[int, ...] = (), measured_difference_bytes: bool = True)
The result of a validate or repair pass.
ok
property
¶
ok: bool
True when nothing worse than a warning is left outstanding.
Findings this pass repaired no longer count against it, so a repair
that resolved everything reports ok. A skipped segment always
does count — an unchecked segment is not a clean one.
unresolved
property
¶
unresolved: tuple[Finding, ...]
Findings that are still outstanding — nothing repaired this pass.
repairable_check_ids
property
¶
repairable_check_ids: list[str]
Distinct check ids that reported something a repair could fix.
Pass these to :func:repair_session to opt in deliberately.
scope_caveat ¶
scope_caveat() -> str
One sentence naming what this run did not cover, or "" if it covered everything.
summary ¶
summary(max_examples: int = 3) -> str
A full human-readable report: counts, then each check that fired.
mef3io.Finding
dataclass
¶
Finding(check_id: str, severity: str, channel: str, segment: int, path: str, field: str, stored: str, expected: str, message: str, repairable: bool, repaired: bool)
One problem, found by one check, in one segment.
Findings about a record (annotation) file at channel or session level carry
segment == -1 (and channel == "" for the session).
mef3io.Check
dataclass
¶
Check(id: str, title: str, description: str, severity: str, repairable: bool)
One check in the registry.
mef3io.available_checks ¶
available_checks() -> list[Check]
Every check, in the order they run. Ids are stable API.
mef3io.describe_check ¶
describe_check(check_id: str) -> Check | None
One check by id, or None if unknown.
mef3io.SkippedSegment
dataclass
¶
SkippedSegment(channel: str, segment: int, path: str, reason: str)
A segment that could not be checked, and why. Never silently dropped.
mef3io.validate_session ¶
validate_session(path: str, *, cancel: CancelToken | None = None, progress: Callable[[int, int], object] | None = None, **kwargs) -> Report
Check a session and return the report. Reads only.
Accepts the same keyword arguments as :class:Validator, plus cancel
and progress as in :meth:Validator.validate.
mef3io.repair_session ¶
repair_session(path: str, check_ids: Sequence[str], backup: bool = True, *, cancel: CancelToken | None = None, progress: Callable[[int, int], object] | None = None, **kwargs) -> Report
Rewrite the declarations named in check_ids — and only those.
This is the only function in mef3io that modifies an existing session, and
it is deliberately not a method of :class:Validator: inspecting a session
and rewriting one should not be reachable through the same object.
Every check still runs, so the returned report describes the whole session;
Finding.repaired marks what was actually written — a repair that
declines to change anything is not counted.
Only declarations are rewritten: metadata section 2 and the universal
headers of .tmet/.tidx/.tdat. Sample data and the block index
are never touched. A segment whose CRCs do not verify is reported and left
alone, and a .mefd.tar archive is refused outright (extract it first).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
A |
required |
check_ids
|
sequence of str
|
Which repairs to apply. Required and non-empty — repairs are never
implicit. :attr: |
required |
backup
|
bool
|
Ignored since 2.0.1: a backup is always kept, in
|
True
|
cancel
|
CancelToken
|
Stop the repair from another thread. Repairs are decided during the
validation pass and written only after it: a cancel during the pass
raises :class: |
None
|
progress
|
callable
|
|
None
|
**kwargs
|
The same keyword arguments as :class: |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
mef3io.recover_session ¶
recover_session(path: str, apply: bool = False, backup: bool = True, password: str = '', *, cancel: CancelToken | None = None, progress: Callable[[int, int], object] | None = None) -> RecoveryReport
Make each segment's block index and data agree after an interrupted write.
This is not :func:repair_session, which only ever rewrites
declarations and is safe to run on anything. Recovery may truncate the
index, rebuild index entries from the data file, and drop a trailing
fragment — so it is a dry run unless you ask for it, and it backs up what it
changes first.
An interrupted append leaves one of two shapes, and they are handled differently because one has lost data and the other has not:
- index ahead of data — entries reference bytes that never landed. Those samples do not exist, so the entries are dropped.
- data ahead of index — blocks reached
.tdatbut the index was not extended. Those samples do exist, so they are recovered: a RED block header carries the sample count, byte count, start time and discontinuity flag, which is everything an index entry needs. Only blocks whose CRC verifies are indexed; a torn tail is not a block.
Both are reachable after a power cut or OS crash during an append: the
writer flushes only each segment's .tmet before renaming it, not the
.tdat or .tidx it extends in place, and this function is what makes
that trade safe. Only the last append of a segment can be affected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
|
required |
apply
|
bool
|
False reports what it would do and writes nothing. |
False
|
backup
|
bool
|
Ignored since 2.0.1: what changes is always saved to
|
True
|
password
|
str
|
Only needed for encrypted sessions. |
''
|
cancel
|
CancelToken
|
Stop recovery from another thread. Every segment is scanned before
anything is written: a cancel during the scan raises
:class: |
None
|
progress
|
callable
|
|
None
|
Returns:
| Type | Description |
|---|---|
RecoveryReport
|
|
Notes
With apply=True the declarations of every segment it changed (and of
a channel's last segment whose .tmet is newer or older than its index,
which a power cut can leave) are re-derived from the index through
:func:repair_session — so recovery alone leaves a valid session after
an unclean shutdown. Only the segments it reports are touched (the dry
run lists their check ids in RecoveredSegment.declarations); a
healthy segment's findings are left to :func:repair_session. A later
segment whose .tmet was lost is rebuilt from the previous segment's.
mef3io.RecoveryReport
dataclass
¶
RecoveryReport(segments: tuple[RecoveredSegment, ...] = (), record_files: tuple[RecoveredRecordFile, ...] = (), skipped: tuple[str, ...] = (), segments_examined: int = 0, applied: bool = False, backup_root: str = '', notes: tuple[str, ...] = ())
What recover_session found, and did or would do.
mef3io.RecoveredSegment
dataclass
¶
RecoveredSegment(channel: str = '', segment: int = 0, path: str = '', blocks_before: int = 0, blocks_after: int = 0, blocks_recovered: int = 0, blocks_dropped: int = 0, tdat_bytes_dropped: int = 0, action: str = '', declarations: tuple[str, ...] = ())
One segment that an interrupted write left inconsistent.
mef3io.RecoveredRecordFile
dataclass
¶
RecoveredRecordFile(path: str = '', records_before: int = 0, records_after: int = 0, bytes_dropped: int = 0, records_damaged: int = 0, index_rebuilt: bool = False, action: str = '')
One annotation file (.rdat + .ridx) an interrupted write left inconsistent.
Recovery keeps every whole record, drops only a torn fragment after the
last whole one (backed up) and rebuilds the index and headers from the
records. A record that fails its CRC (records_damaged) is damage, not
an interrupted write: it is reported and kept, never removed.
Errors¶
mef3io raises mef3io.MefError and its subclasses FormatError (damaged
or malformed files), CrcError (a checksum does not match),
PasswordError (a missing or wrong password), IoError (files, paths,
session naming) and WriteConflictError (a write that would break the
session). All of them subclass RuntimeError. A refusal of
anonymize_session also carries a stable code and details
(Refusal codes); bad arguments raise
ValueError or TypeError. mef3io.Cancelled is deliberately not a
MefError. mef3io.have_cpp_backend() says whether the compiled core
is available.
Warnings¶
Every warning carries its session in w.message.path and never in its text;
see Warnings carry the path as data.
mef3io.MefWarning ¶
MefWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: UserWarning
Base class of every warning mef3io emits.
The message never contains the session's location (its directory or
archive path) or its name: they appear as <session>, also inside an
archive (<session>::<session>/...) and in the session's own file
names (<session>.rdat). Nothing else is scrubbed: channel names and
the names of files inside the session can appear, and those can be
identifying too. A caller with a stricter rule should log only the class
name, or nothing.
What the message summarises is on the warning object, which Python passes
as w.message (warnings.catch_warnings(record=True),
pytest.warns) or raises under -W error:
Attributes:
| Name | Type | Description |
|---|---|---|
path |
str or None
|
The session (or, for a writer, the session being written) the warning
is about, exactly as it was passed in. |
details |
tuple
|
The structured items behind the message: the problem dicts of an unreadable segment, the block-copy mismatches, the unset declarations, and so on (each subclass says which). These MAY contain paths. |
Python's ``"default"`` and ``"module"`` filter actions show a given |
|
|
warning once per session (per calling file), not once per process |
the
|
|
path is no longer in the text, so each session keeps its own registry. |
|
|
Filter every mef3io warning at once with |
:
|
warnings.filterwarnings("ignore", category=mef3io.MefWarning) |
Examples:
>>> with warnings.catch_warnings(record=True) as caught:
... warnings.simplefilter("always")
... r = mef3io.Reader(path)
>>> for w in caught:
... if issubclass(w.category, mef3io.MefWarning):
... log.warning("%s", w.message) # no session path or name
... where = w.message.path # the path, as data
mef3io.SessionDeclarationWarning ¶
SessionDeclarationWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A session leaves size declarations unset that other readers rely on.
Raised on open, once per process per session (Python's default warning filter suppresses a repeat of the same message from the same line). It never affects mef3io's own reads — mef3io sizes every buffer from the block headers themselves — but meflib-based readers (most established MEF tooling) allocate from metadata section 2 before decoding, and cannot tell an unset field from a real measurement.
Silence it like any warning::
warnings.filterwarnings("ignore", category=mef3io.SessionDeclarationWarning)
or per call with Reader(path, warn_declarations=False). To see the
detail, or to fix it, use :class:mef3io.Validator.
Like every :class:MefWarning, the message carries no path:
w.message.path is the session and w.message.details the unset
declarations (dicts with channel, segment and field, as
:attr:Reader.declaration_issues).
mef3io.UnreadableSegmentWarning ¶
UnreadableSegmentWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
Part of this session could not be read and is being skipped.
Only ever raised by a Reader opened strict=False. The skipped segments'
time spans read back as NaN — indistinguishable, in the returned array,
from a genuine recording gap. That is the hazard lenient mode trades for
availability, so it is never silent: see :attr:Reader.problems for the
segment paths and the reason each one failed.
Strict is the default. An unreported CRC failure is how corrupt scaling reaches an analysis unnoticed; lenient mode exists to salvage the intact 99% of an archive, not to relax the checks.
Also raised by :class:Writer (and the compat MefWriter) for segment
metadata it had to skip; details are then the strings of
Writer.unreadable_metadata. From a Reader, details are the
:attr:Reader.problems dicts it is about. w.message.path is the
session; the message itself carries no path (see :class:MefWarning).
mef3io.BlockCopyWarning ¶
BlockCopyWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A block's two stored copies of its start time or sample count disagree.
Every RED block records its start time and sample count TWICE — once in the
.tidx entry and once in the block's own header inside the .tdat.
Nothing in the format keeps them in step. mef3io places data by the HEADER,
which is the copy the per-block CRC protects and the copy meflib and pymef
read, so a file whose copies have drifted apart still reads the same as it
does under the legacy stack.
This warning means the file itself is internally inconsistent — it does not mean the read is wrong. It fires only on such a file: on anything mef3io or the legacy stack wrote, the copies are identical and nothing is emitted.
The structured detail is in the block_copy_mismatches key returned by
:meth:Reader.read_raw. Silence it like any warning::
warnings.filterwarnings("ignore", category=mef3io.BlockCopyWarning)
w.message.details holds the same mismatch dicts and
w.message.path the session; the message carries no path (see
:class:MefWarning).
mef3io.UnscaledSamplesWarning ¶
UnscaledSamplesWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
Values read from a segment whose conversion factor is not set.
A segment stores units_conversion_factor 0 — MEF's "no entry", which
meflib's and pymef's section-2 templates hold until a writer sets it — so
there is no factor to convert its counts into units. :meth:Reader.read,
:meth:Reader.block_envelope, :class:MultiReader and the compat
MefReader.get_data then return those samples as the RAW STORED COUNTS
(factor 1); before 2.0 they read back as zeros. Emitted once per Reader
and channel, the first time such a segment is read.
:meth:Reader.segments shows which segments store 0, and
:meth:Reader.read_raw reports the factor as stored (0) in
units_conversion_factors. w.message.details holds the channel name.
mef3io.LegacyPasswordWarning ¶
LegacyPasswordWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
The password opened this session only through mef3io's OLD byte mapping.
MEF derives a 16-byte key from the password one byte per character.
meflib (and so pymef and mef_tools) keeps the low byte of each
character's Unicode code point; mef3io 1.2.0 and earlier kept the last byte of
its UTF-8 encoding instead. The two agree on ASCII and disagree on
everything else ("é" is 0xE9 in meflib, 0xA9 in old mef3io) — issue #84.
mef3io now writes with meflib's mapping and reads with it first. A session
an older mef3io encrypted with a non-ASCII password still opens, through a
fallback to the old mapping, and this warning says so (on open for the
metadata, from :meth:Reader.records for record files): the data is intact,
but meflib and pymef cannot open that session with that password. To make
it readable everywhere, make a re-keyed copy:
:func:anonymize_session with output_password=(level1, level2)
re-encrypts every file with meflib's mapping. That copy is NOT a pure
re-key: it is de-identified and every time is shifted (by at least the
recording's span + 1 day). A pure re-key is transform_session
(see the transform guide).
The fallback cannot tell who wrote a file: it also accepts a SECOND
password, the one whose old-mapping bytes equal the real key (for a
session keyed with "b©", also "bé"), whoever wrote it. That is
inherent to any compatibility fallback and is why it is announced.
Appending to such a session keeps the existing segments' keys (rewriting them would mean re-encrypting them); new segments use meflib's mapping.
w.message.details lists what matched that way: channel names (on
open), record file paths (from :meth:Reader.records) or segment paths
(from :class:Writer). w.message.path is the session; the message
carries no path (see :class:MefWarning). :func:mef3io.inspect does not
emit it: it reports the same condition as
SessionSummary.legacy_password_mapping.
mef3io.PasswordInteropWarning ¶
PasswordInteropWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A password that other MEF readers handle differently than you may expect.
Raised by :class:Writer for a password longer than 15 characters (meflib
and pymef refuse it, so only mef3io will open the session) or one that
contains a character above U+00FF (MEF keeps only the low byte of each
code point, as meflib does, so a different password can unlock the same
session). The session is still written; the warning only tells you about
its reach. Short Latin-1 passwords (ASCII, or e.g. "é", "ü") raise
nothing.
w.message.path is the session being written; details is
("level-1",) or ("level-2",), which password it is about. The
password itself never appears in the warning.
mef3io.MixedFactorsWarning ¶
MixedFactorsWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A copy holds a channel whose segments use different conversion factors.
Raised by :func:mef3io.merge_sessions and :func:mef3io.transform_session
when the result has such a channel (sources recorded at different gains,
merged or kept as they are). Nothing is wrong with the file: mef3io and
pymef scale each segment by its own factor and read the right values.
Older tools that scale by the channel-level factor, such as the legacy
mef_tools MefReader.get_data, read such a channel as all zeros
(meflib stores 0 at channel level when segments differ). Use mef3io or
pymef to read it, or write it again with one factor per channel.
w.message.path is the result session; details lists the channel
names.
mef3io.MixedUnitsWarning ¶
MixedUnitsWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A merged channel holds segments with different units strings.
Raised by :func:mef3io.merge_sessions when the data the result keeps
carries more than one units string for a channel ("mV" next to
"uV"). Units are free text, so the merge goes ahead: every segment
keeps its own string, samples are never rescaled, and readers label the
whole channel with the FIRST kept segment's units. Check that the
recordings really use the units you expect. "uV", "µV"
(micro sign) and "μV" (Greek mu) count as the same unit and never
warn. The merge report's units_mixed and notes give the same
information; the validator reports it as units.mixed.
w.message.path is the result session; details lists the channel
names.
mef3io.StepFallbackWarning ¶
StepFallbackWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
Writer.write(..., precision="step") on a NEW channel could not use a step.
Raised once per such write. The data was still written, with the default
(inferred power-of-ten) precision — what a default write stores: the
data was not a whole multiple of one step (filtered or resampled data
never is), showed too few distinct values to establish one (a DC level,
a square wave: fewer than 16 distinct magnitudes in the sample), or its
counts would not fit int32. The summary's step_fallback says which.
(On a channel that already has data, "step" never falls back: data
off the channel's factor is refused with WriteConflictError.)
w.message.details is [channel, reason]; w.message.path the
session.
mef3io.ClampedSamplesWarning ¶
ClampedSamplesWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A float write stored some samples as the nearest storable value.
MEF stores int32 counts; a sample whose count at the conversion factor
would leave that range, or that is +inf/-inf, is stored as the
largest count, +-(2**31 - 1) — meflib's own "positive/negative
infinity" values (RED_POSITIVE_INFINITY/RED_NEGATIVE_INFINITY,
meflib.h, which its RED_round clamps to as well) — so it reads back
as +-(2**31 - 1) * factor. The write succeeds; nothing wraps. Typical
cause: a saturation artifact in an append to a channel whose factor is
fine. NaN is never clamped: it stays a gap.
Raised once per such write. w.message.details is
({"channel": ..., "clamped": n, "infinite": k},) (k of the n
were infinite); the text carries counts only, never sample values or
times. The write summary has the same numbers (clamped,
clamped_infinite).
mef3io.WriterFlushWarning ¶
WriterFlushWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
A :class:Writer that was never closed could not flush when it was
garbage-collected.
Only :meth:Writer.close, :meth:Writer.flush and with can raise;
a destructor cannot, so this warning is all it can say. Data written since
the last successful flush may not be on disk: run recover on the
session. w.message.path is the session; details is [reason].
mef3io.BackupAlwaysKeptWarning ¶
BackupAlwaysKeptWarning(message: Any = '', *, path: Optional[str] = None, details: Iterable = ())
Bases: MefWarning
backup=False (or --no-backup) was passed and ignored.
recover_session and repair_session always keep a backup of what
they change; the argument is accepted only so existing callers keep
working.
Metadata objects¶
mef3io.Metadata
dataclass
¶
Metadata(subject: Subject = Subject(), acquisition: Acquisition = Acquisition())
Session-wide metadata: a :class:Subject and an :class:Acquisition
block. Written to every channel; on read, reflects the first channel.
mef3io.Subject
dataclass
¶
Subject(name_1: str = '', name_2: str = '', id: str = '', recording_location: str = '', gmt_offset: int = 0)
Subject / recording-context metadata (MEF section 3, level-2 encrypted).
mef3io.Acquisition
dataclass
¶
Acquisition(session_description: str = '', channel_description: str = '', reference_description: str = '', acquisition_channel_number: int = 1, low_frequency_filter: float = _UNSET_HZ, high_frequency_filter: float = _UNSET_HZ, notch_filter: float = _UNSET_HZ, line_frequency: float = _UNSET_HZ)
Descriptive / acquisition metadata (MEF section 2, level-1 encrypted).
Filter settings default to -1.0 meaning "not recorded".
Legacy mef_tools compatibility¶
Drop-in replacements for mef_tools.io — from mef3io import MefReader,
MefWriter. Same call shapes and defaults as the legacy classes; see the
legacy comparison for the measured differences.
mef3io.compat.MefReader ¶
MefReader(session_path: str, password2: Optional[str] = None, warn_declarations: bool = True)
mef_tools.io.MefReader-compatible reader.
session_path may also be an uncompressed tar archive of a session
(name.mefd.tar, see :func:mef3io.archive_session).
block_envelope ¶
block_envelope(channel: str, t_stamp1=None, t_stamp2=None) -> dict
Per-block minimum and maximum from the block index alone.
Not part of the legacy mef_tools API: the same as
:meth:mef3io.Reader.block_envelope, here so code that holds a
MefReader (such as a server) needs no second reader. Returns NumPy
arrays start_uutc, end_uutc, min, max, one row per
data block overlapping [t_stamp1, t_stamp2), scaled like
:meth:get_data.
get_annotations ¶
get_annotations(channel: Optional[str] = None)
Records as a list of dicts (legacy mef_tools signature).
channel=None returns session-level records; a channel name returns
that channel's CHANNEL-level records only, as the legacy reader did.
Segment-level records are not included here (the native
:meth:mef3io.Reader.records does include them for a channel, or use
Reader.records(channel, level="segment")). Every dict carries
level / channel / segment; see :meth:mef3io.Reader.records
for the fields. Returns None (with a printed warning) on failure,
as the legacy API did.
mef3io.compat.MefWriter ¶
MefWriter(session_path, overwrite=False, password1=None, password2=None, verbose=False, metadata=None, wait_for_disk=False)
mef_tools.io.MefWriter-compatible writer.
Like the legacy writer, appends extend the channel's last segment in place
(in-segment append); pass new_segment=True to start a fresh segment.
Metadata works two ways: mutate the legacy section3_dict /
section2_ts_dict (e.g. w.section3_dict['subject_ID'] = 'Smith') as
with mef_tools, or use the modern object via set_metadata / the
metadata= argument (mef3io.Metadata).
mef_block_len
property
writable
¶
mef_block_len
RED block length in samples (None = derive from fs, like legacy).
max_nans_written
property
writable
¶
max_nans_written: int
Kept for legacy compatibility. mef3io always splits data on NaN runs (never stores NaN as values), i.e. it behaves like the legacy writer's recommended setting of 0; other values are accepted and ignored.
record_offset
property
writable
¶
record_offset: int
Kept for legacy compatibility; non-zero values are accepted and ignored -- mef3io never creates a recording-time offset.
Times are UTC, always. A new session is written at offset 0 with
times stored as -t, as before. A session that already carries a
non-zero offset (e.g. one the legacy stack wrote with
record_offset) keeps it: segments and channels mef3io adds reuse
it, and every time mef3io writes there -- data and annotations -- is
stored as rto - t, so the real date stays hidden and the existing
annotations stay readable. A time at or before the offset cannot be
stored that way and is refused, as is any write while the session's
segments disagree on the offset or it is locked (level-1 password
only).
set_metadata ¶
set_metadata(metadata) -> None
Set session-wide subject/acquisition metadata (a
:class:mef3io.Metadata or a flat dict). Applied on the next write.
get_mefblock_len ¶
get_mefblock_len(fs: float) -> int
Block length that will be used for data at fs (legacy formula).
flush ¶
flush()
Return only when everything written so far is on disk (one
combined wait), as :meth:mef3io.Writer.flush.