mef3io 2.0.0¶

Released 2026-10-26.

A major release: it removes the per-call n_threads argument and the durability option, and changes what a few calls return. Read Upgrading from 1.x before you update. The detailed sections follow it.

What's new¶

Opening a session is parallel, with one thread rule for everything. An object's n_threads covers opening, file access, decoding, decrypting and encoding (0 = automatic, 1 = no extra threads) and can be changed at runtime. On a slow network share a 209-channel session opens in 10.9 s instead of 173 s. See Threads.
Read many channels in one call. r.read(["ch1", "ch2"], t0, t1) returns a 2-D array, each row identical to the single-channel read, and dtype="float32" halves the memory.
Conversion factors are per segment. A channel whose segments store different factors now reads back correctly, and info(), segments() and read_raw() say which factor applies where.
anonymize_session keeps the dates and times unless you give a shift. The identity is removed exactly as before. A kept channel named after the patient is now refused before anything is copied.
precision="step" stores data from an amplifier (for example 0.195 µV per count) as its original counts: 1.52 MB instead of 6.03 MB for an hour at 512 Hz.
Out-of-range samples are clamped, with a warning, instead of wrapping silently into a wrong value.
durability= is gone. Writer.flush(), close() and the opt-in wait_for_disk=True replace it; see When is my data on disk?.
Recovery covers every crash state a power cut can leave after an append, and re-derives the declarations itself.

Upgrading from 1.x¶

Each item says what changed and what to do. The details are in the sections below.

Threads¶

What changed	What to do
`Reader.read` / `read_raw` (and `MultiReader`) no longer take `n_threads=`	Set `r.n_threads = N` instead; it applies from the next call. Passing the argument raises `TypeError`.
`n_threads` must be an int >= 0; a bool or float raises `TypeError`	Pass `0` for automatic.
Negative counts are errors everywhere (before: "all cores"): `ValueError`, C++ `std::invalid_argument`, C ABI `MEF3IO_ERR_ARGUMENT`	Use `0`.
`MultiReader(n_threads=)` is a total shared by its recordings, not a count per recording	Raise it if you relied on `N` threads per recording.
`n_threads=1` makes the open serial too	To keep only decoding serial, open automatically and set `r.n_threads = 1` afterwards.
The default open starts at least 8 threads per open pass	An application opening many Readers from its own pool should pass a small `n_threads`.
C++: `Reader::read` / `read_raw` take `(channel, t0, t1, ctl)`; `Reader::kUseDefaultThreads` is gone; `cancel=`/`progress=` are keyword-only in the C++ binding	Update the calls; set the count with `set_threads`.

Writing and durability¶

What changed	What to do
`Writer(durability=)` / `MefWriter(durability=)` removed (`TypeError`); also C++ `SessionWriter::set_durable` / `durable()` / `SegmentSpec::durable`, C ABI `mef3io_writer_set_durable`, MEX `writer_set_durable`, MATLAB `Writer(Durability=)`	`durability="fast"`: remove it (now the default). `durability="full"`: `wait_for_disk=True`, or the default plus `w.flush()` at the points that must survive a power cut (cheaper).
`close()` and leaving a `with` block now flush, and raise if the flush fails	Nothing, unless you relied on `close()` ignoring a flush error. C++ destructor and `mef3io_writer_close` cannot report a failure: call `flush()` first.
A failed flush is remembered: every later `flush()`/`close()` raises again	Treat the first failure as final.
The default no longer flushes the two headers an append patches, so the `.tmet` can reach the disk before the data it describes	After an unclean shutdown, run `python -m mef3io recover SESSION.mefd` (dry run, lists what it would fix), then add `--apply`.
`recover --apply` no longer runs a session-wide repair, and `--no-repair` is gone	Run `repair` by explicit check id for findings on healthy segments.
`recover` (dry run and `--apply`) now also finds and re-derives stale declarations in each channel's last segment, and recovers an index that fails its own CRC; `RecoveredSegment.declarations` is new	Nothing to change; re-run the dry run on sessions you had marked healthy (it can now list segments 1.x called healthy).

Reading and conversion factors¶

What changed	What to do
`info(ch)["units_conversion_factor"]` is `None` when a channel's segments store different factors (C ABI and MATLAB: NaN; C++ `std::optional`)	Handle `None`; use `segments()` for each segment's factor.
`read_raw()`'s `units_conversion_factor` is `None` when the window mixes factors; `units_conversion_factors` gives the spans	Do not compute `samples * units_conversion_factor` on a mixed window: use the spans, or `read()`.
`MultiReader.read_raw()` no longer refuses recordings with different factors	Use the spans.
`Session.read_runs()` runs carry the factor and split where it changes; the compat `MefReader`'s `ufact` is `[None]` for such a channel	Update code that assumed one run per gap-free stretch.
A segment storing factor 0 ("not set") reads as the raw counts, with `UnscaledSamplesWarning` (it read as all zeros before)	Check your files (`r.segments(ch)`); if you know the real factor, scale `read_raw` counts yourself.
A NaN or infinite stored factor raises `FormatError` (C ABI `MEF3IO_ERR_FORMAT`) on `read()`/`block_envelope()`	Repair or rewrite the file.
A channel with different per-segment factors read back wrong before (first segment's factor for all)	Re-read; nothing to change in files.
`block_envelope()`: `min` and `max` are no longer swapped for a negative factor	None.
`MefReader.get_data(list)` uses the multi-channel read	None (same arrays).

Writing signals¶

What changed	What to do
Sampling frequency 0, negative, NaN or infinite, and `ufact` 0, NaN or infinite raise `ValueError` before anything is written (C++ `std::invalid_argument`, C ABI `MEF3IO_ERR_ARGUMENT`, MATLAB error). A negative factor is still accepted (polarity inversion)	Pass the real factor (or `1.0`) instead of `ufact=0`.
Block length is capped at 858 993 396 samples (`ValueError`); rates below 0.1 Hz write one sample per block	Pass a smaller `block_length`.
A `write_int32` `valid` mask of another length raises `ValueError`	Fix the mask.
Float samples outside int32 at the factor, and `+-inf`, are clamped to `±(2**31 - 1)` counts with one `ClampedSamplesWarning` (before: wrapped silently); the write summary gains `clamped` and `clamped_infinite`; inferred precision requires both ends of the data to fit (so one-sided large data can store a smaller precision, and a different conversion factor, than in 1.x) and ignores `+-inf` (it used to drag the precision to 0)	Replace such samples with `NaN` before writing if you want them as gaps. Pass `precision=` explicitly if you need the 1.x stored factor.
A default-precision append to a channel whose factor is not a power of ten is refused	Use `"step"` or `write_int32`.

De-identification¶

What changed	What to do
`anonymize_session(src, dst)` needs no shift: without one the copy keeps every date and time (`shift_us=None`; C++ `std::optional<si8>`; CLI works without `--shift-us`/`--reference-date`)	If the dates must not be shared, pass `shift_us` (or `--reference-date`/`--shift-us`) explicitly: a script that treated a missing shift as an error now makes a copy with the real dates.
`anonymize.check(src, dst)` without a shift checks the copy that keeps the times, so it no longer reports `rto-disputed`, `time-before-rto` and the other shift-only refusals; `verify` likewise	Pass the shift you will use to check or verify a shifted copy.
`anonymize.redact_text` takes no shift, so the shift-only refusals are gone from it	Use `anonymize.check(..., shift_us=...)` as the pre-check for a shifted copy.
`shift_us=0` is still refused (`shift-zero`)	Leave the shift out to keep the times.
C++: `check_anonymize` lost its `check_shift` parameter; `AnonymizeReport::shift_us` is optional	Update the calls.
New refusal codes `channel-name-identity` (a kept channel named after the patient, raised before copying; code that caught `verify-failed` for it now gets this, same `WriteConflictError`) and `time-range-offset-disputed`	`channel-name-identity` (`anonymize` only): leave that channel out with `exclude_channels`, or rename it. `time-range-offset-disputed` (in both `anonymize.REFUSAL_CODES` and `transform.REFUSAL_CODES`): copy without `time_range`.
Text callbacks (`text=callable`) get `info["time"] = None` for a record whose time cannot be known (no shift, segments disagree on the offset); `AnonymizeReport.source_recording_time_offset` is 0 when segments disagree	Handle `None` in callbacks that do arithmetic on `info["time"]`.
An unclosed `Writer` that fails to flush when garbage-collected emits `WriterFlushWarning`; the compat `MefWriter.mef_block_len` no longer keeps a refused value	Close writers (or use `with`).
Cuts (`time_range`) place each segment and annotation file by its own recording-time offset	None; a transform's cut over disputed offsets is now correct.

Files, cache and bindings¶

What changed	What to do
`cache="auto"` snapshots are named `<hash>.mef3cache`, not `<session name>-<hash>.mef3cache` (the name could identify the subject); the first auto-cache open after upgrading deletes the session's old-named snapshot and builds a new one	Nothing.
C ABI: `mef3io_writer_flush`, `mef3io_writer_set_wait_for_disk`, `mef3io_reader_set_threads` / `mef3io_reader_threads`, `mef3io_writer_threads`, `mef3io_reader_read_raw_factors`, `mef3io_reader_segment_units_conversion_factor`, `mef3io_check_block_length` / `mef3io_check_signal_parameters`, `MEF3IO_MAX_BLOCK_SAMPLES` added; `mef3io_writer_set_durable` removed; `mef3io_channel_info`'s factor is NaN for mixed factors; negative thread counts return `MEF3IO_ERR_ARGUMENT`	Rebuild against the new header.
MATLAB: `Writer(Durability=)` removed (now an error); `Writer(WaitForDisk=true)`, `w.flush()`, `setThreads` / `threads` added; `info` gives NaN for mixed factors; `readRaw` has `units_conversion_factors` (1-based, inclusive); `Writer.write` / `writeInt32` accept sparse input, and the MEX refuses sparse or complex array arguments elsewhere	Remove `Durability`; update `readRaw` users.
C++: `Session`'s constructors take an optional `n_threads` after `strict`; constructors taking a `shared_ptr<const SessionSource>` skip the `.mefd`/`.mefd.tar` naming rule (internal, for tests)	Update calls that passed positional arguments.

Details¶

Breaking: one thread count per object (migration)¶

The per-call n_threads argument of Reader.read and Reader.read_raw is removed (also on MultiReader), with no deprecation period. Set the object's count instead, which is now a settable property:

# before
x = r.read("ch1", t0, t1, n_threads=1)
# now
r.n_threads = 1          # from the next call on
x = r.read("ch1", t0, t1)

cancel= and progress= are keyword-only on the C++ binding's read/read_raw (the Python Reader already required that). - n_threads must be an int >= 0 (0 = automatic). A bool or float raises TypeError; for negative values see below. - MultiReader(n_threads=) is a total, shared by its recordings and never multiplied by them: the open splits it between the recordings it opens at once, a read between the recordings it touches, and a read inside one recording gets all of it. Before, every recording was opened with the full count, so a MultiReader of 10 recordings could run 10 times its budget. Setting MultiReader.n_threads re-splits from the next call. - n_threads=1 now makes the open serial too, not just decoding. If you passed 1 for comparisons with meflib or pymef, the open on a network share is now as slow as before this release's parallel open. Leave the open automatic and set r.n_threads = 1 after opening if you only want serial decoding. - The default open starts threads: at least 8 per open pass, also on small machines (file access waits on round trips, not on cores). An application that opens many Readers from its own thread pool multiplies that; pass a small n_threads there. If a thread cannot be started (a process or container thread limit), the work continues on the threads already running, and in the worst case on the calling thread alone. It never aborts. - Negative counts are errors everywhere: ValueError in Python, std::invalid_argument in C++ (Session, Reader, set_threads, SessionWriter::set_threads), MEF3IO_ERR_ARGUMENT from the C ABI (mef3io_reader_open/_open_ex, mef3io_reader_set_threads, mef3io_writer_set_threads). Before, a negative count meant "all cores". - C++: Reader::read / read_raw take (channel, t0, t1, ctl), and Reader::kUseDefaultThreads is gone. Session's constructors take an optional n_threads after strict. The constructors taking a shared_ptr<const SessionSource> are internal (for tests): they skip the .mefd/.mefd.tar naming rule.

See Threads for the rule.

Breaking: the `durability` option is replaced by flush() / wait_for_disk (migration)¶

Writer(durability=) and MefWriter(durability=) are gone, with no deprecation period: passing it raises TypeError (before anything is written, so overwrite=True never removes a session first). Also removed: C++ SessionWriter::set_durable / durable() and SegmentSpec::durable, the C ABI's mef3io_writer_set_durable, the MEX command writer_set_durable, and MATLAB's Writer(Durability=) (now an unknown-option error).
When is my data on disk? (new section in Long recordings):
write() returns once its data is handed to the operating system;
new Writer.flush() returns only when everything the Writer wrote so far is on disk, in ONE combined wait for all channels (C++ SessionWriter::flush, C ABI mef3io_writer_flush, MATLAB w.flush(), MefWriter.flush());
close() (and leaving a with block) now flushes too — it returns only when everything is on disk, and raises if the flush fails (the C++ destructor and mef3io_writer_close flush but cannot report a failure: call flush() first);
new opt-in Writer(wait_for_disk=True): every write() returns only when its own data is on disk (one combined wait per call; C++ set_wait_for_disk, C ABI mef3io_writer_set_wait_for_disk, MATLAB WaitForDisk=true, MefWriter(wait_for_disk=True)). Migrating from 1.x: durability="fast" → nothing (the default); durability="full" → wait_for_disk=True, or the default plus flush() at the points that must survive a power cut (cheaper).
What the default write still flushes: each append's .tmet before it replaces the old one, and the .tmet of a channel's first segment (or of a new segment or channel in a session that already held data) — a short .tmet would make the whole session fail to open. Nothing else (1.x's default "fast" still flushed the two headers an append patches, and with them the appended data and index; that is gone, so the .tmet can now reach the disk BEFORE the data it describes).
What a power cut can do before a flush: leave the last append of a segment in any mix of old and new bytes — the .tmet newer or older than the data and index, the .tidx header and entries from different writes, entries pointing at data that never landed, a torn .tdat tail, stale header counts, a new segment's .tidx/.tdat empty. recover_session repairs every one of them by itself (python -m mef3io recover SESSION --apply, mef3io.recoverSession(path, Apply=true) in MATLAB), leaving a session that validates clean and holds every sample whose bytes reached the disk; a recorder can then re-send what was lost. Earlier appends are never at risk: nothing rewrites them. The per-state table is in Long recordings.
A failed flush is remembered. After one failed flush, every later flush()/close() of that writer raises again (C++, C ABI, MATLAB, Python): the OS reports a write error once, so a retry that "succeeded" would claim data that is gone. An unclosed Python Writer that cannot flush when it is garbage-collected emits a WriterFlushWarning (the flush now runs with the GIL released).
wait_for_disk=True also makes the published .tmet durable before write() returns (one more combined wait after the rename), and flush()/close() cover every file and new folder a write touched in every mode. macOS flushes folders with F_FULLFSYNC; Windows flushes them through a directory handle where NTFS allows it (see long recordings).
Recovery: publishes a complete, CRC-valid .tmet.mef3io-tmp when the .tmet is lost (even a channel's first segment); rebuilds a MISSING .tidx like an empty one; replaces a zeroed .tdat header on a full-size segment instead of re-sealing it; and numbers a rebuilt index's samples channel-wide.
A write, flush() or close() fails if a flush it waits for fails. 1.x's default ignored flush errors; now the OS error is raised. The flush cost was measured on ext4 only; macOS (F_FULLFSYNC) and Windows (FlushFileBuffers) are unmeasured and typically slower per call.
Measured (ext4, 128 channels × 10-minute blocks at 256 Hz): a round of appends takes 2.2–2.3 s by default, 2.5–2.6 s with one flush() per round, 3.6–3.7 s with wait_for_disk=True — against 3.8 s with 1.x's default and 6.0–6.5 s with "full". A fresh session's first write: 2.4–2.5 s (0.85 s with no flush at all).

Recovery covers every crash state (behaviour change)¶

recover_session(apply=True) re-derives the declarations itself for every segment it changes and for every channel's last segment whose .tmet declares other block, sample or entry counts than its index (a .tmet published before or after its data). It goes through repair_session (the same repairs, backups under <session>.repair-backup/ and provenance). The dry run reports such a segment as needing recovery; 1.x called it healthy. New field RecoveredSegment.declarations lists the check ids.
An index that fails its own CRC is no longer refused: entries the stored CRC covers are kept, then each entry whose RED block header agrees, and the rest is rebuilt from the CRC-checked blocks in .tdat (streaming: one 304-byte header per unverified entry). Intact samples become readable again.
In each channel's last segment, the blocks of the last append (back to its first block, at most 64 MiB) are CRC-checked against their entries, so entries pointing at data that never landed are dropped.
A later segment whose .tmet is empty or short but whose data and index landed (a long-lived Writer's new segment cut off by a power cut, which failed the strict open of the whole session) gets its .tmet REBUILT from the previous segment's plus its own files, then its declarations re-derived. A channel's last segment whose files are missing, empty or short and whose data holds no block is REMOVED (moved to the backup). A channel's first segment with a lost .tmet is skipped with advice; its .tmet is now flushed when written (once per channel). A short or empty .tidx (or a zeroed .tdat header) is REBUILT from the CRC-checked blocks in .tdat, under a header taken from the segment's .tmet.
python -m mef3io recover --apply no longer runs a session-wide repair afterwards, and --no-repair is gone: recovery re-derives the declarations of exactly the segments it reports (the dry run lists the check ids), and a healthy segment's findings are left to repair, by explicit id. A healthy legacy session is left byte for byte.

New¶

One thread rule, everywhere. An object's n_threads is how many threads it may use for everything it does: opening, file access, decoding, decrypting, encoding. 0 = automatic, 1 = no extra threads. "Automatic" is decided in one place in the core: hardware concurrency for CPU work, at least 8 for file access. It is changeable at runtime as Reader.n_threads, MultiReader.n_threads and Writer.n_threads (Python), set_threads (C++, now atomic), mef3io_reader_set_threads / mef3io_reader_threads / mef3io_writer_threads (C ABI; mef3io_writer_set_threads existed), and setThreads / threads on the MATLAB Reader and Writer. A change applies from the next call; calls in flight keep their count.
Sharing a Reader is safe in every case now (#110). A Reader opened with cache= builds its backend once, whichever thread asks first (before, racing threads each built one). close() racing that first build stays closed (before, the session could silently reopen). A skipped record file or one opened through the old password mapping is warned about once per Reader, not once per concurrent call. C++ set_threads and set_index_cache_bytes are atomic, and the metadata loaded at open is read without a lock afterwards.

Reading many channels in one call (#90; part of #68)¶

Reader.read(channels, t0, t1) takes a list of channel names and returns a 2-D array, one row per name in the order given. Fixes #90. Every row is byte-identical to read() of that channel alone (same grid, NaN gaps, per-segment scaling, warnings, errors); the channels must share one sampling rate (ValueError otherwise: read each rate in its own call). Every channel's index is read at once on the Reader's threads, then the data a group of channels at a time with one decode pool per group: with 20 ms per file request (a slow share, simulated), 209 channels x 10 minutes read in 1.2 s on 16 threads instead of 13.5 s for a loop; locally 1.3-1.9x faster from 4 threads on, and 2.9x faster than a Python loop with np.stack on 16 threads (benchmarks). cancel= and progress= cover the whole call. Without t0/t1 the window is the union of the listed channels' spans. An error is the one a loop raises (the first failing channel in list order), and nothing is started for the channels after a channel known to fail.
dtype="float32" on read (one channel or many): the float64 value rounded to the nearest float32, exactly read(...).astype(numpy.float32), in half the memory.
block_envelope(channels) takes a list too and returns a dict keyed by channel name, each value the single-channel dict; the indexes are read concurrently, still index only. For #68 (a multi-channel form for viewers).
MultiReader has the same arguments and returns what a Reader of the merged session returns (it reads the listed channels one after the other, within its thread budget). The legacy MefReader.get_data(list) now uses the multi-channel read (one call per sampling rate and window), with the same arrays as before.
C++: Reader::read_many / read_many_f32 (returning mef3io::ChannelRows<T>: one row-major buffer), Reader::read_f32, Reader::block_envelope_many. read_raw stays one channel per call. Not in the C ABI or MATLAB yet (Python and C++ only).

De-identified copies keep the times unless you ask for a shift¶

anonymize_session no longer requires a shift. Without one (the new default) the copy keeps every date and time exactly, so it still shows when the recording was made; the patient's identity is removed exactly as before (names, ID, location, session name, descriptions, annotation text by default, new identifiers, other programs' private data cleared, provenance anonymize). Give a shift, as before, to hide the dates.

Python: mef3io.anonymize_session(src, dst, shift_us=None, ...), mef3io.anonymize.check(src, dst, shift_us=None, ...) and mef3io.anonymize.verify(src, dst, shift_us=None, ...). None (or leaving the argument out) = times kept; an int = every time moved by it, with all the rules that applied before. AnonymizeReport.shift_us is None for a copy that kept the times, and summary() says so.
Command line: python -m mef3io anonymize SRC DST now works without --shift-us / --reference-date and keeps the times.
C++: AnonymizeOptions::shift_us and AnonymizeReport::shift_us are std::optional<si8> (unset = times kept); verify_anonymized takes a std::optional<si8>; check_anonymize lost its check_shift parameter (the options' shift decides).
What a copy that keeps the times holds: every stored time byte for byte as in the source, and each segment's recording_time_offset (the stored times are read against it, so keeping it is what keeps the times). GMT_offset and the daylight-saving fields are still cleared: they can say where the recording was made, and no time a reader returns depends on them. Its verification requires every time (file headers, block index, data block headers, annotation headers and index, the times inside annotations) and every offset to equal the source's.
Only with a shift: the shift rules (shift-zero, shift-below-floor, shift-not-positive), the recording-time-offset refusals (rto-disputed, rto-unreadable, time-before-rto), id-number-time, time-overflow and the time leak scan. A copy that keeps the times copies each time and offset as stored, so a session with segments that disagree on the offset, or with times in the ambiguous legacy convention, can now be de-identified without a shift (its times read back exactly as the source's).
Cuts under a disputed offset (time_range, both anonymize_session and transform_session): each segment, and each annotation file, is now placed by the offset ITS file is stored against (a segment's own; for a channel or the session, the one all their segments agree on, as the Reader reads it). Before, a transform's cut read everything against one session-wide offset (0 when segments disagree), which could leave out a channel or an annotation inside the window while the copy still passed verification. A channel- or session-level annotation file whose time depends on an offset its segments disagree on cannot be placed and is refused with the new shared code time-range-offset-disputed (copy it without time_range).
Verification of cuts now checks every cut segment's header start and end against the blocks it holds itself (the validator's time checks stand down when a recording-time offset is set), and that every channel left out for "no samples in the window" really has none. A standalone verify(..., time_range=) notes a source channel with samples in the window that the copy does not hold (it may have been excluded on purpose).
Text callbacks (text=callable) get info["time"] = None for a record whose time cannot be known: without a shift, a time stored against an offset the file's segments disagree on. AnonymizeReport. source_recording_time_offset is 0 when the segments disagree (there is no single one), and a note says so.

Migration¶

Code that always passes a shift is unaffected: a shifted copy is made, checked and verified exactly as before.
Anything that relied on the shift being required (a script or form that treated a missing shift as an error) now makes a copy with the REAL dates. If the dates must not be shared, pass shift_us (or --reference-date / --shift-us) explicitly.
anonymize.check(src, dst) without a shift changed meaning. It used to check a shifted copy with the shift still to be chosen; it now checks the copy that keeps the times, so it no longer reports the shift-only refusals listed above (for example rto-disputed or time-before-rto). To check a shifted copy, pass the shift you will use.
anonymize.verify(src, dst) without a shift checks a copy that keeps the times; pass the shift to verify a shifted copy (it was required before).
shift_us=0 is still refused (shift-zero); leave the shift out to keep the times.
anonymize.redact_text takes no shift, so it now follows the copy that keeps the times: it no longer raises the shift-only refusals (rto-disputed, rto-unreadable, time-before-rto) for such sources. A caller that used it as a pre-check for a shifted copy should call anonymize.check(..., shift_us=...) instead.

A kept channel named after the patient is refused before copying¶

A kept channel whose name holds the patient's name or ID, the recording location or the original session name (in a form the copy's check refuses: "Roe" or "Jane_C3" for the patient Jane Roe, "Roe1", "JaneRoe") used to pass anonymize.check and was refused only by the copy's verification, after every block had been copied (verify-failed). It is now refused before anything is written, by check, anonymize_session and redact_text alike, with the new code channel-name-identity (WriteConflictError, details["channel"] = the channel). The pre-flight applies the verification's own channel-name rule to each kept channel's name and to the folder and file names the copy makes from it (#143). Overlaps that the rule only notes ("12" in "LAH12", "P3", a session called "EEG" vs "EEG1") still pass. Leave such a channel out with exclude_channels. Transforms are unaffected.

Segment numbers are no longer read as the subject ID. The copy's check matched the number in a segment folder's name (Fz-000001.segd) against the ID's digits, so a subject PT0001 with more than one segment per channel could not be anonymized at all (verify-failed). The number is mef3io's own counter and is checked on its own; only the channel part of those names is searched now, by the check and by the verification.
Migration: code that caught verify-failed for this case now gets channel-name-identity, raised as the same WriteConflictError, before the copy starts.

Faster¶

Opening a session is parallel (#141). The channel listings, the segment file checks and every metadata file's read and parse run on the Reader's threads. With 20 ms per request (a slow network share, simulated), 209 channels x 10 segments opens in 10.9 s on 16 threads instead of 173 s on one, and 64 x 1 in 0.46 s instead of 6.5 s (benchmarks). The result does not depend on the thread count. The channels, metadata, the lenient problems() (in the same order) and the strict error (the first in listing order, as a serial open meets it) are all the same.
Repeated windowed reads no longer walk every block index (#110). For each index it holds in memory, a Reader remembers the time span the index covers, and skips an index that misses the window. The span is forgotten with its index, so the result is always the same as walking it. Repeated one-second reads of a 2,400-segment channel went from 4.7 ms to 0.2-0.28 ms each; the first read still walks every index once.
Fixed: block_envelope could leave out blocks of a segment whose metadata states an end time earlier than its data, or a start later than it. This happens with files from writers that store a shorter end, and after an interrupted append that was followed by a new segment. It skipped segments by those times; no metadata time is used to skip any more.
Opening a session reads each segment's metadata file with one bounded read of its fixed record size. There is no longer a separate seek to find the file's size, and padding that other writers append after the record is not read. Locally this removes three system calls per segment. On a network file system that re-checks file attributes, it saves one request per segment: about 6% on a simulated slow mount. Results and messages are unchanged. Part of #141.
Opening a .mefd.tar archive no longer scans the whole archive for every folder it lists (the open lists the root and each channel folder once). Folder listings are now indexed once when the archive is opened.
C++: Session and Reader can be opened from a SessionSource, and SessionSource::read_head(path, n) reads the first n bytes of a file in one request.

Changed — check before upgrading (2.0.0)¶

Conversion factors are per segment (#150). MEF stores the units-conversion factor in each segment's section 2, and segments of one channel may store different ones (a new_segment=True write with another precision, or a foreign recorder). The API now says so:
Reader.info(ch)["units_conversion_factor"] (and MultiReader.info) is None when the channel's segments store different factors (meflib's channel aggregate reads NO_ENTRY there). C++ ChannelInfo:: units_conversion_factor is a std::optional; the C ABI's mef3io_channel_info and MATLAB's info give NaN.
Reader.segments(ch) has each segment's units_conversion_factor (C ABI mef3io_reader_segment_units_conversion_factor, MATLAB segments); inspect()'s segment summaries too, and ChannelSummary. units_conversion_factor is None when they differ.
read_raw() gains units_conversion_factors: which factor applies to which samples, as start_index/end_index/units_conversion_factor spans. Its units_conversion_factor is None when the window holds samples of segments with different factors (C++ RawData:: units_conversion_factor is a std::optional, RawData::factor_spans the spans; C ABI mef3io_reader_read_raw_factors; MATLAB readRaw's units_conversion_factors, 1-based and inclusive). Code that computed samples * units_conversion_factor must use the spans, or read().
MultiReader.read_raw() no longer refuses a window over recordings with different factors: it returns the same spans a Reader of the merged session does.
Session.read_runs() runs carry units_conversion_factor and are split where the factor changes.
The compat MefReader's ufact is [None] for such a channel, and get_data scales each sample by its own segment's factor.
A segment whose conversion factor is not set reads as raw counts, with a warning; NaN or infinite factors are refused. 0 is MEF's "no entry" for the factor ("not set"). read(), block_envelope(), MultiReader and the compat MefReader.get_data now return such a segment's samples as the raw stored counts (factor 1) and emit one new mef3io.UnscaledSamplesWarning per Reader and channel the first time such a segment is read. Before, they read back as all zeros. In a channel that mixes such a segment with real factors, every segment is scaled by its own (1 for the unset one). info(), segments() and read_raw()'s units_conversion_factors keep reporting the stored value (0). A factor that is NaN or infinite is invalid: read() and block_envelope() raise mef3io.FormatError (C ABI MEF3IO_ERR_FORMAT) for a window holding such samples, before anything is decoded. read_raw() and toc() still return the counts, and a lenient Reader does not count it as a damaged segment. The C ABI and MATLAB read unset factors as raw counts too, without a warning (C++: Session::unscaled_channels()).

Such files exist — check yours (migration). 0 is what meflib's and pymef's section-2 templates hold until a writer sets the factor (meflib.c:4564; pymef's MefSession._initialize_tmd2). So a session written through pymef or by a meflib-based recorder that never set it, or by mef3io before 2.0 with write_int32(ufact=0), stores 0. mef_tools read those as all zeros (counts * 0), and so did mef3io before 2.0; now they read as the raw counts, with the warning. mef_tools and mef3io's own writers never store 0 by default (mef_tools writes 10**-precision, 1.0 for precision 0). If you know the real factor, scale the counts yourself:

raw = r.read_raw("ch1", t0, t1)
x = np.where(raw["valid"] == 1, raw["samples"] * known_factor, np.nan)

r.segments("ch1") shows which segments store 0 (units_conversion_factor).

Invalid sampling frequencies and conversion factors are refused (#83). Every write path (Writer.write, write_int32, appends, the C ABI and MATLAB) now raises ValueError (C++ std::invalid_argument, C ABI MEF3IO_ERR_ARGUMENT, a MATLAB error) before anything is written when the sampling frequency is 0, negative, NaN or infinite, or the conversion factor (ufact) is 0, NaN or infinite. Before, a rate of 0 or NaN made the write never return, and ufact=0 was stored, after which every sample read back as 0: the signal could not be recovered. 0 is also meflib's "no entry" value for the conversion factor (meflib.h:439). A negative conversion factor is still accepted: it is a polarity inversion, and pymef handles it explicitly (pymef3_file.c:972-978); mef3io, pymef and mef_tools read it back identically (test-pinned). Code that wrote ufact=0 as a placeholder must pass the real factor (or 1.0).
Block length is capped at 858 993 396 samples (a block's byte size is a 32-bit field; the worst case is 5 bytes per sample). Writer(block_length=) above it, or a rate whose default block would be above it (for example 1e12 Hz without block_length), raises ValueError. Rates below 0.1 Hz, whose default block is less than one sample, now write one sample per block instead of never returning. The MATLAB Writer checks BlockLength before it opens (with Overwrite=true, opening removes the existing session); C callers can do the same with the new mef3io_check_block_length / mef3io_check_signal_parameters (and MEF3IO_MAX_BLOCK_SAMPLES). The compat MefWriter.mef_block_len no longer keeps a refused value.
A write_int32 valid mask whose length differs from the data raises ValueError (it was read past its end).
Auto-cache file names no longer contain the session name (#82). The cache="auto" snapshot is now <hash>.mef3cache in the OS cache folder, not <session name>-<hash>.mef3cache: session folders are often named after the subject. The first cache="auto" open of a session after upgrading deletes that session's old-named snapshot (exactly that file, and only if it is a mef3io snapshot) and builds a new one.

Float writes never wrap a count beyond int32: out-of-range samples are clamped¶

Before, a float write whose value needed a count outside int32 at the conversion factor stored a wrapped, wrong value silently (an append of 3.0e6 to a channel at factor 0.001 read back as -1294967.296), and an infinite value wrapped the same way.

Now every float write (Writer.write, MefWriter.write_data, C++ write_float), with the default precision or precision="step", stores such a sample, and +inf/-inf, as the nearest storable value: ±(2**31 - 1) counts, meflib's own RED_POSITIVE_INFINITY / RED_NEGATIVE_INFINITY, which is also what meflib's encoder clamps to. The write succeeds; one out-of-range artifact (say a saturation spike in an append) no longer costs the chunk. The write summary gains clamped and clamped_infinite (C++ WriteSummary::clamped / clamped_infinite), and a write that clamped anything emits ONE ClampedSamplesWarning (a MefWarning; counts only in its text, details per channel). NaN is unchanged: a gap, never clamped. If you prefer to mark such samples as missing, replace them with NaN before writing.

The inferred precision now requires both ends of the data to fit (it used to accept data where only one end overflowed, as mef_tools did), so a default write of one-sided large data picks a smaller precision instead of needing a clamp, and +-inf no longer drags the inferred precision to 0 (it is ignored there, like NaN). precision="step" ignores +-inf when estimating the step too.

Smaller files for data from an amplifier: `precision="step"` (#81)¶

Writer.write(..., precision="step") (also MefWriter.write_data) finds the data's own quantization step, such as an amplifier's 0.195 µV per count, and stores the original counts with that step as the conversion factor. One hour at 512 Hz of 16-bit counts × 0.195: 1.52 MB instead of 6.03 MB with the default path, the same as write_int32 of the counts. Opt-in; the default is unchanged.

The step is estimated from a fixed-size sample (about 10 ms on 50 million samples) and every value is checked during the conversion the write makes anyway: a value stored on its step reads back exactly when the data was counts * step, and within float64 rounding otherwise. The whole write was faster than the default one in our measurements (10 million samples: 0.36 s against 0.57 s).
Data not on one step (filtered, resampled, noise), with too few distinct values to establish one (a DC level, a square wave), or whose counts would not fit int32, is written with the default path, with a new StepFallbackWarning.
A channel keeps one conversion factor: appends and new segments with "step" store data on the channel's factor exactly, and refuse (WriteConflictError, nothing written) data off it rather than round it. A default-precision append to a channel whose factor is not a power of ten is refused with a message saying to use "step" or write_int32.
The write summary has three new keys for every write: units_conversion_factor, quantization ("step", "decimal" or "int32") and step_fallback. C++: SessionWriter::write_float takes a Precision (an int still works; Precision::detect_step() for the step), WriteSummary gains the same fields, plus detect_step / quantize_to_step. Not in the C ABI or MATLAB.

Fixed¶

A channel whose segments store different conversion factors read back wrong (#150). The reader scaled the whole channel by its FIRST segment's factor, so after write(..., precision=2) and write(..., precision=4, new_segment=True) the second segment read 999.0 where 9.99 was written, without a warning. read(), block_envelope() and every value derived from them now scale each block by its own segment's factor (negative factors included); pymef agrees (test-pinned on a session with 0.01, 0.195 and -0.5 in one channel). Copies (transform_session cuts, anonymize_session, merge_sessions) already kept each segment's factor; a cut segment with a negative factor now declares its section-2 native maximum and minimum the right way round.
Appending to a segment folder whose number is not zero-padded (ch1-5.segd, as some writers name them) now extends that folder; it used to look for ch1-000005.segd and refuse. New segments keep the <channel>-NNNNNN form.
Recovery at a non-zero recording-time offset now sets the end time. At a non-zero offset (meflib's date-masking) repair_session leaves times alone, so after recover_session info() still reported the end from before the recovery. Recovery now corrects it from the last block in the three universal headers: only an end that lies before the data's last block (or before rebuilt blocks), never moving it earlier, each header in its own stored form; it notes what it leaves alone. A healthy session is untouched.
Recovery keeps the .tdat checksum when it fixes only headers. It used to reset it to "not computed" whenever it rewrote the .tdat header.
block_envelope() returned min above max for a channel with a negative conversion factor; the pair is now swapped after scaling, as the section-2 native extremes already were.
MATLAB: Writer.write / writeInt32 with a sparse array (data or Valid mask) crashed MATLAB or stored the wrong validity; they now take sparse input, and the MEX refuses sparse or complex array arguments with a clear error (sparse scalars, such as an element of a sparse matrix, are accepted). writeAnnotations accepts string-class ("...") type, text and channel.