Skip to content

mef3io 2.2.0

Released 2026-11-04

This release makes merging long recordings a one-call job that is much easier on network drives, adds a way to leave annotations out of any copy, and fixes two defects: a merge that could fail its own check in the released Linux wheels, and folder names with non-Latin letters on Windows. Output files are the same as before, apart from the random file IDs.

Highlights

  • One-call merge pipeline. merge_sessions can trim each recording to its own time range (time_ranges=[(start, end), None, ...], one entry per source), make the result anonymous (anonymize=True or the options of anonymize_session: shift_us, pseudonym, clear, text), leave channels and annotations out, and write one new session, all in a single pass over the data. Overlap is judged on the trimmed data. With anonymize, the identity of every source is removed and searched for in the result, patient fields that are cleared may differ between the sources, and one shift applies to all of them. The report says which pseudonym and shift were used. merge.check and merge.verify take the same arguments. See Merging recordings.
  • records="drop" for anonymize_session, transform_session and merge_sessions leaves every annotation out of the copy (session, channel and segment level; command line: --drop-records). The copy then has no annotation files, like a recording that never had any; signal data, times, channels and details are as without it. The default, records="keep", behaves as before. The reports gain records_dropped, and check/verify take the same argument: a verification of a dropped copy expects no annotations and fails if it finds any. A channel that held only annotations is left out of such a copy (it would be an empty folder). See Modified copies.
  • Faster merges with n_threads. Sources are opened and checked at once, channels are copied at the same time, and the final verification is parallel too. The output is the same for any thread count. merge_sessions is a single call with no object, so it takes n_threads= on the call (0 = automatic, 1 = no extra thread); this is the one place where a thread count is a per-call argument, see Threads.
  • Copies open each file far fewer times (faster on network drives). merge_sessions, anonymize_session and transform_session used to open every source file about ten times (the check before copying, the copy, the validator and the verification each opened it again) and every file they wrote about six times. Now each source file is opened once for the whole operation, and each written file once to write, flush and close it, plus once to verify it. In a test merge of 4 sources with 576 data files on each side the opens fell from 5,824 to 576 (sources) and from 3,649 to 1,152 (result), and the "does it exist / how big is it" checks from 3,712 to 576 and 3,264 to 576. Nothing is checked less: the verification still reads the sources again and still reads the finished copy back from disk. A read error on a reused network handle is retried once on a fresh open. archive_session makes one check per file instead of two and writes in larger pieces. benchmarks/fs_calls_benchmark.py counts the calls (needs strace); see Benchmarks.

More detail on merges

  • A merge also reports the channels it left out for lack of samples in their range (channels_without_samples), the edge blocks it trimmed and the annotations it left out for lying outside the ranges.
  • A merge now checks overlap on the blocks it actually keeps, including those that extend past a range end.
  • An anonymized merge names no source time in any message. Its report keeps the source paths in memory only (empty sources and source N in notes in to_dict(), pickles and summary(); source_count says how many).
  • New refusals: merge-time-ranges-count, merge-anonymize-metadata, merge-threads-invalid, and (every copy) records-invalid. The single time_range is still refused by a merge (merge-time-range).
  • The sources must not be changed while a copy runs. On Windows they stay open meanwhile, so a source folder cannot be renamed or deleted until it ends. This only concerns the copy itself: other readers and writers of the same files are unaffected and always see current data.

Fixed

  • Joining a recording cut from another one, with notes, could fail its own check. In the released Linux wheels, merge_sessions of a copy made by transform_session (for example a trimmed, password-protected part) with another recording failed with "universal header provenance_UUID does not name the source file" for the session notes (.rdat / .ridx) when the recording carried session- or channel-level notes. Nothing was published (the check refused it), so no data was damaged; a build from source did not show it. The merged note files now always name the file they came from, as every other copied file does. The release workflow now also runs the merge and copy tests against the built wheels.
  • Folder and session names with non-Latin letters work on Windows. A recording whose folder or file name holds characters outside the Windows "ANSI" code page (for example Japanese, or accents on a system set to another language) could fail to open or be written by the Writer, the Reader, appending, annotations, archives, extraction, the validator and repair, recovery, renaming, and the copies (anonymize, transform, merge). Every file call now uses the full Unicode name. Linux and macOS behave exactly as before, and the output files are identical.

Upgrading from 2.1.1

Python code needs no change; nothing that worked is removed.

  • Merge pipeline and n_threads. merge_sessions, merge.check and merge.verify have the new keyword arguments records, time_ranges, anonymize and n_threads; all default to the old behaviour. A merge now refuses a negative n_threads, a time_ranges list that is not one entry per source, and anonymize combined with metadata.
  • records=. The new argument defaults to "keep". Any other value than "keep" or "drop" is refused with records-invalid.
  • Reports. MergeReport, TransformReport and AnonymizeReport gain fields with defaults (records_dropped; MergeReport also channels_without_samples, blocks_trimmed, records_outside_window, source_count, pseudonym, shift_us and the other anonymize fields); reports saved by 2.1.1 still load. Code that reads MergeReport.sources of an anonymized merge now finds it empty; use source_count.
  • Verification. verify of a copy made with records="drop" must be given the same records="drop", otherwise it reports the missing annotations. A merge judges overlap on the data it keeps, so with time_ranges two recordings that overlap only outside their ranges merge.
  • C++. The merge declarations (MergeOptions, MergeReport, merge_sessions, check_merge, verify_merge) moved from mef3io/transform.hpp to the new mef3io/merge.hpp. transform.hpp and anonymize.hpp include it, so existing code compiles unchanged. New members, all defaulted: CopyOptions::drop_records, VerifyOptions::drop_records, CopyReport::records_dropped, MergeOptions::time_ranges, MergeOptions::anonymize, MergeOptions::n_threads, and the anonymize fields of MergeReport.
  • Windows. Paths are UTF-8 everywhere: pass non-Latin names as ordinary Python strings (C++: UTF-8 std::string); no workaround is needed any more.
  • Linux wheels. If you merged cut or password-protected parts that carry notes with 2.1.1 wheels and got the provenance error above, run the merge again with 2.2.0.
  • MATLAB is unchanged.