Skip to content

mef3io — Design & Implementation Plan

Status: agreed design, 2026-07-02 Supersedes: requirements/requirements.md (pure-Python plan, deleted)

This is the original plan

It records the design as agreed before the code was written, and is kept for its reasoning. Section 4.6 (threading) has been updated to the code as built; the rest is the plan. Where the plan differs from what shipped, the Python, C++ and MATLAB guides describe the library as it is. The main differences:

  • the pure-Python backend is a stub; Writer.write takes float data and write_int32 the counts; records are written with write_annotations;
  • the core is C++20; its AES code is mef3io's own (no vendored library); block indices are read into memory and cached under a size budget with least-recently-used eviction, not memory-mapped;
  • the MATLAB binding and the C ABI exist (they were "later" here);
  • one Reader can be shared between threads: its mutable state is locked, not immutable after open (see 4.6);
  • the warm-start cache lives in Python (mef3io.cache), and the module names in 4.1 are not the ones the code uses;
  • features added later are not planned here: inspect, cancelling and progress, the validator with repair and recovery, single-file .mefd.tar sessions, de-identified copies (anonymize_session, see De-identification), modified copies, cut and merge (Modified copies, Merging), and session renaming.

1. Goal

A single C++ core implementing MEF 3.0 read and write, wrapped thinly for Python (now) and MATLAB (later), with the high-level semantics of today's mef_tools (MefReader/MefWriter) living in the C++ layer so every language binding behaves identically. A pure-Python backend with the same API ships in the package as fallback, benchmark reference, and test oracle.

Non-goals: MEF video files (.vidd/.vmet/.vidx) are dropped from the library entirely — never used in practice; could be reconsidered in the future if worthwhile. The only footprint in the code: session traversal silently ignores .vidd directories (with a debug-level note), and video structs/APIs from meflib/pymef are simply not mirrored. Also out of scope (v1): lossy RED encoding (decode is supported), performance parity guarantees against original meflib (we expect to beat it on open time; decode should be comparable).

2. Architecture

┌─ python: mef3io pkg (nanobind) ─┐  ┌─ MATLAB toolbox (MEX, later) ─┐  ┌─ viewer (later)
│  numpy in/out, GIL released     │  │                               │  │
└──────────────┬──────────────────┘  └──────────────┬────────────────┘  │
               │                 stable C ABI shim (later, for MATLAB)  │
┌──────────────┴─────────────────────────────────────┴──────────────────┴───┐
│  HIGH-LEVEL C++ API — mef3io::Reader / mef3io::Writer                     │
│  float64 + NaN-gap semantics, ufact scaling, precision inference,         │
│  int32+ufact primitive write path, append logic, annotations, cache       │
├────────────────────────────────────────────────────────────────────────────┤
│  CORE C++ (format codec, no policy)                                       │
│  fmt structs + CRC · AES + password scheme · RED codec ·                  │
│  mmap IO + dir traversal · Session/Channel/Segment · records ·            │
│  RAII, exceptions, zero globals, thread pool                              │
└────────────────────────────────────────────────────────────────────────────┘

Key decisions (agreed):

  1. High-level ("smart") semantics live in C++, wrappers are thin (<~500 lines).
  2. On-disk format stays byte-identical to meflib output. Code is written from the spec (field tables), not ported from meflib. meflib/pymef, mef3_dump, and mef_tools/reimplementation.py serve as three independent oracles.
  3. Dropped from meflib: global state, exit-on-error, FILT_* filters, e_* allocator wrappers, lossy RED encode, video.
  4. Added: thread safety (planned as immutable-after-open readers; built as locked shared state, see 4.6), context-managed resources, exception-based errors, mmap'd binary-searchable indices, lazy metadata, metadata cache, internal parallelism.

3. Repo layout (monorepo, this repository)

mef_tools/                        (repo root — name unchanged for now)
├── core/                         C++17 library "mef3io_core"
│   ├── include/mef3io/           public headers
│   ├── src/
│   ├── third_party/              tiny-AES-c (vendored, public domain)
│   └── tests/                    Catch2 unit tests
├── bindings/python/              nanobind extension "_mef3io"
├── python/mef3io/                Python package
│   ├── __init__.py               Reader / Writer (C++ backend)
│   ├── pure/                     pure-Python backend (same API)
│   │                             (from reimplementation.py + mef3_dump decode)
│   ├── compat.py                 drop-in MefReader/MefWriter (mef_tools.io API)
│   └── cache.py                  cache location/policy helpers
├── mef_tools/                    legacy package — unchanged, used as oracle
├── tests/                        cross-implementation compatibility matrix
├── benchmarks/                   C++ vs pure vs legacy pymef
└── reference_files/              meflib, pymef, mef3_dump (read-only oracles)

4. C++ core design

4.1 Modules (namespaces)

Namespace Contents
mef3io::fmt POD-ish structs with exact-layout (de)serialization: UniversalHeader (1024 B), MetadataSection1 (1536 B), TimeSeriesMetadataSection2 (10752 B), MetadataSection3 (3072 B), TimeSeriesIndex (56 B), RedBlockHeader (304 B), record header/index + all record types from mefrec.h (Note, EDFA, Seiz+SeizCh, SyLg, LNTP, CSti, ESti, Curs, Epoc)
mef3io::crc CRC-32 Koopman (table-driven), header/body CRC helpers
mef3io::crypto AES-128-ECB (vendored tiny-AES-c, validated against NIST vectors and cryptography in tests), terminal-byte password extraction, L1/L2 validation-field scheme, key derivation
mef3io::red RED decode (range decoder + diff reconstruction + unscale/retrend, incl. lossy-written blocks) and lossless RED encode (diff + range coder, keysamples, per-block stats)
mef3io::io mmap file abstraction (POSIX/Win32), .mefd/.timd/.segd discovery, path/name rules
mef3io::(Session,Channel,Segment) lazy object tree over the directory structure
top level Reader, Writer, Cache, ThreadPool, error hierarchy

4.2 Serialization approach

No reinterpret_cast of packed structs (alignment/UB trap that meflib dances around); each struct has explicit parse(span<const byte>) / serialize(span<byte>) against a field table with static offsets — the same tables already validated in reimplementation.py::FIELD_DEFINITIONS and the deleted requirements doc. static_assert on all offsets/sizes.

4.3 Error handling

Exception hierarchy: MefError ← {FormatError, CrcError, PasswordError, IoError, WriteConflictError}. No error-code returns in the public API; the C ABI shim (later) converts to codes. Bindings map to Python exceptions.

4.4 Reading — designed around partial access

  • Reader::open() reads only: directory tree + the .tmet files needed for channel list, fs, ufact, start/end, nsamp (last-segment shortcut where possible). No .tidx is touched at open.
  • .tidx files are mmap'd, never materialized: entries are fixed 56 B and time/sample ordered → binary search directly on raw bytes for [t0, t1] / [s0, s1] block ranges. O(log n) seek, zero parse cost.
  • read_raw(channel, t0, t1) → preallocated int32 buffer sized round((t1-t0)*fs/1e6); blocks decode in parallel into disjoint slices; gap samples flagged (output carries a validity mask + ufact + actual start).
  • read(channel, t0, t1) → float64 = int32 · ufact, gaps = NaN.
  • toc(channel) → block-level table (start_time, start_sample, n_samples, offset) for viewers/dataloaders.
  • Discontinuity gap length rounds to nearest sample (sub-sample gaps tolerated; never accumulate rounded periods — always t = start_uutc + round(n * 1e6 / fs) with fs as double).
  • recording_time_offset (superseded: the implemented rule is in mef3_format.md, and using the offset as anonymization is deprecated, see anonymization.md): apply per spec (stored = actual − offset). The "corrupted" convention observed in the wild (stored = offset − actual, handled heuristically in mef3_dump) is behind an opt-in leniency flag, off by default, warning when triggered.

4.5 Writing

  • Primitive path: write(channel, span<int32>, ufact, start_uutc, fs) — amplifier counts + V/bit conversion factor stored as units_conversion_factor. No transformation of samples.
  • Convenience path: write(channel, span<double>, start_uutc, fs, precision?) — quantizes to int32 with 10^precision (inferred if absent, same algorithm as mef_tools incl. int32 dynamic-range guard).
  • NaN policy (float path; NaN = discontinuity, blocks are tiled within the segment — NaNs never create new .segd segments, matching current behavior):
  • Default (new API): strict split — every NaN run becomes a true on-disk gap (discontinuous blocks, nothing stored for the gap). No sentinels, no compression penalty.
  • max_nans_allowed=<n> opt-in: NaN runs shorter than n are embedded as the RED_NAN sentinel (INT32_MIN, reads back as NaN) to reduce block count. Today's mef_tools does this implicitly (default 'fs' = 1 s) via the NaN→int32 cast coincidentally hitting RED_NAN; here it is explicit. The compat shim keeps the legacy 'fs' default.
  • All-NaN input: no-op on disk, but explicit — every write returns a summary (samples_written, blocks, gaps_skipped) and warns when samples_written == 0 (today this is silent).
  • int32 primitive path has no NaN: gaps are structural (separate write calls, each an implicit discontinuity, or an optional validity mask). INT32_MIN in int32 input is rejected by default (collides with RED_NAN); opt-in flag to treat it as NaN sentinel.
  • Read side: read() (float64) → NaN for both true gaps and stored RED_NAN; read_raw() (int32) → validity mask distinguishes gap / sentinel / data.
  • Segment control: new_segment flag; append goes to last segment with end-time/fs/ufact consistency checks (same guards as mef_tools).
  • Records: write all supported types at session or channel level; merge with existing records on append (as mef_tools does): write_records always appends in place (record data, index entries, then the headers; body CRCs extended), creating the files when absent; readers sort by time (#156).
  • Encryption on write: none, or fully encrypted (section 2 → L1 key, section 3 → L2 key) — a fixed pairing driven by password presence, not three independent modes. "Level-1 only" is not a valid MEF file; see the canonical explanation in docs/encryption_model.md. Includes section encryption flags, validation fields, and encrypted record bodies.
  • Float fs fully supported (sf8 end-to-end); block length heuristic from fs kept but overridable (mef_block_len).
  • Section-2 buffer sizing is a hard output requirement, not bookkeeping. meflib-based readers allocate from the maximum_* fields before decoding, and 0 is not the NO_ENTRY sentinel for any of them, so every one is measured from the blocks actually written: maximum_difference_bytes from the encoded RED headers, the maximum_contiguous_* trio from runs delimited by the same .tidx discontinuity flag a reader uses. Appends recompute the contiguous trio from the full index (repairing segments written before this was implemented). maximum_difference_bytes stays exact across a chunked write, because the writer keeps the running maximum for the blocks it encoded itself; for a segment reopened from disk, whose earlier blocks it cannot see without a seek each, it takes meflib's RED_MAX_DIFFERENCE_BYTES as a floor instead — an over-declaration only costs a reader memory, while under-declaring truncates the buffer it decodes into. The read path never consults these fields — it sizes from each block's own header — so older or foreign declarations stay readable. See docs/mef3_format.md and docs/legacy_comparison.md.
  • When data is on disk. write() returns once the data is handed to the operating system; only a .tmet is flushed before it is renamed into place (every append, and a channel's first segment), since a short .tmet makes the whole session fail to open. flush() and close() wait for everything written so far in one combined wait (one syncfs on Linux); the opt-in wait_for_disk=True makes each write wait for its own files. A power cut before a flush can tear the last append of a segment; recover_session(apply=True) restores a valid session (index and data itself, declarations through repair_session). See docs/long_recordings.md, "When is my data on disk?".
  • Writes never touch caches; a cached open detects the change (see §6).

4.6 Threading

  • ONE thread rule (Threads): an object's n_threads is how many threads it may use for everything it does; 0 = automatic, decided only by resolve_threads (parallel.hpp: hardware concurrency for CPU work, at least 8 for file access), 1 = no extra threads. Settable at runtime (atomic; the next call uses it). No per-call override (removed in 2.0); a MultiReader splits its total privately (Reader::read_raw_split).
  • No persistent thread pool: each parallel step calls parallel_for.
  • Parallel units: RED block decode and encode — independent blocks, disjoint output slices ⇒ deterministic, race-free — and the open's metadata pass (channel listings, segment file checks, every .tmet read + parse), run with parallel_try_each and assembled in order, so the result, the strict error and the lenient problems() are those of a serial open (the first failure in listing order, never the first in time).
  • Concurrent calls on one Reader from user threads are safe. The metadata parsed at open is immutable afterwards and read without a lock. The Reader is otherwise NOT immutable after open: it caches block indices (with LRU eviction) and, in lenient mode, records problems as it meets them. That state is locked, and each call pins the index it is reading, so another thread's eviction never frees it (core/tests/test_concurrency.cpp, also under TSan). The collect phase of a read takes these locks only around cache access, never across file I/O or decoding.
  • Python bindings release the GIL for every potentially long call: opening (the Reader, Session and SessionWriter constructors, #79), reads, writes, toc, segments, records, write_records, and the session-level functions. Cheap accessors keep it.
  • A session has one writer at a time. That is documented, not enforced: there is no lock file. Within one write call the writer is internally parallel: block boundaries are computed up front (from mef_block_len and the NaN/discontinuity split), all blocks of the call are RED-encoded in parallel into their own buffers, and the .tdat body and .tidx entries are then assembled and written in order on the calling thread. Output is byte-identical regardless of n_threads. Memory scales with one call's encoded data, so very long signals are written in chunks.

5. Python package

import mef3io

with mef3io.Reader("/data/session.mefd", password="...", n_threads=8) as r:
    r.channels                       # list[str]
    r.info("ch1")                    # dict: fs, ufact, start, end, nsamp, units...
    x  = r.read("ch1", t0, t1)       # float64, NaN gaps
    xi = r.read_raw("ch1", t0, t1)   # int32 + mask + ufact
    r.toc("ch1")                     # block table (numpy structured / DataFrame)
    r.records()                      # annotations

with mef3io.Writer("/data/session.mefd", password1=..., password2=...) as w:
    w.write("ch1", counts_int32, ufact=2.5e-7, start=t0, fs=1024.5)
    w.write("ch2", volts_float64, start=t0, fs=256.0, precision=3)
    w.write_records(df)
  • mef3io.pure — same Reader/Writer API, pure Python+numpy(+numba optional). Backend selectable: mef3io.Reader(..., backend="pure"). v1: pure read path complete (exists already across reimplementation.py + mef3_dump); pure write path is a stretch goal after C++ write lands.
  • mef3io.compat — MefReader/MefWriter matching mef_tools.io signatures, defaults, and quirks (e.g. get_data list/str return-shape asymmetry, max_nans_written='fs' default) so it's a drop-in import change; the legacy mef_tools package itself stays untouched in the repo as oracle.

6. Metadata cache ("warm start")

Purpose: collapse the many-small-file cost of session open (mainly on network storage; lazy open + mmap indices already fix the pathological local case).

  • Modes:
  • cache="auto" (default): per-user OS cache dir (platformdirs-style: ~/Library/Caches/mef3io, $XDG_CACHE_HOME, %LOCALAPPDATA%), file <hash16>.mef3cache, named by a hash of the absolute session path ONLY (since 2.0.0; earlier versions prefixed the session folder's stem, which put a name that is often the subject's in a file name; the next cache="auto" open of that session deletes its old-named file, exactly that name and only if it is a mef3io snapshot, and builds a new one). Works with read-only sessions.
  • cache=<path> / explicit persistent: user-requested dump file (e.g. next to the .mefd). Never discovered or used unless explicitly passed — a foreign/stale cache file inside a session dir cannot surprise a default-configured reader.
  • cache=None: off.
  • Contents (original plan, not what was built): per-file fingerprints and the raw, still-encrypted .tmet bytes. The built cache stores decoded channel infos instead, which is why it refuses encrypted sessions outright.
  • As built (python/mef3io/cache.py, cache format 6, issue #64): the snapshot holds the channel infos, the open-time warnings, the exact mef3io version that wrote it, and an O(1)-per-channel signature — no per-file fingerprints and no .tmet bytes. A snapshot only makes sense while it is small and cheaper than the open it replaces, so its size and the cost of checking it are independent of the segment count (both test-pinned).
  • Correctness: checked only when a cached open happens. Refused (session opened normally, fresh snapshot written) on a different mef3io version, or when, per channel, the segment count, the start time in the first segment's .tmet header, the end time and provenance region (offset 960: last operation, version, modification count) in the last segment's .tmet header, or the last segment's .tdat/.tidx sizes differ; or when the set of channel folders or the session-level .rdat/.ridx sizes differ. The two first/last .tmet file UUIDs are compared too (a same-samples overwrite=True with new metadata), and the version includes the compiled extension's identity. No snapshot is saved when segments ≤ channels; a metadata-free marker sends later cached opens straight to a plain open. Blind spot, by design: an in-place change to a non-last segment that moves none of those is not detected: foreign tools, and mef3io's own repair or recovery when its segment-repaired stamp on the last segment was skipped (foreign region, bad CRC, a writer's temp file; the report says so). Opening without cache= always re-reads everything. Last-segment sizes and the session-level record sizes are read through open + fstat, not a bare stat, so an NFS client's attribute cache cannot hide another host's append. Each .timd is always listed; directory timestamps are never trusted (FUSE/object-store mounts report constant ones).
  • Encrypted sessions are never cached (any level). Channel infos hold decrypted metadata, which must not land in a plaintext file. No snapshot or marker is written for one. The order on load is:
  • The cache file is parsed as JSON.
  • A well-formed marker of this build and session is returned at once. It holds no metadata, and the Reader then opens normally. When that open succeeds and finds the session encrypted, the Reader deletes the marker. An open that fails (a missing or wrong password raises PasswordError) leaves it in place; it holds only a hash, so nothing is exposed.
  • For anything else, the encryption check runs before format, version and session are compared. A file that parses as a JSON object with a format key, meaning any mef3io snapshot or marker of any version (1.1.x format-4 snapshots included), is deleted for an encrypted session. Anything else is left alone, because an explicit cache=<path> can name any file.
  • Session identity: snapshots and markers store a SHA-256 of the absolute session path (session_key), never the path itself. It is only compared for equality, and paths can carry a subject's name.
  • Write interaction: none. Write paths are cache-unaware and never touch a snapshot; a stale one is refused at the next cached open.

7. Build & distribution

  • C++17, CMake ≥ 3.26. Dependencies: none beyond vendored tiny-AES-c and the test framework (Catch2, fetched for tests only).
  • Python: nanobind + scikit-build-core; cibuildwheel wheels for manylinux (x86_64/aarch64), macOS (arm64/x86_64), Windows (x86_64), CPython 3.10+.
  • mef3io installs with the compiled backend; if the extension is missing (exotic platform), package still imports with backend="pure".
  • MATLAB (later): flat extern "C" ABI shim over the high-level API + MEX wrappers; packaged toolbox. Deferred by agreement.

8. Testing strategy

  1. Struct layout goldens: byte-level fixtures generated once via legacy pymef/mef_tools; static_assert + parse/serialize round-trip per struct.
  2. The ultimate gate — legacy cross-compatibility matrix (CI):
  3. legacy mef_tools writes → mef3io reads (values, times, gaps equal)
  4. mef3io writes → legacy mef_tools/pymef reads
  5. crossed over: encryption {none, L1, L2} × input {float64+precision, int32+ufact} × NaN/discontinuity patterns × {1, n} segments × fs {integer, fractional} × records {none, mixed types}. Legacy stack runs in a pinned py3.10 CI env (pymef wheel availability).
  6. Backend equivalence: pure vs C++ backend bit-identical outputs on the full matrix.
  7. Codec robustness: RED encode→decode property tests (random signals, extremes, constant, saturating int32), fuzzed truncated/corrupted blocks must raise MefError, never crash; CRC tampering detection.
  8. Crypto vectors: AES-128 NIST vectors; password scheme fixtures generated by meflib.
  9. Concurrency: TSAN job; deterministic-output test across n_threads ∈ {1, 2, max}; parallel-readers stress test.
  10. Benchmarks (benchmarks/): open time (cold/warm/cached), bulk read, random-window read (DL access pattern), write throughput — mef3io-C++ vs mef3io-pure vs legacy pymef.

9. Phases

Phase Deliverable Gate
P0 Scaffolding: CMake, nanobind hello-world wheel, CI skeleton, golden-file generation script (uses legacy mef_tools) wheel builds on 3 OSes
P1 fmt structs + CRC + password/AES (read side) parses all golden .tmet/.tidx byte-identically to oracles
P2 RED decode, mmap IO, lazy Session tree, index binary search reads golden sessions == pymef output exactly
P3 High-level Reader (uutc reads, NaN gaps, scaling, toc) + records read compat matrix read-direction green
P4 RED encode + low-level write path (segments, append, indices, CRCs, encryption write, records write) legacy pymef reads mef3io-written sessions
P5 High-level Writer semantics (precision inference, int32+ufact path, NaN splitting, guards) full compat matrix green
P6 Threading + metadata cache TSAN clean; benchmark targets met
P7 Python package polish: compat shim, pure backend integrated, wheels via cibuildwheel existing mef_tools test suite passes against mef3io.compat
P8 Benchmarks, docs, examples —
P9+ MATLAB C ABI + MEX toolbox; pure write path; viewer TOC API hardening —

10. Open items / risks

  • recording_time_offset corrupted-convention heuristic: opt-in flag (§4.4); confirm against real affected datasets before P3 closes.
  • Exotic record types (LNTP, CSti, ESti): implemented from mefrec.h layouts, round-trip-tested only; Note/EDFA/Seiz/SyLg get full oracle coverage.
  • Writer concurrent-access guard (lock file vs advisory): decide in P4.
  • Legacy pymef in CI depends on installability in a pinned env; fallback is pre-generated golden sessions committed as fixtures.
  • pandas is a dependency only of the compat shim/records convenience, not the core package (numpy-only core).