mef3io — Design & Implementation Plan¶
Status: agreed design, 2026-07-02
Supersedes: requirements/requirements.md (pure-Python plan, deleted)
This is the original plan
It records the design as agreed before the code was written, and is kept for its reasoning. Section 4.6 (threading) has been updated to the code as built; the rest is the plan. Where the plan differs from what shipped, the Python, C++ and MATLAB guides describe the library as it is. The main differences:
- the pure-Python backend is a stub;
Writer.writetakes float data andwrite_int32the counts; records are written withwrite_annotations; - the core is C++20; its AES code is mef3io's own (no vendored library); block indices are read into memory and cached under a size budget with least-recently-used eviction, not memory-mapped;
- the MATLAB binding and the C ABI exist (they were "later" here);
- one
Readercan be shared between threads: its mutable state is locked, not immutable after open (see 4.6); - the warm-start cache lives in Python (
mef3io.cache), and the module names in 4.1 are not the ones the code uses; - features added later are not planned here:
inspect, cancelling and progress, the validator with repair and recovery, single-file.mefd.tarsessions, de-identified copies (anonymize_session, see De-identification), modified copies, cut and merge (Modified copies, Merging), and session renaming.
1. Goal¶
A single C++ core implementing MEF 3.0 read and write, wrapped thinly for
Python (now) and MATLAB (later), with the high-level semantics of today's
mef_tools (MefReader/MefWriter) living in the C++ layer so every
language binding behaves identically. A pure-Python backend with the same API
ships in the package as fallback, benchmark reference, and test oracle.
Non-goals: MEF video files (.vidd/.vmet/.vidx) are dropped from the library
entirely — never used in practice; could be reconsidered in the future if
worthwhile. The only footprint in the code: session traversal silently ignores
.vidd directories (with a debug-level note), and video structs/APIs from
meflib/pymef are simply not mirrored. Also out of scope (v1): lossy RED
encoding (decode is supported), performance parity guarantees against
original meflib (we expect to beat it on open time; decode should be
comparable).
2. Architecture¶
┌─ python: mef3io pkg (nanobind) ─┐ ┌─ MATLAB toolbox (MEX, later) ─┐ ┌─ viewer (later)
│ numpy in/out, GIL released │ │ │ │
└──────────────┬──────────────────┘ └──────────────┬────────────────┘ │
│ stable C ABI shim (later, for MATLAB) │
┌──────────────┴─────────────────────────────────────┴──────────────────┴───┐
│ HIGH-LEVEL C++ API — mef3io::Reader / mef3io::Writer │
│ float64 + NaN-gap semantics, ufact scaling, precision inference, │
│ int32+ufact primitive write path, append logic, annotations, cache │
├────────────────────────────────────────────────────────────────────────────┤
│ CORE C++ (format codec, no policy) │
│ fmt structs + CRC · AES + password scheme · RED codec · │
│ mmap IO + dir traversal · Session/Channel/Segment · records · │
│ RAII, exceptions, zero globals, thread pool │
└────────────────────────────────────────────────────────────────────────────┘
Key decisions (agreed):
- High-level ("smart") semantics live in C++, wrappers are thin (<~500 lines).
- On-disk format stays byte-identical to meflib output. Code is written from
the spec (field tables), not ported from meflib. meflib/pymef,
mef3_dump, andmef_tools/reimplementation.pyserve as three independent oracles. - Dropped from meflib: global state, exit-on-error,
FILT_*filters,e_*allocator wrappers, lossy RED encode, video. - Added: thread safety (planned as immutable-after-open readers; built as locked shared state, see 4.6), context-managed resources, exception-based errors, mmap'd binary-searchable indices, lazy metadata, metadata cache, internal parallelism.
3. Repo layout (monorepo, this repository)¶
mef_tools/ (repo root — name unchanged for now)
├── core/ C++17 library "mef3io_core"
│ ├── include/mef3io/ public headers
│ ├── src/
│ ├── third_party/ tiny-AES-c (vendored, public domain)
│ └── tests/ Catch2 unit tests
├── bindings/python/ nanobind extension "_mef3io"
├── python/mef3io/ Python package
│ ├── __init__.py Reader / Writer (C++ backend)
│ ├── pure/ pure-Python backend (same API)
│ │ (from reimplementation.py + mef3_dump decode)
│ ├── compat.py drop-in MefReader/MefWriter (mef_tools.io API)
│ └── cache.py cache location/policy helpers
├── mef_tools/ legacy package — unchanged, used as oracle
├── tests/ cross-implementation compatibility matrix
├── benchmarks/ C++ vs pure vs legacy pymef
└── reference_files/ meflib, pymef, mef3_dump (read-only oracles)
4. C++ core design¶
4.1 Modules (namespaces)¶
| Namespace | Contents |
|---|---|
mef3io::fmt |
POD-ish structs with exact-layout (de)serialization: UniversalHeader (1024 B), MetadataSection1 (1536 B), TimeSeriesMetadataSection2 (10752 B), MetadataSection3 (3072 B), TimeSeriesIndex (56 B), RedBlockHeader (304 B), record header/index + all record types from mefrec.h (Note, EDFA, Seiz+SeizCh, SyLg, LNTP, CSti, ESti, Curs, Epoc) |
mef3io::crc |
CRC-32 Koopman (table-driven), header/body CRC helpers |
mef3io::crypto |
AES-128-ECB (vendored tiny-AES-c, validated against NIST vectors and cryptography in tests), terminal-byte password extraction, L1/L2 validation-field scheme, key derivation |
mef3io::red |
RED decode (range decoder + diff reconstruction + unscale/retrend, incl. lossy-written blocks) and lossless RED encode (diff + range coder, keysamples, per-block stats) |
mef3io::io |
mmap file abstraction (POSIX/Win32), .mefd/.timd/.segd discovery, path/name rules |
mef3io::(Session,Channel,Segment) |
lazy object tree over the directory structure |
| top level | Reader, Writer, Cache, ThreadPool, error hierarchy |
4.2 Serialization approach¶
No reinterpret_cast of packed structs (alignment/UB trap that meflib dances
around); each struct has explicit parse(span<const byte>) /
serialize(span<byte>) against a field table with static offsets — the same
tables already validated in reimplementation.py::FIELD_DEFINITIONS and the
deleted requirements doc. static_assert on all offsets/sizes.
4.3 Error handling¶
Exception hierarchy: MefError ← {FormatError, CrcError, PasswordError,
IoError, WriteConflictError}. No error-code returns in the public API; the C
ABI shim (later) converts to codes. Bindings map to Python exceptions.
4.4 Reading — designed around partial access¶
Reader::open()reads only: directory tree + the.tmetfiles needed for channel list, fs, ufact, start/end, nsamp (last-segment shortcut where possible). No.tidxis touched at open..tidxfiles are mmap'd, never materialized: entries are fixed 56 B and time/sample ordered → binary search directly on raw bytes for[t0, t1]/[s0, s1]block ranges. O(log n) seek, zero parse cost.read_raw(channel, t0, t1)→ preallocated int32 buffer sizedround((t1-t0)*fs/1e6); blocks decode in parallel into disjoint slices; gap samples flagged (output carries a validity mask + ufact + actual start).read(channel, t0, t1)→ float64 = int32 · ufact, gaps = NaN.toc(channel)→ block-level table (start_time, start_sample, n_samples, offset) for viewers/dataloaders.- Discontinuity gap length rounds to nearest sample (sub-sample gaps
tolerated; never accumulate rounded periods — always
t = start_uutc + round(n * 1e6 / fs)with fs as double). recording_time_offset(superseded: the implemented rule is in mef3_format.md, and using the offset as anonymization is deprecated, see anonymization.md): apply per spec (stored = actual − offset). The "corrupted" convention observed in the wild (stored = offset − actual, handled heuristically in mef3_dump) is behind an opt-in leniency flag, off by default, warning when triggered.
4.5 Writing¶
- Primitive path:
write(channel, span<int32>, ufact, start_uutc, fs)— amplifier counts + V/bit conversion factor stored asunits_conversion_factor. No transformation of samples. - Convenience path:
write(channel, span<double>, start_uutc, fs, precision?)— quantizes to int32 with10^precision(inferred if absent, same algorithm as mef_tools incl. int32 dynamic-range guard). - NaN policy (float path; NaN = discontinuity, blocks are tiled within the
segment — NaNs never create new
.segdsegments, matching current behavior): - Default (new API): strict split — every NaN run becomes a true on-disk gap (discontinuous blocks, nothing stored for the gap). No sentinels, no compression penalty.
max_nans_allowed=<n>opt-in: NaN runs shorter than n are embedded as theRED_NANsentinel (INT32_MIN, reads back as NaN) to reduce block count. Today's mef_tools does this implicitly (default'fs'= 1 s) via the NaN→int32 cast coincidentally hittingRED_NAN; here it is explicit. The compat shim keeps the legacy'fs'default.- All-NaN input: no-op on disk, but explicit — every write returns a summary (samples_written, blocks, gaps_skipped) and warns when samples_written == 0 (today this is silent).
- int32 primitive path has no NaN: gaps are structural (separate
writecalls, each an implicit discontinuity, or an optional validity mask).INT32_MINin int32 input is rejected by default (collides withRED_NAN); opt-in flag to treat it as NaN sentinel. - Read side:
read()(float64) → NaN for both true gaps and storedRED_NAN;read_raw()(int32) → validity mask distinguishes gap / sentinel / data. - Segment control:
new_segmentflag; append goes to last segment with end-time/fs/ufact consistency checks (same guards as mef_tools). - Records: write all supported types at session or channel level; merge with
existing records on append (as mef_tools does):
write_recordsalways appends in place (record data, index entries, then the headers; body CRCs extended), creating the files when absent; readers sort by time (#156). - Encryption on write: none, or fully encrypted (section 2 → L1 key, section 3
→ L2 key) — a fixed pairing driven by password presence, not three
independent modes. "Level-1 only" is not a valid MEF file; see the canonical
explanation in
docs/encryption_model.md. Includes section encryption flags, validation fields, and encrypted record bodies. - Float fs fully supported (sf8 end-to-end); block length heuristic from fs
kept but overridable (
mef_block_len). - Section-2 buffer sizing is a hard output requirement, not bookkeeping.
meflib-based readers allocate from the
maximum_*fields before decoding, and0is not the NO_ENTRY sentinel for any of them, so every one is measured from the blocks actually written:maximum_difference_bytesfrom the encoded RED headers, themaximum_contiguous_*trio from runs delimited by the same.tidxdiscontinuity flag a reader uses. Appends recompute the contiguous trio from the full index (repairing segments written before this was implemented).maximum_difference_bytesstays exact across a chunked write, because the writer keeps the running maximum for the blocks it encoded itself; for a segment reopened from disk, whose earlier blocks it cannot see without a seek each, it takes meflib'sRED_MAX_DIFFERENCE_BYTESas a floor instead — an over-declaration only costs a reader memory, while under-declaring truncates the buffer it decodes into. The read path never consults these fields — it sizes from each block's own header — so older or foreign declarations stay readable. Seedocs/mef3_format.mdanddocs/legacy_comparison.md. - When data is on disk.
write()returns once the data is handed to the operating system; only a.tmetis flushed before it is renamed into place (every append, and a channel's first segment), since a short.tmetmakes the whole session fail to open.flush()andclose()wait for everything written so far in one combined wait (onesyncfson Linux); the opt-inwait_for_disk=Truemakes each write wait for its own files. A power cut before a flush can tear the last append of a segment;recover_session(apply=True)restores a valid session (index and data itself, declarations throughrepair_session). Seedocs/long_recordings.md, "When is my data on disk?". - Writes never touch caches; a cached open detects the change (see §6).
4.6 Threading¶
- ONE thread rule (Threads): an object's
n_threadsis how many threads it may use for everything it does; 0 = automatic, decided only byresolve_threads(parallel.hpp: hardware concurrency for CPU work, at least 8 for file access), 1 = no extra threads. Settable at runtime (atomic; the next call uses it). No per-call override (removed in 2.0); a MultiReader splits its total privately (Reader::read_raw_split). - No persistent thread pool: each parallel step calls
parallel_for. - Parallel units: RED block decode and encode — independent blocks, disjoint
output slices ⇒ deterministic, race-free — and the open's metadata pass
(channel listings, segment file checks, every
.tmetread + parse), run withparallel_try_eachand assembled in order, so the result, the strict error and the lenientproblems()are those of a serial open (the first failure in listing order, never the first in time). - Concurrent calls on one Reader from user threads are safe. The metadata
parsed at open is immutable afterwards and read without a lock. The Reader
is otherwise NOT immutable after open: it caches block indices (with LRU eviction) and,
in lenient mode, records problems as it meets them. That state is locked,
and each call pins the index it is reading, so another thread's eviction
never frees it (
core/tests/test_concurrency.cpp, also under TSan). The collect phase of a read takes these locks only around cache access, never across file I/O or decoding. - Python bindings release the GIL for every potentially long call: opening
(the
Reader,SessionandSessionWriterconstructors, #79), reads, writes,toc,segments,records,write_records, and the session-level functions. Cheap accessors keep it. - A session has one writer at a time. That is documented, not enforced:
there is no lock file. Within one write call the writer is internally
parallel: block boundaries are computed up front (from
mef_block_lenand the NaN/discontinuity split), all blocks of the call are RED-encoded in parallel into their own buffers, and the.tdatbody and.tidxentries are then assembled and written in order on the calling thread. Output is byte-identical regardless ofn_threads. Memory scales with one call's encoded data, so very long signals are written in chunks.
5. Python package¶
import mef3io
with mef3io.Reader("/data/session.mefd", password="...", n_threads=8) as r:
r.channels # list[str]
r.info("ch1") # dict: fs, ufact, start, end, nsamp, units...
x = r.read("ch1", t0, t1) # float64, NaN gaps
xi = r.read_raw("ch1", t0, t1) # int32 + mask + ufact
r.toc("ch1") # block table (numpy structured / DataFrame)
r.records() # annotations
with mef3io.Writer("/data/session.mefd", password1=..., password2=...) as w:
w.write("ch1", counts_int32, ufact=2.5e-7, start=t0, fs=1024.5)
w.write("ch2", volts_float64, start=t0, fs=256.0, precision=3)
w.write_records(df)
mef3io.pure— sameReader/WriterAPI, pure Python+numpy(+numba optional). Backend selectable:mef3io.Reader(..., backend="pure"). v1: pure read path complete (exists already across reimplementation.py + mef3_dump); pure write path is a stretch goal after C++ write lands.mef3io.compat—MefReader/MefWritermatchingmef_tools.iosignatures, defaults, and quirks (e.g.get_datalist/str return-shape asymmetry,max_nans_written='fs'default) so it's a drop-in import change; the legacymef_toolspackage itself stays untouched in the repo as oracle.
6. Metadata cache ("warm start")¶
Purpose: collapse the many-small-file cost of session open (mainly on network storage; lazy open + mmap indices already fix the pathological local case).
- Modes:
cache="auto"(default): per-user OS cache dir (platformdirs-style:~/Library/Caches/mef3io,$XDG_CACHE_HOME,%LOCALAPPDATA%), file<hash16>.mef3cache, named by a hash of the absolute session path ONLY (since 2.0.0; earlier versions prefixed the session folder's stem, which put a name that is often the subject's in a file name; the nextcache="auto"open of that session deletes its old-named file, exactly that name and only if it is a mef3io snapshot, and builds a new one). Works with read-only sessions.cache=<path>/ explicit persistent: user-requested dump file (e.g. next to the.mefd). Never discovered or used unless explicitly passed — a foreign/stale cache file inside a session dir cannot surprise a default-configured reader.cache=None: off.- Contents (original plan, not what was built): per-file fingerprints and
the raw, still-encrypted
.tmetbytes. The built cache stores decoded channel infos instead, which is why it refuses encrypted sessions outright. - As built (
python/mef3io/cache.py, cache format 6, issue #64): the snapshot holds the channel infos, the open-time warnings, the exact mef3io version that wrote it, and an O(1)-per-channel signature — no per-file fingerprints and no.tmetbytes. A snapshot only makes sense while it is small and cheaper than the open it replaces, so its size and the cost of checking it are independent of the segment count (both test-pinned). - Correctness: checked only when a cached open happens. Refused (session
opened normally, fresh snapshot written) on a different mef3io version, or
when, per channel, the segment count, the start time in the first segment's
.tmetheader, the end time and provenance region (offset 960: last operation, version, modification count) in the last segment's.tmetheader, or the last segment's.tdat/.tidxsizes differ; or when the set of channel folders or the session-level.rdat/.ridxsizes differ. The two first/last.tmetfile UUIDs are compared too (a same-samplesoverwrite=Truewith new metadata), and the version includes the compiled extension's identity. No snapshot is saved when segments ≤ channels; a metadata-free marker sends later cached opens straight to a plain open. Blind spot, by design: an in-place change to a non-last segment that moves none of those is not detected: foreign tools, and mef3io's own repair or recovery when itssegment-repairedstamp on the last segment was skipped (foreign region, bad CRC, a writer's temp file; the report says so). Opening withoutcache=always re-reads everything. Last-segment sizes and the session-level record sizes are read throughopen+fstat, not a barestat, so an NFS client's attribute cache cannot hide another host's append. Each.timdis always listed; directory timestamps are never trusted (FUSE/object-store mounts report constant ones). - Encrypted sessions are never cached (any level). Channel infos hold decrypted metadata, which must not land in a plaintext file. No snapshot or marker is written for one. The order on load is:
- The cache file is parsed as JSON.
- A well-formed marker of this build and session is returned at once. It
holds no metadata, and the Reader then opens normally. When that open
succeeds and finds the session encrypted, the Reader deletes the marker.
An open that fails (a missing or wrong password raises
PasswordError) leaves it in place; it holds only a hash, so nothing is exposed. - For anything else, the encryption check runs before format, version
and session are compared. A file that parses as a JSON object with a
formatkey, meaning any mef3io snapshot or marker of any version (1.1.x format-4 snapshots included), is deleted for an encrypted session. Anything else is left alone, because an explicitcache=<path>can name any file. - Session identity: snapshots and markers store a SHA-256 of the absolute
session path (
session_key), never the path itself. It is only compared for equality, and paths can carry a subject's name. - Write interaction: none. Write paths are cache-unaware and never touch a snapshot; a stale one is refused at the next cached open.
7. Build & distribution¶
- C++17, CMake ≥ 3.26. Dependencies: none beyond vendored tiny-AES-c and the test framework (Catch2, fetched for tests only).
- Python: nanobind + scikit-build-core;
cibuildwheelwheels for manylinux (x86_64/aarch64), macOS (arm64/x86_64), Windows (x86_64), CPython 3.10+. mef3ioinstalls with the compiled backend; if the extension is missing (exotic platform), package still imports withbackend="pure".- MATLAB (later): flat
extern "C"ABI shim over the high-level API + MEX wrappers; packaged toolbox. Deferred by agreement.
8. Testing strategy¶
- Struct layout goldens: byte-level fixtures generated once via legacy
pymef/mef_tools;
static_assert+ parse/serialize round-trip per struct. - The ultimate gate — legacy cross-compatibility matrix (CI):
- legacy
mef_toolswrites →mef3ioreads (values, times, gaps equal) mef3iowrites → legacymef_tools/pymef reads- crossed over: encryption {none, L1, L2} × input {float64+precision, int32+ufact} × NaN/discontinuity patterns × {1, n} segments × fs {integer, fractional} × records {none, mixed types}. Legacy stack runs in a pinned py3.10 CI env (pymef wheel availability).
- Backend equivalence: pure vs C++ backend bit-identical outputs on the full matrix.
- Codec robustness: RED encode→decode property tests (random signals,
extremes, constant, saturating int32), fuzzed truncated/corrupted blocks
must raise
MefError, never crash; CRC tampering detection. - Crypto vectors: AES-128 NIST vectors; password scheme fixtures generated by meflib.
- Concurrency: TSAN job; deterministic-output test across n_threads ∈ {1, 2, max}; parallel-readers stress test.
- Benchmarks (
benchmarks/): open time (cold/warm/cached), bulk read, random-window read (DL access pattern), write throughput — mef3io-C++ vs mef3io-pure vs legacy pymef.
9. Phases¶
| Phase | Deliverable | Gate |
|---|---|---|
| P0 | Scaffolding: CMake, nanobind hello-world wheel, CI skeleton, golden-file generation script (uses legacy mef_tools) | wheel builds on 3 OSes |
| P1 | fmt structs + CRC + password/AES (read side) |
parses all golden .tmet/.tidx byte-identically to oracles |
| P2 | RED decode, mmap IO, lazy Session tree, index binary search | reads golden sessions == pymef output exactly |
| P3 | High-level Reader (uutc reads, NaN gaps, scaling, toc) + records read |
compat matrix read-direction green |
| P4 | RED encode + low-level write path (segments, append, indices, CRCs, encryption write, records write) | legacy pymef reads mef3io-written sessions |
| P5 | High-level Writer semantics (precision inference, int32+ufact path, NaN splitting, guards) |
full compat matrix green |
| P6 | Threading + metadata cache | TSAN clean; benchmark targets met |
| P7 | Python package polish: compat shim, pure backend integrated, wheels via cibuildwheel |
existing mef_tools test suite passes against mef3io.compat |
| P8 | Benchmarks, docs, examples | — |
| P9+ | MATLAB C ABI + MEX toolbox; pure write path; viewer TOC API hardening | — |
10. Open items / risks¶
recording_time_offsetcorrupted-convention heuristic: opt-in flag (§4.4); confirm against real affected datasets before P3 closes.- Exotic record types (LNTP, CSti, ESti): implemented from
mefrec.hlayouts, round-trip-tested only; Note/EDFA/Seiz/SyLg get full oracle coverage. - Writer concurrent-access guard (lock file vs advisory): decide in P4.
- Legacy pymef in CI depends on installability in a pinned env; fallback is pre-generated golden sessions committed as fixtures.
- pandas is a dependency only of the compat shim/records convenience, not the core package (numpy-only core).