mef3io vs the legacy stack (pymef / mef_tools)¶
Both implementations read each other's files bit-identically at full access,
and encrypting a session never changes the decoded samples. The differences
below are measured, and each is reproducible by running
examples/08_legacy_compatibility.py
(performance) and
examples/09_encryption_replicability.py
(encryption/access levels); both need pip install "mef3io[test]".
Performance¶
Measured with examples/08: 5 channels × 5 h at 512 Hz (9.2 M samples per
channel, ~88 MB session, band-pass-filtered noise, precision 3), encrypted,
Apple-silicon macOS, Python 3.13:
| Operation | legacy | mef3io | speedup |
|---|---|---|---|
| Write (whole session) | 4.67 s | 0.66 s | 7.1× |
| Open / read headers (legacy-written) | 0.011 s | 0.004 s | 2.5× |
| Read all data (legacy-written file) | 2.78 s | 0.36 s | 7.8× |
| Read all data (mef3io-written file) | 2.80 s | 0.35 s | 8.0× |
| File size | 87.79 MB | 87.79 MB | identical |
Same run confirms cross-compatibility on unencrypted content at full access:
both readers return equal arrays with matching NaN positions on both writers'
files (Data equality: True, NaN positions match: True, all four
writer×reader combinations).
MATLAB vs Python binding (same C++ core)¶
Both bindings sit on the same core, so the language layer is essentially
free. Measured with matlab/benchmark_mef3io.m and
benchmarks/bindings_benchmark.py on the identical workload (5 ch × 5 h @
512 Hz = 46.1 M samples, smoothed noise + NaN gap, precision 3; same machine
as above; MATLAB R2026a / Python 3.13):
| write | read | file size | |
|---|---|---|---|
| Python, plain | 0.62 s (74 MS/s) | 0.34 s (134 MS/s) | 80.2 MB |
| Python, encrypted | 0.62 s (75 MS/s) | 0.34 s (137 MS/s) | 80.2 MB |
| MATLAB, plain | 0.74 s (63 MS/s) | 0.34 s (135 MS/s) | 80.1 MB |
| MATLAB, encrypted | 0.61 s (75 MS/s) | 0.34 s (137 MS/s) | 80.1 MB |
Takeaways: MATLAB and Python are within measurement noise of each other (reads identical at ~135 M samples/s; the one slower MATLAB write is first-run warmup), encryption costs nothing on either binding (it only wraps metadata — the signal codec path is unchanged), and both are the same ~7–8× ahead of the legacy pymef stack shown in the table above. Sessions written by either binding read back bit-identically in the other.
Level-1 password behavior — the main difference¶
MEF 3.0 encrypts section 2 (technical metadata: fs, sample counts, conversion factor) with the level-1 key and section 3 (subject identity, recording-time offset) with the level-2 key; signal blocks themselves are not encrypted. The intended contract: an L1 holder reads the signal and technical metadata but cannot see who the subject is; L2 unlocks everything (see encryption_model.md).
With an L1 password on an encrypted session:
| legacy (pymef / mef_tools 1.2.3) | mef3io | |
|---|---|---|
| Signal | 0 samples returned | bit-identical, complete |
| Technical metadata (s2) | ciphertext read as numbers (e.g. fs = −1.5·10¹⁹⁹) | correct |
| Subject metadata (s3) | no API; garbage used internally | cleanly locked, fields None |
| Annotations | see the writer gap below | refused (UNAVAILABLE) |
| Wrong / missing password | rejected | rejected |
pymef validates an L1 password (the session opens) but never uses the L1 key to decrypt section 2 — so fs and sample counts are ciphertext reinterpreted as float64/int64, and the windowed read matches zero blocks. This happens on legacy-written files as well as mef3io-written ones: it is a reader defect, not a file incompatibility. In practice the legacy L1 password was unusable.
mef3io validates the password, derives the access level, decrypts exactly the
sections that level allows, and reports the rest as unavailable
(Reader.info() → section3_available, subject fields None under L1).
Two-level access therefore works as designed: the L1 password can be given to
signal-processing staff without exposing subject identity.
Annotation encryption — a legacy writer gap¶
The legacy writer stores annotation record bodies unencrypted even in an encrypted session: they are readable with an L1 password — or straight off disk with no password. mef3io encrypts record bodies with the level-2 key (meflib semantics). Treat annotations in legacy-written encrypted sessions as unprotected; rewriting the session with mef3io fixes it.
Quantization — not bit-identical across writers, by design¶
For precision=3 the legacy writer computes np.round(x, 3) followed by a
truncating int32 cast of 1000 * rounded; mef3io stores round(x * 1000)
directly. Boundary samples can therefore differ by up to ~2 quantization
counts between the two writers. Both are valid MEF, each round-trips its own
quantization exactly, and both readers return bit-identical arrays for any
given file.
Section-2 buffer sizing — mef3io is stricter than pymef¶
The maximum_* fields in metadata section 2 tell a meflib-based reader
(CyberPSG and most established MEF tooling) how large a buffer to allocate
before it decodes anything; see
the format reference.
Writers disagree about them:
| Field | legacy stack | mef3io |
|---|---|---|
maximum_difference_bytes |
largest difference_bytes seen |
same |
maximum_contiguous_block_bytes |
the whole .tdat body |
bytes in the longest run between discontinuities |
maximum_contiguous_blocks |
the longest run (same as mef3io) | blocks in the longest run |
maximum_contiguous_samples |
never assigned | samples in the longest run |
number_of_discontinuities |
0 (from the mef_tools wrapper; pymef's own default is -1) |
the real count |
.tdat maximum_entry_size |
a sample count | largest block in bytes (mef3io's reading; meflib defines neither) |
"Legacy stack" matters here: several of these come from the mef_tools
wrapper rather than from pymef. pymef defaults block_interval and
number_of_discontinuities to -1; mef_tools/io.py overwrites them with
0, and 0 is the value that hurts.
pymef computes maximum_contiguous_block_bytes as the whole data body
regardless of the discontinuity flags it wrote, and never assigns
maximum_contiguous_samples. mef3io measures each run against the same .tidx
discontinuity flag a reader uses, and its repair writes that measured value in
BOTH directions — so a legacy file's whole-file over-declaration is brought
down to the longest run, not just raised where it was too small. Under-declaring
is the truncating direction and is rated an error; over-declaring only wastes
memory and is a warning.
Neither reader depends on maximum_difference_bytes — pymef sizes from
RED_MAX_DIFFERENCE_BYTES(maximum_block_samples) and mef3io from each block's
own header — so that divergence changes no decoded value in either direction.
It matters to third-party readers that trust the declarations.
number_of_discontinuities is different in kind: meflib's own
find_discontinuity_indices (meflib.c:3548) mallocs exactly that many
entries and writes one per flagged block, so the legacy 0 is a heap overflow
in any application that calls find_discontinuity_samples.
mef3io ≤ 1.1.2 left maximum_difference_bytes and
maximum_contiguous_block_bytes at 0, which those readers cannot distinguish
from an unset field. Sessions written by that version decode correctly in
mef3io and pymef but can crash a meflib-based reader; appending to such a
segment with a current version repairs the declarations in place, and rewriting
the session fixes them outright.
Confirmed against a meflib-based reader¶
The whole point of the section-2 work is a defect neither mef3io nor pymef can see — both size their buffers from each block's own header rather than from the declarations — so it can only be confirmed by the C library.
On 2026-09-21 the nine-session set from scripts/make_cyberpsg_check.py was
opened in CyberPSG and the traces drawn. Only the deliberate negative control
crashed.
| session | result |
|---|---|
| fresh write by this version | decoded |
| written then extended via the append path | decoded |
legacy mef_tools session, untouched |
decoded |
the same legacy session after repair_session |
decoded |
the 1.1.2 declarations (maximum_difference_bytes = 0) |
crashed |
| the same samples with only the declarations repaired | decoded |
| encrypted | decoded |
.tmet with foreign padding, then appended |
decoded |
rebuilt by recover_session after an interrupted write |
decoded |
The last two rows of that pair are the argument: identical samples, identical blocks, only the declarations differ. That isolates the cause to metadata section 2 about as tightly as it can be isolated, and it confirms that the fix, the repair path and the recovery path all produce output that reader accepts.
"It opens" was never the test — the original failure let ReadSession succeed
and blew up later inside RED_decode, so the traces have to be drawn.