Read EGI simple-binary event channels in blocks - #14250
Open
bruAristimunha wants to merge 2 commits into
Open
Conversation
_read_events walks the file one time sample at a time: for every sample it seeks past the data channels and issues a separate np.fromfile for that sample's event values, so opening a file costs two Python-level I/O calls per sample. Samples are stored as interleaved frames -- n_channels data values followed by n_events event values -- so the same bytes can be read a few MB at a time and the event rows sliced out. Same reads, ~4000x fewer calls. On a 32 MB file (30800 samples, 256 channels) read_raw_egi goes 170.3 -> 10.8 ms (15.7x); the shipped 79 KB test_egi.raw goes 1.96 -> 1.53 ms (1.28x). cProfile put _read_events at 89% of read_raw_egi beforehand (0.194 s of 0.219 s, 30821 np.fromfile calls). The cost is linear in recording length, so it grows with the file: the fixture above is ~2 minutes of data. Memory stays bounded -- the loop reads 4 MiB at a time rather than slurping the whole recording -- so the seeks are not traded for a large allocation. Output is bit-identical, data arrays and annotations both (the annotations are built from exactly these event channels).
bruAristimunha
requested review from
agramfort,
drammock and
larsoner
as code owners
August 29, 2026 19:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this implement/fix?
_read_events()walks a simple-binary EGI file one time sample at a time:for every sample it seeks past the data channels and issues a separate
np.fromfilefor that sample's event values. Opening a file therefore costs twoPython-level I/O calls per sample.
Samples are stored as interleaved frames —
n_channelsdata values followed byn_eventsevent values — so the same bytes can be read a few MB at a time andthe event rows sliced out. Same reads, ~4000x fewer calls.
Numbers
Median of 5 interleaved process pairs against a pristine
mainworktree:test_egi.raw(79 KB)Beforehand
cProfileput_read_eventsat 89% ofread_raw_egi(0.194 s of0.219 s, 30821
np.fromfilecalls). The cost is linear in recording length, soit grows with the file — the fixture above is only ~2 minutes of data.
Correctness
main: data arrays and annotations compared withnp.array_equal(the annotations matter here — they are built from exactlythese event channels).
whole recording, so the seeks are not traded for a large allocation.
pytest mne/io/egi: 26 passed.Caveats
No fixture in the testing dataset is long enough to show this, so CI cannot
demonstrate it. The large file was synthesised by tiling the shipped fixture's
data section and patching
n_samplesin the header; I verified the headerlayout matches the file exactly before tiling.
Additional information
AI disclosure: I directed the work and reviewed and tested every change; Claude
Code (Claude Opus 5) profiled the reader, ran the A/B measurements and made the
edits under my direction.