[FieldTrip] brainvision reading with ft_preprocessing slow

[Konstantinos Tsilimparis]

Hi Roy,

I believe you’re correct that FieldTrip reads data in blocks. For example, it processes the first 5 seconds from all 64 EEG channels, then moving on to the next 5 seconds, and so forth. From my experience with MEG data, reading the data in blocks does seem to improve the speed of data importing. Unfortunately, I’m not as familiar with how EEGLAB handles data imports.

One advantage of FieldTrip that might speed up data reading is that it allows to selectively read the raw data according to pre-defined trial triggers (see https://www.fieldtriptoolbox.org/tutorial/preprocessing/). In contrast, in EEGLAB the whole dataset has to be loaded into memory before it can be processed.

Given that sleep data often requires loading continuous data without triggers, one potential solution could be to segment the continuous data into one-second ‘’fake’’ trials while reading from disk (see: https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces). This is just a thought, and I’m not sure if it will speed up your analysis in the end, but it might be worth a try!

Best,
Konstantinos


From: fieldtrip <fieldtrip-bounces at science.ru.nl> On Behalf Of Roy Cox via fieldtrip
Sent: Tuesday, September 3, 2024 1:25 PM
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Cc: Roy Cox <roycox.roycox at gmail.com>
Subject: [FieldTrip] brainvision reading with ft_preprocessing slow

hi all,

I noticed that FieldTrip's ft_preprocessing is extremely slow reading in our Branvision files (high-density sleep, about 8 GB). I've compared it to eeglab's import tool for Brainvision (bva-io1.71)

%---eeglab (all FieldTrip paths removed)
tic
[bv_folder,bv_file]=fileparts(bf_file);
EEG=pop_loadbv(bv_folder,[bv_file '.vhdr']);
t_eeglab=toc

t_eeglab =

    40.3622

%--fieldtrip (all EEGLAB paths removed)
tic
cfg=[];
cfg.dataset=bf_file;
cfg.continuous = 'yes';
cfg.readbids ='no';
data_ft=ft_preprocessing(cfg);
t_ft=toc

t_ft =

  458.0801

So that's tenfold slower.

One clue that may be of help: as we keep the data on a remote server I can monitor the ethernet throughput. Whereas eeglab's reading function quickly jumps to 1 Mbps (presumably fetching the file with all available bandwith), Fieldtrip's reading operation barely leads to an observable increase in traffic. Without knowing anything, I would speculate that data is fetched piecewise, e.g., by channel or block, leading to notcieable slowing for larger files.

Is there anything that can be done to speed up reading data?

Regards,
Roy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20240906/039780ba/attachment.htm>