[FieldTrip] brainvision reading with ft_preprocessing slow

[Roy Cox]

hi all,

I noticed that FieldTrip's ft_preprocessing is extremely slow reading in
our Branvision files (high-density sleep, about 8 GB). I've compared it to
eeglab's import tool for Brainvision (bva-io1.71)

%---eeglab (all FieldTrip paths removed)
tic
[bv_folder,bv_file]=fileparts(bf_file);
EEG=pop_loadbv(bv_folder,[bv_file '.vhdr']);
t_eeglab=toc

t_eeglab =

    40.3622

%--fieldtrip (all EEGLAB paths removed)
tic
cfg=[];
cfg.dataset=bf_file;
cfg.continuous = 'yes';
cfg.readbids ='no';
data_ft=ft_preprocessing(cfg);
t_ft=toc

t_ft =

  458.0801

So that's tenfold slower.

One clue that may be of help: as we keep the data on a remote server I can
monitor the ethernet throughput. Whereas eeglab's reading function quickly
jumps to 1 Mbps (presumably fetching the file with all available bandwith),
Fieldtrip's reading operation barely leads to an observable increase in
traffic. Without knowing anything, I would speculate that data is fetched
piecewise, e.g., by channel or block, leading to notcieable slowing for
larger files.

Is there anything that can be done to speed up reading data?

Regards,
Roy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20240903/47e6ef93/attachment.htm>