[FieldTrip] brainvision reading with ft_preprocessing slow

[Roy Cox]

hi Konstantinos,

thanks, tried your suggestion, but using ft_definetrial is just as slow as
regular ft_preprocessing when using trialdef.triallength = Inf, and even
slower when setting trialdef.triallength = 1 (or some other number).

additional testing showed that the slow reading only occurs when the
Brainvision subformat (data orientation) is "VECTORIZED". when the
Brainvision subformat is "MULTIPLEXED", FieldTrip's reading is much faster
and on par with eeglab's import functionality. guessing that "MULTIPLEXED"
orientation prevents FieldTrip from reading in blockwise in the first
place. so as a workaround, I could ensure that all our Brainvision data is
multiplexed to begin with (but many other users might not have that option).

while I see the potential advantages of reading data blockwise when not all
data is required for further processing, I think that loading complete
continuous data into memory is a very typical use case that I'm surprised
is not supported. At least for vectorized Brainvision data, blockwise
reading is tremendously slowing down rather than speeding up the pipeline
(and curious whether this affects other data formats too...)

thanks again for your input!
Roy

On Fri, Sep 6, 2024 at 3:28 AM Konstantinos Tsilimparis via fieldtrip <
fieldtrip at science.ru.nl> wrote:

> Hi Roy,
>
>
>
> I believe you’re correct that FieldTrip reads data in blocks. For example,
> it processes the first 5 seconds from all 64 EEG channels, then moving on
> to the next 5 seconds, and so forth. From my experience with MEG data,
> reading the data in blocks does seem to improve the speed of data
> importing. Unfortunately, I’m not as familiar with how EEGLAB handles data
> imports.
>
>
>
> One advantage of FieldTrip that might speed up data reading is that it
> allows to selectively read the raw data according to pre-defined trial
> triggers (see https://www.fieldtriptoolbox.org/tutorial/preprocessing/).
> In contrast, in EEGLAB the whole dataset has to be loaded into memory
> before it can be processed.
>
>
>
> Given that sleep data often requires loading continuous data without
> triggers, one potential solution could be to segment the continuous data
> into one-second ‘’fake’’ trials while reading from disk (see:
> https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces).
> This is just a thought, and I’m not sure if it will speed up your analysis
> in the end, but it might be worth a try!
>
>
>
> Best,
>
> Konstantinos
>
>
>
>
>
> *From:* fieldtrip <fieldtrip-bounces at science.ru.nl> *On Behalf Of *Roy
> Cox via fieldtrip
> *Sent:* Tuesday, September 3, 2024 1:25 PM
> *To:* FieldTrip discussion list <fieldtrip at science.ru.nl>
> *Cc:* Roy Cox <roycox.roycox at gmail.com>
> *Subject:* [FieldTrip] brainvision reading with ft_preprocessing slow
>
>
>
> hi all,
>
>
>
> I noticed that FieldTrip's ft_preprocessing is extremely slow reading in
> our Branvision files (high-density sleep, about 8 GB). I've compared it to
> eeglab's import tool for Brainvision (bva-io1.71)
>
>
>
> %---eeglab (all FieldTrip paths removed)
>
> tic
> [bv_folder,bv_file]=fileparts(bf_file);
> EEG=pop_loadbv(bv_folder,[bv_file '.vhdr']);
> t_eeglab=toc
>
>
>
> t_eeglab =
>
>     40.3622
>
>
>
> %--fieldtrip (all EEGLAB paths removed)
>
> tic
> cfg=[];
> cfg.dataset=bf_file;
> cfg.continuous = 'yes';
> cfg.readbids ='no';
> data_ft=ft_preprocessing(cfg);
> t_ft=toc
>
>
>
> t_ft =
>
>   458.0801
>
>
>
> So that's tenfold slower.
>
>
>
> One clue that may be of help: as we keep the data on a remote server I can
> monitor the ethernet throughput. Whereas eeglab's reading function quickly
> jumps to 1 Mbps (presumably fetching the file with all available bandwith),
> Fieldtrip's reading operation barely leads to an observable increase in
> traffic. Without knowing anything, I would speculate that data is fetched
> piecewise, e.g., by channel or block, leading to notcieable slowing for
> larger files.
>
>
>
> Is there anything that can be done to speed up reading data?
>
>
>
> Regards,
>
> Roy
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20240906/bfd3339e/attachment.htm>