<div dir="ltr">hi Konstantinos,<div><br></div>thanks, tried your suggestion, but using ft_definetrial is just as slow as regular ft_preprocessing when using trialdef.triallength = Inf, and even slower when setting trialdef.triallength = 1 (or some other number).<div><br></div><div>additional testing showed that the slow reading only occurs when the Brainvision subformat (data orientation) is "VECTORIZED". when the Brainvision subformat is "MULTIPLEXED", FieldTrip's reading is much faster and on par with eeglab's import functionality. guessing that "MULTIPLEXED" orientation prevents FieldTrip from reading in blockwise in the first place. so as a workaround, I could ensure that all our Brainvision data is multiplexed to begin with (but many other users might not have that option).</div><div><br></div><div>while I see the potential advantages of reading data blockwise when not all data is required for further processing, I think that loading complete continuous data into memory is a very typical use case that I'm surprised is not supported. At least for vectorized Brainvision data, blockwise reading is tremendously slowing down rather than speeding up the pipeline (and curious whether this affects other data formats too...)</div><div><br></div><div>thanks again for your input!</div><div>Roy</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 6, 2024 at 3:28 AM Konstantinos Tsilimparis via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl">fieldtrip@science.ru.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3392641543496371366">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="m_3392641543496371366WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">Hi Roy,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">I believe you’re correct that FieldTrip reads data in blocks. For example, it processes the first 5 seconds from all 64 EEG channels, then moving
on to the next 5 seconds, and so forth. From my experience with MEG data, reading the data in blocks does seem to improve the speed of data importing. Unfortunately, I’m not as familiar with how EEGLAB handles data imports.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">One advantage of FieldTrip that might speed up data reading is that it allows to selectively read the raw data according to pre-defined trial triggers
(see <a href="https://www.fieldtriptoolbox.org/tutorial/preprocessing/" target="_blank">https://www.fieldtriptoolbox.org/tutorial/preprocessing/</a>). In contrast, in EEGLAB the whole dataset has to be loaded into memory before it can be processed. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">Given that sleep data often requires loading continuous data without triggers, one potential solution could be to segment the continuous data into
one-second ‘’fake’’ trials while reading from disk (see: <a href="https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces" target="_blank">
https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces</a>). This is just a thought, and I’m not sure if it will speed up your analysis in the end, but it might be worth a try!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">Konstantinos<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> fieldtrip <<a href="mailto:fieldtrip-bounces@science.ru.nl" target="_blank">fieldtrip-bounces@science.ru.nl</a>>
<b>On Behalf Of </b>Roy Cox via fieldtrip<br>
<b>Sent:</b> Tuesday, September 3, 2024 1:25 PM<br>
<b>To:</b> FieldTrip discussion list <<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a>><br>
<b>Cc:</b> Roy Cox <<a href="mailto:roycox.roycox@gmail.com" target="_blank">roycox.roycox@gmail.com</a>><br>
<b>Subject:</b> [FieldTrip] brainvision reading with ft_preprocessing slow<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">hi all,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I noticed that FieldTrip's ft_preprocessing is extremely slow reading in our Branvision files (high-density sleep, about 8 GB). I've compared it to eeglab's import tool for Brainvision (<span class="m_3392641543496371366gmail-il">bva</span>-io1.71)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">%---eeglab (all FieldTrip paths removed)<u></u><u></u></p>
</div>
<p class="MsoNormal">tic<br>
[bv_folder,bv_file]=fileparts(bf_file);<br>
EEG=pop_loadbv(bv_folder,[bv_file '.vhdr']);<br>
t_eeglab=toc<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">t_eeglab =<br>
<br>
40.3622<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">%--fieldtrip (all EEGLAB paths removed)<u></u><u></u></p>
</div>
<p class="MsoNormal">tic<br>
cfg=[];<br>
cfg.dataset=bf_file;<br>
cfg.continuous = 'yes';<br>
cfg.readbids ='no';<br>
data_ft=ft_preprocessing(cfg);<br>
t_ft=toc<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">t_ft =<br>
<br>
458.0801<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So that's tenfold slower.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">One clue that may be of help: as we keep the data on a remote server I can monitor the ethernet throughput. Whereas eeglab's reading function quickly jumps to 1 Mbps (presumably fetching the file with all available bandwith), Fieldtrip's
reading operation barely leads to an observable increase in traffic. Without knowing anything, I would speculate that data is fetched piecewise, e.g., by channel or block, leading to notcieable slowing for larger files.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is there anything that can be done to speed up reading data?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Roy<u></u><u></u></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
fieldtrip mailing list<br>
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" target="_blank">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<a href="https://doi.org/10.1371/journal.pcbi.1002202" rel="noreferrer" target="_blank">https://doi.org/10.1371/journal.pcbi.1002202</a><br>
</div></blockquote></div>