<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.gmail-il
{mso-style-name:gmail-il;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hi Roy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">I believe you’re correct that FieldTrip reads data in blocks. For example, it processes the first 5 seconds from all 64 EEG channels, then moving
on to the next 5 seconds, and so forth. From my experience with MEG data, reading the data in blocks does seem to improve the speed of data importing. Unfortunately, I’m not as familiar with how EEGLAB handles data imports.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">One advantage of FieldTrip that might speed up data reading is that it allows to selectively read the raw data according to pre-defined trial triggers
(see <a href="https://www.fieldtriptoolbox.org/tutorial/preprocessing/">https://www.fieldtriptoolbox.org/tutorial/preprocessing/</a>). In contrast, in EEGLAB the whole dataset has to be loaded into memory before it can be processed. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Given that sleep data often requires loading continuous data without triggers, one potential solution could be to segment the continuous data into
one-second ‘’fake’’ trials while reading from disk (see: <a href="https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces">
https://www.fieldtriptoolbox.org/tutorial/continuous/#segmenting-continuous-data-into-one-second-pieces</a>). This is just a thought, and I’m not sure if it will speed up your analysis in the end, but it might be worth a try!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Konstantinos<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> fieldtrip <fieldtrip-bounces@science.ru.nl>
<b>On Behalf Of </b>Roy Cox via fieldtrip<br>
<b>Sent:</b> Tuesday, September 3, 2024 1:25 PM<br>
<b>To:</b> FieldTrip discussion list <fieldtrip@science.ru.nl><br>
<b>Cc:</b> Roy Cox <roycox.roycox@gmail.com><br>
<b>Subject:</b> [FieldTrip] brainvision reading with ft_preprocessing slow<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I noticed that FieldTrip's ft_preprocessing is extremely slow reading in our Branvision files (high-density sleep, about 8 GB). I've compared it to eeglab's import tool for Brainvision (<span class="gmail-il">bva</span>-io1.71)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">%---eeglab (all FieldTrip paths removed)<o:p></o:p></p>
</div>
<p class="MsoNormal">tic<br>
[bv_folder,bv_file]=fileparts(bf_file);<br>
EEG=pop_loadbv(bv_folder,[bv_file '.vhdr']);<br>
t_eeglab=toc<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">t_eeglab =<br>
<br>
40.3622<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">%--fieldtrip (all EEGLAB paths removed)<o:p></o:p></p>
</div>
<p class="MsoNormal">tic<br>
cfg=[];<br>
cfg.dataset=bf_file;<br>
cfg.continuous = 'yes';<br>
cfg.readbids ='no';<br>
data_ft=ft_preprocessing(cfg);<br>
t_ft=toc<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">t_ft =<br>
<br>
458.0801<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So that's tenfold slower.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One clue that may be of help: as we keep the data on a remote server I can monitor the ethernet throughput. Whereas eeglab's reading function quickly jumps to 1 Mbps (presumably fetching the file with all available bandwith), Fieldtrip's
reading operation barely leads to an observable increase in traffic. Without knowing anything, I would speculate that data is fetched piecewise, e.g., by channel or block, leading to notcieable slowing for larger files.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is there anything that can be done to speed up reading data?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Roy<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>