<div dir="ltr">Thank you for sharing your code Julian. Looks like a good approach!<div><br></div><div>Also, thank you for sharing the preprocessing pipeline from your spring school. Can I ask you what the motivation is behind doing the ICA before the electrode interpolation? </div><div><br></div><div>Best,</div><div>Frans</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 15, 2023 at 12:29 PM Julian Keil via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl">fieldtrip@science.ru.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Hello Frans,<div><br></div><div>I don’t know whether there is a specific reason that TF segments first and filters later, but here’s what I do in order to be sure that the filtering is done on the whole epoch (<a href="https://urldefense.com/v3/__https://github.com/BioPsychKiel/SpringSchool2022/blob/main/track_02/01_Code/sc01_preproc.m__;!!HJOPV4FYYWzcc1jazlU!_yF8KxqbQ9i7O04I1jY6rUBExCiDZCkq2juuIOst9PZY0BoNkNcN_XM_EdKi85VuqUKBBTFIkjVBq0cmPuLpKQN0Bg$" target="_blank">https://github.com/BioPsychKiel/SpringSchool2022/blob/main/track_02/01_Code/sc01_preproc.m</a>):</div><div><br></div><div><table><tbody><tr><td id="m_-8233973778444393167LC35"><span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> 1.1. First, read in the header to define the trials</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC36"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg <span>=</span> []; <span><span>%</span> Always clear the configuration</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC37"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.dataset <span>=</span> [<span>inpath</span>,<span>indat</span>(<span>v</span>).name]; <span><span>%</span> Set the dataset</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC39"><font face="Monaco"><span style="font-style:normal;font-size:12px">cfg.trialdef.eventtype <span>=</span> <span><span>'</span>Markers<span>'</span></span>; <span><span>%</span> We now know that the trigger channel is called 'Stimulus'</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC40"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.trialdef.eventvalue <span>=</span> {<span>20</span>,<span>30</span>}; <span><span>%</span> Define the relevant triggers</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC41"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.trialdef.prestim <span>=</span> <span>1.5</span>; <span><span>%</span> Seconds before the stimulus</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC42"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.trialdef.poststim <span>=</span> <span>1.5</span>; <span><span>%</span> Seconds after the stimulus</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC43">
</td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC44"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg <span>=</span> <span>ft_definetrial</span>(<span>cfg</span>); <span><span>%</span> Store the trial definition</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC45">
</td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC46"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> 1.2. Save the trial-definition</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC47"><font face="Monaco"><span style="font-style:normal;font-size:12px"> trl <span>=</span> <span>cfg</span>.<span>trl</span>;</span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC48">
</td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC49"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> 1.3. Then define the entire dataset</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC50"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg <span>=</span> []; <span><span>%</span> Always clear the configuration</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC51"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.dataset <span>=</span> [<span>inpath</span>,<span>indat</span>(<span>v</span>).name]; <span><span>%</span> Set the dataset</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC52"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.trialdef.ntrials <span>=</span> <span>1</span>; <span><span>%</span> One long trial to cover the entire dataset</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC53">
</td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC54"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg <span>=</span> <span>ft_definetrial</span>(<span>cfg</span>); <span><span>%</span> Store the trial definition</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC55"> </td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC56"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%%</span> <span>2. Preprocessing</span></font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC57"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> This step will read, filter and cut the raw data</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC58"> </td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC59"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> 2.1. Read in the raw data and filter</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC60"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.demean <span>=</span> <span><span>'</span>yes<span>'</span></span>; <span><span>%</span> Remove mean across time to get rid of offset</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC61"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.hpfilter <span>=</span> <span><span>'</span>yes<span>'</span></span>; <span><span>%</span> Highpassfilter to remove slow drifts</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC62"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.hpfreq <span>=</span> <span>.1</span>; <span><span>%</span> Highpass frequency. 0.1 removes most slow drifts</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC63"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.hpfilttype <span>=</span> <span><span>'</span>firws<span>'</span></span>; <span><span>%</span> Type of filter</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC64"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.lpfilter <span>=</span> <span><span>'</span>yes<span>'</span></span>; <span><span>%</span> The same for lowpass</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC65"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.lpfreq <span>=</span> <span>45</span>; <span><span>%</span> 50Hz line noise destroys the data, only take data below</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC66"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.lpfilttype <span>=</span> <span><span>'</span>firws<span>'</span></span>; <span><span>%</span> Again the type</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC67">
</td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC68"><font face="Monaco"><span style="font-style:normal;font-size:12px"> data_p <span>=</span> <span>ft_preprocessing</span>(<span>cfg</span>);</span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC69"><br></td></tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC75"> <span style="font-style:normal;font-size:12px"><font face="Monaco"><span>%</span> 2.2. Cut the data according to the trial definition</font></span></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC76"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg <span>=</span> [];</span></font></td>
</tr>
<tr>
</tr></tbody></table><table><tbody><tr><td id="m_-8233973778444393167LC77"><font face="Monaco"><span style="font-style:normal;font-size:12px"> cfg.trl <span>=</span> <span>trl</span>; <span><span>%</span> Use the trl-structure defined above</span></span></font></td>
</tr>
<tr>
</tr></tbody></table><br><table><tbody><tr><td id="m_-8233973778444393167LC78"><font face="Monaco"><span style="font-style:normal;font-size:12px">data_t <span>=</span> <span>ft_redefinetrial</span>(<span>cfg</span>,<span>data_p</span>);</span></font></td></tr></tbody></table><div><br></div><div>Please not that this takes quite a bit longer than doing it the FT-way.</div><div>I hope this helps,</div><div><br></div><div>Julian</div><div><br></div><div><br><blockquote type="cite"><div>Am 15.02.2023 um 11:33 schrieb Frans Nord via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a>>:</div><br><div><div dir="ltr">Hello mailing-list,<div><br></div><div>I have a question regarding the comment about Fieldtrip in the recently published paper "EEG is better left alone" [1]. While I don't agree with the conclusions of the paper there is a specific comment regarding Fieldtrip that I would like some comments about:</div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></div><div><font color="#1d1c1d" face="Slack-Lato, Slack-Fractions, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">"</span></font><span style="font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:18px">We realized that the FieldTrip preprocessing function extracts data epochs before filtering the data when provided with both filter settings and epoch information. After consulting with FieldTrip developers, for all analyses using FieldTrip, we used an alternate multi-step implementation allowing us to extract epochs after the raw data had been filtered."</span></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></div><div>Looking through the code of ft_preprocessing and how the preproc function is used in my own preprocessing pipeline i realized that I seem to do the filtering on raw data epochs with this code:<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></div><div><br></div><div><div style="padding:30px"><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg = [];</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.continuous =<span style="color:rgb(167,9,245)">'yes'</span>;</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.trialfun = <span style="color:rgb(167,9,245)">'ft_trialfun_general'</span>;</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.trialdef.eventtype = <span style="color:rgb(167,9,245)">'STATUS'</span>;</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.trialdef.eventvalue = {1,2,3,4}; <span style="color:rgb(0,128,19)">% Trigger values</span></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.trialdef.prestim =.5; <span style="color:rgb(0,128,19)">% -500 ms before the stimulation</span></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.trialdef.poststim = 2; <span style="color:rgb(0,128,19)">% +2000ms after the stimulation</span></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg.dataset = path_to_dataset; <span style="color:rgb(0,128,19)">% set the name of the dataset</span></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap">cfg = ft_definetrial(cfg); <span style="color:rgb(0,128,19)">% read the list of the specific stimulus</span></span>cfg.hpfilter<span style="white-space:pre-wrap;font-size:10pt"> = </span><span style="white-space:pre-wrap;font-size:10pt;color:rgb(167,9,245)">'yes';</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="white-space:pre-wrap;font-size:10pt">cfg.hpfreq = 1;</span><span style="font-size:10pt;white-space:pre-wrap">[data] =ft_preprocessing(cfg);</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-size:10pt;white-space:pre-wrap"><br></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">I did however think that the code above would apply the high pass filter on the continuous data and not on the epochs but it seems that I was wrong. Or am I? So I'm now consulting the E-mail list to get some clarity about this and a potential recommendation on what to do instead. Basically, how to implement the "multi-step implementation" within Fieldtrip.</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">Thank you for providing the Fieldtrip toolbox</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">Best</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">Frans</span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></span></div><div style="font-size:10pt;font-family:Menlo,Monaco,Consolas,"Courier New",monospace"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">[1] </span><span style="font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:16px">Delorme, A. EEG is better left alone. </span><i style="box-sizing:inherit;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:16px">Sci Rep</i><span style="font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:16px"> </span><span style="font-weight:bolder;box-sizing:inherit;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:16px">13</span><span style="font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:16px">, 2372 (2023). <a href="https://urldefense.com/v3/__https://doi.org/10.1038/s41598-023-27528-0__;!!HJOPV4FYYWzcc1jazlU!4hjKg111dQ7s9-L3G_ddKy01SDzvgiEvo8gLx_R-WDx6Qxj6JJS0HuEwuUx1atexqoHmVc8kvd5ouM4p4smQ$" target="_blank">https://doi.org/10.1038/s41598-023-27528-0</a></span></div></div></div></div>
_______________________________________________<br>fieldtrip mailing list<br><a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br><a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!_yF8KxqbQ9i7O04I1jY6rUBExCiDZCkq2juuIOst9PZY0BoNkNcN_XM_EdKi85VuqUKBBTFIkjVBq0cmPuL82C-wOg$" target="_blank">https://doi.org/10.1371/journal.pcbi.1002202</a><br></div></blockquote></div><br></div></div>_______________________________________________<br>
fieldtrip mailing list<br>
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" target="_blank">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!_aEDEsb7ydsax2OVHHDDpqnHPIUFI-6pcv8G6sYmsU4N6TNd5kEOITO2SHwH7dWoa0Pwsc2gfAKB6js7iloz$" rel="noreferrer" target="_blank">https://doi.org/10.1371/journal.pcbi.1002202</a><br>
</blockquote></div>