<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Alain, I totally agree with your points.
<div class=""><br class="">
</div>
<div class="">I noticed a typo in my earlier e-mail, which should have stated: </div>
<div class=""><br class="">
</div>
<div class="">"However, currently there is no reason NOT to load in the continuous data first, then do the filtering, and then do the epoching."</div>
<div class=""><br class="">
</div>
<div class="">Have a good weekend,</div>
<div class="">JM</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 23 Jun 2023, at 15:06, Alain de Cheveigne via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Hi
Jan-Mathijs, all,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You
are right that with proper precautions one can epoch then filter with no ill effects.<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">However
filtering before epoching remains attractive because simple and relatively safe (less likely to get things wrong). No need to worry about whether the FIR is a good enough approximation to the IIR one had in mind, padding is sufficient, etc.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">For
example, one might want to high-pass with a low cutoff frequency (e.g. 0.1 Hz) to minimize distortion of the ERP. This requires an impulse response duration on the order of tens of seconds. An investigator might find this out of proportion and prefer some
other strategy prone to edge effects.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Note
that an alternative to filtering is detrending, which can be applied before and/or after epoching. Applied before epoching, it reduces the implicit steps at the onset and/or offset of each trial (the origin of the edge effects). Applied after epoching, it
removes the trend local to each trial with minimal impact on the ERP. A typical strategy is to epoch with padding on both sides, fit and remove the trend (possibly with zero weight on the interval of interest), then clip off the padding. </span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Filtering
and detrending are both vulnerable to high-amplitude glitches outside the interval of interest but within the span of the filter or fit. For example, a subject might tend to blink before or after each trial. This can be addressed with robust detrending for
which there is no filtering equivalent (I think).</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Alain</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
On 23 Jun 2023, at 07:54, Schoffelen, J.M. (Jan Mathijs) via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>> wrote:<br class="">
<br class="">
Hi Frans,<br class="">
<br class="">
Historically, due to RAM constraints etc. a tradition FieldTrip pipeline started with epoching, then filtering. However, currently there is no reason to load in the continuous data first, then do the filtering, and then do the epoching. Although I didn’t read
Julian’s reply in detail, I suspect that his code is doing exactly that.<br class="">
<br class="">
There is just still a lot of documentation online which reverses this optimal order.<br class="">
<br class="">
Note, in addition, that if one uses a filter with a finite impulse response, the filtered signal will not depend on the overall length of the data segment that you supply to the filter, once you get sufficiently far away from the edges. In other words, for
instance, the center piece a one minute data segment that has been filtered will look exactly the same as a filtered version of the exact same center piece obtained from the 1-hour continuous recording.<span class="Apple-converted-space"> </span><br class="">
<br class="">
Fieldtrip allows for cfg.padding in ft_preprocessing, which will allow you to obtain epoched and filtered data in a single call to ft_preprocessing, without suffering from the edge effects of the filter. Once the value for cfg.padding exceeds the maximum epoch
length + the filter order (and provided you use a finite impulse response filter) then you should be good.<br class="">
<br class="">
Best wishes,<br class="">
Jan-Mathijs<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On 15 Feb 2023, at 11:33, Frans Nord via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>> wrote:<br class="">
<br class="">
Hello mailing-list,<br class="">
<br class="">
I have a question regarding the comment about Fieldtrip in the recently published paper "EEG is better left alone" [1]. While I don't agree with the conclusions of the paper there is a specific comment regarding Fieldtrip that I would like some comments about:<br class="">
<br class="">
"We realized that the FieldTrip preprocessing function extracts data epochs before filtering the data when provided with both filter settings and epoch information. After consulting with FieldTrip developers, for all analyses using FieldTrip, we used an alternate
multi-step implementation allowing us to extract epochs after the raw data had been filtered."<br class="">
<br class="">
Looking through the code of ft_preprocessing and how the preproc function is used in my own preprocessing pipeline i realized that I seem to do the filtering on raw data epochs with this code:<br class="">
<br class="">
cfg = [];<br class="">
cfg.continuous ='yes';<br class="">
cfg.trialfun =<span class="Apple-converted-space"> </span><br class="">
<br class="">
'ft_trialfun_general'<br class="">
;<br class="">
cfg.trialdef.eventtype =<span class="Apple-converted-space"> </span><br class="">
<br class="">
'STATUS'<br class="">
;<br class="">
cfg.trialdef.eventvalue = {1,2,3,4};<span class="Apple-converted-space"> </span><br class="">
<br class="">
% Trigger values<br class="">
<br class="">
cfg.trialdef.prestim =.5;<span class="Apple-converted-space"> </span><br class="">
<br class="">
% -500 ms before the stimulation<br class="">
<br class="">
cfg.trialdef.poststim = 2;<span class="Apple-converted-space"> </span><br class="">
<br class="">
% +2000ms after the stimulation<br class="">
<br class="">
cfg.dataset = path_to_dataset;<span class="Apple-converted-space"> </span><br class="">
<br class="">
% set the name of the dataset<br class="">
<br class="">
cfg = ft_definetrial(cfg);<span class="Apple-converted-space"> </span><br class="">
<br class="">
% read the list of the specific stimulus<br class="">
cfg.hpfilter<br class="">
=<br class="">
<br class="">
'yes';<br class="">
cfg.hpfreq = 1;[data] =ft_preprocessing(cfg);<br class="">
<br class="">
<br class="">
I did however think that the code above would apply the high pass filter on the continuous data and not on the epochs but it seems that I was wrong. Or am I? So I'm now consulting the E-mail list to get some clarity about this and a potential recommendation
on what to do instead. Basically, how to implement the "multi-step implementation" within Fieldtrip.<br class="">
<br class="">
Thank you for providing the Fieldtrip toolbox<br class="">
Best<br class="">
Frans<br class="">
<br class="">
[1] Delorme, A. EEG is better left alone. Sci Rep 13, 2372 (2023).<span class="Apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://doi.org/10.1038/s41598-023-27528-0__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4qia6CZo$" class="">https://urldefense.com/v3/__https://doi.org/10.1038/s41598-023-27528-0__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4qia6CZo$</a><span class="Apple-converted-space"> </span><br class="">
_______________________________________________<br class="">
fieldtrip mailing list<br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" class="">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br class="">
<a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!-_6L05NY4jlsfrP8C3I8I2jFQwOMSiDKS1tqSQ21aLsqyTB_rndzUnLg-8WiMQbKsVsuFCrx5lJzqbdD1tXsy2Tu8f4-RQxPapetqA$" class="">https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!-_6L05NY4jlsfrP8C3I8I2jFQwOMSiDKS1tqSQ21aLsqyTB_rndzUnLg-8WiMQbKsVsuFCrx5lJzqbdD1tXsy2Tu8f4-RQxPapetqA$</a><span class="Apple-converted-space"> </span><br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
fieldtrip mailing list<br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" class="">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br class="">
<a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4zPBfgKQ$" class="">https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4zPBfgKQ$</a><span class="Apple-converted-space"> </span><br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">fieldtrip
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!61hGn_iqSQTqOUy5TR8ASuo3th2U8mj2LiIj1gmQhqZt2NtnKDt3WqoEVWCmSIHMFnolsu2de5rRpkJnjxYJ4TTv0myofG2Rlu5zVg$" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!61hGn_iqSQTqOUy5TR8ASuo3th2U8mj2LiIj1gmQhqZt2NtnKDt3WqoEVWCmSIHMFnolsu2de5rRpkJnjxYJ4TTv0myofG2Rlu5zVg$</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>