[FieldTrip] High pass filtering before epoching

Fri Jun 23 15:41:51 CEST 2023

Hi Alain, I totally agree with your points.

I noticed a typo in my earlier e-mail, which should have stated:

"However, currently there is no reason NOT to load in the continuous data first, then do the filtering, and then do the epoching."

Have a good weekend,
JM

On 23 Jun 2023, at 15:06, Alain de Cheveigne via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hi Jan-Mathijs, all,

You are right that with proper precautions one can epoch then filter with no ill effects.

However filtering before epoching remains attractive because simple and relatively safe (less likely to get things wrong). No need to worry about whether the FIR is a good enough approximation to the IIR one had in mind, padding is sufficient, etc.

For example, one might want to high-pass with a low cutoff frequency (e.g. 0.1 Hz) to minimize distortion of the ERP. This requires an impulse response duration on the order of tens of seconds. An investigator might find this out of proportion and prefer some other strategy prone to edge effects.

Note that an alternative to filtering is detrending, which can be applied before and/or after epoching. Applied before epoching, it reduces the implicit steps at the onset and/or offset of each trial (the origin of the edge effects). Applied after epoching, it removes the trend local to each trial with minimal impact on the ERP. A typical strategy is to epoch with padding on both sides, fit and remove the trend (possibly with zero weight on the interval of interest), then clip off the padding.

Filtering and detrending are both vulnerable to high-amplitude glitches outside the interval of interest but within the span of the filter or fit. For example, a subject might tend to blink before or after each trial. This can be addressed with robust detrending for which there is no filtering equivalent (I think).

Alain

On 23 Jun 2023, at 07:54, Schoffelen, J.M. (Jan Mathijs) via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hi Frans,

Historically, due to RAM constraints etc. a tradition FieldTrip pipeline started with epoching, then filtering. However, currently there is no reason to load in the continuous data first, then do the filtering, and then do the epoching. Although I didn’t read Julian’s reply in detail, I suspect that his code is doing exactly that.

There is just still a lot of documentation online which reverses this optimal order.

Note, in addition, that if one uses a filter with a finite impulse response, the filtered signal will not depend on the overall length of the data segment that you supply to the filter, once you get sufficiently far away from the edges. In other words, for instance, the center piece a one minute data segment that has been filtered will look exactly the same as a filtered version of the exact same center piece obtained from the 1-hour continuous recording.

Fieldtrip allows for cfg.padding in ft_preprocessing, which will allow you to obtain epoched and filtered data in a single call to ft_preprocessing, without suffering from the edge effects of the filter. Once the value for cfg.padding exceeds the maximum epoch length + the filter order (and provided you use a finite impulse response filter) then you should be good.

Best wishes,
Jan-Mathijs

On 15 Feb 2023, at 11:33, Frans Nord via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hello mailing-list,

I have a question regarding the comment about Fieldtrip in the recently published paper "EEG is better left alone" [1]. While I don't agree with the conclusions of the paper there is a specific comment regarding Fieldtrip that I would like some comments about:

"We realized that the FieldTrip preprocessing function extracts data epochs before filtering the data when provided with both filter settings and epoch information. After consulting with FieldTrip developers, for all analyses using FieldTrip, we used an alternate multi-step implementation allowing us to extract epochs after the raw data had been filtered."

Looking through the code of ft_preprocessing and how the preproc function is used in my own preprocessing pipeline i realized that I seem to do the filtering on raw data epochs with this code:

cfg = [];
cfg.continuous ='yes';
cfg.trialfun =

'ft_trialfun_general'
;
cfg.trialdef.eventtype =

'STATUS'
;
cfg.trialdef.eventvalue = {1,2,3,4};

% Trigger values

cfg.trialdef.prestim =.5;

% -500 ms before the stimulation

cfg.trialdef.poststim = 2;

% +2000ms after the stimulation

cfg.dataset = path_to_dataset;

% set the name of the dataset

cfg = ft_definetrial(cfg);

% read the list of the specific stimulus
cfg.hpfilter
=

'yes';
cfg.hpfreq = 1;[data] =ft_preprocessing(cfg);

I did however think that the code above would apply the high pass filter on the continuous data and not on the epochs but it seems that I was wrong. Or am I? So I'm now consulting the E-mail list to get some clarity about this and a potential recommendation on what to do instead. Basically, how to implement the "multi-step implementation" within Fieldtrip.

Thank you for providing the Fieldtrip toolbox
Best
Frans

[1] Delorme, A. EEG is better left alone. Sci Rep 13, 2372 (2023). https://urldefense.com/v3/__https://doi.org/10.1038/s41598-023-27528-0__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4qia6CZo$
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!-_6L05NY4jlsfrP8C3I8I2jFQwOMSiDKS1tqSQ21aLsqyTB_rndzUnLg-8WiMQbKsVsuFCrx5lJzqbdD1tXsy2Tu8f4-RQxPapetqA$

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!49ARQZhij07QcXbznLgyQvIp4PNcdVSoZ2EsDDB043TbF0wd09QCDkgHP7DO1BXEz9jwCWU0q6yBAjm91FJ0WtPuFdr4zPBfgKQ$

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!61hGn_iqSQTqOUy5TR8ASuo3th2U8mj2LiIj1gmQhqZt2NtnKDt3WqoEVWCmSIHMFnolsu2de5rRpkJnjxYJ4TTv0myofG2Rlu5zVg$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20230623/1ac41231/attachment.htm>