[FieldTrip] filtering continuous vs. time-locked data

[Schoffelen, J.M. (Jan Mathijs)]

Dear Vadim,

Indeed in practice - if you want to highpass filter your data at a low cutoff frequency, as you want - it is perhaps better to filter the ‘continuous’ data. Or, at least to filter the data in longer segments than the ones you want to subsequently analyse. At low highpass cutoff frequencies, the edge artifacts of the filter will be quite large, particularly if the data segments are short, and an IIR filter is used (FT’s firws implementation is a bit better-behaved, but I still wouldn’t use it in practice).

Once upon a time, the available RAM in computers did not allow for reading in an entire recording at once, so that’s why in a lot of tutorials the filtering is still done on the smaller segments. This is in many cases not a big deal, as long as the data is more or less well behaved, and the requested filter settings are not too adventurous.

Note, that, if you apply the filtering at the ft_preprocessing step, combining it with reading in epochs from the raw data file, the cfg.padding option can be used to alleviate edge effects of the filter in the data segments of interest. The underlying idea is that ft_preprocessing reads in a specified longer segment of data, applies the filter, and then cuts off the edges to return the smaller epoch-of-interest. The potential drawback is that artifacts outside your scope-of-interest bleed into the epochs, but this would also happen if you were to filter the continuous data, and cut into epochs later on.

Best wishes,

Jan-Mathijs



On 10 Apr 2022, at 22:13, Vadim Axelrod via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Dear fieldrip forum,

First, as someone originally from Ukraine it was heartwarming for me to see your yellow-blue logo. Thank you.

I am confused with regard to the stage to filter the data of the ERP experiment (hpfilter and lpfilter). According to tutorials, the hpfilter and lpfilter are done at trials level (i.e., not for continuous data):
https://www.fieldtriptoolbox.org/workshop/baci2017/preprocessing/<https://urldefense.com/v3/__https://www.fieldtriptoolbox.org/workshop/baci2017/preprocessing/__;!!HJOPV4FYYWzcc1jazlU!5-jSpu_vkn7LSyzYFlV5CW8TvBrlg29_lb0rY6RJQqkVwRoRbK9uJ_PMNJYOEwC1dek8qeop0hdnl_necuB2qFMQS5vTAXdZxfrYRw$>
https://www.fieldtriptoolbox.org/workshop/oslo2019/introduction/<https://urldefense.com/v3/__https://www.fieldtriptoolbox.org/workshop/oslo2019/introduction/__;!!HJOPV4FYYWzcc1jazlU!5-jSpu_vkn7LSyzYFlV5CW8TvBrlg29_lb0rY6RJQqkVwRoRbK9uJ_PMNJYOEwC1dek8qeop0hdnl_necuB2qFMQS5vTAXfyyFfjlg$>

However, this clearly resulted in an artifact in my case. In the attached graph I compared this result to filtering at the stage of continuous data  and just having a notch electricity line filter (without fieldtrip). For some reason neither bsfilter nor dftfilter work for me (this is a separate issue). Below is the code I used. I am using the fieldtrip version that I downloaded several weeks ago.

Thank you for the help.
Vadim

function FilterTimeLockedData(fileName, condNameArr)

    cfg                    = [];
    cfg.trialdef.prestim   = 0.5;% in seconds
    cfg.trialdef.poststim  = 1; %in seconds
    cfg.trialdef.eventtype = condNameArr;
    cfg.dataset            = fileName;
    cfg_tr_def             = ft_definetrial(cfg);

    cfg                    = [];
    cfg.dataset            = fileName;
    data_continous         = ft_preprocessing(cfg); % read raw data

    data                   = ft_redefinetrial(cfg_tr_def, data_continous);

    cfg                = [];
    cfg.hpfilter       = 'yes';        % enable high-pass filtering
    cfg.lpfilter       = 'yes';        % enable low-pass filtering
    cfg.hpfreq         = 0.2;           % set up the frequency for high-pass filter
    cfg.lpfreq         = 30;          % set up the frequency for low-pass filter

    data               = ft_preprocessing(cfg,data);

  function FilterContinousData(fileName, condNameArr)

    cfg                    = [];
    cfg.trialdef.prestim   = 0.5;% in seconds
    cfg.trialdef.poststim  = 1; %in seconds
    cfg.trialdef.eventtype = condNameArr;
    cfg.dataset            = fileName;
    cfg_tr_def             = ft_definetrial(cfg);

    cfg                    = [];
    cfg.dataset            = fileName;
    cfg.hpfilter           = 'yes';        % enable high-pass filtering
    cfg.lpfilter           = 'yes';        % enable low-pass filtering
    cfg.hpfreq             = 0.2;           % set up the frequency for high-pass filter
    cfg.lpfreq             = 30;          % set up the frequency for low-pass filter
    data_continous         = ft_preprocessing(cfg); % read raw data

    data                   = ft_redefinetrial(cfg_tr_def, data_continous);

<filtering_comparison.jpg>_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!5-jSpu_vkn7LSyzYFlV5CW8TvBrlg29_lb0rY6RJQqkVwRoRbK9uJ_PMNJYOEwC1dek8qeop0hdnl_necuB2qFMQS5vTAXfOobgA4Q$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20220411/b2283d22/attachment.htm>