[FieldTrip] Removing power line

Kloosterman, Niels kloosterman at mpib-berlin.mpg.de
Mon Oct 4 16:29:33 CEST 2021 

Hi Emilie,

Regarding Zapline, we are currently working on a wrapper software called zapline-plus that (1) removes line noise using Zapline on segments of data, allowing it to deal with noise non-stationarities over time, (2) is fully automatic, so it requires no manual setting of N components to remove or the exact line noise frequency, and (3) produces detailed plots of the denoising process, including power spectra so you can see how it performs. It works very well also on mobile EEG data sets that we have tested - we would be curious to know how it works on your data.

See https://github.com/MariusKlug/zapline-plus

Niels

On 4. Oct 2021, at 13:46, fieldtrip-request at science.ru.nl<mailto:fieldtrip-request at science.ru.nl> wrote:

Send fieldtrip mailing list submissions to
fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>

To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
or, via email, send a message with subject or body 'help' to
fieldtrip-request at science.ru.nl

You can reach the person managing the list at
fieldtrip-owner at science.ru.nl

When replying, please edit your Subject line so it is more specific
than "Re: Contents of fieldtrip digest..."


Today's Topics:

  1. Re: Removing power line (Sarang S. Dalal)
  2. Re: Removing power line (Stephen Whitmarsh)
  3. Re: Removing power line (Alain de Cheveigne)
  4. Re: Removing power line (Emilie Caspar)


----------------------------------------------------------------------

Message: 1
Date: Mon, 4 Oct 2021 10:43:01 +0000
From: "Sarang S. Dalal" <sarang at cfin.au.dk>
To: "fieldtrip at science.ru.nl" <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Removing power line
Message-ID:
<95cba4971f21d1e42a6ec3d99c7f9db1da62661b.camel at cfin.au.dk>
Content-Type: text/plain; charset="utf-8"

Hi Emilie,

I agree with Stephen -- absolutely, looking at the power spectrum will help guide your strategy here. It's normally unlikely to see both 50 Hz and 60 Hz in the same place... but I could imagine this may happen if the village gets power from a local electricity grid (likely 50 Hz) but with some buildings using potentially mismatched power inverters on solar panels or diesel generators (can be either 50 Hz or 60 Hz depending on manufacturer). Fluorescent lighting and perhaps some appliances may also generate noise frequencies that are totally unrelated to 50/60 Hz.

Just to clarify, cfg.dftreplace = 'neighbour' uses the spectrum interpolation method that Sabine Leske and I developed. It's actually designed for powerline noise frequencies that are *less* sharp than the default dftfilter method. So while it sounds like you have an extreme situation, it might indeed be an appropriate strategy for your data, depending on what your power spectrum actually looks like.

The design and test scenario was 50 Hz line noise in a typical home environment in Germany, which was actually a precise 50 Hz but of moderately fluctuating amplitude. These sorts of common fluctuations effectively broaden the noise frequency peak from a sharp 50 Hz to, for example, 49-51 Hz. For such a case, you would indeed define cfg.dftbandwidth of 2. The "neighbor" bandwidth is more tricky, but this should represent "clean" frequencies since their amplitude is used to replace the powerline corruption. In this example, a neighbourbandwidth of 1 would imply that the amplitude at 48-49 Hz and 51-52 Hz would be used to "patch" the noise peak at 49-51 Hz.

You may have a more extreme scenario, but nevertheless looking at the power spectrum will help you see how broad each powerline noise peak is (use that to set "dftbandwidth"), and ensure that you pick clean neighbor frequencies (guiding "neighbourbandwidth").

Our paper describing the method has an example of real data and some tips on usage that might be helpful:
https://doi.org/10.1016/j.neuroimage.2019.01.026

Hope this helps!

Cheers,
Sarang


On Sun, 2021-10-03 at 14:45 +0200, Stephen Whitmarsh via fieldtrip wrote:
Dear Emilie,

Those sound indeed like challenging, and interesting, circumstances.
I don't have a direct answer to your question, that would depend on having more information about the data, but here are my 2 cents;

You could first make an FFT plot (e.g. using cfg.method = mtmfft in ft_freqanalysis) to get a better picture of the frequencies in your data (on both 'successful' and 'unsuccessful' data), for several reasons:

 *   You are also trying to remove both 50 and 60Hz, and their harmonics, of which probably only one of those would reflect the noise in the power lines.
 *   You are using cfg.dftreplace = 'neighbour'; which might only work if the frequency band of the noise is sharp enough, or can be estimated precise enough (depending on the duration of the data you are filtering).
 *   It would confirm whether it is line noise at all (which is probable, but more might be at play, e.g. some non-linear combinations and strange harmonics).
 *   You could look for whether the degree of noise is stationary, by e.g. plotting the FFT over different periods, but it might be more important that the frequency is accurately chosen and stable over time.

Depending on your data, the duration of data, whether or not your are looking at trials or the whole data file at once (*), etc., you might also want to use a 'welch method' for estimating the power in the data, i.e. by averaging over several (sliding) windows. Using cfg.method = 'mtmconvol' in ft_freqanalysis, then averaging over time with ft_selectdata (cfg.avgovertime), would be an easy way to do that. Once you have a good picture of the noise, it will be easier to both choose the right filters and to check their efficacy (by comparing the FFT before and after filtering, on both 'successful' and 'unsuccessful' data). In fact, since you are dealing with EEG in such noisy conditions, and assuming you are interested in either ERPs or slow oscillations (<40Hz), you could consider just using a lowpass filter at e.g. 40 Hz.

Good luck, I hope this helps,
Stephen

*) The behaviour of filters depends a lot on the structure/size of the data, so to better understand its behaviour you would need to describe your data(structures) in more detail, i.e. trial length, nr. of trials/channels, and any processing steps until filtering.




Op zo 3 okt. 2021 om 13:12 schreef Emilie Caspar via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>:
Dear Fieldtrippers,

I recently acquired EEG data in Rwanda in complicated testing conditions, especially for the power line : we were testing in rural villages with a single electrical system for the whole village. Our signal is thus parasited by what the villagers were plugging on the electrical system (so probably nonstationary noise) during the testing. The electrical system was also very rudimentary and not correctly grounded.

Usually, I use the following command in Fieldtrip to remove the 50H or 60 Hz power line and it works quite well. Here, it worked for roughly 70% of our acquired data, but it does not work for all the data acquired.

   cfg= [];
   cfg.dftfilter = 'yes';
   cfg.dftfreq = [50 60 100 120 150 180];
   cfg.dftreplace = 'neighbour';
   cfg.dftbandwidth = [2 2 2 2 2 2];
   cfg.dftneighbourwidth = [2 2 2 2 2 2];
   data_intpl = ft_preprocessing(cfg, allData_preprosses);


When I look at the graphs (see figure attached), it really looks like it’s a 60Hz noise, but it seems that the dftfilter function does not remove it. We are certain it’s power noise because when the electricity was cut off because of a storm or else, and we were thus only relying on the batteries to collect our data, the signal was perfect. So in theory we should be able to remove it but we have no cue of what other possibilities to try. Perhaps it’s because the noise is non stationary and dftfilter does not account for that? I know some residual power noise can stay after a dftfiltre, but here it does not remove anything.

Thanks a lot for the help,

Emilie

[cid:17c461876c4c75d62ec1]
---------------------------------------------
Prof. Dr. Emilie Caspar
Associate Professor
Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94
lab’s website: https://moralsocialbrain.com/
personal website: https://emiliecaspar.home.blog/
Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/9a599f3e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PowerLine.jpg
Type: image/jpeg
Size: 37857 bytes
Desc: PowerLine.jpg
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/9a599f3e/attachment-0001.jpg>

------------------------------

Message: 2
Date: Mon, 4 Oct 2021 13:11:58 +0200
From: Stephen Whitmarsh <stephen.whitmarsh at gmail.com>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Removing power line
Message-ID:
<CAFrxm=wMHSY_2VLCON=f6V2+9nP3kum=zuuo=6d+ypb+mfNBvw at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Sarang,

Cheers! Thanks for clarifying the 'neightbour' dft method (and developing
it in the first place :)).
Indeed, I was not aware of the paper/method and assumed a 'standard' use;
but as they say - assuming makes an ... ;-)
Looking forward to reading the paper and seeing how the data will improve.

Best wishes,
Stephen





Op ma 4 okt. 2021 om 12:49 schreef Sarang S. Dalal via fieldtrip <
fieldtrip at science.ru.nl>:

Hi Emilie,

I agree with Stephen -- absolutely, looking at the power spectrum will
help guide your strategy here. It's normally unlikely to see both 50 Hz and
60 Hz in the same place... but I could imagine this may happen if the
village gets power from a local electricity grid (likely 50 Hz) but with
some buildings using potentially mismatched power inverters on solar panels
or diesel generators (can be either 50 Hz or 60 Hz depending on
manufacturer). Fluorescent lighting and perhaps some appliances may also
generate noise frequencies that are totally unrelated to 50/60 Hz.

Just to clarify, cfg.dftreplace = 'neighbour' uses the spectrum
interpolation method that Sabine Leske and I developed. It's actually
designed for powerline noise frequencies that are *less* sharp than the
default dftfilter method. So while it sounds like you have an extreme
situation, it might indeed be an appropriate strategy for your data,
depending on what your power spectrum actually looks like.

The design and test scenario was 50 Hz line noise in a typical home
environment in Germany, which was actually a precise 50 Hz but of
moderately fluctuating amplitude. These sorts of common fluctuations
effectively broaden the noise frequency peak from a sharp 50 Hz to, for
example, 49-51 Hz. For such a case, you would indeed define
cfg.dftbandwidth of 2. The "neighbor" bandwidth is more tricky, but this
should represent "clean" frequencies since their amplitude is used to
replace the powerline corruption. In this example, a neighbourbandwidth of
1 would imply that the amplitude at 48-49 Hz and 51-52 Hz would be used to
"patch" the noise peak at 49-51 Hz.

You may have a more extreme scenario, but nevertheless looking at the
power spectrum will help you see how broad each powerline noise peak is
(use that to set "dftbandwidth"), and ensure that you pick clean neighbor
frequencies (guiding "neighbourbandwidth").

Our paper describing the method has an example of real data and some tips
on usage that might be helpful:
https://doi.org/10.1016/j.neuroimage.2019.01.026

Hope this helps!

Cheers,
Sarang


On Sun, 2021-10-03 at 14:45 +0200, Stephen Whitmarsh via fieldtrip wrote:

Dear Emilie,

Those sound indeed like challenging, and interesting, circumstances.
I don't have a direct answer to your question, that would depend on having
more information about the data, but here are my 2 cents;

You could first make an FFT plot (e.g. using cfg.method = mtmfft in
ft_freqanalysis) to get a better picture of the frequencies in your data
(on both 'successful' and 'unsuccessful' data), for several reasons:

  - You are also trying to remove both 50 and 60Hz, and their harmonics,
  of which probably only one of those would reflect the noise in the power
  lines.
  - You are using cfg.dftreplace = 'neighbour'; which might only work if
  the frequency band of the noise is sharp enough, or can be estimated
  precise enough (depending on the duration of the data you are filtering).
  - It would confirm whether it is line noise at all (which is probable,
  but more might be at play, e.g. some non-linear combinations and strange
  harmonics).
  - You could look for whether the degree of noise is stationary, by
  e.g. plotting the FFT over different periods, but it might be more
  important that the frequency is accurately chosen and stable over time.

Depending on your data, the duration of data, whether or not your are
looking at trials or the whole data file at once (*), etc., you might also
want to use a 'welch method' for estimating the power in the data, i.e. by
averaging over several (sliding) windows. Using cfg.method = 'mtmconvol' in
ft_freqanalysis, then averaging over time with ft_selectdata
(cfg.avgovertime), would be an easy way to do that. Once you have a good
picture of the noise, it will be easier to both choose the right filters
and to check their efficacy (by comparing the FFT before and after
filtering, on both 'successful' and 'unsuccessful' data). In fact, since
you are dealing with EEG in such noisy conditions, and assuming you are
interested in either ERPs or slow oscillations (<40Hz), you could consider
just using a lowpass filter at e.g. 40 Hz.

Good luck, I hope this helps,
Stephen

*) The behaviour of filters depends a lot on the structure/size of the
data, so to better understand its behaviour you would need to describe your
data(structures) in more detail, i.e. trial length, nr. of trials/channels,
and any processing steps until filtering.




Op zo 3 okt. 2021 om 13:12 schreef Emilie Caspar via fieldtrip <
fieldtrip at science.ru.nl>:

Dear Fieldtrippers,

I recently acquired EEG data in Rwanda in complicated testing conditions,
especially for the power line : we were testing in rural villages with a
single electrical system for the whole village. Our signal is thus
parasited by what the villagers were plugging on the electrical system (so
probably nonstationary noise) during the testing. The electrical system was
also very rudimentary and not correctly grounded.

Usually, I use the following command in Fieldtrip to remove the 50H or 60
Hz power line and it works quite well. Here, it worked for roughly 70% of
our acquired data, but it does not work for all the data acquired.

   cfg= [];
   cfg.dftfilter = 'yes';
   cfg.dftfreq = [50 60 100 120 150 180];
   cfg.dftreplace = 'neighbour';
   cfg.dftbandwidth = [2 2 2 2 2 2];
   cfg.dftneighbourwidth = [2 2 2 2 2 2];
   data_intpl = ft_preprocessing(cfg, allData_preprosses);


When I look at the graphs (see figure attached), it really looks like it’s
a 60Hz noise, but it seems that the dftfilter function does not remove it.
We are certain it’s power noise because when the electricity was cut off
because of a storm or else, and we were thus only relying on the batteries
to collect our data, the signal was perfect. So in theory we should be able
to remove it but we have no cue of what other possibilities to try. Perhaps
it’s because the noise is non stationary and dftfilter does not account for
that? I know some residual power noise can stay after a dftfiltre, but here
it does not remove anything.

Thanks a lot for the help,

Emilie

---------------------------------------------
*Prof. Dr. Emilie Caspar*

*Associate Professor *

Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94

lab’s website: https://moralsocialbrain.com/
personal website: https://emiliecaspar.home.blog/

Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32
95

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202


_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/678d1915/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PowerLine.jpg
Type: image/jpeg
Size: 37857 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/678d1915/attachment-0001.jpg>

------------------------------

Message: 3
Date: Mon, 4 Oct 2021 12:29:04 +0100
From: Alain de Cheveigne <alain.de.cheveigne at ens.psl.eu>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Removing power line
Message-ID: <3F456298-66A9-4DB4-9857-ED728A674554 at ens.psl.eu>
Content-Type: text/plain; charset=utf-8

Try ZapLine from the NoiseTools toolbox: http://audition.ens.fr/adc/NoiseTools/.

Alain

On 3 Oct 2021, at 12:03, Emilie Caspar via fieldtrip <fieldtrip at science.ru.nl> wrote:

Dear Fieldtrippers,

I recently acquired EEG data in Rwanda in complicated testing conditions, especially for the power line : we were testing in rural villages with a single electrical system for the whole village. Our signal is thus parasited by what the villagers were plugging on the electrical system (so probably nonstationary noise) during the testing. The electrical system was also very rudimentary and not correctly grounded.

Usually, I use the following command in Fieldtrip to remove the 50H or 60 Hz power line and it works quite well. Here, it worked for roughly 70% of our acquired data, but it does not work for all the data acquired.

  cfg= [];
  cfg.dftfilter = 'yes';
  cfg.dftfreq = [50 60 100 120 150 180];
  cfg.dftreplace = 'neighbour';
  cfg.dftbandwidth = [2 2 2 2 2 2];
  cfg.dftneighbourwidth = [2 2 2 2 2 2];
  data_intpl = ft_preprocessing(cfg, allData_preprosses);


When I look at the graphs (see figure attached), it really looks like it’s a 60Hz noise, but it seems that the dftfilter function does not remove it. We are certain it’s power noise because when the electricity was cut off because of a storm or else, and we were thus only relying on the batteries to collect our data, the signal was perfect. So in theory we should be able to remove it but we have no cue of what other possibilities to try. Perhaps it’s because the noise is non stationary and dftfilter does not account for that? I know some residual power noise can stay after a dftfiltre, but here it does not remove anything.

Thanks a lot for the help,

Emilie

<PowerLine.jpg>
---------------------------------------------
Prof. Dr. Emilie Caspar

Associate Professor

Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94

lab’s website: https://moralsocialbrain.com/
personal website: https://emiliecaspar.home.blog/


Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95


_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202



------------------------------

Message: 4
Date: Mon, 4 Oct 2021 13:46:42 +0200
From: Emilie Caspar <ecaspar at ulb.ac.be>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Removing power line
Message-ID: <E6C52CE1-11C6-4DCD-8E55-7A42A9D27A1D at ulb.ac.be>
Content-Type: text/plain; charset="utf-8"

Dear all,

Thank you all very much for your answers.

We have used a 64-electrode Biosemi system with left and right mastoids as references. The epochs lasts 1.5 s and we will look at averaged data, not single trial data. Regarding the preprocessing of the data, a filter from 0.1 to 40Hz was already applied as follows:

bpfilterOrder = 1; % ft default
bpfilterRange = [.1 40];

cfg = [];
   cfg.dataset    = [ file.name];
   cfg.bpfilter    = 'yes';
   cfg.bpfreq      = bpfilterRange;
   cfg.bpfiltord   = bpfilterOrder;
   cfg.reref       = 'yes';
   cfg.refchannel  = chansRef;
   allData_preprosses = ft_preprocessing(cfg);

Following your suggestions, I performed some additional graphs and analyses, which actually made me a bit confused.

I run different analyses with or without the spectrum interpolation (SI) on two participants whose data where acquired in similar testing conditions: P122 and P123 (graphs in attachement).
- For P122, we clearly see a peak at 50Hz without SI, but when I apply the usual SI at 50Hz with 100 and 150 as harmonics, the noise is clearly removed. This works for roughly 70% of the data.
- for P123, the pattern is a bit more complex. Again, without SI, we see a peak at 50Hz. But for this participant, when I apply the same SI as for P122 at 50Hz and 100 150 as harmonics, then the peak appears to have been increased. I made a second test with a SI for 50HZ and 60Hz, and now on the graph I have two peaks which appeared, one at 50Hz and one at 60hz. And actually the noise is higher in the data after the SI for this participant. This pattern is replicated for the remaining 30% of the data. I am not I understand why the SI works for some participants, but not for others by actually increasing the power noise.  Because when we compare P122 and P123 without SI, it looks like the 50Hz peak is the same between the two participants.

I also tried with broader neighbors for P123 (for instance, 5 instead of 2). But then the peak at 50Hz remains higher in amplitude and in addition becomes larger.

Regarding ZAPline, we started looking at it but it did not remove entirely the noise. But we are still learning to manipulate it to see if it can work with those data.

Thank you!

Emilie



---------------------------------------------
Emilie CASPAR

Associate Professor

Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94

lab’s website: https://moralsocialbrain.com/ <https://moralsocialbrain.com/>
personal website: https://emiliecaspar.home.blog/ <https://emiliecaspar.home.blog/>
Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95


Le 4 oct. 2021 à 12:43, Sarang S. Dalal via fieldtrip <fieldtrip at science.ru.nl> a écrit :

Hi Emilie,

I agree with Stephen -- absolutely, looking at the power spectrum will help guide your strategy here. It's normally unlikely to see both 50 Hz and 60 Hz in the same place... but I could imagine this may happen if the village gets power from a local electricity grid (likely 50 Hz) but with some buildings using potentially mismatched power inverters on solar panels or diesel generators (can be either 50 Hz or 60 Hz depending on manufacturer). Fluorescent lighting and perhaps some appliances may also generate noise frequencies that are totally unrelated to 50/60 Hz.

Just to clarify, cfg.dftreplace = 'neighbour' uses the spectrum interpolation method that Sabine Leske and I developed. It's actually designed for powerline noise frequencies that are *less* sharp than the default dftfilter method. So while it sounds like you have an extreme situation, it might indeed be an appropriate strategy for your data, depending on what your power spectrum actually looks like.

The design and test scenario was 50 Hz line noise in a typical home environment in Germany, which was actually a precise 50 Hz but of moderately fluctuating amplitude. These sorts of common fluctuations effectively broaden the noise frequency peak from a sharp 50 Hz to, for example, 49-51 Hz. For such a case, you would indeed define cfg.dftbandwidth of 2. The "neighbor" bandwidth is more tricky, but this should represent "clean" frequencies since their amplitude is used to replace the powerline corruption. In this example, a neighbourbandwidth of 1 would imply that the amplitude at 48-49 Hz and 51-52 Hz would be used to "patch" the noise peak at 49-51 Hz.

You may have a more extreme scenario, but nevertheless looking at the power spectrum will help you see how broad each powerline noise peak is (use that to set "dftbandwidth"), and ensure that you pick clean neighbor frequencies (guiding "neighbourbandwidth").

Our paper describing the method has an example of real data and some tips on usage that might be helpful:
https://doi.org/10.1016/j.neuroimage.2019.01.026 <https://doi.org/10.1016/j.neuroimage.2019.01.026>

Hope this helps!

Cheers,
Sarang


On Sun, 2021-10-03 at 14:45 +0200, Stephen Whitmarsh via fieldtrip wrote:
Dear Emilie,

Those sound indeed like challenging, and interesting, circumstances.
I don't have a direct answer to your question, that would depend on having more information about the data, but here are my 2 cents;

You could first make an FFT plot (e.g. using cfg.method = mtmfft in ft_freqanalysis) to get a better picture of the frequencies in your data (on both 'successful' and 'unsuccessful' data), for several reasons:
You are also trying to remove both 50 and 60Hz, and their harmonics, of which probably only one of those would reflect the noise in the power lines.
You are using cfg.dftreplace = 'neighbour'; which might only work if the frequency band of the noise is sharp enough, or can be estimated precise enough (depending on the duration of the data you are filtering).
It would confirm whether it is line noise at all (which is probable, but more might be at play, e.g. some non-linear combinations and strange harmonics).
You could look for whether the degree of noise is stationary, by e.g. plotting the FFT over different periods, but it might be more important that the frequency is accurately chosen and stable over time.
Depending on your data, the duration of data, whether or not your are looking at trials or the whole data file at once (*), etc., you might also want to use a 'welch method' for estimating the power in the data, i.e. by averaging over several (sliding) windows. Using cfg.method = 'mtmconvol' in ft_freqanalysis, then averaging over time with ft_selectdata (cfg.avgovertime), would be an easy way to do that. Once you have a good picture of the noise, it will be easier to both choose the right filters and to check their efficacy (by comparing the FFT before and after filtering, on both 'successful' and 'unsuccessful' data). In fact, since you are dealing with EEG in such noisy conditions, and assuming you are interested in either ERPs or slow oscillations (<40Hz), you could consider just using a lowpass filter at e.g. 40 Hz.

Good luck, I hope this helps,
Stephen

*) The behaviour of filters depends a lot on the structure/size of the data, so to better understand its behaviour you would need to describe your data(structures) in more detail, i.e. trial length, nr. of trials/channels, and any processing steps until filtering.




Op zo 3 okt. 2021 om 13:12 schreef Emilie Caspar via fieldtrip <fieldtrip at science.ru.nl <mailto:fieldtrip at science.ru.nl>>:
Dear Fieldtrippers,

I recently acquired EEG data in Rwanda in complicated testing conditions, especially for the power line : we were testing in rural villages with a single electrical system for the whole village. Our signal is thus parasited by what the villagers were plugging on the electrical system (so probably nonstationary noise) during the testing. The electrical system was also very rudimentary and not correctly grounded.

Usually, I use the following command in Fieldtrip to remove the 50H or 60 Hz power line and it works quite well. Here, it worked for roughly 70% of our acquired data, but it does not work for all the data acquired.

   cfg= [];
   cfg.dftfilter = 'yes';
   cfg.dftfreq = [50 60 100 120 150 180];
   cfg.dftreplace = 'neighbour';
   cfg.dftbandwidth = [2 2 2 2 2 2];
   cfg.dftneighbourwidth = [2 2 2 2 2 2];
   data_intpl = ft_preprocessing(cfg, allData_preprosses);


When I look at the graphs (see figure attached), it really looks like it’s a 60Hz noise, but it seems that the dftfilter function does not remove it. We are certain it’s power noise because when the electricity was cut off because of a storm or else, and we were thus only relying on the batteries to collect our data, the signal was perfect. So in theory we should be able to remove it but we have no cue of what other possibilities to try. Perhaps it’s because the noise is non stationary and dftfilter does not account for that? I know some residual power noise can stay after a dftfiltre, but here it does not remove anything.

Thanks a lot for the help,

Emilie

<PowerLine.jpg>
---------------------------------------------
Prof. Dr. Emilie Caspar

Associate Professor

Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94

lab’s website: https://moralsocialbrain.com/ <https://moralsocialbrain.com/>
personal website: https://emiliecaspar.home.blog/ <https://emiliecaspar.home.blog/>

Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95


_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip <https://mailman.science.ru.nl/mailman/listinfo/fieldtrip>
https://doi.org/10.1371/journal.pcbi.1002202 <https://doi.org/10.1371/journal.pcbi.1002202>

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip <https://mailman.science.ru.nl/mailman/listinfo/fieldtrip>
https://doi.org/10.1371/journal.pcbi.1002202 <https://doi.org/10.1371/journal.pcbi.1002202>

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/30b3ca57/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PowerLineNoise.001.jpeg
Type: image/jpeg
Size: 565683 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/30b3ca57/attachment.jpeg>

------------------------------

Subject: Digest Footer

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202


------------------------------

End of fieldtrip Digest, Vol 131, Issue 4
*****************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211004/224fde85/attachment.htm>

More information about the fieldtrip mailing list