[FieldTrip] Dealing with warning message "data contians NaN values"

Schoffelen, J.M. (Jan Mathijs) jan.schoffelen at donders.ru.nl
Fri Jul 19 14:14:59 CEST 2019 

Hi Jeremy,

Consider yourself backed up by the entire fieldtrip community to tell your supervisors (or the powers that are, who told you to downsample first) that you are going to reverse the order of things, and first do all the filtering/epoching/etc. and only downsample afterwards. This will make your life much easier. I’d be happy to hear about good arguments why downsampling should be done first.

Best wishes,

Jan-Mathijs

On 19 Jul 2019, at 13:07, Jeremy Pulmano <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>> wrote:


Sadly, no that doesn't work. I am still getting approximately 1800 trials with an epoch length of 64  samples and 200 trials with an epoch length of 63. Additionally, I've run into extra complications because some trials after downsampling now have the same start and/or end sample indices; this raises error messages later on.

I might just bite the bullet and epoch first then downsample. But if you have any other solutions I would love to try them!

Thanks!

On Fri, Jul 19, 2019 at 12:00 PM Stephen Whitmarsh <stephen.whitmarsh at gmail.com<mailto:stephen.whitmarsh at gmail.com>> wrote:
Hi Jeremy,

A quick reply on your first question during lunch: round(trl/8+0.5)?

Cheers,
s

On Fri, 19 Jul 2019, 12:35 Jeremy Pulmano, <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>> wrote:
Hi Stephen,

Thank you, this is very helpful! However, I am encountering an issue when attempting to divide the cfg.trl by the same ratio (1024/128 = 8) and rounding them off as integers; this results in different sized epochs across trials. In my case, some are 63 samples in length and others are 64 samples in length because of the rounding. Any ideas on how to avoid this? I was thinking that I can simply add one sample to those that become 63 in length but I'm worried that it will skew the data.

In terms of the trial function, I have been doing some experimenting upon writing my own, but unfortunately, I've encountered a few roadblocks. First, the function ft_read_event only works with the original dataset and therefore only identifies the epochs in the raw data. Thus, I cannot seem to find a way to identify the triggers in the downsampled data.

In terms of your second point, yes it is definitely possible to resample after epoching. In my case, however, I have been instructed that downsampling before epoching is the best practice for the dataset I am working with.

Any further suggestions? Thanks again for your help.
J

On Thu, Jul 18, 2019 at 4:54 PM Stephen Whitmarsh <stephen.whitmarsh at gmail.com<mailto:stephen.whitmarsh at gmail.com>> wrote:
Hi Jeremy,

Yes, it seems likely that that would be the problem.

I see a couple of solutions:

1) Take your cfg.trl (trialdefinition which contains the start, end and offset sample) which you get out of ft_definetrial, and divide them with the same ratio as you did your data (and round them off to integers).

2) Resample your trial data after epoching. Honestly not 100% sure this is supported, but only one way to find out :-)

3a) Make your own trialfun*, i.e. function to create your own trl (trial definition), and that uses the appropriate samplerate (to be found in the datastructure field .fsample). An example on making your own trialfun can be found here: http://www.fieldtriptoolbox.org/tutorial/preprocessing/. You can also check http://www.fieldtriptoolbox.org/walkthrough/

3b) Use your fancy corrected trl in combination with ft_redefinetrial to epoch your resampled data.

*) IMHO, creating your own trialfun pays off in the long run, as it allows you to be much flexible, and e.g. enter response times and those kind of things in extra columns in the .trl, which will then be carried trhough your data in a .trialinfo field.

Good fieldtripping!
Stephen



On Thu, 18 Jul 2019 at 16:59, Jeremy Pulmano <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>> wrote:
Hi Stephen,

Thanks so much for your reply. Apologies for the lack of specificity; what I meant to say is that out of 2000 trials, around 1500 of them contain NaNs after epoching. Thus, when I try visual artifact rejection, I only see 500 trials and the other 1500 are omitted completely.

The NaNs appear after epoching. I think you're correct in identifying a mistake with the sampling rates. How can I specify the new sampling frequency in the trial definition? I think there might also be an issue because FieldTrip reads the old sampling frequency from the headerfile (I am using Biosemi .bdf files).

Here is a snippet of the code. I am also attaching it to this email for easier reference. Thanks again.

%% Setup code (abridged)

filename = 'example';
header = ft_read_header(filename);
new_fs = 128;
prestim = 0.1;
poststim = 0.4;
trigger_codes = [ 1 2 3 ];

%% High pass filter

% Define trial for all the continuous data
cfg = [];
cfg.dataset = filename;
cfg.headerfile = header;
cfg.trialdef.triallength = Inf;
cfg.trialdef.ntrials = 1;

cfg = ft_definetrial(cfg);

% Preprocess with high pass filter
order = 100;
cfg.hpfilter = 'yes';
cfg.hpfilttype = 'fir';
cfg.hpfreq = 0.1;
cfg.hpfiltord = order;

data = ft_preprocessing(cfg);
data_hp = data;

%% Downsampling

fprintf('\nDownsampling\n');

cfg = [];
cfg.resamplefs = new_fs;

data = ft_resampledata(cfg, data);
data_res = data;

%% Epoch (NaNs appear after this block)

% Now define the ACTUAL trial
cfg = [];
cfg.continuous = 'yes';
cfg.dataset = filename;
cfg.trialdef.prestim = prestim;
cfg.trialdef.poststim = poststim;
cfg.trialdef.eventtype = 'STATUS';
cfg.trialdef.eventvalue = trigger_codes

cfg = ft_definetrial(cfg);

data = ft_redefinetrial(cfg, data);
data_ep = data;


On Thu, Jul 18, 2019 at 3:06 PM Stephen Whitmarsh <stephen.whitmarsh at gmail.com<mailto:stephen.whitmarsh at gmail.com>> wrote:
Dear Jeremy,

Welcome to the FieldTrip community!

Could you be more specific about "the data looks incorrect in the end.", and perhaps share a screenshot if informative? Sharing the relevant part of your script would be helpful too. For more info about how to get the best response to your questions, please take a look at: http://www.fieldtriptoolbox.org/faq/how_to_ask_good_questions_to_the_community/

Have you checked where the NaN's in the data are, and whether they appear before or after downsampling?

Just an shot in the dark, but have you perhaps defined your trials for epoching (i.e. the samples in the .trl) based on the old samplerate?

Cheers,
Stephen





On Thu, 18 Jul 2019 at 12:35, Jeremy Pulmano <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>> wrote:
Hi,

I've realized that the warning message often comes right after epoching (using ft_definetrial and then ft_redefinetrial). It seems like it successfully breaks the data into segments but still contains NaNs for whatever reason. Here are more details about the pipeline so far:

  1.  High pass filtering (0.1 Hz / FIR / order = 100)
  2.  Downsampling (1024 to 128 Hz)
  3.  Epoching (prestim 0.1, poststim 0.4)
  4.  Base-line correction [ -0.1 0 ]
  5.  Re-referencing (common avg.)
  6.  ICA / component rejection
  7.  Visual artifact removal
  8.  Low pass filter (30 Hz)

Any further help would be greatly appreciated.

Thanks,
Jeremy

On Wed, Jul 17, 2019 at 4:57 PM Jeremy Pulmano <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>> wrote:
Hi Erika,

The original sampling rate is 1024 Hz and the new sampling rate is 128 Hz.

Thanks,
J

On Wed, Jul 17, 2019 at 4:49 PM Erika Puiutta <erika.puiutta at uni-oldenburg.de<mailto:erika.puiutta at uni-oldenburg.de>> wrote:
Hey J,

this is just an educated guess since I don't really know how fieldtrip does the downsampling, but what is your sampling frequency and the sampling frequency you sample down to? Is it possible that your downsampling frequency is not a harmonic of your original sampling frequency (say 1000Hz to 300Hz)? Maybe there are NaNs for the new samples where fieldtrip can't find a corresponding one from the old sampling frequency?

Best of luck,
Erika

Am 17.07.2019 um 17:26 schrieb Jeremy Pulmano <jpulmano at princeton.edu<mailto:jpulmano at princeton.edu>>:

Hi all,

I am new to both neuroscience and FieldTrip and am trying to debug my preprocessing pipeline of EEG data (biosemi .bdf files). These are the current steps:

  1.  High pass filtering
  2.  Downsampling
  3.  Epoching
  4.  Base-line correction
  5.  Re-referencing
  6.  ICA / component rejection
  7.  Visual artifact removal
  8.  Low pass filter (I save the low pass filter for the end so that ICA yields better time course plots).

However, after the first three steps (high pass, downsample, epoch), I continue to get the warning message: "Warning: data contains NaN values, no filtering or preprocessing applied," starting at baseline correction. It repeats itself over and over again. If I ignore these errors, the data looks incorrect in the end. Why does this happen, and how can I resolve this?

Any tips would be appreciated.

Thanks,
J
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
--

[https://docs.google.com/uc?export=download&id=1-cTSY4Ow0kUKbhb45irGhGCGli8wFwgZ&revid=0B4aqG7IAPQqtWVpyeU8vUFBIS1A3RzdaeUh2NzBncThkVFdrPQ]     Jeremy Pulmano
Princeton University Class of 2021
B.S.E. Computer Science
e: jpulmano at princeton.edu<mailto:jpulmano at princeton.edu> | c: (862) 220-5903
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190719/8bfb314b/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

More information about the fieldtrip mailing list