[FieldTrip] Dealing with warning message "data contians NaN values"

Jeremy Pulmano jpulmano at princeton.edu
Fri Jul 19 14:48:02 CEST 2019 

Hi Jan-Mathijs,

I appreciate that. Thanks a lot!

Best,
Jeremy


On Fri, Jul 19, 2019 at 1:22 PM Schoffelen, J.M. (Jan Mathijs) <
jan.schoffelen at donders.ru.nl> wrote:

> Hi Jeremy,
>
> Consider yourself backed up by the entire fieldtrip community to tell your
> supervisors (or the powers that are, who told you to downsample first) that
> you are going to reverse the order of things, and first do all the
> filtering/epoching/etc. and only downsample afterwards. This will make your
> life much easier. I’d be happy to hear about good arguments why
> downsampling should be done first.
>
> Best wishes,
>
> Jan-Mathijs
>
> On 19 Jul 2019, at 13:07, Jeremy Pulmano <jpulmano at princeton.edu> wrote:
>
>
> Sadly, no that doesn't work. I am still getting approximately 1800 trials
> with an epoch length of 64  samples and 200 trials with an epoch length of
> 63. Additionally, I've run into extra complications because some trials
> after downsampling now have the same start and/or end sample indices; this
> raises error messages later on.
>
> I might just bite the bullet and epoch first then downsample. But if you
> have any other solutions I would love to try them!
>
> Thanks!
>
> On Fri, Jul 19, 2019 at 12:00 PM Stephen Whitmarsh <
> stephen.whitmarsh at gmail.com> wrote:
>
>> Hi Jeremy,
>>
>> A quick reply on your first question during lunch: round(trl/8+0.5)?
>>
>> Cheers,
>> s
>>
>> On Fri, 19 Jul 2019, 12:35 Jeremy Pulmano, <jpulmano at princeton.edu>
>> wrote:
>>
>>> Hi Stephen,
>>>
>>> Thank you, this is very helpful! However, I am encountering an issue
>>> when attempting to divide the cfg.trl by the same ratio (1024/128 = 8) and
>>> rounding them off as integers; this results in different sized epochs
>>> across trials. In my case, some are 63 samples in length and others are 64
>>> samples in length because of the rounding. Any ideas on how to avoid this?
>>> I was thinking that I can simply add one sample to those that become 63 in
>>> length but I'm worried that it will skew the data.
>>>
>>> In terms of the trial function, I have been doing some experimenting
>>> upon writing my own, but unfortunately, I've encountered a few roadblocks.
>>> First, the function ft_read_event only works with the original dataset and
>>> therefore only identifies the epochs in the *raw* *data*. Thus, I
>>> cannot seem to find a way to identify the triggers in the downsampled data.
>>>
>>> In terms of your second point, yes it is definitely possible to resample
>>> after epoching. In my case, however, I have been instructed that
>>> downsampling *before* epoching is the best practice for the dataset I
>>> am working with.
>>>
>>> Any further suggestions? Thanks again for your help.
>>> J
>>>
>>> On Thu, Jul 18, 2019 at 4:54 PM Stephen Whitmarsh <
>>> stephen.whitmarsh at gmail.com> wrote:
>>>
>>>> Hi Jeremy,
>>>>
>>>> Yes, it seems likely that that would be the problem.
>>>>
>>>> I see a couple of solutions:
>>>>
>>>> 1) Take your cfg.trl (trialdefinition which contains the start, end and
>>>> offset sample) which you get out of ft_definetrial, and divide them with
>>>> the same ratio as you did your data (and round them off to integers).
>>>>
>>>> 2) Resample your trial data after epoching. Honestly not 100% sure this
>>>> is supported, but only one way to find out :-)
>>>>
>>>> 3a) Make your own trialfun*, i.e. function to create your own trl
>>>> (trial definition), and that uses the appropriate samplerate (to be found
>>>> in the datastructure field .fsample). An example on making your own
>>>> trialfun can be found here:
>>>> http://www.fieldtriptoolbox.org/tutorial/preprocessing/. You can also
>>>> check http://www.fieldtriptoolbox.org/walkthrough/
>>>>
>>>> 3b) Use your fancy corrected trl in combination with ft_redefinetrial
>>>> to epoch your resampled data.
>>>>
>>>> *) IMHO, creating your own trialfun pays off in the long run, as it
>>>> allows you to be much flexible, and e.g. enter response times and those
>>>> kind of things in extra columns in the .trl, which will then be carried
>>>> trhough your data in a .trialinfo field.
>>>>
>>>> Good fieldtripping!
>>>> Stephen
>>>>
>>>>
>>>>
>>>> On Thu, 18 Jul 2019 at 16:59, Jeremy Pulmano <jpulmano at princeton.edu>
>>>> wrote:
>>>>
>>>>> Hi Stephen,
>>>>>
>>>>> Thanks so much for your reply. Apologies for the lack of specificity;
>>>>> what I meant to say is that out of 2000 trials, around 1500 of them contain
>>>>> NaNs after epoching. Thus, when I try visual artifact rejection, I only see
>>>>> 500 trials and the other 1500 are omitted completely.
>>>>>
>>>>> The NaNs appear after epoching. I think you're correct in identifying
>>>>> a mistake with the sampling rates. How can I specify the new sampling
>>>>> frequency in the trial definition? I think there might also be an issue
>>>>> because FieldTrip reads the old sampling frequency from the headerfile (I
>>>>> am using Biosemi .bdf files).
>>>>>
>>>>> Here is a snippet of the code. I am also attaching it to this email
>>>>> for easier reference. Thanks again.
>>>>>
>>>>> %% Setup code (abridged)
>>>>>>
>>>>>> filename = 'example';
>>>>>> header = ft_read_header(filename);
>>>>>> new_fs = 128;
>>>>>> prestim = 0.1;
>>>>>> poststim = 0.4;
>>>>>> trigger_codes = [ 1 2 3 ];
>>>>>>
>>>>>> %% High pass filter
>>>>>>
>>>>>> % Define trial for all the continuous data
>>>>>> cfg = [];
>>>>>> cfg.dataset = filename;
>>>>>> cfg.headerfile = header;
>>>>>> cfg.trialdef.triallength = Inf;
>>>>>> cfg.trialdef.ntrials = 1;
>>>>>>
>>>>>> cfg = ft_definetrial(cfg);
>>>>>>
>>>>>> % Preprocess with high pass filter
>>>>>> order = 100;
>>>>>> cfg.hpfilter = 'yes';
>>>>>> cfg.hpfilttype = 'fir';
>>>>>> cfg.hpfreq = 0.1;
>>>>>> cfg.hpfiltord = order;
>>>>>>
>>>>>> data = ft_preprocessing(cfg);
>>>>>> data_hp = data;
>>>>>>
>>>>>> %% Downsampling
>>>>>>
>>>>>> fprintf('\nDownsampling\n');
>>>>>>
>>>>>> cfg = [];
>>>>>> cfg.resamplefs = new_fs;
>>>>>>
>>>>>> data = ft_resampledata(cfg, data);
>>>>>> data_res = data;
>>>>>>
>>>>>> %% Epoch (NaNs appear after this block)
>>>>>>
>>>>>> % Now define the ACTUAL trial
>>>>>> cfg = [];
>>>>>> cfg.continuous = 'yes';
>>>>>> cfg.dataset = filename;
>>>>>> cfg.trialdef.prestim = prestim;
>>>>>> cfg.trialdef.poststim = poststim;
>>>>>> cfg.trialdef.eventtype = 'STATUS';
>>>>>> cfg.trialdef.eventvalue = trigger_codes
>>>>>>
>>>>>> cfg = ft_definetrial(cfg);
>>>>>>
>>>>>> data = ft_redefinetrial(cfg, data);
>>>>>> data_ep = data;
>>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 18, 2019 at 3:06 PM Stephen Whitmarsh <
>>>>> stephen.whitmarsh at gmail.com> wrote:
>>>>>
>>>>>> Dear Jeremy,
>>>>>>
>>>>>> Welcome to the FieldTrip community!
>>>>>>
>>>>>> Could you be more specific about "the data looks incorrect in the
>>>>>> end.", and perhaps share a screenshot if informative? Sharing the relevant
>>>>>> part of your script would be helpful too. For more info about how to get
>>>>>> the best response to your questions, please take a look at:
>>>>>> http://www.fieldtriptoolbox.org/faq/how_to_ask_good_questions_to_the_community/
>>>>>>
>>>>>> Have you checked where the NaN's in the data are, and whether they
>>>>>> appear before or after downsampling?
>>>>>>
>>>>>> Just an shot in the dark, but have you perhaps defined your trials
>>>>>> for epoching (i.e. the samples in the .trl) based on the old samplerate?
>>>>>>
>>>>>> Cheers,
>>>>>> Stephen
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 18 Jul 2019 at 12:35, Jeremy Pulmano <jpulmano at princeton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've realized that the warning message often comes right after
>>>>>>> epoching (using ft_definetrial and then ft_redefinetrial). It seems like it
>>>>>>> successfully breaks the data into segments but still contains NaNs for
>>>>>>> whatever reason. Here are more details about the pipeline so far:
>>>>>>>
>>>>>>>    1. High pass filtering (0.1 Hz / FIR / order = 100)
>>>>>>>    2. Downsampling (1024 to 128 Hz)
>>>>>>>    3. Epoching (prestim 0.1, poststim 0.4)
>>>>>>>    4. Base-line correction [ -0.1 0 ]
>>>>>>>    5. Re-referencing (common avg.)
>>>>>>>    6. ICA / component rejection
>>>>>>>    7. Visual artifact removal
>>>>>>>    8. Low pass filter (30 Hz)
>>>>>>>
>>>>>>> Any further help would be greatly appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jeremy
>>>>>>>
>>>>>>> On Wed, Jul 17, 2019 at 4:57 PM Jeremy Pulmano <
>>>>>>> jpulmano at princeton.edu> wrote:
>>>>>>>
>>>>>>>> Hi Erika,
>>>>>>>>
>>>>>>>> The original sampling rate is 1024 Hz and the new sampling rate is
>>>>>>>> 128 Hz.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> J
>>>>>>>>
>>>>>>>> On Wed, Jul 17, 2019 at 4:49 PM Erika Puiutta <
>>>>>>>> erika.puiutta at uni-oldenburg.de> wrote:
>>>>>>>>
>>>>>>>>> Hey J,
>>>>>>>>>
>>>>>>>>> this is just an educated guess since I don't really know how
>>>>>>>>> fieldtrip does the downsampling, but what is your sampling frequency and
>>>>>>>>> the sampling frequency you sample down to? Is it possible that your
>>>>>>>>> downsampling frequency is not a harmonic of your original sampling
>>>>>>>>> frequency (say 1000Hz to 300Hz)? Maybe there are NaNs for the new samples
>>>>>>>>> where fieldtrip can't find a corresponding one from the old sampling
>>>>>>>>> frequency?
>>>>>>>>>
>>>>>>>>> Best of luck,
>>>>>>>>> Erika
>>>>>>>>>
>>>>>>>>> Am 17.07.2019 um 17:26 schrieb Jeremy Pulmano <
>>>>>>>>> jpulmano at princeton.edu>:
>>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I am new to both neuroscience and FieldTrip and am trying to debug
>>>>>>>>> my preprocessing pipeline of EEG data (biosemi .bdf files). These are the
>>>>>>>>> current steps:
>>>>>>>>>
>>>>>>>>>    1. High pass filtering
>>>>>>>>>    2. Downsampling
>>>>>>>>>    3. Epoching
>>>>>>>>>    4. Base-line correction
>>>>>>>>>    5. Re-referencing
>>>>>>>>>    6. ICA / component rejection
>>>>>>>>>    7. Visual artifact removal
>>>>>>>>>    8. Low pass filter (I save the low pass filter for the end so
>>>>>>>>>    that ICA yields better time course plots).
>>>>>>>>>
>>>>>>>>> However, after the first three steps (high pass, downsample,
>>>>>>>>> epoch), I continue to get the warning message: "Warning: data contains NaN
>>>>>>>>> values, no filtering or preprocessing applied," starting at baseline
>>>>>>>>> correction. It repeats itself over and over again. If I ignore these
>>>>>>>>> errors, the data looks incorrect in the end. Why does this happen, and how
>>>>>>>>> can I resolve this?
>>>>>>>>>
>>>>>>>>> Any tips would be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> J
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> fieldtrip mailing list
>>>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> fieldtrip mailing list
>>>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Jeremy Pulmano*
>>>>>>>> *Princeton University Class of 2021*
>>>>>>>> B.S.E. Computer Science
>>>>>>>> e: jpulmano at princeton.edu | c: (862) 220-5903
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> fieldtrip mailing list
>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>
>>>>>> _______________________________________________
>>>>>> fieldtrip mailing list
>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>
>>>>> _______________________________________________
>>>>> fieldtrip mailing list
>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>
>>>> _______________________________________________
>>>> fieldtrip mailing list
>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>
>>> _______________________________________________
>>> fieldtrip mailing list
>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>
>> _______________________________________________
>> fieldtrip mailing list
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>> https://doi.org/10.1371/journal.pcbi.1002202
>>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190719/b3a7b77c/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

More information about the fieldtrip mailing list