[FieldTrip] Dealing with warning message "data contians NaN values"

Stephen Whitmarsh stephen.whitmarsh at gmail.com
Fri Jul 19 12:53:32 CEST 2019 

Hi Jeremy,

A quick reply on your first question during lunch: round(trl/8+0.5)?

Cheers,
s

On Fri, 19 Jul 2019, 12:35 Jeremy Pulmano, <jpulmano at princeton.edu> wrote:

> Hi Stephen,
>
> Thank you, this is very helpful! However, I am encountering an issue when
> attempting to divide the cfg.trl by the same ratio (1024/128 = 8) and
> rounding them off as integers; this results in different sized epochs
> across trials. In my case, some are 63 samples in length and others are 64
> samples in length because of the rounding. Any ideas on how to avoid this?
> I was thinking that I can simply add one sample to those that become 63 in
> length but I'm worried that it will skew the data.
>
> In terms of the trial function, I have been doing some experimenting upon
> writing my own, but unfortunately, I've encountered a few roadblocks.
> First, the function ft_read_event only works with the original dataset and
> therefore only identifies the epochs in the *raw* *data*. Thus, I cannot
> seem to find a way to identify the triggers in the downsampled data.
>
> In terms of your second point, yes it is definitely possible to resample
> after epoching. In my case, however, I have been instructed that
> downsampling *before* epoching is the best practice for the dataset I am
> working with.
>
> Any further suggestions? Thanks again for your help.
> J
>
> On Thu, Jul 18, 2019 at 4:54 PM Stephen Whitmarsh <
> stephen.whitmarsh at gmail.com> wrote:
>
>> Hi Jeremy,
>>
>> Yes, it seems likely that that would be the problem.
>>
>> I see a couple of solutions:
>>
>> 1) Take your cfg.trl (trialdefinition which contains the start, end and
>> offset sample) which you get out of ft_definetrial, and divide them with
>> the same ratio as you did your data (and round them off to integers).
>>
>> 2) Resample your trial data after epoching. Honestly not 100% sure this
>> is supported, but only one way to find out :-)
>>
>> 3a) Make your own trialfun*, i.e. function to create your own trl (trial
>> definition), and that uses the appropriate samplerate (to be found in the
>> datastructure field .fsample). An example on making your own trialfun can
>> be found here: http://www.fieldtriptoolbox.org/tutorial/preprocessing/.
>> You can also check http://www.fieldtriptoolbox.org/walkthrough/
>>
>> 3b) Use your fancy corrected trl in combination with ft_redefinetrial to
>> epoch your resampled data.
>>
>> *) IMHO, creating your own trialfun pays off in the long run, as it
>> allows you to be much flexible, and e.g. enter response times and those
>> kind of things in extra columns in the .trl, which will then be carried
>> trhough your data in a .trialinfo field.
>>
>> Good fieldtripping!
>> Stephen
>>
>>
>>
>> On Thu, 18 Jul 2019 at 16:59, Jeremy Pulmano <jpulmano at princeton.edu>
>> wrote:
>>
>>> Hi Stephen,
>>>
>>> Thanks so much for your reply. Apologies for the lack of specificity;
>>> what I meant to say is that out of 2000 trials, around 1500 of them contain
>>> NaNs after epoching. Thus, when I try visual artifact rejection, I only see
>>> 500 trials and the other 1500 are omitted completely.
>>>
>>> The NaNs appear after epoching. I think you're correct in identifying a
>>> mistake with the sampling rates. How can I specify the new sampling
>>> frequency in the trial definition? I think there might also be an issue
>>> because FieldTrip reads the old sampling frequency from the headerfile (I
>>> am using Biosemi .bdf files).
>>>
>>> Here is a snippet of the code. I am also attaching it to this email for
>>> easier reference. Thanks again.
>>>
>>> %% Setup code (abridged)
>>>>
>>>> filename = 'example';
>>>> header = ft_read_header(filename);
>>>> new_fs = 128;
>>>> prestim = 0.1;
>>>> poststim = 0.4;
>>>> trigger_codes = [ 1 2 3 ];
>>>>
>>>> %% High pass filter
>>>>
>>>> % Define trial for all the continuous data
>>>> cfg = [];
>>>> cfg.dataset = filename;
>>>> cfg.headerfile = header;
>>>> cfg.trialdef.triallength = Inf;
>>>> cfg.trialdef.ntrials = 1;
>>>>
>>>> cfg = ft_definetrial(cfg);
>>>>
>>>> % Preprocess with high pass filter
>>>> order = 100;
>>>> cfg.hpfilter = 'yes';
>>>> cfg.hpfilttype = 'fir';
>>>> cfg.hpfreq = 0.1;
>>>> cfg.hpfiltord = order;
>>>>
>>>> data = ft_preprocessing(cfg);
>>>> data_hp = data;
>>>>
>>>> %% Downsampling
>>>>
>>>> fprintf('\nDownsampling\n');
>>>>
>>>> cfg = [];
>>>> cfg.resamplefs = new_fs;
>>>>
>>>> data = ft_resampledata(cfg, data);
>>>> data_res = data;
>>>>
>>>> %% Epoch (NaNs appear after this block)
>>>>
>>>> % Now define the ACTUAL trial
>>>> cfg = [];
>>>> cfg.continuous = 'yes';
>>>> cfg.dataset = filename;
>>>> cfg.trialdef.prestim = prestim;
>>>> cfg.trialdef.poststim = poststim;
>>>> cfg.trialdef.eventtype = 'STATUS';
>>>> cfg.trialdef.eventvalue = trigger_codes
>>>>
>>>> cfg = ft_definetrial(cfg);
>>>>
>>>> data = ft_redefinetrial(cfg, data);
>>>> data_ep = data;
>>>>
>>>
>>>
>>> On Thu, Jul 18, 2019 at 3:06 PM Stephen Whitmarsh <
>>> stephen.whitmarsh at gmail.com> wrote:
>>>
>>>> Dear Jeremy,
>>>>
>>>> Welcome to the FieldTrip community!
>>>>
>>>> Could you be more specific about "the data looks incorrect in the
>>>> end.", and perhaps share a screenshot if informative? Sharing the relevant
>>>> part of your script would be helpful too. For more info about how to get
>>>> the best response to your questions, please take a look at:
>>>> http://www.fieldtriptoolbox.org/faq/how_to_ask_good_questions_to_the_community/
>>>>
>>>> Have you checked where the NaN's in the data are, and whether they
>>>> appear before or after downsampling?
>>>>
>>>> Just an shot in the dark, but have you perhaps defined your trials for
>>>> epoching (i.e. the samples in the .trl) based on the old samplerate?
>>>>
>>>> Cheers,
>>>> Stephen
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 18 Jul 2019 at 12:35, Jeremy Pulmano <jpulmano at princeton.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've realized that the warning message often comes right after
>>>>> epoching (using ft_definetrial and then ft_redefinetrial). It seems like it
>>>>> successfully breaks the data into segments but still contains NaNs for
>>>>> whatever reason. Here are more details about the pipeline so far:
>>>>>
>>>>>    1. High pass filtering (0.1 Hz / FIR / order = 100)
>>>>>    2. Downsampling (1024 to 128 Hz)
>>>>>    3. Epoching (prestim 0.1, poststim 0.4)
>>>>>    4. Base-line correction [ -0.1 0 ]
>>>>>    5. Re-referencing (common avg.)
>>>>>    6. ICA / component rejection
>>>>>    7. Visual artifact removal
>>>>>    8. Low pass filter (30 Hz)
>>>>>
>>>>> Any further help would be greatly appreciated.
>>>>>
>>>>> Thanks,
>>>>> Jeremy
>>>>>
>>>>> On Wed, Jul 17, 2019 at 4:57 PM Jeremy Pulmano <jpulmano at princeton.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Erika,
>>>>>>
>>>>>> The original sampling rate is 1024 Hz and the new sampling rate is
>>>>>> 128 Hz.
>>>>>>
>>>>>> Thanks,
>>>>>> J
>>>>>>
>>>>>> On Wed, Jul 17, 2019 at 4:49 PM Erika Puiutta <
>>>>>> erika.puiutta at uni-oldenburg.de> wrote:
>>>>>>
>>>>>>> Hey J,
>>>>>>>
>>>>>>> this is just an educated guess since I don't really know how
>>>>>>> fieldtrip does the downsampling, but what is your sampling frequency and
>>>>>>> the sampling frequency you sample down to? Is it possible that your
>>>>>>> downsampling frequency is not a harmonic of your original sampling
>>>>>>> frequency (say 1000Hz to 300Hz)? Maybe there are NaNs for the new samples
>>>>>>> where fieldtrip can't find a corresponding one from the old sampling
>>>>>>> frequency?
>>>>>>>
>>>>>>> Best of luck,
>>>>>>> Erika
>>>>>>>
>>>>>>> Am 17.07.2019 um 17:26 schrieb Jeremy Pulmano <
>>>>>>> jpulmano at princeton.edu>:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am new to both neuroscience and FieldTrip and am trying to debug
>>>>>>> my preprocessing pipeline of EEG data (biosemi .bdf files). These are the
>>>>>>> current steps:
>>>>>>>
>>>>>>>    1. High pass filtering
>>>>>>>    2. Downsampling
>>>>>>>    3. Epoching
>>>>>>>    4. Base-line correction
>>>>>>>    5. Re-referencing
>>>>>>>    6. ICA / component rejection
>>>>>>>    7. Visual artifact removal
>>>>>>>    8. Low pass filter (I save the low pass filter for the end so
>>>>>>>    that ICA yields better time course plots).
>>>>>>>
>>>>>>> However, after the first three steps (high pass, downsample, epoch),
>>>>>>> I continue to get the warning message: "Warning: data contains NaN values,
>>>>>>> no filtering or preprocessing applied," starting at baseline correction. It
>>>>>>> repeats itself over and over again. If I ignore these errors, the data
>>>>>>> looks incorrect in the end. Why does this happen, and how can I resolve
>>>>>>> this?
>>>>>>>
>>>>>>> Any tips would be appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> J
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> fieldtrip mailing list
>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> fieldtrip mailing list
>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Jeremy Pulmano*
>>>>>> *Princeton University Class of 2021*
>>>>>> B.S.E. Computer Science
>>>>>> e: jpulmano at princeton.edu | c: (862) 220-5903
>>>>>>
>>>>> _______________________________________________
>>>>> fieldtrip mailing list
>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>
>>>> _______________________________________________
>>>> fieldtrip mailing list
>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>
>>> _______________________________________________
>>> fieldtrip mailing list
>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>
>> _______________________________________________
>> fieldtrip mailing list
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>> https://doi.org/10.1371/journal.pcbi.1002202
>>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190719/2b154dca/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

More information about the fieldtrip mailing list