[FieldTrip] Dealing with warning message "data contians NaN values"
Stephen Whitmarsh
stephen.whitmarsh at gmail.com
Fri Jul 19 15:28:58 CEST 2019
Hi Jeremy,
Any other solution I can come up with would require you to code procedures
yourself, procedures that have already been taken care off by FieldTrip.
I second Lieutenant Colonel Schoffelen that the way of the least resistance
(time, energy, work) is to submit to our greater collective power, let
FieldTrip take care of the horrible administration of trial bookkeeping,
and run free through the fields of analyses in everlasting joy.
Cheers,
Stephen
On Fri, 19 Jul 2019 at 14:42, Schoffelen, J.M. (Jan Mathijs) <
jan.schoffelen at donders.ru.nl> wrote:
> Hi Jeremy,
>
> Consider yourself backed up by the entire fieldtrip community to tell your
> supervisors (or the powers that are, who told you to downsample first) that
> you are going to reverse the order of things, and first do all the
> filtering/epoching/etc. and only downsample afterwards. This will make your
> life much easier. I’d be happy to hear about good arguments why
> downsampling should be done first.
>
> Best wishes,
>
> Jan-Mathijs
>
> On 19 Jul 2019, at 13:07, Jeremy Pulmano <jpulmano at princeton.edu> wrote:
>
>
> Sadly, no that doesn't work. I am still getting approximately 1800 trials
> with an epoch length of 64 samples and 200 trials with an epoch length of
> 63. Additionally, I've run into extra complications because some trials
> after downsampling now have the same start and/or end sample indices; this
> raises error messages later on.
>
> I might just bite the bullet and epoch first then downsample. But if you
> have any other solutions I would love to try them!
>
> Thanks!
>
> On Fri, Jul 19, 2019 at 12:00 PM Stephen Whitmarsh <
> stephen.whitmarsh at gmail.com> wrote:
>
>> Hi Jeremy,
>>
>> A quick reply on your first question during lunch: round(trl/8+0.5)?
>>
>> Cheers,
>> s
>>
>> On Fri, 19 Jul 2019, 12:35 Jeremy Pulmano, <jpulmano at princeton.edu>
>> wrote:
>>
>>> Hi Stephen,
>>>
>>> Thank you, this is very helpful! However, I am encountering an issue
>>> when attempting to divide the cfg.trl by the same ratio (1024/128 = 8) and
>>> rounding them off as integers; this results in different sized epochs
>>> across trials. In my case, some are 63 samples in length and others are 64
>>> samples in length because of the rounding. Any ideas on how to avoid this?
>>> I was thinking that I can simply add one sample to those that become 63 in
>>> length but I'm worried that it will skew the data.
>>>
>>> In terms of the trial function, I have been doing some experimenting
>>> upon writing my own, but unfortunately, I've encountered a few roadblocks.
>>> First, the function ft_read_event only works with the original dataset and
>>> therefore only identifies the epochs in the *raw* *data*. Thus, I
>>> cannot seem to find a way to identify the triggers in the downsampled data.
>>>
>>> In terms of your second point, yes it is definitely possible to resample
>>> after epoching. In my case, however, I have been instructed that
>>> downsampling *before* epoching is the best practice for the dataset I
>>> am working with.
>>>
>>> Any further suggestions? Thanks again for your help.
>>> J
>>>
>>> On Thu, Jul 18, 2019 at 4:54 PM Stephen Whitmarsh <
>>> stephen.whitmarsh at gmail.com> wrote:
>>>
>>>> Hi Jeremy,
>>>>
>>>> Yes, it seems likely that that would be the problem.
>>>>
>>>> I see a couple of solutions:
>>>>
>>>> 1) Take your cfg.trl (trialdefinition which contains the start, end and
>>>> offset sample) which you get out of ft_definetrial, and divide them with
>>>> the same ratio as you did your data (and round them off to integers).
>>>>
>>>> 2) Resample your trial data after epoching. Honestly not 100% sure this
>>>> is supported, but only one way to find out :-)
>>>>
>>>> 3a) Make your own trialfun*, i.e. function to create your own trl
>>>> (trial definition), and that uses the appropriate samplerate (to be found
>>>> in the datastructure field .fsample). An example on making your own
>>>> trialfun can be found here:
>>>> http://www.fieldtriptoolbox.org/tutorial/preprocessing/. You can also
>>>> check http://www.fieldtriptoolbox.org/walkthrough/
>>>>
>>>> 3b) Use your fancy corrected trl in combination with ft_redefinetrial
>>>> to epoch your resampled data.
>>>>
>>>> *) IMHO, creating your own trialfun pays off in the long run, as it
>>>> allows you to be much flexible, and e.g. enter response times and those
>>>> kind of things in extra columns in the .trl, which will then be carried
>>>> trhough your data in a .trialinfo field.
>>>>
>>>> Good fieldtripping!
>>>> Stephen
>>>>
>>>>
>>>>
>>>> On Thu, 18 Jul 2019 at 16:59, Jeremy Pulmano <jpulmano at princeton.edu>
>>>> wrote:
>>>>
>>>>> Hi Stephen,
>>>>>
>>>>> Thanks so much for your reply. Apologies for the lack of specificity;
>>>>> what I meant to say is that out of 2000 trials, around 1500 of them contain
>>>>> NaNs after epoching. Thus, when I try visual artifact rejection, I only see
>>>>> 500 trials and the other 1500 are omitted completely.
>>>>>
>>>>> The NaNs appear after epoching. I think you're correct in identifying
>>>>> a mistake with the sampling rates. How can I specify the new sampling
>>>>> frequency in the trial definition? I think there might also be an issue
>>>>> because FieldTrip reads the old sampling frequency from the headerfile (I
>>>>> am using Biosemi .bdf files).
>>>>>
>>>>> Here is a snippet of the code. I am also attaching it to this email
>>>>> for easier reference. Thanks again.
>>>>>
>>>>> %% Setup code (abridged)
>>>>>>
>>>>>> filename = 'example';
>>>>>> header = ft_read_header(filename);
>>>>>> new_fs = 128;
>>>>>> prestim = 0.1;
>>>>>> poststim = 0.4;
>>>>>> trigger_codes = [ 1 2 3 ];
>>>>>>
>>>>>> %% High pass filter
>>>>>>
>>>>>> % Define trial for all the continuous data
>>>>>> cfg = [];
>>>>>> cfg.dataset = filename;
>>>>>> cfg.headerfile = header;
>>>>>> cfg.trialdef.triallength = Inf;
>>>>>> cfg.trialdef.ntrials = 1;
>>>>>>
>>>>>> cfg = ft_definetrial(cfg);
>>>>>>
>>>>>> % Preprocess with high pass filter
>>>>>> order = 100;
>>>>>> cfg.hpfilter = 'yes';
>>>>>> cfg.hpfilttype = 'fir';
>>>>>> cfg.hpfreq = 0.1;
>>>>>> cfg.hpfiltord = order;
>>>>>>
>>>>>> data = ft_preprocessing(cfg);
>>>>>> data_hp = data;
>>>>>>
>>>>>> %% Downsampling
>>>>>>
>>>>>> fprintf('\nDownsampling\n');
>>>>>>
>>>>>> cfg = [];
>>>>>> cfg.resamplefs = new_fs;
>>>>>>
>>>>>> data = ft_resampledata(cfg, data);
>>>>>> data_res = data;
>>>>>>
>>>>>> %% Epoch (NaNs appear after this block)
>>>>>>
>>>>>> % Now define the ACTUAL trial
>>>>>> cfg = [];
>>>>>> cfg.continuous = 'yes';
>>>>>> cfg.dataset = filename;
>>>>>> cfg.trialdef.prestim = prestim;
>>>>>> cfg.trialdef.poststim = poststim;
>>>>>> cfg.trialdef.eventtype = 'STATUS';
>>>>>> cfg.trialdef.eventvalue = trigger_codes
>>>>>>
>>>>>> cfg = ft_definetrial(cfg);
>>>>>>
>>>>>> data = ft_redefinetrial(cfg, data);
>>>>>> data_ep = data;
>>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 18, 2019 at 3:06 PM Stephen Whitmarsh <
>>>>> stephen.whitmarsh at gmail.com> wrote:
>>>>>
>>>>>> Dear Jeremy,
>>>>>>
>>>>>> Welcome to the FieldTrip community!
>>>>>>
>>>>>> Could you be more specific about "the data looks incorrect in the
>>>>>> end.", and perhaps share a screenshot if informative? Sharing the relevant
>>>>>> part of your script would be helpful too. For more info about how to get
>>>>>> the best response to your questions, please take a look at:
>>>>>> http://www.fieldtriptoolbox.org/faq/how_to_ask_good_questions_to_the_community/
>>>>>>
>>>>>> Have you checked where the NaN's in the data are, and whether they
>>>>>> appear before or after downsampling?
>>>>>>
>>>>>> Just an shot in the dark, but have you perhaps defined your trials
>>>>>> for epoching (i.e. the samples in the .trl) based on the old samplerate?
>>>>>>
>>>>>> Cheers,
>>>>>> Stephen
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 18 Jul 2019 at 12:35, Jeremy Pulmano <jpulmano at princeton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've realized that the warning message often comes right after
>>>>>>> epoching (using ft_definetrial and then ft_redefinetrial). It seems like it
>>>>>>> successfully breaks the data into segments but still contains NaNs for
>>>>>>> whatever reason. Here are more details about the pipeline so far:
>>>>>>>
>>>>>>> 1. High pass filtering (0.1 Hz / FIR / order = 100)
>>>>>>> 2. Downsampling (1024 to 128 Hz)
>>>>>>> 3. Epoching (prestim 0.1, poststim 0.4)
>>>>>>> 4. Base-line correction [ -0.1 0 ]
>>>>>>> 5. Re-referencing (common avg.)
>>>>>>> 6. ICA / component rejection
>>>>>>> 7. Visual artifact removal
>>>>>>> 8. Low pass filter (30 Hz)
>>>>>>>
>>>>>>> Any further help would be greatly appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jeremy
>>>>>>>
>>>>>>> On Wed, Jul 17, 2019 at 4:57 PM Jeremy Pulmano <
>>>>>>> jpulmano at princeton.edu> wrote:
>>>>>>>
>>>>>>>> Hi Erika,
>>>>>>>>
>>>>>>>> The original sampling rate is 1024 Hz and the new sampling rate is
>>>>>>>> 128 Hz.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> J
>>>>>>>>
>>>>>>>> On Wed, Jul 17, 2019 at 4:49 PM Erika Puiutta <
>>>>>>>> erika.puiutta at uni-oldenburg.de> wrote:
>>>>>>>>
>>>>>>>>> Hey J,
>>>>>>>>>
>>>>>>>>> this is just an educated guess since I don't really know how
>>>>>>>>> fieldtrip does the downsampling, but what is your sampling frequency and
>>>>>>>>> the sampling frequency you sample down to? Is it possible that your
>>>>>>>>> downsampling frequency is not a harmonic of your original sampling
>>>>>>>>> frequency (say 1000Hz to 300Hz)? Maybe there are NaNs for the new samples
>>>>>>>>> where fieldtrip can't find a corresponding one from the old sampling
>>>>>>>>> frequency?
>>>>>>>>>
>>>>>>>>> Best of luck,
>>>>>>>>> Erika
>>>>>>>>>
>>>>>>>>> Am 17.07.2019 um 17:26 schrieb Jeremy Pulmano <
>>>>>>>>> jpulmano at princeton.edu>:
>>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I am new to both neuroscience and FieldTrip and am trying to debug
>>>>>>>>> my preprocessing pipeline of EEG data (biosemi .bdf files). These are the
>>>>>>>>> current steps:
>>>>>>>>>
>>>>>>>>> 1. High pass filtering
>>>>>>>>> 2. Downsampling
>>>>>>>>> 3. Epoching
>>>>>>>>> 4. Base-line correction
>>>>>>>>> 5. Re-referencing
>>>>>>>>> 6. ICA / component rejection
>>>>>>>>> 7. Visual artifact removal
>>>>>>>>> 8. Low pass filter (I save the low pass filter for the end so
>>>>>>>>> that ICA yields better time course plots).
>>>>>>>>>
>>>>>>>>> However, after the first three steps (high pass, downsample,
>>>>>>>>> epoch), I continue to get the warning message: "Warning: data contains NaN
>>>>>>>>> values, no filtering or preprocessing applied," starting at baseline
>>>>>>>>> correction. It repeats itself over and over again. If I ignore these
>>>>>>>>> errors, the data looks incorrect in the end. Why does this happen, and how
>>>>>>>>> can I resolve this?
>>>>>>>>>
>>>>>>>>> Any tips would be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> J
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> fieldtrip mailing list
>>>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> fieldtrip mailing list
>>>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Jeremy Pulmano*
>>>>>>>> *Princeton University Class of 2021*
>>>>>>>> B.S.E. Computer Science
>>>>>>>> e: jpulmano at princeton.edu | c: (862) 220-5903
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> fieldtrip mailing list
>>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>>
>>>>>> _______________________________________________
>>>>>> fieldtrip mailing list
>>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>>
>>>>> _______________________________________________
>>>>> fieldtrip mailing list
>>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>>
>>>> _______________________________________________
>>>> fieldtrip mailing list
>>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>>
>>> _______________________________________________
>>> fieldtrip mailing list
>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>> https://doi.org/10.1371/journal.pcbi.1002202
>>>
>> _______________________________________________
>> fieldtrip mailing list
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>> https://doi.org/10.1371/journal.pcbi.1002202
>>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190719/83e1c714/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
More information about the fieldtrip
mailing list