[FieldTrip] Warning with redefining data

Thu Mar 9 13:02:47 CET 2023

Hi Marta,

I would strongly advice against glueing together short snippets of data into longer snippets, unless you are really sure about what you are doing.

The reason for this is that the individual 2-second snippets might have been processed in a way that results in discontinuities at the original snippets’ edges, when glued together. This may have a profound effect on the power spectrum of the signal, even if computed in python.

In sum, even if you would be able to fix the NaN-issue I think that the current pipeline is flawed: if you want to compute power spectra with a sufficiently high spectral resolution, you need to go back to the raw data, and not try to glue together short preprocessed data snippets.

Getting back to the NaN issue: given the warnings that you sketch, and the sequence of processing steps that you describe, I still think that the glueing together of individual snippets hasn’t worked well. introducing the nans in the ft_redefinetrial step.
The fact that you are using data with an odd sampling rate, it may be very well conceivable that numeric rounding errors in the time axes of the trials may be the root of the issue.

Best wishes and good luck,
Jan-Mathijs

On 9 Mar 2023, at 12:19, Marta Stojanović via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hi Jan-Mathjis,

Thank you for your reply, these are very helpful notes.

Indeed, since my pipeline runs through without these warnings when I do not redefine the trials, the issue should be with the steps that attempt to create longer trials from the shorter resting-state trials. Yet, the redefined data does not (at least explicitly) include NaN values, and the outputted structs for these steps look as they should, with the relevant fields and no explicit NaNs.

Exactly as you suggested, when digging into the issue a bit further, it looks like NaN values are first introduced when computing the covariance in ft_timelockanalysis. When running the ft_timelockanalysis function without the covariance configuration I do not receive a warning about NaNs, but I need the covariance for later steps so I of course cannot omit this configuration nor would this solve the issue.

Exactly as you pointed out again, the NaNs in the covariance seem to then affect the rest of the steps downstream including computing the spatial filters and the ft_virtualchannel function.

To answer your question: I want to create longer 10-second trials from the 2-second trials because I need a longer trial length for the analyses I later conduct. I’m running power spectral density analyses, fitting oscillations and one-over-f (FOOOF) in Python, and later entropy analyses, for which 2-second trials simply aren’t long enough to generate “decent” results or to visualize results appropriately.

I know there are multiple ways to redefine trial length, and I have tried several approaches (such as defining cfg.begsample/endsample, cfg.continuous, and reshaping the original data by hand before passing it to ft_redefinetrial). My main question would therefore be how to ensure that the time axes of the original small snippets match up without gaps when redefining, as I’m sure there must be a way. I check the trial, label, and time fields for the outputs when redefining the trials and, again, all looks as it should, without any NaNs that seem to “fill the gaps”.

And an important final note would be then that the NaNs are exactly introduced in the covariance step rather than anywhere earlier (no NaNs in the data itself before or after timelocking), so something seems to be going wrong when computing the covariance.

Thanks a lot again,

Best,
Marta

From: fieldtrip <fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl>> on behalf of "Schoffelen, J.M. (Jan Mathijs) via fieldtrip" <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Reply to: FieldTrip discussion list <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Date: Thursday, 9. March 2023 at 09:17
To: FieldTrip discussion list <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Cc: "Schoffelen, J.M. (Jan Mathijs)" <janmathijs.schoffelen at donders.ru.nl<mailto:janmathijs.schoffelen at donders.ru.nl>>
Subject: Re: [FieldTrip] Warning with redefining data

Hi Marta,

You say: "I’ve run a loop over the trial field in the output of the ft_timelockanalysis as well as ft_preprocessing to check for NaNs in my 10-second data structs, but I didn’t find any when running this loop.”

This is at odds with your earlier statement that you got a warning that the data contains NaNs.

The NaNs have probably been introduced in your attempt to ‘glue together’ the 2-second epochs into longer snippets of 10 seconds. If the time axes of the original small snippets does not entirely match up (i.e. if there were gaps) then ft_redefinetrial might have filled those gaps with NaNs, while creating the 10 second snippets.

FieldTrip functions that operate on data containing NaNs often gracefully ignore them, but in many occasions the NaNs will affect the output. For instance (as is likely in your case), I think that the computation of the covariance in ft_timelockanalysis is not NaN-aware. This may cause downstream problems, when you use an all-NaN covariance matrix for the computation of spatial filters, which consequently will by construction also be all-NaN, as will be the virtual channel data that you try to derive from it.

The relevant question to ask here is: why do you want to stitch together the 2 second snippets of data?

Good luck and best wishes,
Jan-Mathijs

On 7 Mar 2023, at 21:09, Marta Stojanović via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hi Fieldtrip community,

I’m reaching out with a question about a warning I encountered when redefining trial length from resting-state MEG data using ft_definetrial and ft_redefine trial.

Namely, I redefined the length of input data from a resting-state MEG file from two-second trials to longer trials of 10-second length. I successfully used the ft_definetrial and ft_redefinetrial functions to do so, using the ‘trl’ field from the struct where I define the trial to input as a configuration for the ft_redefinetrial function.

With the two-second length I have 144 trials from the data, while with the ten-second length there are 28 trials.

The pipeline and structs look like the following:

%% define trials with 10-second length
clear ten_seconds;
cfg = [];
cfg.dataset = restfile;   % refers to resting-state file of a single subject
cfg.trialdef.beginning = 0; % start at the beginning of the recording
cfg.trialdef.length     = 10; % 10-second trial length
cfg.trialdef.overlap   = 0; % no overlap between trials
ten_seconds = ft_definetrial(cfg);

%% redefine trials to 10 seconds
clear redef_data;
cfg = [];
cfg.trl = ten_seconds.trl; % use trl from defined trial
redef_data = ft_redefinetrial(cfg,data);

%% preprocess redefined 10-second trials
clear ten_sec_trls;
cfg = [];
cfg.demean = 'yes';
cfg.baselinewindow = 'all';
% cfg.baselinewindow = [0 1018];
ten_sec_trls = ft_preprocessing(cfg,redef_data);

These are the associated structs with their fields:

redef_data =

fsample: 508.6275
grad: [1×1 struct]
label: {244×1 cell}
cfg: [1×1 struct]
hdr: [1×1 struct]
trial: {1×28 cell}
time: {1×28 cell}
trialinfo: [28×2 double]
sampleinfo: [28×2 double]

ten_sec_trls =

fsample: 508.6275
grad: [1×1 struct]
hdr: [1×1 struct]
trialinfo: [28×2 double]
trial: {1×28 cell}
time: {1×28 cell}
label: {244×1 cell}
cfg: [1×1 struct]

When I plug the redefined data (i.e., with redefined length, ten_sec_trls) into ft_timelockanalysis, I receive the following warning:

Warning: data contains NaN values
 In ft_preproc_polyremoval at line 75
 In ft_preproc_baselinecorrect at line 52
 In ft_timelockanalysis at line 174

the call to "ft_selectdata" took 0 seconds and required the additional allocation of an estimated 22 MB
Warning: inconsistent sampleinfo
 In makessense at line 115
 In ft_datatype_timelock at line 72
 In ft_checkdata at line 479
 In ft_timelockanalysis at line 207

I’ve run a loop over the trial field in the output of the ft_timelockanalysis as well as ft_preprocessing to check for NaNs in my 10-second data structs, but I didn’t find any when running this loop. I’ve checked the relevant lines from the Fieldtrip functions but couldn’t find a way to solve the warning from there. I’ve also checked the sampleinfo, but I sense the warning may be due to the sampleinfo in the configuration being inconsistent with the actual data due to the removal of non-MEG channels from the sampleinfo in the resting file.

The warning message might influence later analyses, so I’m wondering if anyone has faced the same and/or has an explanation.

Thanks a lot in advance!

All the best,
Marta
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://urldefense.com/v3/__https://eur04.safelinks.protection.outlook.com/?url=https*3A*2F*2Fmailman.science.ru.nl*2Fmailman*2Flistinfo*2Ffieldtrip&data=05*7C01*7Cmarta.stojanovic*40student.uva.nl*7Ca176944561f24559cdde08db2076b219*7Ca0f1cacd618c4403b94576fb3d6874e5*7C0*7C0*7C638139466402430326*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C&sdata=OsJDvoBFn3F3sGMfA0o9ij70Gs0uIamb0T4nX*2FurMJk*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!HJOPV4FYYWzcc1jazlU!-2PB3F72YfAhGO8MRm7edBkmsxtIIj1iHidGe-fn3vBb24hJrp61lSH--nnDL8k83-HbawlwfBwoEkjHHo0Q-aac2JFeVJcIuijxiFw$>
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!44dPKLWAx3oGspX-Q-IZx8b_gfPmQbTd45nbdyM61ROZgOiO2314lg4aY5eGcvDpKtQW3t1ilWOQcQ56Yd02ltXfOme3MNrQw0Ahfg$<https://urldefense.com/v3/__https://eur04.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fdoi.org*2F10.1371*2Fjournal.pcbi.1002202__*3B!!HJOPV4FYYWzcc1jazlU!44dPKLWAx3oGspX-Q-IZx8b_gfPmQbTd45nbdyM61ROZgOiO2314lg4aY5eGcvDpKtQW3t1ilWOQcQ56Yd02ltXfOme3MNrQw0Ahfg*24&data=05*7C01*7Cmarta.stojanovic*40student.uva.nl*7Ca176944561f24559cdde08db2076b219*7Ca0f1cacd618c4403b94576fb3d6874e5*7C0*7C0*7C638139466402430326*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C&sdata=SSG4qTaXpWJjWwF6LzGTFa1t*2BJwucoKKkAla2GnPI8o*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!HJOPV4FYYWzcc1jazlU!-2PB3F72YfAhGO8MRm7edBkmsxtIIj1iHidGe-fn3vBb24hJrp61lSH--nnDL8k83-HbawlwfBwoEkjHHo0Q-aac2JFeVJcIW-6Dbs4$>

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!5Ssql4vDlsZiezE-aJjMUmsU1rVpv6_N7yiWIgxiLmu93eiHASjcyUykgLf95Pi_Xd3fpn-Ak1lAynNEg_T7cVNRz6uXHl4OoOH8FQ$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20230309/36031919/attachment.htm>