[FieldTrip] Invisible data after ft_resampledata
Emilie Caspar
ecaspar at ulb.ac.be
Fri Oct 29 17:04:44 CEST 2021
Dear Jan-Mathijs,
So indeed I confirm that I have checked and there were no NaNs after the downsampling procedure. For clarification, I do the following steps (code below)
I do basically the following:
1) downsampling (datall —> No NaNs)
2) bandpass (0.1-40, order 1)
3) re-referencing (AllData_Preprocesses —> No NaNs)
4) spectral interpolation (data_intpl —> No NaNs)
5) ICA (data —> No NaNs)
6) Epoching —> NaNs appear for just some of them.
7) …
I have checked and after each step of the processing of my data structure, there is not a single NaN. I have also attached a print screen of the « data » structure and a data browser.
Then goes the epoching. For some triggers, there are actually values in each epoch. But for some other triggers, they are replaced by NaN, but just at the moment of the epoching. While in the trialinfo, the triggers do exist.
Of course, the code is exactly the same for each different trigger. When I remove the downsampling phase from my preprocessing, this problem does not happen and I can preprocess my data without future issues. I can’t figure out why, based on a similar structure, some triggers give NaN while other not, something which does not happen when I don’t downsample. And it seems to change from one participant to another (i.e. the data which disappear from some triggers).
Please find below the full code. I have tried two methods: the one you suggested with datr{i}.time = datr{1}.time; but the result is the same and the one below with downsampling as a first step.
Could it be that the downsample function removes, in a way or another, data point but also triggers if they happen to happen at specific data points?
Thank you for the help,
Emilie
bpfilterOrder = 1; % ft default
bpfilterRange = [.1 40];
% % Event details
windows = [-0.2, 1.3]; % get trials from -.70 ms to 200 ms.
baselines = [-0.2 0]; % Baseline from -.70 ms to -.50 ms.
ERPtrialN = [];
ERPs = [];
for s=1
thisSubj = subjID{s};
subjPath = [eegPath thisSubj];
file = dir([subjPath '*.bdf']);
fprintf('\n\n ************** Processing Subject %s **************\n\n ', thisSubj(2:end));
cfg = [];
cfg.dataset = [file.name];
datp = ft_preprocessing(cfg);
cfg = [];
cfg.resamplefs = 512;
cfg.demean = 'no';
fg.detrend = 'yes';
cfg.trials = 'all';
cfg.feedback = 'text';
datall = ft_resampledata(cfg, datp);
cfg = [];
cfg.bpfilter = 'yes';
cfg.bpfreq = bpfilterRange;
cfg.bpfiltord = bpfilterOrder;
cfg.reref = 'yes';
cfg.refchannel = chansRef;
allData_preprosses = ft_preprocessing(cfg, datall);
% Spectrum interpolation
cfg= [];
cfg.dftfilter = 'yes';
cfg.dftfreq = [50 100 150];
cfg.dftreplace = 'neighbour'; % implicates spectrum interpolation
cfg.dftbandwidth = [2 2 2];
cfg.dftneighbourwidth = [2 2 2];
data_intpl = ft_preprocessing(cfg, allData_preprosses);
%%% Remove mastoids
cfg = [];
cfg.channel = chansEEG;
eegData = ft_selectdata(cfg, data_intpl);
% perform the independent component analysis (i.e., decompose the data)
cfg = [];
cfg.numcomponent = 30;
cfg.method = 'runica';
comp = ft_componentanalysis(cfg, eegData);
% %%ICA
figure
cfg = [];
cfg.component = 1:30;
cfg.layout = 'biosemi64.lay';
cfg.comment = 'no';
cfg.colormap=colormap('jet');
ft_topoplotIC(cfg, comp);
%
% % 1 4 15 63
cfg = [];
cfg.layout = 'biosemi64.lay'; % specify the layout file that should be used for plotting
cfg.viewmode = 'component';
cfg.colormap=colormap('jet');
ft_databrowser(cfg, comp);
STOP
cfg = [];
cfg.component = [1,2]; % to be removed component(s)
data = ft_rejectcomponent(cfg, comp, eegData);
finfname = [subjID{s} '_ICA'];
save([ finfname '.mat'], 'data', '-v7.3');
display('Epochs')
for e = 1:length(eventLabel)
fprintf('\n ********* Processing Event: %s ********* \n ', eventLabel{e});
% % % % % Create the trial definition
cfg = [];
cfg.dataset = [ file.name];
cfg.trialdef.eventtype = 'STATUS';
cfg.trialdef.eventvalue = [41,42,43,44];
cfg.trialdef.prestim = abs(windows(1));
cfg.trialdef.poststim = windows(2);
epData_cfg = ft_definetrial(cfg);
cfg = [];
epData = ft_redefinetrial(epData_cfg, data);
%% Baseline
cfg = [];
cfg.demean = 'yes';
cfg.baselinewindow = baselines(e,:);
epData = ft_preprocessing(cfg, epData);
%%%%%%%%%%%% ARTEFACTS %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
cfg = [];
cfg.alim = 100;
cfg.keepchannel = 'yes';
cfg.layout = 'biosemi64.lay';
% cfg.method = 'summary';
% cfg.method = 'trial';
cfg.method = 'channel'; %% Or 'trial'
cfg.metric = 'var';
clean_data = ft_rejectvisual(cfg, epData); %interpolData% Pour rejeter les artefacts (seulement pour donn?es segment?es)
cfg.artfctdef.reject = 'complete'; % Permet de sauvegarder les artefacts rejet?s
cfg.artfctdef.feedback = 'no';
cleandata = ft_rejectartifact(cfg,clean_data);
%%%%% AVERAGING%%%%%%%%
cfg = [];
cfg.trials = 'all';
cfg.covariance = 'yes';
cfg.covariancewindow = [-inf 0]; %it will calculate the covariance matrix
% on the timepoints that are
% before the zero-time point in the trials
Imagine_DOUL = ft_timelockanalysis(cfg, cleandata);
%%%%% AVERAGING%%%%%%%%
end
finfname = [subjID{s} '_Imagine_DOUL'];
save([ finfname '.mat'], 'Imagine_DOUL’);
end
---------------------------------------------
---------------------------------------------
Emilie CASPAR
Associate Professor
Department of Experimental Psychology, Ghent University
office: Henri Dunantlaan, 2 - Floor 2, Room 94
lab’s website: https://moralsocialbrain.com/ <https://moralsocialbrain.com/>
personal website: https://emiliecaspar.home.blog/ <https://emiliecaspar.home.blog/>
Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95
> Le 29 oct. 2021 à 14:51, Schoffelen, J.M. (Jan Mathijs) via fieldtrip <fieldtrip at science.ru.nl> a écrit :
>
> hi Emilie,
>
> Given the lack of responses so far, I think that nobody has encountered this before. I hope that you inspected your data for NaNs? This is the most likely cause of your problem. Also, your use case seems to be quite an exceptional one, since you seem to be reading the channels one at the time (and resample), and then concatenate afterwards, using ft_appenddata. ft_appenddata performs some checks on the sanity of the sampled time axis in order to determine how (and whether) the data can be appended. I suspect that in your case, the channels’ sampling might be slightly different from one channel to the next, which becomes numerically not tolerated once the effective sampling rate is reduced too much.
>
> Possible solutions:
> 1) only downsample after concatenation
> 2) impose a numerically identical time axis across channels, e.g. by including a statement datr{i}.time = datr{1}.time; within the for loop.
>
> Best wishes,
> Jan-Mathijs
>
>
>> On 29 Oct 2021, at 12:52, Emilie Caspar via fieldtrip <fieldtrip at science.ru.nl <mailto:fieldtrip at science.ru.nl>> wrote:
>>
>> Dear Fieldtrippers,
>>
>> Just a follow-up email to know if someone already got this problem of « Invisible trials » after downsampling and has an idea about how to solve the problem.
>>
>> Kind regards,
>>
>> Emilie
>>
>> <Downsampling.001.jpeg>
>> ---------------------------------------------
>> Emilie CASPAR
>>
>> Associate Professor
>>
>> Department of Experimental Psychology, Ghent University
>> office: Henri Dunantlaan, 2 - Floor 2, Room 94
>>
>> lab’s website: https://moralsocialbrain.com/ <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fmoralsocialbrain.com%2F__%3B!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqAQY_tczI%24&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942213822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rF7Md6zG0ZST7IPvrI%2BSunRYU3vfpFsX%2BwvlLqLlO%2B4%3D&reserved=0>
>> personal website: https://emiliecaspar.home.blog/ <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Femiliecaspar.home.blog%2F__%3B!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqAkCkgYUk%24&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942223776%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BuOPHjp%2By9jbnqG0vyPZnwWXNmX3VRGGVtkJOBdhNEQ%3D&reserved=0>
>>
>> Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95
>>
>>
>>> Le 21 oct. 2021 à 14:40, Emilie Caspar via fieldtrip <fieldtrip at science.ru.nl <mailto:fieldtrip at science.ru.nl>> a écrit :
>>>
>>> Dear fieldtrippers,
>>>
>>> I am trying to downsample my EEG data, originally recorded at 2048 Hz.
>>>
>>> I run the following code:
>>>
>>> cfg = [];
>>> %cfg.channel = {'A1', 'B2', 'A5', 'B5', 'B6', 'B8', 'A15', 'B20', 'B15', 'B16', 'A13', 'B18', 'A27', 'A28', 'A31', 'A29', 'A30', 'A32', 'B32', 'A21', 'A31', 'B26', 'EXG1', 'EXG2', 'EXG3', 'EXG4'}%'B15';
>>> cfg.dataset = [ file.name];
>>> cfg.bpfilter = 'yes';
>>> cfg.bpfreq = bpfilterRange;
>>> cfg.bpfiltord = bpfilterOrder;
>>> cfg.reref = 'yes';
>>> cfg.refchannel = chansRef;
>>> allData_preprosses = ft_preprocessing(cfg);
>>>
>>> cfg = [];
>>> cfg.resamplefs = 512;
>>> data_DS = ft_resampledata(cfg, allData_preprosses);
>>>
>>>
>>> When I look at individual trials for artefact rejection, something very weird happens. Basically some trials become ‘invisible’ (sorry, I don’t find any other way to describe it), see figure attached. It seems that the data do exit though because when I average my visible trials to the invisible ones, it results in an invisible ERPs graph. But when I remove the trials containing the invisible data, an ERP graph does appear (please don’t pay attention to the quality of the data, this is just an example without any cleaning).
>>>
>>> It seems there is a logic to the phenomenon, because if I downsample to 1024Hz (so half of my initial 2048 Hz), only half of my data becomes invisible and if I downsample to 512Hz, a 3/4 of the data becomes invisible.
>>>
>>> I have tried the same with a different coding, with this time downsampling before filtering as follows, but the output is exactly the same.
>>>
>>> display('downsample each channel')
>>> cfgp = [];
>>> cfgp.dataset = [file.name];
>>> cfgr = [];
>>> cfgr.resamplefs = 512;
>>> display('Resampling')
>>> for i= chansAll
>>> cfgp.channel = i;
>>> datp = ft_preprocessing(cfgp);
>>> datr{i} = ft_resampledata(cfgr, datp);
>>> clear datp
>>> end
>>> cfg = [];
>>> datall = ft_appenddata(cfg, datr{:});
>>>
>>> display('Filters')
>>>
>>> cfg = [];
>>> cfg.dataset = [ file.name];
>>> cfg.bpfilter = 'yes';
>>> cfg.bpfreq = bpfilterRange;
>>> cfg.bpfiltord = bpfilterOrder;
>>> cfg.reref = 'yes';
>>> cfg.refchannel = chansRef;
>>> data_DS = ft_preprocessing(cfg, datall);
>>>
>>>
>>> Does anyone have any idea regarding this phenomenon?
>>>
>>> Thank you in advance!
>>>
>>> Best regards,
>>>
>>> Emilie
>>>
>>> <Downsampling.001.jpeg>
>>> ---------------------------------------------
>>> Emilie CASPAR
>>>
>>> Associate Professor
>>>
>>> Department of Experimental Psychology, Ghent University
>>> office: Henri Dunantlaan, 2 - Floor 2, Room 94
>>>
>>> lab’s website: https://moralsocialbrain.com/ <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Fmoralsocialbrain.com*2F%26data%3D04*7C01*7Cemilie.caspar*40ulb.be*7C9eaa583fc9f2496f3d9708d99494a638*7C30a5145e75bd4212bb028ff9c0ea4ae9*7C0*7C0*7C637704188432821811*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26sdata%3DX9emmQ51jam6Zr*2FzU1ntTu4F*2FAWsPVUwbwMPtsxLOhw*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJQ!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqAxN_xjhw%24&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942233734%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=97ffhkfBhxqQTqVVe93tJ%2F%2BscIaQbbl2RBAq1Txxo%2Fo%3D&reserved=0>
>>> personal website: https://emiliecaspar.home.blog/ <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Femiliecaspar.home.blog*2F%26data%3D04*7C01*7Cemilie.caspar*40ulb.be*7C9eaa583fc9f2496f3d9708d99494a638*7C30a5145e75bd4212bb028ff9c0ea4ae9*7C0*7C0*7C637704188432831766*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26sdata%3DRSBTcOxejkzdWKA9QOyJZKXBD*2F5bGyFC12*2FKCpKl8GY*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJQ!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqAk0r1eNs%24&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942243688%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=k8evpeTT%2FhyMZOCsUHpjOPZW5BR%2FDxmwbB3jPJ9HVUE%3D&reserved=0>
>>>
>>> Université libre de Bruxelles (office & contact): DB10.138 / +32 2 650 32 95
>>>
>>>
>>> _______________________________________________
>>> fieldtrip mailing list
>>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942243688%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fssxL2sleRPNFIHBaV6eHJ4fgsa%2B6x5rKhV8n4vYNAQ%3D&reserved=0>
>>> https://doi.org/10.1371/journal.pcbi.1002202 <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202__%3B!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqA3PXLTtU%24&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942253648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FSbbGE9ncIe8ayTRcxDJCRyC%2BsL67pSL6x8ah6%2BzeIA%3D&reserved=0>
>>
>> _______________________________________________
>> fieldtrip mailing list
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=04%7C01%7Cemilie.caspar%40ulb.be%7Cc85af862d2004c60d8cf08d99ae0a6b2%7C30a5145e75bd4212bb028ff9c0ea4ae9%7C0%7C0%7C637711111942263600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jGMIbV6xpw%2Bcphs0t8Qn7H7G4w0FoVQ9sZpAwilsKcU%3D&reserved=0>
>> https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!vIs6cDO9UI_eSxJ-ejSlr1oqs47g4UtnsVqQOLcRSfvkRYPZQz1BrYgADkAqrz-vRfTJ2oqA3PXLTtU$
>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Capture d?e?cran 2021-10-29 a? 16.45.55.png
Type: image/png
Size: 21596 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Capture d?e?cran 2021-10-29 a? 16.52.59.png
Type: image/png
Size: 85955 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Capture d?e?cran 2021-10-29 a? 17.02.12.png
Type: image/png
Size: 26340 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Capture d?e?cran 2021-10-29 a? 17.01.58.png
Type: image/png
Size: 150955 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Capture d?e?cran 2021-10-29 a? 17.00.31.png
Type: image/png
Size: 46064 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20211029/c14db527/attachment-0009.png>
More information about the fieldtrip
mailing list