[FieldTrip] Cluster-Based Permutation Tests with Mixed Design 2x2

[Arnaud Delorme]

Hi Ascen,

Robert (Oostenveld) pointed me recently to the page below. You should test for differences in the differences.

http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests/

If it does not fulfill your need, then the only alternative I know of is to use LIMO (https://github.com/LIMO-EEG-Toolbox/limo_tools/wiki) which allows you to create arbitrarily complex designs and perform cluster based permutation tests.

Best wishes,

Arno

> On Oct 5, 2020, at 2:12 AM, Pagan, Ascension (Dr.) <appc1 at leicester.ac.uk> wrote:
> 
>  
> Dear All,
> I have a mixed design (2 conditions x 2 groups), with 44 participants in group1, and 30 participants in group2, and I am planning to run cluster-based permutation tests on ERP data to test:
> 	• The main effect of condition (cond1 vs. cond2)
> 	• The main effect of group (group1 vs. group2)
> 	• The effect of condition in group1 (group1 cond1 vs. group1 cond2)
> 	• The effect of condition in group2 (group2 cond1 vs. group2 cond2)
> 	• The interaction effect between condition and group (group1 cond1 - group1 cond2) vs. (group2 cond1 - group2 cond2)
>  
> I have used ft_timelockanalysis to create data structures for COND1_ft, COND2_ft, group1_ft, group2_ft; COND1group1_ft, COND2group1_ft; COND1group2_ft, COND2group2_ft but I am having some doubts regarding design matrix and cfg.statistic. I have read the suggestions and comments in the discussion list, but I’m still not sure if I’m setting them up correctly. Below, I specify what I’m trying to do and how I have done it so far: 
>  
> First, my understanding from a message dated some years ago https://mailman.science.ru.nl/pipermail/fieldtrip/2008-March/014371.html is that to perform permutation tests on a mixed design we should have a design matrix with Row 1 (condition), Row 2 (subject number), and Row 3 (group). In my case:
>  
> sbj = 74 %(i.e., 44 sbjs in group1, 30 subjs in group2);
> design = zeros(3, 2*sbj);
>  
> %Set up Row 1
> design(1, 1:sbj) = 1; %(Row 1 with cond1 from 1:74 column)
> design(1,sbj+1:2*sbj) = 2; %(Row 1 with cond2 from 75:148 column)
>  
> %Set up Row 2
> for i= 1:sbj
>          design(2,i) = i; %(Row 2 with sbj number from 1:74 column)
> end
> for i=1:sbj
>          design(2,sbj+i)=i; %(Row 2 with sbj number from 75:148 column)
> end
>  
> %Set up Row 3
> design(3, [1:44 75:118]) = 1; %(Row 3 with group1 from 1:44 and 75:118 column)
> design(3, [45:74 119:sbj*2]) = 2; %(Row 3 with group2 from 45:74 and 119:148 column)
>  
> cfg.design = design;
>  
> With the following configuration:
> cfg.method = 'montecarlo';
> cfg.ivar = 1 %(row with independent variable, i.e., condition number)
> cfg.uvar = 2 %(row with unit variables, i.e., subject number)
> cfg.wvar = 3 %(row with within-cell variable, i.e., group number)
>  
> And, with the rest of the configuration set up as follows: 
> cfg.minnbchan = 2; 
> cfg.neighbours = neighbours;  
> cfg.alpha = 0.05; 
> cfg.tail = 0; %two-sided test
> cfg.correcttail = 'alpha'; %conceptually equivalent to performing a Bonferroni correction for the two tails
> cfg.numrandomization = 10000; 
> cfg.correctm = 'cluster';
> cfg.clusterstatistic = 'maxsum'; 
> cfg.clusteralpha = 0.05; 
> cfg.clustertail = 0;
>  
> However, I am not sure about the design matrix because when I try to run the previous code, I get the following error messages:
> Error using resampledesign (line 153)
> the design matrix variables should be constant within a block
>  
> Error in ft_statistics_montecarlo (line 215)
> resample = resampledesign(cfg, design);
>  
> Error in ft_timelockstatistics (line 182)
>   [stat, cfg] = statmethod(cfg, dat, design);
>  
> Second, I am not sure which cfg.statistic I should be using in the mixed design. If I use subject number as my unit of observation (UO), then, in one case, UO is assigned to only one of two groups (between UO-design; independent samples), but at the same time, every UO is assigned to two experimental conditions (within UO-design; dependent samples) (http://www.fieldtriptoolbox.org/tutorial/cluster_permutation_timelock/?s). 
>  
> So, my question is: Is there any way to use a single cfg.statistic with a mixed design or should I run two separate tests: one with cfg.statistic = 'depsamplesT'; to test the main effect of condition (cond1 vs. cond2), the effect of condition in group1 (group1 cond1 vs. group1 cond2), and the effect of condition in group2 (group2 cond1 vs. group2 cond2), and another test cfg.statistic = 'indepsamplesT'; to test the main effect of group (group1 vs. group2)?
>  
> After this, I would then test:
> %Main effect of Condition
> [statCOND] = ft_timelockstatistics(cfg, COND1_ft{:}, COND2_ft{:});  
>  
> %Main effect of Group
> [statGROUP] = ft_timelockstatistics(cfg, group1_ft{:}, group2_ft{:});
>  
> %Main Effect of Condition in Group1
> [statCONDgroup1] = ft_timelockstatistics(cfg, COND1group1_ft{:},COND2group1_ft{:}); 
>  
> %Main Effect of Condition in Group2
> [statCONDgroup2] = ft_timelockstatistics(cfg, COND1group2_ft{:},COND2group2_ft{:});
>  
> Finally, to test the interaction effect ((group1 cond1 - group1 cond2) vs. (group2 cond1 - group2 cond2)), I was planning to use cfg.statistic = 'indepsamplesT’
> (as in (http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests/)  and then, first, calculate the differences as follows:
> cfg  = [];
> cfg.operation = 'subtract';
> cfg.parameter = 'avg';
> for i =1:length(COND1group1_ft)
> CONDgroup1DIFF_ft{i} = ft_math(cfg, COND1group1_ft{i}, COND2group1_ft{i});
> end
> for i = 1:length(COND1group2_ft)
> CONDgroup2DIFF_ft{i} = ft_math(cfg, COND1group2_ft{i}, COND2group2_ft{i});
> end
>  
> Second, set up the configuration as above and run the following:
> %Test Interaction Effect between Condition and Group
> [statCONDbyGROUP] = ft_timelockstatistics(cfg,CONDgroup1DIFF_ft{:},CONDgroup2DIFF_ft{:});
>  
>  
> Apologise for the long email, but I hope it makes sense. I’d be grateful to any advice or suggestion about the design matrix and cfg.statistic.
>  
> Best wishes,
> Ascen
>  
>  
> --------------------------------
> Dr Ascensión Pagán
> Lecturer, Department of Psychology
> Faculty of Health and Life Sciences
> Northumbria University
> Newcastle upon Tyne
> NE1 8ST
> UK
> ascension.pagan at northumbria.ac.uk
>  
>  
> This message is intended solely for the addressee and may contain confidential and/or legally privileged information. Any use, disclosure or reproduction without the sender’s explicit consent is unauthorised and may be unlawful. If you have received this message in error, please notify Northumbria University immediately and permanently delete it. Any views or opinions expressed in this message are solely those of the author and do not necessarily represent those of the University. Northumbria University email is provided by Microsoft Office365 and is hosted within the EEA, although some information may be replicated globally for backup purposes. The University cannot guarantee that this message or any attachment is virus free or has not been intercepted and/or amended. 
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202