[FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

[Blume Christine]

Maybe an example will stimulate some discussion ;-).

I test an interaction among two factors with two levels each with regard to the ERPs elicited by each stimulus.
Let's say we present two stimuli (house vs. face) in colour (black/white vs. colour). The interaction is significant and we want to know which factor levels differ using post hoc tests using the following cfg.

cfg = [];
cfg.latency = [0 1];
cfg.method = 'montecarlo';
cfg.statistic = 'ft_statfun_depsamplesT';
cfg.correctm = 'cluster';
cfg.clusteralpha = 0.05;
cfg.clusterstatistic = 'maxsum';
cfg.minnbchan = 2;
cfg.tail = 0;
cfg.clustertail = 0;
cfg.alpha = 0.025;
cfg.numrandomization = 1000;
cfg_neighb.method = 'distance';
cfg.neighbours = ft_prepare_neighbours(cfg_neighb, data_sample);
cfg.design = design;
cfg.ivar = 2;
cfg.uvar = 1;

What we get is the following.

stat_erp.FACE_COLOURvsFACE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.FACE_BW);
stat_erp.FACE_COLOURvsFACE_BW.posclusters.prob % 18 clusters, p > 0.4076
stat_erp.FACE_COLOURvsFACE_BW.negclusters.prob % 12 clusters, p > 0.2957
stat_erp.FACE_COLOURvsHOUSE_COLOUR = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_COLOUR);
stat_erp.FACE_COLOURvsHOUSE_COLOUR.posclusters.prob % 10 clusters, p > 0.4326
stat_erp.FACE_COLOURvsHOUSE_COLOUR.negclusters.prob % 12 clusters, p = 0.0599; p > 0.8631
stat_erp.FACE_COLOURvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.FACE_COLOURvsHOUSE_BW.posclusters.prob % 16 clusters, p = 0.0360, p > 0.7742
stat_erp.FACE_COLOURvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0300, p > 0.5105
stat_erp.HOUSE_COLOURvsFACE_BW = ft_timelockstatistics(cfg, grandavg_erp.HOUSE_COLOUR, grandavg_erp.FACE_BW);
stat_erp.HOUSE_COLOURvsFACE_BW.posclusters.prob % 16 clusters, p > 0.2697
stat_erp.HOUSE_COLOURvsFACE_BW.negclusters.prob % 11 clusters, p = 0.0090, p > .0.9291
stat_erp.HOUSE_COLOURvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.HOUSE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.HOUSE_COLOURvsHOUSE_BW.posclusters.prob % 19 clusters, p = 0.0020, p = 0.0979, p > 0.2468
stat_erp.HOUSE_COLOURvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0480, p = 0.0699, p > 0.3606
stat_erp.FACE_BWvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.FACE_BWvsHOUSE_BW.posclusters.prob % 16 clusters, p = 0.0410, p > 0.7732
stat_erp.FACE_BWvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0320, p > 0.4725

Now the question is: how do we correct for these 6 follow-up tests? If we say we want to use the Bonferroni-Holm correction for example, we have to sort p-values according to size. However, we have two p-values per test, one for positive and one for negative clusters and Bonferroni-Holm assumes we only have one per test.

Any ideas or suggestions how to best handle this?

Best,
Christine

Von: Blume Christine
Gesendet: Montag, 28. November 2016 17:04
An: FieldTrip discussion list (fieldtrip at science.ru.nl)
Betreff: AW: [FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

Sorry, I forgot to mention that I am using

cfg.method = 'montecarlo';

Best,
Christine

Von: fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl> [mailto:fieldtrip-bounces at science.ru.nl] Im Auftrag von Blume Christine
Gesendet: Montag, 28. November 2016 16:46
An: FieldTrip discussion list (fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>)
Betreff: [FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

Dear community,

I am using the cluster-based permutation test for statistical evaluation of my data. Besides main effects I also test the interaction between them. When following up on a significant interaction with further permutation tests I feel I should correct my alpha level for multiple comparisons. Is that correct?

Best,
Christine

---------------------------------------------------------------------------------
Dipl.-Psych. Christine Blume
University of Salzburg
Department of Psychology
Centre for Cognitive Neuroscience (CCNS)
Laboratory for Sleep, Cognition and Consciousness Research
Hellbrunner Str. 34
A-5020 Salzburg
Email: christine.blume at sbg.ac.at<mailto:christine.blume at sbg.ac.at>
T: +43 (0) 662 - 8044 5146
www.sleepscience.at<http://www.sleepscience.at/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20161130/014e9349/attachment-0002.html>