[FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

Wed Nov 30 16:56:41 CET 2016

Hi Christine,

I don’t have the answer to your specific question, but I want to raise a few points:

Although the permutation framework as implemented in FT outputs 1 p-value per cluster (for a two-sided test, both for the ‘negative’ and for the ‘positive’ clusters), Only the smallest p-value counts for the statistical inference. This is because your inferential procedure is about making a binary decision, you either reject or accept your null-hypothesis.

Also, note that in the permutation framework, without explicit adjustment, the p-values that come out reflect one-sided p-values. For a valid inference, you need to Bonferroni correct these (i.e. multiply them by 2), or adjust the critical alpha level.

This being said, I would say that what you want to achieve (i.e. doing post-hoc tests) does not need to be done within the cluster-based framework. The clusters are just byproducts of your inferential procedure.

Some general background on how to deal with the output of the tests can be found on:

http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test

In your specific case, where as a first step you have evaluated the interaction as a difference of differences, I would think it’s fine to use this result to justify a selection of channel + time points, across which you average the condition specific ERP, and which you subject to your post hoc tests.

Best,
Jan-Mathijs

On 30 Nov 2016, at 09:57, Blume Christine <christine.blume at sbg.ac.at<mailto:christine.blume at sbg.ac.at>> wrote:

Maybe an example will stimulate some discussion ;-).

I test an interaction among two factors with two levels each with regard to the ERPs elicited by each stimulus.
Let’s say we present two stimuli (house vs. face) in colour (black/white vs. colour). The interaction is significant and we want to know which factor levels differ using post hoc tests using the following cfg.

cfg = [];
cfg.latency = [0 1];
cfg.method = 'montecarlo';
cfg.statistic = 'ft_statfun_depsamplesT';
cfg.correctm = 'cluster';
cfg.clusteralpha = 0.05;
cfg.clusterstatistic = 'maxsum';
cfg.minnbchan = 2;
cfg.tail = 0;
cfg.clustertail = 0;
cfg.alpha = 0.025;
cfg.numrandomization = 1000;
cfg_neighb.method = 'distance';
cfg.neighbours = ft_prepare_neighbours(cfg_neighb, data_sample);
cfg.design = design;
cfg.ivar = 2;
cfg.uvar = 1;

What we get is the following.

stat_erp.FACE_COLOURvsFACE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.FACE_BW);
stat_erp.FACE_COLOURvsFACE_BW.posclusters.prob % 18 clusters, p > 0.4076
stat_erp.FACE_COLOURvsFACE_BW.negclusters.prob % 12 clusters, p > 0.2957
stat_erp.FACE_COLOURvsHOUSE_COLOUR = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_COLOUR);
stat_erp.FACE_COLOURvsHOUSE_COLOUR.posclusters.prob % 10 clusters, p > 0.4326
stat_erp.FACE_COLOURvsHOUSE_COLOUR.negclusters.prob % 12 clusters, p = 0.0599; p > 0.8631
stat_erp.FACE_COLOURvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.FACE_COLOURvsHOUSE_BW.posclusters.prob % 16 clusters, p = 0.0360, p > 0.7742
stat_erp.FACE_COLOURvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0300, p > 0.5105
stat_erp.HOUSE_COLOURvsFACE_BW = ft_timelockstatistics(cfg, grandavg_erp.HOUSE_COLOUR, grandavg_erp.FACE_BW);
stat_erp.HOUSE_COLOURvsFACE_BW.posclusters.prob % 16 clusters, p > 0.2697
stat_erp.HOUSE_COLOURvsFACE_BW.negclusters.prob % 11 clusters, p = 0.0090, p > .0.9291
stat_erp.HOUSE_COLOURvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.HOUSE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.HOUSE_COLOURvsHOUSE_BW.posclusters.prob % 19 clusters, p = 0.0020, p = 0.0979, p > 0.2468
stat_erp.HOUSE_COLOURvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0480, p = 0.0699, p > 0.3606
stat_erp.FACE_BWvsHOUSE_BW = ft_timelockstatistics(cfg, grandavg_erp.FACE_COLOUR, grandavg_erp.HOUSE_BW);
stat_erp.FACE_BWvsHOUSE_BW.posclusters.prob % 16 clusters, p = 0.0410, p > 0.7732
stat_erp.FACE_BWvsHOUSE_BW.negclusters.prob % 13 clusters, p = 0.0320, p > 0.4725

Now the question is: how do we correct for these 6 follow-up tests? If we say we want to use the Bonferroni-Holm correction for example, we have to sort p-values according to size. However, we have two p-values per test, one for positive and one for negative clusters and Bonferroni-Holm assumes we only have one per test.

Any ideas or suggestions how to best handle this?

Best,
Christine

Von: Blume Christine
Gesendet: Montag, 28. November 2016 17:04
An: FieldTrip discussion list (fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>)
Betreff: AW: [FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

Sorry, I forgot to mention that I am using

cfg.method = 'montecarlo';

Best,
Christine

Von: fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl> [mailto:fieldtrip-bounces at science.ru.nl] Im Auftrag von Blume Christine
Gesendet: Montag, 28. November 2016 16:46
An: FieldTrip discussion list (fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>)
Betreff: [FieldTrip] Cluster-based permutation: Interactions & Alpha adjustment

Dear community,

I am using the cluster-based permutation test for statistical evaluation of my data. Besides main effects I also test the interaction between them. When following up on a significant interaction with further permutation tests I feel I should correct my alpha level for multiple comparisons. Is that correct?

Best,
Christine

---------------------------------------------------------------------------------
Dipl.-Psych. Christine Blume
University of Salzburg
Department of Psychology
Centre for Cognitive Neuroscience (CCNS)
Laboratory for Sleep, Cognition and Consciousness Research
Hellbrunner Str. 34
A-5020 Salzburg
Email: christine.blume at sbg.ac.at<mailto:christine.blume at sbg.ac.at>
T: +43 (0) 662 – 8044 5146
www.sleepscience.at<http://www.sleepscience.at/>

_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl<mailto:fieldtrip at donders.ru.nl>
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20161130/b78938c4/attachment-0002.html>