[FieldTrip] Evaluation under the permutation distribution positive/negative for two-tailed analysis

[Oliver Hermann]

Dear Jan-Mathijs,

Thank you for your response. Yes, that makes sense and is consistent with applying a Bonferroni Correction (/2) to the final p-value.

Thanks,
Oliver


-------------------
Oliver Hermann
PhD Researcher
Department of Psychology
University of Bath

From: fieldtrip <fieldtrip-bounces at science.ru.nl> On Behalf Of Schoffelen, J.M. (Jan Mathijs) via fieldtrip
Sent: 13 June 2025 10:19
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Cc: Schoffelen, J.M. (Jan Mathijs) <janmathijs.schoffelen at donders.ru.nl>
Subject: Re: [FieldTrip] Evaluation under the permutation distribution positive/negative for two-tailed analysis

You don't often get email from fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
CAUTION:  This email came from outside of the University. To keep your account safe, only click on links and open attachments if you know the person who sent the email, or you expected to receive this communication.


Dear Oliver,

Let me give it a shot:

I think the underlying idea of the statement in the referred paper is that the permutation distribution of the test-statistic (in this case a T-statistic) is symmetric around 0. Thus, under the assumption that a complete permutation distribution can be created (i.e. one can take all possible random splits of the individual observations to one of the conditions, and subsequently compute the T-statistics, followed by thresholding and clustering), the permutation distribution obtained by retaining (for each permutation) the largest positive cluster-based test statistic, will be exactly mirrored by a permutation distribution of the largest negative cluster-based test statistics. I would assume that this justifies the idea to use the absolute cluster statistics and test these against its permutation distribution.

However, in reality these permutation distributions will not necessarily be based on test statistics that 'behave nicely (i.e. symmetrically) around 0'. For that reason, and this is how it is done in the code, a two-sided (cluster-based) permutation test actually compares the observed test statistics with 2 permutation distributions, obtained by taking the most extreme (positive and negative) permuted test statistics.

Best wishes,
Jan-Mathijs




On 12 Jun 2025, at 16:08, Oliver Hermann via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hi all,

I wonder if anyone can clear up some confusion regarding evaluation of summed-cluster statistics following permutation.

Specifically, as of Maris and Oostenveld (2007), evaluation is performed for all clusters, based on the permutation distribution of the max (absolute) cluster statistic, see "It is important to note that the p-values for all eight clusters are calculated under the permutation distribution of the maximum (absolute value) cluster-level statistic and not under the permutation distribution of the second largest, third largest, etc."

However, when I look at the guide at https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fieldtriptoolbox.org%2Ftutorial%2Fstats%2Fcluster_permutation_timelock%2F&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C14fb2dd67cc443c1d01208ddaca9eec6%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638856568050624143%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=hpLhzLXr2yG27mhl%2BlOdy5gNFo2Sieo46csraRk0xJs%3D&reserved=0

the following recommendation is given:  And third, the critical value for the cluster-level test statistic (determined by cfg.alpha; see further) is now two-sided: negative cluster-level statistics must be compared with the negative critical value, and positive cluster-level statistics must be compared with the positive critical value.

To me, these two statements seem contradictory. Please could someone clarify the correct approach (I assume the latter) and why the discrepancy?

Thanks,
Oliver





-------------------
Oliver Hermann
PhD Researcher
Department of Psychology
University of Bath


_______________________________________________
fieldtrip mailing list
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C14fb2dd67cc443c1d01208ddaca9eec6%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638856568050664752%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rmF08lwvEx72rP6Selmr5eG8Enh6yRSfuBgmPRAqtoE%3D&reserved=0
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C14fb2dd67cc443c1d01208ddaca9eec6%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638856568050682941%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RZ4oo3swKZek85veC9iUJihz2GkKduEpSBTMfjC9Ufg%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20250616/9dad1d91/attachment.htm>