[FieldTrip] Theoretical question about cluster-based permutation tests

[Snijders, T.M. (Tineke)]

Hi Anne,

All the tests you do initially and the threshold that is used to select samples for clustering do not affect the FA because they only determine whether a sample-point will be included in a cluster, they do not determine significance.
In the end you are only doing ONE test to determine significance, and that is to test whether such a large cluster could have occurred by chance. And you do that by the permutation test, by permuting the data and looking at how large the clusters get when the data would have been random. Then you compare your cluster-statistic to the random distribution of cluster-statistics, and if that large a cluster only happens in <5% of the cases you call it significant. So that's only one test.
Note that also in the random data there will be clusters formed of which the sample-points survive the initial thresholds, and if you use e.g. a lower threshold for selecting samples for clustering, this means you will get larger clusters. But, then also your 'real' cluster should be larger to survive the 'significance' question: "how probable is it to see such a large cluster in random data (using these settings)?"

Hope this helps,

Cheers,
Tineke


________________________________
From: fieldtrip-bounces at science.ru.nl [fieldtrip-bounces at science.ru.nl] on behalf of Anne Mickan [amickan1990 at gmail.com]
Sent: Friday, June 17, 2016 6:43 PM
To: fieldtrip at science.ru.nl
Subject: [FieldTrip] Theoretical question about cluster-based permutation tests

Dear all,

I have a theoretical question about cluster-based permutation tests. I've been reading through the website and through Maris & Oostenveld (2007) and I watched the video, all of which left me with the feeling I have a good understanding of what the permutation tests is doing (I've also implemented it successfully for my own data).

However, I don't yet entirely understand how this test statistic avoids the multiple comparison problem. Particularly I'm wondering how it is that all the tests done initially on a sample-by-sample-basis (step 1 as described in Maris & Oostenveld, p. 180) and the threshold that is used to select samples for clustering (step 2) does not affect the FA. It is referred to a later section, which, however, does not clear it up for me. So I was hoping to get another explanation to fully understand this issue.

Thanks a lot in advance.

Best,
Anne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20160621/32415200/attachment-0002.html>