[FieldTrip] time windows and number of channels in cluster based permutation tests

[Michele Scaltritti]

Dear Stephen,

thank you so much for the clarification. Indeed, I only thought about how
the number of channels/timepoints affects the "real" clusters, without
considering the implications for permutations. Thank you again for your
help, I really appreciate it.

On Fri, Feb 17, 2017 at 12:49 PM, Stephen Politzer-Ahles <
politzerahless at gmail.com> wrote:

>
> Dear Michele,
>
> The general idea is, when you get a p-value out of the cluster test, it's
> evaluated against a permutation distribution---the 'real' cluster is
> getting compared against all the fake clusters that pop up when you shuffle
> the data randomly. Roughly speaking, when you analyze more timepoints and
> more channels, there's more chances for random nonsense clusters to pop up
> in each permutation of randomly shuffled data; thus, there's more chances
> to get random nonsense clusters that have a bigger test statistic than your
> real cluster. Since the p-value is just the proportion of random clusters
> that have a bigger test statistic than the real cluster, testing more
> timepoints and/or more channels means you might end up with a higher
> p-value.
>
> Anyway, the selection of channels and time ranges has to be chosen *a
> priori*, not based on the data (choosing which window to analyze after
> you've seen where the effect is would be a non-independent / double-dipping
> analysis), so since you seem to have found a significant effect based on
> the samples you had identified a priori, I don't think you have to worry
> about the whole-head analysis.
>
> Best,
> Steve
>
> ---
> Stephen Politzer-Ahles
> The Hong Kong Polytechnic University
> Department of Chinese and Bilingual Studies
> http://www.mypolyuweb.hk/~sjpolit/
> <http://www.nyu.edu/projects/politzer-ahles/>
>
> On Fri, Feb 17, 2017 at 7:00 PM, <fieldtrip-request at science.ru.nl> wrote:
>
>> Send fieldtrip mailing list submissions to
>>         fieldtrip at science.ru.nl
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>> or, via email, send a message with subject or body 'help' to
>>         fieldtrip-request at science.ru.nl
>>
>> You can reach the person managing the list at
>>         fieldtrip-owner at science.ru.nl
>>
>>
>
>>
>> Message: 3
>> Date: Fri, 17 Feb 2017 10:33:52 +0100
>> From: Michele Scaltritti <michele.scaltritti at gmail.com>
>> To: fieldtrip at science.ru.nl
>> Subject: [FieldTrip] time windows and number of channels in cluster
>>         based   permutation tests
>> Message-ID:
>>         <CABGEo=TM-TGqh3bQubOD_29z-cxQ93grvXQHNMo70zqLcpjjEg at mail.
>> gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Dear FieldTrip users,
>>
>> I have a rather general question regarding cluster-based permutation test.
>>
>> In the tutorial for Cluster-based permutation tests on event related
>> fields, I read that sensitivity of the analysis depends on the length of
>> the time interval that is analyzed. Elsewhere, I found a similar reasoning
>> regarding channels, in the sense that it the exclusion of the sensors
>> where
>> no effect is likely to be present should yield a more sensitive analysis.
>>
>> In my experiment, I analyzed a limited set of central electrodes (where
>> previous evidence suggests that the difference should be located) and
>> found
>> a significant positive cluster, and a (smaller) significant negative one.
>> When doing the same analysis considering all the electrodes, no
>> significant
>> negative cluster is found, just a positive one. Among other things, this
>> may reflect a different sensitivity as a function of the number of
>> channels
>> considered in the analysis. I explored a bit the different parameters
>> (such
>> as neighbors channels definition), but I did not identify the source of
>> the
>> difference between the analyses.
>>
>> Leaving aside my specific case, in more general terms I cannot understand
>> how sensitivity is increased by focusing on a smaller time-windows, or on
>> a
>> specific set of electrodes. After all, the number of comparisons we make
>> should not affect the number of samples surpassing the threshold, nor the
>> clustering as a function of the spatial and temporal adjacency of these
>> supra-theshold samples. I understand that the same cluster may vary when
>> other samples are considered in the analyses (for example, samples from
>> other electrodes may get included), but again, I am failing to see how
>> this
>> may be related to the sensitivity of the analysis. The cluster-level
>> statistics, for example, should not be weakened by this.
>>
>> I apologize if this message underlies any gross misunderstanding.
>>
>> Michele Scaltritti
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachment
>> s/20170217/c376010b/attachment-0001.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>
>> End of fieldtrip Digest, Vol 75, Issue 19
>> *****************************************
>>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20170217/4af473d9/attachment.html>