[FieldTrip] Correcting for multiple cluster analyses?

[Schoffelen, J.M. (Jan Mathijs)]

Hi David,

If you want to be strict about the multiple tests you are performing, you need to Bonferroni correct for the number of tests, not for the number of clusters as such. The clusters are a ‘byproduct’ of the inferential procedure, and are a means to reduce the number of comparisons per test to 1. Actually, you are right that in 2-sided testing you are performing 2 tests, one for each side (i.e. testing the largest positive and negative cluster in the observed data against their respective randomization distributions. The interpretation of the p-values associated with the most extreme clusters (i.e. the ones that are associated with the actual inferential decision), depends on the settings of the cfg.correcttail option. See https://www.fieldtriptoolbox.org/faq/why_should_i_use_the_cfg.correcttail_option_when_using_statistics_montecarlo/<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fieldtriptoolbox.org%2Ffaq%2Fwhy_should_i_use_the_cfg.correcttail_option_when_using_statistics_montecarlo%2F&data=05%7C01%7Cfieldtrip%40science.ru.nl%7Cc44000c71e984799a1bc08dbd39e426b%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638336448940467345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Tkk%2BWcEZeYNlJ2R3mx5262sL94GJ2CfPEmX2UN8YHG8%3D&reserved=0> for more information.

I hope this helps,
Jan-Mathijs



On 17 Oct 2023, at 19:47, David Prete via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hello Everyone,

I asked this question last week, but it didn't show up in my Fieldtrip
emails so I'm trying to send this again to see if it works this time.
Sorry if you're seeing this twice.

I have run multiple cluster-based permutation analyses and I am
wondering if I need to correct for multiple comparisons and how
exactly would I do that? I know there was a previous thread that
suggested using Bonferroni correction but not how to apply the
correction. If I run the cluster analysis 9 times in total based on
different time windows and comparing different conditions, would I
divide 0.05 by 9 to get a corrected p-value of  0.00556? Or do I need
to correct based on the total number of clusters found from the 9
times I ran the analysis? So, for example let's say each time I ran
the analysis I found a positive and negative cluster leading to 18
different p-values, would the correction be 0.05 divided by 18?

Thank you,
David Prete (he/him)
Ph.D. Candidate
McMaster University


_______________________________________________
fieldtrip mailing list
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C01%7Cjanmathijs.schoffelen%40donders.ru.nl%7Cc6ac3241c5e3439f7d7808dbcf3c5540%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638331630260706725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=THIhFBgc6VtNF%2BxCVK5gPFo4dd1oG%2FF2UvjsFP6Zg1o%3D&reserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C01%7Cfieldtrip%40science.ru.nl%7Cc44000c71e984799a1bc08dbd39e426b%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638336448940467345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5qvQIDcHAiqOH7MiDksSfdJOnZc5yiTSQemrHWeP0z0%3D&reserved=0>
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C01%7Cjanmathijs.schoffelen%40donders.ru.nl%7Cc6ac3241c5e3439f7d7808dbcf3c5540%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638331630260706725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cZJOHWv%2FQPqbZQoQV0sSlryklBJOZpjggCL24PpGjLs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20231023/046947cc/attachment.htm>