[FieldTrip] Statistics question alpha cutoff adjustment

[Schoffelen, J.M. (Jan Mathijs)]

Hi Paul,

You asked:

> Should my alpha cutoff in this final comparison of the real sum(t) HEP effect against surrogate sum(t) effects stay at 5%, or should I adjust it because I test both, positive and negative directions?

The answer: yes, you should adjust the alpha level for a two-sided test.

The fact that a positive/negative going effect may have the same underlying dipolar source is physiological knowledge/assumption, which the statistical machinery does not know/care about.
If you want to optimally pool information across sensors (assuming you are working with MEG magnetometer data (or axial gradiometers) so that you can optimally leverage the spatial clustering heuristic, you may consider to do a synthetic planar gradient transform (followed by ft_combineplanar).

Best wishes,
Jan-Mathijs


> On 6 Aug 2024, at 07:37, Paul Steinfath via fieldtrip <fieldtrip at science.ru.nl> wrote:
>
> Dear Fieldtrippers,
>
> I have a question regarding multiple comparison correction in a case of permutation based statistics.
> In the field of Heartbeat Evoked Potential (HEP) research, it is a common practice to perform a “surrogate heartbeat analysis” to assess if effects are truly locked to the heartbeat, or merely a result of fluctuations in ongoing brain activity.
>
> The procedure can work like this:
> 1. Compare HEPs between condition A and condition B using cluster based permutation t-test and identify any significant clusters.
>
> 2. Create surrogate data by shuffling the HEP onset triggers per condition and repeating the cluster based permutation test >100 times.
> Each time, keeping the sum(t) value of the largest clusters.
>
> 3. Compare the sum(t) value of the original comparison using real HEPs to the distribution of maximum sum(t) values obtained from surrogate data.
>
> For reference, here is an example study describing the approach:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F30738205%2F&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C4356c9c7bd4f455c9c9008dcb77b204b%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638586986652184682%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=5JhmZcNU3Zee1tZrDZzE07DfKjYka8NPJwxhdmVNu2U%3D&reserved=0
>
> Now, let's say I initially find two HEP effects, one in positive and one in negative direction (could be two directions of the same dipole).
> I would compare the real positive sum(t) against the surrogate positive max sum(t) and vice versa for the negative direction.
>
> Should my alpha cutoff in this final comparison of the real sum(t) HEP effect against surrogate sum(t) effects stay at 5%, or should I adjust it because I test both, positive and negative directions?
>
> I am getting a bit confused in this multi-level statistic approach…
>
> Thank you very much and all the best
> Paul
>
> _______________________________________________
> fieldtrip mailing list
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C4356c9c7bd4f455c9c9008dcb77b204b%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638586986652184682%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wts9TIBygPETlD%2BNWjlR6nfp4wwJbaF8s2qeu9%2Bubqc%3D&reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C4356c9c7bd4f455c9c9008dcb77b204b%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638586986652184682%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vjQm0O7dhUvU2jUGe270amRvphJRDHtu4ItcqwRRmT8%3D&reserved=0