[FieldTrip] Statistics question alpha cutoff adjustment

[Paul Steinfath]

Dear Fieldtrippers,

I have a question regarding multiple comparison correction in a case of permutation based statistics.
In the field of Heartbeat Evoked Potential (HEP) research, it is a common practice to perform a “surrogate heartbeat analysis” to assess if effects are truly locked to the heartbeat, or merely a result of fluctuations in ongoing brain activity.

The procedure can work like this:
1. Compare HEPs between condition A and condition B using cluster based permutation t-test and identify any significant clusters.

2. Create surrogate data by shuffling the HEP onset triggers per condition and repeating the cluster based permutation test >100 times.
Each time, keeping the sum(t) value of the largest clusters.

3. Compare the sum(t) value of the original comparison using real HEPs to the distribution of maximum sum(t) values obtained from surrogate data.

For reference, here is an example study describing the approach:
https://pubmed.ncbi.nlm.nih.gov/30738205/

Now, let's say I initially find two HEP effects, one in positive and one in negative direction (could be two directions of the same dipole).
I would compare the real positive sum(t) against the surrogate positive max sum(t) and vice versa for the negative direction.
 
Should my alpha cutoff in this final comparison of the real sum(t) HEP effect against surrogate sum(t) effects stay at 5%, or should I adjust it because I test both, positive and negative directions?

I am getting a bit confused in this multi-level statistic approach…

Thank you very much and all the best
Paul