[FieldTrip] Threshold free cluster enhancement for a Repeated Measures design

Tue May 14 09:25:15 CEST 2024

Hi Ash,

Thank you so much for your response! I have some follow-up questions:
1)    True, using F-statistics makes more sense for my design. But to clarify, you are recommending using F-statistics (ft_statfun_depsamplesFunivariate) and then doing pairwise tests for the main effects to get around the fact that the depsamplesF functions only take 1xN factorial design? Or do you mean I should continue using T-statistics, and do the pairwise tests?

Well, I don’t suggest anything. I am not sure whether there’s a right or wrong here. Both F-statistics as T-statistics can be used as test-statistic. I think that overall one should keep an eye on the ball, i.e. the scientific question that one wants to answer. The risk exists that a discussion leads to conclusions due to technical/algorithmic constraints (e.g. let’s do T-stats because there is not (yet) an existing implementation for the F-statistics for a higher order design -> which is not the way forward: if there’s a scientific reason that such a function is needed (which is not what I am saying here), it should just be implemented). I don’t know the specific details of your experimental design, but it seems as if the 2x3 can be split into twice a 2x2 design, where the no cycling condition serves as a ‘baseline’ for the 2 cycling conditions. If that is the case, then an overall (hypothetical NxM F-test will lead to a very ‘easy’ significant effect, due to the presence of the baseline. In this particular case - but again, that depends on the taste of you and your supervisory team - you could consider to stick to FieldTrip for the non-parametric statistics, and test
 1) the main effects (+interaction) of Rest versus the (average of) the cycling conditions,

and
 2) the main effects + interaction using the 2x2 cycling conditions.

I think that in principle this is defendible and valid.

2)    Thank you for pointing out that bit of documentation about testing interactions effects, I totally missed that before. All things considered now I am leaning towards not getting into interaction effects, since it doesn’t seem to be recommended for my design and especially if I end up using F-statistics, these will not inform me about the pattern in the data responsible for the interaction effect. But I am still a bit uncertain as to what I should do for the main effects - should I just compare both main effects, of cycling (averaging across vis and invis, and comparing R, L, H) and of perception (averaging across R/L/H and comparing vis versus invis), just like in a two-way RM-ANOVA and leave it at that? What would you recommend?

3)    I was clearly confusing cluster based and TFCE approaches. Also, I apologise for my poor choice of wording – I have read Benedikt Ehinger’s post of TFCE as well as the FieldTrip page on how not to interpret results from cluster-based permutation tests, so I do understand it doesn’t make sense to talk about significant clusters. But one thing I am uncertain about – will I be able to plot an effect topographically using ft_clusterplot when I run an analysis with TFCE? Or is that only possible for other methods of correcting for multiple comparisons in cluster based permutation analyses? I am asking this because the TFCE example on the FieldTrip website is only for on one channel (https://www.fieldtriptoolbox.org/example/threshold_free_cluster_enhancement/<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fieldtriptoolbox.org%2Fexample%2Fthreshold_free_cluster_enhancement%2F&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Cb45c619ba1d14f79ebfa08dc73e70039%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638512683174870590%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=9hI%2BQESPnrNne9ZJl6RRgMhyjQdHjuCmbGRCXJWHB9s%3D&reserved=0>) and I was wondering what it looks like scaled up for multiple channels. I actually tried running the TFCE analysis for all channels on the dataset used in the example but I couldn’t plot the effect using ft_clusterplot. But maybe, this is something that doesn’t even make sense to attempt?

I have not used TFCE for a long time, but ft_clusterplot in general can only be used for the ’traditional’ clusters. And even then, I wouldn’t use it. The function is there for historical reasons, but I must say that I don’t find its output very informative. If you managed to run the statistics using tfce, then you could perhaps use ft_topo/singleplotER/TFR for the ’stat’ parameter which will visualize the test statistics. You could then mask the ’stat’ field with the stat.mask, or stat.prob, to highlight the data points with low p-values. Also, you could consider doing something that e.g. is used in CosmoMVPA-toolbox, where the output p-value is converted back into a Z-statistic. I think that this latter step is motivated by the fact that TFCE-values can have strange distributions in terms of their values.

4)    The reason I was asking about this analysis without a prior constraints was also to further my understanding about these analyses. I am puzzled by the fact the same test in not significant on data obtained for very similar paradigms with a very similar number of subjects. So in essence, if in the previous experiment, the cluster-based tests were not significant and we didn’t have a prior constraints to limit our analysis then, we would have likely missed the effect in the time window and channels that we are now using as a prior constraints in the current experiment.

This is also relevant to me because I will also be performing a similar cluster based analysis for other parameters (e.g. heart-evoked potential amplitudes) on the same dataset, where I don’t have any a priori topographical constraints, and I am wondering if there are additional tests I should perform for fear of missing out on some effect that is not evident in cluster based tests?

I am not sure whether I understand this well. I think the logic would be: if based on previous literature, and or experiments there’s prior information (i.e. latencies/frequency ranges) to include into the statistical approach, this might boost the sensitivity. On the other hand, if there are earlier experiments with null-findings based on cluster-based permutations without spatiotemporalspectral constraints, and no other literature then you should have good arguments to now all of a sudden use additional priors. If those priors happen to be poorly chosen, then that’s too bad.
However, if there are good reasons to assume that you were underpowered in the past (i.e. motivating the null finding then), one should repeat the experiment with more power (i.e. more subjects, and/or more data per subject), and might consider using prior constraints based on the previous experiment, where the prior constraints are not strictly inspired by data points that survive the multiple comparison thresholding, but inspired by another defendible heuristic, e.g. the non-correct p-values.

Best wishes,
Jan-Mathijs

Best,
Ash

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Cb45c619ba1d14f79ebfa08dc73e70039%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638512683175026873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qFLm5Nysons8hbvqs6vJajLBmx%2F2dcd%2B829Xdb0vl%2BM%3D&reserved=0>
https://doi.org/10.1371/journal.pcbi.1002202<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Cb45c619ba1d14f79ebfa08dc73e70039%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638512683175026873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=7rvR7rpDucq%2Fj8Wb6ZfFq5I78rX1xp78PKEG4ATTM2I%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20240514/e48afadc/attachment.htm>