[FieldTrip] Statistical testing for classification metrics

[Schoffelen, J.M. (Jan Mathijs)]

Hi Elif,

Cluster-based permutation tests can be done using ft_timelock/freq/sourcestatistics, using ‘montecarlo’ as a method. If you use ‘mvpa’ as a method, you won’t be able to get a multiple comparison correct p-value AND a classification result in a single shot.

What you can do, though, is to obtain some performance metric per subject, using the mvpa functionality, and then come up with a meaningful second-level test that performs statistical inference of the performance metric across subjects, using the clustering philosophy.

Best wishes,
Jan-Mathijs


On 23 May 2022, at 16:29, Elif Tülay via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Dear fieldtrip community

I have a question about how I can perform statistical testing for selected metrics (such as for accuracy). Actually I want to obtain p-values for my accuracy results via cluster-based permutation test.
When I check the link below
https://github.com/fieldtrip/fieldtrip/blob/master/ft_statistics_mvpa.m<https://urldefense.com/v3/__https://github.com/fieldtrip/fieldtrip/blob/master/ft_statistics_mvpa.m__;!!HJOPV4FYYWzcc1jazlU!-mRuDeeFEAj91C2mt6bRmziZJszE2ZSUUxwWZMjrNc18GwC67yM1N3gOeAylbpXClko1JVWMsaJ-YpsA3ul_-5-14ZbjtY_lMkrg1w$>
I can not see anything about it. But it says
The function supports cross-validation, searchlight analysis, generalization, nested preprocessing, a variety of classification and regression metrics, as well as statistical testing of these metrics.

Here is some part of my script;
cfg.method          = 'mvpa';
cfg.design          = [ones(length(GROUP1),1); 2*ones(length(GROUP2),1)];
cfg.features        = 'chan';
cfg.mvpa            = [];
cfg.mvpa.classifier = 'lda';
cfg.mvpa.metric     = 'accuracy';
cfg.mvpa.k          = 10;

stat = ft_freqstatistics (cfg, data1, data2, data3, ...) where the data obtained from ft_freqanalysis for each subject.



Thank you in advance for your consideration. I am looking forward to hearing from you.



Best regards,


Elif Tülay

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!-mRuDeeFEAj91C2mt6bRmziZJszE2ZSUUxwWZMjrNc18GwC67yM1N3gOeAylbpXClko1JVWMsaJ-YpsA3ul_-5-14ZbjtY8VExj9YA$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20220524/abef2567/attachment.htm>