[FieldTrip] Statistical testing for classification metrics

Schoffelen, J.M. (Jan Mathijs) janmathijs.schoffelen at donders.ru.nl
Tue May 24 08:36:27 CEST 2022

Hi Elif,

Cluster-based permutation tests can be done using ft_timelock/freq/sourcestatistics, using ‘montecarlo’ as a method. If you use ‘mvpa’ as a method, you won’t be able to get a multiple comparison correct p-value AND a classification result in a single shot.

What you can do, though, is to obtain some performance metric per subject, using the mvpa functionality, and then come up with a meaningful second-level test that performs statistical inference of the performance metric across subjects, using the clustering philosophy.

Best wishes,

On 23 May 2022, at 16:29, Elif Tülay via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Dear fieldtrip community

I have a question about how I can perform statistical testing for selected metrics (such as for accuracy). Actually I want to obtain p-values for my accuracy results via cluster-based permutation test.
When I check the link below
I can not see anything about it. But it says
The function supports cross-validation, searchlight analysis, generalization, nested preprocessing, a variety of classification and regression metrics, as well as statistical testing of these metrics.

Here is some part of my script;
cfg.method          = 'mvpa';
cfg.design          = [ones(length(GROUP1),1); 2*ones(length(GROUP2),1)];
cfg.features        = 'chan';
cfg.mvpa            = [];
cfg.mvpa.classifier = 'lda';
cfg.mvpa.metric     = 'accuracy';
cfg.mvpa.k          = 10;

stat = ft_freqstatistics (cfg, data1, data2, data3, ...) where the data obtained from ft_freqanalysis for each subject.

Thank you in advance for your consideration. I am looking forward to hearing from you.

Best regards,

Elif Tülay

fieldtrip mailing list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20220524/abef2567/attachment.htm>

More information about the fieldtrip mailing list