[FieldTrip] Statistical testing for classification metrics
Schoffelen, J.M. (Jan Mathijs)
janmathijs.schoffelen at donders.ru.nl
Tue May 24 08:36:27 CEST 2022
Cluster-based permutation tests can be done using ft_timelock/freq/sourcestatistics, using ‘montecarlo’ as a method. If you use ‘mvpa’ as a method, you won’t be able to get a multiple comparison correct p-value AND a classification result in a single shot.
What you can do, though, is to obtain some performance metric per subject, using the mvpa functionality, and then come up with a meaningful second-level test that performs statistical inference of the performance metric across subjects, using the clustering philosophy.
On 23 May 2022, at 16:29, Elif Tülay via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:
Dear fieldtrip community
I have a question about how I can perform statistical testing for selected metrics (such as for accuracy). Actually I want to obtain p-values for my accuracy results via cluster-based permutation test.
When I check the link below
I can not see anything about it. But it says
The function supports cross-validation, searchlight analysis, generalization, nested preprocessing, a variety of classification and regression metrics, as well as statistical testing of these metrics.
Here is some part of my script;
cfg.method = 'mvpa';
cfg.design = [ones(length(GROUP1),1); 2*ones(length(GROUP2),1)];
cfg.features = 'chan';
cfg.mvpa = ;
cfg.mvpa.classifier = 'lda';
cfg.mvpa.metric = 'accuracy';
cfg.mvpa.k = 10;
stat = ft_freqstatistics (cfg, data1, data2, data3, ...) where the data obtained from ft_freqanalysis for each subject.
Thank you in advance for your consideration. I am looking forward to hearing from you.
fieldtrip mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the fieldtrip