[FieldTrip] permutation test: multiple comparison correction for the p-values

[Steinmann, Iris]

Hi Phillipp,

The idea was, that I don’t have to recalculate my statistics if I just want to look at the results with a higher threshold. So, thanks a lot for your additional information, it helps me to think over my analysis in the first place …

thanks, have a nice day!
Iris



From: fieldtrip-bounces at science.ru.nl [mailto:fieldtrip-bounces at science.ru.nl] On Behalf Of Philipp Ruhnau
Sent: Freitag, 13. November 2015 17:29
To: FieldTrip discussion list
Subject: Re: [FieldTrip] permutation test: multiple comparison correction for the p-values

Hey Iris,
you could use matlabs inbuilt function (mafdr) to produce corrected p-values,

then again if you just want to check your outcome with a higher threshold you might as well feed that into the fieldtrip statistics function with cfg.alpha

It is possible to estimate corrected p-values by hand, the logic being the following
as in Julian’s example for 5 tests bonferroni adjustment is alpha_level/5. but you could also multiply your p-value with 5 and keep the original threshold.

similarly for a simple Benjamini-Hochberg fdr if you have a vector with p-values (original_pvals) you do

pvals = sort(original_pvals); % sort

% correct p-vals by hand
n = numel(pvals);
cor_pvals = pvals .* (n./(1:n));


so following the fdr logic you correct each p-val by its own factor.
but note that this is not the fdr correction that is implemented in fieldtrip (see fieldtrips fdr.m for the reference)
and that some p-values now can exceed 1…
Also I think Arnaud Delorme’s fdr function in eeglab had the option to estimate corrected p-values (but I haven’t checked for quite a while)

but yeah, I’m with Julian on this one. as the t-value doesn’t change it certainly makes more sense to quote fdr-corrected p < 0.05 then an exact but corrected p-value

hope this helps
philipp
On 13 Nov 2015, at 16:41, Steinmann, Iris <iris.steinmann at med.uni-goettingen.de<mailto:iris.steinmann at med.uni-goettingen.de>> wrote:

Hi Julian,

Thanks a lot for your advice! But I wane keep all the p-values. Maybe to check what will happen when I would go with a more liberal threshold (maybe p = 0.06 instead of p = 0.05). So, is there any possibility to correct my p-values directly instead of only the mask?

Best wishes!
Iris



From: fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl> [mailto:fieldtrip-bounces at science.ru.nl] On Behalf Of Julian Keil
Sent: Freitag, 13. November 2015 09:51
To: FieldTrip discussion list
Subject: Re: [FieldTrip] permutation test: multiple comparison correction for the p-values

Hi Iris,

I'm not sure what you want to do with the p-values in the following, but you can multiply your (FDR-corrected) mask with the prob.
This would look like this: stat_TF.prob_new = stat_TF.prob .* stat_TF.mask;

Now you only have the significant p-values left in your prob_new-field, everything else is set to 0.

Hope this helps

Julian

********************
Dr. Julian Keil

AG Multisensorische Integration
Psychiatrische Universitätsklinik
der Charité im St. Hedwig-Krankenhaus
Große Hamburger Straße 5-11, Raum A007
10115 Berlin

Telefon: +49-30-2311-1879
Fax:        +49-30-2311-2209
http://psy-ccm.charite.de/forschung/bildgebung/ag_multisensorische_integration

Am 13.11.2015 um 09:41 schrieb Steinmann, Iris:



Hi everybody,

I' m using the 'ft_freqstatistics' function to find significant differences between two time-frequency spectra
(see the used options down below).
While fieldtrip calculates the permutation test it throws the following information:

"performing FDR correction for multiple comparisons
the returned probabilities are uncorrected, the thresholded mask is corrected"

Since I'm using the p-values for the following analysis and not the thresholded mask, I was wondering how I
get my p-values corrected for multiple comparison?

Does anyone has an idea? Thanks in advance!


I used the following options:

cfg = [];
cfg.channel     = 'all';
cfg.latency     = [2.3 2.8];
cfg.avgoverchan = 'yes';
cfg.avgovertime = 'no';
cfg.frequency = [9 14];
cfg.parameter   = 'powspctrm';
cfg.alpha = 0.05;
cfg.tail = 0;
cfg.correctm = 'fdr';
cfg.correcttail = 'prob';
cfg.ivar = 1;
cfg.statistic = 'ft_statfun_indepsamplesT';
cfg.method = 'montecarlo';
cfg.design = design; % defined in at the beginning of the function
cfg.numrandomization = 1000;

stat_TF = ft_freqstatistics(cfg, TF_remove, TF_noremove)
_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl<mailto:fieldtrip at donders.ru.nl>
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl<mailto:fieldtrip at donders.ru.nl>
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20151116/0b917c48/attachment-0002.html>