[FieldTrip] permutation test: multiple comparison correction for the p-values

[Philipp Ruhnau]

Hey Iris,
you could use matlabs inbuilt function (mafdr) to produce corrected p-values,

then again if you just want to check your outcome with a higher threshold you might as well feed that into the fieldtrip statistics function with cfg.alpha

It is possible to estimate corrected p-values by hand, the logic being the following 
as in Julian’s example for 5 tests bonferroni adjustment is alpha_level/5. but you could also multiply your p-value with 5 and keep the original threshold.

similarly for a simple Benjamini-Hochberg fdr if you have a vector with p-values (original_pvals) you do

pvals = sort(original_pvals); % sort

% correct p-vals by hand
n = numel(pvals);
cor_pvals = pvals .* (n./(1:n));


so following the fdr logic you correct each p-val by its own factor. 
but note that this is not the fdr correction that is implemented in fieldtrip (see fieldtrips fdr.m for the reference)
and that some p-values now can exceed 1…
Also I think Arnaud Delorme’s fdr function in eeglab had the option to estimate corrected p-values (but I haven’t checked for quite a while)

but yeah, I’m with Julian on this one. as the t-value doesn’t change it certainly makes more sense to quote fdr-corrected p < 0.05 then an exact but corrected p-value

hope this helps
philipp
> On 13 Nov 2015, at 16:41, Steinmann, Iris <iris.steinmann at med.uni-goettingen.de> wrote:
> 
> Hi Julian,
>  
> Thanks a lot for your advice! But I wane keep all the p-values. Maybe to check what will happen when I would go with a more liberal threshold (maybe p = 0.06 instead of p = 0.05). So, is there any possibility to correct my p-values directly instead of only the mask?
>  
> Best wishes!
> Iris
>  
>  
>  
> From: fieldtrip-bounces at science.ru.nl [mailto:fieldtrip-bounces at science.ru.nl] On Behalf Of Julian Keil
> Sent: Freitag, 13. November 2015 09:51
> To: FieldTrip discussion list
> Subject: Re: [FieldTrip] permutation test: multiple comparison correction for the p-values
>  
> Hi Iris,
>  
> I'm not sure what you want to do with the p-values in the following, but you can multiply your (FDR-corrected) mask with the prob.
> This would look like this: stat_TF.prob_new = stat_TF.prob .* stat_TF.mask;
>  
> Now you only have the significant p-values left in your prob_new-field, everything else is set to 0.
>  
> Hope this helps
>  
> Julian
>  
> ********************
> Dr. Julian Keil
>  
> AG Multisensorische Integration
> Psychiatrische Universitätsklinik
> der Charité im St. Hedwig-Krankenhaus
> Große Hamburger Straße 5-11, Raum A007
> 10115 Berlin
> 
> Telefon: +49-30-2311-1879
> Fax:        +49-30-2311-2209 
> http://psy-ccm.charite.de/forschung/bildgebung/ag_multisensorische_integration <http://psy-ccm.charite.de/forschung/bildgebung/ag_multisensorische_integration>
>  
> Am 13.11.2015 um 09:41 schrieb Steinmann, Iris:
> 
> 
> Hi everybody,
>  
> I' m using the 'ft_freqstatistics' function to find significant differences between two time-frequency spectra
> (see the used options down below).
> While fieldtrip calculates the permutation test it throws the following information:
>  
> "performing FDR correction for multiple comparisons
> the returned probabilities are uncorrected, the thresholded mask is corrected"
>  
> Since I'm using the p-values for the following analysis and not the thresholded mask, I was wondering how I
> get my p-values corrected for multiple comparison?
>  
> Does anyone has an idea? Thanks in advance!
>  
>  
> I used the following options:
>  
> cfg = [];
> cfg.channel     = 'all';
> cfg.latency     = [2.3 2.8];
> cfg.avgoverchan = 'yes';
> cfg.avgovertime = 'no';
> cfg.frequency = [9 14];
> cfg.parameter   = 'powspctrm';
> cfg.alpha = 0.05;
> cfg.tail = 0;
> cfg.correctm = 'fdr';
> cfg.correcttail = 'prob';
> cfg.ivar = 1;     
> cfg.statistic = 'ft_statfun_indepsamplesT';
> cfg.method = 'montecarlo';            
> cfg.design = design; % defined in at the beginning of the function
> cfg.numrandomization = 1000;
>  
> stat_TF = ft_freqstatistics(cfg, TF_remove, TF_noremove)
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl <mailto:fieldtrip at donders.ru.nl>
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip <http://mailman.science.ru.nl/mailman/listinfo/fieldtrip>
>  
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20151113/1daa6e7c/attachment-0002.html>