[FieldTrip] Problem with permutation testing and FDR correction

[Michael Wibral]

Dear Fieldtrip users,

I think we detected an error with FDR correction and permutation testing. When increasing the number of permutations, the number of significant voxels goes DOWN, on the other hand when decreasing the number of permutations the number of significant voxels goes up. In my opinion the relationship should be the other way round. The theoretical background is as follows:

With FDR correction, the best p-value should survive bonferroni correction (if I am not completely mistaken here), the threshold  for the other p-values is then decreased successively.
Hence, the p-value assigned to the best (most significant) statistical result plays a crucial role here. This best p-value in permutation tests can never be better than 1/numpermutations, i.e. when I do ten permutations, the best p I can possibly get is 0.1 EVEN IF ALL PERMUTED VALUES ARE LESS EXTREME. So to test with FDR at 275 sensors and 1 timepoint at a threshold of 0.05 we need for anything to get significant a p-value of 0.05/275 = 0.000181818.... to be able to reach this in the best case (remebering that p-values can never be better than  1/numpermutations) we need at least (0.000181818....)^-1 = 5500 permutations. In other words with anything less than this number of permutations we should not be able to get any significances. However, we do in fact get  alot of significant values at least in the fieldtrip versions tested up to 16th of January, e.g. in a freqstatistics test on 275 sensors, 50 frequencies and 26 timepoints I get 20760 significant voxels using only 10 permutations (!!). I assume that in the stats module the p-value is simply taken as the fraction of permutations that was more extreme than the actual value. This is correct as long as this fraction is not 0. In the case of a 0 fraction, however, this "0" should be replaced by "1/numpermutations", otherwise you get everything signifcant by just using 10 permutations. An alternative would be to issue an ERROR that the number of permutations is insufficient to perform the desired test with fdr correction.


Example: for 4300 source points at 0.05 the number of permutations should at least be (0.05/4300)^-1=86000.

For now one should compute the p-value for a bonferroni correction manually, invert this value and take the resulting number as the number of permutations to be at least mathematically on the safe side (practically it seems to be advisable to multiply by another factor of 100 to have stable results, e.g 2000 permutations for uncorrceted testing at 0.05)

Please disregard this mail if you are sure that this behaviour has been fixed in the latest fieldtrip (past 16th of January) versions.

Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Michael Wibral.vcf
Type: text/x-vcard
Size: 637 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20110203/d3fb9a88/attachment-0002.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Michael Wibral.vcf
Type: text/x-vcard
Size: 637 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20110203/d3fb9a88/attachment-0003.vcf>