Freqstatistics Yields Zero Significant Clusters?

[Erick Britis Ortiz]

Dear fieldtrip users,
(with special thanks to Michael Wibral's assessment)


Following up on the discussion, we have also tried to use the cluster
option for multiple comparisons correction in a previous study and
thought that our results were not significant. Now, we tried again with
1000 randomizations and cfg.correctm = 'cluster', to no avail. Since we
are interested in p<0.05, 1000 iterations seem large enough, but most of
the probabilities calculated are near or equal to 100%. Further testing
with 10000 permutations yielded similar results.

We are fairly sure that there is something there, because calculating
analytically we see big clusters that extend in a consistent way to
neighboring frequency and time bins, even at low p thresholds (<10^-4).
Since Bonferroni is over-strict, Montecarlo is the option of choice for
a corrected result that we will trust. But now something seems off.

Our data is 275 channels x 13 time x 31 frequency bins. The attached
graphics show:
Fig.1: old study, histogram of montecarlo results
Fig.2: present study, histogram of analytical results
Fig.3: present study, histogram of montecarlo results
Fig.4: present study, imagesc of montecarlo results (sum with channels
with prob.<0.001
for each time and freq.)

Questions:
1) What do these results mean?
2) How to get a mask for a new p threshold without having to recalculate
everything? (the probabilities are valid, but since the thresholds are
dynamic, we cannot make a meaningful comparison with a fixed value)

Any ideas will be appreciated!


Thank you in advance,
Erick Britis Ortiz
MEG-Zentrum, University of Tübingen


Michael Wibral wrote:
> Hi Charles,
>
> from your output: ... computing statistic 100 from 100 performing
> Bonferoni correction for multiple comparisons ...
>
> it seems that you're only computing 100 randomizations. It follows
> that the best p-value you could EVER get is 0.01. You then do
> bonferroni correction (not the cluster based correction you
> intended!). So if you set an alpha of 0.9 and divide this by - say -
> 2000 for your bonferoni corrcetion you alpha is 0.9/2000=0.00045. You
> see that you'll never reach this limit given that you do only 100
> randomizations and by defibition cannot get below p=0.01. In addition
> you won't get clusters when you use Bonferroni.
>
> I suggest using:
>
> cfg.numrandomization = 5000; cfg.corectm='cluster'; % or 'fdr'
>
>
> Good Luck! Michael




----------------------------------
The aim of this list is to facilitate the discussion between users of the FieldTrip  toolbox, to share experiences and to discuss new ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/archives/fieldtrip.html and http://www.ru.nl/neuroimaging/fieldtrip.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Fig1_Study1_hist_mc.png
Type: image/png
Size: 2650 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20090605/558ada64/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Fig2_Study2_hist_an.png
Type: image/png
Size: 3083 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20090605/558ada64/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Fig3_Study2_hist_mc.png
Type: image/png
Size: 2853 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20090605/558ada64/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Fig4_Study2_montecarlo_imagesc.png
Type: image/png
Size: 4950 bytes
Desc: not available
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20090605/558ada64/attachment-0011.png>