[FieldTrip] ROC_based permutation test

[Shen-Mou Hsu]

Dear Eelke,

Many thanks for your helpful and detailed answers. I think that there is an
error in the documentation about the configuration option for performing
ROC analysis. The correct one should be cfg.statistic = 'ft_statfun_roc'.
Otherwise, it will call the matlab built-in ROC function.

Meanwhile, just to clarify my concept about the ROC_based permutation
tests. In the initial stage, does the test calculate whether for every
sample, the area under ROC curve is significant from 0.5. In this sense,
should I specify cfg.clusteralpha = 0.5?

Best regards,

Shen-Mou Hsu


On Tue, Jun 11, 2013 at 4:18 PM, Eelke Spaak <eelke.spaak at donders.ru.nl>wrote:

> Dear Shen-Mou Hsu,
>
> With regards to your first question, I do not know the answer, so
> someone else might help you there.
>
> In response to your second question, regarding the error "could not
> determine the parametric critical value for clustering", this is
> caused by the value of cfg.clusterthreshold used. The default value
> there is 'parametric', meaning that the statistics routine will ask
> your 'statfun' to compute a parametric threshold for considering a
> (time/frequency/channel)-voxel a cluster-member candidate. This can be
> done by e.g. depsamplesT or indepsamplesT, as it is possible to
> analytically compute a T value corresponding to p < 0.05. However, in
> the case of the ROC statistic, no such parametric estimate can be
> computed (or perhaps it can be in some way, I don't know, but at least
> I know the FT implementation does not).
>
> Fortunately, the statistics routines also allow you to use a
> nonparametric threshold for cluster-member candidates, based on the
> generated distribution of the test statistic under the null
> hypothesis. To use this, simply specify cfg.clusterthreshold =
> 'nonparametric_individual' or cfg.clusterthreshold =
> 'nonparametric_common'. The difference between the two is that the
> former computes a threshold per voxel, and the latter uses the same
> threshold for all voxels. Which one is appropriate for you I don't
> know. (Good reasons for using 'nonparametric_individual' might be a
> strong variation of your test statistic with frequency. I know for a
> fact this is the case with certain quantifications of phase-amplitude
> coupling; these show much higher values in the low frequencies even
> when computed on noise.)
>
> Hope this helps.
>
> Best,
> Eelke
>
> On 11 June 2013 09:48, Shen-Mou Hsu <explena at gmail.com> wrote:
> > Dear all,
> >
> > I was trying to perform signle-trial ROC-based permutation tests using
> the
> > statfun_roc. However I encountered two questions and wondered if anyone
> > could kindly shed some light on the issues. First, is it necessary to
> > perform baseline normalization for each trial before the tests? Second,
> an
> > error message returned stating "Error using roc. Too many input
> arguments.
> > Error using ft_statistics_montecarlo (line 223) could not determine the
> > parametric critical value for clustering", after running the following
> > script:
> >
> > load (['t_RF_EpoRejDePow']); load (['t_RN_EpoRejDePow']);
> >
> > cfg = [];
> > cfg.channel     = 'MEG';
> > cfg.latency     = [-0.35 0.55];
> > cfg.frequency   = [8 12];
> > cfg.parameter   = 'powspctrm';
> > cfg.method      = 'montecarlo';
> > cfg.statistic   = 'roc';
> > cfg.alpha       = 0.025;
> > cfg.tail = 0;
> > cfg.correctm    = 'cluster';
> > cfg.clusteralpha = 0.05;
> > % cfg.correcttail = 'prob';
> > cfg.clustertail = 0;
> > cfg.numrandomization = 1000;
> > cfg.minnbchan        = 2;
> > cfg_neighb.method    = 'distance';
> > cfg.neighbours       = ft_prepare_neighbours(cfg_neighb,
> t_RN_EpoRejDePow);
> > cfg.logtransform = 'yes';
> >
> > design = [1*ones(1,size(t_RF_EpoRejDePow.powspctrm,1))
> > 2*ones(1,size(t_RN_EpoRejDePow.powspctrm,1))]; % the first dimension of
> > these variable is the trial number.
> > cfg.design  = design;
> >
> > P_ROC_t_RFvsRN =
> ft_freqstatistics(cfg,t_RF_EpoRejDePow,t_RN_EpoRejDePow);
> >
> >
> > Any help is greatly appreciated.
> >
> > Best regards,
> >
> > Shen-Mou Hsu
> >
> >
> >
> > _______________________________________________
> > fieldtrip mailing list
> > fieldtrip at donders.ru.nl
> > http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20130613/1425604d/attachment.html>