[FieldTrip] a case of double dipping (circular analysis) ???

[Vladimir Litvak]

Dear Yair,

There is double dipping in the way you select your SOI because once you
have already established that there is an effect in the time window you are
averaging over, your second test no longer controls for false positives at
the level you set. This should not be critical because this ROI
identification is separate from your main test for the effect of interest,
but I would understand why the reviewer is not completely comfortable with
that. If you can do a cluster-based test over both time and sensors and
then average over the cluster, it'd be more elegant
.
Regarding your main test, there is a subtle point that could make a
difference. It's whether you first computed averages for each condition and
then averaged the averages or you just pooled trials across all conditions
and averaged for your ROI identification. If the numbers of trials in
conditions A, B and C are equal then the two procedures are equivalent and
you should not worry. But if the numbers are unequal, this can lead to
bias. This is discussed in the Kriegeskorte paper but not in a very
explicit way, Intuitively, you cannot introduce a bias if your ROI test is
completely uninformed by what the conditions are (the pooling case) but if
you 'inject' some information about conditions by computing separate
averages first it could possibly be problematic. My colleague Howard Bowman
(CCed) has been working on a paper explaining this point but it is not yet
published. He might be able to share the draft with you.

So to sum up, my recommendation would be to pool all the trials first
across A.B.C, do a test across both time and sensors and then compare
conditions with respect to the average in the identified cluster.

Best,

Vladimir




On Wed, Apr 10, 2019 at 7:19 PM Yair Dor-Ziderman <yairem at gmail.com> wrote:

> Dear Fieldtrip users,
>
> I have just recieved a major revision request for a MEG analysis, with the
> concern that I was double dipping, citing (Kriegeskorte et al., 2009,
> Circular analysis in systems neuroscience - the dangers of double dipping,
> Nature neuroscience, 12(5), 535-540).
>
> I ran a MEG visual MIsmatch Negativity experiment (n=24) with standard and
> deviant trials for, say, conditions A, B and C.
> I conducted my analysis in three data-driven steps (all adequately
> corrected for multiple comparisons):
> 1) Over all conditions (A, B, and C), and over all sensors, but not over
> time, I compared the standard and  deviant trials to determine the time of
> interest (TOI, .when deviant trials deferred from standard trials).
> 2) Having found the TOI (~250-300 ms post stimulus presentation), I
> averaged over all conditions, and over the time-of-interest, but not over
> sensors, I performed a cluster-based permutation test to find the sensors
> exhibiting the effect (SOI, difference between standard and deviant trials)
> 3) Finally, for each subject, I averaged over the TOI and SOI, and
> separated the data into conditions.
>
> The reviewer argues that "The authors extracted time points and sensors
> that exhibited significant differences between standard and deviant trials,
> and subsequently analyzed this data under the null hypothesis of no effect.
> This seems like a case of circular analysis, or "double dipping""
>
> To my modest understanding, standard and deviant are mathematically
> orthogonol to the study's conditions. However, I do have to say, that
> closely reading the paper cited above - it appears that even in such cases
> there may be concern for double dipping.
>
> Has anyone encountered this problem? I this justified ?
>
> Thanks,
>
> Yair
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190411/4018bf20/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202