[FieldTrip] a case of double dipping (circular analysis) ???

[Yair Dor-Ziderman]

Thanks Vladimir for the comprehensive answer. It is very helpful!

Best wishes,

Yair

On Thu, 11 Apr 2019, 13:13 Vladimir Litvak, <litvak.vladimir at gmail.com>
wrote:

> Dear Yair,
>
> There is double dipping in the way you select your SOI because once you
> have already established that there is an effect in the time window you are
> averaging over, your second test no longer controls for false positives at
> the level you set. This should not be critical because this ROI
> identification is separate from your main test for the effect of interest,
> but I would understand why the reviewer is not completely comfortable with
> that. If you can do a cluster-based test over both time and sensors and
> then average over the cluster, it'd be more elegant
> .
> Regarding your main test, there is a subtle point that could make a
> difference. It's whether you first computed averages for each condition and
> then averaged the averages or you just pooled trials across all conditions
> and averaged for your ROI identification. If the numbers of trials in
> conditions A, B and C are equal then the two procedures are equivalent and
> you should not worry. But if the numbers are unequal, this can lead to
> bias. This is discussed in the Kriegeskorte paper but not in a very
> explicit way, Intuitively, you cannot introduce a bias if your ROI test is
> completely uninformed by what the conditions are (the pooling case) but if
> you 'inject' some information about conditions by computing separate
> averages first it could possibly be problematic. My colleague Howard Bowman
> (CCed) has been working on a paper explaining this point but it is not yet
> published. He might be able to share the draft with you.
>
> So to sum up, my recommendation would be to pool all the trials first
> across A.B.C, do a test across both time and sensors and then compare
> conditions with respect to the average in the identified cluster.
>
> Best,
>
> Vladimir
>
>
>
>
> On Wed, Apr 10, 2019 at 7:19 PM Yair Dor-Ziderman <yairem at gmail.com>
> wrote:
>
>> Dear Fieldtrip users,
>>
>> I have just recieved a major revision request for a MEG analysis, with
>> the concern that I was double dipping, citing (Kriegeskorte et al., 2009,
>> Circular analysis in systems neuroscience - the dangers of double dipping,
>> Nature neuroscience, 12(5), 535-540).
>>
>> I ran a MEG visual MIsmatch Negativity experiment (n=24) with standard
>> and deviant trials for, say, conditions A, B and C.
>> I conducted my analysis in three data-driven steps (all adequately
>> corrected for multiple comparisons):
>> 1) Over all conditions (A, B, and C), and over all sensors, but not over
>> time, I compared the standard and  deviant trials to determine the time of
>> interest (TOI, .when deviant trials deferred from standard trials).
>> 2) Having found the TOI (~250-300 ms post stimulus presentation), I
>> averaged over all conditions, and over the time-of-interest, but not over
>> sensors, I performed a cluster-based permutation test to find the sensors
>> exhibiting the effect (SOI, difference between standard and deviant trials)
>> 3) Finally, for each subject, I averaged over the TOI and SOI, and
>> separated the data into conditions.
>>
>> The reviewer argues that "The authors extracted time points and sensors
>> that exhibited significant differences between standard and deviant trials,
>> and subsequently analyzed this data under the null hypothesis of no effect.
>> This seems like a case of circular analysis, or "double dipping""
>>
>> To my modest understanding, standard and deviant are mathematically
>> orthogonol to the study's conditions. However, I do have to say, that
>> closely reading the paper cited above - it appears that even in such cases
>> there may be concern for double dipping.
>>
>> Has anyone encountered this problem? I this justified ?
>>
>> Thanks,
>>
>> Yair
>> _______________________________________________
>> fieldtrip mailing list
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>> https://doi.org/10.1371/journal.pcbi.1002202
>>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://doi.org/10.1371/journal.pcbi.1002202
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190411/f0c2eb21/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202