[FieldTrip] a case of double dipping (circular analysis) ???

[Yair Dor-Ziderman]

Dear Fieldtrip users,

I have just recieved a major revision request for a MEG analysis, with the
concern that I was double dipping, citing (Kriegeskorte et al., 2009,
Circular analysis in systems neuroscience - the dangers of double dipping,
Nature neuroscience, 12(5), 535-540).

I ran a MEG visual MIsmatch Negativity experiment (n=24) with standard and
deviant trials for, say, conditions A, B and C.
I conducted my analysis in three data-driven steps (all adequately
corrected for multiple comparisons):
1) Over all conditions (A, B, and C), and over all sensors, but not over
time, I compared the standard and  deviant trials to determine the time of
interest (TOI, .when deviant trials deferred from standard trials).
2) Having found the TOI (~250-300 ms post stimulus presentation), I
averaged over all conditions, and over the time-of-interest, but not over
sensors, I performed a cluster-based permutation test to find the sensors
exhibiting the effect (SOI, difference between standard and deviant trials)
3) Finally, for each subject, I averaged over the TOI and SOI, and
separated the data into conditions.

The reviewer argues that "The authors extracted time points and sensors
that exhibited significant differences between standard and deviant trials,
and subsequently analyzed this data under the null hypothesis of no effect.
This seems like a case of circular analysis, or "double dipping""

To my modest understanding, standard and deviant are mathematically
orthogonol to the study's conditions. However, I do have to say, that
closely reading the paper cited above - it appears that even in such cases
there may be concern for double dipping.

Has anyone encountered this problem? I this justified ?

Thanks,

Yair
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190410/c2a345b4/attachment-0001.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202