[FieldTrip] Fwd: a case of double dipping (circular analysis) ???

Fri Apr 12 09:24:38 CEST 2019

---------- Forwarded message ---------
From: Howard Bowman <H.Bowman at kent.ac.uk>
Date: Thu, Apr 11, 2019 at 9:24 PM
Subject: RE: [FieldTrip] a case of double dipping (circular analysis) ???
To: Vladimir Litvak <litvak.vladimir at gmail.com>, FieldTrip discussion list <
fieldtrip at science.ru.nl>

Hi Vladimir: Thanks for copying me in.

Hi Yair: from your explanation of your analysis, it does also sound to me
that you have a double-dipping bias in your analysis. As Vladimir
indicates, I have been working through some of the maths on this, but you
do not actually need to go that far. The bias (which inflates the false
positive/ type I error rate) is easy to demonstrate by simulating the null.
I have enclosed our first paper on this, which was published in 2017. Where
I would say our paper goes beyond the Kriegeskorte et al Double Dipping
paper is the issue that Vladimir highlighted about imbalances of trial
counts across conditions, which is the norm in EEG and MEG experiments, for
reasons such as artefact rejection.

As Vladimir said, the approach we advocate is to aggregate your data across
conditions and search for a window in that. This is bias free, but in the
context of trial count asymmetries, only if you aggregate your data in a
fashion that ensures that every trail in every condition counts equally in
the aggregated average – that is our Aggregated Average of Trials. When we
were first looking at this, I think I even put a bet down that this
aggregation at the trial level **would** be biased, but it turns out it is
not and it is the right approach. This is shown in simulation in Brooks et
al, Psychophysiology, 2017; see figures 2 and 3.

We also now have a mathematical proof of this bias-freeness, but that
paper, which Vladimir is involved with, is still on my “to submit” stack.
[If not this weekend, then I’ll certainly get it in next, Vladimir.]

I suspect the simulation results will be enough for you Yair, but if you
want to see the mathematical details, let me know.

Good luck with your analysis.

Howard

--------------------------------------------

Professor Howard Bowman (PhD)

Professor of Cognition & Logic in CS at Uni Kent, and Professor of
Cognitive Neuroscience in Psychology at Uni Birmingham

Centre for Cognitive Neuroscience and Cognitive Systems and the School of
Computing, University of Kent at Canterbury, Canterbury, Kent, CT2 7NF,
United Kingdom

email: H.Bowman at kent.ac.uk

WWW: http://www.cs.kent.ac.uk/people/staff/hb5/

School of Psychology,

University of Birmingham, Edgbaston, Birmingham B15 2TT

*From:* Vladimir Litvak <litvak.vladimir at gmail.com>
*Sent:* 11 April 2019 10:34
*To:* FieldTrip discussion list <fieldtrip at science.ru.nl>; H.Bowman <
H.Bowman at kent.ac.uk>
*Subject:* Re: [FieldTrip] a case of double dipping (circular analysis) ???

Dear Yair,

There is double dipping in the way you select your SOI because once you
have already established that there is an effect in the time window you are
averaging over, your second test no longer controls for false positives at
the level you set. This should not be critical because this ROI
identification is separate from your main test for the effect of interest,
but I would understand why the reviewer is not completely comfortable with
that. If you can do a cluster-based test over both time and sensors and
then average over the cluster, it'd be more elegant

.

Regarding your main test, there is a subtle point that could make a
difference. It's whether you first computed averages for each condition and
then averaged the averages or you just pooled trials across all conditions
and averaged for your ROI identification. If the numbers of trials in
conditions A, B and C are equal then the two procedures are equivalent and
you should not worry. But if the numbers are unequal, this can lead to
bias. This is discussed in the Kriegeskorte paper but not in a very
explicit way, Intuitively, you cannot introduce a bias if your ROI test is
completely uninformed by what the conditions are (the pooling case) but if
you 'inject' some information about conditions by computing separate
averages first it could possibly be problematic. My colleague Howard Bowman
(CCed) has been working on a paper explaining this point but it is not yet
published. He might be able to share the draft with you.

So to sum up, my recommendation would be to pool all the trials first
across A.B.C, do a test across both time and sensors and then compare
conditions with respect to the average in the identified cluster.

Best,

Vladimir

On Wed, Apr 10, 2019 at 7:19 PM Yair Dor-Ziderman <yairem at gmail.com> wrote:

Dear Fieldtrip users,

I have just recieved a major revision request for a MEG analysis, with the
concern that I was double dipping, citing (Kriegeskorte et al., 2009,
Circular analysis in systems neuroscience - the dangers of double dipping,
Nature neuroscience, 12(5), 535-540).

I ran a MEG visual MIsmatch Negativity experiment (n=24) with standard and
deviant trials for, say, conditions A, B and C.

I conducted my analysis in three data-driven steps (all adequately
corrected for multiple comparisons):

1) Over all conditions (A, B, and C), and over all sensors, but not over
time, I compared the standard and  deviant trials to determine the time of
interest (TOI, .when deviant trials deferred from standard trials).

2) Having found the TOI (~250-300 ms post stimulus presentation), I
averaged over all conditions, and over the time-of-interest, but not over
sensors, I performed a cluster-based permutation test to find the sensors
exhibiting the effect (SOI, difference between standard and deviant trials)

3) Finally, for each subject, I averaged over the TOI and SOI, and
separated the data into conditions.

The reviewer argues that "The authors extracted time points and sensors
that exhibited significant differences between standard and deviant trials,
and subsequently analyzed this data under the null hypothesis of no effect.
This seems like a case of circular analysis, or "double dipping""

To my modest understanding, standard and deviant are mathematically
orthogonol to the study's conditions. However, I do have to say, that
closely reading the paper cited above - it appears that even in such cases
there may be concern for double dipping.

Has anyone encountered this problem? I this justified ?

Thanks,

Yair

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190412/57eb6c41/attachment-0002.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202