Question about nonparametric statistical testing of coherence differences

[Eric Maris]

Dear Matthew,


Your questions relate to the core concepts behind statistical inference
based on permutation.


> My question involves a situation in which one is comparing the coherence
> between two signals in two different conditions, and the coherence is
> generally high in both conditions, but we would like to say that the
> coherence is higher in one condition than the other. After some
> thought about this issue, I wonder if shuffling trials between the two
> conditions allows for a fair comparison to be made to the coherence in
> the raw data? For example, if a pair of signals has different relative
> phases in the two conditions, the overall coherence that exists in
> both conditions will be decreased by the trial shuffling. It's also
> conceivable, though perhaps less likely, that the condition-shuffled
> data could have more coherence than either condition alone. In either
> case, the distribution of the cluster statistics of the coherence
> differences in the condition-shuffled data would certainly provide
> some sort of estimate of a null distribution of the coherence
> difference that does uses the actual data, but since the overall
> coherence values of the condition-shuffled and raw data could be so
> different, is it fair to say that this is a null distribution of the
> difference in coherence between the unshuffled conditions?


It is important to always keep in mind which null hypothesis is tested using
permutation inference. This null hypothesis is exchangeability. For the case
that you describe, exchangeability involves that the probability
distributions of bivariate time series (or, equivalently, their Fourier
coefficients) in the two experimental conditions are identical. You describe
a situation in which this null hypothesis does not hold, and the reason for
this is not a single but two parameters: coherence and relative phase. Both
parameters differ across the two conditions that you describe. Now, if you
want to test whether coherence differs across the two conditions and you
don't want your inference to be affected by a difference in relative phase,
then you have to come up with a test statistic that has the appropriate
sensitivity.  Such a test statistic might be the following: calculate the
coherence difference between the two conditions using Fourier coefficients
of which the phases were adjusted such that relative phase in the two
conditions are both equal to zero. (This is a simple operation, involving a
condition-specific phase shift applied to all Fourier coefficients of one of
the two channels.)

This example shows both the strength and the weakness of permutation
inference. The strength is that you can construct your own test statistic
such that it is sensitive to the physiological phenomenon in which you are
interested. And the weakness is that the null hypothesis (exchangeability)
is much more general than this specific physiological phenomenon. This means
that a small permutation p-value allows you to infer that the two conditions
most likely differ, but that inferring the precise cause of this difference
depends on how clever you have been in designing a test statistic with the
desired specificity.

By the way, I did not understand the following sentence in your email: "It's
also
conceivable, though perhaps less likely, that the condition-shuffled
data could have more coherence than either condition alone." For the
situation that you describe, the coherence difference between the conditions
(which is the relevant quantity for the phenomen in which you are
interested) will on average be less for the permuted data than for the
original data.


>
> If you have the time to respond, I'm curious to know what your
> thoughts are regarding this issue and specifically in a situation in
> which it is necessary to assess significance on one session.
> Performing the clustering and shuffling of coherence values across
> sessions obviously avoids this problem. For a test within a session, I
> wonder if it would be possible to use a bootstrapping or jackknife
> method combined with clustering to get a fair estimate of the
> variability of the cluster level statistics and perform hypothesis
> tests using that.

Here the answer is simple. Permutation inference can only be used for
comparing experimental conditions. However, I want to argue that, for
investigating coherence, comparing experimental conditions is the only
sensible thing one can do. The reason is that, for simple biophysical
reasons, the null hypothesis of zero coherence within a single condition,
will never be true, at least not in electrophysiological studies. This is
because the potentials that are recorded in the two channels will always be
affected by the physiological activity of common sources, simply because all
potentials are volume conducted.


Best,

Eric

dr. Eric Maris
Donders Institute for Brain, Cognition and Behavior
Center for Cognition and F.C. Donders Center for Cognitive Neuroimaging
Radboud University
P.O. Box 9104
6500 HE Nijmegen
The Netherlands
T:+31 24 3612651
Mobile: 06 39584581
F:+31 24 3616066
E: e.maris at donders.ru.nl

MSc Cognitive Neuroscience: www.ru.nl/master/cns/

----------------------------------
The aim of this list is to facilitate the discussion between users of the FieldTrip  toolbox, to share experiences and to discuss new ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/archives/fieldtrip.html and http://www.ru.nl/neuroimaging/fieldtrip.