about cluster randomization analysis

Eric Maris maris at NICI.RU.NL
Tue Nov 8 13:17:50 CET 2005

Hi Marco,

  thank you for your accurate responses. I fully understand from your arguments that temporally zooming on clusters is definitely wrong. Still, I wonder whether and how it is possible to use cluster randomization analysis cases in which it is difficult to formulate a precise hypothesis about when to expect an effect  (for example, in infants), or cases in which an unexpected effect arises from a t-test. Do you think it would be correct to slide a relatively large (width of 200ms? 400ms? to be chosen a priori of course) window through the epochs and compute cluster randomization analysis for each latency to explore dubious significant t-test clusters? 

If you have no hypothesis about where to expect an effect, you should use the complete latency window in which it may occur. Of course, this will reduce the sensitivity (statistical power) of your test (in comparison with the situation in which you do know when the effect can occur). As a rule, prior knowledge increases sensitivity.

    Another related question: I computed a post-hoc non kosher tuning of the window around the most significative cluster in my data, and I saw that it is significative (p<0.05) if the window edges exceed of about 50 ms the cluster edges (since the cluster is about 70 ms long, the whole window is about 170 ms long); but if I take longer windows, the p-value increases quite rapidly (I'm running at least 500 random draws for each window, and checking that the result does not depend on the number of draws). Do you have such instabilities in your data or should I think that the effect relative to my cluster is definitely too weak? Or maybe my data are not clean enough? 

  This phenomenon is not an instability, it is what I would expect. Imagine your trials are 10 seconds long and there is an effect in the latency window between 1.3 and 1.35 seconds (i.e., less than 1 percent of trial length). If you ask clusterrandanalysis to compare the conditions over the complete trial length, it may very well miss the effect in the window between 1.3 and 1.35 seconds, because it has to use a large critical value in order to control for false positives in the time window where there is no effect (i.e., 99 percent of the 10 second trial). 


  Eric Maris

  On 10/28/05, Eric Maris <maris at nici.ru.nl> wrote: 
    Dear Marco,

    > The procedure I am following now is a sort of two-steps method: in the
    > first place, I choose a wide time interval and a low minimum number of 
    > channels. I end up with many clusters that are far from being
    > significative. I then shorten the time interval to include just one
    > cluster (starting from the most significant one), and increase the minimum 
    > number of channels, and run the analysis again. In this case, I eventually
    > got a significative cluster where I was expecting it from a simple
    > observation of the t-test. Do you think this procedure is right or am I 
    > doing something wrong? Is it correct to temporally focus on a cluster to
    > check its significance?

    Clusterrandanalysis only controls the false alarm (type I error) rate if you
    choose the "tuning parameters" (latency interval, channel subset, the 
    minnbchan-parameter; and if you use on TFRs, also the frequency interval)
    independent of the data. Instead, if you play around with these tuning
    parameters until you find a cluster whose p-value exceeds the critical 
    alpha-level, you are not controlling the false alarm rate. In this case, the
    chosen tuning parameters depend on the data.

    An extreme example illustrates this even better. Assume you calculate
    T-statistics for all (channel, time point)-pairs and you select the pair 
    with the largest T-statistic. Then, you select the latency interval that
    only contains this time point and the channel subset that only contains this
    channel. With these tuning parameters, you reduce your data to a single cell 
    in the spatiotemporal matrix, and clusterrrandanalysis will produce a
    p-value that is very close to the p-value of a T-test. Since you have
    selected this (channel, time point)-pair on the basis of its T-statistic, 
    this p-value is strongly biased.

    > Another couple of questions:
    > 1) Minnbchan. I understood it is the minimum number of significative
    > neighbor (channel,time) points for a  (channel,time) point to enter a 
    > cluster, no matter if adjacency is more in channel space or time
    > direction. Am I right? Since time and channel space are quite different
    > dimension, would it be better to set a minimum channel number separately 
    > for the two?

    Minnbchan should also be chosen independent of the data. I introduced this
    tuning parameter because it turned out that in 3-dimensional analyses on
    TFRs (involving the dimensions time, space ( i.e., sensors) and frequency),
    sometimes a cluster appeared that consisted of two or more 3-dimensional
    "blobs" that were connected by a single (channel, time, frequency)-element.
    From a physiological perspective, such a cluster does not make sense. To 
    remove these physiologically implausible (and therefore probably random)
    connections, I introduced the minnbchan parameter. Because of this
    physiological rationale, I apply the minimum number criterium to the
    spatial, and not to the temporal dimension. Short-lived phenomena are very
    well possible from a physiological perspective, whereas effects at spatially
    isolated sensors are not.

    > 2) Maybe because my data are average-referenced, I often end up with a 
    > positive and negative cluster emerging almost at the same time. Have you
    > thought about any way to include the search of dipole-like configurations?

    I have not thought about it, but it certainly makes sense to incorporate 
    biophysical constraints (such dipolar patterns) in the test statistic.

    One should be aware of the fact that different hypotheses are tested before
    and after rereferencing. This is physical and not a statistical issue. As 
    you most certainly know, EEG-signals are potential DIFFERENCES and therefore
    the underlying physiological events that are measured by EEG depend on the
    reference channel(s). If the experimental manipulation affects the current 
    reference channel, then rereferencing to another channel (or set of
    channels) that is not affected by the experimental manipulation makes a
    difference for the result of the statistical test.


    Eric Maris

  Marco Buiatti - Post Doc

  Cognitive Neuroimaging Unit  - INSERM U562
  Service Hospitalier Frederic Joliot, CEA/DRM/DSV 
  4 Place du general Leclerc, 91401 Orsay cedex, France
  Telephone: +33 1 69 86 77 65    Fax: +33 1 69 86 78 16
  E-mail: marco.buiatti at gmail.com    Web: www.unicog.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20051108/34ca3ab1/attachment-0002.html>

More information about the fieldtrip mailing list