about cluster randomization analysis

[Vladimir Litvak]

Dear Marco,

A quick comment about two of your points from someone who had similar
experiences.

1) It is definitely wrong to zoom in on clusters the way you did. If you
know in advance where you expect the effect you should make a window as
narrow as possible around this time and then you will have higher
detection power. If you take a large enough window you can get clusters
of any size just by chance and then if you 'zoom in' on those clusters
they will be significant but this is meaningless, the same as doing 100
t-tests and taking the 5 that crossed the threshold by chance as
significant.

2) The clusterrandanalysis procedure can definitely be manipulated by
re-referencing. So if you have a positive and a negative pole of the
dipole none of which crosses the threshold, you can rereference to one
of them and then there is a good chance the other will be significant.
Again, this kind of manipulations is not quite 'kosher'. When I
discussed that with Robert and Eric they said that one should decide in
advance about a reasonable reference that is likely to reveal an effect.


Best,

Vladimir Litvak