Thanks! / Question regarding test statistics

Thu Jan 28 11:21:06 CET 2010

Hi Roemer,

thanks a lot for the answer. So I will allow alpha and clusteralpha to be
varied independently.

You are correct in suggesting that I was referring to "cfg.clustertail",
which I understand from the code (error message in clusterstat.m, l.32) to
be required to equal "cfg.tail". Sorry about the typo!

All the best,

Andrea

  _____

From: FieldTrip discussion list [mailto:FIELDTRIP at NIC.SURFNET.NL] On Behalf
Of Roemer van der Meij
Sent: Donnerstag, 28. Januar 2010 11:07
To: FIELDTRIP at NIC.SURFNET.NL
Subject: Re: [FIELDTRIP] Thanks! / Question regarding test statistics

Hi Andrea,

Sorry for my late reply (compared to my previous replies), I was a bit too
busy. I can't seem to find any reference on cfg.testtail, are you referring
to cfg.clustertail instead?

We are thinking at the moment on changing how the correcting of
p-values/alpha-values work, but for the moment you are correct in assuming
that you need to divide the cfg.alpha manually by 2 in case of a two-sided
test. This will most likely change in the near-future and be made more
explicit in the documentation, but for now, the tutorials are correct.

It is very important to make the distinction between cfg.clusteralpha and
cfg.alpha. Cfg.clusteralpha only refers to when certain values where you
want to cluster over are able to be selected for clustering. When clustering
over t-values, cfg.clusteralpha only sets the threshold on when to cluster.
A cfg.clusteralpha = 0.05 would allow more t-values to be selected for
clustering, most likely producing bigger clusters, then a cfg.clusteralpha =
0.01. This does not affect the false alarm rate of the statistical test at
the cluster-level, this is the cfg.alpha you mentioned. So, to control your
false-alarm rate when doing a two-sided cluster-level test, cfg.alpha is the
only one that is important. The selection of values to cluster over however,
does change when you do a two-sided test.

You are correct in stating that, when using the defaults, the overall
alpha-level that you are testing against is 0.10, whereas the alpha-level
per tail is 0.05. I guess it's a matter of preference which way you look at
it. But it is something that can be made more explicit in the
documentation/tutorials, thanks for bringing this up.

Thanks for noticing the old alphathresh field, the tutorial is updated now.

I hope this helps!

Best,
Roemer

On 1/21/2010 1:33 PM, Andrea Ostendorf wrote:

Dear Roemer,

Thanks a lot. My previous version worked with your proposed corrections and
I downloaded the new version today.

Sorry, I have yet another question, this time regarding the parameters
controlling statistical analysis for the cluster-based test statistic.

When asking myself whether it made sense to set cfg.testtail and cfg.tail
independently, I found the error message in clusterstat.m, l.32, which
states that the parameters should be equal. Could you perhaps mention this
requirement in the tutorials?

Also, regarding cfg.alpha and cfg.clusteralpha - just to make sure that I
have got things right: For a two-sided test, I have to split cfg.alpha
manually so as to equal half the desired significance level but I do not
have to change cfg.clusteralpha explicitly, is this correct? I gathered so
from the tutorials and it looks so from the code (in clusterstat and the
statfun functions, the alpha parameter is halved automatically). If nothing
is specified, both parameters are set to 0.05. As a two-sided test is the
default, does this mean that the false alarm rate is set at the 10% level if
the user does not specify anything, or have I overlooked something? (I have
to admit that I am impressed with the intricacies but that I have not really
grasped the details.)

By the way, the tutorial on cluster-based permutation tests on event related
fields mentions a parameter cfg.alphathresh, which I only found in the
clusterrand functions and which I took to mean cfg.clusteralpha.

All the best to all of you

Andrea

  _____

From: FieldTrip discussion list [mailto:FIELDTRIP at NIC.SURFNET.NL] On Behalf
Of Roemer van der Meij
Sent: Mittwoch, 20. Januar 2010 14:08
To: FIELDTRIP at NIC.SURFNET.NL
Subject: Re: [FIELDTRIP] Warning in ft_topoplotER

Hi Andrea,

Thanks for the data/script, it was exactly what I needed. There was a silly
bug in ft_clusterplot where several calls to cfg.highlight were not changed
to cfg.highlightchannel during the big update, and I failed to notice them
during browsing. Things should be fixed now (will be on the ftp-server
tonight). You can fix it in your own version easily by changing in line 267
and line 285 cfg.highlight >>> cfg.highlightchannel. Clusterplot should work
as intended afterwards.

I also realised now that any additional options that go into clusterplot
will not be used (like cfg.marker), these are fixed now as well (on
ftp-server tonight).

Thanks for pointing out these bugs!

Best,
Roemer

Andrea Ostendorf wrote:

Dear Roemer,

thanks a lot for your help and the time you are taking to assist me! Your
remarks have been really helpful.

The warning is from topoplotER, which I call indirectly through
ft_clusterplot. Thanks for pointing out that I am not to use the options
directly. I saw some options in the help of ft_clusterplot together with the
hint that more options were to be found in topoplotER.

Since I keep having trouble with uploading files to your server, I attach a
test MATLAB file (stripped down to the basics) and my data. I hope that is
okay, it is really quite a small amount of data. Otherwise, I would try my
private Ubuntu computer.

The warning is

Warning: cfg.highlight is now used for specifing highlighting-mode, use
cfg.highlightchannel instead of cfg.highlight for specifiying channels

> In ft_topoplotER at 214

  In topoplotER at 17

  In ft_clusterplot at 277

  In test_clusterplot at 5

Thanks a lot!

Best regards

Andrea

  _____

From: FieldTrip discussion list [mailto:FIELDTRIP at NIC.SURFNET.NL] On Behalf
Of Roemer van der Meij
Sent: Mittwoch, 20. Januar 2010 12:52
To: FIELDTRIP at NIC.SURFNET.NL
Subject: Re: [FIELDTRIP] Thanks / warning in ft_topoplotER

Hi Andrea,

Always happy to help. And thank you for helping us squashing bugs!

Are you sure you are using the most recent fieldtrip version? There was a
bug several weeks ago that gave the warning (which used to be on line 214,
but is now on line 215) even when cfg.highlight was not numerical. (if it's
numerical, it is assumed cfg.highlight is used in the old way). If you have
a version that is newer than 19-12-2009 (date of previous bug fix), could
you post the exact code you use to call ft_topoplotER? This could give me a
better idea on where in the code the problem lies. Could you then also post
the exact warning matlab spits out? (if it's not a version problem,
something complicated is going on and all information is helpful). Thanks!

Concerning ft_clusterplot, you're not actually supposed to use the
highlighting options directly. As clusterplot tries to determine the
clustering automatically, this could possibly interfere with it's normal
functioning. Thanks for pointing this out. I now added a check for these
'forbidden' options and they will be removed automatically. The updated
clusterplot will be uploaded to the ftp-server somewhere late in the evening
today.

However, if you want to manage you're own clustering directly, you can also
do this using topoplotER/TFR. It might be a bit difficult, but you have to
put all the highlighting options in its own cell (including the ones that
were already in cells). There is 'hidden' documentation about this in
topoplotER (you won't see it with the help/doc command, but you will if you
edit the function), you can find it just below the normal documentation.
I copy-pasted it here for easy reference:
***********
% It is possible to use multiple highlight-selections (e.g.: multiple
statistical clusters of channels)
% To do this, all the content of the highlight-options (including
cfg.highlight) should be placed in a cell-array
% (even if the normal content was already in a cell-array). Specific marker
settings (e.g. color, size) are defaulted when
% not present.
% Example (3 selections):
% cfg.highlight               = {'labels', 'labels', 'numbers'}
% cfg.highlightchannel    = {{'MZF03','MZC01','MRT54'}, [1:5], 'C*'}
% cfg.highlightsymbol    = {'o',[],'+'}          % the empty option will be
defaulted
% cfg.highlightcolor       = {'r',[0 0 1]};      % the missing option will
be defaulted
% cfg.highlightsize         = [];                      % will be set to
default, as will the missing cfg.highlightfontsize
***********

Using these options you should be able to get the same individual plots
clusterplot can give you. If you do not want to see the other channels, set
cfg.marker = 'off';. On the use of the different routines, topoplotTFR is
identical to topoplotER, but is just present because of our present
naming-scheme. Clusterplot however, is a wrapper around topoplotER/TFR
designed to quickly and without much configuring plot your statistical
clusters (which requires statistics output as input). By using the 'hidden'
options above, you can make topoplotER/TFR do exactly the same as
clusterplot, but then you can specify your own clusters.

We are currently looking into your question number 3, as it appears things
go a little deeper there. There are at least some situations where
cfg.neighbours = [] apparently doesn't work, but we are still figuring out
why. For the moment, a really ugly and time consuming workaround would be to
make a for-loop over single channels, but it would prevent the clustering
over channels and should provide (per channel) the exact same output as
cfg.neighbours = [];

I hope all of the above helps. Thanks for bringing these bugs to our
attention!

Kind regards,

Roemer

__________ Information from ESET NOD32 Antivirus, version of virus signature
database 4788 (20100120) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com

----------------------------------