cluster statistic on one sample

Mon Oct 20 18:00:47 CEST 2008

Hey Eric,

I am not sure I understand your point against the bootstrap. My main
source of information about robust statistics is Wilcox, R. R. (2005).
Introduction to Robust Estimation and Hypothesis Testing (2nd Ed.
ed.): Academic Press.
In this book, Wilcox makes an extensive use of the bootstrap
technique. The validation of the technique, when it has been
performed, relies on Monte-Carlo simulations. Also, over almost 600
pages, Wilcox spends only one page on permutations, basically saying
that it is a special case of the bootstrap and that there is no
particular reason to use it.
Do you have references showing validation tests with a direct
comparison of bootstrap and permutation? My understanding is that such
comparisons do not exist for EEG/MEG data.
Also, one must keep in mind that bootstrap is particularly efficient
when applied to robust measures of central tendency, like trimmed
means and M-estimators, see my recent EEG paper for instance (http://www.journalofvision.org/8/12/3/
).
Finally, Wilcox provides a large number of recipes to test
significance of linear regression results, that could be applied to
the problem outlined earlier about the hypothesis test against zero.

Best,

GAR

On 20 Oct 2008, at 15:02, Eric Maris wrote:

> Dear Fieldtrip-list-readers,
>
>
>> What about performing a nonparametric test, based on the bootstrap
>> distribution of the beta weights under the null-hypothesis?
>> This problem sounds similar to one I came across recently (and which
>> I still have to write something about on fieldtrip's wiki-page (sorry
>> Eric)), which has to do with the testing of the significance of the
>> F-
>> value for interaction in a 2x2 repeated measure anova. Also in this
>> case, one also wants to test a parametric null-hypothesis, as Eric
>> phrased it in his last e-mail. One way to test this (I don't have the
>> reference at hand), is to test the observed F-statistic against a
>> null-distribution, obtained from bootstrapping your data, which you
>> preconditioned as to impose the null-hypothesis (in the case of an
>> anova it would be to remove from each of the observations the mean of
>> the cell to which the observation belongs). I don't know yet how to
>> impose the null-hypothesis in the regression case, but would this
>> line of thought be a possibility?
>> As to a potential implementation: Robert and I are pretty close to
>> have the bootstrapping implemented.
>
> Again, I can only try to clarify some points here. I will not be
> able to
> offer a solution for your problems.
>
> 1. Contrary to the permutation test, there is no useful statistical
> theory
> for statistical tests based on the bootstrap distribution. By
> "useful", I
> mean a theory that allows one to specify a scientifically
> interesting null
> hypothesis (such as, "An expected value equal to 0") under which the
> false
> alarm rate of a boostrap-p-value-based test can be controlled.
>
> 2. The bootstrap distribution has a nice intuitive appeal, because the
> procedure to generate it (sampling with replacement) mimicks the
> sampling
> process behind the sampling distribution (which is the ultimate
> "thing to
> get" if you want to quantify the reliability of some quantity). But
> that is
> not a proof of false alarm rate control!
>
> 3. I think the bootstrap distribution can be useful in situations
> where
> parametric statistical tests do not exists, but I know of no rigourous
> statistical argument to substantiate this claim.
>
>
> Greetings,
>
> Eric Maris
>
>
>>
>> Yours,
>>
>> Jan-Mathijs
>>
>>
>> On Oct 20, 2008, at 11:05 AM, Vladimir Litvak wrote:
>>
>>> Dear Floris and Eric,
>>>
>>> Parametric tests at scalp level taking into account spatial
>>> relationship between sensors can be done in SPM (with RFT
>>> correction).
>>> That'll require using some low-level functions to convert
>>> coefficients to images but in principle shouldn't be that difficult.
>>>
>>> Best,
>>>
>>> Vladimir
>>>>
>>>> On Mon, Oct 20, 2008 at 10:17 AM, Eric Maris
>>>> <e.maris at donders.ru.nl> wrote:
>>>>> Dear Floris,
>>>>>
>>>>>
>>>>>
>>>>>> I have a question about statistical analysis on the sensor level.
>>>>>> I would like to make use of the cluster size thresholding of the
>>>>>> clusterrand routine in Fieldtrip. Unfortunately, in the current
>>>>>> wrapper, it seems there is no option for a one-sample T-test?
>>>>>> There is
>>>>>> an activation-baseline test, and a (in)dependent samples test
>>>>>> between
>>>>>> two conditions, but what I want to do is simply test whether a 14
>>>>>> (subjects) x 275 (channels) matrix is different from zero,
>>>>>> taking into
>>>>>> account the spatial relations between adjacent sensors. (The data
>>>>>> points are regression weights from a multiple-regression
>>>>>> analysis, so
>>>>>> there's no easy way to split it into two parts.)
>>>>>> I assume this should be easy to tweak, but I couldn't come up
>>>>>> with any
>>>>>> smart ideas how to do it.
>>>>>> Anyone any ideas?
>>>>>
>>>>> I'm afraid that I have to disappoint you, Floris. Your null
>>>>> hypothesis is a
>>>>> typical parametric null hypothesis; the expected value of some
>>>>> (matrix-valued) variable being equal to zero. The null hypothesis
>>>>> that is
>>>>> tested by a nonparametric permutation test is equality across
>>>>> experimental
>>>>> conditions of the probability distribution from which the
>>>>> (condition-specific) data are drawn. Since you have single
>>>>> condition only, I
>>>>> see no way of applying the theory behind nonparametric
>>>>> permutation testing
>>>>> (of the type described by Maris & Oostenveld, 2007) to your data.
>>>>>
>>>>> To solve your problem we need a brilliant theoretical insight.
>>>>>
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Floris
>>>>>>
>>>>>> ----------------------------------
>>>>>> The aim of this list is to facilitate the discussion between
>>>>>> users of the
>>>>> FieldTrip
>>>>>> toolbox, to share experiences and to discuss new ideas for MEG
>>>>>> and EEG
>>>>> analysis.
>>>>>> See also http://listserv.surfnet.nl/archives/fieldtrip.html and
>>>>>> http://www.ru.nl/fcdonders/fieldtrip.
>>>>>
>>>>> ----------------------------------
>>>>> The aim of this list is to facilitate the discussion between
>>>>> users of the FieldTrip  toolbox, to share experiences and to
>>>>> discuss new ideas for MEG and EEG analysis. See also http://
>>>>> listserv.surfnet.nl/archives/fieldtrip.html and http://www.ru.nl/
>>>>> fcdonders/fieldtrip.
>>>>>
>>>>>
>>>>
>>>
>>> ----------------------------------
>>> The aim of this list is to facilitate the discussion between users
>>> of the FieldTrip  toolbox, to share experiences and to discuss new
>>> ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/
>>> archives/fieldtrip.html and http://www.ru.nl/fcdonders/fieldtrip.
>>
>> ----------------------------------
>> The aim of this list is to facilitate the discussion between users
>> of the
> FieldTrip
>> toolbox, to share experiences and to discuss new ideas for MEG and
>> EEG
> analysis.
>> See also http://listserv.surfnet.nl/archives/fieldtrip.html and
>> http://www.ru.nl/fcdonders/fieldtrip.
>
> ----------------------------------
> The aim of this list is to facilitate the discussion between users
> of the FieldTrip  toolbox, to share experiences and to discuss new
> ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/archives/fieldtrip.html
>  and http://www.ru.nl/fcdonders/fieldtrip.

************************************************************************************
Guillaume A. Rousselet, Ph.D.

Lecturer

Centre for Cognitive Neuroimaging (CCNi)
Department of Psychology
Faculty of Information & Mathematical Sciences (FIMS)
University of Glasgow
58 Hillhead Street
Glasgow, UK
G12 8QB

The University of Glasgow, charity number SC004401

http://web.me.com/rousseg/GARs_website/

Email: g.rousselet at psy.gla.ac.uk
Fax. +44 (0)141 330 4606
Tel. +44 (0)141 330 6652
Cell +44 (0)791 779 7833

"no test based upon a theory of probability can by itself
provide any valuable evidence of the truth or falsehood
of a hypothesis.

But we may look at the purpose of tests from another
viewpoint. Without hoping to know whether each separate
hypothesis is true or false, we may search for
rules to govern our behaviour with regard to them, in
following which we insure that, in the long run of
experience, we shall not often be wrong."

	Neyman J & Pearson E, 1933
************************************************************************************

----------------------------------
The aim of this list is to facilitate the discussion between users of the FieldTrip  toolbox, to share experiences and to discuss new ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/archives/fieldtrip.html and http://www.ru.nl/fcdonders/fieldtrip.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20081020/1ffd99cd/attachment-0001.html>