[FieldTrip] Question about cluster-based permutation tests on linear mixed models

Fri Oct 21 19:38:51 CEST 2016

Dear Eric and Alik,

thanks a lot for your helpful responses!

I will have a close look at the faqs, Eric, and test the approaches you
outlined. I am curious, anyway, as to how different results will be for
simple regressions compared to the multilevel results of the linear-mixed
models.

Like Alik, I am also curious about other people's opinions on the general
question if there are theoretical reasons against a combination of the
approaches like Alik suggested. We also thought about this approach but
haven't fully tested it yet because of the very long calculation times.

Thanks again and have a nice weekend!
Elisabeth

2016-10-20 12:49 GMT+02:00 Alik Widge <alik.widge at gmail.com>:

> Eric, I don't think I understand why you would say "I do not see how these
> models could be combined with permutation-based inference; they are just
> different statistical frameworks". As you somewhat hint, the (G)LMM is a
> regression, and the beta coefficient for the independent-variable of
> interest at each voxel/vertex/sensor x timepoint can be interpreted as "how
> much does the independent variable explain the brain activity?" In that
> framework, it seems to me that one could do the following:
>
> for n=1:1000
>    1) Permute the condition labels (within subjects) of the individual
> trials
>    2) Re-fit the LMM at each (voxel,timepoint), creating a beta map and
> corresponding t-map
>    3) Threshold and construct cluster mass statistic as usual
> end
> 4) Identify cluster in the original (unpermuted) analysis and report
> cluster p-value
>
>
> Now, the main thing that has come up when we've tried to do this is that
> re-fitting a (voxel x time) GLM 1000 times by the standard iterative
> maximum-likelihood engines is remarkably slow. In fieldtrip, I can imagine
> it would require rewriting at least a statfun, maybe other pieces of the
> code. (We had an idea that, since the betas  likely should vary smoothly
> over time and space, one could use the output of one GLM as the seed to the
> next, which would speed up convergence.) So it still does not seem like a
> good idea, but based on the above, is there actually a *theoretical* reason
> it wouldn't work?
>
>
> Alik Widge, MD, PhD
> Director, Translational NeuroEngineering Laboratory
> Division of Neurotherapeutics, Massachusetts General Hospital
> Assistant Professor of Psychiatry, Harvard Medical School
> Clinical Fellow, Picower Institute for Learning & Memory (MIT)
> awidge at partners.org
> http://scholar.harvard.edu/awidge/
> 617-643-2580
>
> Alik Widge
> alik.widge at gmail.com
> (206) 866-5435
>
>
> On Thu, Oct 20, 2016 at 6:08 AM, Maris, E.G.G. (Eric) <
> e.maris at donders.ru.nl> wrote:
>
>> Note: this is the second time I post this reply, and the reason is that I
>> forgot to add an appropriate Subject (for findability) to my email (shame
>> on me…(-;)
>>
>> *From: *Elisabeth May <elisabethsusanne.may at gmail.com>
>> *Subject: **[FieldTrip] Question about cluster-based permutation tests
>> on linear mixed models*
>> *Date: *27 September 2016 at 14:46:55 GMT+2
>> *To: *<fieldtrip at science.ru.nl>
>> *Reply-To: *FieldTrip discussion list <fieldtrip at science.ru.nl>
>>
>>
>> Dear FieldTripers,
>>
>> I have a question about the potential use of cluster-based permutation
>> tests for results obtained using linear mixed models.
>>
>> We are working with data from a 10 min EEG experiment on source level
>> with the aim to quantify the relationship of brain activity in different
>> frequency bands with continous perceptual ratings across 20 subjects in
>> different experimental conditions. Thus, we have 10 min time courses of
>> brain activity and ratings for each voxel for different conditions and want
>> to test a) if there are significant relationships in the single conditions
>> and b) if these relationships differ between two conditions. To this end, I
>> have calculated linear mixed models in R using the lme4 toolbox. For both
>> the single condition relationships and the condition contrasts, they result
>> in a single t-value (and a corresponding p-value), which is based on
>> information on both the single subject and the group level (i.e. we perform
>> a multi-level analysis). However, with more than 2000 voxels, we have a lot
>> of t-values and are wondering if there is a way to apply cluster-based
>> tests to correct for multiple comparisons.
>>
>> The main problem I see is that I only have one multilevel t-value for the
>> effect across all subjects, i.e. I don't have single subjects values, which
>> I could then e.g. randomize between conditions as normally done in
>> cluster-based permutation tests. (Or rather, I would be able to extract
>> single subject values but would then loose the advantage of the multi-level
>> analysis.)
>>
>> I found an old thread in the mailinglist archive where it was suggested
>> to flip the signs of the t-statistic for cluster-level correction (
>> https://mailman.science.ru.nl/pipermail/fieldtrip/2012-July/005375.html).
>> I understand that, in our case, I would do this randomly for all voxels in
>> each randomization and then build spatial clusters on the resulting (partly
>> flipped) t-values. However, I am not sure if that is a valid approach based
>> on the null hypothesis that there are no significant relations in my single
>> conditions (a) or no significant relationship differences in my condition
>> contrasts (b).
>>
>> For the condition contrasts, I would be able to permute the condition
>> labels as normally done in cluster-based permutation tests,I think, but
>> would then have to recalculate the linear mixed models for all voxels in
>> every permutation. This would result in a very high computational load.
>>
>> Does anyone have any experience with this kind of analysis? Would the
>> flipping of t-values be a valid approach (and if yes, is there anything to
>> keep in mind in particular)? Can you think of other ways to combine linear
>> mixed models with a multiple comparison correction on the cluster level?
>>
>>
>> Hi Elisabeth,
>>
>> I’m not an expert on linear mixed modelling, at least not with respect to
>> the different ways in which they can be used to deal with correlated
>> observations (typically, time series). However, from a theoretical point of
>> view, I do not see how these models could be combined with
>> permutation-based inference; they are just different statistical
>> frameworks. However, it IS possible to answer your questions ("we have
>> 10 min time courses of brain activity and ratings for each voxel for
>> different conditions and wan to test a) if there are significant
>> relationships in the single conditions and b) if these relationships differ
>> between two conditions.”) within the framework of cluster-based permutation
>> tests. Question b) is the most straightforward because it amounts to a
>> cluster-based permutation test using the depsamplesT statfun applied to the
>> regression coefficients in each of the two conditions. Answering question
>> a) requires that you bin your ratings in a number of categories, calculate
>> the trial-averaged EEG data for each of the categoreies, and test the
>> difference between them using a cluster-based permutation test using the
>> depsamplesregrT statfun. Both of these approaches have been described
>> previously on this discussion list, and for the depsamplesregrT statfun
>> (your question a), it was Vladimir Litvak who used it first (actually, I
>> implemented it for him). The approach for question b) is actually a variant
>> on the general approach for testing interactions using cluster-based
>> permutation tests.
>>
>> Have a look here:
>> http://www.fieldtriptoolbox.org/faq/how_can_i_test_for_corre
>> lations_between_neuronal_data_and_quantitative_stimulus_and_
>> behavioural_variables
>> and
>> http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_intera
>> ction_effect_using_cluster-based_permutation_tests
>>
>> These tutorials provide all the necessary concepts, although they do not
>> answer your question in a recipe-like fashion.
>>
>> best,
>> Eric Maris
>>
>>
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20161021/9df58684/attachment-0001.html>