<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Alik,
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">
<p dir="ltr" class="">So, two further questions. </p>
<p dir="ltr" class="">1) Anyone reading this thread have some code for a reasonable ERP generator, especially one that models different ERPs per subject? I'm getting increasingly interested in doing some simulations to test my claims. (I'm thinking to do the
1d case just because it's faster to run. ) However, I'd like to do that on stimulated data that properly models, E.g., inter-subject variability of the precise timing of major peaks.
</p>
<p dir="ltr" class="">2) Eric, why are you describing the below as a fixed effects model? If we do a mixed model (with subject as a random intercept) at each time/frequency/sensor point, that seems like it handles the random effect that most people care about.
Are you saying that you think each predictor variable would also need to be modeled as a per-subject slope?
</p>
</div>
</blockquote>
<div>If you are permuting trials across conditions within every subject, this corresponds to the null hypothesis that WITHIN EVERY SUBJECT, there is no association between biological data and the condition labels. This is the permutation-version of a fixed-effects
test.</div>
<div><br class="">
</div>
<div>Keep in mind that you use your LMM t-stats only for thresholding and not for inference.</div>
<div><br class="">
</div>
<div>I’m aware that this may be confusing at first sight. Actually, the topic (fixed versus random effects tests in the permutation framework) have not been described in a paper yet. I’m thinking about writing one, though...</div>
<blockquote type="cite" class="">
<div class="">
<p dir="ltr" class="">I guess I'm also thinking here that (to your second point), if the resulting values are used for thresholding to create cluster statistics for inference, inflation in the t/p values should be controlled for because the null distribution
created during the permutation will have the same inflation. However, I don't believe I have a formal mathematical backup for that intuition, hence (1) above.
</p>
<div class="gmail_extra"><br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>The formal proof of the unbiasedness of the permutation test is in Maris & Oostenveld (2007), section "4.3.3. The permutation test controls the false alarm rate unconditionally”. I’m aware that very few readers go through this section, but it is one of
the 2 reasons for the popularity of the method. The other reason is its sensitivity, which is the result of clustering.</div>
<div><br class="">
</div>
<div>best,</div>
<div>Eric</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="gmail_extra">
<div class="gmail_quote">On Oct 27, 2016 7:58 AM, "Maris, E.G.G. (Eric)" <<a href="mailto:e.maris@donders.ru.nl" class="">e.maris@donders.ru.nl</a>> wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Dear colleagues,
<div class=""><br class="">
</div>
<div class="">@alik : </div>
<div class="">1. The approach you propose is a so-called fixed-effects approach, of which the outcome may depend on just a few subjects (provided the number of trials is high). Some neuroscientists consider a fixed-effects approach insufficient to support a
scientific claim. E.g., the whole neuroimaging community does so. </div>
<div class="">2. Your approach is actually a genuine permutation test, in which the LLM-derived t-stats are only used for thresholding (and not for inference).</div>
<div class=""><br class="">
</div>
<div class="">@david : </div>
<div class="">1. There is nothing wrong with using FDR correction, if you think the false discovery rate is the quantity that one should control. Others may disagree though, stating the more strict family-wise error rate is the relevant quantity.</div>
<div class="">2. FDR correction assumes that the sample-specific (sample = a channel-time-frequency triplet) p-values are unbiased. Because the unbiasedness of these p-values depends on auxiliary assumptions, there may be good reasons not to trust them. This
is supported by the recent Ekstrom et al paper on the inflated type 1 error rate in neuroimaging studies.</div>
<div class=""><br class="">
</div>
<div class="">best,</div>
<div class="">Eric Maris</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;margin:0px" class="">
<span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">From:<span class="m_-3435853957900395105Apple-converted-space"> </span></b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">David
Groppe <<a href="mailto:david.m.groppe@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">david.m.groppe@gmail.com</a>><br class="">
</span></div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;margin:0px" class="">
<span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Subject:<span class="m_-3435853957900395105Apple-converted-space"> </span></b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class=""><b class="">Re:
[FieldTrip] Question about cluster-based permutation tests on linear mixed models</b><br class="">
</span></div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;margin:0px" class="">
<span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Date:<span class="m_-3435853957900395105Apple-converted-space"> </span></b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">26
October 2016 at 19:35:03 GMT+2<br class="">
</span></div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;margin:0px" class="">
<span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">To:<span class="m_-3435853957900395105Apple-converted-space"> </span></b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">FieldTrip
discussion list <<a href="mailto:fieldtrip@science.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@science.ru.nl</a>>, <<a href="mailto:alik.widge@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">alik.widge@gmail.com</a>><br class="">
</span></div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;margin:0px" class="">
<span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Reply-To:<span class="m_-3435853957900395105Apple-converted-space"> </span></b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">FieldTrip
discussion list <<a href="mailto:fieldtrip@science.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="">
<div class="">P.S. If you want to explore using FDR control to correct for multiple comparisons, I would not recommend limiting yourself to FieldTrip's FDR correction code (fdr.m). It only implements the<span class="m_-3435853957900395105Apple-converted-space"> </span><span class="m_-3435853957900395105gmail-pl-c">Benjamini-Yekutieli
FDR control procedure, which is guaranteed to control the FDR at or below the desired level, but tends to be quite overly conservative in practice. The more popular FDR control algorithm by<span class="m_-3435853957900395105Apple-converted-space"> </span></span><span class="m_-3435853957900395105gmail-pl-c">Benjamini
& Hochberg is not always<span class="m_-3435853957900395105Apple-converted-space"> </span></span><br class="">
<span class="m_-3435853957900395105gmail-pl-c">guaranteed to control the FDR at or below the desired level, but it is much less conservative and tends to accurately control FDR in practice. Here is some code for the<span class="m_-3435853957900395105Apple-converted-space"> </span></span><br class="">
<span class="m_-3435853957900395105gmail-pl-c">Benjamini & Hochberg algorithm:<br class="">
<br class="">
</span></div>
<span class="m_-3435853957900395105gmail-pl-c"><a href="https://www.mathworks.com/matlabcentral/fileexchange/27418-fdr-bh" style="color:purple;text-decoration:underline" target="_blank" class="">https://www.mathworks.com/<wbr class="">matlabcentral/fileexchange/<wbr class="">27418-fdr-bh</a><br class="">
<br class="">
</span></div>
<span class="m_-3435853957900395105gmail-pl-c">MATLAB's mafdr.m function that is part of the Bioinformatics toolbox also implements the<span class="m_-3435853957900395105Apple-converted-space"> </span></span><br class="">
<span class="m_-3435853957900395105gmail-pl-c">Benjamini & Hochberg algorithm.</span>
<div class=""><span class="m_-3435853957900395105gmail-pl-c"><br class="">
<br class="">
</span></div>
</div>
<div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br class="">
<div class="gmail_quote">On Tue, Oct 25, 2016 at 3:28 PM, David Groppe<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:david.m.groppe@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">david.m.groppe@gmail.<wbr class="">com</a>></span><span class="m_-3435853957900395105Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">I would definitely recommend running some simulations.<br class="">
<br class="">
</div>
<div class="">It might be simpler to use bootstrap samples rather than permutations to generate your null distribution. Bootstrapping in also asymptotically accurate.<span class="m_-3435853957900395105HOEnZb"><font color="#888888" class=""><br class="">
</font></span></div>
<span class="m_-3435853957900395105HOEnZb"><font color="#888888" class="">
<div class=""> -David<br class="">
<br class="">
<br class="">
</div>
</font></span></div>
<div class="m_-3435853957900395105HOEnZb">
<div class="m_-3435853957900395105h5">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Tue, Oct 25, 2016 at 1:29 PM, Alik Widge<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:alik.widge@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">alik.widge@gmail.com</a>></span><span class="m_-3435853957900395105Apple-converted-space"> </span>w<wbr class="">rote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">Thanks, that was super interesting! Was not aware of those.<span class="m_-3435853957900395105Apple-converted-space"> </span><br class="">
<br class="">
Have been meditating this afternoon on this and related Anderson papers. What's interesting is that he appears to think my suggestion below *would* be asymptotically acceptable -- *if* one specifically permutes the dependent variable (power/ERP observation)
rather than permuting each column of the independent variables separately (i.e., if one preserves any correlational structure that exists between the independent variables). That's the Manly (1997) method, and it appears that the only reason it breaks down
sometimes is if there's an outlier in the independent variable. This could presumably be a problem in the ecological sciences, for which he's writing, where one can't control things like temperature in a season or numbers of eels that swim past a given sensor.
In cognitive neuroscience, where the predictor/independent variables are usually dummy coded properties of the trial, this seems like we might be on firmer ground.<span class="m_-3435853957900395105Apple-converted-space"> </span><br class="">
<br class="">
</div>
Opinion based on reading and reasoning, of course, and not to be trusted until and unless I or someone else were to back it up by doing some simulated-data experiments...<br class="">
<br class="">
</div>
<div class="gmail_extra"><span class=""><br clear="all" class="">
<div class="">
<div class="m_-3435853957900395105m_-5148554780153790566m_3649430948774947823gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">Alik Widge<br class="">
<a href="mailto:alik.widge@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">alik.widge@gmail.com</a><br class="">
<a href="tel:%28206%29%20866-5435" value="+12068665435" style="color:purple;text-decoration:underline" target="_blank" class="">(206) 866-5435</a><br class="">
<br class="">
</div>
</div>
</div>
<br class="">
</span>
<div class="">
<div class="m_-3435853957900395105m_-5148554780153790566h5">
<div class="gmail_quote">On Tue, Oct 25, 2016 at 11:30 AM, David Groppe<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:david.m.groppe@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">david.m.groppe@gmail.<wbr class="">com</a>></span><span class="m_-3435853957900395105Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">
<div class="">Hi Elisabeth and Alik,<br class="">
Permutation methods applied to multiple regression models are not generally guaranteed to be accurate because testing individual terms in such models (e.g., partial correlation coefficients) requires accurate knowledge of other terms in the model (e.g.,
the slope coefficients for all the other predictors in the multiple regression). Because such parameters have to be estimated from the data, permutation tests are only ‘‘asymptotically exact’’ for such tests (Anderson, 2001; Good, 2005). Though there are special
cases (e.g., a two factor ANOVA with two levels of each factor), where permutation methods do guarantee accuracy.<br class="">
</div>
In lieu of permutation testing, you might want to try using one of Benjamini and colleagues' false discovery rate (FDR) control algorithms to control for multiple comparisons. In my tests on simulated ERP data (Groppe et al., 2011), FDR correction was nearly
as powerful as cluster-based permutation testing for detecting a very broadly distributed effect (e.g., a P300-like effect) and it was far more sensitive than cluster-based testing for an effect with a very limited distribution (e.g., an N170-like effect).
FDR correction is also very computationally efficient.<br class="">
</div>
hope this is helpful,<br class="">
</div>
-David<br class="">
<br class="">
<br class="">
</div>
Refs:<br class="">
Anderson, M. J. (2001). Permutation tests for univariate or multivariate analysis of variance and regression.<span class="m_-3435853957900395105Apple-converted-space"> </span><i class="">Canadian journal of fisheries and aquatic sciences</i>,<span class="m_-3435853957900395105Apple-converted-space"> </span><i class="">58</i>(3),
626-639.<br class="">
<br class="">
Good, P. I. (2005). Permutation, Parametric and Bootstrap Tests of Hypotheses: A Practical Guide to Resampling Methods for Testing Hypotheses.<br class="">
<br class="">
Groppe, D. M., Urbach, T. P., & Kutas, M. (2011). Mass univariate analysis of event‐related brain potentials/fields II: Simulation studies.<span class="m_-3435853957900395105Apple-converted-space"> </span><i class="">Psychophysiology</i>,<span class="m_-3435853957900395105Apple-converted-space"> </span><i class="">48</i>(<wbr class="">12),
1726-1737.<br class="">
<div class="">
<div class="">
<div class=""><br class="">
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Fri, Oct 21, 2016 at 1:38 PM, Elisabeth May<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:elisabethsusanne.may@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">elisabethsusanne.may@<wbr class="">gmail.com</a>></span><span class="m_-3435853957900395105Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">Dear Eric and Alik,
<div class=""><br class="">
</div>
<div class="">thanks a lot for your helpful responses!</div>
<div class=""><br class="">
</div>
<div class="">I will have a close look at the faqs, Eric, and test the approaches you outlined. I am curious, anyway, as to how different results will be for simple regressions compared to the multilevel results of the linear-mixed models. </div>
<div class=""><br class="">
</div>
<div class="">Like Alik, I am also curious about other people's opinions on the general question if there are theoretical reasons against a combination of the approaches like Alik suggested. We also thought about this approach but haven't fully tested it yet
because of the very long calculation times. </div>
<div class=""><br class="">
</div>
<div class="">Thanks again and have a nice weekend!</div>
<div class="">Elisabeth</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">2016-10-20 12:49 GMT+02:00 Alik Widge<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:alik.widge@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">alik.widge@gmail.com</a>></span>:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">Eric, I don't think I understand why you would say "I do not see how these models could be combined with permutation-based inference; they are just different statistical frameworks". As you somewhat hint, the (G)LMM is a regression, and the beta
coefficient for the independent-variable of interest at each voxel/vertex/sensor x timepoint can be interpreted as "how much does the independent variable explain the brain activity?" In that framework, it seems to me that one could do the following:<br class="">
<br class="">
</div>
<div class="">for n=1:1000<br class="">
</div>
<div class=""> 1) Permute the condition labels (within subjects) of the individual trials<br class="">
</div>
<div class=""> 2) Re-fit the LMM at each (voxel,timepoint), creating a beta map and corresponding t-map<br class="">
</div>
<div class=""> 3) Threshold and construct cluster mass statistic as usual<br class="">
</div>
<div class="">end<br class="">
</div>
<div class="">4) Identify cluster in the original (unpermuted) analysis and report cluster p-value<br class="">
</div>
<br class="">
<br class="">
Now, the main thing that has come up when we've tried to do this is that re-fitting a (voxel x time) GLM 1000 times by the standard iterative maximum-likelihood engines is remarkably slow. In fieldtrip, I can imagine it would require rewriting at least a statfun,
maybe other pieces of the code. (We had an idea that, since the betas likely should vary smoothly over time and space, one could use the output of one GLM as the seed to the next, which would speed up convergence.) So it still does not seem like a good idea,
but based on the above, is there actually a *theoretical* reason it wouldn't work?<br class="">
<br class="">
<br class="">
Alik Widge, MD, PhD<br class="">
Director, Translational NeuroEngineering Laboratory<br class="">
Division of Neurotherapeutics, Massachusetts General Hospital<br class="">
Assistant Professor of Psychiatry, Harvard Medical School<br class="">
Clinical Fellow, Picower Institute for Learning & Memory (MIT)<br class="">
<a href="mailto:awidge@partners.org" style="color:purple;text-decoration:underline" target="_blank" class="">awidge@partners.org</a><br class="">
<a href="http://scholar.harvard.edu/awidge/" style="color:purple;text-decoration:underline" target="_blank" class="">http://scholar.harvard.edu/awi<wbr class="">dge/</a><br class="">
<a href="tel:617-643-2580" value="+16176432580" style="color:purple;text-decoration:underline" target="_blank" class="">617-643-2580</a><br class="">
</div>
<div class="gmail_extra"><br clear="all" class="">
<div class="">
<div class="m_-3435853957900395105m_-5148554780153790566m_3649430948774947823m_5020537499326629206m_-2394943662401627580m_-6489643051603551350gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">Alik Widge<br class="">
<a href="mailto:alik.widge@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">alik.widge@gmail.com</a><br class="">
<a href="tel:%28206%29%20866-5435" value="+12068665435" style="color:purple;text-decoration:underline" target="_blank" class="">(206) 866-5435</a><br class="">
<br class="">
</div>
</div>
</div>
<br class="">
<div class="gmail_quote">
<div class="">
<div class="m_-3435853957900395105m_-5148554780153790566m_3649430948774947823m_5020537499326629206m_-2394943662401627580h5">
On Thu, Oct 20, 2016 at 6:08 AM, Maris, E.G.G. (Eric)<span class="m_-3435853957900395105Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:e.maris@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">e.maris@donders.ru.nl</a>></span><span class="m_-3435853957900395105Apple-converted-space"><wbr class=""> </span>wrote:<br class="">
</div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">
<div class="m_-3435853957900395105m_-5148554780153790566m_3649430948774947823m_5020537499326629206m_-2394943662401627580h5">
<div style="word-wrap:break-word" class="">
<div dir="auto" style="word-wrap:break-word" class="">Note: this is the second time I post this reply, and the reason is that I forgot to add an appropriate Subject (for findability) to my email (shame on me…(-;)</div>
<div dir="auto" style="word-wrap:break-word" class=""><br class="">
</div>
<div dir="auto" style="word-wrap:break-word" class="">
<div style="word-wrap:break-word" class="">
<blockquote type="cite" class="">
<div class="">
<div style="margin:0px" class=""><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">From: </b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">Elisabeth
May <<a href="mailto:elisabethsusanne.may@gmail.com" style="color:purple;text-decoration:underline" target="_blank" class="">elisabethsusanne.may@gmail.co<wbr class="">m</a>><br class="">
</span></div>
<div style="margin:0px" class=""><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Subject: </b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class=""><b class="">[FieldTrip]
Question about cluster-based permutation tests on linear mixed models</b><br class="">
</span></div>
<div style="margin:0px" class=""><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Date: </b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">27
September 2016 at 14:46:55 GMT+2<br class="">
</span></div>
<div style="margin:0px" class=""><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">To: </b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class=""><<a href="mailto:fieldtrip@science.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<div style="margin:0px" class=""><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif;color:rgb(127,127,127)" class=""><b class="">Reply-To: </b></span><span style="font-family:-webkit-system-font,'Helvetica Neue',Helvetica,sans-serif" class="">FieldTrip
discussion list <<a href="mailto:fieldtrip@science.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<br class="">
<br class="">
<div dir="ltr" class=""><span style="font-size:12.8px" class="">Dear FieldTripers, </span>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">I have a question about the potential use of cluster-based permutation tests for results obtained using linear mixed models. </div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">We are working with data from a 10 min EEG experiment on source level with the aim to quantify the relationship of brain activity in different frequency bands with continous perceptual ratings across 20 subjects in different
experimental conditions. Thus, we have 10 min time courses of brain activity and ratings for each voxel for different conditions and want to test a) if there are significant relationships in the single conditions and b) if these relationships differ between
two conditions. To this end, I have calculated linear mixed models in R using the lme4 toolbox. For both the single condition relationships and the condition contrasts, they result in a single t-value (and a corresponding p-value), which is based on information
on both the single subject and the group level (i.e. we perform a multi-level analysis). However, with more than 2000 voxels, we have a lot of t-values and are wondering if there is a way to apply cluster-based tests to correct for multiple comparisons. </div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">The main problem I see is that I only have one multilevel t-value for the effect across all subjects, i.e. I don't have single subjects values, which I could then e.g. randomize between conditions as normally done in cluster-based
permutation tests. (Or rather, I would be able to extract single subject values but would then loose the advantage of the multi-level analysis.)</div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">I found an old thread in the mailinglist archive where it was suggested to flip the signs of the t-statistic for cluster-level correction (<a href="https://mailman.science.ru.nl/pipermail/fieldtrip/2012-July/005375.html" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl<wbr class="">/pipermail/fieldtrip/2012-July<wbr class="">/005375.html</a>).
I understand that, in our case, I would do this randomly for all voxels in each randomization and then build spatial clusters on the resulting (partly flipped) t-values. However, I am not sure if that is a valid approach based on the null hypothesis that there
are no significant relations in my single conditions (a) or no significant relationship differences in my condition contrasts (b). </div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">For the condition contrasts, I would be able to permute the condition labels as normally done in cluster-based permutation tests,I think, but would then have to recalculate the linear mixed models for all voxels in every
permutation. This would result in a very high computational load. </div>
<div style="font-size:12.8px" class=""><br class="">
</div>
<div style="font-size:12.8px" class="">Does anyone have any experience with this kind of analysis? Would the flipping of t-values be a valid approach (and if yes, is there anything to keep in mind in particular)? Can you think of other ways to combine linear
mixed models with a multiple comparison correction on the cluster level?</div>
<div style="font-size:12.8px" class=""><br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Hi Elisabeth,
<div class=""><br class="">
</div>
<div class="">I’m not an expert on linear mixed modelling, at least not with respect to the different ways in which they can be used to deal with correlated observations (typically, time series). However, from a theoretical point of view, I do not see how these
models could be combined with permutation-based inference; they are just different statistical frameworks. However, it IS possible to answer your questions ("<span style="white-space:pre-wrap" class="">we have 10 min time courses of brain activity and ratings
for each voxel for different conditions and wan to test a) if there are significant relationships in the single conditions and b) if these relationships differ between two conditions.”) within the framework of cluster-based permutation tests. Question b) is
the most straightforward because it amounts to a cluster-based permutation test using the depsamplesT statfun applied to the regression coefficients in each of the two conditions. Answering question a) requires that you bin your ratings in a number of categories,
calculate the trial-averaged EEG data for each of the categoreies, and test the difference between them using a cluster-based permutation test using the depsamplesregrT statfun. Both of these approaches have been described previously on this discussion list,
and for the depsamplesregrT statfun (your question a), it was Vladimir Litvak who used it first (actually, I implemented it for him). The approach for question b) is actually a variant on the general approach for testing interactions using cluster-based permutation
tests. </span></div>
<div class=""><span style="white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span style="white-space:pre-wrap" class="">Have a look here:</span></div>
<div class=""><span style="white-space:pre-wrap" class=""><a href="http://www.fieldtriptoolbox.org/faq/how_can_i_test_for_correlations_between_neuronal_data_and_quantitative_stimulus_and_behavioural_variables" style="color:purple;text-decoration:underline" target="_blank" class="">http://www.fieldtriptoolbox.or<wbr class="">g/faq/how_can_i_test_for_corre<wbr class="">lations_between_neuronal_data_<wbr class="">and_quantitative_stimulus_and_<wbr class="">behavioural_variables</a></span></div>
<div class=""><span style="white-space:pre-wrap" class="">and </span></div>
<div class=""><span style="white-space:pre-wrap" class=""><a href="http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests" style="color:purple;text-decoration:underline" target="_blank" class="">http://www.fieldtriptoolbox.or<wbr class="">g/faq/how_can_i_test_an_intera<wbr class="">ction_effect_using_cluster-bas<wbr class="">ed_permutation_tests</a></span></div>
<div class=""><span style="white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span style="white-space:pre-wrap" class="">These tutorials provide all the necessary concepts, although they do not answer your question in a recipe-like fashion.</span></div>
<div class=""><span style="white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span style="white-space:pre-wrap" class="">best,</span></div>
<div class=""><span style="white-space:pre-wrap" class="">Eric Maris</span></div>
</div>
</div>
<div class=""><br class="">
</div>
</div>
</div>
<br class="">
</div>
</div>
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
<br class="">
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
<br class="">
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
<br class="">
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" style="color:purple;text-decoration:underline" target="_blank" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" style="color:purple;text-decoration:underline" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" class="">fieldtrip@donders.ru.nl</a><br class="">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" target="_blank" class="">https://mailman.science.ru.nl/<wbr class="">mailman/listinfo/fieldtrip</a><br class="">
</blockquote>
</div>
</div>
<br class="">
<br class="">
<br class="">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">From:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Müller, K. (Katja) <<a href="mailto:K.Muller@psych.ru.nl" class="">K.Muller@psych.ru.nl</a>><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Subject:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class=""><b class="">[FieldTrip] MNE single trial time courses</b><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Date:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">28 October 2016 at 15:06:59 GMT+2<br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">"<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>" <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Reply-To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">FieldTrip discussion list <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<br class="">
<br class="">
Dear list,<br class="">
<br class="">
What is the current way to go (if there is one) to obtain *single trial* source time courses from source reconstructions from Minimum Norm Estimate? Is there a tutorial?<br class="">
<br class="">
I used the MNE tutorial as a basis, i.e. I have a FreeSurfer based source model. In old mailing list messages there is some information about options like .rawtrial='yes' or .singletrial='yes', but some of them are very old and I am unable to extract which
way to go at the moment. E.g. .singletrial is deprecated.<br class="">
<br class="">
<br class="">
Best regards,<br class="">
Katja<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">From:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">mehdy dousty <<a href="mailto:mehdy.dousty@gmail.com" class="">mehdy.dousty@gmail.com</a>><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Subject:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class=""><b class="">[FieldTrip] Covaraince matrix for MEG resting state</b><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Date:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">28 October 2016 at 18:49:03 GMT+2<br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">"<a href="mailto:hcp-users@humanconnectome.org" class="">hcp-users@humanconnectome.org</a>" <<a href="mailto:hcp-users@humanconnectome.org" class="">hcp-users@humanconnectome.org</a>>,
<<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class="">
<span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(127, 127, 127, 1.0);" class=""><b class="">Reply-To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">FieldTrip discussion list <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<br class="">
<br class="">
<div dir="ltr" class="">Hi all
<div class="">I am using the resting state MEG signal for constructing the inverse problem by eLoreta, and as it is resting state, the covariance matrix needs to be used, so I am going to use HCP R-noise to construct the covariance matrix, but there are two
main issues which I need your help. 1- as I am going to use preprocess data for resting state, do I need process the noise as well? if it is so, I can't find the time series of the noise in the raw data after using ft_read_header. 2- do I need to redefine
the time series which add the noise time series at the beginning of the signal and then add the resting state time series so I can use the below code for computing the covariance?</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<pre class="inbox-inbox-inbox-inbox-m_-5691756089817090439inbox-inbox-code gmail_msg" style="color:rgb(33,33,33);padding:0.5em;font-size:12px;border:1px dashed rgb(204,204,204);overflow:hidden;font-family:consolas,"andale mono",menlo,monaco,monospace;background-color:rgb(247,249,250);white-space:pre-wrap;width:640px;text-align:justify">cfg= []</pre>
<pre class="inbox-inbox-inbox-inbox-m_-5691756089817090439inbox-inbox-code gmail_msg" style="color:rgb(33,33,33);padding:0.5em;font-size:12px;border:1px dashed rgb(204,204,204);overflow:hidden;font-family:consolas,"andale mono",menlo,monaco,monospace;background-color:rgb(247,249,250);white-space:pre-wrap;width:640px;text-align:justify"> cfg.covariance = 'yes'; % I dont know how to enter the noise covariance to the data</pre>
<pre class="inbox-inbox-inbox-inbox-m_-5691756089817090439inbox-inbox-code gmail_msg" style="color:rgb(33,33,33);padding:0.5em;font-size:12px;border:1px dashed rgb(204,204,204);overflow:hidden;font-family:consolas,"andale mono",menlo,monaco,monospace;background-color:rgb(247,249,250);white-space:pre-wrap;width:640px;text-align:justify">cfg.covariancewindow = [-inf 0];
timelockanalysis = ft_timelockanalysis(cfg, inputdata);
</pre>
</div>
<div class=""><br class="">
</div>
<div class="">I look forward to hearing from you.</div>
<div class="">
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class=""><br class="inbox-inbox-Apple-interchange-newline">
Mehdy Dousty</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">Hotchkiss Brain Institute</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">University of Calgary</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">HSC Building, Room 2932B</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">3330 Hospital Drive NW</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">Calgary, AB T2N 4N1</b></span></font></div>
<div style="font-size:16px;font-family:calibri,arial,helvetica,sans-serif" class="">
<font face="Helvetica" size="1" class=""><span style="font-size:12px" class=""><b class="">Email <a href="mailto:Mehdy.Dousty@Ucalgary.ca" class="">Mehdy.Dousty@Ucalgary.ca</a></b></span></font></div>
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<br class="">
_______________________________________________<br class="">
fieldtrip mailing list<br class="">
<a href="mailto:fieldtrip@donders.ru.nl" class="">fieldtrip@donders.ru.nl</a><br class="">
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>