<div dir="ltr">Hi Eric,<div><br></div><div>I tried the simulations as you suggested. Rather I simulated several different types of 2X2 datasets. For all of them, permutation analysis worked just fine. Of course, it's possible that I'm doing something wrong, so I'd appreciate your feedback.</div>
<div><br></div><div>Here's how I created the data:</div><div><br></div><div>
<b>between/between</b>. Normally-distributed data in each of 4 cells. The effect size for e1 and e2 were each set to 1, with a SD of 1. The interaction (if present) was .75 with an SD of 1. There were 40 subjects per cell. <div class="gmail_extra">
<br></div><div class="gmail_extra"><b>within/within. </b>For each participant, I generated a random intercept (M=0, SD=.25) and random slopes for both e1 and e2 (M=1, SD=.25) and for the interaction, if present (M=.75, SD=.25). Having generated that, for each participant in each condition, I drew a single sample where the mean was the sum of the effects just listed and the SD=1. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><b>within/between. </b>The first factor was within subjects and the second was between. Each subject had a random intercept (SD=.25) and a random slope for factor 1 (M=1, SD=.25). The subjects in Level 1 for both factors had a random intercept (M=.75, SD=.25). The between-subject factor was 1. As above, I then generated a single datapoint from a normal distribution (M=1, SD=1). </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">In each case, I ran 500 simulations with an interaction and 500 without. For each, I analyzed either with an ANOVA (ezANOVA in R) or a 500-sample permutation test, as follows: Permutations respected the structure of the data. So in the between/between case, condition labels were permuted freely. In the within/within case, for each subject, I randomly flipped the levels of each factor, preserving structure. That is, each subject had two cells where factor 1 was 0 and two where factor 1 was 1. If the codes switched, both the 1s were turned to 0s and both the 0s were turned to 1s. The same was done for Factor 2. I dealt with the within/between data in an analogous fashion, with the constraint that the same number of subjects be in the each of the between-subject conditions. Having done my permuting, I then calculated the F-stat for the interaction. I then compared the actual F-stat against the resulting distribution.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">The short description of the result is I got basically the same results for the permutation tests and the ANOVA. For instance, in the within/within case, when there was an actual interaction in the generative model, I got an average p-value of .0844 using ezANOVA and .0858 using permutations. The Type II error was .318 and .312, respectively. When there was no interaction, I got average p-values of .5052 and .5066, respectively, and a Type I error of .036 and .038, respectively. I got analogous results for between/between and within/within. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Incidentally, I understand that it isn't strictly necessary to permute condition codes for both factors. But it doesn't seem to do any harm, either. I actually tried the within/between case permuting only the between factor, with similar results. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra">Josh</div><div class="gmail_extra"><br></div><div class="gmail_extra"> <br></div><div class="gmail_extra"><div class="gmail_quote">
On Sun, Jan 26, 2014 at 4:44 AM, <span dir="ltr"><<a href="mailto:fieldtrip-request@science.ru.nl" target="_blank">fieldtrip-request@science.ru.nl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Steve and Josh,<br>
<br>
<br>
Josh writes<br>
<br>
> > labels. I'm sure there's a proof somewhere for why this doesn't work,<br>
> > and it would be great to see it.<br>
<br>
In general, questions like these are very hard to answer satisfactorily on a<br>
discussion list. It is dealt with much more easily in person, say at one of<br>
the Fieldtrip courses. However, let me give it a try.<br>
<br>
To prove that something does not work it suffices to produces a single<br>
example that shows the contrary.<br>
<br>
Try the following:<br>
<br>
Generate random data in a 2-by-2 between-subjects design (say, normally<br>
distributed within every cell). Add large main effects (relative to the<br>
within-cell variance; say, MS_beween 50 times larger than MS_within) and no<br>
interaction effect. Take a small number of subjects (say, 5 per cell). Now,<br>
calculate a permutation p-value for the interaction-effect F-statistic by<br>
permuting across all 4 cells. Do this for a large number of simulated data<br>
set. My prediction is that, on average, the F-statistic p-value is less than<br>
0.05, which it should be (because there is no interaction effect).<br>
<br>
I have not run this simulation study myself. Let me know if it does not<br>
produce the predicted result. (I cannot guarantee that I'm not missing<br>
something when producing this recipe.)<br>
<br>
<br>
<br>
Best,<br>
<br>
Eric<br>
<br>
<br>
<br>
<br>
<br>
<br>
> -----Original Message-----<br>
> From: Stephen Politzer-Ahles [mailto:<a href="mailto:politzerahless@gmail.com" target="_blank">politzerahless@gmail.com</a>]<br>
> Sent: zondag 26 januari 2014 8:25<br>
> To: <a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a><br>
> Subject: Re: [FieldTrip] interactions<br>
><br>
> Hi Josh,<br>
><br>
> Have you seen this [admittedly pretty old now] message from the<br>
> archives: <a href="http://mailman.science.ru.nl/pipermail/fieldtrip/2011-" target="_blank">http://mailman.science.ru.nl/pipermail/fieldtrip/2011-</a><br>
> January/003447.html<br>
> ? My understanding was that it is ok to test interactions in within-<br>
> subjects designs, and that you could do it by faking a dataset that<br>
> represents the interaction (step 3 in that message) and then doing a<br>
> dependent samples t-test. I had never heard before that interactions<br>
> can't be tested in a within-subjects design, but also it's been a long<br>
> time since I've looked at this issue--I'd definitely be interested to<br>
> hear if this is no longer the recommended way to test interactions. I<br>
> have seen messages saying that it doesn't work for between-subjects<br>
> designs (e.g.<br>
> <a href="http://mailman.science.ru.nl/pipermail/fieldtrip/2011-" target="_blank">http://mailman.science.ru.nl/pipermail/fieldtrip/2011-</a><br>
> September/004244.html),<br>
> but I'm not sure if that's still current. Hopefully someone on the list<br>
> can offer more insight about the second question.<br>
><br>
> Best,<br>
> Steve<br>
><br>
> ><br>
> > Message: 2<br>
> > Date: Fri, 24 Jan 2014 10:54:10 -0500<br>
> > From: Joshua Hartshorne <<a href="mailto:jkhartshorne@gmail.com" target="_blank">jkhartshorne@gmail.com</a>><br>
> > To: <a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a><br>
> > Subject: [FieldTrip] interactions<br>
> > Message-ID:<br>
> ><br>
> > <CA+3amhe+x4+TNUY1tf0aXe-cf-AB1kTE+ZHTpuRJxNQ=<a href="mailto:bNioUQ@mail.gmail.com" target="_blank">bNioUQ@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="iso-8859-1"<br>
> ><br>
> > Hi List!<br>
> ><br>
> > I have seen around a dozen comments in the archives that interactions<br>
> > can't be tested by permutation for within-subject designs. I haven't<br>
> > been able to find a thread that explains why not. It seems like in a<br>
> > 2x2 design, you could still pick one of the conditions and permute<br>
> the<br>
> > labels. I'm sure there's a proof somewhere for why this doesn't work,<br>
> > and it would be great to see it.<br>
> ><br>
> > Similarly, for the mixed design, why permute the between-subject<br>
> labels?<br>
> > Why not permute the within-subject labels instead? Actually, why not<br>
> > do both? I follow the reasoning why permuting both is overkill, but<br>
> > not why it's wrong.<br>
> ><br>
> > If someone could explain, it would be much appreciated. Knowing what<br>
> > to do is good, but it would be even better to understand why.<br>
> ><br>
> > Thanks,<br>
> > Josh<br>
> > -------------- next part -------------- An HTML attachment was<br>
> > scrubbed...<br>
> > URL:<br>
> ><br>
> <<a href="http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140124/" target="_blank">http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140124/</a><br>
> b885cb4a/attachment-0001.html><br>
> ><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sun, 26 Jan 2014 10:43:58 +0100<br>
From: Azeez Adebimpe <<a href="mailto:ayobimpe2004@hotmail.com" target="_blank">ayobimpe2004@hotmail.com</a>><br>
To: FieldTrip discussion list <<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a>><br>
Subject: Re: [FieldTrip] Urgent: Error in Source Statistics, Group<br>
level<br>
Message-ID: <DUB111-W130FC2F5C9CE7B2035F0BE4CDA30@phx.gbl><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hi Chaitanya ,<br>
I would suggest you try analyitcs instead of montecarlo and use stat= ft_sourcestatitics(cfg, source1a, source2a .................., source1b,source2b.............);a and b are for the conditions.<br>
Azeez Adebimpe<br>
<br>
<br>
Date: Sun, 26 Jan 2014 09:46:03 +0100<br>
From: <a href="mailto:chaitanya.pro@gmail.com" target="_blank">chaitanya.pro@gmail.com</a><br>
To: <a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a><br>
Subject: Re: [FieldTrip] Urgent: Error in Source Statistics, Group level<br>
<br>
Hi Eelke,<br>
<br>
No significant results then in my data. I wonder how my boss takes it :P.<br>
Anyway, thanks for your help on a Sunday that too.<br>
>From your reply I also understand that the code doesn't have any mistakes :)<br>
<br>
===============================================<br>
<br>
<br>
<br>
Best RegardsChaitanya Srinivas Lanka<br>
<br>
<br>
Wiss. Mitarbeiter PhD Student<br>
<br>
Functional and Restorative Neurosurgery Neural Information Processing<br>
Neurosurgical University Hospital Graduate Training Center for Neuroscience<br>
<br>
Eberhard Karls University Eberhard Karls University<br>
<br>
Otfried-Mueller-Str.45 ?sterbergstr. 3<br>
<br>
D-72076 Tuebingen D-72074 Tuebingen<br>
<br>
Mobile Phone Number : <a href="tel:%2B49-176-79035731" value="+4917679035731" target="_blank">+49-176-79035731</a><br>
===============================================<br>
<br>
<br>
<br>
<br>
<br>
On Sun, Jan 26, 2014 at 9:40 AM, Eelke Spaak <<a href="mailto:eelke.spaak@donders.ru.nl" target="_blank">eelke.spaak@donders.ru.nl</a>> wrote:<br>
<br>
Hi Chaitanya,<br>
stat.prob reflects the 'p-values' resulting from your statistical test. So voxels expressing e.g. stat.prob < 0.05 should be considered reflecting a significant difference between conditions. The NaNs correspond to voxels outside the brain.<br>
<br>
<br>
Since stat.mask is all zeros (which by default is just stat.prob < 0.05), this indicates there are no significant differences between your conditions. There is nothing we can help you with in this respect :)<br>
<br>
<br>
Best,Eelke<br>
<br>
On 26 January 2014 09:06, Chaitanya Srinivas <<a href="mailto:chaitanya.pro@gmail.com" target="_blank">chaitanya.pro@gmail.com</a>> wrote:<br>
<br>
<br>
Hi Eelke,<br>
<br>
I looked at the stat.stat values if that is what you mean. There are some NaNs , but also some values. Similarly in stat.prob, there are some 1's. The stat.mask is all zeros as you say.<br>
<br>
<br>
<br>
<br>
Any further suggestions from you?<br>
Thank you<br>
<br>
===============================================<br>
<br>
<br>
<br>
Best RegardsChaitanya Srinivas Lanka<br>
<br>
<br>
<br>
<br>
Wiss. Mitarbeiter PhD Student<br>
<br>
<br>
<br>
Functional and Restorative Neurosurgery Neural Information Processing<br>
Neurosurgical University Hospital Graduate Training Center for Neuroscience<br>
<br>
<br>
<br>
Eberhard Karls University Eberhard Karls University<br>
<br>
<br>
<br>
Otfried-Mueller-Str.45 ?sterbergstr. 3<br>
<br>
<br>
<br>
D-72076 Tuebingen D-72074 Tuebingen<br>
<br>
<br>
<br>
Mobile Phone Number : <a href="tel:%2B49-176-79035731" value="+4917679035731" target="_blank">+49-176-79035731</a><br>
===============================================<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Sun, Jan 26, 2014 at 8:53 AM, Eelke Spaak <<a href="mailto:eelke.spaak@donders.ru.nl" target="_blank">eelke.spaak@donders.ru.nl</a>> wrote:<br>
<br>
<br>
<br>
Dear Chaitanya,<br>
Perhaps an obvious question: do you find any significant differences in the statistics step (inspect the stat structure)? If not, the mask will consist of all zeroes, hence giving you a 'blank' plot.<br>
<br>
<br>
<br>
<br>
Best,Eelke<br>
<br>
On 26 January 2014 08:46, Chaitanya Srinivas <<a href="mailto:chaitanya.pro@gmail.com" target="_blank">chaitanya.pro@gmail.com</a>> wrote:<br>
<br>
<br>
<br>
<br>
Dear fieldtrip users,<br>
I would like to do sourcestatistics on a group level with eeg data. I have a<br>
pre and post intervention measurement for each of my 10 subjects<br>
. After source reconstruction using an DICS beamformer<br>
and volume normalization, I calculated the sourcegrandaverage for the pre and<br>
post condition and i have avg.pow for each subject.<br>
<br>
However, when I use the grandaverage results in ft_sourcestatistics in the<br>
configuration shown below and plot the result I just get a blank anatomical<br>
mri. It only runs with cfg.parameter="pow" .I tried with cfg.parameter = 'avg.pow' it doesnt run.<br>
Do I have to set any additional parameters or am I making some mistake?<br>
<br>
<br>
cfg=[];<br>
cfg.dim = grandAVGsourcePre.dim;<br>
cfg.method = 'montecarlo';<br>
cfg.statistic = 'depsamplesT';<br>
cfg.parameter = 'pow';<br>
cfg.correctm = 'cluster';<br>
cfg.numrandomization = 1000;<br>
cfg.alpha = 0.05;<br>
cfg.tail = 0;<br>
<br>
nsubj=length(sourcePre.trial);<br>
cfg.design(1,:) = [1:nsubj 1:nsubj];<br>
cfg.design(2,:) = [ones(1,nsubj) ones(1,nsubj)*2];<br>
cfg.uvar = 1;<br>
cfg.ivar = 2;<br>
stat = ft_sourcestatistics(cfg, grandAVGsourcePre, grandAVGsourcePost);<br>
and next interpolation<br>
<br>
cfg = [];<br>
<br>
<br>
<br>
<br>
<br>
cfg.voxelcoord = 'no';<br>
cfg.parameter = 'mask';<br>
cfg.interpmethod = 'nearest';<br>
cfg.coordsys = 'mni';<br>
<br>
<br>
<br>
<br>
<br>
mask = ft_sourceinterpolate(cfg,stat,mri);<br>
statplot.mask = mask.mask;<br>
<br>
<br>
and then for plotting<br>
<br>
<br>
<br>
<br>
<br>
cfg = [];<br>
cfg.method = 'slice';<br>
cfg.funparameter = 'stat';<br>
cfg.maskparameter = 'mask';<br>
cfg.funcolorlim = [-0.1 0.1];<br>
cfg.opacitylim = [-0.1 0.1];<br>
figure<br>
ft_sourceplot(cfg, statplot);<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
===============================================<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Best RegardsChaitanya Srinivas Lanka<br>
<br>
Wiss. Mitarbeiter PhD Student<br>
<br>
<br>
<br>
<br>
<br>
Functional and Restorative Neurosurgery Neural Information Processing<br>
Neurosurgical University Hospital Graduate Training Center for Neuroscience<br>
<br>
<br>
<br>
<br>
<br>
Eberhard Karls University Eberhard Karls University<br>
<br>
<br>
<br>
<br>
<br>
Otfried-Mueller-Str.45 ?sterbergstr. 3<br>
<br>
<br>
<br>
<br>
<br>
D-72076 Tuebingen D-72074 Tuebingen<br>
<br>
<br>
<br>
<br>
<br>
Mobile Phone Number : <a href="tel:%2B49-176-79035731" value="+4917679035731" target="_blank">+49-176-79035731</a><br>
===============================================<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
<br>
fieldtrip mailing list<br>
<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<br>
<br>
<br>
_______________________________________________<br>
<br>
fieldtrip mailing list<br>
<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<br>
<br>
<br>
_______________________________________________<br>
<br>
fieldtrip mailing list<br>
<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<br>
<br>
<br>
_______________________________________________<br>
<br>
fieldtrip mailing list<br>
<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<br>
<br>
<br>
_______________________________________________<br>
fieldtrip mailing list<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140126/9c7c30c0/attachment.html" target="_blank">http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140126/9c7c30c0/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: image.png<br>
Type: image/png<br>
Size: 23195 bytes<br>
Desc: not available<br>
URL: <<a href="http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140126/9c7c30c0/attachment.png" target="_blank">http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140126/9c7c30c0/attachment.png</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
fieldtrip mailing list<br>
<a href="mailto:fieldtrip@donders.ru.nl" target="_blank">fieldtrip@donders.ru.nl</a><br>
<a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip" target="_blank">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<br>
End of fieldtrip Digest, Vol 38, Issue 49<br>
*****************************************<br>
</blockquote></div><br></div></div></div>