[FieldTrip] interactions

[Eric Maris]

Hi Steve and Josh,


Josh writes

> > labels. I'm sure there's a proof somewhere for why this doesn't work,
> > and it would be great to see it.

In general, questions like these are very hard to answer satisfactorily on a 
discussion list. It is dealt with much more easily in person, say at one of 
the Fieldtrip courses. However, let me give it a try.

To prove that something does not work it suffices to produces a single 
example that shows the contrary.

Try the following:

Generate random data in a 2-by-2 between-subjects design (say, normally 
distributed within every cell). Add large main effects (relative to the 
within-cell variance; say, MS_beween 50 times larger than MS_within) and no 
interaction effect. Take a small number of subjects (say, 5 per cell). Now, 
calculate a permutation p-value for the interaction-effect F-statistic by 
permuting across all 4 cells. Do this for a large number of simulated data 
set. My prediction is that, on average, the F-statistic p-value is less than 
0.05, which it should be (because there is no interaction effect).

I have not run this simulation study myself. Let me know if it does not 
produce the predicted result. (I cannot guarantee that I'm not missing 
something when producing this recipe.)



Best,

Eric






> -----Original Message-----
> From: Stephen Politzer-Ahles [mailto:politzerahless at gmail.com]
> Sent: zondag 26 januari 2014 8:25
> To: fieldtrip at science.ru.nl
> Subject: Re: [FieldTrip] interactions
>
> Hi Josh,
>
> Have you seen this [admittedly pretty old now] message from the
> archives: http://mailman.science.ru.nl/pipermail/fieldtrip/2011-
> January/003447.html
> ? My understanding was that it is ok to test interactions in within-
> subjects designs, and that you could do it by faking a dataset that
> represents the interaction (step 3 in that message) and then doing a
> dependent samples t-test. I had never heard before that interactions
> can't be tested in a within-subjects design, but also it's been a long
> time since I've looked at this issue--I'd definitely be interested to
> hear if this is no longer the recommended way to test interactions. I
> have seen messages saying that it doesn't work for between-subjects
> designs (e.g.
> http://mailman.science.ru.nl/pipermail/fieldtrip/2011-
> September/004244.html),
> but I'm not sure if that's still current. Hopefully someone on the list
> can offer more insight about the second question.
>
> Best,
> Steve
>
> >
> > Message: 2
> > Date: Fri, 24 Jan 2014 10:54:10 -0500
> > From: Joshua Hartshorne <jkhartshorne at gmail.com>
> > To: fieldtrip at science.ru.nl
> > Subject: [FieldTrip] interactions
> > Message-ID:
> >
> > <CA+3amhe+x4+TNUY1tf0aXe-cf-AB1kTE+ZHTpuRJxNQ=bNioUQ at mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hi List!
> >
> > I have seen around a dozen comments in the archives that interactions
> > can't be tested by permutation for within-subject designs. I haven't
> > been able to find a thread that explains why not. It seems like in a
> > 2x2 design, you could still pick one of the conditions and permute
> the
> > labels. I'm sure there's a proof somewhere for why this doesn't work,
> > and it would be great to see it.
> >
> > Similarly, for the mixed design, why permute the between-subject
> labels?
> > Why not permute the within-subject labels instead? Actually, why not
> > do both? I follow the reasoning why permuting both is overkill, but
> > not why it's wrong.
> >
> > If someone could explain, it would be much appreciated. Knowing what
> > to do is good, but it would be even better to understand why.
> >
> > Thanks,
> > Josh
> > -------------- next part -------------- An HTML attachment was
> > scrubbed...
> > URL:
> >
> <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20140124/
> b885cb4a/attachment-0001.html>
> >