[FieldTrip] Cluster-based permutation tests on time-frequency and size of conditions

[David Groppe]

Hi Martina,
   Balanced sample sizes are typically recommended for conventional
parametric independent samples tests (e.g., t-tests, ANOVAs) because it
makes the tests less sensitive to differences in variation between the
populations being compared. If the populations being compared differ in
variance, having more observations from the population with less
variability will make these tests overly permissive (i.e., the true false
positive rate will be greater than your nominal alpha level). If you have
more observations from the population with greater variability, the tests
become overly conservative.
   A few years ago, my colleagues and I simulated some EEG data and found
that permutation tests exhibit a qualitatively similar sensitivity to
differences in variance between populations (see below). If you're
concerned about such a difference in your data you could do as has already
been suggested and use a subset of data so that the number of observations
between samples is the same. Alternatively you could use a permutation test
based on variants of the t-statistic that are less sensitive to differences
in variance. In our paper below, we investigated two variants, Welch's t
and t_dif. Welch's t proved a bit less sensitive to differences in variance
and was only slightly less powerful than the conventional t-statistic.
t_dif was markedly insensitive to differences in variance but was
significantly less powerful. However, I would guess that using t_dif or
Welch's t are likely more powerful than discarding trials (though we didn't
investigate that option in the paper).
      cheers,
         -David

Groppe, D.M., Urbach, T.P., & Kutas, M. (2011) *Mass univariate analysis of
event-related brain potentials/fields II: Simulation studies*.
*Psychophysiology*, 48(12) pp. 1726-1737, DOI:
10.1111/j.1469-8986.2011.01272.x.
www.cogsci.ucsd.edu/~dgroppe/PUBLICATIONS/mass_uni_preprint2.pdf





On Thu, Mar 19, 2015 at 4:06 AM, Martina Rossi <martina.rossi76 at yahoo.it>
wrote:

> Dear Stephen and Joram,
>
> Thank so much for your feedback,
> I will check out the suggested material,
>
> Best,
> Martina
>
>
>   Il Mercoledì 18 Marzo 2015 20:43, Stephen Whitmarsh <
> stephen.whitmarsh at gmail.com> ha scritto:
>
>
> You can also check out this video of Robert. Apologies for the quality -
> not of the talk, but of the recording :-)
>
> At 14:45 he actually mentions unequal number of trials between conditions.
>
> https://www.youtube.com/watch?v=vOSfabsDUNg
>
> Cheers,
> Stephen
>
>
>
>
> On 18 March 2015 at 19:33, Stephen Whitmarsh <stephen.whitmarsh at gmail.com>
> wrote:
>
> I should add that (1) is typically done *within *subjects, and (2) *over *
> subjects.
>
> Cheers,
> Stephen
>
> On 18 March 2015 at 19:20, Stephen Whitmarsh <stephen.whitmarsh at gmail.com>
> wrote:
>
> Dear Martina,
>
> It might help to distinguish two aspects of cluster-based statistic.
>
> 1) the statistical approuch that you will use to determine whether a
> time-channel-datapoint / time-frequency-channel-datapoint /
> time-frequency-voxel-datapoint is considered significant different between
> conditions.
> 2) the statistical approuch that you will use to determine whether *a
> cluster of* time-channel-datapoints / time-frequency-channel-datapoints /
> time-frequency-voxel-datapoints is considered significantly different
> between conditions.
>
> When you talk about *cluster statistics*, you probably think about the
> second part. But this might not be what you should initially be concerned
> with when thinking about e.g. different numbers of trials between
> conditions. Rather, consider what statistical tests you (can) use to
> compare your time-frequency values between conditions *(within subjects).*
> This can be, e.g., a t-test, a nonparametric (e.g. montecarlo) test, or any
> test, for that matter. As far as my limited knowledge of statistics goes,
> in most simple and non-extreme cases, unequal number of trials that does
> not have to increase your chance of type I errors, rather that of type 2
> (you'll be insensitive to differences if you don't have enough observations
> in one condition due to noisy estimate of means/distribution). But in any
> case it's a simple question to google or ask a statistician.
>
> Now, after you are happy with and confident about the between conditions
> statistical test, consider how the cluster statistics might help you.
> First of all, how does it determine whether a cluster is significantly
> different between conditions? There are different options, but the gist is
> that it takes your significant statistical numbers of step (1), adds them
> up when they belong to the same cluster (based on whether they are
> neigbourings in time/freq/space with other significant numbers), takes the
> maximum of these summed up clusters (there might be more than one cluster),
> and then compares this one value to the same taken from a*(non-parametric)
> monte-carlo distribution *of the null hypothesis based on permuting the
> values over conditions (and then calculating the maximum sum). The Maris
> and Oostenveld paper explains it in more detail.
>
> The reason for doing cluster-statistics is that its a smart way of dealing
> with multiple comparisons over many time x frequency x channels (or space).
> The method is blind for your decisions about how its computed for each
> point in time x frequency x channels (or space).
>
> I find the FieldTrip statistics functions, their configurations etc., and
> the way they interact confusing at times, but I hope this helps to clear it
> up a bit.
>
> Long story short - I think your question does not limit itself to cluster
> statistics and at the same time is much simpler. It's all about (1).
>
> Best wishes,
> Stephen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> There are two separate steps cluster statistics (as implemented in
> FieldTrip, but in general as well).
>
>
>
>
> On 18 March 2015 at 14:51, Joram van Driel <joramvandriel at gmail.com>
> wrote:
>
> Hi Martina,
>
> In general, I'd advice to do some kind of trial-selection procedure when
> comparing error versus correct trials, in order to trial-count-match the
> two conditions. Otherwise you run into problems considering: SNR (higher
> for the correct condition), and RT (errors are usually faster, resulting in
> a time-on-task confound). What I always do is pick from the correct
> condition a similar number of trials that are close to the RT distribution
> of the error trials (i.e. the faster correct trials). That way you solve
> both problems at once (and probably the cluster-based permutation test in
> field trip will work as well, as a bonus ;)).
>
> Best,
> Joram
>
> On Wed, Mar 18, 2015 at 2:31 PM, Martina Rossi <martina.rossi76 at yahoo.it>
> wrote:
>
> Dear All,
>
> I would like to get some feedback from the community about a statistical
> analysis problem I need to tackle with my study.
> I want to apply the cluster-based permutation tests on time-frequency data
> considering two conditions (correct vs error).
> Unfortunately, these two conditions have different sizes (correct >>
> error).
> Right now, I am only considering subjects having a ratio "error/correct"
> bigger than 1/5, yet this is only an arbitrary threshold I set.
> The question is the following:
> is there a formal way to identify a threshold by which two conditions can
> be realiably compared with the cluster-based permutation tests?
> If the cluster-based approach is not suitable in this scenario, is there
> any other approach you would suggest?
> I shall perhaps point out that I am working on EEG data recorded with a 32
> channel system (impedance levels < 10 kΩ).
>
> Looking forward to hear your feedback :)
>
> Kind Regards,
> Martina Rossi
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
>
>
>
> --
> Joram van Driel
> Postdoc @ Vrije Universiteit Amsterdam
> Cognitive Psychology
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
>
>
>
>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20150322/c85b93bf/attachment-0001.html>