[FieldTrip] Cluster-based permutation tests on time-frequency and size of conditions

Thu Mar 26 08:48:38 CET 2015

Hi David,
Thank so much for the reference and all the useful details, indeed very helpful :)

Best,Martina

     Il Lunedì 23 Marzo 2015 5:03, David Groppe <david.m.groppe at gmail.com> ha scritto:

 Hi Martina,
   Balanced sample sizes are typically recommended for conventional parametric independent samples tests (e.g., t-tests, ANOVAs) because it makes the tests less sensitive to differences in variation between the populations being compared. If the populations being compared differ in variance, having more observations from the population with less variability will make these tests overly permissive (i.e., the true false positive rate will be greater than your nominal alpha level). If you have more observations from the population with greater variability, the tests become overly conservative.
   A few years ago, my colleagues and I simulated some EEG data and found that permutation tests exhibit a qualitatively similar sensitivity to differences in variance between populations (see below). If you're concerned about such a difference in your data you could do as has already been suggested and use a subset of data so that the number of observations between samples is the same. Alternatively you could use a permutation test based on variants of the t-statistic that are less sensitive to differences in variance. In our paper below, we investigated two variants, Welch's t and t_dif. Welch's t proved a bit less sensitive to differences in variance and was only slightly less powerful than the conventional t-statistic. t_dif was markedly insensitive to differences in variance but was significantly less powerful. However, I would guess that using t_dif or Welch's t are likely more powerful than discarding trials (though we didn't investigate that option in the paper).
      cheers,
         -David

Groppe, D.M., Urbach, T.P., & Kutas, M. (2011) Mass univariate analysis of event-related brain potentials/fields II: Simulation studies. Psychophysiology, 48(12) pp. 1726-1737, DOI: 10.1111/j.1469-8986.2011.01272.x.
www.cogsci.ucsd.edu/~dgroppe/PUBLICATIONS/mass_uni_preprint2.pdf

On Thu, Mar 19, 2015 at 4:06 AM, Martina Rossi <martina.rossi76 at yahoo.it> wrote:

Dear Stephen and Joram,
Thank so much for your feedback,I will check out the suggested material,
Best,Martina 

     Il Mercoledì 18 Marzo 2015 20:43, Stephen Whitmarsh <stephen.whitmarsh at gmail.com> ha scritto:

 You can also check out this video of Robert. Apologies for the quality - not of the talk, but of the recording :-)

At 14:45 he actually mentions unequal number of trials between conditions.

https://www.youtube.com/watch?v=vOSfabsDUNg

Cheers,
Stephen

On 18 March 2015 at 19:33, Stephen Whitmarsh <stephen.whitmarsh at gmail.com> wrote:

I should add that (1) is typically done within subjects, and (2) over subjects.

Cheers,
Stephen  

On 18 March 2015 at 19:20, Stephen Whitmarsh <stephen.whitmarsh at gmail.com> wrote:

Dear Martina,

It might help to distinguish two aspects of cluster-based statistic.

1) the statistical approuch that you will use to determine whether a time-channel-datapoint / time-frequency-channel-datapoint / time-frequency-voxel-datapoint is considered significant different between conditions.
2) the statistical approuch that you will use to determine whether a cluster of time-channel-datapoints / time-frequency-channel-datapoints / time-frequency-voxel-datapoints is considered significantly different between conditions.
When you talk about cluster statistics, you probably think about the second part. But this might not be what you should initially be concerned with when thinking about e.g. different numbers of trials between conditions. Rather, consider what statistical tests you (can) use to compare your time-frequency values between conditions (within subjects). This can be, e.g., a t-test, a nonparametric (e.g. montecarlo) test, or any test, for that matter. As far as my limited knowledge of statistics goes, in most simple and non-extreme cases, unequal number of trials that does not have to increase your chance of type I errors, rather that of type 2 (you'll be insensitive to differences if you don't have enough observations in one condition due to noisy estimate of means/distribution). But in any case it's a simple question to google or ask a statistician.

Now, after you are happy with and confident about the between conditions statistical test, consider how the cluster statistics might help you. 
First of all, how does it determine whether a cluster is significantly different between conditions? There are different options, but the gist is that it takes your significant statistical numbers of step (1), adds them up when they belong to the same cluster (based on whether they are neigbourings in time/freq/space with other significant numbers), takes the maximum of these summed up clusters (there might be more than one cluster), and then compares this one value to the same taken from a(non-parametric) monte-carlo distribution of the null hypothesis based on permuting the values over conditions (and then calculating the maximum sum). The Maris and Oostenveld paper explains it in more detail. 

The reason for doing cluster-statistics is that its a smart way of dealing with multiple comparisons over many time x frequency x channels (or space). The method is blind for your decisions about how its computed for each point in time x frequency x channels (or space). 

I find the FieldTrip statistics functions, their configurations etc., and the way they interact confusing at times, but I hope this helps to clear it up a bit. 

Long story short - I think your question does not limit itself to cluster statistics and at the same time is much simpler. It's all about (1).  

Best wishes,
Stephen

There are two separate steps cluster statistics (as implemented in FieldTrip, but in general as well).

On 18 March 2015 at 14:51, Joram van Driel <joramvandriel at gmail.com> wrote:

Hi Martina,
In general, I'd advice to do some kind of trial-selection procedure when comparing error versus correct trials, in order to trial-count-match the two conditions. Otherwise you run into problems considering: SNR (higher for the correct condition), and RT (errors are usually faster, resulting in a time-on-task confound). What I always do is pick from the correct condition a similar number of trials that are close to the RT distribution of the error trials (i.e. the faster correct trials). That way you solve both problems at once (and probably the cluster-based permutation test in field trip will work as well, as a bonus ;)).
Best,Joram
On Wed, Mar 18, 2015 at 2:31 PM, Martina Rossi <martina.rossi76 at yahoo.it> wrote:

Dear All,
I would like to get some feedback from the community about a statistical analysis problem I need to tackle with my study.I want to apply the cluster-based permutation tests on time-frequency data considering two conditions (correct vs error).Unfortunately, these two conditions have different sizes (correct >> error).Right now, I am only considering subjects having a ratio "error/correct" bigger than 1/5, yet this is only an arbitrary threshold I set.The question is the following:is there a formal way to identify a threshold by which two conditions can be realiably compared with the cluster-based permutation tests?If the cluster-based approach is not suitable in this scenario, is there any other approach you would suggest?I shall perhaps point out that I am working on EEG data recorded with a 32 channel system (impedance levels < 10 kΩ).
Looking forward to hear your feedback :)
Kind Regards,Martina Rossi

_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-- 
Joram van DrielPostdoc @ Vrije Universiteit AmsterdamCognitive Psychology

_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20150326/fa65abae/attachment-0002.html>