[FieldTrip] sample size calculations - nonparametric cluster permutation tests.

[Robert Oostenveld]

Hi Martin,

Let me CC this to the FieldTrip mailing list, since people there might want to chip in.

I am not aware of a way to calculate sample sizes for a non-parametric test in general, although for specific tests there are estimates (see e.g. http://www.stat.purdue.edu/~bacraig/notes582/noether.pdf <http://www.stat.purdue.edu/~bacraig/notes582/noether.pdf>). The challenge with the cluster-based permutation test is that the effect is not a-priori specified, and hence the effect size is not quantified. If you don’t know the effect size, you cannot estimate the likelihood of observing that effect (given the number of subjects), or number of subjects (given the desired likelihood of observing the effect). Actually, to be more precise: with the non-parametric permutation test you are not even making an inference about a specific effect, but you are making an inference on the data being exchangeable in general (see also this FAQ <http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/> on the FieldTrip website).  

If you do know the effect (as you seem to do), then a non-parametric test might actually not be the most optimal. Your follow up study would have a better power if you were not to ask the question "is there a difference in the distribution of the data in general” (i.e. H0: there is no difference, versus H1 there is a difference), but rather “is there a difference in the channel-time-frequency range (ie.. cluster) that I previously identified”. In the 2nd case it might be possible to average over the region of interest. Still, you may have the issue that the distribution of the statistic-of-interest is not according to a known distribution (or cannot be assumed to have normally distributed error terms), so a non-parametric test might still be in place. 

Pragmatically I think that you can indeed use a resampling strategy to estimate the required sample size from previously acquired data, asssuming that your previous study is sufficiently overpowered (and of course assuming that your previous study is not a false alarm, but a true rejection of the H0). Another approach (although not 100% addressing your question) is to specify in the new study design to specify that a certain number of subjects are going to be set aside to optimize the analysis pipeline and determine the effect of interest, after which that optimized pipeline will be used to evaluate the effect in all other subjects. However, that does not allow a-priori specification of the number of subjects; it just replaces the effect size estimate from the previous study with an effect size estimate from the pilot group.

Many studies actually benefit from having specific hypotheses amd being able to specifying the effect of interest prior to testing that effect. E.g. that is also what is done here <http://www.fieldtriptoolbox.org/tutorial/beamformer/#preprocessing>. In general, initiating new neuroscience studies again-and-again as if we don’t know anything about the data is simply not efficient. So please do use your existing data in the analysis plan of your new study.

best
Robert
 

> On 15 Feb 2019, at 01:22, Gallagher, Martin J <martin.j.gallagher at vumc.org> wrote:
> 
>  
> Hello Dr. Oostenveld,
>  
> Thank you for developing the nonparametric permutation tests in Fieldtrip and your reviews of the subjects.
>  
> For a grant, my study section is asking for sample size calculations.  Is there a way to estimate this?   I know it’s probably simple-minded, but could I use permutation tests I’ve run in the past and keep scaling back the number of trials until I get to a N value that still has a p value < 0.025?
>  
> Thank you  for considering!
>  
> _______________________________
> Martin J. Gallagher MD, PhD
> Associate Professor
> Dept. of Neurology, Epilepsy Division
> Vanderbilt University School of Medicine
> The information contained in this message and any attachments is intended solely for the individual or entity to which it is addressed and may contain confidential and/or privileged material.  If you have received this email in error, please contact the sender and delete the material from your system.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20190215/0fae75dd/attachment-0001.html>
-------------- next part --------------
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202