<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div>Dear discussion list readers & contributors (especially Christine Blume),</div>
<div><br class="">
</div>
<div>There have been many questions (not only on the FT discussion list) about the calculation of effect size measures in the context of cluster-based permutation tests. I will continue my reply under the quotes below.</div>
<div><br class="">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">
<span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif; color: rgb(127, 127, 127);" class=""><b class="">From:<span class="Apple-converted-space"> </span></b></span><span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif;" class="">Blume
Christine <<a href="mailto:christine.blume@sbg.ac.at" style="color: purple; text-decoration: underline;" class="">christine.blume@sbg.ac.at</a>><br class="">
</span></div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">
<span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif; color: rgb(127, 127, 127);" class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b></span><span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif;" class=""><b class="">Re:
[FieldTrip] Effect size measure for cluster-based permutation tests</b><br class="">
</span></div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">
<span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif; color: rgb(127, 127, 127);" class=""><b class="">Date:<span class="Apple-converted-space"> </span></b></span><span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif;" class="">4
September 2017 at 14:28:47 GMT+2<br class="">
</span></div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">
<span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif; color: rgb(127, 127, 127);" class=""><b class="">To:<span class="Apple-converted-space"> </span></b></span><span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif;" class="">FieldTrip
discussion list <<a href="mailto:fieldtrip@science.ru.nl" style="color: purple; text-decoration: underline;" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">
<span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif; color: rgb(127, 127, 127);" class=""><b class="">Reply-To:<span class="Apple-converted-space"> </span></b></span><span style="font-family: -webkit-system-font, 'Helvetica Neue', Helvetica, sans-serif;" class="">FieldTrip
discussion list <<a href="mailto:fieldtrip@science.ru.nl" style="color: purple; text-decoration: underline;" class="">fieldtrip@science.ru.nl</a>><br class="">
</span></div>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Hi Alik,<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Thanks a lot for your suggestion, which I hoped would prompt more answers. Does anyone have suggestions on how exactly to implement the calculation
of an effect size measure?<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Best,<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Christine<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span lang="DE" style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">Von:</span></b><span lang="DE" style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span><a href="mailto:fieldtrip-bounces@science.ru.nl" style="color: purple; text-decoration: underline;" class="">fieldtrip-bounces@science.ru.nl</a><span class="Apple-converted-space"> </span>[<a href="mailto:fieldtrip-bounces@science.ru.nl" style="color: purple; text-decoration: underline;" class="">mailto:fieldtrip-bounces@science.ru.nl</a>]<span class="Apple-converted-space"> </span><b class="">Im
Auftrag von<span class="Apple-converted-space"> </span></b>Alik Widge<br class="">
<b class="">Gesendet:</b><span class="Apple-converted-space"> </span>Mittwoch, 23. August 2017 16:42<br class="">
<b class="">An:</b><span class="Apple-converted-space"> </span>FieldTrip discussion list<br class="">
<b class="">Betreff:</b><span class="Apple-converted-space"> </span>Re: [FieldTrip] Effect size measure for cluster-based permutation tests<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
My naive answer, which perhaps will provoke Eric to provide a better one: you have the actual cluster statistic and its permutation distribution under the null hypothesis. It seems as though that distribution could be assumed Gaussian and something like Cohen's
d calculated. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Aug 23, 2017 9:35 AM, "Blume Christine" <<a href="mailto:christine.blume@sbg.ac.at" style="color: purple; text-decoration: underline;" class="">christine.blume@sbg.ac.at</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class="">Dear all,</span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class="">I came across a question posted by someone about a year ago, which concerned effect size measures for cluster-based permutation tests. Unfortunately, the question does not seem to have been answered…</span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class="">Q: I am using cluster-based permutation tests (depsamplesT, on time-frequency data) and am wondering how to best calculate an effect size from that.</span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class="">Best,</span><o:p class=""></o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-GB" class="">Christine</span><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class=""><br class="">
</div>
<div class="">Giving a useful answer to this question requires that one knows for what this effect size measure will be used. Typically, a standardised effect size measure is required to perform a power calculation. A power calculation is possible for a number
of parametric statistical tests such as the T- and the F-test. As input for this power calculation, Cohen’s d is required. A sensible value for Cohen’s d can sometimes be found in published studies (preferably with large sample sizes). </div>
<div class=""><br class="">
</div>
<div class="">Cohen’s d can easily be obtained from the outcome of a cluster-based permutation test: </div>
<div class="">
<ol class="MailOutline">
<li class="">Calculate the non-standardised effect sizes by averaging the (sensor, frequency, time)-specific effects within the cluster of interest. Typically, the (sensor, frequency, time)-specific effects are raw differences between the subject averages for
the experimental conditions that are being compared.</li><li class="">Calculate the standard deviation over the subjects of these non-standardised effect sizes.</li><li class="">Calculate Cohen’s d by dividing the grand average of the non-standardised effect sizes by the standard deviation obtained in 2.</li></ol>
<div class=""><br class="">
</div>
</div>
<div class="">Unfortunately, Cohen’s d calculated in this way, will be biased, and therefore cannot be used for a power calculation. This type of bias is sometimes denoted as “double dipping”. </div>
<div class=""><br class="">
</div>
<div class="">In general, it is extremely challenging to perform a power calculations for statistical analyses that involve high-dimensional data. This does not only hold for electrophysiological, but also for fMRI data. To get idea about the difficulties that
one encounters, have a look at this paper from the fMRI community: <a href="http://www.biorxiv.org/content/early/2016/04/20/049429" class="">
http://www.biorxiv.org/content/early/2016/04/20/049429</a>. For the analysis of high-dimensional electrophysiological data, quite some statistical work still has to be done.</div>
<div class=""><br class="">
</div>
<div class="">best,</div>
<div class="">Eric Maris</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</body>
</html>