<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Verdana","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Dear Artemy,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><br>Importantly, these constants also enter in the permutation distribution that is used to evaluated the significance of the maximum cluster-mass statistic, to the effect that the Bullmore-style and the Fieldtrip-style permutation distributions are shifted versions of each other. As a result, the p-values that roll out of the two approaches are identical.</span><o:p></o:p></p></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>If I understand correctly, having the same resulting p-values could only be if the two methods assign the same rank-ordering to a given a set of clusters. But I don't think that is the case. Let's imagine that the t-statistic cutoff 'c' is equal to 1, and the data contains two suprathreshold clusters (let's say this is a spatial test and the clusters are composed of electrodes):<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- The first cluster has 10 electrodes, each one with a t-statistic equal to 1.1<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- The second cluster has 2 electrodes, both with a t-statistic equal to 3<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>As I understand, Bullmore's method would assign cluster 1 a mass of 10*(1.1-1) = 1 and cluster 2 a mass of 2*(3-1)=4 , while your method would assign cluster 1 a mass of 10*1.1 = 11 and cluster 2 a mass of 2*3 = 6. Hence, given a null distribution, it should be possible to choose a cluster-based threshold that indicates as significant only cluster 1 under Bullmore's method, and only cluster 2 under yours.<o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I think your reasoning is correct: when the data contain more than one suprathreshold cluster, my argument does not apply anymore. Your example shows that the Bullmore- and Fieldtrip-style cluster statistics have different sensitivities. Thank you for pointing this out. For every test statistic, the decisions based on the permutation p-value controls the type-I error rate, but the type-II error rate (the complement of sensitivity) depends on the exact test statistic.</span><o:p></o:p></p></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks for confirming that up. I should note, though, that this is an issue even in data without multiple suprathreshold clusters. The same logic as above -- which shows that the two measures gives different ranks to same set of clusters -- also applies to the distribution of clusters under the null hypothesis. Thus one can imagine a single cluster in the data that would be judged significant under Fieldtrip's method and not significant under Bullmore, or vice-versa. I believe that generally, in comparison to Bullmore's method, Fieldtrip's method would tend to favor judging-as-significant large clusters (with many electrodes). <o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Sure, it also applies to the distribution of clusters under the null hypothesis and, therefore, can result in different p-values for the Fieldtrip and the Bullmore style cluster statistics, even when there is only single cluster in the data. I should have made this more explicit in my reply, especially since the distribution of clusters under the null hypothesis is random (and therefore a focus on the single-cluster-in-the-data situation is not a very useful one).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div><div><p class=MsoNormal>I personally think of the distinction not so much in terms of controlling sensitivity, but rather as concerning the definition of what counts as a cluster of interest. Though both methods look for spatiotemoprally contiguous regions of electrodes that exceed threshold, for Bullmore the cluster is the sum of suprathreshold statistic values , while for Fieldtrip it's the sum of the entire statistic values in the region. I'm quite interested in the question of which gives more justifiable/better results in real-world settings, though unfortunately I have not seen any work done on the matter. From what I have seen in my brief forays into the extensive analytic + numerical studies of cluster-based significance testing in the fMRI literature, in that field they always refer to Bullmore-style clusters.<o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I do think that sensitivity is a crucial concept here. Neurobiological data are almost always high-dimensional (data arrays with dimensions space, time and/or frequency), but the statistical test only answers the question whether there is some direction in this high-dimensional space along which the experimental conditions differ. It is crucial that our test statistics are chosen such that they have a high sensitivity in the directions that are neurobiologically plausible. In a recent paper in Psychophysiology (2012), I took some time to explain this issue.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Note that others in the field (e.g., Karl Friston) often put forward the claim that parametric statistical tests (e.g., the t-statistic) are always more sensitive than nonparametric statistical tests. This claim only holds for scalar observations (e.g., an electrical potential measured at one electrode and one post-stimulus time), which is not the type of data neurobiological studies are typically interested in.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Thanks for your interest in this issue. You ask the right questions. Also, my apologies for my earlier sloppy replies.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Eric<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Verdana","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks again & happy holidays...<o:p></o:p></p></div><div><p class=MsoNormal>-Artemy<o:p></o:p></p></div></div></div></div></body></html>