<div dir="ltr">Dear all, <br><br>I have a theoretical question about shuffling strategies in permutation tests using linear mixed effects models.<br><br>I have created a minimal working example, so that you can refer to specific points with code if necessary here: <a href="https://ozika.github.io/perm_test_lme/">https://ozika.github.io/perm_test_lme/</a> <br><br><h3 class="gmail-S2" style="margin:15px 10px 5px 4px;padding:0px;line-height:18px;min-height:0px">
<font color="#3c3c3c" face="Helvetica, Arial, sans-serif"><span style="font-size:17px;white-space:pre-wrap">Scenario 1:</span></font></h3><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px">(I just made these examples up, i tried to keep the dimensions unique in size for identifiability)</div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Dependent variable (DV)</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">pupil size (measured every second after onset, for 5 seconds)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Fixed effects</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">background color (<span style="font-weight:bold">within subject, </span>2 levels)</li><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">congruency (<span style="font-weight:bold">within subject, </span>2 levels)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Random effects</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">participant (10 participants/levels)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px">Each participant completes 40 trials, 10 one of each kind in the 2x2 design. The overall DV dataset is therefore 400x5 matrix.</div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><br></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Analysis (code [here](</span><a href="https://github.com/ozika/perm_test_lme">https://github.com/ozika/perm_test_lme</a><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">)</span><span style="font-weight:bold">  to be specific):</span><br></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">1) Run LMM for each data point using: pupil ~ bgcolor + congruency + bgcolor*congruency + (1|sub) </span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">2) Collect t-values</span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">3) Identify clusters</span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">4) Generate null distribution by shuffling labels of bgcolor and congruency </span><span style="font-weight:bold;font-style:italic">within </span><span style="font-style:italic">each subject. </span><span style="font-weight:bold;font-style:italic">Include (?) </span><span style="font-style:italic">permutations where no cluster was above the threshold and assign them 0 cluster mass. </span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">5) Calculate desired threshold based on the null distribution</span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">6) Evaluate actual clusters for each effect. </span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold"></span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"></div><h3 class="gmail-S5" style="margin:3px 10px 5px 4px;padding:0px;line-height:18px;min-height:0px;white-space:pre-wrap;color:rgb(60,60,60);font-family:Helvetica,Arial,sans-serif;font-size:17px"><br></h3><h3 class="gmail-S5" style="margin:3px 10px 5px 4px;padding:0px;line-height:18px;min-height:0px;white-space:pre-wrap;color:rgb(60,60,60);font-family:Helvetica,Arial,sans-serif;font-size:17px">--- </h3><div><br></div><h3 class="gmail-S5" style="margin:3px 10px 5px 4px;padding:0px;line-height:18px;min-height:0px;white-space:pre-wrap;color:rgb(60,60,60);font-family:Helvetica,Arial,sans-serif;font-size:17px">Scenario 2:</h3><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Dependent variable (DV)</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">pupil size (measured every second after onset, for 5 seconds)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Fixed effects</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">background color (<span style="font-weight:bold">within subject, </span>2 levels)</li><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">drug/placebo (<span style="font-weight:bold">between subject, </span>2 levels)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-weight:bold">Random effects</span></div><ul class="gmail-S3" style="margin:10px 0px 20px;padding-left:0px;font-family:Helvetica,Arial,sans-serif;font-size:14px;color:rgb(0,0,0)"><li class="gmail-S4" style="margin-left:56px;line-height:21px;min-height:0px;text-align:left;white-space:pre-wrap">participant (10 participants/levels)</li></ul><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">This is the same scenario as above but one factor is now a between subject variable. Would the right randomization strategy be to shuffle bgcolor within subject and then shuffle labels for entire subjects? (e.g. if each sub has 100 sampels altogether, shuffle the entire blocks of 100)</span></div><div class="gmail-S1" style="margin:2px 10px 9px 4px;padding:0px;line-height:21px;min-height:0px;white-space:pre-wrap;color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:14px"><span style="font-style:italic">Is there a disadvantage in shuffling ALL variables across all samples/subjects? </span></div><br>Thank you for any comments that you might have!<br><br>Ondrej<br> </div>