<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Sebastian,
<div class=""><br class="">
</div>
<div class="">I don’t understand why you think that the variance of ft_timelockgrandaverage leads to wrong statistics. As I said in an earlier e-mail, the traditional way to do second-level statistics is to compute the variance of the mean-per-subject, and
not to compute the variance of the individual-trials-concatenated-across-subjects.</div>
<div class=""><br class="">
</div>
<div class="">The computation that you call ‘real mean and variance’ assume the unit of observation to be the trials, while the unit-of-observation that ft_timelockgrandaverage uses for the mean and variance computation is the subject. While the mean is not
going to change (assuming the same number of trials per subject), the variance will turn out differently, because that results from a non-linear operation (squaring).</div>
<div class=""><br class="">
</div>
<div class="">In your numeric example, the 'FT-code' leads to a rejection of the null-hypothesis due to the fact that you use the incorrect n1 and n2 in the numerator of the t-statistic, these should be 2 and 2, rather than 100 and 100.</div>
<div class=""><br class="">
</div>
<div class="">Best wishes,</div>
<div class="">Jan-Mathijs</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 1 Oct 2022, at 08:48, <a href="mailto:Sebastian.Neudek@med.uni-duesseldorf.de" class="">
Sebastian.Neudek@med.uni-duesseldorf.de</a> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;" class="">
<p style="margin-top: 0px; margin-bottom: 0px;" class=""></p>
<div class="">Hi Jan-Mathijs,<br class="">
<br class="">
thank you for your suggestion of mixed-effect models. Probably those are the perfect fit for my case.<br class="">
But I still think the variance implementation in fieldtrip in ft_timelockgrandaverage is not correct (only calculating the variance of means of subjects as group variance). I looked a little bit deeper into the implementations and it seems the degress of freedom
are the number of trials. Therefore the implemented mean in ft_timelockgrandaverage already is the mean from a pooled variance.<br class="">
<br class="">
Here you have an example matlab script of which I think the variance of ft_timelockgrandaverage leads to wrong statistics. There are only four subjects who belong to 2 groups (Subject 1 and 2 in group A, Subject 3 and 4 in group B). The trials from all subjects
are drawn from the same distribution. All have same mean and same variance. Therefore the real group means are the same and don't differ. Here a t-test is evaluated, to test whether the means of the groups differ.<span class="Apple-converted-space"> </span><br class="">
<br class="">
<br class="">
<br class="">
rng(0,'twister');<br class="">
<br class="">
% all people have the same number of trials and the same(!) mean<br class="">
N = 100;<br class="">
<br class="">
n1 = N; % Person 1 belongs to group A<br class="">
mu1 = 0;<br class="">
sigma1 = 1;<br class="">
<br class="">
n2 = N; % Person 2 belongs to group A<br class="">
mu2 = 0;<br class="">
sigma2 = 1;<br class="">
<br class="">
n3 = N; % Person 3 belongs to group B<br class="">
mu3 = 0;<br class="">
sigma3 = 1;<br class="">
<br class="">
n4 = N; % Person 4 belongs to group B<br class="">
mu4 = 0;<br class="">
sigma4 = 1;<br class="">
<br class="">
% normal distributed random numbers as trials for each person<br class="">
trials_1 = mu1 + randn([1 n1])*sigma1;<br class="">
trials_2 = mu2 + randn([1 n2])*sigma2;<br class="">
trials_3 = mu3 + randn([1 n3])*sigma3;<br class="">
trials_4= mu4 + randn([1 n4])*sigma4;<br class="">
<br class="">
% calculate mean and variance for each person<br class="">
<br class="">
avg1 = mean(trials_1);<br class="">
avg2 = mean(trials_2);<br class="">
avg3 = mean(trials_3);<br class="">
avg4 = mean(trials_4);<br class="">
<br class="">
var1 = var(trials_1);<br class="">
var2 = var(trials_2);<br class="">
var3 = var(trials_3);<br class="">
var4 = var(trials_4);<br class="">
<br class="">
% real mean and variance<br class="">
<br class="">
avg_A = mean([trials_1 trials_2]); % avg for group A<br class="">
avg_B = mean([trials_3 trials_4]); % avg for group B<br class="">
var_A = var([trials_1 trials_2]);<br class="">
var_B = var([trials_3 trials_4]);<br class="">
<br class="">
% mean and variance of fieldtrip<br class="">
<br class="">
avg_A_ft = mean([avg1 avg2]);<br class="">
avg_B_ft = mean([avg3 avg4]);<br class="">
var_A_ft = var([avg1 avg2]);<br class="">
var_B_ft = var([avg3 avg4]);<br class="">
<br class="">
% now a t-test, to check if the mean of group A: {person 1 and person 2}<br class="">
% and group B: {person 3 and person 4} are different<br class="">
% H0: both groups share the same mean<br class="">
% H1: both groups have different mean<br class="">
<br class="">
%<span class="Apple-converted-space"> </span><a href="https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Student*27s_t-test__;JQ!!HJOPV4FYYWzcc1jazlU!6Mx3xotEc6n0WD6ucT1uJXuGoGRb3nBniS8bvTkgu5bBo2C7RGs6aXqagsx4CtsBeIlIV_8l_5FZv7IoifABgNu85rFnFfmYfPKd5WAnGaP2sxaQRB88G4Q$" class="">https://en.wikipedia.org/wiki/Student%27s_t-test</a><br class="">
alpha = 0.05;<br class="">
<br class="">
t = avg_A-avg_B/(sqrt((var_A+var_B)/2)*sqrt(2/(n1+n2))); % this formula is only correct for same group sizes<br class="">
%p = 1-tcdf(t,n1+n2+n3+n4-4) % right tailed t-test<br class="">
%p = tcdf(t,n1+n2+n3+n4-4) % left tailed t-test<br class="">
p = 2*tcdf(-abs(t),n1+n2+n3+n4-4); % two tailed t-test<br class="">
<br class="">
fprintf('Result:\n');<br class="">
if p < alpha<span class="Apple-converted-space"> </span><br class="">
fprintf('H0 rejected. Significant difference of means between Group A and B (P = %.4f, T = %.4f)\n\n',p ,t);<br class="">
else<br class="">
fprintf('H0 not rejected. No significant difference of means between Group A and B (P = %.4f, T = %.4f)\n\n',p ,t);<br class="">
end<br class="">
<br class="">
<br class="">
% t value with fieldtrips variance<br class="">
<br class="">
t_ft = avg_A_ft-avg_B_ft/(sqrt((var_A_ft+var_B_ft)/2)*sqrt(2/(n1+n2))); % this formula is only correct for same group sizes<br class="">
% p_ft = 1-tcdf(t_ft,n1+n2+n3+n4-4) % right tailed t-test<br class="">
% p_ft = tcdf(t_ft,n1+n2+n3+n4-4) % left tailed t-test<br class="">
p_ft = 2*tcdf(-abs(t_ft),n1+n2+n3+n4-4); % two tailed t-test<br class="">
<br class="">
<br class="">
<br class="">
fprintf('Result using variance of ft_timelockgrandaverage:\n');<br class="">
if p_ft < alpha<span class="Apple-converted-space"> </span><br class="">
fprintf('H0 rejected. Significant difference of means between Group A and B (P = %.4f, T = %.4f)\n\n',p_ft ,t_ft);<br class="">
else<br class="">
fprintf('H0 not rejected. No significant difference of means between Group A and B (P = %.4f, T = %.4f)\n\n',p_ft ,t_ft);<br class="">
end<br class="">
</div>
<div class=""><br class="">
<br class="">
Best,<br class="">
Sebastian<br class="">
<br class="">
<br class="">
<br class="">
</div>
<br class="">
<p style="margin-top: 0px; margin-bottom: 0px;" class=""></p>
</div>
<hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 674.234375px;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">Von:</b><span class="Apple-converted-space"> </span>fieldtrip <<a href="mailto:fieldtrip-bounces@science.ru.nl" class="">fieldtrip-bounces@science.ru.nl</a>> im Auftrag von Schoffelen,
J.M. (Jan Mathijs) via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>><br class="">
<b class="">Gesendet:</b><span class="Apple-converted-space"> </span>Freitag, 30. September 2022 13:46:40<br class="">
<b class="">An:</b><span class="Apple-converted-space"> </span>FieldTrip discussion list<br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Schoffelen, J.M. (Jan Mathijs)<br class="">
<b class="">Betreff:</b><span class="Apple-converted-space"> </span>[Extern] - Re: [FieldTrip] Calculating the variance of a group with given variances of each subject</font>
<div class=""> </div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Hi Sebastian,
<div class=""><br class="">
</div>
<div class="">I would say that for visualization purposes (and for traditional parametric second-level statistics) you are interested in the variance across your units-of-observation. In this case across subjects. This is indeed what ft_timelockgrandaverage
gives you. If you are into more sophisticated estimates, it smells a bit as if you want to be moving into the direction of mixed-effect models, which are gaining quite a bit of momentum these days. Since I am a bit old-school I have not eaten any cheese from
that (<a href="https://urldefense.com/v3/__http://www.dwotd.nl/2008/02/360-daar-heb-ik-geen-kaas-van-gegeten.html__;!!HJOPV4FYYWzcc1jazlU!6Mx3xotEc6n0WD6ucT1uJXuGoGRb3nBniS8bvTkgu5bBo2C7RGs6aXqagsx4CtsBeIlIV_8l_5FZv7IoifABgNu85rFnFfmYfPKd5WAnGaP2sxaQG6s86NA$" class="">http://www.dwotd.nl/2008/02/360-daar-heb-ik-geen-kaas-van-gegeten.html</a>),
so perhaps somebody else wants to chime in here.</div>
<div class=""><br class="">
</div>
<div class="">Best wishes,</div>
<div class="">Jan-Mathijs</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 26 Sep 2022, at 21:36, Sebastian Neudek via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" class="">fieldtrip@science.ru.nl</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div id="divtagdefaultwrapper" dir="ltr" class="" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">
<p class="" style="margin-top: 0px; margin-bottom: 0px;"></p>
<div class="">Hi all,<br class="">
<br class="">
I am currently writing my master's thesis using FieldTrip and might need a bit of your expertise for a little hurdle I encountered.<br class="">
<br class="">
First a few words about what I want to do:<br class="">
I have data from a MEG-go/nogo study: There are multiple subjects and multiple trials for each condition. I now want to plot the ERF for each condition averaged across all subjects. But together with the ERF I also want to plot the standard error or variance
or confidence interval or standard deviation. Something like this(Figure 3a):<br class="">
<a href="https://urldefense.com/v3/__https://www.researchgate.net/figure/ERP-results-including-P3-amplitudes-a-Grand-average-and-standard-errors-of-ERP_fig2_327651209__;!!HJOPV4FYYWzcc1jazlU!4AKDCxWgDNOVGlWwimPV3Sc-5suVMj59sPhwC-eHwI7h0FQr10-YxE5fzliZsQft3R0I3s5Swpei12UfNPYgMdNHwshEvUe7t8p0Gw$" class="OWAAutoLink" id="LPlnk347301" previewremoved="true">https://www.researchgate.net/figure/ERP-results-including-P3-amplitudes-a-Grand-average-and-standard-errors-of-ERP_fig2_327651209</a><br class="">
Independent which of these measures I finally use, I need to calculate the variance of the ERF at each timepoint.<br class="">
To do this I first used ft_timelockanalysis to calculate the averages of each subject. ft_timelockanalysis returns not only the average, but also the variance of the ERF. Afterwards I use ft_timelockgrandaverage to calculate the group average of the ERF.<br class="">
<br class="">
And this is where my troubles begin. I get how ft_timelockgrandaverage calulates the new average: It averages over the mean values of the subjects (it also accounts for the degrees of freedom). But for the variance it just calculates the variance of the means.
I thought, it will also account for the variances of each subject.<br class="">
In the extreme case it leads to the following:<br class="">
Imagine you have two subjects and you measure very close averages avg(sub1)=1 and avg(sub2)=1.0001. But their variances are extremly high: var(sub1)=var(sub2)=100. Then the variance calculated by ft_timelockgrandaverage is almost zero, when in reality it should
be something like 100.<br class="">
Maybe this should be noted in ft_timelockgrandaverage, because it can lead to wrong statistics if the user isn't aware of this effect.<br class="">
<br class="">
Back to my task:</div>
<div class="">This is surely not the variance I want to use for my ERF plots and therefore I searched for a different calculation.<br class="">
I found on wikipedia a formula for so called 'pooled variance' (at the bottom of the page for sample-based statistics):<br class="">
<a href="https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Pooled_variance__;!!HJOPV4FYYWzcc1jazlU!4AKDCxWgDNOVGlWwimPV3Sc-5suVMj59sPhwC-eHwI7h0FQr10-YxE5fzliZsQft3R0I3s5Swpei12UfNPYgMdNHwshEvUeBLHOpZA$" class="OWAAutoLink" id="LPlnk563115" previewremoved="true">https://en.wikipedia.org/wiki/Pooled_variance</a><br class="">
<br class="">
But I am not 100% happy with this pooled variance.<br class="">
First of all the deegrees of freedom is not accounted for. Then there is a different mean calculated than in ft_timelockgrandaverage: The mean is accounting for the number of trials for each subject. I don't know if the different mean is a pro or a con (On
the one hand, means with a higher precisision because of a higher number of trials are weighted higher than means with a lower trial number. On the other hand, each subject on the group should be weighted same).<br class="">
<br class="">
I also found a second formula on stackexchange, which also calculates a variances of variances:<br class="">
<a href="https://urldefense.com/v3/__https://stats.stackexchange.com/questions/300392/calculate-the-variance-from-variances__;!!HJOPV4FYYWzcc1jazlU!4AKDCxWgDNOVGlWwimPV3Sc-5suVMj59sPhwC-eHwI7h0FQr10-YxE5fzliZsQft3R0I3s5Swpei12UfNPYgMdNHwshEvUdcUgTl6A$" class="">https://stats.stackexchange.com/questions/300392/calculate-the-variance-from-variances</a><br class="">
<br class="">
Therefore I am uncertain which formula to use, because as it seems, they differ.<br class="">
<br class="">
<br class="">
I hope somebody of you is a little more experienced in statistics than I am and can help me out finding the correct or best calculation for this variance. Best case would be if it is already implemented in fieldtrip and I only missed it, but I can also implement
it myself.<span class="Apple-converted-space"> </span><br class="">
<br class="">
Best,<br class="">
Sebastian<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
<br class="">
<p class="" style="margin-top: 0px; margin-bottom: 0px;"></p>
</div>
<span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">fieldtrip
mailing list</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" class="" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<a href="https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!4AKDCxWgDNOVGlWwimPV3Sc-5suVMj59sPhwC-eHwI7h0FQr10-YxE5fzliZsQft3R0I3s5Swpei12UfNPYgMdNHwshEvUdau-41SA$" class="" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!4AKDCxWgDNOVGlWwimPV3Sc-5suVMj59sPhwC-eHwI7h0FQr10-YxE5fzliZsQft3R0I3s5Swpei12UfNPYgMdNHwshEvUdau-41SA$</a></div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>