# [FieldTrip] IMPORTANT: Bug in ft_freqstatistics

Harsimrat Singh singhharsimrat at gmail.com
Mon Mar 25 17:47:45 CET 2013

```Hi All

I think I am missing something with ft_selectdata. I investigated a bit
more while redoing the analysis with new version of fieldtrip.

I have 4 class data but i am just doing classification (multivariate
analysis) between the first two classes with 13 and 20 trials each.
After the ft_freqanalysis - all the trials are contained in tf - and
tf.powsctrm is 4D (73*1*3*4) - [trials, channel, freq, time] - Here, I have
one channel as I removed others.
When I use
tf1  = ft_selectdata(tf, 'rpt', data1.trlvec(1:end));
to get trials for only the first class
I get
label: {32x1 cell}
dimord: 'rpt_chan_freq_time'
freq: [10.5000 12.5000 14.5000]
time: [0.3000 0.4000 0.5000 0.6000]
powspctrm: [4-D double]
cumtapcnt: [13x3 double]
cfg: [1x1 struct]
Now, the size of tf1.powspctrm is (73, 1, 2, 3) which is same as
tf.powspctrm
Similarly, for trials of class 2
tf2  = ft_selectdata(tf, 'rpt', data2.trlvec(1:end));
Again, the size of tf2 is same as tf1 which is same as tf
The only difference in the structure is
tf2.cumtapcnt which is 20 for tf2, 13 for tf1 and 73 for tf.

The first question is-
How does ft_freqstatistics handle tf1.cumtapcnt, when I use
stat        = ft_freqstatistics(cfg,tf1,tf2)
Is tf1. cumtapcnt telling the ft_freqstatistics which trials to use of tf1?
This only works if I design the label as
cfg. design =
[ones(size(tf1.powspctrm,1),1);2*ones(size(tf2.powspctrm,1),1)]'
its size is 73*2 =146

But my total relevant trials are just 13 (class 1) + 20(class2) = 33 for
both classes
I suspect I making a mistake somewhere as in ideal case I should be
designing cgf.design = [ones(size(tf1.cumtapcnt,1),1);
2*ones(size(tf2.cumtapcnt,1),1)]'
size(cg.design) should be 33
and tf1.powspctrm should have size [13 1 3 4] and tf2 should have size [20
1 3 4]

So ft_freqstatistics should be handling 33 trials  - with 13 from first
class, 20 from other class.
But now it has full 146 trials even after doing ft_selectdata.
I will be very grateful for any help.

Best regards
Harsimrat

On 21 March 2013 08:34, Eelke Spaak <eelke.spaak at donders.ru.nl> wrote:

> Hi Ingrid,
>
> Thanks for your question; it made me realize that my e-mail was not
> clear enough. In your case, when you repaired channels that were still
> present in the data, the output channel order of ft_channelrepair will
> be the same as the input channel order.
>
> The problem only occurs if you use ft_channelrepair to repair
> *missing* channels, so e.g. channels switched off during acquisition
> or that were discarded during preprocessing.
>
> Groeten,
> Eelke
>
> On 20 March 2013 20:55, Ingrid Nieuwenhuis <inieuwenhuis at berkeley.edu>
> wrote:
> > Hi Eelke,
> >
> > Thanks for reporting. I did do ft_channelrepair on my EEG data for some
> > subjects and subsequently used ft_freqstatistics. However, the channel
> order
> > is indentical in all my subjects in the freq.
> >
> > I repaired bad channels using the default setting (method='nearest'). The
> > bad channels were included in the input data to ft_channelrepair (so they
> > were not missing). I just checked, and in data_out after
> ft_channelrepair,
> > the order of the labels of the data is identical to the data_in, thus the
> > repaired channels were NOT appended to the end of the data. So apparently
> > ft_channelrepair not always appends the repaired channels at the end. I
> > suppose this is good news for me, meaning that the bug did not affect me,
> > since the channels were ordered identically when calling
> ft_freqstatistics.
> > Do you agree, or am I missing something? And for others that used
> > ft_channelrepair, might be worth checking if the ordering is indeed
> changed
> > after calling it, because it appears not always to be the case.
> >
> > Thanks,
> > Ingrid
> >
> >
> > On 3/20/2013 2:01 AM, Eelke Spaak wrote:
> >>
> >> Dear FieldTrip community,
> >>
> >> Short version: if you did ft_freqstatistics across subjects with a
> >> different channel ordering across data sets (which is the case if you
> >> the following carefully.
> >>
> >> Yesterday, we discovered a possibly severe bug in a utility function
> >> (ft_selectdata_old) which was used by ft_freqstatistics. The bug
> >> entails that ft_selectdata_old assumes that the channels in all data
> >> input arguments are ordered identically, and does not take into
> >> account the order of the data.label field. This means that when
> >> several data input arguments with different channel orderings are
> >> passed to ft_selectdata_old, the resulting output structure (with a
> >> newly added 'rpt' dimension) will contain wrong data (i.e. data for
> >> channel X might end up in another channel Y). Importantly, the same
> >> thus holds for ft_freqstatistics: when doing statistics across
> >> subjects, if the different subjects' data sets have different channel
> >> orderings, the resulting statistics will be wrong.
> >>
> >> We expect that in many cases, the channel ordering across different
> >> data sets will be identical, and in those cases this bug will not have
> >> had any effect. However, one important case in which channel ordering
> >> *will* be different is when one does ft_channelrepair on a data set,
> >> since this function appends the repaired channels to the end of the
> >> data. So, if you have done ft_channelrepair on only a subset of your
> >> subjects' data, and/or used ft_channelrepair to repair different
> >> channels in different subjects, and subsequently passed the repaired
> >> data sets to ft_freqstatistics, there is a good chance the results you
> >> got were wrong.
> >>
> >> Most likely, this bug was introduced with revision 4101, in september
> >> 2011, so it has been in the code for quite a while. Note that
> >> ft_channelrepair only supports repairing missing channels since
> >> revision 5216 (february 2012), and that this was advised against until
> >> revision 6100 (june 2012), when the algorithm for doing so sensibly
> >> was implemented.
> >>
> >> The fix has been implemented and is now available from the FTP server
> >> as of fieldtrip-20130319 (Donders users: the fix is automatically
> >> available on the /home/common/ version of FieldTrip). Note that
> >> ft_selectdata_old still does not work correctly, this is something we
> >> will fix in the near future (ft_selectdata_old is deprecated). We
> >> fixed the behaviour of ft_appendfreq (the preferred replacement
> >> function for ft_selectdata_old when appending freq structures) and
> >> changed ft_freqstatistics to use this function instead.
> >>
> >> The other statistics functions, ft_timelockstatistics and
> >> ft_sourcestatistics, were not affected by this bug. We strongly advise
> >> described above applies to you, you will need to redo your
> >> channel-level (time-)frequency statistics analyses.
> >>
> >> Details on the bug (and a test script) can be found here:
> >> http://bugzilla.fcdonders.nl/show_bug.cgi?id=2069 If you find any
> >> problems related to the bug, please post them there.
> >>
> >> We sincerely apologise for this bug and any trouble it might have
> caused.
> >>
> >> Best,
> >> On behalf of the FT core development team,
> >>
> >> Eelke
> >> _______________________________________________
> >> fieldtrip mailing list
> >> fieldtrip at donders.ru.nl
> >> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> >
> >
> > --
> > Ingrid Nieuwenhuis PhD
> > Postdoctoral Fellow
> > Sleep and Neuroimaging Laboratory
> > Department of Psychology
> > University of California, Berkeley
> > California 94720-1650
> > Tolman Hall, room 5305
> >
> >
> > _______________________________________________
> > fieldtrip mailing list
> > fieldtrip at donders.ru.nl
> > http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20130325/6c2b283b/attachment-0002.html>
```