Questions on High pass filter and Padding in Preprocessing

[gao zai]

Dear Robert,

Thanks very much for your great answer. As you mentioned, I tried to read
the continuous data into memory, it was slow. I will try to read them in
group of channels.

BTW, what do you mean by "the padding option in ft_preprocessing pads the
data _to_ the specified length, not _with_ the indicated length. " I checked
the script about ft-preprocessing, it seems to me that if the padding length
is shorter than the specified length (in my case, 2.5s),  as you point it
out, it will not work. If the padding is longer than that, then fieldtrip
will read a longer epoch than the specified length, so if I use a
cfg.padding=10, it will input a 10s epoch (and the middle part of 2.5s is
the trial I am focused) into memory, am I right? If I am right, can I just
use the padding here (e.g., cfg.padding=10) instead of filtering the
continuous data, since it is equal to filtering a longer trial and then drop
the padding part after completion?

Millions of thanks.

Best,
Feng



On Fri, Mar 19, 2010 at 12:21 PM, Robert Oostenveld <
r.oostenveld at fcdonders.ru.nl> wrote:

> Dear Feng,
>
>
> On 12 Mar 2010, at 11:07, gao zai wrote:
>
>> I am now working on the MEG data in 4D format. Since it was collected in
>> DC, so I want to first use a bandpass filter [0.1 80] to filter my data.
>> However, I found that the latter part of averaged results drifted away
>> relative to the  results when no filter or only a low-pass filter is used
>> (see the attached Fig.1). I thought it may be related to the end effect, so
>> I used cfg.padding=1, it didn't work.
>>
>
> the padding option in ft_preprocessing pads the data _to_ the specified
> length, not _with_ the indicated length. In your case according to the
> figure the trial was ~2.5 seconds long, i.e. already longer than 1 second,
> so cfg.padding=1 did not have an efefct (whereas cfg.padding=10 does).
>
>
>  Latter I turned to use cfg.padding=10 (1/.1=10) it works and no obvious
>> drift was found.
>>
>> However, when tried to only apply a high pass filter to the raw data,
>> regardless of using cfg.padding=10, the results always showed a square wave
>> (see Fig.2). And there is a warning:"Matrix is close to singular or badly
>> scaled. Results may be inaccurate. RCOND = 6.669292e-17."I don't understand
>> why this happened. Does anybody know what is the problem with it? How can I
>> get a right result in this case? Thank you very much.
>>
>
> You specified cfg.hpfreq=0.1, i.e. that frequencies below 0.1 Hz should be
> removed. These frequencies correspond to dripfts in the data of about 10
> seconds. The data segment you are working with is only 2.5 seconds, so
> removing the 10 second drift fails. You can increase the cfg.hpfreq to
> around 1/2.5 = 0.4 Hz, which would be the lowest frequency that can be
> estimated from the 2.5 second data segment.
>
> Alternatively, you could use the following pipeline
>
>
> cfg=[];
> cfg.dataset ='c,rfDC';
> cfg.continuous = 'yes';
> cfg.hpfilter = 'yes';
> cfg.hpfreq = 0.1;
> datacnt = ft_preprocessing(cfg);
>
> to read it into memory as one continuous segment, and apply the filter on
> the complete data.
>
> Then you can do
>
>
> cfg=[];
> cfg.dataset ='c,rfDC';
> cfg.trialdef.eventtype  = 'TRIGGER';
> cfg.trialdef.eventvalue =42;
> cfg.trialdef.prestim    = 1;
> cfg.trialdef.poststim   = 1.5;
> cfg.trialdef.offset     = 1;
> cfg.trialfun            = 'mytrialfun';
> cfg = definetrial(cfg);
>
> followed by
>
> datatrl = ft_redefinetrial(cfg, datacnt);
>
> Reading all data (also the inter-trial interval, the breaks, and the
> instruction periods) into memory requires a lot of RAM. If it does not fit
> all at once, you could do the above section for small groups of channels
>
> cfg.channel = 1:10;
> % cfg.channel = 11:20;
> % cfg.channel = 21:30;
> % ...
>
> and then call ft_appenddata([], datatrl_changroup1, datatrl_changroup2,
> ...)
>
> best,
> Robert
>
>
>
>
>
>
>
>
>
>
> ----------------------------------
> The aim of this list is to facilitate the discussion between users of the
> FieldTrip  toolbox, to share experiences and to discuss new ideas for MEG
> and EEG analysis. See also
> http://listserv.surfnet.nl/archives/fieldtrip.html and
> http://www.ru.nl/neuroimaging/fieldtrip.
>

----------------------------------
The aim of this list is to facilitate the discussion between users of the FieldTrip  toolbox, to share experiences and to discuss new ideas for MEG and EEG analysis. See also http://listserv.surfnet.nl/archives/fieldtrip.html and http://www.ru.nl/neuroimaging/fieldtrip.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20100319/d43c4647/attachment-0002.html>