[FieldTrip] problems with speed of processing

Thu Jul 12 12:00:02 CEST 2012

Thank you all for this interesting advices. I am not using Fieldtrip in
this case to read the raw data (for this I am using a function from
BCI2000), just for signal processing.
The qsubcellfun function is new for me. I will implement it, this will
solve the problem of memory as Giovanni mentioned.

Thanks again,

Eliana

On Thu, Jul 12, 2012 at 8:17 AM, Gio Piantoni <g.piantoni at nin.knaw.nl>wrote:

> Hi,
>
> Please, note that "clear all" clears variables, but does not free up
> memory, especially for files read from disk. You can check the memory
> usage with the command "top" in Linux command line. You'll see that
> nothing changes even if you use "clear all".
> I agree with Stephen that you need to write smaller functions for each
> subjects. Something like:
>
> for i = 1:numel(subj)
>   do_analysis(subj)
> end
> Where "do_analysis.m" is a function you wrote to do all the analysis.
> In this way, Matlab takes care of memory management by itself (it
> frees the memory once the function has been executed).
>
> A more elaborate solution is to use the "qsub" functions included in
> fieldtrip. Try this:
> addpath /path/to/fieldtrip/qsub
> fname = 'do_analysis';
> subj    = {1, 2, 3, 4, 5};
> qsubcellfun(fname, x1, 'memreq', 1024^3, 'timreq', 60*60, 'backend',
> 'local');
>
> This will start a new Matlab session (and use a new Matlab license)
> and it will close it when the function has been executed. There should
> be no memory leaks at all. See "help qsubcellfun" for more info.
> Remember to use 'backend' 'local' unless you have access to Oracle
> Grid Engine or Torque.
>
> Hope this helps.
> Gio
>
> --
> Giovanni Piantoni, MSc
> Dept. Sleep & Cognition
> Netherlands Institute for Neuroscience
> Meibergdreef 47
> 1105 BA Amsterdam (NL)
>
> +31 20 5665492
> gio at gpiantoni.com
> www.gpiantoni.com
>
> On Thu, Jul 12, 2012 at 1:33 AM, Bill Budd <Bill.Budd at newcastle.edu.au>
> wrote:
> > Hi Eliana
> >
> >
> >
> > I see something similar when using SPM8 to process data. I’ve heard that
> > this might be because SPM8 uses fieldtrip functions to import my
> continuous
> > EEG files (Biosemi/bdf) and these functions may not release memory in
> matlab
> > after closing. When looping my preprocessing  functions over multiple
> > subjects I have the exactly same problem in SPM that you see in
> fieldtrip.
> > This may not be causing your problem (or mine) but wondered if the
> > processing your refer to also involves importing raw MEG data?
> >
> >
> >
> > The PCs I’m using have plenty of RAM/disk space (Win7/matlab 2011b).
> >
> >
> >
> > Be great to find a solution to this as it slows done processing multi
> > subject data significantly!
> >
> >
> >
> > Cheers
> >
> >
> >
> > -Bill
> >
> >
> >
> >
> >
> > From: fieldtrip-bounces at science.ru.nl
> > [mailto:fieldtrip-bounces at science.ru.nl] On Behalf Of Eliana Garcia
> > Sent: Wednesday, 11 July 2012 2:40 AM
> > To: FieldTrip discussion list
> > Subject: Re: [FieldTrip] problems with speed of processing
> >
> >
> >
> > Dear Stephen,
> >
> >
> >
> > Thanks for the suggestion. I have already one function that is doing all
> the
> > preprocessing and saving as well. But this seams not to solve the issue.
> >
> >
> >
> > Thanks,
> >
> > Eliana
> >
> >
> >
> > On Tue, Jul 10, 2012 at 6:01 PM, Stephen Whitmarsh
> > <stephen.whitmarsh at gmail.com> wrote:
> >
> > Dear Eliana,
> >
> >
> >
> > You could try creating a function that does all this preprocessing and
> then
> > writes the output to file.
> >
> > If you then call this function in a (subject) loop you are sure that
> Matlab
> > will free up the memory it used for that function.
> >
> >
> >
> > Perhaps it helps,
> >
> >
> >
> > Stephen
> >
> >
> >
> > On 10 July 2012 16:57, Eliana Garcia <elilife at gmail.com> wrote:
> >
> > Hello Dear Fieldtrip Community,
> >
> >
> >
> > I am having now problems with processing my data. I am analyzing a quite
> big
> > data set with 400 trials that are composed by 10 sec (re-sampled at
> 150Hz).
> > I am doing first a demean, then planar gradient transformation,
> > time-frequency analysis using multitapers and then combine the planar
> > gradient again. For the first subject always runs normally, but for the
> > second one matlab gets really slow. I am taking care in deleting the
> > variables that I create with big amount of data after I don't need them
> any
> > more, so I am using just one big file that is being safe at the end
> (average
> > time-frequency across trials) for each of the subjects. Every time matlab
> > starts to analyze a new subject I am using clear all (with exceptions
> like
> > directory and name of subject) but stills with every new subject it gets
> > slower and slower.
> >
> >
> >
> > Do you have any advises to make the analysis faster? Sometimes is too
> slow
> > and the computer normally crashes after 3 subjects.
> >
> >
> >
> > Thank you very much for the attention.
> >
> >
> >
> > Best,
> >
> >
> >
> > --
> >
> > Eliana García Cossio
> > Applied Neurotechnology Lab.
> >
> > Graduate School of Neural and Behavioural Science - Max Planck Research
> > School
> > Institute of Medical Psychology and Behavioural Neurobiology
> > Universität Tübingen
> >
> > (+0049) 01 577-8587604
> > Otfried-Müller-Str. 47, 72076
> > Tübingen- Germany
> >
> >
> >
> >
> >
> > _______________________________________________
> > fieldtrip mailing list
> > fieldtrip at donders.ru.nl
> > http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> >
> >
> >
> >
> > _______________________________________________
> > fieldtrip mailing list
> > fieldtrip at donders.ru.nl
> > http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> >
> >
> >
> >
> >
> > --
> >
> > Eliana García Cossio
> > Applied Neurotechnology Lab.
> >
> > Graduate School of Neural and Behavioural Science - Max Planck Research
> > School
> > Institute of Medical Psychology and Behavioural Neurobiology
> > Universität Tübingen
> >
> > (+0049) 01 577-8587604
> > Otfried-Müller-Str. 47, 72076
> > Tübingen- Germany
> >
> >
> >
> >
> > _______________________________________________
> > fieldtrip mailing list
> > fieldtrip at donders.ru.nl
> > http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>

-- 
Eliana García Cossio
Applied Neurotechnology Lab.
Graduate School of Neural and Behavioural Science - Max Planck Research
School
Institute of Medical Psychology and Behavioural Neurobiology
Universität Tübingen
(+0049) 01 577-8587604
Otfried-Müller-Str. 47, 72076
Tübingen- Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20120712/336be378/attachment-0002.html>