[FieldTrip] problems with speed of processing

[Gio Piantoni]

Hi,

Please, note that "clear all" clears variables, but does not free up
memory, especially for files read from disk. You can check the memory
usage with the command "top" in Linux command line. You'll see that
nothing changes even if you use "clear all".
I agree with Stephen that you need to write smaller functions for each
subjects. Something like:

for i = 1:numel(subj)
  do_analysis(subj)
end
Where "do_analysis.m" is a function you wrote to do all the analysis.
In this way, Matlab takes care of memory management by itself (it
frees the memory once the function has been executed).

A more elaborate solution is to use the "qsub" functions included in
fieldtrip. Try this:
addpath /path/to/fieldtrip/qsub
fname = 'do_analysis';
subj    = {1, 2, 3, 4, 5};
qsubcellfun(fname, x1, 'memreq', 1024^3, 'timreq', 60*60, 'backend', 'local');

This will start a new Matlab session (and use a new Matlab license)
and it will close it when the function has been executed. There should
be no memory leaks at all. See "help qsubcellfun" for more info.
Remember to use 'backend' 'local' unless you have access to Oracle
Grid Engine or Torque.

Hope this helps.
Gio

-- 
Giovanni Piantoni, MSc
Dept. Sleep & Cognition
Netherlands Institute for Neuroscience
Meibergdreef 47
1105 BA Amsterdam (NL)

+31 20 5665492
gio at gpiantoni.com
www.gpiantoni.com

On Thu, Jul 12, 2012 at 1:33 AM, Bill Budd <Bill.Budd at newcastle.edu.au> wrote:
> Hi Eliana
>
>
>
> I see something similar when using SPM8 to process data. I’ve heard that
> this might be because SPM8 uses fieldtrip functions to import my continuous
> EEG files (Biosemi/bdf) and these functions may not release memory in matlab
> after closing. When looping my preprocessing  functions over multiple
> subjects I have the exactly same problem in SPM that you see in fieldtrip.
> This may not be causing your problem (or mine) but wondered if the
> processing your refer to also involves importing raw MEG data?
>
>
>
> The PCs I’m using have plenty of RAM/disk space (Win7/matlab 2011b).
>
>
>
> Be great to find a solution to this as it slows done processing multi
> subject data significantly!
>
>
>
> Cheers
>
>
>
> -Bill
>
>
>
>
>
> From: fieldtrip-bounces at science.ru.nl
> [mailto:fieldtrip-bounces at science.ru.nl] On Behalf Of Eliana Garcia
> Sent: Wednesday, 11 July 2012 2:40 AM
> To: FieldTrip discussion list
> Subject: Re: [FieldTrip] problems with speed of processing
>
>
>
> Dear Stephen,
>
>
>
> Thanks for the suggestion. I have already one function that is doing all the
> preprocessing and saving as well. But this seams not to solve the issue.
>
>
>
> Thanks,
>
> Eliana
>
>
>
> On Tue, Jul 10, 2012 at 6:01 PM, Stephen Whitmarsh
> <stephen.whitmarsh at gmail.com> wrote:
>
> Dear Eliana,
>
>
>
> You could try creating a function that does all this preprocessing and then
> writes the output to file.
>
> If you then call this function in a (subject) loop you are sure that Matlab
> will free up the memory it used for that function.
>
>
>
> Perhaps it helps,
>
>
>
> Stephen
>
>
>
> On 10 July 2012 16:57, Eliana Garcia <elilife at gmail.com> wrote:
>
> Hello Dear Fieldtrip Community,
>
>
>
> I am having now problems with processing my data. I am analyzing a quite big
> data set with 400 trials that are composed by 10 sec (re-sampled at 150Hz).
> I am doing first a demean, then planar gradient transformation,
> time-frequency analysis using multitapers and then combine the planar
> gradient again. For the first subject always runs normally, but for the
> second one matlab gets really slow. I am taking care in deleting the
> variables that I create with big amount of data after I don't need them any
> more, so I am using just one big file that is being safe at the end (average
> time-frequency across trials) for each of the subjects. Every time matlab
> starts to analyze a new subject I am using clear all (with exceptions like
> directory and name of subject) but stills with every new subject it gets
> slower and slower.
>
>
>
> Do you have any advises to make the analysis faster? Sometimes is too slow
> and the computer normally crashes after 3 subjects.
>
>
>
> Thank you very much for the attention.
>
>
>
> Best,
>
>
>
> --
>
> Eliana García Cossio
> Applied Neurotechnology Lab.
>
> Graduate School of Neural and Behavioural Science - Max Planck Research
> School
> Institute of Medical Psychology and Behavioural Neurobiology
> Universität Tübingen
>
> (+0049) 01 577-8587604
> Otfried-Müller-Str. 47, 72076
> Tübingen- Germany
>
>
>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
>
>
>
>
> --
>
> Eliana García Cossio
> Applied Neurotechnology Lab.
>
> Graduate School of Neural and Behavioural Science - Max Planck Research
> School
> Institute of Medical Psychology and Behavioural Neurobiology
> Universität Tübingen
>
> (+0049) 01 577-8587604
> Otfried-Müller-Str. 47, 72076
> Tübingen- Germany
>
>
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip