[FieldTrip] FT doesn't close .fif files

[Robert Oostenveld]

Hi Laurence

thanks for the reply to Scott. I had typed my email on the train  
yesterday evening (while offline) and sent it out this morning,  
without checking whether there was any news.

Does fiff_setup_read_raw take a significant amount of time? I.e. is  
there a severe performance impact if we would call it on each  
ft_read_data and fclose also on each call?

If Scott follows standard FieldTrip practise, he could very well call  
ft_read_header in his script but he cannot pass the hdr into the  
subsequent ft_read_data calls (because those are done by   
ft_preprocessing, which will read it's own copy of the header info  
anyway).

Is the position of the file pointer relevant in the subsequent calls  
to ft_read_data? If not, then we could also do a plain fopen instead  
of fiff_setup_read_raw on subsequent calls to ft_read_data and also  
fclose on each call. A fopen/fseek/fclose does not take any measurable  
time on a modern operating system and file system (because the file is  
read-cached in memory anyway) and the advantage of the stateless  
implementation is worth more than lossing a milisecond on an operation  
that takes a second or more anyway (which is reading the actual data  
data and pumping it to memory).

What we could also do (alternative to the two options above) is to  
have a persistent variable for the filename and for the fid in  
ft_read_header. Using those, it could close the previous file it it  
detects that a new file is opened.

best
Robert



On 8 Feb 2011, at 16:33, Laurence Hunt wrote:

> Hi Scott -
>
> Thanks for the e-mail - it's a good point.
>
> The difficulty arises because fiff_setup_read_raw (called by  
> ft_read_header) leaves the .fif file open to be read (it stores the  
> fid in its output structure raw.fid, i.e. hdr.orig.raw.fid for  
> fieldtrip). Unfortunately if we closed it at this stage (i.e. within  
> ft_read_header), fiff_read_raw_segment, which is called by  
> ft_read_data/ft_read_event, doesn't know where to look for the file  
> - it expects a structure to be returned by fiff_setup_read_raw with  
> an open file. Again, we could close the file after calling  
> fiff_read_raw_segment in ft_read_data, but this would assume that  
> ft_read_data/ft_read_event is only going to be called once, whereas  
> some users might want to use hdr=ft_read_header once and then keep  
> on using ft_read_data with same hdr structure, in which case we need  
> to keep the file open.
>
> Unless you can think of a better solution, my suggestion would be  
> first to call
>
> hdr=ft_read_header(fname);
>
> in your script, then supply this hdr to your calls to ft_read_event/ 
> ft_read_data (this should hopefully prevent the file being opened  
> multiple times), and then once you have done all the reading, close  
> that file with
>
> fclose(hdr.orig.raw.fid);
>
> Hopefully that will stop you clogging up matlab with open files. Let  
> me know if it works or not.
>
> Cheers,
> Laurence
>
> ===========================================
> Laurence Hunt, DPhil Student
>  Centre for Functional MRI of the Brain (FMRIB),
> University of Oxford
> lhunt at fmrib.ox.ac.uk
> Phone: (+44)1865-(2)22738
> ===========================================
>
> On 4 Feb 2011, at 16:12, Scott Burns wrote:
>
>> FieldTrip Experts -
>>
>> I'm building a FieldTrip pipeline for use on M/EEG data collected  
>> on a Neuromag 306 system.  Because I'm looping over subjects (13  
>> currently), event type (5 unique events), and number of runs (4),  
>> there's lots of preprocessing to do.  After processing a certain  
>> amount of files, MATLAB refuses to open more files.
>>
>> I ran 'lsof MATLAB | grep "/autofs/cluster/kuperberg/" (to only  
>> include the data files I'm using, not MATLAB's system files) and  
>> found that for every innermost loop, my script is opening the .fif  
>> file twice and not releasing it.  I use a custom function for  
>> ft_definetrial and in that I use ft_read_event (that operates on  
>> the .fif file). After defining trials, I use ft_preprocessing.
>>
>> I would venture to guess that both ft_read_event and ft_read_data  
>> (ft_preprocessing:line 492) are either 1) not closing files they've  
>> opened (presumably through MNE functions), or 2) they're using  
>> shared code that doesn't close files.  I use MNE and haven't run  
>> into this issue before.  That being said, fclose('all') does close  
>> all these files but I think it's still worth looking into.
>>
>> FWIW I'm using CentOS 5 and MATLAB R2010B (glnxa64).
>>
>>
>> Scott Burns
>> Kuperberg Lab
>> Martinos Center for Biomedical Imaging
>> sburns at nmr.mgh.harvard.edu
>>
>> The information in this e-mail is intended only for the person to  
>> whom it is addressed. If you believe this e-mail was sent to you in  
>> error and the e-mail contains patient information, please contact  
>> the Partners Compliance HelpLine at http://www.partners.org/complianceline 
>>  . If the e-mail was sent to you in error but does not contain  
>> patient information, please contact the sender and properly dispose  
>> of the e-mail.
>>
>>
>>
>>
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip