[FieldTrip] FT doesn't close .fif files
r.oostenveld at donders.ru.nl
Wed Feb 9 10:43:55 CET 2011
thanks for the reply to Scott. I had typed my email on the train
yesterday evening (while offline) and sent it out this morning,
without checking whether there was any news.
Does fiff_setup_read_raw take a significant amount of time? I.e. is
there a severe performance impact if we would call it on each
ft_read_data and fclose also on each call?
If Scott follows standard FieldTrip practise, he could very well call
ft_read_header in his script but he cannot pass the hdr into the
subsequent ft_read_data calls (because those are done by
ft_preprocessing, which will read it's own copy of the header info
Is the position of the file pointer relevant in the subsequent calls
to ft_read_data? If not, then we could also do a plain fopen instead
of fiff_setup_read_raw on subsequent calls to ft_read_data and also
fclose on each call. A fopen/fseek/fclose does not take any measurable
time on a modern operating system and file system (because the file is
read-cached in memory anyway) and the advantage of the stateless
implementation is worth more than lossing a milisecond on an operation
that takes a second or more anyway (which is reading the actual data
data and pumping it to memory).
What we could also do (alternative to the two options above) is to
have a persistent variable for the filename and for the fid in
ft_read_header. Using those, it could close the previous file it it
detects that a new file is opened.
On 8 Feb 2011, at 16:33, Laurence Hunt wrote:
> Hi Scott -
> Thanks for the e-mail - it's a good point.
> The difficulty arises because fiff_setup_read_raw (called by
> ft_read_header) leaves the .fif file open to be read (it stores the
> fid in its output structure raw.fid, i.e. hdr.orig.raw.fid for
> fieldtrip). Unfortunately if we closed it at this stage (i.e. within
> ft_read_header), fiff_read_raw_segment, which is called by
> ft_read_data/ft_read_event, doesn't know where to look for the file
> - it expects a structure to be returned by fiff_setup_read_raw with
> an open file. Again, we could close the file after calling
> fiff_read_raw_segment in ft_read_data, but this would assume that
> ft_read_data/ft_read_event is only going to be called once, whereas
> some users might want to use hdr=ft_read_header once and then keep
> on using ft_read_data with same hdr structure, in which case we need
> to keep the file open.
> Unless you can think of a better solution, my suggestion would be
> first to call
> in your script, then supply this hdr to your calls to ft_read_event/
> ft_read_data (this should hopefully prevent the file being opened
> multiple times), and then once you have done all the reading, close
> that file with
> Hopefully that will stop you clogging up matlab with open files. Let
> me know if it works or not.
> Laurence Hunt, DPhil Student
> Centre for Functional MRI of the Brain (FMRIB),
> University of Oxford
> lhunt at fmrib.ox.ac.uk
> Phone: (+44)1865-(2)22738
> On 4 Feb 2011, at 16:12, Scott Burns wrote:
>> FieldTrip Experts -
>> I'm building a FieldTrip pipeline for use on M/EEG data collected
>> on a Neuromag 306 system. Because I'm looping over subjects (13
>> currently), event type (5 unique events), and number of runs (4),
>> there's lots of preprocessing to do. After processing a certain
>> amount of files, MATLAB refuses to open more files.
>> I ran 'lsof MATLAB | grep "/autofs/cluster/kuperberg/" (to only
>> include the data files I'm using, not MATLAB's system files) and
>> found that for every innermost loop, my script is opening the .fif
>> file twice and not releasing it. I use a custom function for
>> ft_definetrial and in that I use ft_read_event (that operates on
>> the .fif file). After defining trials, I use ft_preprocessing.
>> I would venture to guess that both ft_read_event and ft_read_data
>> (ft_preprocessing:line 492) are either 1) not closing files they've
>> opened (presumably through MNE functions), or 2) they're using
>> shared code that doesn't close files. I use MNE and haven't run
>> into this issue before. That being said, fclose('all') does close
>> all these files but I think it's still worth looking into.
>> FWIW I'm using CentOS 5 and MATLAB R2010B (glnxa64).
>> Scott Burns
>> Kuperberg Lab
>> Martinos Center for Biomedical Imaging
>> sburns at nmr.mgh.harvard.edu
>> The information in this e-mail is intended only for the person to
>> whom it is addressed. If you believe this e-mail was sent to you in
>> error and the e-mail contains patient information, please contact
>> the Partners Compliance HelpLine at http://www.partners.org/complianceline
>> . If the e-mail was sent to you in error but does not contain
>> patient information, please contact the sender and properly dispose
>> of the e-mail.
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
More information about the fieldtrip