[FieldTrip] FT doesn't close .fif files
Laurence Hunt
lhunt at fmrib.ox.ac.uk
Wed Feb 9 13:22:18 CET 2011
p.s. I think this only applies to continuous data - evoked responses are closed in ft_read_header already, I think.
L
On 9 Feb 2011, at 12:21, Laurence Hunt wrote:
> Hi Robert,
>
> (i) Re-running fiff_setup_read_raw in ft_read_data might slow down operations with multiple calls to ft_read_data (e.g. reading multiple trials from a raw dataset), as it takes about half a second to execute fiff_setup_read_raw once.
>
> (ii) Just using fopen and fclose would be much faster, but fiff_setup_read_raw does indeed move the file cursor to a new position, so I think the best solution is to store this position after the file has been opened by fiff_setup_read_raw. So, within ft_read_header, add:
>
> raw.pos = ftell(raw.fid);
> fclose(raw.fid);
>
> and in ft_read_data, add
>
> hdr.orig.raw.fid = fopen(filename,'rb','ieee-be');
> fseek(hdr.orig.raw.fid,hdr.orig.raw.pos,'bof');
>
> before the call to fiff_read_raw_segment, and fclose it afterwards. This seems to work fine if I make these amendments to my version of the code.
>
> Hope this helps,
> Laurence
>
> On 9 Feb 2011, at 09:43, Robert Oostenveld wrote:
>
>> Hi Laurence
>>
>> thanks for the reply to Scott. I had typed my email on the train yesterday evening (while offline) and sent it out this morning, without checking whether there was any news.
>>
>> Does fiff_setup_read_raw take a significant amount of time? I.e. is there a severe performance impact if we would call it on each ft_read_data and fclose also on each call?
>>
>> If Scott follows standard FieldTrip practise, he could very well call ft_read_header in his script but he cannot pass the hdr into the subsequent ft_read_data calls (because those are done by ft_preprocessing, which will read it's own copy of the header info anyway).
>>
>> Is the position of the file pointer relevant in the subsequent calls to ft_read_data? If not, then we could also do a plain fopen instead of fiff_setup_read_raw on subsequent calls to ft_read_data and also fclose on each call. A fopen/fseek/fclose does not take any measurable time on a modern operating system and file system (because the file is read-cached in memory anyway) and the advantage of the stateless implementation is worth more than lossing a milisecond on an operation that takes a second or more anyway (which is reading the actual data data and pumping it to memory).
>>
>> What we could also do (alternative to the two options above) is to have a persistent variable for the filename and for the fid in ft_read_header. Using those, it could close the previous file it it detects that a new file is opened.
>>
>> best
>> Robert
>>
>>
>>
>> On 8 Feb 2011, at 16:33, Laurence Hunt wrote:
>>
>>> Hi Scott -
>>>
>>> Thanks for the e-mail - it's a good point.
>>>
>>> The difficulty arises because fiff_setup_read_raw (called by ft_read_header) leaves the .fif file open to be read (it stores the fid in its output structure raw.fid, i.e. hdr.orig.raw.fid for fieldtrip). Unfortunately if we closed it at this stage (i.e. within ft_read_header), fiff_read_raw_segment, which is called by ft_read_data/ft_read_event, doesn't know where to look for the file - it expects a structure to be returned by fiff_setup_read_raw with an open file. Again, we could close the file after calling fiff_read_raw_segment in ft_read_data, but this would assume that ft_read_data/ft_read_event is only going to be called once, whereas some users might want to use hdr=ft_read_header once and then keep on using ft_read_data with same hdr structure, in which case we need to keep the file open.
>>>
>>> Unless you can think of a better solution, my suggestion would be first to call
>>>
>>> hdr=ft_read_header(fname);
>>>
>>> in your script, then supply this hdr to your calls to ft_read_event/ft_read_data (this should hopefully prevent the file being opened multiple times), and then once you have done all the reading, close that file with
>>>
>>> fclose(hdr.orig.raw.fid);
>>>
>>> Hopefully that will stop you clogging up matlab with open files. Let me know if it works or not.
>>>
>>> Cheers,
>>> Laurence
>>>
>>> ===========================================
>>> Laurence Hunt, DPhil Student
>>> Centre for Functional MRI of the Brain (FMRIB),
>>> University of Oxford
>>> lhunt at fmrib.ox.ac.uk
>>> Phone: (+44)1865-(2)22738
>>> ===========================================
>>>
>>> On 4 Feb 2011, at 16:12, Scott Burns wrote:
>>>
>>>> FieldTrip Experts -
>>>>
>>>> I'm building a FieldTrip pipeline for use on M/EEG data collected on a Neuromag 306 system. Because I'm looping over subjects (13 currently), event type (5 unique events), and number of runs (4), there's lots of preprocessing to do. After processing a certain amount of files, MATLAB refuses to open more files.
>>>>
>>>> I ran 'lsof MATLAB | grep "/autofs/cluster/kuperberg/" (to only include the data files I'm using, not MATLAB's system files) and found that for every innermost loop, my script is opening the .fif file twice and not releasing it. I use a custom function for ft_definetrial and in that I use ft_read_event (that operates on the .fif file). After defining trials, I use ft_preprocessing.
>>>>
>>>> I would venture to guess that both ft_read_event and ft_read_data (ft_preprocessing:line 492) are either 1) not closing files they've opened (presumably through MNE functions), or 2) they're using shared code that doesn't close files. I use MNE and haven't run into this issue before. That being said, fclose('all') does close all these files but I think it's still worth looking into.
>>>>
>>>> FWIW I'm using CentOS 5 and MATLAB R2010B (glnxa64).
>>>>
>>>>
>>>> Scott Burns
>>>> Kuperberg Lab
>>>> Martinos Center for Biomedical Imaging
>>>> sburns at nmr.mgh.harvard.edu
>>>>
>>>> The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> fieldtrip mailing list
>>>> fieldtrip at donders.ru.nl
>>>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>>
>>> _______________________________________________
>>> fieldtrip mailing list
>>> fieldtrip at donders.ru.nl
>>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>>
>
> ===========================================
> Laurence Hunt, DPhil Student
> Centre for Functional MRI of the Brain (FMRIB),
> University of Oxford
> lhunt at fmrib.ox.ac.uk
> Phone: (+44)1865-(2)22738
> ===========================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20110209/102214ea/attachment-0002.html>
More information about the fieldtrip
mailing list