<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Dear Scott,<div><br></div><div>thanks for reporting this problem and for already tracking it down to files not being closed. </div><div><br></div><div>ft_read_data does the following on every read-request (line 772), </div><div><div> case {'neuromag_fif' 'neuromag_mne'}</div><div> % check that the required low-level toolbox is available</div><div> ft_hastoolbox('mne', 1);</div><div> if (hdr.orig.iscontinuous)</div><div> dat = fiff_read_raw_segment(hdr.orig.raw,begsample+hdr.orig.raw.first_samp-1,endsample+hdr.orig.raw.first_samp-1,chanindx);</div></div><div><br></div><div>from this I understand that the file is already supposed to be open and represented in hdr.orig.raw (hdr.orig is in general where fieldtrip stores the original header details, i.e. including all system specific information).</div><div><br></div><div>and prior to that it does (line 223)</div><div><div>% read the header if it is not provided</div><div>if isempty(hdr)</div><div> hdr = ft_read_header(filename, 'headerformat', headerformat);</div><div>end</div></div><div><br></div><div>I would expect the part on ft_read_header not to be executed on every call to ft_read_data, because in general ft_read_header has already been called outside ft_read_data (e.g. to select the channels prior to start the reading). So ft_preprocessing calls ft_read_header once and ft_read_data many times. </div><div><br></div><div>and in ft_read_header it does</div><div><div><div> orig = fiff_read_meas_info(filename);</div><div>and</div><div> raw = fiff_setup_read_raw(filename);</div></div></div><div><br></div><div>Diagnosis: ft_read_header opens the file, ft_read_data keeps it open. Then you move to the next file, and that one is also opened (and kept open). Hmm, indeed a problem. The design idea is that reading from files shoudl be stateless, i.e. prior and after each read operation the file should be closed. On modern file systems there is no penalty associated with that, and it keeps the code much cleaner. That is why fieldtrip does not have a ft_open_file and ft_close_file function, and why you cannot close the file (except with fclose('all')).</div><div><br></div><div>Do you happen to know whether fiff_read_meas_info and fiff_setup_read_raw are supposed to keep the files open? I could not find a fiff_close function, do you know whether there is one? </div><div><br></div><div>best</div><div>Robert</div><div><br></div><div>PS let's do the follow up discussion via the bug tracking system and not on the email list... you should have received a mail from that by now.</div><div><br></div><div><br><div><div>On 4 Feb 2011, at 17:12, Scott Burns wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">FieldTrip Experts -<div><br></div><div>I'm building a FieldTrip pipeline for use on M/EEG data collected on a Neuromag 306 system. Because I'm looping over subjects (13 currently), event type (5 unique events), and number of runs (4), there's lots of preprocessing to do. After processing a certain amount of files, MATLAB refuses to open more files. </div><div><br></div><div>I ran 'lsof MATLAB | grep "/autofs/cluster/kuperberg/" (to only include the data files I'm using, not MATLAB's system files) and found that for every innermost loop, my script is opening the .fif file twice and not releasing it. I use a custom function for ft_definetrial and in that I use ft_read_event (that operates on the .fif file). After defining trials, I use ft_preprocessing.</div><div><br></div><div>I would venture to guess that both ft_read_event and ft_read_data (ft_preprocessing:line 492) are either 1) not closing files they've opened (presumably through MNE functions), or 2) they're using shared code that doesn't close files. I use MNE and haven't run into this issue before. That being said, fclose('all') does close all these files but I think it's still worth looking into.</div><div><br></div><div>FWIW I'm using CentOS 5 and MATLAB R2010B (glnxa64).</div><div><br></div><div><br><div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Scott Burns</div><div>Kuperberg Lab</div><div>Martinos Center for Biomedical Imaging</div><div><a href="mailto:sburns@nmr.mgh.harvard.edu">sburns@nmr.mgh.harvard.edu</a></div><div><br></div><div><span class="Apple-style-span" style="border-collapse: collapse; "><font class="Apple-style-span" face="Arial" size="1"><span class="Apple-style-span" style="font-size: 9px; ">The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at </span></font></span><span class="Apple-style-span" style="border-collapse: collapse; "><a href="http://www.partners.org/complianceline" target="_blank"><font class="Apple-style-span" face="Arial" size="1"><span class="Apple-style-span" style="font-size: 9px; "><font class="Apple-style-span">http://www.partners.org/<wbr>complianceline</font></span></font></a></span><span class="Apple-style-span" style="border-collapse: collapse; "><font class="Apple-style-span" face="Arial" size="1"><span class="Apple-style-span" style="font-size: 9px; "> . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.</span></font></span></div><div><br></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div></div>_______________________________________________<br>fieldtrip mailing list<br><a href="mailto:fieldtrip@donders.ru.nl">fieldtrip@donders.ru.nl</a><br>http://mailman.science.ru.nl/mailman/listinfo/fieldtrip<br></blockquote></div><br></div></body></html>