<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">p.s. I think this only applies to continuous data - evoked responses are closed in ft_read_header already, I think.<div><br></div><div>L</div><div><div><br><div><div>On 9 Feb 2011, at 12:21, Laurence Hunt wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Robert,<div><br></div><div>(i) Re-running fiff_setup_read_raw in ft_read_data might slow down operations with multiple calls to ft_read_data (e.g. reading multiple trials from a raw dataset), as it takes about half a second to execute fiff_setup_read_raw once.</div><div><br></div><div>(ii) Just using fopen and fclose would be much faster, but fiff_setup_read_raw does indeed move the file cursor to a new position, so I think the best solution is to store this position after the file has been opened by fiff_setup_read_raw. So, within ft_read_header, add:</div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; "><br></span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; ">raw.pos = ftell(raw.fid);</span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; ">fclose(raw.fid);</span></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Courier; min-height: 14px; "><br></div></div><div>and in ft_read_data, add</div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; "><br></span></div><div><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; ">hdr.orig.raw.fid = fopen(filename,</span><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; "><span style="color: #bf00f7">'rb'</span></span><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; ">,</span><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; "><span style="color: #bf00f7">'ieee-be'</span></span><span class="Apple-style-span" style="font-family: Courier; font-size: 10px; ">);</span></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">fseek(hdr.orig.raw.fid,hdr.orig.raw.pos,<span style="color: #bf00f7">'bof'</span>);</div></div><div><br></div><div><div>
</div>
before the call to fiff_read_raw_segment, and fclose it afterwards. This seems to work fine if I make these amendments to my version of the code.</div><div><br></div><div>Hope this helps,</div><div>Laurence</div><div><br><div><div>On 9 Feb 2011, at 09:43, Robert Oostenveld wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi Laurence<br><br>thanks for the reply to Scott. I had typed my email on the train yesterday evening (while offline) and sent it out this morning, without checking whether there was any news.<br><br>Does fiff_setup_read_raw take a significant amount of time? I.e. is there a severe performance impact if we would call it on each ft_read_data and fclose also on each call?<br><br>If Scott follows standard FieldTrip practise, he could very well call ft_read_header in his script but he cannot pass the hdr into the subsequent ft_read_data calls (because those are done by ft_preprocessing, which will read it's own copy of the header info anyway).<br><br>Is the position of the file pointer relevant in the subsequent calls to ft_read_data? If not, then we could also do a plain fopen instead of fiff_setup_read_raw on subsequent calls to ft_read_data and also fclose on each call. A fopen/fseek/fclose does not take any measurable time on a modern operating system and file system (because the file is read-cached in memory anyway) and the advantage of the stateless implementation is worth more than lossing a milisecond on an operation that takes a second or more anyway (which is reading the actual data data and pumping it to memory).<br><br>What we could also do (alternative to the two options above) is to have a persistent variable for the filename and for the fid in ft_read_header. Using those, it could close the previous file it it detects that a new file is opened.<br><br>best<br>Robert<br><br><br><br>On 8 Feb 2011, at 16:33, Laurence Hunt wrote:<br><br><blockquote type="cite">Hi Scott -<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks for the e-mail - it's a good point.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The difficulty arises because fiff_setup_read_raw (called by ft_read_header) leaves the .fif file open to be read (it stores the fid in its output structure raw.fid, i.e. hdr.orig.raw.fid for fieldtrip). Unfortunately if we closed it at this stage (i.e. within ft_read_header), fiff_read_raw_segment, which is called by ft_read_data/ft_read_event, doesn't know where to look for the file - it expects a structure to be returned by fiff_setup_read_raw with an open file. Again, we could close the file after calling fiff_read_raw_segment in ft_read_data, but this would assume that ft_read_data/ft_read_event is only going to be called once, whereas some users might want to use hdr=ft_read_header once and then keep on using ft_read_data with same hdr structure, in which case we need to keep the file open.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Unless you can think of a better solution, my suggestion would be first to call<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">hdr=ft_read_header(fname);<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">in your script, then supply this hdr to your calls to ft_read_event/ft_read_data (this should hopefully prevent the file being opened multiple times), and then once you have done all the reading, close that file with<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">fclose(hdr.orig.raw.fid);<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hopefully that will stop you clogging up matlab with open files. Let me know if it works or not.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers,<br></blockquote><blockquote type="cite">Laurence<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">===========================================<br></blockquote><blockquote type="cite">Laurence Hunt, DPhil Student<br></blockquote><blockquote type="cite"> Centre for Functional MRI of the Brain (FMRIB),<br></blockquote><blockquote type="cite">University of Oxford<br></blockquote><blockquote type="cite"><a href="mailto:lhunt@fmrib.ox.ac.uk">lhunt@fmrib.ox.ac.uk</a><br></blockquote><blockquote type="cite">Phone: (+44)1865-(2)22738<br></blockquote><blockquote type="cite">===========================================<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On 4 Feb 2011, at 16:12, Scott Burns wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">FieldTrip Experts -<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I'm building a FieldTrip pipeline for use on M/EEG data collected on a Neuromag 306 system. Because I'm looping over subjects (13 currently), event type (5 unique events), and number of runs (4), there's lots of preprocessing to do. After processing a certain amount of files, MATLAB refuses to open more files.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I ran 'lsof MATLAB | grep "/autofs/cluster/kuperberg/" (to only include the data files I'm using, not MATLAB's system files) and found that for every innermost loop, my script is opening the .fif file twice and not releasing it. I use a custom function for ft_definetrial and in that I use ft_read_event (that operates on the .fif file). After defining trials, I use ft_preprocessing.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I would venture to guess that both ft_read_event and ft_read_data (ft_preprocessing:line 492) are either 1) not closing files they've opened (presumably through MNE functions), or 2) they're using shared code that doesn't close files. I use MNE and haven't run into this issue before. That being said, fclose('all') does close all these files but I think it's still worth looking into.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">FWIW I'm using CentOS 5 and MATLAB R2010B (glnxa64).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Scott Burns<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Kuperberg Lab<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Martinos Center for Biomedical Imaging<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:sburns@nmr.mgh.harvard.edu">sburns@nmr.mgh.harvard.edu</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at <a href="http://www.partners.org/complianceline">http://www.partners.org/complianceline</a> . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">fieldtrip mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:fieldtrip@donders.ru.nl">fieldtrip@donders.ru.nl</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">fieldtrip mailing list<br></blockquote><blockquote type="cite"><a href="mailto:fieldtrip@donders.ru.nl">fieldtrip@donders.ru.nl</a><br></blockquote><blockquote type="cite"><a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br></blockquote><br>_______________________________________________<br>fieldtrip mailing list<br><a href="mailto:fieldtrip@donders.ru.nl">fieldtrip@donders.ru.nl</a><br><a href="http://mailman.science.ru.nl/mailman/listinfo/fieldtrip">http://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br><br></div></blockquote></div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">===========================================</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">Laurence Hunt, DPhil Student</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "> Centre for Functional MRI of the Brain (FMRIB), </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">University of Oxford</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; "><a href="mailto:lhunt@fmrib.ox.ac.uk">lhunt@fmrib.ox.ac.uk</a></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">Phone: (+44)1865-(2)22738</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font-size: 12px; ">===========================================</div></div></div></blockquote></div><br></div></div></body></html>