[FieldTrip] A Fieldtrip data format?

[Schmiedt, Joscha]

Hi Gio,

Thanks for the info. The fcdc_matbin approach indeed would be suitable data for raw data. However, I’m looking for a fast solution for data that is already processed and cut into trials, in particular with varying trial length. 

Joscha

> On Mar 31, 2015, at 22:39, Gio Piantoni <gio at gpiantoni.com> wrote:
> 
> Hi Joscha,
> 
> FieldTrip already has its own simple uncompressed file format.
> fcdc_matbin "It is not an official file format, but was invented here
> at the FCDC. It consists of two files: a *.mat matlab file that
> contains the header (and optionally the events) and a *.bin binary
> file that contains the data."
> http://www.fieldtriptoolbox.org/faq/reading_is_slow_can_i_write_my_raw_data_to_a_more_efficient_file_format
> 
> If you want to look at the code:
> https://github.com/fieldtrip/fieldtrip/blob/master/fileio/ft_write_data.m#L275
> 
> You should be able to read it as usual with ft_preprocessing. You can
> convert your files to the fcdc_matbin using ft_preprocessing as well:
> https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L581
> Be careful about not losing precision though.
> 
> The only catch is that, as far as I know, you cannot export your
> preprocessed data through ft_preprocessing at the moment, because
> ft_write_data is now inside this if-part:
> https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L252
> but it's only a matter of adapting that to your needs.
> 
> Would this work for you?
> 
> -g
> 
> On Tue, Mar 31, 2015 at 9:09 AM, Schmiedt, Joscha
> <joscha.schmiedt at esi-frankfurt.de> wrote:
>> Hi,
>> 
>> When working with Fieldtrip it is very convenient to store the data and
>> analyses using MATLAB’s save and load functions. However, for large and/or
>> complex data with many channels (>2GB) MATLAB enforces compression, which is
>> pretty useless for electrophysiology data and slows down the load and save
>> performance by a factor of up to 8 (see e.g.
>> http://undocumentedmatlab.com/blog/improving-save-performance). The MATLAB
>> file format is based on HDF5, which is generally an open and future-proof
>> data format, but unfortunately MATLAB doesn’t allow you to disable the
>> compression.
>> 
>> An option to overcome this could be to develop a simple data format for
>> Fieldtrip data that is also based on HDF5. Is or has there been any
>> development going into that direction? Would there be any interest? Of
>> course, creating yet another data format is almost never a good idea
>> (https://xkcd.com/927/), but since HDF5 is well-documented and readable with
>> almost any software, it might be worth thinking about it.
>> 
>> I’d be happy to hear your thoughts.
>> 
>> Joscha
>> 
>> ---------------------------------
>> Joscha Schmiedt
>> PhD Student
>> 
>> Ernst Strüngmann Institute (ESI) for Neuroscience
>> in Cooperation with Max Planck Society
>> Deutschordenstraße 46
>> 60528 Frankfurt am Main
>> Germany
>> 
>> Tel.: +49 (0)69 96769 241
>> 
>> Sitz der Gesellschaft: Frankfurt am Main
>> Registergericht: Amtsgericht Frankfurt - HRB 84266
>> Geschäftsführer: Prof. Dr. Pascal Fries
>> 
>> 
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> 
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip