[FieldTrip] A Fieldtrip data format?

[Gio Piantoni]

Hi Joscha,

FieldTrip already has its own simple uncompressed file format.
fcdc_matbin "It is not an official file format, but was invented here
at the FCDC. It consists of two files: a *.mat matlab file that
contains the header (and optionally the events) and a *.bin binary
file that contains the data."
http://www.fieldtriptoolbox.org/faq/reading_is_slow_can_i_write_my_raw_data_to_a_more_efficient_file_format

If you want to look at the code:
https://github.com/fieldtrip/fieldtrip/blob/master/fileio/ft_write_data.m#L275

You should be able to read it as usual with ft_preprocessing. You can
convert your files to the fcdc_matbin using ft_preprocessing as well:
https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L581
Be careful about not losing precision though.

The only catch is that, as far as I know, you cannot export your
preprocessed data through ft_preprocessing at the moment, because
ft_write_data is now inside this if-part:
https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L252
but it's only a matter of adapting that to your needs.

Would this work for you?

-g

On Tue, Mar 31, 2015 at 9:09 AM, Schmiedt, Joscha
<joscha.schmiedt at esi-frankfurt.de> wrote:
> Hi,
>
> When working with Fieldtrip it is very convenient to store the data and
> analyses using MATLAB’s save and load functions. However, for large and/or
> complex data with many channels (>2GB) MATLAB enforces compression, which is
> pretty useless for electrophysiology data and slows down the load and save
> performance by a factor of up to 8 (see e.g.
> http://undocumentedmatlab.com/blog/improving-save-performance). The MATLAB
> file format is based on HDF5, which is generally an open and future-proof
> data format, but unfortunately MATLAB doesn’t allow you to disable the
> compression.
>
> An option to overcome this could be to develop a simple data format for
> Fieldtrip data that is also based on HDF5. Is or has there been any
> development going into that direction? Would there be any interest? Of
> course, creating yet another data format is almost never a good idea
> (https://xkcd.com/927/), but since HDF5 is well-documented and readable with
> almost any software, it might be worth thinking about it.
>
> I’d be happy to hear your thoughts.
>
> Joscha
>
> ---------------------------------
> Joscha Schmiedt
> PhD Student
>
> Ernst Strüngmann Institute (ESI) for Neuroscience
> in Cooperation with Max Planck Society
> Deutschordenstraße 46
> 60528 Frankfurt am Main
> Germany
>
> Tel.: +49 (0)69 96769 241
>
> Sitz der Gesellschaft: Frankfurt am Main
> Registergericht: Amtsgericht Frankfurt - HRB 84266
> Geschäftsführer: Prof. Dr. Pascal Fries
>
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip