[FieldTrip] A Fieldtrip data format?

[Schmiedt, Joscha]

Hey,

For your information: there is a concerted effort by many big labs to create a standard data format for electrophysiology: https://crcns.org/NWB/Overview

It will actually be based on HDF5.

Joscha

________________________________________
From: fieldtrip-bounces at science.ru.nl [fieldtrip-bounces at science.ru.nl] on behalf of Schmiedt, Joscha [joscha.schmiedt at esi-frankfurt.de]
Sent: Wednesday, April 01, 2015 2:03 PM
To: FieldTrip discussion list
Subject: Re: [FieldTrip] A Fieldtrip data format?

Hi Gio,

Thanks for the info. The fcdc_matbin approach indeed would be suitable data for raw data. However, I’m looking for a fast solution for data that is already processed and cut into trials, in particular with varying trial length.

Joscha

> On Mar 31, 2015, at 22:39, Gio Piantoni <gio at gpiantoni.com> wrote:
>
> Hi Joscha,
>
> FieldTrip already has its own simple uncompressed file format.
> fcdc_matbin "It is not an official file format, but was invented here
> at the FCDC. It consists of two files: a *.mat matlab file that
> contains the header (and optionally the events) and a *.bin binary
> file that contains the data."
> http://www.fieldtriptoolbox.org/faq/reading_is_slow_can_i_write_my_raw_data_to_a_more_efficient_file_format
>
> If you want to look at the code:
> https://github.com/fieldtrip/fieldtrip/blob/master/fileio/ft_write_data.m#L275
>
> You should be able to read it as usual with ft_preprocessing. You can
> convert your files to the fcdc_matbin using ft_preprocessing as well:
> https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L581
> Be careful about not losing precision though.
>
> The only catch is that, as far as I know, you cannot export your
> preprocessed data through ft_preprocessing at the moment, because
> ft_write_data is now inside this if-part:
> https://github.com/fieldtrip/fieldtrip/blob/master/ft_preprocessing.m#L252
> but it's only a matter of adapting that to your needs.
>
> Would this work for you?
>
> -g
>
> On Tue, Mar 31, 2015 at 9:09 AM, Schmiedt, Joscha
> <joscha.schmiedt at esi-frankfurt.de> wrote:
>> Hi,
>>
>> When working with Fieldtrip it is very convenient to store the data and
>> analyses using MATLAB’s save and load functions. However, for large and/or
>> complex data with many channels (>2GB) MATLAB enforces compression, which is
>> pretty useless for electrophysiology data and slows down the load and save
>> performance by a factor of up to 8 (see e.g.
>> http://undocumentedmatlab.com/blog/improving-save-performance). The MATLAB
>> file format is based on HDF5, which is generally an open and future-proof
>> data format, but unfortunately MATLAB doesn’t allow you to disable the
>> compression.
>>
>> An option to overcome this could be to develop a simple data format for
>> Fieldtrip data that is also based on HDF5. Is or has there been any
>> development going into that direction? Would there be any interest? Of
>> course, creating yet another data format is almost never a good idea
>> (https://xkcd.com/927/), but since HDF5 is well-documented and readable with
>> almost any software, it might be worth thinking about it.
>>
>> I’d be happy to hear your thoughts.
>>
>> Joscha
>>
>> ---------------------------------
>> Joscha Schmiedt
>> PhD Student
>>
>> Ernst Strüngmann Institute (ESI) for Neuroscience
>> in Cooperation with Max Planck Society
>> Deutschordenstraße 46
>> 60528 Frankfurt am Main
>> Germany
>>
>> Tel.: +49 (0)69 96769 241
>>
>> Sitz der Gesellschaft: Frankfurt am Main
>> Registergericht: Amtsgericht Frankfurt - HRB 84266
>> Geschäftsführer: Prof. Dr. Pascal Fries
>>
>>
>> _______________________________________________
>> fieldtrip mailing list
>> fieldtrip at donders.ru.nl
>> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip


_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip