[FieldTrip] importing large edf data
Stephen Politzer-Ahles
stephen.politzer-ahles at ling-phil.ox.ac.uk
Sat Dec 19 15:06:46 CET 2015
Hi Leila,
I'm not sure why it didn't work for you. I tried the following code (on a
sample EDF file I found at http://www.teuniz.net/edf_bdf_testfiles/) and it
worked perfectly for me:
cfg = [];
cfg.dataset = 'C:\Users\Spolitzerahles\Desktop\ma0844az_1-1+.edf';
cfg.trl = [1 500 0];
cfg.continuous = 'yes';
data_500samples = ft_preprocessing(cfg);
cfg = [];
cfg.dataset = 'C:\Users\Spolitzerahles\Desktop\ma0844az_1-1+.edf';
cfg.trl = [1 1000 0];
cfg.continuous = 'yes';
data_1000samples = ft_preprocessing(cfg);
Here's the output:
>> data_500samples
data_500samples =
hdr: [1x1 struct]
label: {37x1 cell}
time: {[1x500 double]}
trial: {[37x500 double]}
fsample: 200
sampleinfo: [1 500]
cfg: [1x1 struct]
>> data_1000samples
data_1000samples =
hdr: [1x1 struct]
label: {37x1 cell}
time: {[1x1000 double]}
trial: {[37x1000 double]}
fsample: 200
sampleinfo: [1 1000]
cfg: [1x1 struct]
Best,
Steve
---
Stephen Politzer-Ahles
University of Oxford
Language and Brain Lab
Faculty of Linguistics, Phonetics & Philology
http://users.ox.ac.uk/~cpgl0080/
> ------------------------------
>
> Message: 5
> Date: Fri, 18 Dec 2015 13:40:59 +0000 (UTC)
> From: Leila Ayoubian <leilayou_54 at yahoo.com>
> To: FieldTrip discussion list <fieldtrip at science.ru.nl>
> Subject: Re: [FieldTrip] importing large edf data
> Message-ID:
> <2054001626.533069.1450446059200.JavaMail.yahoo at mail.yahoo.com>
> Content-Type: text/plain; charset="utf-8"
>
> ?Thanks Steve for your reply.I think your code for segmenting the data is
> the way to go for me ! :) And I do appreciate you gave me an example.?
> Thanks
> However I tried it with two different cfg.trl = [1 1000 0]; and cfg.trl =
> [1 10000 0]; to see the results, and it looked the same!
>
> So this is what I get:
>
>
> ?? hdr: [1x1 struct]
> ???????? label: {51x1 cell}
> ????????? time: {1x50710 cell}
> ???????? trial: {1x50710 cell}
> ?????? fsample: 1024
> ??? sampleinfo: [50710x2 double]
> ?????????? cfg: [1x1 struct]
>
> so the actual data which is in data.trial? is this in both cases!
> size(data.trial)
>
> ans =
>
> ?????????? 1?????? 50710
>
> How could that happen if you are segmenting it in two different chunks?
> Best regards
>
> ___________________________________________________ Leila Ayoubian
>
>
>
> From: Stephen Politzer-Ahles <
> stephen.politzer-ahles at ling-phil.ox.ac.uk>
> To: fieldtrip at science.ru.nl
> Sent: Friday, December 18, 2015 12:37 PM
> Subject: Re: [FieldTrip] importing large edf data
>
> Hello Leila,
>
> I think there are a few possible options for you. If you're not interested
> in analyzing high frequencies, then you could import the data one sample at
> a time, downsampling each channel, and then concatenate the channels into
> one structure, as described in
> http://www.fieldtriptoolbox.org/faq/how_can_i_preprocess_a_dataset_that_is_too_large_to_fit_into_memory.
> I don't know what your sampling rate was, but if your sampling rate was
> extremely high then this can save you a lot of space. For example, if
> you're only interested in frequencies up to high gamma (around 90-100 Hz),
> then your data only need to be sampled at around 250 Hz (per the Nyquist
> theorem, plus a little extra wiggle room), so if you sampled at e.g. 2500
> Hz (which our EDF system does by default) or 5000 Hz then you have way more
> samples than necessary; downsampling like this will greatly reduce your
> array size.
>
>
>
> If that is not an option, then you can read in a segment of data as a
> 'trial'. ft_preprocessing understands trial start and end times from
> cfg.trl, which minimally is an Nx3 matrix (N is the number of trials to
> read, and the columns are the start sample for each trial, the end sample
> for each trial, and the trigger value associated with the trial; you could
> just put 0 if there's no trigger). So let's say you want to read the first
> through thousandth sample of the recording, then you would set up your cfg
> as normal, then add:
>
> cfg.trl = [1 1000 0];
>
> and then use ft_preprocessing.
>
> By way of example, here is code that reads in raw data as a series of
> 1-second "trials":
>
> cfg = [];
> cfg.dataset = 'C:\Users\Steve\Desktop\subj2.vhdr';
> cfg.trialdef.triallength = 1; % see documentation for ft_definetrial for
> an explanation of these parameters
> cfg.trialdef.ntrials = Inf;
> cfg = ft_definetrial( cfg );
>
> cfg.continuous = 'yes';
> data = ft_preprocessing(cfg);
>
> After doing whatever you need to these trials, you could concatenate them
> back into continuous data (see
> http://mailman.science.ru.nl/pipermail/fieldtrip/2011-June/003971.html
> for details).
>
> Best,
> Steve
>
> ---
> Stephen Politzer-Ahles
> University of Oxford
> Language and Brain Lab
> Faculty of Linguistics, Phonetics & Philology
> http://users.ox.ac.uk/~cpgl0080/
> On Fri, Dec 18, 2015 at 11:00 AM, <fieldtrip-request at science.ru.nl> wrote:
>
> Send fieldtrip mailing list submissions to
> ? ? ? ? fieldtrip at science.ru.nl
>
> To subscribe or unsubscribe via the World Wide Web, visit
> ? ? ? ? http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> or, via email, send a message with subject or body 'help' to
> ? ? ? ? fieldtrip-request at science.ru.nl
>
> You can reach the person managing the list at
> ? ? ? ? fieldtrip-owner at science.ru.nl
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of fieldtrip digest..."
>
>
> Today's Topics:
>
> ? ?1. importing large edf data (Leila Ayoubian)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 18 Dec 2015 09:43:43 +0000 (UTC)
> From: Leila Ayoubian <leilayou_54 at yahoo.com>
> To: "fieldtrip at science.ru.nl" <fieldtrip at science.ru.nl>
> Subject: [FieldTrip] importing large edf data
> Message-ID:
> ? ? ? ? <807603762.476165.1450431823359.JavaMail.yahoo at mail.yahoo.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi:Thanks for providing us with this amazing toolbox.
> I am new to fieldtrip. I am trying to import EEG data in the format of
> *.edf for the purpose of seizure detection for epileptic patients. I can't
> break up the data as you understand the detection has to have continuous
> data. However I could load section of the data at a time and clear what is
> already used and reload again.?
> So trying to load the data like this:
> cfg.dataset= ('mydata.edf');
> rawdata=ft_preprocessing(cfg);
> This is the error message I get :
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> Error using zeros
> Requested 51x51927040 (19.7GB) array exceeds maximum array size
> preference. Creation of arrays greater than this limit may take a long time
> and cause MATLAB to become unresponsive. See array size limit or
> preference panel for more information.
>
> Error in read_edf (line 403)
> ? dat = zeros(length(chanindx),nepochs*epochlength);
>
> Error in ft_read_data (line 622)
> ??? dat = read_edf(filename, hdr, begsample, endsample, chanindx);
>
> Error in ft_preprocessing (line 566)
> ????? dat = ft_read_data(cfg.datafile, 'header', hdr, 'begsample',
> begsample, 'endsample', endsample, 'chanindx', rawindx, 'checkboundary',
> ????? strcmp(cfg.continuous, 'no'), 'dataformat', cfg.dataformat);
>
> Error in ReadFedfiles (line 8)
>
> rawdata=ft_preprocessing(cfg);%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> I tried the ft_redefinetrial, but since the input to this file is the
> output from ft-PREPROCESSING it gives me the same error.
>
>
> I also tried to use ft_read_data to choose the samples I want to load
> :?filename???? = 'mydata.edf'
> ?hdr?? = ft_read_header(filename);
> ?sample.start=1
> ?sample.end=100
> dat = ft_read_data(filename,'sample.start','1','sample.end','100')
> and here is again the error message:
> Error using zeros
> Requested 51x51927040 (19.7GB) array exceeds maximum array size
> preference. Creation of arrays greater than this limit may take a
> long time and cause MATLAB to become unresponsive. See array size limit or
> preference panel for more information.
>
> Error in read_edf (line 403)
> ? dat = zeros(length(chanindx),nepochs*epochlength);
>
> Error in ft_read_data (line 622)
> ??? dat = read_edf(filename, hdr, begsample, endsample, chanindx);
>
> Error in ReadFedfiles (line 31)
> dat = ft_read_data(filename,'sample.start','1','sample.end','100')
> ?
> >>
>
> Could you please direct me as to which functions and in which order I
> should be using the functions to load data in smaller samples.Some examples
> would be useful.
> Thanks again for your support and assistance.
> We appreciate your effort.
> Kind regards
>
> ___________________________________________________ Dr. Leila Ayoubian
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20151218/6dfa7ce1/attachment-0001.html
> >
>
> ------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20151219/3b1c1290/attachment-0002.html>
More information about the fieldtrip
mailing list