[FieldTrip] importing large edf data
Leila Ayoubian
leilayou_54 at yahoo.com
Thu May 5 16:35:07 CEST 2016
Hi,I am trying to concatenate two EDF files in Matlab and convert it back to EDF. (basically sticking the original data to on piece)Here is my code. It does create the 'a' file but it does not open as EDF.Any suggestion?
clc;clear all;close all;%%[fileName1, pathName1] = uigetfile({'*.edf;*.e', 'Nicolete files (*.edf;*.e)'; '*.*', 'All files (*.*)'}, 'Select edf file');file1 = fullfile(pathName1, fileName1);cfg1 = [];%cfg.trl = [1 1000 0];%45min*60=2700 sec % 2700sec*1000=2700000 mscfg1.trialdef.triallength = 5;cfg1.dataset =file1; %'For leila.edf';cfg1= ft_definetrial( cfg1 );cfg1.continuous = 'yes';datastr1 = ft_preprocessing(cfg1);data1=datastr1.trial{1,1};
Fs=datastr1.fsample;
hdr=datastr1.hdr;%%
[fileName2, pathName2] = uigetfile({'*.edf;*.e', 'Nicolete files (*.edf;*.e)'; '*.*', 'All files (*.*)'}, 'Select edf file');file2 = fullfile(pathName2, fileName2);cfg2 = [];%cfg.trl = [1 1000 0];%45min*60=2700 sec % 2700sec*1000=2700000 mscfg2.trialdef.triallength = 5;cfg2.dataset =file2; %'For leila.edf';cfg2= ft_definetrial( cfg2 );cfg2.continuous = 'yes';datastr2 = ft_preprocessing(cfg2);
data2=datastr2.trial{1,1};
Fs2=datastr2.fsample;
% EEG=cat(2,data1,data2)% EEG=EEG';% HDR=hdr1% out=EDFCreator(EEG);
dat = cat(2,data1,data2);%%% write_edf('a', hdr, dat);
ft_write_data('a', dat, 'header', hdr,'dataformat','edf' ) __________________________________________________ Leila Ayoubian, PhD.
Institute of Psychiatry, Psychology and Neuroscience
King's College London
From: Stephen Politzer-Ahles <stephen.politzer-ahles at ling-phil.ox.ac.uk>
To: fieldtrip at science.ru.nl
Sent: Monday, December 21, 2015 11:39 AM
Subject: Re: [FieldTrip] importing large edf data
Hi Leila,
cfg.trialdef.triallength is a parameter for ft_definetrial(), which is not needed if all you want to do is read a single segment of continuous data based on predetermined latencies; you may notice that, in the sample code snippets I sent in my last message, I did not use this function at all, I just hard-coded the latency range in cfg.trl (and then passed this cfg structure to ft_preprocessing() ). So, if this is all you need to do, there is no need to worry about these additional parameters.
If you are trying to read multiple segments of data as separate epochs into one dataset (which is what you would accomplish by using cfg.trialdef.triallength = 1 and ft_definetrial() ), then I'm not sure why it's not working for you at the moment; I would have to see your code to be able to diagnose what might be going on.
Best,
Steve
---
Stephen Politzer-Ahles
University of Oxford
Language and Brain Lab
Faculty of Linguistics, Phonetics & Philology
http://users.ox.ac.uk/~cpgl0080/
Message: 2
Date: Mon, 21 Dec 2015 09:36:27 +0000 (UTC)
From: Leila Ayoubian <leilayou_54 at yahoo.com>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] importing large edf data
Message-ID:
<1459455402.1301519.1450690587900.JavaMail.yahoo at mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"
Hi Steve,Thanks for checking for me. You are indeed right. Not sure why I got a different result at first.
But now I see thecfg2.trialdef.triallength = 1;?
there is no difference in the way data is loaded once I change the number to let's say 5! Where should I see that difference??Thanks again.Kind regards
?__________________________________________________ Leila Ayoubian, PhD.
Institute of Psychiatry, Psychology and Neuroscience
King's College London
From: Stephen Politzer-Ahles <stephen.politzer-ahles at ling-phil.ox.ac.uk>
To: fieldtrip at science.ru.nl
Sent: Saturday, December 19, 2015 2:06 PM
Subject: Re: [FieldTrip] importing large edf data
Hi Leila,
I'm not sure why it didn't work for you. I tried the following code (on a sample EDF file I found at http://www.teuniz.net/edf_bdf_testfiles/) and it worked perfectly for me:
cfg = [];
cfg.dataset = 'C:\Users\Spolitzerahles\Desktop\ma0844az_1-1+.edf';
cfg.trl = [1 500 0];
cfg.continuous = 'yes';
data_500samples = ft_preprocessing(cfg);
cfg = [];
cfg.dataset = 'C:\Users\Spolitzerahles\Desktop\ma0844az_1-1+.edf';
cfg.trl = [1 1000 0];
cfg.continuous = 'yes';
data_1000samples = ft_preprocessing(cfg);
Here's the output:
>> data_500samples
data_500samples =
?????????? hdr: [1x1 struct]
???????? label: {37x1 cell}
????????? time: {[1x500 double]}
???????? trial: {[37x500 double]}
?????? fsample: 200
??? sampleinfo: [1 500]
?????????? cfg: [1x1 struct]
>> data_1000samples
data_1000samples =
?????????? hdr: [1x1 struct]
???????? label: {37x1 cell}
????????? time: {[1x1000 double]}
???????? trial: {[37x1000 double]}
?????? fsample: 200
??? sampleinfo: [1 1000]
?????????? cfg: [1x1 struct]
Best,
Steve
---
Stephen Politzer-Ahles
University of Oxford
Language and Brain Lab
Faculty of Linguistics, Phonetics & Philology
http://users.ox.ac.uk/~cpgl0080/?
------------------------------
Message: 5
Date: Fri, 18 Dec 2015 13:40:59 +0000 (UTC)
From: Leila Ayoubian <leilayou_54 at yahoo.com>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] importing large edf data
Message-ID:
? ? ? ? <2054001626.533069.1450446059200.JavaMail.yahoo at mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"
?Thanks Steve for your reply.I think your code for segmenting the data is the way to go for me ! :) And I do appreciate you gave me an example.? Thanks
However I tried it with two different cfg.trl = [1 1000 0]; and cfg.trl = [1 10000 0]; to see the results, and it looked the same!
So this is what I get:
?? hdr: [1x1 struct]
???????? label: {51x1 cell}
????????? time: {1x50710 cell}
???????? trial: {1x50710 cell}
?????? fsample: 1024
??? sampleinfo: [50710x2 double]
?????????? cfg: [1x1 struct]
so the actual data which is in data.trial? is this in both cases!
size(data.trial)
ans =
?????????? 1?????? 50710
How could that happen if you are segmenting it in two different chunks?
Best regards
___________________________________________________ Leila Ayoubian
? ? ? From: Stephen Politzer-Ahles <stephen.politzer-ahles at ling-phil.ox.ac.uk>
?To: fieldtrip at science.ru.nl
?Sent: Friday, December 18, 2015 12:37 PM
?Subject: Re: [FieldTrip] importing large edf data
Hello Leila,
I think there are a few possible options for you. If you're not interested in analyzing high frequencies, then you could import the data one sample at a time, downsampling each channel, and then concatenate the channels into one structure, as described in http://www.fieldtriptoolbox.org/faq/how_can_i_preprocess_a_dataset_that_is_too_large_to_fit_into_memory. I don't know what your sampling rate was, but if your sampling rate was extremely high then this can save you a lot of space. For example, if you're only interested in frequencies up to high gamma (around 90-100 Hz), then your data only need to be sampled at around 250 Hz (per the Nyquist theorem, plus a little extra wiggle room), so if you sampled at e.g. 2500 Hz (which our EDF system does by default) or 5000 Hz then you have way more samples than necessary; downsampling like this will greatly reduce your array size.
If that is not an option, then you can read in a segment of data as a 'trial'. ft_preprocessing understands trial start and end times from cfg.trl, which minimally is an Nx3 matrix (N is the number of trials to read, and the columns are the start sample for each trial, the end sample for each trial, and the trigger value associated with the trial; you could just put 0 if there's no trigger). So let's say you want to read the first through thousandth sample of the recording, then you would set up your cfg as normal, then add:
cfg.trl = [1 1000 0];
and then use ft_preprocessing.
By way of example, here is code that reads in raw data as a series of 1-second "trials":
cfg = [];
cfg.dataset = 'C:\Users\Steve\Desktop\subj2.vhdr';
cfg.trialdef.triallength = 1; % see documentation for ft_definetrial for an explanation of these parameters
cfg.trialdef.ntrials = Inf;
cfg = ft_definetrial( cfg );
cfg.continuous = 'yes';
data = ft_preprocessing(cfg);
After doing whatever you need to these trials, you could concatenate them back into continuous data (see http://mailman.science.ru.nl/pipermail/fieldtrip/2011-June/003971.html for details).
Best,
Steve
---
Stephen Politzer-Ahles
University of Oxford
Language and Brain Lab
Faculty of Linguistics, Phonetics & Philology
http://users.ox.ac.uk/~cpgl0080/
On Fri, Dec 18, 2015 at 11:00 AM, <fieldtrip-request at science.ru.nl> wrote:
Send fieldtrip mailing list submissions to
? ? ? ? fieldtrip at science.ru.nl
To subscribe or unsubscribe via the World Wide Web, visit
? ? ? ? http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
or, via email, send a message with subject or body 'help' to
? ? ? ? fieldtrip-request at science.ru.nl
You can reach the person managing the list at
? ? ? ? fieldtrip-owner at science.ru.nl
When replying, please edit your Subject line so it is more specific
than "Re: Contents of fieldtrip digest..."
Today's Topics:
? ?1. importing large edf data (Leila Ayoubian)
----------------------------------------------------------------------
Message: 1
Date: Fri, 18 Dec 2015 09:43:43 +0000 (UTC)
From: Leila Ayoubian <leilayou_54 at yahoo.com>
To: "fieldtrip at science.ru.nl" <fieldtrip at science.ru.nl>
Subject: [FieldTrip] importing large edf data
Message-ID:
? ? ? ? <807603762.476165.1450431823359.JavaMail.yahoo at mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"
Hi:Thanks for providing us with this amazing toolbox.
I am new to fieldtrip. I am trying to import EEG data in the format of *.edf for the purpose of seizure detection for epileptic patients. I can't break up the data as you understand the detection has to have continuous data. However I could load section of the data at a time and clear what is already used and reload again.?
So trying to load the data like this:
cfg.dataset= ('mydata.edf');
rawdata=ft_preprocessing(cfg);
This is the error message I get :
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Error using zeros
Requested 51x51927040 (19.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time
and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in read_edf (line 403)
? dat = zeros(length(chanindx),nepochs*epochlength);
Error in ft_read_data (line 622)
??? dat = read_edf(filename, hdr, begsample, endsample, chanindx);
Error in ft_preprocessing (line 566)
????? dat = ft_read_data(cfg.datafile, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', rawindx, 'checkboundary',
????? strcmp(cfg.continuous, 'no'), 'dataformat', cfg.dataformat);
Error in ReadFedfiles (line 8)
rawdata=ft_preprocessing(cfg);%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
I tried the ft_redefinetrial, but since the input to this file is the output from ft-PREPROCESSING it gives me the same error.
I also tried to use ft_read_data to choose the samples I want to load :?filename???? = 'mydata.edf'
?hdr?? = ft_read_header(filename);
?sample.start=1
?sample.end=100
dat = ft_read_data(filename,'sample.start','1','sample.end','100')
and here is again the error message:
Error using zeros
Requested 51x51927040 (19.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a
long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in read_edf (line 403)
? dat = zeros(length(chanindx),nepochs*epochlength);
Error in ft_read_data (line 622)
??? dat = read_edf(filename, hdr, begsample, endsample, chanindx);
Error in ReadFedfiles (line 31)
dat = ft_read_data(filename,'sample.start','1','sample.end','100')
?
>>
Could you please direct me as to which functions and in which order I should be using the functions to load data in smaller samples.Some examples would be useful.
Thanks again for your support and assistance.
We appreciate your effort.
Kind regards
___________________________________________________ Dr. Leila Ayoubian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20151218/6dfa7ce1/attachment-0001.html>
------------------------------
_______________________________________________
fieldtrip mailing list
fieldtrip at donders.ru.nl
http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20160505/61a9a8aa/attachment-0002.html>
More information about the fieldtrip
mailing list