[FieldTrip] Question about how to reduce the file size

[Robert Oostenveld]

Hi Casper,

Biosemi files are often problematic. The files themselves are 24 bit, which makes them efficient on disk (although though they are still huge on disk), but once in memory they take 64 bit per sample. So your 3GB becomes 8GB in memory, not accounting for any overhead. Depending on the analysis pipeline, it might well be that two copies of the data are needed in memory (so 16GB), plus further overhead.

Note that downampling requires that a low-pass filter is applied prior to downsampling to avoid aliassing (http://en.wikipedia.org/wiki/Aliasing). This happens automatically in ft_resampledata (look for cfg.resamplemethod and related comments in the code).

You can use a strategy like this

cfg1 = [];
cfg1.dataset = yourfilename;
cfg1 = ...
cfg1 = ft_definetrial(cfg1); % this part is optional, without it it results in continuous data in memory 

cfg2 = [];
cfg2.resamplefs = 500;

for i=1:nchan
  cfg1.channel = i; % you can use a number as well as a string
  temp = ft_preprocessing(cfg1);
  singlechan{i} = ft_resampledata(cfg2, temp);
  clear temp;
end % for all channels

data = ft_appenddata([], singlechan{:});

This reads and downsamples the data one channel at a time. 


best regards,
Robert



On 05 Jan 2015, at 17:02, Casper van Heck <caspervanheck at gmail.com> wrote:

> Dear Emilie,
> 
> I'm using a Windows-based wreck with 8GB ram and 1.5GB files, which never pops over an usage of 4GB, so I am a bit surprised that you're getting issues with your data. Also; reducing the sampling rate that much should reduce the memory footprint to something close to 400mb, at least, which to my mind should not produce issues of any kind. Could you post a bit more of your code?
> 
> I've been lowering the sampling rate too (5000 to 500), but I'm also cutting the data into smaller pieces, based on markers, effectively splitting the data into four parts, and throwing away more than 80%. Cutting the data into pieces can provide a workaround for memory issues. Detail: while I do filter (and some other details) before the resampling, and I'm only resampling due to time constraints, not crashing behaviour.
> 
> Also check the memory tutorial: fieldtrip.fcdonders.nl/tutorial/memory
> 
> Does this help?
> 
> Casper
> 
> On Fri, Dec 12, 2014 at 11:24 PM, Caspar, Emilie <e.caspar at ucl.ac.uk> wrote:
> Dear Fieltrippers, 
> 
> I did a pilot study on one participant today. Now that I'm trying to analyze my data, I realize that the size file is too big for my computer (size = 3Gb). Even after one hour, the filters (high pass + low pass) were not yet achieved. 
> 
> So I would like to see how to reduce the size of my sample BEFORE the filters.
> 
> I know that there is "ft_resampledata", and I did it to reduce the actual sample rate (= 2048) to 256. However, even with this procedure my computer is crashing (even with 16 Go RAM). In addition, I'm not sure that I can resample before filtering (I read different informations). 
> 
> Another way I was thinking about was to pre-select electrodes that I need (only 6 electrodes on 64). But here I have two questions:
> - Can I pre-process only some electrodes? Does it really reduce size for next preprocesses?
> - Is this the correct way to ask? As it crashes, not sure it works.
> 
>     cfg             = [];
>     cfg.dataset     = [ file.name];
>     cfg.channel     = 'B5', 'B6', 'B15', 'B16';
>     allData_prepross = ft_preprocessing(cfg);
>     cfg.resamplefs = 256;
>     DataSample = ft_resampledata(cfg, allData_prepross)
> 
> 
> I would appreciate pieces of advice!
> 
> Many thanks :)
> 
> Emilie
> 
> 
> 
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> 
> _______________________________________________
> fieldtrip mailing list
> fieldtrip at donders.ru.nl
> http://mailman.science.ru.nl/mailman/listinfo/fieldtrip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20150105/1972444f/attachment-0001.html>