[FieldTrip] Run ft_prepare_vol_sens on multiple cores

[Richards, John]

Frans.

I was able to do this.  There are two places where there is an iterative part of the ft_prepare_leadfield.  I think one of these is the main program, and another is the sb_transfer.  I was able to recode that section and use parfor for the parallel processing.  We do 3mm hex source volumes for adults, or dense tetra source/head models, both which have something like 100K source voxels; and 2mm hex source models for infants, or dense tetra source/head models.  We run this on 48-core machines and it takes 90 min or so to do the LF.  I used to do these with the single-threaded matlab and it took something like 18 hrs. 

I don't remember offhand exactly which sections I did this in.  I have customized versions for the files below.  I would be glad to give you a further description if you are interested.

John 

./JERLab_ft_prepare_leadfield.m
./external/simbio/JERLab_sb_transfer.m
./forward/JERLab_ft_prepare_vol_sens_notransfer.m
./forward/JERLab_ft_prepare_vol_sens.m
./forward/private/JERLab_leadfield_simbio.m
Also in sb_solve I commented out the disp because they were distracting on the matlab output, but I don't think I changed this module.

It takes a few steps to upgrade to a new fieldtrip version because I have these extra programs.  The last time I upgraded was june 2021 to the fieldtrip-20210330 version.  These are the notes I used to upgrade, so they might be a good hint where to look for parallelization with MATLAB.
ERLab_ft_prepare_leadfield
  add computeparallel to the cfg file
  will do the parallel, OR, use the original code if computeparallel is not set
private/prepare_headmodel
 adds parfor based on computeparallel in the cfg
JERLab_ft_prepare_vol_sens, substitute JERLab_sb_transfer
external/simbio/JERLab_sb_transfer
  adds the parfor for the parallel pool
external/simbio/sb_solve 
  removed the disp's, so don't print during the parallel
  clears are ok in subroutines, but not in external parfor loop


This used to be ft_prepare_vol_sens_notransfer, but is a JERLab program, so rename.
it is used in private/prepare_headmodel
x forward/JERLab_ft_prepare_vol_sens_notransfer.m
 

***********************************************
John E. Richards
Carolina Distinguished Professor
Department of Psychology
University of South Carolina
Columbia, SC  29208
Dept Phone: 803 777 2079
Fax: 803 777 9558
Email: richards-john at sc.edu
https://jerlab.sc.edu
*************************************************


----------------------------------------------------------------------

Message: 1
Date: Wed, 13 Apr 2022 10:48:41 +0000
From: "Schoffelen, J.M. (Jan Mathijs)"
	<janmathijs.schoffelen at donders.ru.nl>
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Run ft_prepare_vol_sens on multiple cores
Message-ID: <34E8671E-BDFF-40B0-A8FC-51B6DE832330 at donders.ru.nl>
Content-Type: text/plain; charset="utf-8"

Hi Frans,

Perhaps the simbio folks who are reading along can comment on this?

Best wishes,
Jan-Mathijs


On 12 Apr 2022, at 17:51, Frans Nord via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Hello all,

In the article "The FieldTrip-SimBio pipeline for EEG forward modelling" (https://pubmed.ncbi.nlm.nih.gov/29580236/<https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-2af725325d4aef4b&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F29580236%2F__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34BCpR5Yg%24>) it is stated that they were able to reduce the computational time substantially when calling ft_prepare_vol_sens and ft_calculate_leadfield through running it on all 16 cores.

Looking at the code I would assume that it's the call to sb_transfer that takes a long time to compute but I don't know how to attempt any parallelizing there. Anyone who has any idea on how to do that?

Best regards and thank you for developing a great toolbox!
//Frans
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-6da7080247d95362&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34U9MFCNg%24