<div dir="ltr">Thank you John, that helped a lot! <div><br></div><div>Using a parfor in the sb_transfer function seems to provide the main reduction in computation time as far as I can see. Have you looked into adding a parfor in the for loops calling ft_compute_leadfield as well? Seems like it could possibly reduce the computation time even further.</div><div><br></div><div>Best regards</div><div>//Frans</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 14, 2022 at 6:56 PM Richards, John via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl">fieldtrip@science.ru.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Frans.<br>
<br>
I was able to do this. There are two places where there is an iterative part of the ft_prepare_leadfield. I think one of these is the main program, and another is the sb_transfer. I was able to recode that section and use parfor for the parallel processing. We do 3mm hex source volumes for adults, or dense tetra source/head models, both which have something like 100K source voxels; and 2mm hex source models for infants, or dense tetra source/head models. We run this on 48-core machines and it takes 90 min or so to do the LF. I used to do these with the single-threaded matlab and it took something like 18 hrs. <br>
<br>
I don't remember offhand exactly which sections I did this in. I have customized versions for the files below. I would be glad to give you a further description if you are interested.<br>
<br>
John <br>
<br>
./JERLab_ft_prepare_leadfield.m<br>
./external/simbio/JERLab_sb_transfer.m<br>
./forward/JERLab_ft_prepare_vol_sens_notransfer.m<br>
./forward/JERLab_ft_prepare_vol_sens.m<br>
./forward/private/JERLab_leadfield_simbio.m<br>
Also in sb_solve I commented out the disp because they were distracting on the matlab output, but I don't think I changed this module.<br>
<br>
It takes a few steps to upgrade to a new fieldtrip version because I have these extra programs. The last time I upgraded was june 2021 to the fieldtrip-20210330 version. These are the notes I used to upgrade, so they might be a good hint where to look for parallelization with MATLAB.<br>
ERLab_ft_prepare_leadfield<br>
add computeparallel to the cfg file<br>
will do the parallel, OR, use the original code if computeparallel is not set<br>
private/prepare_headmodel<br>
adds parfor based on computeparallel in the cfg<br>
JERLab_ft_prepare_vol_sens, substitute JERLab_sb_transfer<br>
external/simbio/JERLab_sb_transfer<br>
adds the parfor for the parallel pool<br>
external/simbio/sb_solve <br>
removed the disp's, so don't print during the parallel<br>
clears are ok in subroutines, but not in external parfor loop<br>
<br>
<br>
This used to be ft_prepare_vol_sens_notransfer, but is a JERLab program, so rename.<br>
it is used in private/prepare_headmodel<br>
x forward/JERLab_ft_prepare_vol_sens_notransfer.m<br>
<br>
<br>
***********************************************<br>
John E. Richards<br>
Carolina Distinguished Professor<br>
Department of Psychology<br>
University of South Carolina<br>
Columbia, SC 29208<br>
Dept Phone: 803 777 2079<br>
Fax: 803 777 9558<br>
Email: <a href="mailto:richards-john@sc.edu" target="_blank">richards-john@sc.edu</a><br>
<a href="https://jerlab.sc.edu" rel="noreferrer" target="_blank">https://jerlab.sc.edu</a><br>
*************************************************<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 13 Apr 2022 10:48:41 +0000<br>
From: "Schoffelen, J.M. (Jan Mathijs)"<br>
<<a href="mailto:janmathijs.schoffelen@donders.ru.nl" target="_blank">janmathijs.schoffelen@donders.ru.nl</a>><br>
To: FieldTrip discussion list <<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a>><br>
Subject: Re: [FieldTrip] Run ft_prepare_vol_sens on multiple cores<br>
Message-ID: <<a href="mailto:34E8671E-BDFF-40B0-A8FC-51B6DE832330@donders.ru.nl" target="_blank">34E8671E-BDFF-40B0-A8FC-51B6DE832330@donders.ru.nl</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Frans,<br>
<br>
Perhaps the simbio folks who are reading along can comment on this?<br>
<br>
Best wishes,<br>
Jan-Mathijs<br>
<br>
<br>
On 12 Apr 2022, at 17:51, Frans Nord via fieldtrip <<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a><mailto:<a href="mailto:fieldtrip@science.ru.nl" target="_blank">fieldtrip@science.ru.nl</a>>> wrote:<br>
<br>
Hello all,<br>
<br>
In the article "The FieldTrip-SimBio pipeline for EEG forward modelling" (<a href="https://pubmed.ncbi.nlm.nih.gov/29580236/" rel="noreferrer" target="_blank">https://pubmed.ncbi.nlm.nih.gov/29580236/</a><<a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-2af725325d4aef4b&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F29580236%2F__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34BCpR5Yg%24" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-2af725325d4aef4b&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F29580236%2F__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34BCpR5Yg%24</a>>) it is stated that they were able to reduce the computational time substantially when calling ft_prepare_vol_sens and ft_calculate_leadfield through running it on all 16 cores.<br>
<br>
Looking at the code I would assume that it's the call to sb_transfer that takes a long time to compute but I don't know how to attempt any parallelizing there. Anyone who has any idea on how to do that?<br>
<br>
Best regards and thank you for developing a great toolbox!<br>
//Frans<br>
_______________________________________________<br>
fieldtrip mailing list<br>
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" target="_blank">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-6da7080247d95362&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34U9MFCNg%24" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-31327df5-454455534531-6da7080247d95362&q=1&e=57cfd234-1f5c-425d-a911-de4097b63ba0&u=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202__%3B%21%21HJOPV4FYYWzcc1jazlU%214B2vOR7eEquYu23rZZqYB7aXimMxJhGdqndb2U4xGpnQc91i_jchiCZIl946JbymXnDbqWfKxg16WtwFDXZTXNI-VKqad34U9MFCNg%24</a><br>
<br>
<br>
<br>
_______________________________________________<br>
fieldtrip mailing list<br>
<a href="https://mailman.science.ru.nl/mailman/listinfo/fieldtrip" rel="noreferrer" target="_blank">https://mailman.science.ru.nl/mailman/listinfo/fieldtrip</a><br>
<a href="https://doi.org/10.1371/journal.pcbi.1002202" rel="noreferrer" target="_blank">https://doi.org/10.1371/journal.pcbi.1002202</a><br>
</blockquote></div>