[FieldTrip] delays on computing server

[Roy Cox]

hi Jan-Mathijs,

Yes, we're using Matlab's parallel computing toolbox, and our computing
server provides regular gui-based interactive Matlab (no job scheduling
etc). I've generally set up code such that a recording is processed by a
"recording-level" function, taking a custom cfg as input that specifies all
kinds of things that happen inside the function (e.g., filepaths, channels
to read, flags for rereferencing and filtering, and so on). Then there's a
"group-level" script that either uses a regular for loop across recordings
(allowing breakpoints), or a parfor loop with each of however many workers
you have taking care of one recording (at the expense of no/few debugging
options). As I'm sure you know, there's some tinkering required with
declaring variables to accommodate correct parfor behavior, but once set up
it's relatively easy to switch from serial to parallel and back if e.g.
recording 37 randomly fails and you need to step through the code.

Anyway, I had hoped to avoid the profiling route, also because parallel
profiling is yet another thing (mpiprofile), but I can look into it. Are
you sure there's no crude way to turn off all FT communication? If that
would work, at least I'd know that my issue is in fact related to FT (and
not something more general with Matlab and/or the server configuration).

Side note: I'm mostly using SleepTrip, but I see the same behavior when
using a recent version of FT.

Best,
Roy

On Fri, Mar 10, 2023 at 8:56 AM Schoffelen, J.M. (Jan Mathijs) via
fieldtrip <fieldtrip at science.ru.nl> wrote:

> Hoi Roy,
>
> The compute server sounds like an interesting use case. Are you using
> mathworks’ parallel server functionality for this? We have recently started
> exploring and documenting a bit the possibilities for using matlab’s
> parallel processing tools in FieldTrip, in collaboration with Mathworks and
> Vortech
> <https://urldefense.com/v3/__https://www.vortech.nl__;!!HJOPV4FYYWzcc1jazlU!-ld--Q_DRiwfhl22L94A2cbFEL4iDJa8LCyV2Ls36yIpVS1PQUwCeJr2qhHPVca3kmUF9i5SbYThi2-qSoZrd5IhA5p_cPqJGO001A$> .
> Please see
> https://urldefense.com/v3/__https://www.fieldtriptoolbox.org/tutorial/distributedcomputing_parfor/__;!!HJOPV4FYYWzcc1jazlU!_klyOEK25CIa-hZ6MJhzDhgAg55vCXGDlPRKfU5uRURFVHMhjX4U_YCbEu3Onol3LlwcKh1l_5as6kcfeQSeDd_Z_hGD$ 
> <https://urldefense.com/v3/__https://www.fieldtriptoolbox.org/tutorial/distributedcomputing_parfor/__;!!HJOPV4FYYWzcc1jazlU!-ld--Q_DRiwfhl22L94A2cbFEL4iDJa8LCyV2Ls36yIpVS1PQUwCeJr2qhHPVca3kmUF9i5SbYThi2-qSoZrd5IhA5p_cPp0n-bR4w$> for
> some initial documentation. We would be very happy to receive insights/use
> cases/best practices/tips ’n tricks from the FieldTrip users, to be
> documented on the website as well.
>
> Now, regarding to your question, I don’t have a pasklaar answer, but I
> would start with profiling the execution of the slow functions: this will
> give you an indication which lines take most time to execute, so that you
> can tweak (if possible) the ‘overhead’-options. Typing ‘help profile’ and
> ‘help profview’ should get you started.
>
> Met groet,
> Jan-Mathijs
>
>
> On 9 Mar 2023, at 15:07, Roy Cox via fieldtrip <fieldtrip at science.ru.nl>
> wrote:
>
> hello all,
>
> We recently started batching EEG analyses, involving FieldTrip code, on a
> computing server (essentially a big Windows workstation).
>
> Mostly great, but I've noticed that there are sometimes lengthy delays
> (tens of seconds) when executing a FieldTrip command.
>
> Possibly related, I sometimes see warnings like these:
>
> *Warning: could not send tracker information for "startup"*
>
> It's all very poorly reproducible: the exact same script operating on the
> same recording sometimes runs fast and sometimes takes forever.
>
> One reason I can think of is that this has something to do with FieldTrip
> trying to send some information to its developers for tracking purposes,
> but perhaps this isn't suitable for a computing server running many jobs in
> parallel.
>
> Supposing that's indeed the issue: is there a way to disable any and all
> communication attempts by FieldTrip to the outside world? I looked at
> ft_defaults but not sure which options to choose.
>
> And any other suggestions as to what might be going on are welcome of
> course.
>
> Best,
> Roy
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
>
> https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!64qKXsw6Gxwuj46geg5MTQV45TvFoLrpT77shzu8cOjjweyBCX-RYZ8FTM2DU9huMl1gG6E2Qmgp8XH-87zpgqpQhd8kZWkKZJBnNw$
>
>
> _______________________________________________
> fieldtrip mailing list
> https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
> https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!HJOPV4FYYWzcc1jazlU!_klyOEK25CIa-hZ6MJhzDhgAg55vCXGDlPRKfU5uRURFVHMhjX4U_YCbEu3Onol3LlwcKh1l_5as6kcfeQSeDQkEPnPV$ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20230310/d0052154/attachment.htm>