[FieldTrip] Query regarding stat.prob and permutation tests

Mon Aug 11 13:36:04 CEST 2025

Hi Cihan,

Thanks for looking into this in such an amount of detail. To me, this functional behavior does not make sense (i.e. the overwriting of the non-parametrically derived p-value).

Would you mind raising this as an issue on the fieldtrip github repository https://github.com/fieldtrip/fieldtrip<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffieldtrip%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Ca7d2bae8b7d244aa71d508ddd8cb415c%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638905089661508669%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=E%2BYy%2FBXhh1wR9B4eakdExJHvGXw%2FKeqInB1Com3hmu8%3D&reserved=0>, and copy over the relevant information from this e-mail thread?

I did not write the original code, so it could be that the person who wrote this had a particular reason to have the code behave as it currently does, or it is a genuine ‘bug’.

Best wishes and thanks,

Jan-Mathijs

On 8 Aug 2025, at 12:21, Cihan Dogan <cdogan1994 at hotmail.co.uk> wrote:

You don't often get email from cdogan1994 at hotmail.co.uk<mailto:cdogan1994 at hotmail.co.uk>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hi Jan,

Of course – here’s my setup:

    cfg = [];
    cfg.latency = TIME_WINDOW;
    cfg.channel = 'eeg';
    cfg.statistic = 'ft_statfun_depsamplesT';
    cfg.method = 'montecarlo';
    cfg.correctm = 'cluster';
    cfg.neighbours = neighbours;
    cfg.clusteralpha = 0.025;
    cfg.numrandomization = n_permutations;
    cfg.tail = 0;
    cfg.design = design_matrix;
    cfg.computeprob = 'no'; <<<<< only factor I change
    cfg.alpha = 0.05;
    cfg.correcttail = 'alpha';
    cfg.clusterthreshold = 'nonparametric_common';
    cfg.uvar = 1;
    cfg.ivar = 2;
    stat = ft_timelockstatistics(cfg, group_one{:}, group_two{:});

    probs = stat.prob < if computeprob = ‘Yes’ p-values are much lower on average, when compared to computeprob = ‘No’.

I then had a look through ft_statfun_depsamplesT<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffieldtrip%2Ffieldtrip%2Fblob%2Fmaster%2Fstatfun%2Fft_statfun_depsamplesT.m%23L128&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Ca7d2bae8b7d244aa71d508ddd8cb415c%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638905089661523671%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=5Cx9T5MLvXo1PjWrPSkvaTOiJfrIhy4Fok15zqljHFU%3D&reserved=0> function, and it seems what actually happens is when computeprob is True, it calculates p-values from this parametric test (line 128). Then at the very end of the ft_statistics_montcarlo<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffieldtrip%2Ffieldtrip%2Fblob%2Fmaster%2Fft_statistics_montecarlo.m%23L528&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Ca7d2bae8b7d244aa71d508ddd8cb415c%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638905089661537979%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=FaWv%2FYeHzNs%2FZhOTtYLiq9%2FH7jBDYpg0Iia4pxqKsIc%3D&reserved=0> (line 528) if computeprob was set to true, the parametric two sample p-values are copied over the nonparametric p-values and returned.

I have highly skewed non-gaussian data, therefore I think this is why the p-values in stat.prob are much lower across time and space when computeprob is True, vs. when it is not.

I just wanted to confirm if my understanding was correct.

Thank you,
Cihan
________________________________
From: fieldtrip <fieldtrip-bounces at science.ru.nl> on behalf of Schoffelen, J.M. (Jan Mathijs) via fieldtrip <fieldtrip at science.ru.nl>
Sent: Friday, August 8, 2025 11:40
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Cc: Schoffelen, J.M. (Jan Mathijs) <janmathijs.schoffelen at donders.ru.nl>
Subject: Re: [FieldTrip] Query regarding stat.prob and permutation tests

Hi Cihan,

Difficult to comment on this. Perhaps some example code and data might be helpful to demonstrate.

On 8 Aug 2025, at 10:53, cxd425--- via fieldtrip <fieldtrip at science.ru.nl> wrote:

Dear Jan,

Thank you for the response, this is helpful.

May I ask then, if cfg.method=‘montecarlo’ and cfg.computeprob=True the p-values, are much lower in stat.prob compared to when cfg.method=‘montecarlo’ and cfg.computeprob=False – they are mostly 1s when computeprob is False. I’m not referring to the cluster p-values (2nd level), but more so the sample (1st) level p-values.

Thank you in advance for the help,
Cihan
________________________________
From: fieldtrip <fieldtrip-bounces at science.ru.nl> on behalf of Schoffelen, J.M. (Jan Mathijs) via fieldtrip <fieldtrip at science.ru.nl>
Sent: Wednesday, August 6, 2025 12:26 pm
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Cc: Schoffelen, J.M. (Jan Mathijs) <janmathijs.schoffelen at donders.ru.nl>
Subject: Re: [FieldTrip] Query regarding stat.prob and permutation tests

Hi Cihan,

cfg.computeprob is an option that is used by (some of) the ’statfuns’, to convert a test-statistic with a known shape into a p-value, based on the shape of the parameter, (e.g. a T or F statistic).

The computational algorithm with which the p-value is computed (+returned in the output of a high level function call to ft_timelock/freq/sourcestatistics) depends on a combination of the contents of cfg.method (indeed if you would use cfg.method=‘montecarlo’, then the returned p-values are based on a permutation test, if you would  use cfg.method=‘analytic’ the p-values will be based on the known distributions), and cfg.correctm.

I  hope that this helps,

best wishes,
Jan-Mathijs

On 31 Jul 2025, at 17:03, cxd425--- via fieldtrip <fieldtrip at science.ru.nl> wrote:

Hi FieldTrip community,

I hope you’re doing well.

I was just curious how the p-values are computed at the first level - I couldn’t find any documentation explaining the behaviour of stat.prob when cfg.computeprob is set to true or false. When I set computeprob = false, the resulting stat object still contains p-values. Could it be that when cfg.computeprob = true, the p-values at the first level are computed using a permutation test, whereas when it’s set to false, a simpler non-permutation statistical test (e.g., a two-sample t-test) is used instead?

Any clarification would be greatly appreciated.

Best regards,
Cihan

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Ca7d2bae8b7d244aa71d508ddd8cb415c%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638905089661552865%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UYVYAqwQvLPe5cgkgQk3LutfAT7Ka5ogOw4qUtzVXus%3D&reserved=0>
https://doi.org/10.1371/journal.pcbi.1002202<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C02%7Cfieldtrip%40science.ru.nl%7Ca7d2bae8b7d244aa71d508ddd8cb415c%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638905089661566000%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UhPcZpuf7TKqov0si6B%2BUjPEmZuTR0b1ejHlMy3XEa4%3D&reserved=0>

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20250811/3bc2da1d/attachment.htm>