[FieldTrip] Inconsistency between stat.mask and p-values in ft_freqstatistics

[Schoffelen, J.M. (Jan Mathijs)]

Guillermo,

Is there any new information in this e-mail? I recall that you asked the same question, in more or less the same e-mail last week, and that one of our colleagues - Mikkel - has taken the time to answer it. I might have missed it, but I did not see any follow up on Mikkel’s reply on this list.

In contrast to the previous message, you now write: "Upon reviewing the results, I observed that stat.mask did not align with the p-values in stat.prob as expected. I anticipated that stat.mask would indicate values lower than cfg.alpha, but this was not consistent with the p-values. Additionally, I noticed that when stat.cirange contains values of "0", corresponding values of "1" appear in stat.mask, despite the p-value not being lower than cfg.alpha.”  This is still way to vague in order to give the readership any handles to be able to think along with you. Note, that we are all scientists that are keen to help each other out, but that this e-mail list is not a helpdesk.


I don’t understand well why
-you specify ‘montecarlo’ as method, but you don’t specify a number of randomizations
-you think that you need to look at the cirange in the output

Moreover, I don’t know whether
-it makes sense to do second-level statistics with only 3 subjects


But more generally: Have you reviewed the documentation on the fieldtrip website about how to properly conduct a statistical test in fieldtrip? The code that you pasted uses inconsistently datafftpre and data as input to ft_freqstatistics, and while you mention that datafftpre contains 3 subjects worth of data, the call to ft_freqanalysis that precedes in your pasted code example the call to ft_freqstatistics suggests otherwise.

Just as a general problem-solving strategy, after you have studied all documentation about performing statistical comparisons in FieldTrip, I suggest that you look up in the codebase the lines of code (in ft_statistics_montecarlo), where the mask (and cirange) are computed. This will most likely take most of the magic away. If you would do this, you would see that probably the only situation in which the mask,  p-values and alpha level do not relate to one another in a predictable way, is when cfg.correctm = ‘fdr’. But since you didn’t mention this in the code you pasted, it will probably not be the cause for your observed discrepancy (which to repeat myself is still not clear to the readership).

And now it’s time for BWV891.

Good luck,

Jan-Mathijs




On 25 Mar 2024, at 17:20, Guillermo Sanchez-Garrido Campos via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:


Hello FieldTrip team,

I've been utilizing the ft_freqstatistics function to conduct statistical analysis on my LFP data, recorded using the Neuropixels system. During my analysis, I encountered a discrepancy between the values in stat.mask and the p-values in stat.prob, which has led to confusion in interpreting the results.

Detailed Description:

Power Spectrum Calculation:

cfg = [];
cfg.method       = 'mtmfft';
cfg.output       = 'pow';
cfg.taper        = 'hanning';
cfg.foilim       = [1 100]; % Frequency range
cfg.polyremoval  = -1;
cfg.pad          = 'nextpow2';
datafftpre = ft_freqanalysis(cfg, datapre); % Same for post

Statistical Analysis Configuration:

cfg = [];
cfg.method           = 'montecarlo';
cfg.statistic        = 'ft_statfun_depsamplesT';
cfg.correctm         = 'no';
cfg.alpha            = 0.05;
cfg.tail             = 0;
cfg.correcttail      = 'prob';
cfg.computestat      = 'yes';
cfg.computeprob      = 'yes';
cfg.computecritval   = 'yes';

% Design configuration for dependent groups
nsubj = size(data, 1);
design = zeros(2, 2 * nsubj);
design(1, 1:nsubj)         = 1;
design(1, nsubj+1:2*nsubj) = 2;
design(2, :) = repmat(1:nsubj, 1, 2);

cfg.design = design;
cfg.ivar   = 1;
cfg.uvar   = 2;

stat = ft_freqstatistics(cfg, datafftpre, data);


Observations:

Upon reviewing the results, I observed that stat.mask did not align with the p-values in stat.prob as expected. I anticipated that stat.mask would indicate values lower than cfg.alpha, but this was not consistent with the p-values. Additionally, I noticed that when stat.cirange contains values of "0", corresponding values of "1" appear in stat.mask, despite the p-value not being lower than cfg.alpha.

Input Data:

My data is structured into two 3D matrices, datafftpre and datafftpost, each representing LFP recordings for 3 subjects (1st dimension), across 310 channels (rows), and 649 frequencies (columns). Each cell contains the power spectrum value for the corresponding subject, channel, and frequency.

I would greatly appreciate any guidance or suggestions on correctly interpreting the results of my analysis.

Thank you for your attention and support.

Warm regards,

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C02%7Cfieldtrip%40science.ru.nl%7C29eb0f0a8410484acd5e08dc4cef8cd7%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638469838931444997%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=tcOvWBxVpAKT1VDqv3yiV8yurbtXJ1xTxRaP%2FKGjDjk%3D&reserved=0>
https://doi.org/10.1371/journal.pcbi.1002202

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20240325/5dae8776/attachment.htm>