[FieldTrip] Artifact rejection problem due to samples occurring twice in the data

Wed Nov 1 09:14:04 CET 2023

Dear Maria,

Ah, that’s too bad. I realize that I misread the code a little bit. What seems to be missing from the call to ft_fetch_data (in ft_artifact_threshold), is the option to pass on the possibility to specify ‘allowoverlap’ to be anything else than what is the default in ft_fetch_data (which is false). Therefore, with the current code, your use case is not going to work.

What you could do, however, is to call ft_artifact_threshold without a data structure as a second input argument. If you specify the dataset, an appropriate ’trl’ matrix (could be different from the one that defines the eventual epochs of interest, in your specific case perhaps indicating the starts and ends of the individual sentences or so), and optionally some preprocessing settings (for filtering, in cfg.artfctdef.threshold), then the output of ft_artifact_threshold can be used to clean the data structure as per the call to ft_rejectartifact that you pasted in your earlier e-mail.

I hope this helps, otherwise feel free to drop by if you’re on campus.

BW,
JM

On 31 Oct 2023, at 08:47, Hartog, M.M.J. den (Maria) via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>> wrote:

Dear Jan-Mathijs,

Thank you for your reply.

The error is indeed thrown by ft_artifact_threshold. Unfortunately, specifying cfg.continuous = 'yes' does not solve the problem. The error message after this change is the same as before:

Error using ft_fetch_data
some of the requested samples occur twice in the data and have conflicting values

Error in ft_artifact_threshold (line 186)
    dat = ft_fetch_data(data,        'header', hdr, 'begsample', trl(trlop,1), 'endsample', trl(trlop,2), 'chanindx', chanindx, 'checkboundary', strcmp(cfg.continuous, 'no'));

Is there anything else I can try?

Best,
Maria

------------------------------------------------------------------------

Date: Fri, 27 Oct 2023 08:04:42 +0000
From: "Schoffelen, J.M. (Jan Mathijs)"
        <janmathijs.schoffelen at donders.ru.nl<mailto:janmathijs.schoffelen at donders.ru.nl>>
To: FieldTrip discussion list <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Subject: Re: [FieldTrip] Artifact rejection problem due to samples
        occurring twice in the data
Message-ID: <244FB7A1-D7BD-4DDE-9B5A-486E940B4259 at donders.ru.nl<mailto:244FB7A1-D7BD-4DDE-9B5A-486E940B4259 at donders.ru.nl>>
Content-Type: text/plain; charset="utf-8"

Dear Maria,

Thanks for your clear message and diagnostics so far. Just for the clear picture: the error is thrown by ft_artifact_threshold, isn’t it? If I read through the code of this function, I get the impression that you might get yourself back into the game, if you specify cfg.continuous = ‘yes’ for the cfg into ft_artifact_threshold. Although, this feels a bit at odds with the representation of the data that you give as an input to the function, because the time-axes of the individual trials clearly is not continuous. However, the sampelinfo field of the data object is still consistent with a continuous data representation (on disk). It’s the sampleinfo field that ft_fetch_data uses under the hood to determine the overlapping samples. I think that the error will convert into a warning with cfg.continuous = ‘yes’; (unless you have already tried this and can disconfirm my hunch). So, long story short, I’d recommend cfg.continuous = ‘yes’.

Best wishes,
Jan-Mathijs

------------------------------------------------------------------------

On 26 Oct 2023, at 11:31, Hartog, M.M.J. den (Maria) via fieldtrip <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl><mailto:fieldtrip at science.ru.nl>> wrote:

Dear community,

My name is Maria den Hartog and I am a PhD candidate at the Centre for Language Studies (CLS) at Radboud University in Nijmegen. I am currently preprocessing data from an ERP reading experiment about the processing of pronouns of address (e.g. he, she, they) within a text.

I want to use automatic artifact rejection to remove full trials with signal exceeding +-100 µV as part of my preprocessing pipeline, but I get an error message indicating that 'some of the requested samples occur twice in the data and have conflicting values'. This is true, and it is a consequence of the naturalistic reading paradigm I used. My trials run from -200 ms prestimulus to 1000 ms poststimulus. The t = 0 of each trial is the onset of a pronoun within a text, and the pronouns can occur less than 1000 ms after each other (e.g., the stimuli are he and his in 'He looked at his cat.'). I have baselined my trials to the 200 ms window up to the stimulus, meaning the same sample can have different values in different trials.

The code I use for artifact rejection is:
    cfg = [];
    cfg.continuous = 'no';
    cfg.artfctdef.threshold.min = -100;
    cfg.artfctdef.threshold.max = 100;
    art = ft_artifact_threshold(cfg,data);

    cfg = [];
    cfg.artfctdef.xxx.artifact = art.artfctdef.threshold.artifact;
    data_forERP = ft_rejectartifact(cfg,data);

The error I get:

Error using ft_fetch_data
some of the requested samples occur twice in the data and have conflicting values

Error in ft_artifact_threshold (line 186)
    dat = ft_fetch_data(data,        'header', hdr, 'begsample', trl(trlop,1), 'endsample', trl(trlop,2), 'chanindx', chanindx, 'checkboundary', strcmp(cfg.continuous, 'no'));

Is there a way to adjust the cfg for ft_rejectartifact so that the function will accept that samples do indeed occur twice in the data and that the same sample can have different values in different trials (and data from the correct trial will be fetched)?

Here are a few solutions that I have tried based on other discussion list questions, that I have found to be unsatisfactory or ineffective:

  *   For ft_databrowser, it is possible to use cfg.allowoverlap = 'yes' to avoid errors due to conflicting sample values, but this setting is not available for ft_rejectartifact.
  *   I do not want to remove or adjust the sampleinfo from the data, because I want to be able to relate trials to the original samples if necessary.
  *   I have found that downsampling my signal will lead ft_rejectartifact to reconstruct the sampleinfo assuming that the trials are consecutive segments of a continuous recording, but I do not want to 'hack' my way around the problem in this way.

Hope you can help me out!

Best,
Maria

_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.science.ru.nl%2Fmailman%2Flistinfo%2Ffieldtrip&data=05%7C01%7Cfieldtrip%40science.ru.nl%7C0a324d0066ee4f023f6a08dbdab28372%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638344232470203562%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nvrtRAMUxvtm%2B5%2FmFGB0OF6V%2BQbQM29V9fIvvSWU%2BCg%3D&reserved=0>
https://doi.org/10.1371/journal.pcbi.1002202<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1371%2Fjournal.pcbi.1002202&data=05%7C01%7Cfieldtrip%40science.ru.nl%7C0a324d0066ee4f023f6a08dbdab28372%7C084578d9400d4a5aa7c7e76ca47af400%7C1%7C0%7C638344232470359829%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fOhsrhstklJOQ6Nfg5BYzWaCHG6cWcri7DFPjyEJseU%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20231101/1b3cd8af/attachment.htm>