[FieldTrip] Warning OPENEDF: Physical Minimum larger than Maximum\n

Fri Nov 20 16:02:42 CET 2020

Hi Alex and Katarina,

According to the EDF standard, inverted physical min/max values are used to implement the negative-up convention in clinical neurophysiology: https://www.edfplus.info/specs/edfplus.html#:~:text=5.%20'Digital%20maximum'%20must%20be%20larger,by%200'%20errors%20by%20some%20viewers).

Therefore, it would appear (without further evidence to the contrary) that your data is ok, and the headers are simply indicating a negative gain.

If you're trying to only copy the data, I'd suggest doing it digitally (ie. without any conversion into physical units) using lower-level functions. This will ensure the data is not "altered". In our lab, we also have a similar use case where we extract only scalp electrodes during specified times into separate EDF files. We do this in Python using pyedflib, mainly because some of our other analysis code was already in Python, not for any particular advantage of the language or toolbox over Fieldtrip: https://gist.github.com/dtxe/a74bd43f4106281b86b9171bf5efe73d

I'm speculating the reason the data changes after the 2nd decimal place is due to rounding:

  1.  Data is stored in EDF format as 16-bit integers (ref)<https://www.edfplus.info/specs/edf.html#:~:text=each%20sample%20value%20is%20represented%20as%20a%202%2Dbyte%20integer>
  2.  On import, the data is converted into physical units using digital min/max, physical min/max
  3.  On export, the data is converted back into digital units
     *   The header field for physical min/max only accommodates 8 characters, hence lines 91-95
     *   Fieldtrip optimizes the dynamic range by using the physical signal's min/max, which is likely different than the original EDF file's physical min/max
     *   After converting back into digital units, the data is rounded to the nearest int16.

I hope this helps,

Simeon

[cid:2cc287ba-1b09-4751-9231-2940fe7f4f40]

Simeon Wong

Ibrahim Lab

Neurosciences & Mental Health

686 Bay Street, Toronto ON  M5G 0A4

P 416.813.7654  x303697

________________________________
From: fieldtrip <fieldtrip-bounces at science.ru.nl> on behalf of Lepauvre, Alex <alex.lepauvre at ae.mpg.de>
Sent: November 20, 2020 06:12
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Warning OPENEDF: Physical Minimum larger than Maximum\n

Dear Fieldtrip community,

Following on the messages below, I performed comparison between the saved signal and the original signal. To give a little more context, our single goal here is to load our EDF signal in fieldtrip to then chop it into pieces to be able to save each piece separately. We are required to do so as we are having very long EDFs and because we have to format our data in these specific pieces for data ingestion in XNAT.

The written function (see attached) loads the EDF of interest and using a photodiode trigger channel, we parse our signal in “blocks”, enabling the user of the code to adjust detected onsets manually to account for misdetection by our algorithm. The EDF signal is cut from onsets to onsets and stored in separated files. We need to make sure that this procedure doesn’t alter our raw signal in any ways. To check whether this happens, following the data saving, the saved data are being reloaded and compared to the original data that were parsed. If the comparison yields anything else than 0s, then it means that the data were altered in some way.

When running the function, the first thing I notice is that following saving, the size of the signal.trial{1,1} matrices has been changed. Following some digging in the edf write function, I noticed that the data are being saved to the EDF in 1 seconds chunks, while it is not in our original signal (0.625seconds instead). I assume that this is not a big deal as this doesn’t really alter the data. However, what I believe is problematic is that the signal gets altered after the second decimal place across all channels. I am assuming that this has something to do with the conversion to uint16? I am a bit puzzled about why this is the case and how we can get around this issue to save our data and keep them pristine. Has anyone been encountering similar issues and/or knows a way?

Kind regards,

Alex

From: fieldtrip <fieldtrip-bounces at science.ru.nl> On Behalf Of Lepauvre, Alex
Sent: Thursday, 19 November 2020 14:01
To: FieldTrip discussion list <fieldtrip at science.ru.nl>
Subject: Re: [FieldTrip] Warning OPENEDF: Physical Minimum larger than Maximum\n

Dear Simeon, deaf Fieldtrip community

Thanks a lot for your helpful feedback. To give you a little bit more context, our EDFs are from iEEG data.  I have been digging a little bit more into this issue with our data. First of all, it turns out that the warning isn’t thrown by the opnedEDF function, but by the read_edf.m one. It seems that in our files, the accessed PhysMin and PhysMax are reversed. Indeed, most values in the PhysMax are negative (-8711uV, except for some misc channels), while the PhysMin is the exact opposite (+8711), which seems to be the prime reason for this error.  I am not sure why that is the case. According to what you were saying @Simeon, this might be due to our EDF export software doing something odd with the headers? Can this harm our data?

Also, another aspect I am confused about is that we are currently using fieldtrip to chop our single big EDFs into smaller more manageable files. We therefore load the data triggering the warning, then chop them and adjust the headers, and save the chunks separately. However, when reloading the newly saved data, the warning isn’t there anymore, which has me worried that the data have been altered in some way.

After digging in the function write_edf a little bit more, I am supposing that the issue doesn’t appear anymore because fieldtrip compute the max and min amplitude of the signal, and fill it in in the PhyMax and PhysMin headers itself, correct?

Finally, another issue we are facing is that for some of our EDF files, the write_EDF function doesn’t work with some of our EDFs (I am unfortunately not allowed to share any data due to strong data privacy policies In our project, my apologies). While for some of our files it did work, for at least one, the write_edf returns the following error:

[cid:image003.jpg at 01D6BF33.9D87D350]

I again dived into the function write_edf and located the issue here:

[cid:image005.jpg at 01D6BF33.9D87D350]

I am not entirely sure what the purpose of these statements is. It seems to reduce the decimal places of the physical minimum (and then maximum) to find which approximation fits the expected size (8*nchans). However, none of the 5 options fit. Surprisingly enough, going one more would work: in my signal %-8.2g fits exactly nchans*8. I am not sure why that is the case nor what it would entail for my data if I were to change it in the code myself. Could someone enlighten me?

From: fieldtrip <fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl>> On Behalf Of Katarina Bendtz
Sent: Wednesday, 18 November 2020 20:20
To: FieldTrip discussion list <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Subject: Re: [FieldTrip] Warning OPENEDF: Physical Minimum larger than Maximum\n

Dear Simeon!

Thank you so much for your quick reply! I understand!

Do you think I can safely ignore this or should I try to find out which channels this applies to? That is, if the trigger channels are set to identical values, will this in some way compromise/change the data?

best,

Kat

Katarina Bendtz, PhD
Research Fellow
Kreiman Lab
Boston Children's Hospital
Harvard University

+1 (617) 390 45 16
3 Blackfan Circle
Boston, USA

On Nov 18, 2020, at 1:22 PM, Simeon Wong <simeon.wong at sickkids.ca<mailto:simeon.wong at sickkids.ca>> wrote:

Hi Katarina,

The fread command advances the file pointer. Therefore the 4 lines are reading consecutive values from the EDF file.

https://www.mathworks.com/help/matlab/ref/fread.html?searchHighlight=fread&s_tid=doc_srchtitle#d122e421874:~:text=and%20positions%20the%20file%20pointer%20after%20the%20last%20value%20read<https://urldefense.com/v3/__https://www.mathworks.com/help/matlab/ref/fread.html?searchHighlight=fread&s_tid=doc_srchtitle*d122e421874:*:text=and*20positions*20the*20file*20pointer*20after*20the*20last*20value*20read__;I34lJSUlJSUlJSU!!D0zGoin7BXfl!r4QJaRIZ-3FdVMHleuLbLlEDUz6WEdLbbNCQa6ENyxDrfsWOU5abThVtTfOyf5ZSROzo$>

With that being said, some EDF export software (for better or for worse) set identical physical values if channels are switched off or for channels without physical units (ie. trigger channels or misc channels).

All the best,

Simeon

<Outlook-gb44xqtx.png>

Simeon Wong

Ibrahim Lab

Neurosciences & Mental Health

686 Bay Street, Toronto ON  M5G 0A4

P 416.813.7654  x303697

________________________________

From: fieldtrip <fieldtrip-bounces at science.ru.nl<mailto:fieldtrip-bounces at science.ru.nl>> on behalf of Katarina Bendtz <katarina.bendtz at gmail.com<mailto:katarina.bendtz at gmail.com>>
Sent: November 18, 2020 12:50
To: fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl> <fieldtrip at science.ru.nl<mailto:fieldtrip at science.ru.nl>>
Subject: [FieldTrip] Warning OPENEDF: Physical Minimum larger than Maximum\n

Dear experts:

Me and my colleague at another university idependently I get this warning when we try to open an edf file:

Warning OPENEDF: Physical

Minimum larger than Maximum\n

I’m using fieldtrip 2020113. Looking at the source code

https://github.com/fieldtrip/fieldtrip/blob/master/private/openedf.m<https://urldefense.com/v3/__https:/github.com/fieldtrip/fieldtrip/blob/master/private/openedf.m__;!!D0zGoin7BXfl!tHOzBhC87IDZICVPmzh1QeJQI1pBz7wzlyRm7uCUqe_WlgHxjVh2sShWYk0yjAhO9UAo$>

I read:

if (any(EDF.PhysMin >= EDF.PhysMax))

fprintf(2,'Warning OPENEDF: Physical Minimum larger than Maximum\n');

EDF.PhysMin = EDF.DigMin;

        EDF.PhysMax = EDF.DigMax;

end

but then a couple of lines up, I see both EDF.PhysMin and EDF.PhysMax, (and also EDF.DigMin and EDF.DigMax) being set to the same thing:

EDF.PhysMin= str2num(char(fread(EDF.FILE.FID,[8,EDF.NS],'char')'));

EDF.PhysMax= str2num(char(fread(EDF.FILE.FID,[8,EDF.NS],'char')'));

EDF.DigMin = str2num(char(fread(EDF.FILE.FID,[8,EDF.NS],'char')')); %

EDF.DigMax = str2num(char(fread(EDF.FILE.FID,[8,EDF.NS],'char')')); %

So it is quite reasonable that I will get this warning. I’m puzzled.

1. What’s purpose of this warning?

2. Why are the variables set to the same value?

3. Can I safely ignore this warning?

4. Or is there something I misunderstood.

Thanks a lot,

Best,

Kat

Katarina Bendtz, PhD
Research Fellow
Kreiman Lab
Boston Children's Hospital
Harvard University

+1 (617) 390 45 16
3 Blackfan Circle
Boston, USA

________________________________

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.
_______________________________________________
fieldtrip mailing list
https://mailman.science.ru.nl/mailman/listinfo/fieldtrip<https://urldefense.com/v3/__https://mailman.science.ru.nl/mailman/listinfo/fieldtrip__;!!D0zGoin7BXfl!r4QJaRIZ-3FdVMHleuLbLlEDUz6WEdLbbNCQa6ENyxDrfsWOU5abThVtTfOyf5sM8xsl$>
https://doi.org/10.1371/journal.pcbi.1002202<https://urldefense.com/v3/__https://doi.org/10.1371/journal.pcbi.1002202__;!!D0zGoin7BXfl!r4QJaRIZ-3FdVMHleuLbLlEDUz6WEdLbbNCQa6ENyxDrfsWOU5abThVtTfOyfw9x0Kj7$>

________________________________

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20201120/53c652af/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 69416 bytes
Desc: image003.jpg
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20201120/53c652af/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 94776 bytes
Desc: image005.jpg
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20201120/53c652af/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-53mcdynt.png
Type: image/png
Size: 1756 bytes
Desc: Outlook-53mcdynt.png
URL: <http://mailman.science.ru.nl/pipermail/fieldtrip/attachments/20201120/53c652af/attachment-0001.png>