Support the Arctic Sea Ice Forum and Blog

Author Topic: Arctic Penguin (Wipneus) data files contain "nan" values  (Read 800 times)

Alphabet Hotel

  • Frazil ice
  • Posts: 444
    • View Profile
  • Liked: 175
  • Likes Given: 162
Arctic Penguin (Wipneus) data files contain "nan" values
« on: June 29, 2021, 08:18:34 AM »
I downloaded the file nsidc_arc_nt_detail.txt from the Arctic Penguin site and I've been loading it into a database. I just ran into a problem converting some of the values into numbers: 43 rows contain the string "     nan" instead of numbers ("nan" means "not a number"?) I guess I can just convert these into nulls (or drop these rows entirely) but is the data really unavailable?

$ grep nan nsidc_arc_nt_detail.txt | tail -3
1988-01-12 12:00,  12.00,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan
2021-02-20 12:00,  51.75,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan
2021-02-21 12:00,  52.75,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan,     nan
« Last Edit: June 29, 2021, 02:28:39 PM by kassy »

oren

  • First-year ice
  • Posts: 7661
    • View Profile
  • Liked: 3001
  • Likes Given: 2952
Re: Arctic Penguin extent/area files have illegal values
« Reply #1 on: June 29, 2021, 08:24:21 AM »
You can write a PM to Wipneus to make sure, but I think these are dates with no available data.

oren

  • First-year ice
  • Posts: 7661
    • View Profile
  • Liked: 3001
  • Likes Given: 2952
Re: Arctic Penguin extent/area files have illegal values
« Reply #2 on: June 29, 2021, 08:27:21 AM »
The images say no data - but there is an update of the spreadsheets.
With some hesitation - here is the update with data to 22nd Feb


Better yet, check the NSIDC files directly, maybe the data exists there. Feb 20-21 this year was an outage but maybe they restored the data later.

oren

  • First-year ice
  • Posts: 7661
    • View Profile
  • Liked: 3001
  • Likes Given: 2952
Re: Arctic Penguin extent/area files have illegal values
« Reply #3 on: June 29, 2021, 08:34:08 AM »
This is my most recent link for the NSIDC files, but IIRC you need some ftp software now to download. This is for the regional file but I assume the ftp directory has all the other files as well.

ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/seaice_analysis/Sea_Ice_Index_Regional_Daily_Data_G02135_v3.0.xlsx

SimonF92

  • Grease ice
  • Posts: 523
    • View Profile
  • Liked: 200
  • Likes Given: 86
Re: Arctic Penguin extent/area files have illegal values
« Reply #4 on: June 29, 2021, 08:48:42 AM »
Also, dont drop the rows as this could lead to index mismatching..Nans will help keep the data in the correct shape. Most commonly used data interpretation tools can handle nans.
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

Alphabet Hotel

  • Frazil ice
  • Posts: 444
    • View Profile
  • Liked: 175
  • Likes Given: 162
Re: Arctic Penguin extent/area files have illegal values
« Reply #5 on: June 29, 2021, 10:14:02 AM »
Also, dont drop the rows as this could lead to index mismatching..Nans will help keep the data in the correct shape. Most commonly used data interpretation tools can handle nans.

SQL wants nulls, but otherwise I see what you mean. I'll keep the empty values.

gerontocrat

  • Multi-year ice
  • Posts: 12938
    • View Profile
  • Liked: 4426
  • Likes Given: 34
Re: Arctic Penguin extent/area files have illegal values
« Reply #6 on: June 29, 2021, 12:12:19 PM »
I downloaded the file nsidc_arc_nt_detail.txt from the Arctic Penguin site and I've been loading it into a database. I just ran into a problem converting some of the values into numbers: 43 rows contain the string "     nan" instead of numbers ("nan" means "not a number"?) I guess I can just convert these into nulls (or drop these rows entirely) but is the data really unavailable?

is the data really unavailable? Yes. I believe there was a problem with the satellites and took a long time for NSIDC to sort out. Change of satellite and recalibration for consistent data?

ps: Wipneus uses a diffrent map from NSIDC to define the boundaries of the central arctic seas. This affects the CAB, the Beaufort, the Chukchi the ESS, the Laptev, and to a lesser extent the Kara and CAA.
« Last Edit: June 29, 2021, 12:28:43 PM by gerontocrat »
"Para a Causa do Povo a Luta Continua!"
"And that's all I'm going to say about that". Forrest Gump
"Damn, I wanted to see what happened next" (Epitaph)

gerontocrat

  • Multi-year ice
  • Posts: 12938
    • View Profile
  • Liked: 4426
  • Likes Given: 34
Re: Arctic Penguin extent/area files have illegal values
« Reply #7 on: June 29, 2021, 12:24:05 PM »
This is my most recent link for the NSIDC files, but IIRC you need some ftp software now to download. This is for the regional file but I assume the ftp directory has all the other files as well.

ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/seaice_analysis/Sea_Ice_Index_Regional_Daily_Data_G02135_v3.0.xlsx
I have found that if using Google I go to https://nsidc.org/arcticseaicenews/sea-ice-tools/ and then scroll down to the list of files all I need to do is click on the files(s) I want and they download without a problem. A few months ago it didn't work anymore but now it does.

The other way is to copy that ftp address directly into Windows Explorer (not Internet Explorer) and hit enter. The file list comes up, choose your files, right click and choose copy to folder. It works though Microsoft say they have dumped supporting ftp.



"Para a Causa do Povo a Luta Continua!"
"And that's all I'm going to say about that". Forrest Gump
"Damn, I wanted to see what happened next" (Epitaph)

oren

  • First-year ice
  • Posts: 7661
    • View Profile
  • Liked: 3001
  • Likes Given: 2952
Re: Arctic Penguin extent/area files have illegal values
« Reply #8 on: June 29, 2021, 01:51:13 PM »
I suggest to edit the first post title to:
Arctic Penguin (Wipneus) data files contain "nan" values

This will help others in the future who encounter the same issue.

kassy

  • Moderator
  • Young ice
  • Posts: 4064
    • View Profile
  • Liked: 1612
  • Likes Given: 1590
Re: Arctic Penguin (Wipneus) data files contain "nan" values
« Reply #9 on: June 29, 2021, 02:30:12 PM »
Done.
Þetta minnismerki er til vitnis um að við vitum hvað er að gerast og hvað þarf að gera. Aðeins þú veist hvort við gerðum eitthvað.