Calculating area and extent from gridded concentration data

[Wipneus]

Follow-up of a discussion in the "2014 sea ice area and extent data" thread about such calculations.

[Jim Hunt]

Thanks for this Wipneus!

Thanks too for all your hard work. I'm really looking forward to trying this out for myself,  for a change.

[Neven]

Follow-up of a discussion in the "2014 sea ice area and extent data" thread about such calculations.

This discussion starts here.

[Pmt111500]

thanks for starting this, and for work you've already done... I once tried to do an ellipsoidal equation system for converting from one grid system to another, and would very much welcome any shortcuts possible. I'm not aqcuainted with the grid systems involved in these products, but maybe me or some other amateurs will try to do their own system to get consistent total values of sea ice area.

[Jim Hunt]

Back on the original thread there were some queries about the OSI-SAF data.

Chris - The "near real time" archives only go back to 2005. However there is also a "reprocessed concentration"  dataset "covering the period from October 1978 to October 2009"

Wipneus - I think the following means they use a 10 km x 10 km grid!

The resolution or footprint size of the SSM/I 1937 measurements are approximately 25 km while for SSM/I 85 it is approximately 12.5km. As the analysis is performed on a 10 km grid we would like to keep the higher resolution information provide by SSM/I 85. However, SSM/I 85 is much more affected by atmospheric noise than the SSM/I 1937. The ASCAT input data is provided on a 12.5 km grid and by that comparable in resolution with SSM/I 85.

[Neven]

This isn't turning out to be quite so straightforward as I expected.

The attached images show my calculated extent and the error with NSIDC Extent, both are for 2012.

Initially I left the 'Pole Hole' out, just to get things working, the result is shown in the first image. As concentration and area drops in the summer my calculated area drops substantially below NSIDC Exent (down to 4%). So following the description on this page:
https://nsidc.org/data/smmr_ssmi_ancillary/area_extent.html
Where it is stated:

There is a circular section over the Northern Hemisphere pole (known as the "pole hole") which is never measured due to orbit inclination. For the purposes of ice extent, pixels under the pole hole are always considered to be at least 15 percent. For total ice-covered area, the pixels under the pole hole are not used.

I simply take the 'Pole Hole' as being over 15% concentration. Now the error leaps to +6% in the summer, as seen in the second graph.

I've tried: Both V2 and V3 area files (psn25area_v3.dat and psn25area_v2.dat). Landmask has a negligible effect, but is now included in all calculations. The range of concentration is from 0 to 250, so 15% is 37.5, there is negligible difference between using 38 and 37 as the cut offs for including a grid cell area in the total extent calculation.

Wipneus,

Do you know if NSIDC Exent is calculated using the 25km grid, or if they use the 12.5km grid?

Chris, 25km is the one to use.

[Wipneus]

Chris,

Make sure not to include lake ice.

Then there is the difference between nrt and final data. In the attached image you see the error I get with the nrt data, from 1 Jan 2013 onward. I cannot explain the summer dip, but the constant error in the rest of the year gives some confidence in the method.

Errors in the final data set are much bigger (upto +150k), I haven't looked into it yet but I suspect you have to understand/implement some processing steps that remove false ice.

It is something that I still want to do anyway. Start here:

http://nsidc.org/data/docs/noaa/g02135_seaice_index/

I have my doubts, whether CT  does such re-processing.

[crandles]

If Wipneus releases the daily figures I might add his home brew AMSR2 area/extent to my list.

I have put this together:
https://sites.google.com/site/arctischepinguin/home/amsr2/data/UH_AMSR2_3.125km_Area_Extent-v0.0.txt

Is it useful, can I do better? Let me know.

Alexander Beitsch has let me know that we can expect that all data we be reprocessed at some time, so all this is still prematurely and experimental.

Quoting here seemed appropriate.

That is great Wipneus!

Maybe someone will do some graphs of 2014 vs 2013.

.

For prediction purposes, I like to have a long record. Not sure whether removing that size of grid boxes issue in CT area will improve things much but it would be nice to see if some improvement can be made.

[Wipneus]

I have put this together:
https://sites.google.com/site/arctischepinguin/home/amsr2/data/UH_AMSR2_3.125km_Area_Extent-v0.0.txt

Is it useful, can I do better? Let me know.

Alexander Beitsch has let me know that we can expect that all data we be reprocessed at some time, so all this is still prematurely and experimental.

Quoting here seemed appropriate.

That is great Wipneus!

Maybe someone will do some graphs of 2014 vs 2013.

For prediction purposes, I like to have a long record. Not sure whether removing that size of grid boxes issue in CT area will improve things much but it would be nice to see if some improvement can be made.

My AMSR2 ASIv6 area data differ from CT-area in more important points than grid cell area calculation.

1. CT-area is based on the NASA Team algorithm, ASIv6 stands for ARTIST Sea Ice algorithm version 6 (ARTIST is yet another acronym, but seldom spelled out).  In short: the NT algorithm gives significant lower concentrations than most other algorithms, especially when the ice is not dry.
2. CT uses a grid of 25x25 km, but uses wavelength where the antenna footprint is over 40 km wide. The ASI algorithm is based on microwave ~90 GHz band, on the AMSR2 this has a natural resolution of 5 km, with some over sampling a 3.125 km grid width is justified;
3. My calculation only includes only the 14 well-known regions, CT also extra regional ice as well as lake ice;

Sea ice concentration calculated with ASI is available from 1988 (from top of my head), using data from the SSM/I, AMSR-E and SSMIS but it is clearly not homogeneous with the  AMSR2 ASIv6 concentration data that I have now. I intend to wait for a more finalized version until making the attempt to make a long range series.

[crandles]

Thanks again Wipneus.

The . was meant to indicate I had moved onto something different. I was aware of the much better resolution of your AMSR2 data and that it is only available more recently - didn't know there was any chance of going back to 1988. Other differences are also good to know about. 

So my comments about a long record were thinking that only an effort by Chris Reynolds or yourself with NSIDC data would get a long record of area without the grid size problem. Being able to test different options like including or excluding lakes and/or other areas outside regions to see what is better for prediction would be interesting.

[Jim Hunt]

Regarding DMI's interpretation of the OSI-SAF data, they have now confirmed to me something I had rather suspected:

The Great Lakes are included in both the old and the new version of the OSISAF data. However, because of the near coast masking in the old product there were not many ice or open water pixels left in the Great Lakes.

Satellite measurements near land are affected by the land emission and the computed ice concentration is therefore less reliable along the coast. In the old version of the product reliability was weighted higher than providing more uncertain ice concentrations. We have realised that many application suffer from the lack of data near land, at the same time we are now providing uncertainties along with the ice concentration data. Therefore, the new product includes data near land even though the uncertainty is higher.

[ChrisReynolds]

So my comments about a long record were thinking that only an effort by Chris Reynolds or yourself with NSIDC data would get a long record of area without the grid size problem. Being able to test different options like including or excluding lakes and/or other areas outside regions to see what is better for prediction would be interesting.

Having the PIOMAS data broken down into regions has proven (at least to myself) very useful. For some time I've wanted a long term record of area/extent in regions - in a numerical format. The problem with what Wipneus has found with CT Area is that the use of a single grid cell area makes the entire series fundamentally unphysical - IMO. Something better has to be done, and using NSIDC gridded data (NASA Team) is an opportinity worth taking. What Wipneus is doing with greater resolution data is still of value, I just want a series back as far as possible to give current changes context.

I've got tomorrow off work to have another crack at this, tiredness has meant I've made a mental note of issues like the Great Lakes (and other lakes) then failed to consider. Thanks for prompting me Wipneus. Omitting the lakes (by editing the land mask manually - in a spreadsheet ) gives the extents, and error shown in the attached image. The error is still at least an order of magnitude greater than that with which I would be satisfied. Ideally I'd like to get no difference, but as I've found with the PIOMAS data, an error may be present due to factors like rounding.

I spent last night using a hex editor to examine the concentration and grid box area files and ensuring I could indentify values from the arrays into which they're loaded and the expected position in the source binary files - the upshot being that I am correctly loading arrays with data that reflects the source files. So it's not an issue with some mismatch there. The closeness of my extent calculations and the NSIDC Extent series argues against such an error anyway, but I wanted to be sure before moving on.

PS, due to the extreme losses in 2012 I also think it worth downloading a less extreme year and looking at errors in such a year.

[ChrisReynolds]

Michael Yorke has given me a great tip for downloading the data. From the main directory:
ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/daily/

Take the link for a single year.
ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/daily/2003/

And replace the final '/' with '.tar'
ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/daily/2003.tar

That link downloads a TAR compressed file with all the year's data.

[ChrisReynolds]

I've just had an afternoon totally on the problem, no improvement. A hobby is meant to be enjoyable, programming isn't enjoyable for me, so I'm dropping the matter of a publicly available set of regional data.

[Jim Hunt]

I must be crazy. Other people do sudoku or crossword puzzles for fun. I programme computers instead!

[ChrisReynolds]

Jim,

Horses for courses.

The only reason I've worked stuff out from PIOMAS and NCEP/NCAR gridded data is that I want to understand the loss of ice in the Arctic. For me programming is a necessary evil.

[Wipneus]

OSISAF have updated their sea ice concentration product:

The new developments are a direct result of the dialog with our users. The primary improvements in the new product are:

    Temporally and spatially varying uncertainty estimates on every measured ice concentration pixel are computed. The total uncertainty squared is given as the squared sum of the representativeness uncertainty and the algorithm uncertainty. The two components are given in the data-file, together with the total uncertainty.
    A land spill-over correction scheme, which is independent of climatology, is used. This means that, unlike the old product, ice concentrations are now estimated in the coastal zone. A climatology approach is used in OSI-409 for land-spill-over correction. A physical-statistical methodology is now used in the new product, in preference to the climatological one.
    Dynamical tie-points are used to improve the sea ice concentration estimates during summer melt. Dynamical tie-points are also used to reduce sensitivity to interannual and climatic variability in the ice and water signatures, and sensor drift. Additionally, the dynamical tie-points enable quicker integration of new microwave radiometer data and eases the combination of different sensors in the same product, in order to improve coverage.

The second point  makes the product  (IMHO) usable in comparison with alternatives by including sea ice within 50km of the coast.

The third point could potentially mean large changes, to the point where we can actually say that not enough is known about the product. OSISAF seems to share my opinion:

Both the new and old products will be available in an overlapping period, for users to test the new product before the new product fully replaces the existing product.

http://osisaf.met.no/p/ice/new_ice_conc.html

[Wipneus]

OSISAF have updated their sea ice concentration product:

After a first look at the data:
- seems to be only available from 2015-06-11. The archived data is still the same;
- pole hole has been filled in (not just 100% or another constant, could be intelligent);
- the old and the new product are visibly (in concentration maps) different within the ice pack.

Neither the coastal zone correction nor the pole hole filling in are described in the Algorithm or User Guide documents.

[Jim Hunt]

In some related news:

The Air Force has stopped trying to recover a two-year-old weather satellite after operators lost the ability to command the spacecraft last month, an Air Force spokesman said March 24.

Operators at the 50th Space Wing at Schriever Air Force Base in Colorado Springs, Colorado have “ceased all recovery efforts” of the Defense Meteorological Satellite Program Flight 19 satellite, Andy Roake, a spokesman for Air Force Space Command, said in a March 24 email to SpaceNews.

The DMSP constellation requires at least two primary satellites and two backup satellites to gather cloud imagery. As a result of the problem, the Air Force has reassigned an older satellite, DMSP Flight 17, which launched in 2006 and had been serving as a backup, into a primary role.

Apparently DMSP F13 "exploded in orbit" last month also

[Tealight]

I'm developing an Arctic albedo model and want to use gridded concentration data to improve the accuracy. Since a few forum members have already experience with gridded data I would like to know what program you prefer to obtain the data and then use it for further calculations.

I read that Wipneus is or was using "R" and Chris Reynold used Visual basic, but maybe there are even better suited programs available now.

My only programming experience is in C++ and Java from school so I wouldn't mind starting with a new language.

The NSIDC mentions Fortran routines for their tools. Does that mean I should use a program that supports Fortran executables?
http://nsidc.org/data/polar-stereo/tools_geo_pixel.html
 

[Wipneus]

I'm developing an Arctic albedo model and want to use gridded concentration data to improve the accuracy. Since a few forum members have already experience with gridded data I would like to know what program you prefer to obtain the data and then use it for further calculations.

I read that Wipneus is or was using "R" and Chris Reynold used Visual basic, but maybe there are even better suited programs available now.

Most of my own newer program development are now in Python with some R thrown in for the graphics.
It does not really matter, what you need is a programming environment that handles arrays manipulations with efficiency and ease. IMO Python has the edge here.

My only programming experience is in C++ and Java from school so I wouldn't mind starting with a new language.

The NSIDC mentions Fortran routines for their tools. Does that mean I should use a program that supports Fortran executables?
http://nsidc.org/data/polar-stereo/tools_geo_pixel.html

Fortran programs should be compiled with a fortran compiler. I have not used any of those .for files ever. The data files can be read using most programming languages. Fortran was the language that I first used to program computers (using punch cards) and is still used a lot in academic environments. I do not seen much use for it outside those.

[Tealight]

Most of my own newer program development are now in Python with some R thrown in for the graphics.
It does not really matter, what you need is a programming environment that handles arrays manipulations with efficiency and ease. IMO Python has the edge here.

Thanks for the information. I think Python would suit me very well, because SNAP can be configured to work with Python.

Any suggestions for a compiler or is the standard CPython good enough?

[carmiac]

Python isn't really a compiled language, but I know what you mean. Python should be fine, and I would suggest the Anaconda distribution of is, since it has SciPy built in, which is what you want for doing lots of math and making charts.

https://www.continuum.io/downloads

[Tealight]

Hi,

I use a small ring around the pole hole, 2 or 3 pixels wide I think.

Say the pole hole has radius R, calculate average concentration of all valid grid cells within a distance R+2 from the pole.

BTW, CT area does the same, except the ring is much wider than 2 or 3 pixels.

BTW2, there is a dedicated thread in the developer corner for such discussions:
http://forum.arctic-sea-ice.net/index.php/topic,782.0.html

Ok lets continue the polehole discussion here.

I never do any calculations on the shaped array, which allows me to use just one for loop for all pixels. For the polehole it means I have to use the 1D array location. Does your calculation look similar to this? It definitely works for me and makes the polehole quite hard to find on concentration maps.

In my program every pixelvalue is part of the array icedavf

Code: [Select]

icepoleedge = [69465,69466,69767,69768,69769,69770,69771,69772,70070,70071,70072,70075,70076,70077,70372,70373,70374,70381,70382,70383,70676,70677,70678,70685,70686,70687,70979,70980,70981,70990,70991,70992,71283,71284,71285,71294,71295,71296,71588,71589,71590,71597,71598,71599,71892,71893,71894,71901,71902,71903,72198,72199,72200,72203,72204,72205,72503,72504,72505,72506,72507,72508,72809,72810]
icepole = [70073,70074,70375,70376,70377,70378,70379,70380,70679,70680,70681,70682,70683,70684,70982,70983,70984,70985,70986,70987,70988,70989,71286,71287,71288,71289,71290,71291,71292,71293,71591,71592,71593,71594,71595,71596,71895,71896,71897,71898,71899,71900,72201,72202]
icepolecon = 0


for val in range(0,len(icepoleedge),1):
icepolecon = icepolecon+ice[icepoleedge[val]] /len(icepoleedge)

for val2 in range(0,len(icepole),1):
ice[icepole[val2]] = icepolecon

[Wipneus]

I never do any calculations on the shaped array, which allows me to use just one for loop for all pixels. For the polehole it means I have to use the 1D array location. Does your calculation look similar to this? It definitely works for me and makes the polehole quite hard to find on concentration maps.

Using 2-d arrays makes it possible to be explicit which cells are withing some arbitrary circle with radius R and center x,y (the pole in this case). Some masking hen makes the calculations quite simple (no loops). I will try to dig up of some codes of mine soon to illustrate.

[Tealight]

Using 2-d arrays makes it possible to be explicit which cells are withing some arbitrary circle with radius R and center x,y (the pole in this case). Some masking hen makes the calculations quite simple (no loops). I will try to dig up of some codes of mine soon to illustrate.

Can you estimate efficiency of using this 2-D method? I only used one big loop for the whole array to find all relevant pixels and saved them in a list (took maybe 1-2 seconds)

To actually calculate the polehole I use the lists and these two small "for loops". For 366 days it took maybe a second or less so one day only needs 2.7ms of calculation. My download tool now takes care of formatting all new files. Before a file is saved on my disk I remove the header, calculate the polehole and give it a more accessible name.