Support the Arctic Sea Ice Forum and Blog

Author Topic: Home brew AMSR2 extent & area calculation  (Read 1983615 times)

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3450 on: August 17, 2020, 04:02:16 AM »
The horizontal palette below below is tricky to make and resize from the original tiff file because they open as indexed color in Gimp which precludes many RGB operations. The result displayed can easily be rotated to a vertical bar.

Either way, an accurate palette has to be embedded as an integral part of the image, as it is in AMSR2 UHH. The yellow separator is easily removed or replaced with another color by a single click.

The second image shows what happens when the palette is differenced from itself by an offset of one palette color. As these differences are very small, the outcome is complex however still interpretable. As an HSV grayscale, the palette values decline monotonically but in a slightly flawed manner. Possibly it increments properly in HSL luminance.

Scientific palettes should always be built up from a stepped grayscale. The color can furnish a pleasing aesthetic tint but at the end of the day, it is necessary for accurate graphical arithmetic operations to have that underlying easily recoverable grayscale.

The final image shows a five day difference, |Aug 15th - Aug 10th| linearly compressed to 50% to lighten. The before and after palettes are included. It needs a click to display colors accurately. Areas of melt activity and wind-driven motion show up clearly whereas the black of the central CAB means little change in AWI AMSR2 concentration over the five days.
« Last Edit: August 17, 2020, 04:08:38 AM by A-Team »

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3451 on: August 17, 2020, 08:27:42 AM »
ATeam
I appreciate your posts. Do you know how to automate gimp? I do not know how to get started. With VBA I could record macros and go from their. I just do not know where to start. Most products use AMSR2 sensor these days. Are you referring to MASIE 4 K, Hamburg 3.125 K or something else?
Edit: Clearly I missed the earlier posts.
« Last Edit: August 17, 2020, 09:10:09 AM by interstitial »

Sepp

  • New ice
  • Posts: 66
    • View Profile
  • Liked: 40
  • Likes Given: 108
Re: Home brew AMSR2 extent & area calculation
« Reply #3452 on: August 17, 2020, 09:23:36 AM »
I appreciate your posts. Do you know how to automate gimp?

Chapter 2 and 3 in this link might be a good start: https://docs.gimp.org/2.10/en/gimp-scripting.html

Since the Gimp Script Fu syntax is not that simple, several things might be easier using ImageMagick in the command line mode: https://imagemagick.org/script/command-line-processing.php

For ImageMagick, there is also a Python API available, but I am not aware of its quality.

oren

  • Moderator
  • First-year ice
  • Posts: 9819
    • View Profile
  • Liked: 3589
  • Likes Given: 3943
Re: Home brew AMSR2 extent & area calculation
« Reply #3453 on: August 17, 2020, 10:00:20 AM »
Welcome Sepp!

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3454 on: August 17, 2020, 11:19:27 AM »
thanks Sepp


uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3455 on: August 17, 2020, 03:51:23 PM »
pixel count update. This glitch was filled manually to smooth out the open water count. There are other small sporadic phantom ice and the 'swath ends' left untouched.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3456 on: August 17, 2020, 10:38:50 PM »
Quote
interstitial: Do you know how to automate gimp?
It's doable to a certain extent but I don't recommend going that route. I found the time spent automating greatly exceeded what it took to perform the task manually many times over, that I invariably wanted to do the task differently after a few days (or lost interest in it altogether), and that even minor version upgrades come with no assurance of backward compatibility to protect the investment. Gimp is thinly staffed by volunteers who are often pre-occupied for years with esoteric priorities whose mainstream utility escapes me.

ImageJ macros are a lot easier to work with. The plugin java structure has engaged a large number of individual contributors, mostly people like us with similar scientific agendas (only in cell biology). You may find a pre-existing macro does about what you want or just needs a few tweaks or additions to transparent script or an email to the originator. No knowledge of programming languages is needed if you can work off examples.

There are thousands (?) of plugins that don't come preinstalled with the download that may do what you want. Even if these are sub-perfect in some way, they don't do any real damage (unlike in a version upgrade elsewhere).

https://imagej.nih.gov/ij/plugins/index.html

Panoply is a very nice closed gui alternative for netCDF command line. I've written the staff of one with feature requests but while helpful every time, improvement priorities arise from his host institution and important user groups. While you could do keystroke capture macros off your OS, it would make more sense to delve into netCDF libraries to see if you can alter something in in there to better pre-condition your input file to Panoply.

We've previously delved into things like automating Nullschool products but a bazillion practical issues come up like the server being late or down or just not responding, before even getting to the graphical slice'n'dice which takes quite a few steps to get done right.

My view is that AMSR2_UHH is by far the best product out there for visualizing ice concentration (etc) for purposes of the current melt or freeze season. It just does not pay to focus on poor resolution legacy satellites products unless your entire focus is on long term trend continuity. A lot of the latter don't even offer a netCDF much less put error maps within it.

AMSR2_UHH is slated for termination; its PI now works at AWI on an improved product, factoring in observational regimes of the Polarstern expressly designed for algorithmic interpretive improvements. Best of all, end user input is being considered; change is about impossible to bring about on older pipelines set in concrete.

AMSR2_AWI still has its share of weather-related artifacts in summer. These are almost invariably white bias but sometimes a diffuse blue appears. That being the case, options exist to correct that bias using better weather on earlier days or by masking with cloud-cover sensors like the new NOAA-20 VIIRS.

The key here is that the speed of wind-driven ice is about 1% that of surface wind. In other words, bias-producing weather generally moves on faster than the ice can get out of positional register. So by stacking a few days in gimp and correcting presumptive white bias in the latest date by setting layers iteratively to gimp  'darken only' mode which, the way the palettes are constructed in both legacy AMSR2_UHH and AMSR2_AW, is tantamount to blue over-ride.

That is, if the weather ever did clear over a particular area of ice during a time that the sensor was making its swath there, the algo gets a chance to process actual ice data and produce an accurate pixels, maybe still white but more likely bluer if a lower concentration was indeed warranted.

The attached example shows blue over-ride on the 16 Aug 2020 AMSR2_AWI for purposes of a more accurate comparison to a large Sept 2012 AMSR2_UHH rescaled down by 68.38% to fit. The most interesting difference is 2020 low ice between the pole and Greenland to which the 2012 has no counterpart at all.
« Last Edit: August 18, 2020, 04:18:57 AM by A-Team »

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3457 on: August 17, 2020, 11:40:12 PM »
Thanks A-Team as crude as it will be I would like to get a volume estimate off of Hycom thickness. If I don't automate it I will likely only do 1 or 2. I have used Image J in the past and it sounds like that might be the best way to go. I knew there were plugins for it but I didn't think to search for them.
I looked for AMSR2 UHH but the website I was directed to was in German. It wouldn't translate so I couldn't really find it. The image you posted looks great. The older maps would be nice to have for comparison but oh well. I checked the AMSR2 AWI link and I wanted it to have the support documentation but since they are just starting to develop it that wouldn't make sense.

oren

  • Moderator
  • First-year ice
  • Posts: 9819
    • View Profile
  • Liked: 3589
  • Likes Given: 3943
Re: Home brew AMSR2 extent & area calculation
« Reply #3458 on: August 18, 2020, 02:35:14 AM »
This is the link I have for AMSR2 UHH.
sorted oldest to newest, scroll down the end of the directory and you will find Arctic images for each date.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3459 on: August 18, 2020, 12:46:43 PM »
Those links being:

ftp://ftp-projects.cen.uni-hamburg.de/seaice/AMSR2/3.125km/ stable for now
http://seaice.de/Arc_AWI_AMSR2_latest.tiff daily change in 'latest' no archive yet

Quote
I would like to get a volume estimate off of Hycom thickness. If I don't automate it ...
It would take 20 seconds a day to do manually but several hours to automate so a couple of years will elapse before any time is saved, assuming there's still volume to measure. Either way, there's a fair amount of one-time spreadsheet set up to do because the volume scale bar (palette) is not quite in synch with the dithered colors used in the ice thickness display.

This is the same project uniquorn is working on above, only to get AMSR2_AWI for concentration statistics (palette equalization is optimization). It starts with loading an indexed color file into ImageJ, either the Hycom gif or the AMSR2 tif. The trick then is to list the color data values and their frequencies of occurrence using Analysis --> Histogram. These need to be coordinated with the ordering of the map palette using Image -->Color --> Edit LUT and Image -->Color --> Show LUT --> List.

The numbers all end up as spreadsheet columns, Excel or any free online tool will do. Build a new column as the product of occurrence x thickness for each color. Assume wrongly that each pixel has the same area. The sum at the bottom is total volume for that day. Repeat down the Hycom stack to get each of the 30 days.

Hycom needs extra fiddling with the colorpicker radius. The palette has been greatly improved and now consists of 100 clean blocks. At radius 5, clicking on a single palette color block will select the 'same' thickness on the map. Check that clicking on all the blocks in turn selects all of the ice pixels; if not, bump the radius. This process assigns a thickness to each map color. The resolution is 20 colors per meter of thickness. After completing set-up of the ss template, the daily update is just a click and paste.

In some ways, making the associated data file is the point of netCDF and related. I've not seen this for Hycom products so it has to made off the graphic. As you say, this won't be as accurate and adds on to the model thickness error which isn't great to begin with. The resolution and ice area are so llow today the Arctic Ocean ice is represented in Hycom by only 13,000 pixels (vs 413,000 in AMSR2_AWI).
« Last Edit: August 18, 2020, 01:06:40 PM by A-Team »

oren

  • Moderator
  • First-year ice
  • Posts: 9819
    • View Profile
  • Liked: 3589
  • Likes Given: 3943
Re: Home brew AMSR2 extent & area calculation
« Reply #3460 on: August 18, 2020, 12:59:09 PM »
Thanks A-Team, I now see I forgot to post the actual link...

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3461 on: August 18, 2020, 06:48:15 PM »
Lars asked if there is end user interest in netCDF versions of the new AMSR2_AWI. The original tif files seem to be in 8-bit geoTiff format (ie pixels have geo-referenced tags to their lat lon relative to polar stereographic projection).

It seems this was verified in QGIS but not GDAL, the acronym for Geospatial Data Abstraction Library, a computer software library for reading and writing raster and vector geospatial data formats from command line. Fortunately someone put up a free front end for some of GDAL, notably an online batch tool for gdalinfo() that converts geoTiff into netCDF format which we want for the menu-driven Panoply viewer for various reasons.

This greatly blows up file size, in part because the AWI file covers a gargantuan area outside of the Arctic that does not normally concern the forum. Cropping the image to the Arctic still gives a valid nc file but the georeferencing has been lost (2D rather than geo2D in Panoply). The preliminary full Panoply map below shows that the conversion process works (but maybe could be improved).
« Last Edit: August 18, 2020, 06:57:53 PM by A-Team »

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3462 on: August 18, 2020, 07:28:33 PM »
A-Team
Hycom does offer a 50 GB daily data file but you are right that the resolution on the gif is not very high. I wish they offered something in between. AWI sounds like it is/will be a great area and concentration product. I did not realize Uniquorn is already doing Hycom thickness. I may decide to do something else.

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3463 on: August 18, 2020, 07:36:43 PM »
I like distinct color changes in images so I can assign ranges as I look at them. To many colors makes it unreadable as well. Maybe 6 colors with blending in between.

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3464 on: August 18, 2020, 08:30:49 PM »
I did not realize Uniquorn is already doing Hycom thickness. I may decide to do something else.
I'm not analysing hycom, just the new awi amsr2. Please go ahead and cover it.

Today's pixel count (from the imagej histogram)
A big increase in 100% conc at the expense of 99%. Different/thicker clouds or a small algo adjustment?

I quite like this LUT, it picks out 1% and 99% concentration (if the 100px colours correlate to conc)
It looks like the swath edges have gone, so this is probably a design change.
Coastal areas are cleaner too.
All the low end pixels take a large drop today, so best to take the open water chart with a pinch of salt.
« Last Edit: August 18, 2020, 08:49:46 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3465 on: August 18, 2020, 09:02:10 PM »
So the harmonics have been ironed out too. You can't really tell, but we are looking at a quite different presentation. (The 1 day open water chart shown doesn't have the glitch correction for the 14th)
Spurious ice artefacts (in warm open water) have gone too. That could explain some of the drop in low conc.
« Last Edit: August 18, 2020, 09:09:34 PM by uniquorn »

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3466 on: August 18, 2020, 10:24:14 PM »
Quote
“you can't really tell, but we are looking at a quite different presentation”
Indeed. A major refinement and a very troubling new picture of the current state of the ice (assuming accuracy). The ice looks just terrible today along atlantic side … and elsewhere (gold areas in 2nd frame; click for full size and both frames):

interstitial

  • Young ice
  • Posts: 2900
    • View Profile
  • Liked: 574
  • Likes Given: 96
Re: Home brew AMSR2 extent & area calculation
« Reply #3467 on: August 20, 2020, 05:59:24 PM »
Any Chance AWI graphs will include thickness down to 0% rather than cutting off at 15%? I doubt it would make much volume difference but ice in the water makes a difference to melting/freezing.

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3468 on: August 20, 2020, 10:01:43 PM »
Flashing in and out in the chukchi over the last 3 days. I'm not sure how that issue can be resolved.
pixel count had a different curve yesterday, has reverted today. Only showing days since the update.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3469 on: August 21, 2020, 12:02:17 AM »
This is really a treat having such a nice visualization of concentration in the AMSR2_AWI (though we don't know yet how close it is to being in final form). Below, I wondered where the nearly open water ice was (clicked on all the dark palette colors, then replaced with yellow). These will be next to melt out.

At the other end, I looked at where the CAB is not yet showing any signs of opening up (last of light palette squares) and then sought to highlight ice areas where there's some early indications of action.

There is still a big central block of seemingly unaffected ice on the Wrangel side of the pole. It has no firm connection with the CAA coastline but seemingly is floating free out in the central ocean which has never been seen before.

Color exploration to bring out features a whole lot easier to do than on AMSR2_UHH. In fact, no real knowledge at all is needed of gimp: just upload the file, paste in the palette, convert to RGB, and color away. (Crops are ok but do not resize once in RGB.)

The base map here was de-clouded a bit as described previously. If you don't do that, there will be a fair amount 'white bias' artifacts (depending on weather) that make day to day comparisons erratic.
« Last Edit: August 21, 2020, 04:02:53 AM by A-Team »

Phil.

  • Grease ice
  • Posts: 542
    • View Profile
  • Liked: 76
  • Likes Given: 11
Re: Home brew AMSR2 extent & area calculation
« Reply #3470 on: August 21, 2020, 02:48:49 AM »
Looks great A-Team.  Is gimp similar to Image-J?

jdallen

  • Young ice
  • Posts: 3412
    • View Profile
  • Liked: 651
  • Likes Given: 245
Re: Home brew AMSR2 extent & area calculation
« Reply #3471 on: August 21, 2020, 07:50:42 AM »
<snip>
There is still a big central block of seemingly unaffected ice on the Wrangel side of the pole. It has no firm connection with the CAA coastline but seemingly is floating free out in the central ocean which has never been seen before.
<snip>
... and the fact it's disconnected from any land-fast ice ... that there ISN'T any land fast ice... should scare the hell out of everyone.

That by itself sets this year apart from every other including 2012 and tells me we have just entered a new regime.

It doesn't matter if 2020 doesn't pass 2012.  The damage has been done and we are on a short path to a BOE.
This space for Rent.

marcel_g

  • Frazil ice
  • Posts: 127
    • View Profile
    • Art by Marcel Guldemond
  • Liked: 54
  • Likes Given: 425
Re: Home brew AMSR2 extent & area calculation
« Reply #3472 on: August 21, 2020, 02:58:25 PM »
<snip>
There is still a big central block of seemingly unaffected ice on the Wrangel side of the pole. It has no firm connection with the CAA coastline but seemingly is floating free out in the central ocean which has never been seen before.
<snip>
... and the fact it's disconnected from any land-fast ice ... that there ISN'T any land fast ice... should scare the hell out of everyone.

That by itself sets this year apart from every other including 2012 and tells me we have just entered a new regime.

It doesn't matter if 2020 doesn't pass 2012.  The damage has been done and we are on a short path to a BOE.

Agreed. We already have a (Significant) fractional BOE.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3473 on: August 22, 2020, 01:21:49 PM »
Quote
Oren: the new product from AWI that is replacing UHH is using 6.5(??) km or some other number higher than 3.125.
That's correct. The new AMSR2_AWi product has a resolution of 5.0 km that's intermediate to the two previous AMSR2_UHH products of 3.215 and 6.25. All these were developed by Lars Kaleschke and colleagues.

The scale can be determined in any image program simply by drawing a rectangle from the Bering Strait to Norway on the two images and noting the widths. That comes out here to 1654:2654 pixels or 0.623. Dividing that into 3.125 gives 5.011. The map is the usual NSIDC polar stereographic centered on the pole with standard latitude 70º.

The scale really matters close to islands and narrow straits, for example visualizing post-collapse Milne Ice Shelf or Lincoln Sea floe drift. Putting a square pixel down on a curved coastline divides its footprint between land and sea. The larger scale UHH sought to improve the fit but it involved a certain amount of extrapolation beyond the 6.25 km raw data.

The land map on AMSR2_AWI is much improved so the resolution chosen for the server archive could be dialed back into a single 5.0 km product with a new focus on a better data processing algo.

Kaleschke recently wrote up Sea Ice Ticker Nr. 50 about observational efforts during leg 3 of Mosaic to improve AMSR2 data quality using the same satellite sensors but at ground level on the ice. Snow drifts in the lee of equipment and linear polarization of low angle specular solar reflection made comparison to the satellite view quite problematic.

We do not know at this time whether AMSR2_AWI fully assimilates this new information or is still a work in progress. However it is currently the best product out there for accurately measuring open water and ice area via concentration.

Given the shocking new photos of the north pole melt ponds, what can be expected from overlaying 5x5 km pixels? Ponds and leads took up half the surface on Aug 19th but individual features had characteristic dimensions in the tens or hundreds of meters, not km.

A 5 km pixel would thus have a whole lot of small melt ponds in it in addition to unaffected ice. Although Mosaic didn't share the overhead imagery tan, it is still possible to make a visualization of AMSR2_AWI pixels sitting over this landscape. The 3-frame animation below simulates this for two grid cell sizes; the larger cells are like pixel footprints from AMSR2_AWI at the north pole.

https://www.meereisportal.de/en/mosaic/sea-ice-ticker/
« Last Edit: August 22, 2020, 02:02:31 PM by A-Team »

gandul

  • Guest
Re: Home brew AMSR2 extent & area calculation
« Reply #3474 on: August 22, 2020, 01:55:28 PM »
This is really a treat having such a nice visualization of concentration in the AMSR2_AWI (though we don't know yet how close it is to being in final form). Below, I wondered where the nearly open water ice was (clicked on all the dark palette colors, then replaced with yellow). These will be next to melt out.

At the other end, I looked at where the CAB is not yet showing any signs of opening up (last of light palette squares) and then sought to highlight ice areas where there's some early indications of action.

There is still a big central block of seemingly unaffected ice on the Wrangel side of the pole. It has no firm connection with the CAA coastline but seemingly is floating free out in the central ocean which has never been seen before.

Color exploration to bring out features a whole lot easier to do than on AMSR2_UHH. In fact, no real knowledge at all is needed of gimp: just upload the file, paste in the palette, convert to RGB, and color away. (Crops are ok but do not resize once in RGB.)

The base map here was de-clouded a bit as described previously. If you don't do that, there will be a fair amount 'white bias' artifacts (depending on weather) that make day to day comparisons erratic.

The big central block is the region that remained static close to the July Anticyclone center but in its less warm side, and was very much compacted.
Part of the story of 2021 and 2022 will be to watch how this region is flushed out to the Atlantic Ocean.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3475 on: August 22, 2020, 02:55:59 PM »
No. Ice motion largely followed the circling winds in the late July anti-cyclone as shown decisively by the OsiSaf time series. No evidence has been presented for anti-cyclonic compaction of central ice -- in fact the north pole photos above show the exact opposite: no pressure ridges, lots of open water.

A weak inward Coriolis force proportional to the utterly minuscule ice velocity was completely dwarfed by other considerations. As explained many times, the lengths (magnitudes) of displacement arrows on OsiSaf have to be divided by 6 to get actual ice displacement per 24 hours.

This does not affect their direction. Wind power density during the anti-cyclone was unremarkable, not sustained very long at any given location; maximum ice speeds were on the order of 0.3 km/hr on the periphery. While the Coriolis force pointed inward, its magnitude was thus negligible relative to centrifugal. Consequently the ice simply moved with the wind as shown by direct observation (OsiSaf doesn't lie). This discussion applies unchanged to related melt season commentary concerning outward dispersion for cyclones.

http://osisaf.met.no/p/osisaf_hlprod_qlook.php?year=2020&month=07&day=28&action=d%2B&prod=LR-Drift&area=NH&size=100%25

Compaction is a real physical phenomenon that cannot be usefully defined or represented in summer by unilaterally declaring it to mean the 'area/extent ratio' as done on long-trend forums. 'Extent' is a legacy concept from the early days of low resolution satellite record; its numbers are highly flawed in summer months because of weather for reasons given many times by Oren. Dividing by 'extent' is an invitation to uncontrolled error.

The central CAB lies outside the TransPolar Drift. It barely moves at all from year to year as shown decisively by eight years of daily Ascat time series. Export out the Fram involves ice that forms in the NSI region of the Laptev and sometimes the Kara. Export ceased in mid-May this year; no one has any idea when or if it will resume its past character.

This has been a very unusual year with many unforeseen developments, most recently the totally unexpected appearance of high latitude melt ponds with over 50% surface coverage. This inability to even understand the present, much less predict the state of the ice even a month out, makes speculation about years even farther in the future unproductive.

Case in point: another doomsday 10 day weather forecast that -- once again -- never happened has preoccupied the melt season forum. Without knowledge of future weather, it's not possible to move beyond broad-brush climate change trend continuation.

However an exception might be made for gradual oceanic processes such Arctic-specific  Atlantification; vox noted a pair of very good papers based on multi-year mooring data, not models.

Arctic Ocean Moorings Shed Light On Winter Sea Ice Loss
https://phys.org/news/2020-08-arctic-ocean-winter-sea-ice.html
« Last Edit: August 23, 2020, 05:27:17 PM by A-Team »

gandul

  • Guest
Re: Home brew AMSR2 extent & area calculation
« Reply #3476 on: August 22, 2020, 03:10:26 PM »
Yeah, you are telling me this area was not very much compacted after staying 20 days under the center of a 1030+ hPa, when physics says the opposite. Or that the effect of this compaction was insignificant after a while. And I don’t believe you.
This region surface also showed much less wetness than surroundings at the end of July.
SMOS at the end of July, pre-cyclone.

binntho

  • Nilas ice
  • Posts: 2214
    • View Profile
  • Liked: 881
  • Likes Given: 239
Re: Home brew AMSR2 extent & area calculation
« Reply #3477 on: August 22, 2020, 03:40:09 PM »
Perhaps you should define "compactness" and how it is measured and quantified.

EDIT: Although A-Team is one of the very few posters here who knows what they are talking about, that does not make them right all the time. But their view re. compactness (or rather, the absence of same) during this melting season resonates with my own, and if I remember correctly, it was pretty much in line with the majority view.
« Last Edit: August 22, 2020, 05:16:13 PM by binntho »
because a thing is eloquently expressed it should not be taken to be as necessarily true
St. Augustine, Confessions V, 6

oren

  • Moderator
  • First-year ice
  • Posts: 9819
    • View Profile
  • Liked: 3589
  • Likes Given: 3943
Re: Home brew AMSR2 extent & area calculation
« Reply #3478 on: August 22, 2020, 05:24:23 PM »
Gandul, please keep the discussion civil.
This might be better discussed in the general thread, but defining compaction could definitely help. If you mean ice floes moved generally towards the middle, yeah sure, though it was mostly a circular motion. If you mean ice floes were pressured together causing thickening in the process, certainly no.
Looking at a sample OSI-SAF drift map from mid-July, perhaps the region that appears better on AMSR is the region where the ice moved less during the anti-cyclone, and thus suffered from less movement-related bottom melt. This may have helped preserve the thinner glue ice, thus improving satellite appearance.
Click to enlarge and zoom.

gandul

  • Guest
Re: Home brew AMSR2 extent & area calculation
« Reply #3479 on: August 22, 2020, 05:56:42 PM »
Perhaps you should define "compactness" and how it is measured and quantified.

EDIT: Although A-Team is one of the very few posters here who knows what they are talking about, that does not make them right all the time. But their view re. compactness (or rather, the absence of same) during this melting season resonates with my own, and if I remember correctly, it was pretty much in line with the majority view.

Yes I could, if I had the time/resources: for the period july 3 to 20 approx, integrate the fluxes of the ice drift velocity field along the contour that I mark in red over the first frame of the gif. That would give you a total area, which, if it is negative, means compactness, if positive, means divergence. This area could be expressed as a ratio of the total area bounded. For instance, in that period a divergence of -X% happened over that region...

More practical than this circle is to choose a region based on meridians and parallels, for instance, the “triangle” with NP as vertex, formed by 135W, 150E meridians and 80N parallel.

This will characterize the ice compaction well for that region in that period of time because the ice pack in that region does not really displace much in average, otherwise this formula would not really reflect the real compaction. To test this, simply obtain the average of ice drift velocity over the region and integrate it from day 3 to day 20. If the total displacement is much smaller (1/10 or less) than the root square of the region area, then we are good.

There I leave it for somebody with computational resources with access to some nc files and with enough curiosity on this particular issue. I doubt anybody will spend the time. I wish I could.

Click to play
« Last Edit: August 22, 2020, 06:19:09 PM by gandul »

binntho

  • Nilas ice
  • Posts: 2214
    • View Profile
  • Liked: 881
  • Likes Given: 239
Re: Home brew AMSR2 extent & area calculation
« Reply #3480 on: August 22, 2020, 06:31:36 PM »
So by "compactness" you mean an area where the motion causes a movement that tends to a lesser area, i.e. inwards. Fair enough. Individual floes would gregariously flock together, and any leads would get narrower or even close up.

But I think most people would understand "compaction" as meaning "compression", although that is not a necessary meaning. Compact can well mean "less dispersed" as well. The difference is important when the forces are parallel to the surface of an aggregate of thin pieces. Do the pieces crumble and stack, or do they just float closer together?

But you do not discern between the two? I.e. did the July weather lower dispersion, or did it compress the ice (both of which would count as compaction)? This is where the divergence in opinions occurs. Most posters (me included) refuse to countenance compression, the forces involved were simply too small and the general state oft the ice too fluid in aggregate.

So if we agree that the ice was "compacted" in the sense that dispersion was lessened, then we agree. But I'd prefer that we used some other term for it!
because a thing is eloquently expressed it should not be taken to be as necessarily true
St. Augustine, Confessions V, 6

gandul

  • Guest
Re: Home brew AMSR2 extent & area calculation
« Reply #3481 on: August 22, 2020, 08:11:55 PM »
Yes I think we agree in less dispersion. I also thought in July that, when very little space is left between the floes, some sort of piling up had to occur, instead of the gregarious thinking that “melting just takes care of creating the space”. Nobody explained how but anyway, in that sequence of 20 days there is such a level of extent reduction that the ice had to find ways to stay together: crushing, floe tilting, and maybe what Michael Hauber conjectured and automatically was disregarded: some stacking of ice.

I can’t prove or disprove that, but be sure that pure exact mathematical melting was not the responsible of leaving space and the total extent reduction at that time, especially in the red region I mark, that also stayed cooler than the Atlantic side.

oren

  • Moderator
  • First-year ice
  • Posts: 9819
    • View Profile
  • Liked: 3589
  • Likes Given: 3943
Re: Home brew AMSR2 extent & area calculation
« Reply #3482 on: August 22, 2020, 08:37:17 PM »
Gandul please continue this discussion in the stacking thread started for that purpose.

binntho

  • Nilas ice
  • Posts: 2214
    • View Profile
  • Liked: 881
  • Likes Given: 239
Re: Home brew AMSR2 extent & area calculation
« Reply #3483 on: August 23, 2020, 05:47:18 AM »
instead of the gregarious thinking that “melting just takes care of creating the space”.
I'm not sure if that is "gregarious thinking" but my point as I have expressed it frequently is that the forces were not strong enough to overcome the rigidity of the ice to cause ridging or stacking. So once the leads closed up, the inward movement stopped.
because a thing is eloquently expressed it should not be taken to be as necessarily true
St. Augustine, Confessions V, 6

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3484 on: August 23, 2020, 09:15:41 PM »
awi amsr2(dev) histogram pixel count, 1-99% concentration to highlight smaller curve differences. Just a hint of harmonic at 50
« Last Edit: August 23, 2020, 09:25:14 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3485 on: August 23, 2020, 11:40:55 PM »
Worldview tm, beaufort aug23, amsr2awi(dev) aug22 inset. Can do a better comparison tomorrow

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3486 on: August 24, 2020, 12:22:57 AM »
Quote
Can do a better comparison tomorrow
Nice! What about the companion cloud shot, M3 M11 M13 one of those red ones. NOAA-20 VIIRS is new, maybe better. Swathes are synched on the same satellite. Key to masking the visible which has some lite clouds and standing waves.

Here is the history of peripheral melt that goes with that, as told by all the AMSR_AWi available to date (Aug 6th to Aug 22th). The lines show the extent of open water.

Day to day melt does not seem that dramatic but over time it adds up. Note very pronounced regional differences: the Atlantic side from Svalbard to NSI in the Laptev is orderly and incremental, a big bite proceeds in the Chukchi but not so much in the Beaufort which however has internal growth. The north Greenland open water event appears over.

If you download, just delete all the intermediate frames with transparencies. This lets you look at the 3 full frames at your own pace.

Note though that ice concentration inside is also declining. Because of the need for de-clouding weather, re-coloring the palette for emphasis, and gif optimization not being applicable, that needs to be done separately as a mp4.

This is made from a tiled stack of the AMSR_AWi rescued from the daily 'latest' by uniquorn. Open water was selected across the whole tile and its boundary saved in a new layer by shrinking the full selection by 1 pxl and then colored successively by the gradient tool, re-stacked with a few key re-insertions of the latest ice concentration and some delays set for special views: all, none, 17 day.
« Last Edit: August 24, 2020, 01:41:36 AM by A-Team »

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: Home brew AMSR2 extent & area calculation
« Reply #3487 on: August 24, 2020, 12:48:57 AM »
You have a github A-team?
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3488 on: August 24, 2020, 01:55:58 AM »
Quote
working on a large project at the moment (Scottish snow, not Arctic stuff), but heres the mask i'm using easy distinction between clouds and ice/snow ... can separate ice (yellow) from clouds (white) in python with...rain clouds moving over ice, surface water affects sensors ...Bremen amsr2 showing rain all over the ice.
No github. Not sure what I would put up there for an open source pj. These .xcf files are a one-off format; they lose the algo steps (aka undo) upon saving. I rarely if ever repeat steps making complex graphics so no versioning. Might be good to automate some of them for the standing graphics page though.

nanning

  • Nilas ice
  • Posts: 2487
  • 0Kg CO₂, 37 KWh/wk,125L H₂O/wk, No offspring
    • View Profile
  • Liked: 273
  • Likes Given: 23170
Re: Home brew AMSR2 extent & area calculation
« Reply #3489 on: August 24, 2020, 07:57:17 AM »
Re: the gif above SimonF92's post.
What a wonderful visual A-Team. And great choice of colours imho.


edit: corrected name because there's also a Simon on the forum
« Last Edit: August 24, 2020, 06:31:27 PM by nanning »
"It is preoccupation with possessions, more than anything else, that prevents us from living freely and nobly" - Bertrand Russell
"It is preoccupation with what other people from your groups think of you, that prevents you from living freely and nobly" - Nanning
Why do you keep accumulating stuff?

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: Home brew AMSR2 extent & area calculation
« Reply #3490 on: August 24, 2020, 09:18:29 AM »
Ah, i understand.

That has to be one of the best visualisations of melt progression I have ever seen!
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

Jim Hunt

  • First-year ice
  • Posts: 6284
  • Don't Vote NatC or PopCon, Save Lives!
    • View Profile
    • The Arctic sea ice Great White Con
  • Liked: 895
  • Likes Given: 87
Re: Home brew AMSR2 extent & area calculation
« Reply #3491 on: August 24, 2020, 02:03:48 PM »
You have a github A-team?

The ASIF has its very own Developers Corner:

https://forum.arctic-sea-ice.net/index.php/board,24.0.html

Topic #24!
"The most revolutionary thing one can do always is to proclaim loudly what is happening" - Rosa Luxemburg

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: Home brew AMSR2 extent & area calculation
« Reply #3492 on: August 24, 2020, 07:29:16 PM »
You have a github A-team?

The ASIF has its very own Developers Corner:

https://forum.arctic-sea-ice.net/index.php/board,24.0.html

Topic #24!

Thanks for this Jim.

Ive worked on a project there before with some of the members. I was just hoping to "pinch" A-team's code for another snow project im working on (the details of which ill share in the developers corner when ready)
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: Home brew AMSR2 extent & area calculation
« Reply #3493 on: August 24, 2020, 07:34:16 PM »
Quote
github visualizations from GIMP
In essence, the infinite undo in Gimp (vs none in ImageJ) stores a reverse order macro but only until the file is saved. So it should not be a giant step to set it aside for later use. However it's not clear whether it saves the step in enough detail (eg 'resize' is kinda vague) or saves the intermediate files with an incomplete recollection how it got there.

It would be great to automate a rolling (or extensional) update of the three time series below. However most of the manual time goes into trying alternate visualizations and discarding them as less effective. So recording keystrokes would just be asking for trouble.

Once the final path through the gimp menu is settled on, it takes very little time to provide the daily update. While that's a nice resource in some sense, it has lost all shock value for the viewer. So usually I would rather try an altogether fresh approach.

The ice in the Beaufort isn't melting out for them on the melt season forum but on this forum it is.

The first animation shows the odd-day OsiSaf progression since Aug 1st. The display is simplified to dark blue open water, light blue dodgy ice, and white for solid ice. The ice motion arrows can be ignored since the purple ones have large uncertainty on the intermediate ice and of course are missing altogether on the open water.

There is a lot to be said for simplification: if the end viewer can just follows the light blue, the point about rapidly melting ice in the Beaufort is made.

That's down to vulnerable individual floes and a lot of wispy ice as can be seen in uniquorn's WorldView capture. Sunlight at 75º latitude on Aug 24 is still bringing in 42% of the energy it did on June 21st; even if it's reflected away or blocked by clouds, the water has already warmed.

The new AMSR2_AWI provides a much more nuanced view of sea ice concentration. That's worth presenting at face value but it's asking a lot of the viewer to discount late summer weather and wetness artifacts. So it's better to strip off those effects to some degree, maybe at the risk of introducing new ones.

Here I used 'darken only' graphics on the 18-day stack of files available. The trick here is that the 100-color palette comes monotonically graded by blue so the operational resultant is trapped within the LUT. That is, 'darken only' replaces a whitish pixel if a blueish one occurs on an earlier but still recent date and that blue has to correspond to one of the palette squares. A lot of other operations, like bicubic resizing, will land a lot of the pixels elsewhere (dithered).

The free parameter here is the number of days back allowed to overrule the date being corrected. Here I set it to 3, the second trick being to tile up the original set, then add layers of the same width by cyclic permutation, darken only the whole tile, then later delete the 3 unwanted frames from the cyclic.

The other really useful element of AMSR2_AWI file design is the ability to re-color. This is like contouring but exact. The free parameter is the range; the choices made are conveniently displayed in the embedded altered palette. Here i set the range to a contiguous eight palette squares after finding this group picked out the next round of ice likely to melt within the bigger block of CAB ice.

So that is really the role for github here: code that precomputes daily this ZxZ two-parameter lattice of small non-negative integers for me the decision-maker, not me qua low-life graphics guy.
« Last Edit: August 25, 2020, 12:45:27 AM by A-Team »

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3494 on: August 24, 2020, 10:14:00 PM »
pixel count update. The y axis has been reduced here to get a better look at the lower concentration pixels which increased significantly today. I thought that may be due to clearer weather over the beaufort. Not sure.

awi(dev) beaufort histogram, aug17-23

seaice.de

  • Frazil ice
  • Posts: 130
    • View Profile
  • Liked: 86
  • Likes Given: 21
Re: Home brew AMSR2 extent & area calculation
« Reply #3495 on: August 25, 2020, 01:43:34 PM »
The new AMSR2 product can from now on be found here. Thanks for all the feedback!

ftp://ftp.awi.de/sea_ice/product/amsr2/

Jim Hunt

  • First-year ice
  • Posts: 6284
  • Don't Vote NatC or PopCon, Save Lives!
    • View Profile
    • The Arctic sea ice Great White Con
  • Liked: 895
  • Likes Given: 87
Re: Home brew AMSR2 extent & area calculation
« Reply #3496 on: August 25, 2020, 02:26:35 PM »
The new AMSR2 product can from now on be found here.

Thanks very much for the NetCDF Lars.

However here's a bit more feedback for you. Panoply 4.11.4 tells me:
"The most revolutionary thing one can do always is to proclaim loudly what is happening" - Rosa Luxemburg

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3497 on: August 25, 2020, 03:15:47 PM »
It opens for me using panoply4.115 but I've lost the option to change projection etc (the map tab)
Appears to be using x,y rather than geo
« Last Edit: August 25, 2020, 03:21:22 PM by uniquorn »

seaice.de

  • Frazil ice
  • Posts: 130
    • View Profile
  • Liked: 86
  • Likes Given: 21
Re: Home brew AMSR2 extent & area calculation
« Reply #3498 on: August 25, 2020, 03:24:25 PM »
The new AMSR2 product can from now on be found here.

Thanks very much for the NetCDF Lars.

However here's a bit more feedback for you. Panoply 4.11.4 tells me:
Thanks for the feedback! I tried to reproduce this. However, with Panoply 4.11.6 I can view the data (attached).

Another question: do you need the latitude, longitude grid included in the NetCDF file? Or are you fine with X,Y? Could the lat, lon grid be provided in a separate file? Including lat, lon would increase the file size.

uniquorn

  • First-year ice
  • Posts: 5136
    • View Profile
  • Liked: 2176
  • Likes Given: 388
Re: Home brew AMSR2 extent & area calculation
« Reply #3499 on: August 25, 2020, 04:18:37 PM »
using grid point index as units gives me more options. Need to play with it for a while.