AWI AMSR2

[Juan C. García]

I think that this topic is important enough to have a new thread, so, if you allow me, Seaice.de:

A new freezing thread and time to announce a new independent AMSR2 data source. Have a look at:

ftp://ftp.awi.de/sea_ice/product/amsr2/v106/analysis_nh/

It includes a table of the regional extent and area derived from AWI AMSR2 SIC product
nh_awi_amsr2_regional_extent_area.csv

Full reprocessing is not yet finished but data will be regularly updated.

See ftp://ftp.awi.de/sea_ice/product/amsr2/README.txt for a quick introduction.

Feedback and comments are very welcome!

It opens fine in the outdated Internet explorer.

Read me:

ftp://ftp.awi.de/sea_ice/product/amsr2/README.txt

AWI AMSR2 ice concentration product
-----------------------------------
Contact: lars.kaleschke@awi.de

Introduction
------------

The AWI AMSR2 ice concentration product is based on previous developments at the University of Bremen (UB) and the University of Hamburg (UHH) [1,2,3,4]. The level 1 brightness temperature data are provided by the Japan Aerospace Exploration Agency’s (JAXA) Advanced Microwave Scanning Radiometer 2 (AMSR2) [5].

Main features
-------------
 * Visually improved ice concentration with lead occurrences
 * 3 km resolution
 * Compressed GEOTIFF with optimized color scheme [6]
 * New landmask
 * Twice daily processing

Product specifications
----------------------
The product is generated twice daily for two different start times (filenames with AM or PM).

    [PM] 12 - 12 UTC. The PM product starts with data after noon and is available in the evening.

    [AM] 00 - 24 UTC. The AM product is available in the morning of the next day.
...

[Jim Hunt]

Assuming for the moment that Lars is watching:

Feedback and comments are very welcome!

There are obviously still some large gaps in the data, and less obviously some small ones. In the CSV files could you fill missing dates with NaN or -1 or somesuch?

A personal preference, but could you format area/extent values in million km2?

Is there a binary mask of Wipenus / Cryosphere today regions somewhere?

I assume Wipneus must have one somewhere! As a corollary, could you make the 2432x3584 regional mask you are presumably using available?

[seaice.de]

Hi Jim, thanks for the feedback!

The data gaps will be filled when a full reprocessing is finished. I hesitate to use NaN or other placeholders. This will be used when there are real data gaps not for incomplete processing.

Earlier you pointed out the ordering. I missed a .sort_index() which is now included.

I updated the README.txt to describe the mask used:
Regional sea ice extent and area are calculated based on the upsampled 25 km "NSIDC" regional mask "Arctic_region_mask_Meier_AnnGlaciol2007.msk" available at https://nsidc.org/data/g02135

The old 25 km mask is certainly not the best choice for the 3.125 km product because of inconsistencies with the land mask.

Regarding the format I leave it to the users to divide by 1e6. Using python the data can be easily used this way:

import pandas as pd
amsr2=pd.read_csv(extent_merged,index_col='Date',parse_dates=[0])/1e6
amsr2_1day=amsr2.resample("1d").mean()

Assuming for the moment that Lars is watching:

Feedback and comments are very welcome!

There are obviously still some large gaps in the data, and less obviously some small ones. In the CSV files could you fill missing dates with NaN or -1 or somesuch?

A personal preference, but could you format area/extent values in million km2?

Is there a binary mask of Wipenus / Cryosphere today regions somewhere?

I assume Wipneus must have one somewhere! As a corollary, could you make the 2432x3584 regional mask you are presumably using available?

[Jim Hunt]

Thanks for the additional information Lars. I await the complete set of regional data with much interest!

In the meantime I've grafted your numbers on to Wipneus's Arctic wide UH derived data, with the following results:

https://GreatWhiteCon.info/2021/08/the-2021-arctic-sea-ice-minimum-extent/#Sep-17

The join is invisible, and the high resolution AWI AMSR2 minimum Arctic sea ice extent for 2021 currently stands at 4.38 million km2 on September 13th.

E&OE!

[seaice.de]

Thanks, I expected that you will do this! 

Thanks for the additional information Lars. I await the complete set of regional data with much interest!

In the meantime I've grafted your numbers on to Wipneus's Arctic wide UH derived data, with the following results:

https://GreatWhiteCon.info/2021/08/the-2021-arctic-sea-ice-minimum-extent/#Sep-17

The join is invisible, and the high resolution AWI AMSR2 minimum Arctic sea ice extent for 2021 currently stands at 4.38 million km2 on September 13th.

E&OE!

[A-Team]

1. Would it be possible to get the PM file to be consistently distinct from the AM file? It seems one of the swaths gets used twice -- Fig.1 black shows the same swath used in both.

2. Is it feasible to add time stamps to the constituent swaths -- are these the same every day? We have a way of getting the timing of WorldView Modis visible so this would give a better idea of synchronization.

3. Can 2021 be prioritized for completion? It seems almost all of July and earlier are missing. Fig.2 shows files only back to July 28.

4. On the Wandel Sea, the embedded palette is too narrow. 20x20 pixels per color would be preferable. We color-pick palette colors to re-bin, re-color and collect statistics on image time series (which is done better as a cvs provided in the hosting file system).

5. On the Wandel Sea, can lat/lon of the center be embedded in the image? Actually a consistent North Pole center might be better as it allows stacking of different satellite resources. At this time, only nullschool GFS allows re-centering polar stereographic nadir view. Not all of the products provide netCDFs.

6. Any comment or implications for the AWI archive? Bremen is saying they have a way of combining Modis and AMSR2 to give 1 km resolution, to be offered as of Oct 1st. There is great interest in getting OsiSaf sea ice motion down to one day or half day from its current two-day.

7. That is great that the archive goes back to include the GAC of August 2012. The Fig.3 gif frames below show the probability (10% bins) of ice at any given location defined as the average occupancy over the last 10 years, either color or grayscale. The core area has been near-solid ice every year; the creep has been in the periphery. Fig.4 just shows the AWI for each of the ten years.

[Jim Hunt]

I use Filezilla for manual FTP on Windoze, but I've just stumbled across this potentially useful information on the Woods Hole web site:

https://www2.whoi.edu/site/itp/data/data-products/

Enable FTP URLs in Chrome:
Open Chrome and type “chrome://flags” in the address bar.

Once in the flags area, type “enable-ftp” in the search bar stating “search flags”.
When you see the “Enable support for FTP URLs” option tap where it says “Default”.
Tap “Enable” option.
Hit “Relaunch Now” option at the bottom of the page.
Firefox also disabled ftp:// types of URLS as a default. To allow them, please do the following:

Restore FTP in Firefox 88 and 89

Load about:config in the Firefox address bar.
Confirm that you will be careful.
Search for network. ftp. enabled.
Set the value to TRUE by double-clicking on the line, or with a left-click on the toggle icon.

[interstitial]

I use Filezilla for manual FTP on Windoze, but I've just stumbled across this potentially useful information on the Woods Hole web site:

https://www2.whoi.edu/site/itp/data/data-products/

Enable FTP URLs in Chrome:
Open Chrome and type “chrome://flags” in the address bar.

Once in the flags area, type “enable-ftp” in the search bar stating “search flags”.
When you see the “Enable support for FTP URLs” option tap where it says “Default”.
Tap “Enable” option.
Hit “Relaunch Now” option at the bottom of the page.
Firefox also disabled ftp:// types of URLS as a default. To allow them, please do the following:

Restore FTP in Firefox 88 and 89

Load about:config in the Firefox address bar.
Confirm that you will be careful.
Search for network. ftp. enabled.
Set the value to TRUE by double-clicking on the line, or with a left-click on the toggle icon.

I tried in chrome it appears to have been removed.

[oren]

https://www.coywolf.news/productivity/chrome-removing-ftp/

Google has released a timeline for the complete removal of FTP from Chrome.

Chrome 80 (stable in February 2020) will disable FTP by default for non-enterprise versions of Chrome, but it can be reenabled with chrome://flags
Chrome 81 (stable in March 2020) will disable FTP by default for all versions of Chrome, but it can still be reenabled with chrome://flags
Chrome 82 (stable in April 2020) will completely remove FTP support

(My Chrome is already 93)


https://www.ghacks.net/2021/04/16/firefox-90-wont-handle-ftp-sites-anymore/

Firefox 90 won't handle FTP sites anymore

Mozilla announced today that Mozilla's Firefox web browser won't support the FTP protocol from Firefox 90 onward. It was clear that FTP support would be removed from the browser, but it was not clear until today when that would happen.

FTP code is not removed in Firefox 88, and users may flip a preference to restore FTP support in Firefox 88 and 89. Mozilla will remove FTP entirely when Firefox 90 is released later this year.

(My Firefox is already 92)

[Jim Hunt]

Google has released a timeline for the complete removal of FTP from Chrome.

FileZilla it is then.

And/or wget on Cygwin!

[gerontocrat]

So where can I find old versions of firefox or chrome that I can then use for accessing ftp sites?

Any ideas, anybody?

ps: the science community use ftp a lot. Is there any commmunity discussion about this?

[oren]

Why not use Internet Explorer? You can tell Chrome to open IE when clicking on ftp links.

[seaice.de]

This works perfectly fine for me:
wget ftp://ftp.awi.de/sea_ice/product/amsr2/v106/analysis_nh/*

As an answer to A-team: the GIFs are not meant for data analysis. The color palette is reduced to 16 colors which gives still a reasonable quality while the file size remains small enough to be posted on Twitter.

So where can I find old versions of firefox or chrome that I can then use for accessing ftp sites?

Any ideas, anybody?

ps: the science community use ftp a lot. Is there any commmunity discussion about this?

[Aluminium]

On openSUSE, Firefox opens ftp links without any problem.

[Phil.]

So where can I find old versions of firefox or chrome that I can then use for accessing ftp sites?

Any ideas, anybody?

ps: the science community use ftp a lot. Is there any commmunity discussion about this?

I'm able to access ftp on Safari.

[gerontocrat]

Thanks for all the tips.

But why did they dump ftp. I've read explanations but they are supposed to be The Masters of Cyberspace so I do not believe they could not fix any problems.

[Sepp]

But why did they dump ftp. I've read explanations but they are supposed to be The Masters of Cyberspace so I do not believe they could not fix any problems.

I read a while ago, that the Firefox FTP code is quite a mess and vulnerable to several potential security problems. And since hardly nobody needs it anymore, I think it was the right (but for us annoying) choice. But I have no serious reference for this statement, so maybe it is wrong.

[HapHazard]

So where can I find old versions of firefox or chrome that I can then use for accessing ftp sites?

https://sourceforge.net/projects/portableapps/files/Mozilla%20Firefox%2C%20Portable%20Ed./

Portable versions of firefox. Might be a better idea; won't interfere with any present installs (no profile folder, etc). Can run from USB.

Or, every old regular install version you can shake a stick at:

https://ftp.mozilla.org/pub/firefox/releases/  (OFC it's an FTP link LOL)

[uniquorn]

Central Arctic extent and area.
sic-leads, sep1-25
https://oden.geo.su.se/map/ to show bathymetry

[uniquorn]

Beaufort and Chukchi

[uniquorn]

stats update from aug1

[uniquorn]

1.stats update

2. sic-leads, oct9-16.
Appears to be a good representation of leads visible using suomi/npp viirs brightness temp band15. Too cloudy to check all of them.  https://go.nasa.gov/3vxB4o9

Looking forward to seeing this back to 2012

[uniquorn]

1. stats update
2. sic-leads, oct16-22

[uniquorn]

1. regional stats update
2. sic-leads, oct23-29 (3MB)

[Sepp]

Thank you for those graphs.

The dark blue lines within the pack is not open water, right?

[uniquorn]

Long answer to short question...
Black is open water, darker blue is lower concentration/thin ice. As you are probably aware, for a short while a new lead will be open water and in winter, depending on the air temperature, begin to thicken. For some time it will likely be thinner than the surrounding floes. The colour look up table (LUT) is the first image.

extract from the version 1.06 readme:

Leads: difference between SIC, LEADS and SIC-LEADS
--------------------------------------------------
There are three different products with the filename including SIC, LEADS and SIC-LEADS. The first is the traditional ASI ice concentration, the second contains the lead ice fraction [4], and the latter product contains the ice concentration minus half the lead fraction. The AMSR lead detection does not provide reasonable features during the melting period but is calculated anyhow throughout the year

[4]An algorithm to detect sea ice leads by using AMSR-E passive microwave imagery
J. Röhrs and L. Kaleschke      Published: 27 Mar 2012
The Cryosphere, 6, 343–352, https://doi.org/10.5194/tc-6-343-2012

Abstract.
Leads are major sites of energy fluxes and brine releases at the air-ocean interface of sea-ice covered oceans. This study presents an algorithm to detect leads wider than 3 km in the entire Arctic Ocean. The algorithm detects 50 % of the lead area that was visible in optical MODIS satellite images. Passive microwave imagery from the Advanced Microwave Scanning Radiometer – Earth Observation System (AMSR-E) is used, allowing daily observations due to the fact that AMSR-E does not depend on daylight or cloud conditions. Using the unique signatures of thin ice in the brightness temperature ratio between the 89 GHz and 19 GHz channels, the algorithm is able to detect thin ice areas in the ice cover and is optimized to detect leads. Leads are mapped for the period from 2002 to 2011 excluding the summer months, and validated qualitatively by using MODIS, Envisat ASAR, and CryoSat-2 data. Several frequently recurring large scale lead patterns are found, especially in regions where sea ice is known to drift out of the Arctic Ocean.

Discussion
This study presents a way to distinguish thin ice from other ice types by using microwave imagery. Two different frequencies from the AMSR-E sensor are combined to one ratio in which thin ice theoretically has significant values above one. The ratio involves two channels with different sensor footprints. The resolution of the 18.7 GHz channel is generally too coarse to resolve lead structures. Thus, the ratio usually represents a mixture of surface types. The algorithm detects leads smaller than the 6 km ×4 km resolution of the 89.0 GHz footprint. This ability is caused by the strong emissivity contrast between thin ice and thicker ice and additionally is enhanced by the use of a high-pass filter. We note that lead structures can also be detected using the 89.0 GHz brightness temperature and the high-pass filter alone. However, using the ratio has the advantage of the respective tie-points being more stable. Because of this high-pass filter, the lead detection algorithm presented here does not detect large areas of thin ice.
For example, coastal polynyas can be seen in Fig. 5a and c.

At the broadest part of the polynya in Fig. 5a, the TIC decreases to low values. The large polynya in Fig. 5c is marked with zero TIC. At narrow parts of the polynyas, the TIC has reasonable high values. The TIC algorithm does not detect large regions of thin ice due to the fact that the algorithm uses a spatial high-pass filter. It enables the detection of sub-pixel scale leads but causes the algorithm to overlook larger thin ice areas. The lead detection presented here is optimized to detect narrow leads. Other techniques exist to recognize large areas of thin ice and polynyas (Cavalieri, 1994; Markus and Burns, 1995; Kern et al., 2005; Tamura and hshima,2011). The ratio of the 89.0 GHz and 18.7 GHz channels used here could in principle be applied to detect large areas of thin ice if the spatial filter would not be implemented. However, the significant influence of the atmospheric variability restrains this approach

1. SIC-LEADS look up table
2. fig5
3. fig6
4. https://go.nasa.gov/3bCua7P for a more recent comparison, with sic leads inset.

[uniquorn]

example of the LEADS product, oct27-nov1

the latter product contains the ice concentration minus half the lead fraction

Lars may appreciate feedback on ratio of the lead fraction applied. Leads do seem a bit overstated on the SIC-LEADS product, visually. But then, we are nearly always looking through thin cloud during daylight availability on worldview.

Corrected Reflectance (True Color)
Temporal coverage: 24 February 2000 - Present
True Color: Red = Band 1, Green = Band 4, Blue = Band 3
These images are called true-color or natural color because this combination of wavelengths is similar to what the human eye would see. The images are natural-looking images of land surface, oceanic and atmospheric features. The downside of this set of bands is that they tend to produce a hazy image.

my emhpasis

[seaice.de]

example of the LEADS product, oct27-nov1

the latter product contains the ice concentration minus half the lead fraction

Lars may appreciate feedback on ratio of the lead fraction applied. Leads do seem a bit overstated on the SIC-LEADS product, visually. But then, we are nearly always looking through thin cloud during daylight availability on worldview.

Corrected Reflectance (True Color)
Temporal coverage: 24 February 2000 - Present
True Color: Red = Band 1, Green = Band 4, Blue = Band 3
These images are called true-color or natural color because this combination of wavelengths is similar to what the human eye would see. The images are natural-looking images of land surface, oceanic and atmospheric features. The downside of this set of bands is that they tend to produce a hazy image.

my emhpasis

Thank you, the Wordview comparison is nice! I am still working on the question what is actually detected as leads with the AMSR2 method. Very likely it includes thicker types of grey ice and not only nilas and open water.

[uniquorn]

Overlay of sic-leads nov2pm, enlarged 474% onto worldview suomi/npp brightness temp, band15night, nov3  https://go.nasa.gov/3bE0l6H

Apart from the slight drift difference the representation is very good, though thinner lead width especially is somewhat exaggerated. sic-leads takes the enlargement well too.
click for defaults

Some interesting differences, sic-leads tends to pick out colder parts of the floes and sometimes highlights an older refrozen lead more than a newer one. I'll try to find an area with a few clear days to run an animation.

[oren]

Great comparison. Yeah it seems the sic-leads product somewhat exaggerates lead width but is overall quite accurate.

[uniquorn]

Probably a better way of doing this in pre-processing but here we go...

LEADS colours have been inverted, white set to fully transparent then overlaid onto SIC at 35% transparency to 'soften' them.
It reminds me of the earlier demo versions which, of course, we immediately enhanced

1. animation of the last few days
2. demo sic-leads

There are all those empty LUT's waiting to be filled

[uniquorn]

Thank you, the Wordview comparison is nice! I am still working on the question what is actually detected as leads with the AMSR2 method. Very likely it includes thicker types of grey ice and not only nilas and open water.

A bit off topic, but some here may be interested in what is going on beneath the land mask, particularly Greenland watchers (or Siberian snow). Accepted that the algorithm isn't tuned to snow/ice on land, but every little helps.

[uniquorn]

Ventured south to look at the Pine island and Thwaites glacier area
SIC, aug27-nov3  5.7MB

[uniquorn]

colour inverted overlaid leads at 35% transparent, 50% size
aug3-nov3  (11MB)

comparison with https://go.nasa.gov/3k704hv north of NSI

[uniquorn]

1. regional stats update
2. sic, oct30-nov4

[uniquorn]

Rightly some discussion elsewhere about the validity of averaging SIC images by intensity where the palette is not appropriate. The new AWI version allows us to use a stepped LUT that may better retain the concentration information throughout the averaging procedure. We can then change the LUT back to the better looking AWI product afterwards.

SIC-LEADS, nov6-11
gs stepped LUT by A-Team

Shorter time scale averages might be useful for other visual analysis, monthly average from 2013 onwards etc. Once the series has been fully processed.

SIC-LEADS, nov6-11, averaged

added SIC-LEADS, nov6-11, averaged, reverted to original LUT

and https://go.nasa.gov/3oksNAt of the Mclure strait for a lead comparison

While some may miss the graticule, I prefer to have a version without it. Providing a separate LEADS product is also highly desirable.

[uniquorn]

regional stats update

[uniquorn]

sic-leads comparison with sic, atlantic side, nov8-15.
A huge improvement in drift information and floe tracking through cloud. Worldview overlay onto bathy showing the long central lead is apparently unrelated to Gakkel Ridge. Position is midway between shelf break and Lomonosov, possible second lead setting up at ~3/4 harmonic.
Could just be coincidence, the possible parallel leads structure stretching north of Ellesmere I.
Fram funnel shows as a separate curved lead structures
https://go.nasa.gov/3cdCM5b
yep, should have brightened that up a bit

[uniquorn]

regional stats update

SIC, nov13-26, Bremen SMOS inset (7MB)
colour inverted leads overlaid at 33% transparent

[uniquorn]

regional stats update

SIC, nov20-dec3 (5.6MB)
colour/linear inverted LEADS overlaid at 33% transparent

[uniquorn]

sic-leads providing a good representation so far of the Nares Strait/Lincoln Sea arch dynamics.

Example from dec1-6

[uniquorn]

Bering and central Arctic drops, Hudson gains.

[uniquorn]

regional stats update

                                Total_e      Total_a
2021-12-07 00:00   10686801   10201773
2021-12-07 12:00   10727242   10210250
2021-12-08 00:00   10745572   10249900

[uniquorn]

Laptev comparison of sic-leads (nov15-dec9) and worldview https://go.nasa.gov/3Gsx4cs (dec1-9)

Mostly looking at the persistent coastal leads but also interesting to see more leads being detected offshore as the ice thickens (or changes type)

[uniquorn]

regional stats update

SIC, dec4-17 (7.5MB)
gimp/colour/linear inverted LEADS overlaid at 33% transparent

[uniquorn]

Using GIMP linear invert loses too much detail from the LEADS product. Here is an update using standard invert which inverts 'perceptually'
dec5-18 (8.4MB)
edit: reindexed

[uniquorn]

Will be interesting to see what the LEADS product makes of the Lincoln Sea tomorrow.
https://go.nasa.gov/3JcB5Eg  dec20-22

edit: why wait? The dec21PM already shows it

added a rammb, dec18-22 for comparison. (9.7MB)  https://col.st/A3jZN

[uniquorn]

Lincoln Sea still not settled down. This event looking similar to the one above.

1. sic-leads v106, jan1-7
2. rammb jan6-8  https://col.st/FL2pg
3. worldview, jan5-8  https://go.nasa.gov/3FcAjDP
4. Mercator (model) current 0m, sep21-jan7
http://static-bulletin.mercator-ocean.fr/img/9/psy4qv3r1/20220107/arc/A/psy4qv3r1_20220107_arc_current_0m.png

[uniquorn]

example of the LEADS product, oct27-nov1

the latter product contains the ice concentration minus half the lead fraction

Lars may appreciate feedback on ratio of the lead fraction applied. Leads do seem a bit overstated on the SIC-LEADS product, visually. But then, we are nearly always looking through thin cloud during daylight availability on worldview.

Corrected Reflectance (True Color)
Temporal coverage: 24 February 2000 - Present
True Color: Red = Band 1, Green = Band 4, Blue = Band 3
These images are called true-color or natural color because this combination of wavelengths is similar to what the human eye would see. The images are natural-looking images of land surface, oceanic and atmospheric features. The downside of this set of bands is that they tend to produce a hazy image.

my emhpasis

Thank you, the Wordview comparison is nice! I am still working on the question what is actually detected as leads with the AMSR2 method. Very likely it includes thicker types of grey ice and not only nilas and open water.

Maybe more clues to ice type when looking at the Antarctic. Here focusing on the pig/thwaites area. LEADS once again colour inverted and overlaid at 50% transparent this time as there is a lot more drift. B-22 occasionally visible as lower(?) concentration ice. I assume that's similar to the surface features or weather effects often seen in the Arctic amsr2 product.
aug3-jan8  (9MB)

[uniquorn]

A couple of sh masking issues.