Support the Arctic Sea Ice Forum and Blog

Author Topic: Test space  (Read 4522 times)

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #50 on: June 09, 2019, 05:28:38 PM »
I am testing below for differences between U Bremen's SMOS vs SMOS-SMAP maps for ice thinness. Recall the M in both stands for soil moisture; their data (like Ascat's) has been re-purposed for sea ice. The latter brings in synergistic data from a second satellite to correct and enhance the former.

Since the advent of v205 of SMOS-SMAP, there seems to be no justification for using plain SMOS, other than it has a much longer consistent archive (ie same algo versioning) and a one day  shorter lag-to-archive posting. SMOS is riddled with flash artifacts. Both are observational and vetted in situ, unlike Piomas.

SMOS-SMAP is provided at a vastly better resolution (2.83x), measured as Gimp pixel counts on the (polar stereographic) line between St Lawrence Island in the Bering Sea and Bear Island in the Barents (these are consistently visible in both pure imagery and in archival base maps).

That distance is 4738 km (2558 nm) as measured in WGS84 on Google Earth Pro. Accurate pixel counts are the key to re-sizing images to matching overlays:

satellite      pxl       %      ~km/pxl    ~km^2/pxl
SMOS          397.5    75.815   11.92      5.06
Ascat         524.3   100.000    9.04      6.67
OsiSaf       1075.5   204.711    4.41     13.66
SMOS-SMAP    1146.0   218.577    4.13     14.59
AMSR2        1197.6   228.419    3.96     15.24


Both satellites max out (to beige) in the central Arctic during the freeze season, as the ice pack quickly thickens beyond their 0.5m sensing capability. However, certain peripheral areas do not thicken to this depth at any time during recent winters. Thus SMOS-SMAP provides nuanced data during months when AMSR2, Ascat and WorldView see nothing but ice surface.

It's not possible yet to see how the two thinness maps compare during melt season, nor is it clear what they measure or conflate in summer, the possibilities being ice thinness, melt ponds, rafted ice, refrozen melted snow, slushy floes, liquid cloud water, or artifacts from passing storms.

Rather than throw the baby out with the bathwater (not use in summer), it might be better to take SMOS-SMAP colors as the thin dodgy ice, that ice most vulnerable to melt-out. This interpretation interpret very well with the peripheral location of thinness colors both during fall freeze-up of 2018 and into early June of 2019.

The mp4 below is a 266-day hybrid map showing SMOS from 15 Sep to 31 Dec 2018 concatenated to SMOS-SMAP from 01 Jan to 07 June 2019. I originally made it to 700 pixel width but because the center stays so dark and is so large, it views better at 550 width. While moderately difficult to make, the final product only takes up 4-5 MB at 16-20 fps frame rate.

Note the very rapid expansion of dodgy ice in the last few days in the Beaufort-Chukchi and above the Svalbard-FJL-SZ line.

The freeze/melt cycle is largely a story of peripheral ice freezing, moving with the wind, and then melting to an extent determined by whatever weather summer brought. In most years the sub-central ice pack is not noticeably affected (though in fact multi-year ice is on a disappearing trend).

This year the lower CAB has experienced unusual displacement towards the CAA, Fram and Beaufort arm. The area has been cloudy enough that the main region of thick old ice has rarely been visible in Worldview, though intact ice with brittle healed leads can be seen after enhancement.

The contribution of SMOS-SMAP is thus to the near-miss zone: ice that thinned to 0.5m and below but did not quite melt out (or become visible as low concentration by AMSR2) by end of season. It is likely more informative than sketchy albedo and melt pond products earlier in the season as these do not measure either top or bottom melt.

The last three weeks of Ascat are also showing weather streaking across the scene, sometimes leaving permanent effects on radar brightness (ice near-surface dielectric), both lightning (less polarizable constituents) and darkening (more saline or more liquid).
« Last Edit: June 11, 2019, 02:38:19 AM by A-Team »

be cause

  • ASIF Middle Class
  • Posts: 531
  • Citizenship .. a Lurker gets asylum
    • View Profile
  • Liked: 69
  • Likes Given: 77
Re: Test space
« Reply #51 on: June 11, 2019, 02:54:33 AM »
All I can say A .. I'm in awe !

  I've managed to negotiate double my GB's   now I can revisit your domain as often as I want :)

 thanks for your commitment .. and the humour .. I too shout 'over here !' .. bon noite   b.c.
2007 + 5 = 2012 + 4 = 2016 + 3 = 2019 ...

johnm33

  • ASIF Upper Class
  • Posts: 1198
    • View Profile
  • Liked: 70
  • Likes Given: 21
Re: Test space
« Reply #52 on: June 11, 2019, 02:51:06 PM »
Testing


A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #53 on: June 12, 2019, 01:57:33 AM »
Here I am testing the effectiveness of a backwards-running mp4 of the Nares funnel (or spillway), the ice in the Lincoln Sea caught up in its extended surface current (driven by sea level differences) above the Nares Strait.

The movie uses near-daily Sentinel-1 images furnished by DMI from 01 May to 11 June 2019. Be sure to set the movie option to 'loop'.

It would be feasible to extend the mp4 back to 15 Sept 2018 though that would require down-scaling and/or faster frame rates to keep file size manageable. The firehose effect may have pauses deep in winter and not persist the whole time, even though water under the ice does flow down the Nares year round at a "steady sectionally averaged mean of about 0.06 m/s, a volume flux of 0.72 million cubic meters per second" [A Münchow 2008].

Although DMI's interest is the waters around Greenland, the images do go far enough offshore to capture the entire funnel 165 km to the north and east, but not enough past Ellesmere Island to really determine where the exported ice originates (from along the coast or up towards the pole?).

In any event, the ice being lost is some of the thickest and oldest ice left in the Arctic Ocean. This ice sector is seldom directly set in motion by the wind (per Osisaf ice motion vectors) but this season it as been strongly pressed down against the western CAA coast by the persistent CW rotation (that has recently reversed in the Beaufort).

According to WorldView, the funnel had an area of 18,400 sq km on 10 Jun 2019 which is 1/54 of a million sq km. It takes roughly 48 days for the area shown in the funnel to flush. At this pace, some 140,000 sq km per year of ice area would be exported, only a fraction of ice loss via the Fram.

The Terra visual counterparts have quite different properties from Sentinel radar and even from the nominally identical WorldView scenes. The last 4-5 days have seen a peculiar darkening of funnel ice in Sentinel and Ascat radar; it leads to a stained glass look at optical wavelengths after reprocessing for feature-following in Gimp (crop before equalizing).

The large floe off Ellesmere entered the scene 48 days ago. It is losing ice on the margins but still is too big to fit down the strait at 37 km height.

For year-round flows through the Nares and the overall freshened water budget history of the Arctic Ocean, see:

https://journals.ametsoc.org/doi/full/10.1175/JPO-D-15-0093.1
https://journals.ametsoc.org/doi/full/10.1175/2007JCLI1748.1
« Last Edit: June 12, 2019, 09:06:02 PM by A-Team »

uniquorn

  • ASIF Middle Class
  • Posts: 996
    • View Profile
  • Liked: 485
  • Likes Given: 57
Re: Test space
« Reply #54 on: June 12, 2019, 02:32:11 PM »
<For year-round flows through the Nares> Thanks for these.
Testing the full size ascat mask from upthread with interferometry, 1 day difference, sep17-jun7. Poor choice of date storage and may possibly be improved further by using amsr2 open water mask. I think there is still some life left in it :)
used -crf 27 option to reduce file size
ffmpeg -i intil.gif -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -level 3 -crf 27 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" interf4.mp4

@johnm33. I like the moon phase idea
« Last Edit: June 12, 2019, 02:38:26 PM by uniquorn »

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #55 on: June 12, 2019, 09:04:41 PM »
On grabbing date overlays as in #54, there is an easy way to get control of the background making text easier to read. In ImageJ, if taking dates off (renamed or crop-able) file names:

-- early on, duplicate the stack of initial upload
-- crop it down to 13 x 50 or so
-- select all, fill all with sub-black, eg 10 instead of 0 on [0,255]
-- set text color to white
-- make montage, checks on labels and foreground color
-- slice montage back to new date stack, crop to text with 1-2 pixels of boundary
-- later, when you know final width of imagery, adjust canvas checking 'zero fill' and 'center'
-- set 'paste control' to black as transparent
-- then paste on imagery montage, position dates to suit, click to flatten in.
-- on mp4, the controller defaults to covering dates if on the bottom, have to mouse away.

Life is better in Gimp where a text file of vertical dates in monospaced font can be pasted/rendered in a blank image that is the right pixel multiple for number of layers. This allows both day number (Ascat) and calendar date (everything else) using https://www.epochconverter.com/days/2019. Adjust canvas size and use filmstrip to montage to a floating layer over the imagery montage. There is better control of font color and layer transparency so dates don't jump out too boldly.

It is feasible to do this just once for 2018-19 and crop to the date range of individual projects (attached).  This was a nuisance to make as "AI" insists it knows better than you how dates should be formatted, just like it knows better than you how to drive a car. The CSV starts off like this:

01 01 19   Tue    Day 01   01 01 18   Mon   Day 01
02 01 19   Wed   Day 02   02 01 18   Tue   Day 02
« Last Edit: June 12, 2019, 10:10:18 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #56 on: June 12, 2019, 11:19:45 PM »
Here I am testing integration of satellite imagery, almost all of which comes to us in polar stereographic projection, into google earth pro (GEP) which can import and align images but is itself in plate carrée projection.

I drew a polygon in the general shape of remnant old thick ice the CAB next to the CAA and adjusted its vertices to make it a million sq km in area. (GEP is quite good at adjusting boundaries and computing WGS84 lengths, perimeters and areas). This draft polygon was a moderately good fit to the main area of whitish Ascat ice in the same region (animation shows stages of this).

The GEP import 'handle' interface is a nightmare for images of large areas, especially those that include the North Pole. It is essential to know the lat-long coordinates of the corners of the images before importing them.

The attached Ascat is cropped to four corners whose lat,lon can be collected in GEP. If it could be overlaid on the GEP earth and a polygon drawn over it, the area of core remnant ice could be determined under various scenarios. Maybe we could even make Nasa-style ice movie as seen from space by importing a time series of Ascat. However their area is just too big given the differences in projections for importation to work.

Now GDAL is an open source cartographic program that can change projectons on png images (ie warp them), with code that looks something like this (EPSG catalog numbers needed for pc and ps):

Step1) gdal_translate -of Gtiff -a_srs EPSG:4326 pc_map_piece.png pc_map_piece.tiff
Step2) gdalwarp -s_srs EPSG:4326 -t_srs EPSG:3995 pc_map_piece.tiff ps_map_piece.tiff

However, I looked for something easier like a free online tool that would compute areas of polygons on PS gifs given the North Pole center and the bounding edge latitude. However all the offerings involved deep dives into software manuals just to chase down a one-time fixed warp.

PS is conformal and so not equal area but is not really that far off at narrow ranges of high latitude, especially if subtracting two areas. So I looked at just counting pixels within a given selection (~polygon) to see what fraction of the Arctic Basin's pixels they constituted.

Since the latter's area is 7.109 million sq km as drawn in GEP coastlines, the product gives a low tech estimate of the selection's area for each commonly used satellite image (see table in #50).

Three minor points here: the world is not going over to NSIDC's lambert azimuthal equal area projection, few data sources provide necessary geo-referenced Geo2D files that Panoply could recast into any of its 120 stock projections and output as kmz, and the satellite archival services could but won't process raw satellite data directly into GEP format which would greatly improve resolution.

What I really wanted to get at here is a melting season essential: how much of the thick multi-year ice is getting chewed up? The same regions of peripheral ice freeze and thas each year with weather variation and an Arctic Amplification downward trend but that is largely beside the point.

It is only the attack on older ice that matters in terms of a blowout summer from which the ice will never recover that would be catastrophic for the planetary climate. Unprecedented ce motion so far in 2018-19 has set the ice up for a weather pattern (warmth, cyclone, waves) that could make a very large dent in the core and so herald the end: older ice has been advected far from its zones of stability into regions where it surely melt out or be exported by September.
« Last Edit: June 12, 2019, 11:43:58 PM by A-Team »

uniquorn

  • ASIF Middle Class
  • Posts: 996
    • View Profile
  • Liked: 485
  • Likes Given: 57
Re: Test space
« Reply #57 on: June 12, 2019, 11:32:20 PM »
Nice.^
Ran out of time today attempting the open water mask so experimented with a simpler and possibly softer option of amsr2 white to alpha overlaid at 50% over ascat then remasked with the circle. This dulls the ascat weather over water and adds more information from the amsr2 lower concentration. Cheated with the amsr2 date and went from blue to 8bit.
The scaling numbers upthread make it much easier to overlay.

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #58 on: June 14, 2019, 12:43:00 AM »
I need to look around for a avi --> mp4 converter that allows lower image quality -- the 271 day sea ice motion movie below is really pushing it on forum file size. Didn't help that much to go grayscale or faster fps, it seems like number of days is the real driver. Doesn't really work here to skip every other day.

Note Osisaf is good on both open and dodgy ice in addtion to its sea ice motion vectors. The netCDF files are in an open archive and Panoply has a five minute walk-through tutorial on how to make vectors from such files.

https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Create%20Wind%20Vector%20Plots%20with%20Panoply

Note the near-complete absence of Beaufort Gyre ice circulation the last nine months. Magenta near the pole represents persistent pole hole uncertainty. Lots of other things here to see if you set on loop.

Keep in mind, Osisaf is the real thing, not guesswork or weak proxy like sea level pressure or 2m winds. The latter are just models, not observed and not factually grounded since there are no instruments anywhere in the 7,109,000 sq km of Arctic Ocean. Osisaf is taken directly from daily radar fly-overs.

The ice motion vectors are displayed like airport runway wind roses in the bottom image. The Beaufort has the strongest winds blowing ice west off the Alaskan coast. There is also a consistent CW rotation off Siberia that continues to the Fram. The CAB off the CAA sees very little wind and very little ice motion.

That transparency ^ is a nice touch, mellows the colors. Note AMSR2 has a fair number of transient artifacts (diffuse middle blues) in the middle of the pack that need suppression. AMSR2 actually makes very little use of its 100-color palette: open water blue + 100% concentration white make up a huge percentage.

You cannot reduce to say 5 or 10 colors by posterization as the palette is non-linear; however that is easily done manually with the color wand radius on a palette in the montage. Here you can set the Gimp color wand to radius 30, click on the bluest blue and it will highlight everything below 15%, similarly with white 255. I do this on the montage to process all the day layers simultaneously, then replace colors and re-slice.

I've been wanting to reduce these AMSR2 to three colors (low, high, all the rest), then overlay all but the open water with SMOS-SNAP thinness (which has to be a hybrid with SMOS rfi for dates prior to 01 Jan 19).

Circular and elliptical masks not only cut down on distracting off-topic regions but also significantly reduce file size because mp4 compression is 3D and would pick up on the z constancy whatever the frame rate. Thus a circle of radius 1 has area 3.1415 but its bounding square 2x2 has area 4, so there’s a 21.5% savings in file size, eg 4 mb instead of 5. Gimp provides good control over ellipses so once a bounding box is drawn, the ratio of sides can be used to draw the ellipse inside.

Gimp has a really quick way to make multiple copies of a single file and montage it horizontally to a masking layer: simply hold down command-shift-D and watch the layer count grow in the title bar. If you overshoot, command-Z undo to the right number. Then hit ‘filmstrip’ with all its features shut off. Picking a color or transparency with the non-contiguous color wand then lets you fill or otherwise transfer the mask to layers above or below, for example drop a circular mask.

Ascats have a number of programmatic peculiarities: the use of 1 for black instead of 0, poor placement of the sigma-zero brightness palette (which is useless as it is just standard 256 grayscale gradient) an over-contrasted small rectangle off the upper right hand corner of the pole hole that they at first denied though later admitted to but refused to fix (a 95% brightness squeeze is all it needs), an 1170 x 1170 size for an image really of dimension 1153 x 1153 according to ‘zealous crop’ (which is good because odd numbers are essential to have a single pixel define the North Pole namely 577,577 in the cropped or 585,585 in the original.

Note the 9 x 9 pole hole is off-center at 576,576 and incorrectly dithered. It would have been better had they used a white dot or cross-hairs like AMSR2, indicated the meridional termini, specified the bounding latitude and EPSG, and provided an accurate land mask. Had a few basic cartographic parameters like radial pixls per degree of latitude, precise alignment with other satelllite data sources could be immediately automated.

There are a fair number of defective or missing files in the satellite archives. This doesn’t matter too much in a single time series but in multi-source overlays it does. There needs to be 271 files between 15 Sept 2018 and 12 June 2019 in each layer or there will be mismatches of days.. For missing dates, I simply duplicate the earlier date and give it a faux file name. If two consecutive dates are missing, I duplicate the later date for the second.

To repair a file with some missing data (typically a black polygon), I layer it over the preceding date, select and delete its missing data which lets the earlier data show through, then use ‘make new layer from visible’ in Gimp and replace the original defective file with it.

Ascat defectives:

2018 338.gif
2019 005.gif
2019 006.gif
2019 007.gif
2019 044.gif
2019 045.gif
2019 059.gif
2019 068.gif
2019 069.gif
2019 070.gif
2019 147.gif

AMSR2 defectives:

Arc_20181216_res3.125.png
Arc_20190127_res3.125.png
Arc_20190222_res3.125.png
Arc_20190316_res3.125.png
Arc_20190318_res3.125.png

SMOS-SMAP defectives:

2019 04 19.png
2019 04 20.png
2019 04 21.png
2019 06 08.png

SMOS defectives:
none

Osisaf defectives:
none

Sentinel-1 DMI:
1-2 per week
« Last Edit: June 14, 2019, 03:29:49 AM by A-Team »

sidd

  • ASIF Governor
  • Posts: 4535
    • View Profile
  • Liked: 155
  • Likes Given: 0
Re: Test space
« Reply #59 on: June 14, 2019, 06:54:12 AM »
Re: avi --> mp4

ffmpeg ?

sidd

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #60 on: June 14, 2019, 08:00:48 AM »
>> avi --> mp4  ffmpeg ?

Correct, sidd. Some people above are doing command line. However there are dozens of point and click web conversion front ends, all of which i would guess are just ffmpeg under the hood. The better sites presumably implement the most useful commands in ffmpeg.

We may need to go to thumbnails on these forums with outside links to real files on the cloud. Since not everyone has an account there, the question becomes one site with a shared password. However that might encounter vandalism, non-forum use, or site policy issues.

As we go to longer time series, the files become far too big for gifs. I'm concerned that we are hitting the wall with mp4s too as we try to do whole years. I find the mp4s don't load that consistently now which may be an aspect of forum software rather than internet connection.

I have much nicer versions on my desktop than anything I can distribute in a post. A single forum page can really become unwieldy with 30 posts, each of which might have 2-3 mp4s or big images. And what is the point if they are just looking on a phone screen?

With 4-5 overlays, the serious mp4 providers are manipulating a couple thousand individual files. Those can really drag on performance before they can be cropped and resized. Algos that were fast enough before don't scale well with load. The slightest keyboard carelessness can trigger long cpu hang-ups. The new March 2019 iMacs are no faster really than the 2014's.

However there are workarounds. I think we need to start with where we need to be on the science side and then figure out how to get there, without compromising the data but maintaining some level of practicality and accessibility.

Attached is just a nice WorldView png of Wrangell and the ESS that butts up against forum width constraints, a related issue with mp4s that want 720 pixels width.

sidd

  • ASIF Governor
  • Posts: 4535
    • View Profile
  • Liked: 155
  • Likes Given: 0
Re: Test space
« Reply #61 on: June 14, 2019, 09:25:34 AM »
Some people on this forum have access to pretty serious hardware. But diskspace is a big issue. I see archival becoming a problem already, even with mp4 compression.

Bandwidth is quite cheap, at least if we limit to the users of this forum, but that might be an issue if access is made public. coralcache might be a solution for bandwidth.

Anyone know inside contacts at ESA or panagea ? they might be willing to provide public access archival. I shall ask around the US GFDL and NOAA and edu sites where i know some people. Someone might be willing to donate disk at least, if not too much bandwidth.

I know that microsoft, google, and amazon have discounted or free programs for science. But i hate relying on commercial entity.

archive.org is another possibility.

sidd

A-Team

  • ASIF Upper Class
  • Posts: 2300
    • View Profile
  • Liked: 242
  • Likes Given: 14
Re: Test space
« Reply #62 on: June 15, 2019, 01:13:14 AM »
Thx for looking into this, Sidd. It would be great if we could get institutional space somewhere since we are simply reworking and crediting their open source archival graphical material.

NASA SVS has posting very advanced Arctic visualizations for years. They have a real doozy out now that has to be seen to be believed. I am testing various adaptations of it that might work for Arctic sea ice, though there's a tradeoff between eye-catching glitz and actually conveying information.

Climate science journals all refuse to carry gifs, even the arXiv pre-print server won't. Talking here about a ubiquitous graphics standard adopted in 1989. Some do allow .mov but those must be in Supplemental and are not peer-reviewed. Notably, even 'online only' journals still expect you to print a pdf! So there is no distribution outlet there.

http://svs.gsfc.nasa.gov/30701 eight active layers inside a beachball!

Slice 1: Sea Surface Currents & Temperature (ECCO2 model)
Slice 2: Winds (GEOS-5 model)
Slice 3: Precipitable Water (GEOS-5 model)
Slice 4: Clouds (GEOS-5 model)
Slice 5: Precipitation (IMERG data)
Slice 6: Soil Moisture (SMAP data)
Slice 7: Biosphere (Multiple satellite datasets)
Slice 8: Blue Marble (MODIS data)

Meanwhile back at the melt season wx forum, some remarkable happenings are being overlooked, possibly because they are merely observed and not predicted by the alternative reality provided by models.

-- The Nares Strait won't by blocked by the large floe along Ellesmere; it is weeks away and already cracking up according to optical and radar imagery.

-- The disintegration of old thick ice northeast of Banks Island has accelerated. The region of isolated floes now extends 230 km poleward of the central CAA coast (and possibly farther as clouds are masking the WV view).

-- Ascat has been losing its ability to image features for the last four days, especially in the critical central CAB region. This may be due to melt ponding, moist weather systems, or possibly loss of surface roughness. It may still be possible to track the massive open lead system (cracks). It may also get better if previous weather returns. AMSR2 has not been affected.

-- Smos-Smap is showing massive colors here which cannot plausibly represent ice thinness; however its artifacts seem fewer than Smos alone. Both are showing large scale color development along and well out from the Siberian coast.

-- The last 25 days are shown. These could be posted as a 7MB gif instead of mp4; gifs are more desirable as they capture each day's data in a separate frame. mp4 to gif does not work well at recovering this information.

-- Osisaf was still able to post its gridded sea ice motion vectors without Ascat but with more uncertainty (magenta arrows). I have never been able to locate the other png archives it uses.
« Last Edit: June 15, 2019, 07:36:32 AM by A-Team »

sidd

  • ASIF Governor
  • Posts: 4535
    • View Profile
  • Liked: 155
  • Likes Given: 0
Re: Test space
« Reply #63 on: June 15, 2019, 06:04:05 AM »
One possibility for bandwidth might be bittorrent. a lot of unix distributions use it to distribute entire operating systems and applications.

sidd