Please support this Forum and Neven's Blog

Author Topic: Ice Concentration Images and Animations  (Read 3532 times)

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #50 on: August 07, 2017, 08:35:19 AM »
Speaking of which, I posted a 2 week animation on the 2017 melting thread earlier today, up to Aug 5th (before Aug 6th was available), but here is one that includes Aug 6th. Looks like another big drop on the CAB north of the Chukchi, inside the ice perimiter.

This filter is 5-days most recent under 90 (v1.2 in the descriptions above).

Also, for comparison's sake, I have made the start (and end) dates the same as that of the animation posted by Romett in the 2017 thread:
http://forum.arctic-sea-ice.net/index.php/topic,1834.msg124167.html#msg124167 .
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #51 on: August 07, 2017, 08:44:13 AM »
Vs. unfiltered:
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #52 on: August 07, 2017, 08:51:30 AM »
vs. median:
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #53 on: August 07, 2017, 08:58:05 AM »
And to emphasize the point even further, have a look at the graphic that Wipneus just posted: http://forum.arctic-sea-ice.net/index.php/topic,382.msg124175.html#msg124175 .
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Neven

  • Administrator
  • ASIF Governor
  • *****
  • Posts: 4051
    • View Profile
    • Arctic Sea Ice Blog
Re: Ice Concentration Images and Animations
« Reply #54 on: August 07, 2017, 09:23:47 AM »
Like I said before, I think these animations are particularly useful around now, in August, because the effect of melt ponding becomes negligible (as they either drain or freeze over). Either way, it's a cool method, even though I have trouble wrapping my mind around it (but that's me).
Il faut cultiver notre jardin

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #55 on: August 07, 2017, 09:43:26 AM »
even though I have trouble wrapping my mind around it (but that's me).

I probably just didn't explain it very well. It works like this. A period of days is set (e.g. 3 or 5). For each pixel, the concentration values for the 3 or 5 days (or whatever) ending in the current day are looked at. To filter it, some algorithm is applied to that set of values, e.g. minimum or median, and the resulting value is displayed on the filtered map. Is that any clearer?

My original idea was basically just to try remove the fast-moving "purple noise" from the maps, which I find very distracting.

I agree that these are mostly useful in areas undergoing active melt. Not sure about the melt ponds aspect yet -- there may be some temporal pattern that can be used to help distinguish between melt ponds and real water, or maybe not.

For instance, the "breathing" I mentioned earlier that is currently happening on the Pacific side is I think an indicator of a real phenomenon, which has been mentioned elsewhere ("bounce back"). Obviously there isn't new ice forming, but perhaps some ice is drying out after being drenched by waves, which would be a useful thing to know about.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

magnamentis

  • ASIF Upper Class
  • Posts: 1229
    • View Profile
    • Nexpaq Modular ARA iOS Software Mobile Computing Phones Philosophy Ethics Numerology Mikrocirkulation Vaskular Therapie Gesundheit Blut Gesundheit Schmerzen Multipelsklerose Diabetes Immunsystem Fibromyalgie
Re: Ice Concentration Images and Animations
« Reply #56 on: August 07, 2017, 11:50:39 AM »
Thanks Sterks. I have similar thoughts. For long-term animations the median is probably is best, nice and smooth. It may even filter out some of the low concentration artifacts in the original images. On the other hand, it is relatively insensitive to day-to-day changes. For instance, if you watch the pacific side, you can see it "breathing" in some of the other animations, which I think does reflect real physical changes to the ice (although maybe not exactly just concentration). So for short-term animations, I am leaning towards one of the noisier versions (to better watch the horse race  ;D ).

your entire idea and approach with using filters on those graphs is a great improvement to get the bigger picture of how things went and are currently developing. great thanks for that, something really new and noteworthy. wouldn't be surprised if sooner or later some of the idea would be adopted by one or several of the main data providers.
http://magnamentis.com
Knowledge, Understanding & Insight Are Among The Best Sources For Personal Freedom & Vitality !

sedziobs

  • ASIF Lurker
  • Posts: 94
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #57 on: August 07, 2017, 04:03:57 PM »
These animations are a really nice addition the forum.  Thanks gd2 for taking on this project.

Would it make sense to subtract 2 days from the label in the median method?  It seems like the August 6 image is more representative of the ice on August 4, even though it is influenced by data past that date.  As you said earlier, perhaps the median method is better suited to longer time periods, where the exact date is less important.  I think some higher speed animations that span multiple months would be interesting to see.  So, overall I agree that the minimum method is better for time spans of less than a week, and the median method for longer.

One suggestion I have is to add a pause on the last frame of each animation.   

magnamentis

  • ASIF Upper Class
  • Posts: 1229
    • View Profile
    • Nexpaq Modular ARA iOS Software Mobile Computing Phones Philosophy Ethics Numerology Mikrocirkulation Vaskular Therapie Gesundheit Blut Gesundheit Schmerzen Multipelsklerose Diabetes Immunsystem Fibromyalgie
Re: Ice Concentration Images and Animations
« Reply #58 on: August 07, 2017, 06:06:31 PM »
These animations are a really nice addition the forum.  Thanks gd2 for taking on this project.

One suggestion I have is to add a pause on the last frame of each animation.
[/b]

this one can easily suggest to 99% of all the gif makers, not only is it good for the eyes to rest for second or two to be prepared to digest the succession but also it would be much easier to distinguish the last from the first picture, including the content of the two.
http://magnamentis.com
Knowledge, Understanding & Insight Are Among The Best Sources For Personal Freedom & Vitality !

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #59 on: August 07, 2017, 09:16:10 PM »
Thanks sedziobs and magnamentis for your suggestions.

Regarding the median filter, while I agree that it tends to look somewhat like older pictures, what it actually shows is the middle value of the last N days (half way between max and min), not the value of the middle day. In regions where there is an ongoing melt, this would indeed make the image look similar to a smoothed version of the middle day. So I see what you're saying, but I'm not sure relabelling would make sense (to me).

Pausing is definitely a good idea. I have actually been adding a brief pause (first and last frame are doubled), clearly it needs to be longer. :)
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #60 on: August 08, 2017, 01:21:37 AM »
Here is an overlay of the latest (Aug 6th) v1.2 Bremen filter with a Wipneus PIOMAS thickness map and a Cryosat map that Michael just posted showing the difference between PIOMAS thickness and what Cryosat detected, both from April. (I haven't been able to find a map directly showing the Cryosat measured thickness.)

Visually, it looks to me like the Cryosat April thickness does a very good job at predicting the melt at this point as well as other features of the Bremen map, including the high concentration arm extending towards the ESS.

http://forum.arctic-sea-ice.net/index.php/topic,119.msg112164.html#msg112164
http://forum.arctic-sea-ice.net/index.php/topic,8.msg124238.html#msg124238
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #61 on: August 08, 2017, 06:10:43 AM »
Found a good example of a current low concentration anomaly, a yellow streak in a 90+% concentration area of the Beaufort that appears to be caused by an odd-looking cloud that barely moves over a period of 2 days.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Peter Ellis

  • ASIF Middle Class
  • Posts: 548
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #62 on: August 08, 2017, 09:59:33 AM »
Thanks for preparing all the different versions - to my mind the median looks clearest and is also the most defensible from a mathematical standpoint as it's not making any unjustified assumptions about the nature and direction of the noise.

It's worth pointing out that ALL the microwave ice products do something like this internally anyway when they average the satellite readings over 24 hours (or 48, or whatever window they use) - plus a lot more pre-processing besides to compare different wavelengths etc in order to help filter out clouds.

In terms of relating this data to the official quantitated products out there, a 5-day window makes most sense to me, to line up with the NSIDC's graph that uses 5-day averaging (which is probably not median-based, but if the noise is randomly distributed that should make very little odds).

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #63 on: August 08, 2017, 07:56:23 PM »
You're welcome Peter. It's a fun project and the products are useful at least to me in figuring out what's going on with the ice.

I agree that the median filter is the most mathematically elegant and gives the smoothest result. Also, unlike the other filters, it would work equally well during refreeze (although ice concentration probably isn't very useful during refreeze, since I guess it will quickly become 100% in most areas). However, it suffers from the disadvantage that it is slow to respond to daily changes.

As to justification, seaice.de offered a technical justification here:
http://forum.arctic-sea-ice.net/index.php/topic,1834.msg123495.html#msg123495

Personally, I think the results are justification enough -- at least for use as an amateur tool. Compare the filtered images to the originals and to other evidence of the ice concentration such as satellite images. I find that the filters succeed in removing the obvious, fast-moving noise (apparently caused by cloud) and allow me to better see where the ice is thickening, thinning, or approaching the 15% threshold.

To demonstrate this again, attached is a gif of recent days, unfiltered. Look for example at the ice near the Chukchi/ESS around July 28 and again around Aug 4. Large areas become purple for a day or so and then revert back to lower concentrations. If you check WorldView, you will see that these 'purple flashes' correspond to areas and dates of thick cloud cover.

Then look at the filtered versions -- attached to following posts. They eliminate most of the purple flashes, and instead show a record of an earlier value (exactly what value they show depends on the algorithm).

This approach obviously would do a bad job at reflecting reality if low concentration anomalies were anywhere near as pervasive as high concentration anomalies. But they don't seem to be. I have been searching for them and although I do find one occasionally (see previous posts), they are rare. Indeed, this can be easily verified by looking at the high concentration area near the CAA. If low concentration artifacts were common, the filtered images would not succeed in retaining this large purple area. But, apart from a few small blemishes, they do succeed.

For additional justification about the "direction of the noise", I tried applying some inverse filters. That is, instead of for example filtering out concentrations above 90%, I filtered below 90%. The results of this exercise do not come anywhere near to reality and show just how common high concentration artifacts really are. (See following posts.)

Here is the unfiltered version:
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #64 on: August 08, 2017, 08:10:01 PM »
Median (5 days):
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #65 on: August 08, 2017, 08:18:03 PM »
Most recent under 90% (5 days).

Compared to the median version above, this version does a better job at showing daily changes, for example the alternating loss and recovery cycle ("breathing") on the Pacific side, and the big extent gaps forming on the last frame (Aug 7). That's why I prefer this filter for daily use. Median is better suited for long-term trends.

By the way, I found a bug in a boundary condition which resulted in red (85%) being filtered out in previous versions. Fixed here. Also, in previous versions I had been replacing dark purple with lighter purple in some cases, which I am no longer doing.
« Last Edit: August 08, 2017, 08:23:12 PM by greatdying2 »
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Sterks

  • ASIF Citizen
  • Posts: 149
  • Member # 1000
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #66 on: August 08, 2017, 08:29:21 PM »
This is great
There is, however, an issue which is how filter affects motion (translation or massive drift as Tor Benjar points out it happened today).
See, when a package translates and one uses a filter that does not take into account advective phenomena such as flows, drifts, or waves, funny things may result. The median filter just blurs the package in the direction of the movement. But a selective filter would tend to maintain a "wake" behind the translating package if it is low concentration (just like the ping pong character moving in the old phosphorescent monitors), and tend to make disappear a high concentration package.
All this would be difficult to evaluate in this animations, admittingly, but this is no artifice to be eliminated: drift exists and is very real, the same for high concentration or low concentration ice.
Excuse the long explanation

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #67 on: August 08, 2017, 08:30:45 PM »
I also tried most recent under 95%. The results are passable but not as useful I think as the 90% threshold.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

magnamentis

  • ASIF Upper Class
  • Posts: 1229
    • View Profile
    • Nexpaq Modular ARA iOS Software Mobile Computing Phones Philosophy Ethics Numerology Mikrocirkulation Vaskular Therapie Gesundheit Blut Gesundheit Schmerzen Multipelsklerose Diabetes Immunsystem Fibromyalgie
Re: Ice Concentration Images and Animations
« Reply #68 on: August 08, 2017, 08:34:54 PM »

Then look at the filtered versions -- attached to following posts. They eliminate most of the purple flashes, and instead show a record of an earlier value (exactly what value they show depends on the algorithm).

i'd have to repeat myself but that exactly makes this so valuable for many of us and considering the consistency of retread by getting rid of the noise makes it a real tool that provides better information about the ice than any other i have seen. laymen or not, facts remain exacxtly that and these images show the short/mid term development in a most comprehensive way, at least for me and some others as it seems.
http://magnamentis.com
Knowledge, Understanding & Insight Are Among The Best Sources For Personal Freedom & Vitality !

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #69 on: August 08, 2017, 08:36:11 PM »
This is great
There is, however, an issue which is how filter affects motion (translation or massive drift as Tor Benjar points out it happened today).
See, when a package translates and one uses a filter that does not take into account advective phenomena such as flows, drifts, or waves, funny things may result. The median filter just blurs the package in the direction of the movement. But a selective filter would tend to maintain a "wake" behind the translating package if it is low concentration (just like the ping pong character moving in the old phosphorescent monitors), and tend to make disappear a high concentration package.
All this would be difficult to evaluate in this animations, admittingly, but this is no artifice to be eliminated: drift exists and is very real, the same for high concentration or low concentration ice.
Excuse the long explanation

Good point. For drifting low concentration ice, the "most recent" filters can capture it -- provided it has not been obscured by a cloud artifact -- because they just use the most recent value as long as it's below (e.g. 90%). These filters would however fail for high concentration ice drifting quickly through low concentration areas. The high concentration ice would basically get erased by the filter if it was moving quickly enough.

Edit: Oh I see now what you mean by a wake. Yes, low concentration ice drifting through a high concentration area would indeed leave a low concentration wake when using these filters.

Edit2: I think I should compile a list of caveats to post along with these filtered images, which should include at least: 1.) Low concentration cloud artifacts are exaggerated; 2.) High concentration ice that is moving quickly through low concentration ice can be erased by the filter; and 3.) Low concentration ice that is moving quickly through high concentration ice can leave a wake.

Any additional caveats?
« Last Edit: August 08, 2017, 08:46:47 PM by greatdying2 »
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #70 on: August 08, 2017, 08:48:26 PM »
facts remain exacxtly that and these images show the short/mid term development in a most comprehensive way

Thanks magnamentis, I'm glad that I'm not the only one who finds them useful!  ;D
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Peter Ellis

  • ASIF Middle Class
  • Posts: 548
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #71 on: August 08, 2017, 08:51:55 PM »
Most recent under 90% (5 days).

Compared to the median version above, this version does a better job at showing daily changes, for example the alternating loss and recovery cycle ("breathing") on the Pacific side

What is your basis for believing this apparent cycle (i.e. where an unusually high/low value goes away the next day) is real, and therefore this filter is better?  There seems to be no plausible physical mechanism that would cause it. On the contrary, this is exactly what you expect from random noise. Large excursions away from the true value will typically revert towards the mean the following day.

Edit:  "Most recent under 90" also clearly has a problem in areas where the true concentration is over 90, such as the area around 135W, 85-90N.  You can clearly see a spiral artifact appear over the course of 1-2 August as the core of the weather system passes overhead.  This then leaves a false "deposit" of low concentration pixels that lasts for several days - because the real concentration is over 95% and thus cannot overwrite the false low values.  The artifact is even larger (and just as false) in the "Most recent under 95" animation.  If you look at the Earthdata images in the right wavelengths, you can clearly see that the ice sheet in this region is unbroken and snow covered, at ~100% concentration.
https://go.nasa.gov/2vM23QD
« Last Edit: August 08, 2017, 09:02:36 PM by Peter Ellis »

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #72 on: August 08, 2017, 09:21:06 PM »
Most recent under 90% (5 days).

Compared to the median version above, this version does a better job at showing daily changes, for example the alternating loss and recovery cycle ("breathing") on the Pacific side

What is your basis for believing this apparent cycle (i.e. where an unusually high/low value goes away the next day) is real, and therefore this filter is better?  There seems to be no plausible physical mechanism that would cause it. On the contrary, this is exactly what you expect from random noise. Large excursions away from the true value will typically revert towards the mean the following day.

These occur after big storms, which are very real. Typically the ice concentration appears to drop sharply immediately after the storm, then rebound somewhat (and afterwards often continue to fall). I am not certain exactly what physical process is reflected in the rebound, but it is a pattern that can be predicted, not random noise. As I said in an earlier post, it's clearly not "real" in the sense that new ice is not forming, but as a guess it may result from ice that has been drenched in waves drying out and thus is useful to detect. If it was just random noise, the timing wouldn't coincide with storms.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Peter Ellis

  • ASIF Middle Class
  • Posts: 548
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #73 on: August 08, 2017, 09:25:34 PM »
These occur after big storms, which are very real. Typically the ice concentration appears to drop sharply immediately after the storm, then rebound somewhat (and afterwards often continue to fall). I am not certain exactly what physical process is reflected in the rebound, but it is a pattern that can be predicted, not random noise.
Storms also have more atmospheric moisture - both as clouds and also just as more humid air.  So this pattern fits with a false low value (caused by cloud / atmospheric moisture) that then goes away as the clouds move, followed by continued decline - because it's melt season.  Personally I think this is the sort of process that ought to be filtered out if we are interested in real melting.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #74 on: August 08, 2017, 09:29:27 PM »
Edit:  "Most recent under 90" also clearly has a problem in areas where the true concentration is over 90, such as the area around 135W, 85-90N.  You can clearly see a spiral artifact appear over the course of 1-2 August as the core of the weather system passes overhead.  This then leaves a false "deposit" of low concentration pixels that lasts for several days - because the real concentration is over 95% and thus cannot overwrite the false low values.  The artifact is even larger (and just as false) in the "Most recent under 95" animation.  If you look at the Earthdata images in the right wavelengths, you can clearly see that the ice sheet in this region is unbroken and snow covered, at ~100% concentration.
https://go.nasa.gov/2vM23QD

Yes, I agree that this is an artifact. Furthermore, basically all of the "yellow streaks" in the "large purple area" are artifacts (as I suggested above).

Luckily, these are easy to see and to ignore. And besides, they are not in an area of much interest -- unless someone thinks a polynya might be forming here. The main purpose of these filters is to better watch the evolution of marginal ice zones during the melt season without being distracted by frequent cloud artifacts.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #75 on: August 08, 2017, 09:34:18 PM »
These occur after big storms, which are very real. Typically the ice concentration appears to drop sharply immediately after the storm, then rebound somewhat (and afterwards often continue to fall). I am not certain exactly what physical process is reflected in the rebound, but it is a pattern that can be predicted, not random noise.
Storms also have more atmospheric moisture - both as clouds and also just as more humid air.  So this pattern fits with a false low value (caused by cloud / atmospheric moisture) that then goes away as the clouds move, followed by continued decline - because it's melt season.  Personally I think this is the sort of process that ought to be filtered out if we are interested in real melting.

Yes and no. I think different filters are better for different different things. But you make a good argument, so how about this. In future I will post (at least) 3 versions: unmodified, most recent under 90, and median. Then people can look at their favourite, or even better,  compare them and draw their own conclusions.  :)
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #76 on: August 09, 2017, 12:03:09 AM »
I forgot, I was going to post a version to show what happens if you select the most recent over 90% instead of under. Here it is (5 days).

Needless to say, this map is not very useful (except in helping to understand artifacts).
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #77 on: August 09, 2017, 07:14:06 PM »
It is instructive to compare these different versions with this animation produced by Wipneus:

http://forum.arctic-sea-ice.net/index.php/topic,382.msg124478.html#msg124478
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #78 on: August 13, 2017, 07:12:38 PM »
Wipneus just posted a very useful animation of the basin ice:
http://forum.arctic-sea-ice.net/index.php/topic,382.msg125109.html#msg125109

For me, this re-confirms the usefulness of the last-under-90 filter. Take a good look at the details, all across the basin. The filtered image looks like reality.

The median filter does quite well too, but it is a bit blurry, a bit behind, and I think overestimates the concentrations a bit (because high concentration artifacts are far more frequent than low concentration ones -- I wonder if maybe a quartile filter might work... hmmm).

Attached are the latest full-sized last-under-90 and median filter images (5-day). (I'll leave it to you to guess which is which.)
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #79 on: August 13, 2017, 08:22:36 PM »
For fun, here are the first (Q1) and third quartiles (Q3). Compare to the median above.

The period is 5 days, so Q1 is the second lowest value and Q3 the second highest. (Median is the third lowest / third highest.)
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #80 on: August 13, 2017, 08:34:34 PM »
Actually, I like Q1. It is basically a not-quite-minimum filter. Does a little smoothing, in regions of active melt is only 1 day "behind", filters out low concentration 1-day artifacts, and filters out high concentration artifacts even if they persist for 3 of 5 days.

Attached is Q1 for the last 2 weeks.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Tor Bejnar

  • ASIF Upper Class
  • Posts: 1493
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #81 on: August 13, 2017, 08:46:45 PM »
I'm enjoying your experimentation.  Can you put a numbers to the ratio of high and low concentration artifacts?  ("high concentration artifacts are far more frequent than low concentration ones")  Is this fairly consistent over different 5-day periods?  Armed with this information, you might find a sweet spot.  If the ratio was 2:1, then maybe a 66% filter would be 'best' (or there may be a better mathematically determined sweet spot, given a specific ratio). 
Arctic ice is healthy for children and other living things.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #82 on: August 13, 2017, 08:55:00 PM »
Hmm... it's a good idea to try to quantify that. So far it has just been from eyeballing it -- although backed up by the results, compare for example Q1 vs. Wipneus animation and Q3 vs. the same animation.

To quantify it, we would have to define mathematically what is meant by an artifact, which could be tricky. I suppose I could try to apply some kind of extreme value test... but that would assume that the artifacts only occur infrequently, which may not be true in this case.

Let me think about it and get back to you in a day or two. I don't have a lot of time today to play -- need to study for a job interview tomorrow  ;D .

If you (or anyone) have any suggestions about how to quantify it, I'm all ears.
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #83 on: August 13, 2017, 09:11:53 PM »
Last bit of fun for now -- using the last-under-90 5-day filter, here is all the ice over 90 coloured white and under 90 coloured blue.

(NB: I don't think we will get anywhere close to this unless perhaps there is a late-season GAC, which looks unlikely based on current forecasts. But it's interesting to look at.)
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

Neven

  • Administrator
  • ASIF Governor
  • *****
  • Posts: 4051
    • View Profile
    • Arctic Sea Ice Blog
Re: Ice Concentration Images and Animations
« Reply #84 on: August 13, 2017, 10:36:40 PM »
This is what this year could've looked like had there been more melting momentum (as in 2010 and 2012).
Il faut cultiver notre jardin

Peter Ellis

  • ASIF Middle Class
  • Posts: 548
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #85 on: August 13, 2017, 10:45:26 PM »
For a more realistic estimate, I wonder if it's possible to look at previous years and work out the probability that a pixel with 10%/20%/30%/40%/50%/(etc.) concentration will melt out by the time of the summer minimum, and then apply those probabilities to the current map.

A-Team

  • ASIF Upper Class
  • Posts: 1872
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #86 on: August 13, 2017, 10:57:26 PM »
probability that a pixel with 10%/ /(etc.) concentration will melt out and then apply those probabilities to the current map.

Nice idea but it requires very precise tracking of ice movement over six weeks (not feasible), the pixel being fixed but the ice moving past it. Furthermore ice of the same concentration can be of very different thicknesses -- indeed a cell a mosaic -- yet edge melt alone is only a small fraction of melt at most floe size distributions (which aren't available and might not be comparable year-on-year or at different locations). Then there is the matter of comparable weather and near-surface water temperatures, not to mention variable pre-conditioning during the previous freeze season.

Here is yet another AMSR2, from R Saldo archives at DTU, that we haven't looked at previously. It goes up to Aug 13th, just barely. Below that is a UH AMSR2 3.1k wipneus made to Aug 11th on the extent forum. Here 90-100% bin is replaced with yellow in the final frame.

Meanwhile, the 1 km S1AB active radar swaths from today show solid ice below the pole; consult the full image for areas other than the Ellesmere centered CAA shown below. (Only the upper left corner appears to be peppered with small areas of open water.)

http://north.seaice.dk/2017/08/01/20170801.amsr2.n.ice.gif

http://www.seaice.dk/latest/todays-sentinel1-n-1daymos.jpg
« Last Edit: August 13, 2017, 11:25:34 PM by A-Team »

oren

  • ASIF Upper Class
  • Posts: 1571
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #87 on: August 14, 2017, 01:00:34 AM »
Hmm... it's a good idea to try to quantify that. So far it has just been from eyeballing it -- although backed up by the results, compare for example Q1 vs. Wipneus animation and Q3 vs. the same animation.

To quantify it, we would have to define mathematically what is meant by an artifact, which could be tricky. I suppose I could try to apply some kind of extreme value test... but that would assume that the artifacts only occur infrequently, which may not be true in this case.

If you (or anyone) have any suggestions about how to quantify it, I'm all ears.
To quantify the amount of "artifacts" as you define them in the original image, you need to count the number of pixels your algorithm actually replaced. Those that were replaced by a lower value shall be known as "high concentration artifacts" while those that were bumped up shall be known as "low concentration artifacts".

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #88 on: August 14, 2017, 02:19:51 AM »
For a more realistic estimate, I wonder if it's possible to look at previous years and work out the probability that a pixel with 10%/20%/30%/40%/50%/(etc.) concentration will melt out by the time of the summer minimum, and then apply those probabilities to the current map.

Good idea. I have an idea along those lines too -- using classifiers (machine learning) to categorize pixels as melt / not based on the date, using previous years to train. Probably would use the original data instead of the concentration buckets used for the images. Would expect the classification to become more accurate as the melting season progressed. But no time now, so it's going to have to wait -- unless someone else wants to take a stab at it?

Also, A-team makes a very good point about the limitations of this approach on a pixel basis (as opposed to ice floes), and other problems. Not sure...
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.

greatdying2

  • ASIF Middle Class
  • Posts: 574
    • View Profile
Re: Ice Concentration Images and Animations
« Reply #89 on: August 14, 2017, 02:25:40 AM »
To quantify the amount of "artifacts" as you define them in the original image, you need to count the number of pixels your algorithm actually replaced. Those that were replaced by a lower value shall be known as "high concentration artifacts" while those that were bumped up shall be known as "low concentration artifacts".

That would certainly be easy and worthwhile to do. Not sure how it would help in choosing the "best" threshold, but it would be interesting to see the numbers. Note that the last-under-90 filter does not replace any low concentration pixels (but the quartile-based ones do).
The Permian–Triassic extinction event, a.k.a. the Great Dying, occurred about 250 million years ago and is the most severe known extinction event. Up to 96% of all marine species and 70% of terrestrial vertebrate species became extinct; it is also the only known mass extinction of insects.