Polar see-saw

[Ice Cool Kim]

As part of my continuing look at length of melting periods I took a second look at comparing Arctic and Antarctic trends.

The tendency for the two poles to behave in opposite fashions is well recognised and often referred to as the polar see-saw (though if someone is aware of a more scientific term please let me know).

My initial plot on the other thread discussing Arctic melting/freezing seasons showed both. They are  hardly a mirror image of each other but generally seem to act in opposite sense.

However, in evaluating the effect of the length of the filter I was using to detect the melting duration I found something quite remarkable.

I will make some observations below.

[Edit] just noted error in graph title , this was done on ice area, not extent.

[Ice Cool Kim]

Observations:

1) step change in Antarctic becomes more evident as filter length is increase. Shorter filters accentuate the one year anti-correlation  and tend to mask the abrupt change that started in 1986

2) Arctic shows a similar (but opposite) event but it happens two years later.

3) The above show that the spike in AO and arctic melting season in 1989 were not local phenomenon but manifestations of a global event.

Any ideas what this may represent?

[Wipneus]

Kim, you filtering constants have considerable influence on the averages ( a good filtering should only reduce the noise, not the mean).

This is event more pronounced for the antarctic, vs the arctic.

In your case I would look for a better filter.

[Ice Cool Kim]

Thanks for your comment Wipneus.

I take your point but this is not just a filter. It is not affecting the mean it is affecting the difference between the zero crossings in rate of change.

I am equally sceptical of the way that Antarctic result moves, however. That needs dissecting.

I don't see this happening in Arctic, do you?

The shorter filter results are more sensitive , but it looks like scaling the deviation for 0.5 would make them pretty coincident.

[ChrisReynolds]

Notice the severe asymmetry (time domain) in Antarctic extent,
http://arctic.atmos.uiuc.edu/cryosphere/IMAGES/seaice.recent.antarctic.png
as compared to Arctic
http://arctic.atmos.uiuc.edu/cryosphere/IMAGES/seaice.recent.arctic.png
Changing filter length could cause this to shift the average as the filter would include more of the asymmetry the longer it gets.

FWIW I suspect a similar effect is causing the decrease in melt season length you've described on another thread. I've reproduced your results but haven't managed to find the time to analyse this possibility.

[Ice Cool Kim]

Thanks Chris.

I was aware of the lop-sided symmetry but I'm not sure I can see how that would explain the shift since LP filtering should move both min and max to earlier dates. Perhaps more related to the different roundness of the min/max peaks. One being shifted more that the other.

Also the minima look to be more well behaved. The aim of the filtering is remove that noise and the kind of double peak we see in 2013. Short filters will allow that to produce a later max date. Since the minima are smoother that will shift one end only and affect the derived melt/freeze period.

A very quick scan seems to show pre-85 peaks were sharp and well defined, later ones less so.
http://arctic.atmos.uiuc.edu/cryosphere/IMAGES/seaice.area.antarctic.png

I'm going to have a closer look now.

[Ice Cool Kim]

On the "length of melting " thread , I've posted a plot showing the mean of melting and (1-freezing) compared to AO.  This  reduces errors a bit further and probably shows the closest correlation.

So I thought I'd try the same idea on Antarctic to see whether it removed step ( it pretty much has to in a way ).

The result was rather surprising.

In post '2 here , I noted that the "87" event was global , not just Arctic and happened first down under. But here we see just how similar the trace is.

Here I've lagged Antarctic data by a year to see how well they line up. Note that post 2002 variations line up without the lag.  2010 dip is synchronous in both hemispheres, SH does rebound unlike NH.

It's a little strange to see SH matches that last feature of AO better than the Arctic itself. In fact it correlates to '87 period considerably more closely than Arctic as well.

[Wipneus]

I take your point but this is not just a filter. It is not affecting the mean it is affecting the difference between the zero crossings in rate of change.

I was under the impression that you used some gaussian filter on the daily time series. Perhaps I am mistaken?

I don't see this happening in Arctic, do you?

Shortest time filter results are in 18 years clearly above, in 3 clearly below the other. I would say it is fairly obvious.

The shorter filter results are more sensitive , but it looks like scaling the deviation for 0.5 would make them pretty coincident.

You are using a filter method that is not just reducing the noise but also affecting the signal that you are looking for. That needs to be demonstrated not to affect the conclusions.

In your case I would:
1) try filter methods more applicable to non-linear signals, "Loess" is would be my favorite choice here.
 or 2) separate the data in trend+anomaly and apply the filter only on the anomaly

[Ice Cool Kim]

Shortest time filter results are in 18 years clearly above, in 3 clearly below the other. I would say it is fairly obvious.

Could you rephrase that, please? You seem to be seeing something you consider obvious but I really don't understand what you mean by this comment. It's obvious in SH, what are you seeing that is obvious in Arctic?

In your case I would:
1) try filter methods more applicable to non-linear signals, "Loess" is would be my favorite choice here.
 or 2) separate the data in trend+anomaly and apply the filter only on the anomaly

1) I am simply trying to remove short term variations from a roughly sinusoidal formed annual cycle. In what way would a filter like LOESS which does not even have a definable frequency response be an advantage? In what way is a gaussian not a suitable choice?

2) To separate a "trend" you have to assume a model. That choice will heavily affect the result. How should that model be chosen?
2b) Having assumed what the long term variation is and filtered out the short term , what can be gained from the remaining 'medium' term signal. The aim here is to examine the long term behaviour not to remove it.

I have an engineering/science background. The kind of processing you are suggesting is a lot of what I see as problematic in climate science. However, if you have answers to those questions and show that there is good grounds for that kind of approach, I'm always willing to learn a new technique. However, I have to be able to justify using it before I start.

[Ice Cool Kim]

I would add that I don't like the MF plot having a mean around 0.56 , if it was not for the correlation with AO I would be inclined to think the whole thing might be an artefact.

There is something odd going on it seems. Though finding Antarctic ice follows AO better than Arctic is a killer if I haven't just slipped up somewhere.

[ChrisReynolds]

There is something odd going on it seems. Though finding Antarctic ice follows AO better than Arctic is a killer if I haven't just slipped up somewhere.

Carl Wunsch, 2007, Abrupt climate change: An alternative view.
http://ocean.mit.edu/~cwunsch/papersonline/abrupt2006.pdf

The major problem in tuning or wiggle-matching is that of
‘‘false-positives’’—the visual similarity between records that are
in truth unrelated. A good deal is known (e.g., Barrow and
Bhavasar, 1987; Newman et al., 1994) about the tendency of the
human eye to seek, and to often find, patterns in images that are
tricks of the human brain. (The classical example is the
conviction of a large number of astronomers that they could
perceive ‘‘canali’’, or lines, on the Martian surface). A related
problem is the tendency to attribute importance to rare events
that occur no more often than statistics predicts (e.g., Kahneman
et al., 1982; Diaconis and Mosteller, 1989). It is for this reason
that statisticians have developed techniques for determining the
significance of patterns independent of the human eye.
An example of the problem is shown in Figure 2. The black
curve represents the three-month running average of monthly
maximum temperature in Oxford, UK between 1861 and 1903,
and the gray curve is the same physical variable, but between
1936 and 1978 (the annual cycle having been suppressed). This
arbitrary example was chosen because it is a simple way of
obtaining two real physical records with nearly identical
spectral densities, but for which there is no plausible
mechanism by which they should be correlated or coherent.
The ‘‘event’’ in the black record in 1880 might be identified
with the weaker minimum occurring in the gray curve just
slightly ‘‘earlier,’’ and some physical hypothesis for the delay,
or for age-model alignment, made. More generally, if there
were some uncertainty of the age-models for these two records
(there is not any), one might be strongly tempted to argue that
the degree of alignment that can be achieved by comparatively
small age-model adjustments is too great to occur by chance.
But, here it does occur by chance, and is a consequence solely
of the common frequency (spectral) content.

[Ice Cool Kim]

It is a good point that we have to be forever on guard against the pattern matching tendency of brain. (This may have some value for animal  survival but is not helpful in science).

However, I don't think the level of correlation from 1984-1997 can be dismissed as an optical illusion or psychological trick.

The correlation from 2003 to present (without the 1y lag) would also seem significant. That's a good 2/3 of the satellite record that shows quite a tight correlation.

That does not seem to be comparable to the somewhat tenuous correspondences noted in that paper.


I do take your point though. There is a natural tendency (probably with good darwinian founding) to see and match patterns.

[Wipneus]

Shortest time filter results are in 18 years clearly above, in 3 clearly below the other. I would say it is fairly obvious.

Could you rephrase that, please? You seem to be seeing something you consider obvious but I really don't understand what you mean by this comment. It's obvious in SH, what are you seeing that is obvious in Arctic?

Sorry if I'm too fast. I am looking at the curve called "length of the Arctic melting season13d" and compare it with the others. Filtered properly, there is no reason (AFAICS) that apart from random fluctuations this curve should lie above or below the others. Yet counting with the eyes, I get out of 33 years:
in 18 year the curve lies clearly above;
in 3 years the curve lies clearly below;
in the other years does not clearly depart from the lines with the longer time filters.

That is not random. It may be a small effect, but now I see the Antarctic results where the effects are even higher I am not sure.

Also the Antarctic effect is in the opposite direction: the lower filter time lies below where in the Arctic it lied above. That is a curious thing, might be interesting.

In your case I would:
1) try filter methods more applicable to non-linear signals, "Loess" is would be my favorite choice here.
 or 2) separate the data in trend+anomaly and apply the filter only on the anomaly

1) I am simply trying to remove short term variations from a roughly sinusoidal formed annual cycle. In what way would a filter like LOESS which does not even have a definable frequency response be an advantage? In what way is a gaussian not a suitable choice?

It is only my suggestion, feel free to use something better.

This is how loess reconstructs a sine:

About everything looks good: amplitude, phase, zero crossing, begin- and end point. That is not going to work as good with gaussian smooth.

And yes, it needs the "tuning" to be right-ish, between following all the wiggles and a straight line. What would you expect?

2) To separate a "trend" you have to assume a model. That choice will heavily affect the result. How should that model be chosen?
2b) Having assumed what the long term variation is and filtered out the short term , what can be gained from the remaining 'medium' term signal. The aim here is to examine the long term behaviour not to remove it.

Uhh, perhaps you did not understand that the final result is trend+filtered_anomaly, from which you can calculate the begin and endpoint of the melting period.
The only reason for the split is not to filter the "sinusoid" and hope the anomaly will be mostly made up by random noise.  A simple average would probably be sufficient, especially for the Antarctic where the "sinusoid" stays fairly constant in amplitude.

[Ice Cool Kim]

I get out of 33 years:
in 18 year the curve lies clearly above;
in 3 years the curve lies clearly below;

OK, thanks for the clarification.

Re. Arctic: I already commented on this effect. The shorter filters are more sensitive to change, that is expected and is pretty much the reason for testing different filters and seeing how this alters the pattern.

In effect, it makes little difference to the overall pattern which shows that the result is not an artefact the of choice of filter length. What is affected is the magnitude of effect. It is expected that deviations will be damped with a longer filter.

The only significant diviation is that some detail is lost in the post '87 rebound, with the longer filters.

What you have identified is not (IMO) a bias offset but magnitude variation. The reason that there are a majority of points that are greater is because, for the majority of the satellite period, the melting period was longer than the 'neutral' value.

I'm still looking into Antarctic plot which I am dubious about. Maybe you are correct that there is some linkage.

[Ice Cool Kim]

Uhh, perhaps you did not understand that the final result is trend+filtered_anomaly, from which you can calculate the begin and endpoint of the melting period.
The only reason for the split is not to filter the "sinusoid" and hope the anomaly will be mostly made up by random noise.  A simple average would probably be sufficient, especially for the Antarctic where the "sinusoid" stays fairly constant in amplitude.

Indeed, I did not realise that was what you were suggesting. Thanks for being more explicit.

I have already noted the presence of a pseudo cyclic variation and stated that as being one of main reasons for this filtering exercise. It is clear that 'hoping' the anomaly is random is not justified and that an average , or worse, a running average filter would be an awful choice.

Such processing would inevitably introduce aliasing effects into the result which would be far from obvious. This will at best introduce spurious variations and quite possibly lead to false conclusions being drawn.

This is signal processing 101. The module that most climate scientists seem to have missed out on.

Anomalies can help to visualise a dataset with a large repetitive pattern, however applying any filter to what is left is highly questionable. It would be more legitimate to filter first, then take the anomaly if that's what you want to look at.

This is a classic case of applying a "smooth" without recognition of the fact that it is a filter and giving due consideration to the frequency response of what is being done.

Since the signal we are trying to analyse is deviations in annual cycle what is the effect of filtering the residual anomaly? 

Thinking about it: due to linearity of convolution filters (and the subtraction of anomaly processing) , filtering before taking anomaly would be the same as filtering the anomaly and applying the same filter to the annual 'climatology' that was subtracted.

Adding it back would be identical to just filtering the signal but NOT identical to filtering just the anomaly and adding back in the unfiltered climatology.

Doing what you suggest would leave part of the signal unfiltered. I don't that as being beneficial (or even intentional).




Re. LOESS:

About everything looks good: amplitude, phase, zero crossing, begin- and end point.

The fact that this filter always ends up going through the end points is one of the reasons I don't like it. That clearly has no validity. It is seems to be doing something similar to padding the window in a convolution filter ( only worse ).

The fact that it has an indeterminate frequency response is another problem. Fine, the result look "smoother" but what have we done to the data? We don't actually know. 

So, thanks for your suggestions, ideas and suggestions are welcome, but I don't think I see any reason that LOESS would be a better choice than gaussian. Rather the opposite.

[Ice Cool Kim]

Here is an (extreme) example of the situations I'm trying to resolve.

An unfortunate short term , small scale variation causes two zero crossings, even with 13d filter. Unfiltered there are loads.  Which one you choose to be the "real" onset of melting? The first , the last , maybe some average, none of the above?

The literature has various solutions bases of various proxies of ice cover. Some directly from radar reflection data.

The aim here is regard the neither the local trough nor peak to be indicative of the underlying annual cycle and to filter it out to obtain a zero-crossing that is representative of annual cycle.

It also shows how the length of the filter can change the date of onset and hence shorten/lengthen the resulting melting period.  The annual minima are much more prone to noise than the maxima.  Small changes in a small number are more significant than small changes in a big number. There is probably a physical aspect to this as well, the ice a maximum being more solid an compact is less susceptible to weather induced variations.

Detection of the date of minimum is thus more sensitive to both noise/weather and the method and criteria used to define the turning point.

There may still be another issue I'm chasing down.

[Ice Cool Kim]

One note on the above. With the lighter filter (red) choosing either zero crossing (or the minima directly in the area data) would be misleading.

20d day filter is better than either individual zero-crossing of 13d, but still offset from the underlying annual cycle.

 Taking the mean of the two 10d results would be close to 30 day filter (blue line) result.

Since these small pseudo cyclic variations are fairly symmetrical averaging the two is quite a good solution. This will clearly not resolve less regular noise that requires filtering.

[Wipneus]

The fact that ...

The fact that ...

Products of your imagination are presented as facts. None of them is. EOD

[Ice Cool Kim]

Any filter which produces results up to the end of the data, I regards with suspicion.  There is no magic way to do that without changing both frequency and phase response.

Last time I tried to find a definition of the frequency response of LOESS it seemed there is none. Since the way it works depends on the data it would seem obvious that there is not way to determine the frequency response.

Thus my statement that it has an indeterminate frequency response would seem to be accurate.

Now if you know better I'm always willing to be corrected by facts. Simple contradictory statements are less convincing.

[Peter Ellis]

An unfortunate short term , small scale variation causes two zero crossings, even with 13d filter. Unfiltered there are loads.  Which one you choose to be the "real" onset of melting? The first , the last , maybe some average, none of the above?

None of the above.  The Arctic is a large place, and different parts of it will start melting while others are still increasing, and vice versa.  If you think you can pin down either "the" date of melt onset or "the" date of re-freeze onset with any accuracy less than about a week either way, you're talking out of an uncharacteristic orifice.

[Ice Cool Kim]

Very amusing terminology Peter. Yet this is what several studies have attempted to do already.

As you rightly point out this will have regional variations as well as temporal ones. This is precisely why I have attempted to use intelligently designed filters remove these short-term variations to detect the underlying annual cycles and to examine how those vary from year to year and on a decadal scale.

My approach is indeed the "none of the above" option.

It was not immediately apparent from the tone of your comment that you considered yourself to be in agreement with me.