Please support this Forum and Neven's Blog

Author Topic: Cutting through the noise  (Read 597 times)

Dundee

  • ASIF Lurker
  • Posts: 39
    • View Profile
Cutting through the noise
« on: August 22, 2016, 08:28:39 PM »
I think we are all aware daily ice metrics have a fair bit of noise. This makes for exciting discussions as each day's numbers become available, but it does not help us sort out what might really be going on.

The most common way to reduce noise is by averaging data across multiple days. NSIDC, for example publishes five day moving averages and encourages emphasis on monthly averages vice daily values.

The problem with moving averages is that they are not especially good at suppressing short term noisy events and they introduce distortion (flattening of peak values). Of note, the standard deviation in annual NSIDC September minimums – the equivalent of a 30 day average – is actual slightly higher than the average of computed daily standard deviations for the month. This is the opposite of intuitive expectation, and means that reliance on the monthly number rather than the flightier daily figures is not actually providing us with more clarity into minimum ice numbers (which are in fact not normally distributed).

One method of removing short duration noise is with a low pass filter. A common type of filter combines a fraction of samples of the signal, drawn at equal intervals of time, and balanced to meet the needs of the filter designer. A five day moving average is actually a very simple example of this type of filter - twenty percent of each of five consecutive samples are totaled to produce output for any given day. By altering the fraction of each sample added to the mix, and using more samples, we can improve the suppression of high frequencies (noise) and reduce the distortion of low frequencies (signal).

One way to design a filter is with the Parks-McClellan algorithm. Given the desired pass frequency, stop frequency, and desired distortion/noise suppression goals, it determines the minimum number of samples (taps) and fraction of each sample (gain coefficients).

On online tool for filter design is available at:

http://t-filter.engineerjs.com/

I've been fiddling with this for a while now. A filter that stops changes with a period of four days or less and passes changes with a period of a week has about the same ability to reduce very short term noise as a five day moving average (which is not as effective as we'd like).

The fastest changing event in the annual ice cycle is the change from thaw to freezing at the September minimum. Only a trial and error result, but setting the pass band to 45 days induced visible distortion in second and third derivatives of September ice extent, at 42 days, fidelity looked OK.

The number of samples in the filter is a function of the desired attenuation of noise, the acceptable distortion of signal, and the ratio of the stop and pass frequencies (periods). I don’t claim it is the optimum, but I have converged on a 133 day filter, passing 42 day signals and stopping 21 day signals. This gives 50dB of suppression with 0.01dB of distortion (noise cut by a factor of 10,000, and about one quarter of one percent distortion). Peak reduction of impulse (one day) noise for this filter is about the same as a 14 day moving average, but the overall reduction of noise is better and mathematically well behaved, and introduced distortion is much less. Sharpening the cutoff period to 35 days (83% of the pass band, rather than fifty percent) can be done, bet requires more than 600 taps. This no harder to implement in software than a shorter filter, but yields a filtered record that is nearly two years shorter than the raw data (since the first day with data available for each tap would be more than 300 days into the available data span, with a similar loss at the end).

Below is a graph of a constructed signal with sine waves of increasing periods, and then random noise added to 64 day period signal. A five day running average and pass42/stop21 day filtered average are shown for comparison. I’ve also included a graph of daily changes (the equivalent of the first time derivative) of the plots. Of note – removing high frequency noise removes it for higher derivatives as well, since derivation is in effect a phase operation and does not affect the frequency or amplitude of a sine signal.

Filtering cleans up annual graphs somewhat but its greatest value is the effect on derived statistics, such as daily rate, anomaly, and derived concentration and thickness (these graphs can be noisy to the point of being nearly unusable).

I encourage anyone displaying daily time series ice data that is noisy (and in particular, data that has been manipulated in ways that increase the proportion of noise, through subtraction or other combining of multiple data sets) consider filtering to improve visibility of the underlying fundamental trends. Filtering is simple to implement and, at the level of daily summary values, not computationally demanding.

The last two images are screen shots of the frequency response and impulse response (in effect, a plot of the filter coefficients) for the 133 tap pass42/stop21 day filter I’ve been working with. Frequency is in cycles per year, 10 corresponds to a period of 36.5 days.
« Last Edit: August 22, 2016, 09:18:49 PM by Dundee »

budmantis

  • ASIF Upper Class
  • Posts: 1073
  • "Be the change you wish to see in this world." MG
    • View Profile
Re: Cutting through the noise
« Reply #1 on: August 28, 2016, 03:57:56 PM »
This stuff is way above my pay grade Dundee, but I have a thought or two.

One of the biggest draws to the forum are the daily updates on sea ice extent and area. For myself, I look forward with great anticipation to the daily updates during the summer months. At the same time, I realize that some viewers and members tend to overreact to large drops or unexpected increases which in turn creates a lot of noise at times. I've done the same myself.

FWIW, I think we need to take the daily numbers with a grain of salt, but without them it would be a lot less exciting.
"To predict the behavior of ordinary people in advance, you only have to assume that they will always try to escape a disagreeable situation with the smallest possible expenditure of intelligence." Nietzsche