Accuracy of poll predictions

[Ned W]

In the thread for the June poll for the IJIS/JAXA daily minimum, there has been some discussion about the "accuracy" of our collective predictions.  I went back and looked at all of the June/July/August polls for the NSIDC monthly and CT or IJIS daily minimums, from 2013-2016 ... that's 24 polls.

First up, here are the distributions of responses by how they ended up (too high, correct, or too low). 





In 23 of the 24 cases, a majority of poll responses were on the "too-low" side.  The one exception is the August 2014 poll for CT Area daily minimum. 

Clearly recent events can have a big effect.  In 2013, the first year after the big 2012 record, every single participant in the poll erred on the too-low side, for four of the six polls that year.

Here are graphs comparing the median prediction from each poll to the final (observed) value for that year:





Overall, we do better in August than in June (duh!).  For June polls, our best results were in 2015.

And there's a very strong bias towards under-predicting ice extent.

[Neven]

Thanks for this, Ned.

It'd be interesting if we could give the bias a number, and then use that to predict the minimum. ASIF prediction + ASIF average bias = X. 

[Richard Rathbone]

Thanks for this, Ned.

It'd be interesting if we could give the bias a number, and then use that to predict the minimum. ASIF prediction + ASIF average bias = X. 

Post 2013 I use it as a sanity check. If my prediction isn't rather higher than the forum, I need to doublecheck my reasoning and the indicators I am using.

[slow wing]

X would be an estimate of the bias only and probably not a very good one as it's only 4 years of data and Mother Nature has had a big say.

  The actual bias could be defined as the difference between what the forum votes and what some idealised observer with essentially perfect knowledge of the physics would vote.

  The trouble is, the weather fluctuates away from the climate mean so the idealised observer also won't predict the exact answer each year. (It's a chaotic system so the fluctuation is not predictable even with essentially perfect knowledge of the physics.)

  2013 was a rebound year. Was that mostly physics or fluctuation? Given the resumption of the downward trend culminating in the record low maximum volume this year, I suspect the latter. So even the idealised observer would have voted high[EDIT]low - perhaps in all 6 polls for 2013.

  For the other 3 years, its a more difficult question whether the fluctuation was upwards or downwards.

  In my view we can't even rule out that the idealised observer would have voted high[EDIT]low on the other 3 years as well - and maybe in 23 out of the 24 polls. In that case, there would be no evidence whatsoever for bias in the forum.

  It's a shame we don't have polls from 2012 as well. I suspect 2012 was a downwards fluctuation and so the idealised observer would have voted high in 2012, maybe in all 6 polls.

  Maybe the forum would have voted high as well in most or all of the 6 hypothetical polls for 2012. That would have given a very different look to the graphs in the opening post. Does anyone remember if the forum was generally thinking higher or lower than the final numbers for 2012?
 
  In summary, it's easy to over-interpret the graphs in the opening post and difficult to establish bias from only 4 years data in a complicated and chaotic system.

[Ned W]

Thanks for this, Ned.

It'd be interesting if we could give the bias a number, and then use that to predict the minimum. ASIF prediction + ASIF average bias = X. 

That would be interesting.  But as slow wing says it's a small sample of noisy data.  And there are oter problems:

1. Hopefully, over time we get better at predictions, so the bias should shrink. In fact, the act of observing this and speculating about it might stimulate a change in the bias.       So the bias is non stationary and the act of observing it renders the observations obsolete.

2. Unfortunately, by the time we get a large enough sample size (say 2030s or so) there will probably be "ice-free" days in most Septembers ... making it easy to reliably predict the daily min.  So the whole thing becomes moot.

3. While waiting for that to happen, characterizing the bias requires being able to compare predictions across multiple years.  But that's hard to do when people change the format of the poll (... cough NSIDC cough OVERLAPPING BINS cough JUNE 2017 cough ...)

[Neven]

3. While waiting for that to happen, characterizing the bias requires being able to compare predictions across multiple years.  But that's hard to do when people change the format of the poll (... cough NSIDC cough OVERLAPPING BINS cough JUNE 2017 cough ...)

Is that really a problem? The overlapping bins, I mean. You still get an average and a median, etc, right?

If it is, I'll do regular bins for the next poll, and you'll just have to homogenize or extrapolate, or apply some other statistical techniques, for this one poll, Ned. 

Maybe the forum would have voted high as well in most or all of the 6 hypothetical polls for 2012. That would have given a very different look to the graphs in the opening post. Does anyone remember if the forum was generally thinking higher or lower than the final numbers for 2012?

The Forum started in 2013, so no polls for 2012 there. But I believe I ran polls on the blog. If I have time later today, I'll go and check.

[EgalSust]

Thanks for the effort, Ned W!

For comparison, here are some results from Sea Ice Prediction Network:


Figure 13: June, July, and August 2016 SIO contributions of the September pan-Arctic sea ice extent as box plots, broken down by type of method. Boxes show medians and interquartile ranges. Colors identify method types, and n denotes the number of contributions. Individual boxes for each method represent, from left to right, contributions to the June, July, and August SIO. The heavy gray line shows the 2016 observed September sea ice extent from the NSIDC index (as in Figure 1). Figure updated from Hamilton and Stroeve (2016). (source)


Figure 7. Distributions of June, July, and August 2015 Outlook contributions as a series of box and whisker plots, broken down by general type of method. The box color depicts contribution method with the number below indicating total number of contributions by method over the three months. The individual boxes for each method represent, from left to right, each month of June, July, and August. (source)


Figure 5. Distribution of Pan-Arctic Outlook values (August Report) for September 2015 extent. (source same as above)


Figure 3. Box plots show the median, interquartile range, highest, and lowest values of sea ice outlook contributions from June, July, and August, broken down by type of method used. As a group neither the statistical nor modeling methods shows a clear advantage, but some contributions from both groups came close to the observed mean September extent. (source)


Figure 2. Distribution of individual Pan-Arctic Outlook values (August Report) for September 2014 sea ice extent compared to the observed value. Labels on the bar graph are rounded to the tenths for readability. Refer to the Individual Outlooks at the bottom of this report for the full details of individual submissions. (source same as above)


Figure 5. Distribution of 2013 Sea Ice Outlook predictions by method and month, including the August 1 cycle which consists of an informal update. Box plots show median and approximate interquartile range (middle 50%) of each group. The observed mean September 2013 extent, 5.35 million km2, is marked by a horizontal line. Median SIO predictions from all three methods shifted upwards in the August 1 cycle, reflecting adjustments to mid-summer conditions but also changing contributor pools. A more detailed analysis with comparisons to earlier years and other datasets has been drafted by Stroeve and Hamilton (in review), and will be summarized in a later post. (source


Figure 4a. Distributions of Outlook estimates for September 2013 arctic sea ice extent for the June report (based on May data). (source: 2013 post-season report, linked above)


Figure 4b. Distributions of Outlook estimates for September 2013 arctic sea ice extent for the July report (based on June data). (source as above)


Figure 3a. Distributions of Outlook estimates for September 2012 arctic sea ice extent for the June report (based on May data). (source)


Figure 3b. Distributions of Outlook estimates for September 2010 arctic sea ice extent for the July report (based on June data). (source as above)

Late Andrew Slater (R.I.P.) has made some comparisons of August predictions:


At least two out of the last three years (source)


The last two years (source as above)


Three out of the last three years (source as above)

[magnamentis]

Thanks for this, Ned.

It'd be interesting if we could give the bias a number, and then use that to predict the minimum. ASIF prediction + ASIF average bias = X. 

chances are high that if we do that this year things will be the other way around which would see for a huge bias towards the other end. i'm confident that this year could even run the predictions out on the lower side and even more confident that the majority is at or in the vicinity of being spot on.

[Steven]

Here are graphs comparing the median prediction from each poll to the final (observed) value for that year:

Thanks.  I modified your graph to include the median predictions by the sea ice prediction network, indicated by the red horizontal line segments:

[Chuck Yokota]

I tend to vote pessimistically, as I would rather be pleasantly surprised than the opposite.

[Feeltheburn]

I tend to vote pessimistically, as I would rather be pleasantly surprised than the opposite.

That is commendable. <snip, removed the remainder for unnecessary belligerence; J.>

[Juan C. García]

Have you think in the accuracy in which NSIDC makes their monthly averages?
Do you agree that:

Arctic sea ice extent during September 2016 averaged 4.72 million square kilometers (1.82 million square miles), the fifth lowest in the satellite record.
http://nsidc.org/arcticseaicenews/2016/10/

From my point of view, it is completely wrong to convert a 15%+ average on the month to make it 100% on a given grid.

They should -instead- make the statistical average of their daily values (they can use their "one daily value" or the "5 day moving average").

So, part of the inaccuracy that this Forum has, comes from the strange way the NSIDC makes their monthly average (that is not an average at all).
 

[Juan C. García]

Arctic sea ice extent for December 2016 averaged 12.10 million square kilometers (4.67 million square miles), the second lowest December extent in the satellite record. This is 20,000 square kilometers (7,700 square miles) above December 2010, the lowest December extent,...
http://nsidc.org/arcticseaicenews/2017/01/

      

[Juan C. García]

Because it's still so early, I'm going to wait the full 10 days before voting. Right now, I have no idea which way this could go. 

Between 3.25 and 3.75 million km2.
A little lower than 2012. Just to vote for a new record (given the low volume that we have now) and because I know that I will not know, 10 days after today... 

I was only joking, because I do not believe in the accuracy of the NSIDC Monthly Average at all... 

I definitely prefer to bet on an IJIS or NSIDC daily value 

[Juan C. García]

Have you think in the accuracy in which NSIDC makes their monthly averages?
Do you agree that:

Arctic sea ice extent during September 2016 averaged 4.72 million square kilometers (1.82 million square miles), the fifth lowest in the satellite record.
http://nsidc.org/arcticseaicenews/2016/10/

Year    Month    Day        Extent
YYYY       MM     DD    10^6 sq km
2016   9   1   4.278
2016   9   2   4.211
2016   9   3   4.263
2016   9   4   4.258
2016   9   5   4.201
2016   9   6   4.203
2016   9   7   4.083
2016   9   8   4.104
2016   9   9   4.137
2016   9   10   4.16
2016   9   11   4.209
2016   9   12   4.236
2016   9   13   4.341
2016   9   14   4.357
2016   9   15   4.378
2016   9   16   4.363
2016   9   17   4.309
2016   9   18   4.434
2016   9   19   4.495
2016   9   20   4.533
2016   9   21   4.593
2016   9   22   4.627
2016   9   23   4.814
2016   9   24   4.899
2016   9   25   4.916
2016   9   26   5.071
2016   9   27   5.143
2016   9   28   5.174
2016   9   29   5.175
2016   9   30   5.187
         
Statistical Monthly Average:       4.505
Official Monthly Average:            4.720
Difference on km2:                 214,933

[Juan C. García]

Back in November 9th, 2012 (I had a bad size on NSIDC grid, but all the analysis seems fine to me):

I have been thinking about the method used to calculate the NSIDC monthly average and the impacts that could have in the future, so let’s image the following scenario:
We are at Sep. 1st, 2019. The NSIDC shows an Arctic sea ice extent (SIE) of one million km2, concentrated basically at the Arctic Basin. The ice extend is decreasing and by September 10th, the sea ice is almost gone. So, we have Arctic free of ice for the first time in several thousand years. By September 20th, some ice starts to build up around the continental coasts. By the end of the month, one million km2 has freeze.
On October, everybody is surprised that the NSIDC has announced a September SIE monthly average of 1.18 million km2. There was not a day with more than one million km2 at September. So, what will be the reason for this monthly average? The answer will be the method used to calculate the NISDC monthly average.
The normal NSIDC satellite grid at the arctic is 35km.* 35km. = 1,225 km2. The 15% of that is 183.75 km2. So basically, if a normal grid has a monthly average of 184 km2 or more, it accounts as 1,225 km2 for that month.

There are several ways in which a grid can qualify as a 100% SIE. By example, a grid at the Arctic Basin:

Day(%)- % - Sea Ice (km2)- Daily graph (km2)
Sep 1 - 100% - 1,225 - 1,225
Sep 2 - 100% - 1,225 - 1,225
Sep 3 - 85% - 1,041 - 1,225
Sep 4 - 65% - 796 - 1,225
Sep 5 - 50% - 613 - 1,225
Sep 6 - 30% - 368 - 1,225
Sep 7 - 14% - 172 - 0
Sep 8 - 10% - 123 - 0
Sep 9 - 8% - 98 - 0
Sep 10 - 3% - 37 - 0

Total: 5,696 km2
Average (divided by 30): 189.9 km2
It adds to monthly average: 1,225 km2

The NSIDC affirms that the ice at the beginning of September at the Arctic Basin accounts for 720,000 km2 and the freeze at continental coasts accounts for the rest, so there is a monthly average of 1.18 million km2 for September 2019.
Let’s have another example. A 325 km2 ice floe is moving from one grid to another on a sea free of ice (except for that floe). It stays 6 days in one grid, 15 days in second one and 9 days in a third one. At the end, it has made that 3 grids count as with 100% ice, so it accounts for 1,225 km2 * 3 = 3,675 km2. That looks incredible, but it counts for more than 10 times his size.
So, if someone is looking for the negative feedbacks that will make the Gompertz curve, the NSIDC method for calculating the monthly average seems that will be one important negative feedback. Rather artificial, but as I understand the method, it will work that way.

http://neven1.typepad.com/blog/2012/11/ct-sia-finally-above-2-million-km2-anomaly-mark.html?cid=6a0133f03a1e37970b017c3343058c970b#comment-6a0133f03a1e37970b017c3343058c970b

[Ned W]

Juan, it looks to me like you are applying the 15% concentration threshold over time (in terms of the monthly average) in addition to over space (within a grid cell). I don't think that's right. 

As I understand it, for your first example, the grid cell would be reported as having 1225 km2 on the first six days, and 0 km2 on the remaining 24.  In that case, it would contribute 245 km2 to the monthly average, not 1225 km2. 

Likewise, in your second example, the ice floe only raises each of the three grid cells to 1225km2 for part of the month.  The first cell would end up contributing (1225 * 6 / 30), the second one (1225 * 15 / 30) and the third one (1225 * 9 / 30) ... which adds up to 1225 total, not three times that. 

[oren]

Juan, it looks to me like you are applying the 15% concentration threshold over time (in terms of the monthly average) in addition to over space (within a grid cell). I don't think that's right. 

We all agree it's not right, but it's still the way NSIDC calculate monthly averages. The averaging algorithm assumes immobile ice, when the ice is mobile (or melts here and freezes elsewhere) their algorithm produces absurdly high numbers.
If I recall correctly they announced they will create an improved algorithm for caculating the monthly averages, but I'm not sure where that stands.

[Ned W]

Thanks, Oren.  Despite following this stuff in a casual way for over a decade, I haven't paid that close attention to some of the details. 

Looking at the explanation here, it sounds like for the monthly extent they do apply a 15% threshold over time, but based on the daily concentration measurements not the daily extent measurements:

On monthly extent images, ice ends and water begins where the concentration estimates of grid cells in the gridded average, or mean, concentration field for that month drop below 15 percent.

That's slightly different from what I remembered, and it makes Juan's first example correct.  The monthly average concentration in his grid cell is 15.5% so it counts as "ice" for the month.

But his second example still doesn't seem to work.  His drifting ice floe represents 26.5% of whatever grid cell it's in.  Multiply that by 6/30, 15/30, and 9/30, and none of the three grid cells it spends time in would reach an average concentration of 15% for the month -- so all three grid cells would show up as non-ice.  In this example, the algorithm is underestimating the extent.

Disclaimer: both my understanding and my calculations could be wrong.

[Juan C. García]

But his second example still doesn't seem to work.  His drifting ice floe represents 26.5% of whatever grid cell it's in.  Multiply that by 6/30, 15/30, and 9/30, and none of the three grid cells it spends time in would reach an average concentration of 15% for the month -- so all three grid cells would show up as non-ice.  In this example, the algorithm is underestimating the extent.
Disclaimer: both my understanding and my calculations could be wrong.

You are right, Ned. It has to be a drifting ice floe with a bigger area, to work in this example.
Thank you for making the correction!

Even though, NSIDC makes us have the wrong conclusions, with the algorithm that they are using to calculate their monthly average.

Going back to the discussion in this topic, my point of view is that daily extent depends on winds and currents (or how the ice floes are separeted of each other), so it is not the better way of measuring the ASI when our planet is approaching an ice-free Arctic summer. It gets worst with the NSIDC monthly average algorithm.

So is it there an accuracy in the poll predictions? Luckly, we had some kind of rebound on 2013-2014 (that make our polls very incorrect). But 2015-2016 are not that good. If we see the low extent that we have on May 2016, well, it was difficult to forecast that the tendency will change for September. Even though, I prefer the ADS (JAXA or IJIS) conclusion, that 2016 is the second lowest year on record, than the NSIDC conclusion of fifth lowest. On NSIDC daily values, 2016 was also the second lowest, very close to 2007.

At the end, I believe that the tendency in volume is the one that is going to bring an ice-free Arctic, and if we follow this tendency, that will happen in the near future. Next 10 years? Even less? When it happens, extent will have a dramatic drop, even that it will also depend on "the statistical noise" of how the floes are separated of each other. I like to compare the ice measure in area, at least on September.

So the polls on this Forum are a better way to make a forecast (because in this Forum we are checking the status of the ice on a daily basis and in a way, we also know that an ice-free Arctic is going to happen in the near future), than the tendency that NSIDC says (based on their monthly average algorithm) and sadly, with the IPCC definition of "ice-free Arctic when there is less than 1 million km2 of extent in three consecutive years", well, the true is that they are also misleading.

[Steven]

Juan, it looks to me like you are applying the 15% concentration threshold over time (in terms of the monthly average) in addition to over space (within a grid cell). I don't think that's right. 

We all agree it's not right, but it's still the way NSIDC calculate monthly averages. The averaging algorithm assumes immobile ice, when the ice is mobile (or melts here and freezes elsewhere) their algorithm produces absurdly high numbers.

As shown in the table below, the difference between the official NSIDC September monthly extent (second column) and the average of the daily NSIDC extent values for September (third column) is for most years about 0.1 million km² (fourth column), and at worst 0.2 million km².  So I think this is only a relatively minor problem, and it is definitely not the reason for the bias on this forum.


                   NSIDC September extent:
Year              Official       DailyAverage       Difference
2005               5.59               5.50               0.09 million km²
2006               5.95               5.86               0.09
2007               4.32               4.27               0.05
2008               4.74               4.69               0.05
2009               5.39               5.26               0.13
2010               4.93               4.87               0.06
2011               4.63               4.56               0.07
2012               3.63               3.57               0.06
2013               5.35               5.21               0.14
2014               5.29               5.22               0.07
2015               4.68               4.62               0.06
2016               4.72               4.51               0.21

[Juan C. García]

As shown in the table below, the difference between the official NSIDC September monthly extent (second column) and the average of the daily NSIDC extent values for September (third column) is for most years about 0.1 million km² (fourth column), and at worst 0.2 million km².  So I think this is only a relatively minor problem, and it is definitely not the reason for the bias on this forum.

I believe that it is a subject that is important on this Forum, but it is not polite to talk about it. But waiting to have an ice-free Arctic -instead of talking about it- should be disgusting to the majority of the human beings.

I brought the subject, because I believe it is important in evaluating the accuracy of the polls. Now I am going to be off topic, so I will only make this comment and I will stop talking about it, at least under this topic.

In 2013 I made some analysis on the importance of this difference between the NSIDC statistical and official averages. And the difference are important in other months, specially in October.
What it means to me, is that when we have an ice-free Arctic, it could extent to the middle of October, at least.

[Ned W]

I thought this graph was interesting:



It shows the variance (er, standard deviation) of the drop in extent from [Day X] to the September daily minimum (for JAXA daily extent).

During the first half of July, the "unpredictability" of the minimum is relatively high and relatively constant -- there's not much difference in the variance of the expected drop to minimum from July 1 vs from July 14.  But after July 15 that variance starts to drop fast. 

Also, interestingly the uncertainty in the eventual minimum seems to be slightly lower in mid-June than in July. 

Oh, yeah -- this is based on the 2007-2016 decade.

[Steven]

The NSIDC September extent for 2017 was 4.87 million km2:

ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/monthly/data/N_09_extent_v2.1.csv

Looking at the June poll on this forum, this means that 98% of the votes were too low, and 1% of the votes were too high.  Moreover, the median of the votes was too low by about 1.4 million km2.

In the July poll, 92% of the votes were too low and 1% were too high.  And in the August poll, 88% of the votes were too low and 0% were too high.

Here is a graphic (updated from an earlier graphic by Chris Reynolds last year) showing the distribution of the votes in the June polls on this forum in the last 5 years.  The correct bin for each year has a fluo-green background.





(Note: the 2017 poll had overlapping bins.  To make this consistent with the polls in previous years, I split each of the overlapping bins in two equal halves and distributed the votes evenly over the two halves.)

Finally, here is a similar graphic for the JAXA extent June polls in the last two years.

[Neven]

That's a great graph, Steven. Could you please post it again at the start of next year's June poll? 

[ChrisReynolds]

I agree with Neven, good graph Steven. There is rather a low bias here, but once one is aware that's not an issue. And after all it should all be for fun.

[Steven]

Updated graphic for the June polls for JAXA extent:





Caveat: the June 2018 poll had a different format than the 2017 and 2016 polls:

(1) The 2018 poll had overlapping bins of width 0.5 million km2, whereas the previous years' polls had bins of width 0.25 million km2.  For consistency with previous years, I split each of the overlapping bins in the 2018 poll in two equal halves and distributed the votes in the bin evenly over its two halves.

(2) The 2018 poll had no bins below 2.75 million km2, except for a single bin labeled "New Record".  I decided to treat this "New Record" bin as 2.5-3.0.

[Tor Bejnar]

(2) The 2018 poll had no bins below 2.75 million km2, except for a single bin labeled "New Record".  I decided to treat this "New Record" bin as 2.5-3.0.

I was thinking, "Gosh, most of the crazy low voters went away or aren't so crazy any more."  Then I read the real reason.  There's a saying that come to me: 

[Dharma Rupa]

I think the whole subject of poll predictions is stupid...and analyzing their accuracy is the most stupid part.

[Neven]

I think the whole subject of poll predictions is stupid...and analyzing their accuracy is the most stupid part.

I disagree. Should we start a poll on this?

Yes, we should.

[litesong]

"Gosh, most of the crazy low voters went away....  Then I read the real reason.

I know the real reason, but I've been sworn to secrecy.... or something like that.

[mostly_lurking]

Updated graphic for the June polls for JAXA extent:

What this poll consistently shows is that most people are more on the alarmist side and always expect the situation to be worse than it is  

[oren]

What this poll consistently shows is that most people are more on the alarmist side and always expect the situation to be worse than it is

No (imho). It shows that some people always vote zero/new record as a statement, regardless of any current evidence (the bottom lump), but that most people do their best to guess the actual outcome (the apparent bell curve).
Why does the bell curve seem skewed to the downside? I believe that downside surprises, while rather rare (10-20% chance annually), have a larger departure from the trend, so any year where a 2012-like shock does not happen is a year that will appear somewhat above the median of the bell curve. I don't think it's alarmist to think that other such shocks are possible, and to factor that into the predictions.
Also note that as we move further in time from 2012, the implied downside surprise probability/magnitude is getting smaller, as people revise down the probability estimate following the new annual data. After 2012 many thought it could happen every year, now many think it's a fluke. A reasonable deduction in both cases.

[Stephan]

So there may be a kind of "educative" training in the next years, when those members which were wrong in their predictions on the far too low side will vote for higher, more realistic bins in the future?!?

[oren]

I added a new paragraph to my post while you were posting. But to add to that - those that are "stuck" at zero/new record will probably remain there. The rest are undergoing a gradual adaptive process. I expect the next 2012 to easily come below the median bell curve.

[mostly_lurking]

What this poll consistently shows is that most people are more on the alarmist side and always expect the situation to be worse than it is

No (imho). It shows that some people always vote zero/new record as a statement, regardless of any current evidence (the bottom lump), but that most people do their best to guess the actual outcome (the apparent bell curve).
Why does the bell curve seem skewed to the downside? I believe that downside surprises, while rather rare (10-20% chance annually), have a larger departure from the trend, so any year where a 2012-like shock does not happen is a year that will appear somewhat above the median of the bell curve. I don't think it's alarmist to think that other such shocks are possible, and to factor that into the predictions.
Also note that as we move further in time from 2012, the implied downside surprise probability/magnitude is getting smaller, as people revise down the probability estimate following the new annual data. After 2012 many thought it could happen every year, now many think it's a fluke. A reasonable deduction in both cases.

I think the "implied downside surprise" was smaller this year because someone(don't remember who) stated that in previous years the poll was skewed to what I called "alarmist" and people right away changed their vote accordingly (less alarmist)-(he might have stated this in a different related thread)

[Pmt111500]

*temporarily abandons statistics as predictive method to allow weather to do it's thing*

[DavidR]

So there may be a kind of "educative" training in the next years, when those members which were wrong in their predictions on the far too low side will vote for higher, more realistic bins in the future?!?

We are in an environment where there is a high probability that the result will be midrange amongst  other recent years and a low probability that it will be exceptionally low. If the extent is one bin higher than out estimate nine times out of ten and 9 bins lower once out of ten then we are actually quite realistic. 

As a group we are concerned that extent is declining and therefore as a group we have a bias towards greater losses. We are not betting our children's inheritance on this,  just the worlds inheritance.

[Klondike Kat]

So there may be a kind of "educative" training in the next years, when those members which were wrong in their predictions on the far too low side will vote for higher, more realistic bins in the future?!?

We are in an environment where there is a high probability that the result will be midrange amongst  other recent years and a low probability that it will be exceptionally low. If the extent is one bin higher than out estimate nine times out of ten and 9 bins lower once out of ten then we are actually quite realistic. 

As a group we are concerned that extent is declining and therefore as a group we have a bias towards greater losses. We are not betting our children's inheritance on this,  just the worlds inheritance.

David, I think you summarized it quite well.  The more people that are concerned, the greater proportion of pessimistic responses.  I believe that is the main source of the bias.  Combine that with a short-term deviation from the intermediate trend, and the bias becomes compounded.

[Archimid]

Or it could be that the situation looked worst in 2016 and 2017 than it did in 2018.  It makes sense since 2015-2017 there was a peak planetary temperature and 2018 was the corresponding trough.

We'll see what happens next. My guess is the hiatus will continue until the next step change happens. The next step might or might not be the last.

[Dharma Rupa]

But the predictions are meaningless.  The weather will do what it will do year after year.  The only prediction which has any real-world significance is whether the end will come suddenly or slowly.

[Archimid]

Predictions are not meaningless, they are all wrong but some of them are useful. Predictions like the ones used by SIPN are good, educated guesses that use very good data and amazing models to make the best educated guess possible. I'm thankful that they exist and get better every year.

That said, models use the slow changing world of the past to predict a rapidly changing world. As old patterns change into new unpredictable patterns they lose their power until revised.

The fact that at all levels punches must be pulled or else climate change becomes unbelievable makes me distrust the models even more.

[FishOutofWater]

I don't participate in the predictions here because I don't have a scientific basis yet for making a prediction. The opinions of a group of people who are mostly non-scientists have, unsurprisingly, proven to be biased and inaccurate. Of course, people who aren't using scientific models but who are excited by sea ice loss have been biased to larger losses than have been observed. It's a prediction based on emotion.

[Dharma Rupa]

I don't participate in the predictions here because I don't have a scientific basis yet for making a prediction. The opinions of a group of people who are mostly non-scientists have, unsurprisingly, proven to be biased and inaccurate. Of course, people who aren't using scientific models but who are excited by sea ice loss have been biased to larger losses than have been observed. It's a prediction based on emotion.

I've stopped participating in the predictions, but I don't think the models are scientific.  In fact, I don't think the models can be scientific, at least not until we have a few hundred Earths or more than a few dozen ice ages to works with.

There's a reason that Economics is called the dismal science, and any "science" of a climate that we can effect by our actions suffers from the same basic fault.  Our actions change the laws.

We might come up with heuristics that seem to work, but we cannot come up with fixed rules that actually define what is happening.

P.S.  Working with heuristics is Engineering, not Science.

[magnamentis]

What this poll consistently shows is that most people are more on the alarmist side and always expect the situation to be worse than it is

and if it is as you say it makes totally sense, ask yourself why this forum exists and why most of us are here ??

BECAUSE WE ARE ALARMED (and rightly so)

and i think it's good that this is how it is, the rest is interpretation of what's going on, depending on what we pay more attention to.

looking at graphs and images one could expect a faster melt-out and looking at a variation of feedbacks and whether patterns etc. one could be a bit closer to what becomes true.

as i said earlier, this year we're not as far of the mark as before and it will be to be seen whether
there is a learning process or it's only a coincidence.

however, considering that quite a lot is going down the river on space ship earth, i can't see why "alarmist" can be used to express something negative.

humanity as a whole should perhaps be a bit (much more) alarmed, not less!

[RoxTheGeologist]

IMHO the poll is very useful, as useful as doing Lab QC with known test results; you can identify bias in methods, standards, procedures and personnel. It's not a waste of time to do analysis.

Why does the forum over-predict ice loss? Maybe it's a lesson for the climate change policy advocates among us. Use data and analysis rather than guess work. We gain credibility if our arguments and predictions are harder to shoot down.

[DavidR]

We are just following an established technique for identifying an answer when better data is not available:  https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

Until we can predict the late summer weather this is about good as we can expect.  SIPN uses a similar approach: https://www.arcus.org/sipn.

Most of their contributers went low in August:

https://www.arcus.org/sipn/sea-ice-outlook/2018/august

[Steven]

With 2 days to go, it seems almost certain that the NSIDC monthly average sea ice extent for September 2018 will be slightly below 4.75 million km2.  More precisely, 4.72 +/- 0.01 million km2.

Here is the updated graphic showing the distribution of the votes in the June polls for NSIDC September extent on this forum in the past 6 years:





Caveats:

(1) The June 2018 and 2017 polls had wide overlapping bins whereas previous years had smaller, non-overlapping bins.  I split the overlapping bins in the same way as described upthread.

(2) In October 2017, NSIDC changed their methodology for calculating the monthly average extent.  The revised values are slightly lower than in the previous version of the NSIDC dataset and this means that the September 2013 and 2014 extent are now in the 5-5.25 range rather than 5.25-5.5.  However, I think it's more fair to use the original values that were used back then, rather than a revision several years later with different methodology.

[Steven]

I have also made a variant of a graph that Ned W posted in the topmost post of this thread (and updated it with the last 2 years of data).

The blue line segments in the graph below show the median predictions for NSIDC September extent in the June, July and August polls on this forum in the past 6 years.  The red line segments show the median predictions by the Sea Ice Prediction Network.  Finally, the diamond symbols in the graph show the observed NSIDC September extent for each year.  (I used version 2.1 of the NSIDC monthly extent dataset for 2013-2017).

[Phil.]

(2) In October 2017, NSIDC changed their methodology for calculating the monthly average extent.  The revised values are slightly lower than in the previous version of the NSIDC dataset and this means that the September 2013 and 2014 extent are now in the 5-5.25 range rather than 5.25-5.5.  However, I think it's more fair to use the original values that were used back then, rather than a revision several years later with different methodology.

Although I think it's worth pointing out that this change was in response to users who queried why the monthly NSIDC results didn't agree with their intuitive calculation based on the daily data (some of them posters on this site).  It's possible that some of those participating in the poll were using that more intuitive method when making their predictions.