Accuracy of poll predictions

[Steven]

It's possible that some of those participating in the poll were using that more intuitive method when making their predictions.

Possibly.  But I think most people base their vote on the September monthly extent values for previous years.  The choice of methodology used by NSIDC affects the monthly values for previous years too, so people probably adjust their expectations to it.

Not that it matters much.  It only affects the bins for the 2013 and 2014 September extent.  For the other years the bin remains the same for the old and new version.  My graphics above would change as follows when using the revised NSIDC September extent values for 2013-2017: https://i.imgur.com/Q1lgGeN.png,  https://i.imgur.com/3jnsLn4.png.

[gerontocrat]

Way back in late May / early June my thought was simple - 2017 extent loss really low because of cool cloudy summer, if 2018 is even a little bit more "normal" extent at minimum must be a good bit, say 250k+ lower than 2017, what with the late Feb early March warmth bomb weakening ice formation / thickening at a really cold time of year.

I was right. My timing was off, is all. Jaxa SIE at 30 Sept is 309k less than 2017 on that date. So what is a mere 9 days? I claim a prize for being wrongly right or rightly wrong.

[LDorey]

I have also made a variant of a graph that Ned W posted in the topmost post of this thread (and updated it with the last 2 years of data).

The blue line segments in the graph below show the median predictions for NSIDC September extent in the June, July and August polls on this forum in the past 6 years.  The red line segments show the median predictions by the Sea Ice Prediction Network.  Finally, the diamond symbols in the graph show the observed NSIDC September extent for each year.  (I used version 2.1 of the NSIDC monthly extent dataset for 2013-2017).

Nice...so prediction skill is increasing with experience, implying a better informed forum ( not me so much, my predictions still suck) but looks like on average the forum is getting better, which is cool ...so good job everyone (seriously it's awesome to see)..., so, I mean summer has always been the big draw, but lately the freeze up (or lack therefor of) is getting pretty cool topic(or should that be hot topic, or above average tempurature topic? ) Anyway how about some winter poles (phunny or too much? )

How fast jaxa will hit x m km3 or when (if?) the barrens will freez or hit some number that means it effectively froze? The october average? the discussions / justifications are always facinating.

Liam

[Ned W]

I have also made a variant of a graph that Ned W posted in the topmost post of this thread (and updated it with the last 2 years of data).

The blue line segments in the graph below show the median predictions for NSIDC September extent in the June, July and August polls on this forum in the past 6 years.  The red line segments show the median predictions by the Sea Ice Prediction Network.  Finally, the diamond symbols in the graph show the observed NSIDC September extent for each year.  (I used version 2.1 of the NSIDC monthly extent dataset for 2013-2017).

Thanks for doing this, Steven.  As always, your "variant" of my graph is better and nicer than the original. 

[magnamentis]

So there may be a kind of "educative" training in the next years, when those members which were wrong in their predictions on the far too low side will vote for higher, more realistic bins in the future?!?

would have been a great post without restricting it the the "far too low side" voters.

if you mean your statements honestly then it has to and indeed applies to ALL voters (that they can learn) who were not spot on.

the way you express it it sounds like a high voter who wants to be right and bash the low voters.

yes i'm a low while not extremely low voter because ther trend is down, we all know it's going down and nobody can predict when, how much each year and in what kind of ups and downs between seasons and intra-seasons.

[Steven]

The JAXA extent minimum for 2019 was 3.96 million km2.

Here is an updated graphic, showing the distribution of the votes in the June polls for JAXA extent on this forum in the past few years.  The correct bin for each year is indicated with a fluo green background.




People on this forum tend to vote too low in the polls.  But the results this year were pretty good, especially compared to previous years.

The July and August polls were a bit worse than June this year:  The median of the votes in the June 2019 poll was 3.73 million km2, in the July poll it was 3.51 million km, and in the August poll it was 3.64.

---

Caveat: note that the format of these polls changed in 2018:

(1) From 2018 onward, the polls had overlapping bins of width 0.5 million km2, whereas in previous years the polls had bins of width 0.25 million km2.  For consistency with previous years, I split each of the overlapping bins in two equal halves and distributed the votes in the bin evenly over its two halves.

(2) The lowest bin in the 2019 poll was labeled "Under 2.00 million km^2".  I decided to treat this bin as 1.5-2.0.  Similarly, the 2018 poll had a lowest bin around 2.5-3.0.  In contrast, the 2017 and 2016 polls had bins all the way down to 0.

[Stephan]

Thank you for that intersting evaluation. Makes me wonder about the outcome of the 2020 melting season polls, but we have to wait for it for a while now...

[HapHazard]

IDK, outside of 2017 I'd say the voters are normally close enough.

[oren]

2017 had extremely low volume, so voters had good reasons to bet low. But it turns out initial volume is not the deciding factor, as the melting weather variability + export variability affects more thickness than the winter variability. Of course when all factors are aligned a nasty surprise could manifest.

[Klondike Kat]

IDK, outside of 2017 I'd say the voters are normally close enough.

I would disagree.
In 2016, 91% voted too low.
In 2017, 96.8% were too low.
In 2018, 78.5% were too low,
This year, only 51.3%, but that required a last minut push to fall below 4.
Hence, only this year was close.

[HapHazard]

It's all in how we translate the data, innit. 

[RoxTheGeologist]

My typical method is to pick the bin above the median.

[snrjon]

A better poll might be to ask for P10, P50 and P90 estimates. At least then there's a chance that the result might get "in range" 😉

[Pmt111500]

My typical method is to pick the bin above the median.

My typical method is to pick the bin below the median.

[Darvince]

I think framing the possible values as between a non-zero value (2.5M for 2018 and 2M for 2019) and 5 million km² has done a lot of the work in improving the poll results, more than any learning that has gone on among the population of the forum, though the even-headedness here, at least in 2018, seemed to be higher than for previous years. This year seemed to have had more alarm about the ice than last year, but it also, more than 2015 and '17, was borne out as justified.

[SimonF92]

Great post Stephen, thanks for taking the time to plot it.

This is the first year we didn't get any renegades going >5 or <1. Data looks most normally distributed too, which indicates opinions were more cohesive in 2019.

[Juan C. García]

Great post Stephen, thanks for taking the time to plot it.

This is the first year we didn't get any renegades going >5 or <1. Data looks most normally distributed too, which indicates opinions were more cohesive in 2019.

On the <1, maybe it is because I just gave the option: "Under 2.00 million km^2" 

[Steven]

Here is the updated graphic for NSIDC September extent.  The 2019 value was 4.32 million km2.




Notes:

(1) From 2017 onward the polls have overlapping bins, whereas previous years had smaller, non-overlapping bins.  I split the overlapping bins as described upthread.

(2) In October 2017, NSIDC changed their methodology for calculating the monthly average extent.  The revised values are slightly lower than in the previous version of the NSIDC dataset.  This means that the September 2013 and 2014 extent are now slightly below 5.25 rather than above 5.25.  However, I think it's more fair to use the original values that were used back then, rather than a revision several years later with different methodology.

(3) The September 2018 extent was reported last year as 4.71, but recently this number has been revised upward to 4.79.  In this graph I prefer to keep the original value that was reported back then.

[Steven]

Here is an update of the graph for NSIDC September extent in Reply #48 upthread.




The blue line segments show the median predictions in the NSIDC extent polls on this forum. 

The red line segments show the median predictions by the Sea Ice Prediction Network (SIPN). 

The diamond symbols show the observed NSIDC September extent for each year.

[blumenkraft]

Those are cool! Thank you, Steven.

[Klondike Kat]

Seems about right.

[Steven]

I've updated the graphics to include 2020.  The poll predictions were pretty accurate this year.

JAXA June polls:


NSIDC June polls:


NSIDC poll medians:

[Stephan]

Very interesting, indeed.
Thank you for the evaluation.
+

[Steven]

The JAXA extent minimum for 2021 was 4.61 million km2.

This was not a good year for the accuracy of the poll predictions on this forum.  The median prediction in the JAXA June 2021 poll was 3.66 million km2, in the July poll it was 3.77 and in the August poll it was 4.16 million km2.

[Stephan]

Thank you for the evaluation. Very much appreciated.