Support the Arctic Sea Ice Forum and Blog

Author Topic: September Predictions Challenge  (Read 28033 times)

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
September Predictions Challenge
« on: June 09, 2018, 07:47:24 PM »
This is a challenge which accumulates scores across all the monthly polls on September sea ice to come up with an overall rating of how good the predictions made are. In addition to making a prediction, entrants are required to rate their confidence in that prediction. The higher the confidence, the narrower the margin of error you are allowed, but the higher score you get if the September ice ends up where you predicted it (and the bigger penalty you take if you miss).

Currently included polls are: JAXA daily minimum area, NSIDC September average.
Other polls may be added during the course of the season if their structure fits this challenge.

Points are scored as follows:
Very High Confidence: 10 points if you pick the correct bin, -10 points for all other bins.
High Confidence: 6 points for the correct bin, 2 points if one bin out, -2 points if two bins out, -6 points for all other bins
Medium Confidence: 4 points for the correct bin, 2 points if one bin out, 1 point if two bins out, -1 if three bins out, -2 if four bins out, -4 points for all other bins
Low Confidence: 2 points for the correct bin, 1 point if within 3 bins, -1 point if 4-6 bins out, -2 points for all other bins
Very Low Confidence: 1 point if in the correct bin, no score (or penalty) for any other bin.

Added for July.
Note on portmanteau and end of range bins: these are excluded from the challenge, you may either select a bin within the portmanteau range, or extend the range beyond the poll endpoint for your challenge entry. If the September values fall out of the normal range, scores will be assessed by extending the bin structure. e.g. an end result of 5.9 counts as two bins out for a 5-5.5 entry (currently the top bin in the July polls)

To enter, post guess and confidence in this thread before the closing date of the poll. Editting a post to change the prediction before the closing date is allowed, editting for any reason after the closing date for a poll will result in disqualification from the challenge.

Currently active polls are:
None

There is no requirement to enter all polls or to match your votes in the predictions threads.

Entries on the following polls have been completed
 June JAXA daily minimum area, https://forum.arctic-sea-ice.net/index.php/topic,2322.0.html
 June NSIDC SIE September average, https://forum.arctic-sea-ice.net/index.php/topic,2324.0.html
 July JAXA daily minimum area, https://forum.arctic-sea-ice.net/index.php/topic,2339.0.html
 July NSIDC SIE September average, https://forum.arctic-sea-ice.net/index.php/topic,2340.0.html
August poll for JAXA daily minimum https://forum.arctic-sea-ice.net/index.php/topic,2362.0.html
August poll for NSIDC Sept average https://forum.arctic-sea-ice.net/index.php/topic,2361.0.html

JAXA daily minimum was 4.46
NSUDC Sept average was 4.71

Compilation of entries

Richard Rathbone
June
JAXA - 3.75-4.25, Medium 2
NSDIC - 4.0-4.5, Medium 2
July
NSIDC av 4.5-5.0,  High 6
JAXA min 4.25-4.75, High 6
August
NSIDC: 4.75-5.25, High 2
JAXA: 4.25-4.75, High 6

Total 24

Paddy
June
JAXA - 3.75-4.25, Low 1
NSDIC - 4.25-4.75, Low 2
July
JAXA - 4.0-4.5, Low 2
NSDIC - 4.5-5.0, Low 2
August
JAXA: 4.0 to 4.5, Medium 4
NSIDC: 4.5 to 5.0, Medium 4

Total 15

Brigantine

June
JAXA - 4.00-4.50, Medium 4
NSIDC - 4.25-4.75, Medium 4
July entry:
JAXA - 4.25-4.75 Medium 4
NSIDC - 4.50-5.00 Medium 4
August entry:
JAXA - 4.00-4.50, High 6
NSIDC - 4.50-5.00, High 6

Total 28

Neven
June
NSIDC: Between 4.5 and 5.0 million km2, None 1
JAXA: 4.0 - 4.5 M km^2, None 1
July
JAXA minimum extent:      4.25-4.75,  Medium 4
NSIDC September extent:  4.5-5.0,  Medium 4
August
JAXA minimum extent:      4.0-4.5,  Medium 4
NSIDC September extent:  4.5-5.0,  High 6

Total 20

EgalSust
June
JAXA - 4.0-4.5, Medium 4
NSIDC - 4.5-5.0, Medium 4
July
JAXA - 4.25-4.75, High 6
NSIDC - 4.75-5.25, High 2
August
JAXA - 4.25-4.75, High 6
NSIDC - 4.75-5.25, High 2

Total 24

jdallen
June:
JAXA - 3.75-4.25, Medium 2
NSDIC - 4.0-4.5, Medium 2

Total 4

Daniel B.
June
NSIDC  4.5-5.0, High 6
JAXA  4.0-4.5, Medium 4

Total 10

Stephan
June
JAXA 4.0 to 4.5 Medium 4
NSIDC 4.5 to 5.0 High 6

Total 10

Viggy
June
JAXA - New record, Very High (this is ambiguous, 3.18 fits in two bins, but due to the comment I am taking it as 2.75-3.25, though 3.0 - 3.5 H would also have been a possible choice by the reasoning expressed there) -10
NSIDC - 2.75 - 3.25, High -6
August
JAXA: 3.25 to 3.75, High -6
NSIDC: 3.75 to 4.25, Very High -10

Total -32

gerontocrat
June
NSIDC - 4.0 to 4.5, none 0
JAXA   -  3.75 to 4.25, none 0

RealityCheck
June
JAXA min 3.75-4.25; high conf 2
NSIDC av 4.25-4.75; high conf 6

Total 8

Juan C. Garcia
June:
JAXA - 3.75-4.25, Medium 2
NSDIC - 4.0-4.5, Medium 2
July
JAXA minimum extent:      4.25-4.75,  Medium 4
NSIDC September extent:  4.5-5.0,  Medium 4

Total 12

uniquorn
June
JAXA min 3.5-4.0; medium 1
NSIDC av 3.5-4.0; low 1
July
JAXA min 3.5-4.0; medium 1
NSIDC av 3.75-4.25; medium 1
August
JAXA - 3.5-4.0, medium 1
NSIDC - 3.75-4.25 medium 1

Total 6

Steven
June
JAXA:    3.75-4.25,  Medium 2
NSIDC:  4.0-4.5,  Medium 2
July
JAXA minimum extent:      4.25-4.75,  Medium 4
NSIDC September extent:  4.75-5.25,  Medium 2
August
JAXA minimum extent:      4.0-4.5, High 6
NSIDC September extent:  4.5-5.0,  High 6

Total 22

Tetra
August
JAXA: 4.0 to 4.5, Medium 4
NSIDC: 4.5 to 5.0, Medium 4

Total 8

liefde
JAXA ASI September daily minimum: Between 3.5 and 4.0 million km^2, Very High -10
NSIDC SIE September average: Between 4.25 and 4.75 million km^2, Very High 10

Total 0

« Last Edit: October 06, 2018, 01:08:13 PM by Richard Rathbone »

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #1 on: June 09, 2018, 07:53:35 PM »
JAXA - 3.75-4.25, Medium
NSDIC - 4.0-4.5, Medium

i.e. JAXA will score positive points for me if it ends up between 3.25-4.75 and NSDIC if it ends up between 3.5 and 5.0


Tor Bejnar

  • Young ice
  • Posts: 4606
    • View Profile
  • Liked: 879
  • Likes Given: 826
Re: September Predictions Challenge
« Reply #2 on: June 09, 2018, 08:10:19 PM »
I "predict" (defined as "guess") ASI extent, area and volume will just beat out the current records in September.  My confidence is 'extremely' low, so I should get no points for guessing right (or wrong), and 1 point for honesty. ::)
Arctic ice is healthy for children and other living things because "we cannot negotiate with the melting point of ice"

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #3 on: June 10, 2018, 01:43:30 PM »
JAXA - 3.75-4.25, Low
NSDIC - 4.25-4.75, Low

I'm betting on "something similar to 2016" for both polls, but with wide confidence intervals that should cover most of the plausible outcomes.  I was tempted by "Medium" confidence as offering greater rewards in terms of points, but at this point "Low" felt more honest to the actual range of outcomes that could lie ahead.

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #4 on: June 12, 2018, 03:43:39 AM »
JAXA - 4.00-4.50, Medium
NSIDC - 4.25-4.75, Medium

I approve of this game! It's a great idea.
« Last Edit: June 13, 2018, 01:45:18 AM by Brigantine »

Neven

  • Administrator
  • First-year ice
  • Posts: 9712
    • View Profile
    • Arctic Sea Ice Blog
  • Liked: 1372
  • Likes Given: 620
Re: September Predictions Challenge
« Reply #5 on: June 12, 2018, 10:04:59 AM »
Okay, I'll play too.  :)

June polls

NSIDC: Between 4.5 and 5.0 million km2
JAXA: 4.0 - 4.5 M km^2
Make money, not peace

be cause

  • Young ice
  • Posts: 2584
    • View Profile
  • Liked: 1079
  • Likes Given: 1139
Re: September Predictions Challenge
« Reply #6 on: June 12, 2018, 10:44:39 AM »
hi Neven ; I note you have no confidence in your predictions .. :) b.c.
We live in a Quantum universe . Do you live like you do ?

EgalSust

  • New ice
  • Posts: 36
    • View Profile
  • Liked: 1
  • Likes Given: 1
Re: September Predictions Challenge
« Reply #7 on: June 12, 2018, 05:07:58 PM »
June:

JAXA - 4.0-4.5, Medium
NSIDC - 4.5-5.0, Medium

Low confidence would have been the honest choice...

edit: NSDIC -> NSIDC
« Last Edit: June 12, 2018, 06:16:21 PM by EgalSust »

gerontocrat

  • Multi-year ice
  • Posts: 22166
    • View Profile
  • Liked: 5435
  • Likes Given: 70
Re: September Predictions Challenge
« Reply #8 on: June 12, 2018, 05:56:02 PM »
At what time (GMT) on what date do these polls close ?

Waiting to the last moment to get the last ounce of data ( from Mystic Meg and The Farmers' Almanac ) before posting my wholly scientific prediction.
"Para a Causa do Povo a Luta Continua!"
"And that's all I'm going to say about that". Forrest Gump
"Damn, I wanted to see what happened next" (Epitaph)

jdallen

  • Young ice
  • Posts: 3433
    • View Profile
  • Liked: 670
  • Likes Given: 254
Re: September Predictions Challenge
« Reply #9 on: June 12, 2018, 06:04:22 PM »
June:
JAXA - 3.75-4.25, Medium
NSDIC - 4.0-4.5, Medium
This space for Rent.

Daniel B.

  • Grease ice
  • Posts: 659
    • View Profile
  • Liked: 3
  • Likes Given: 29
Re: September Predictions Challenge
« Reply #10 on: June 12, 2018, 06:18:24 PM »
NSIDC  4.5-5.0, High (6 of the past 10 years have been in this bin, averaging right in the middle, with only one year more than one bin away)

JAXA  4.0-4.5, Medium (Much greater spread in the data over the past decade.  Although 5 out of 10 have been in this bin, 4 out of 10 being more than one bin away)

Stephan

  • Young ice
  • Posts: 2786
    • View Profile
  • Liked: 782
  • Likes Given: 490
Re: September Predictions Challenge
« Reply #11 on: June 12, 2018, 10:04:59 PM »
This is how I voted in the two threads:
JAXA 4.0 to 4.5 Medium
NSIDC 4.5 to 5.0 High
It is too late just to be concerned about Climate Change

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #12 on: June 12, 2018, 10:51:20 PM »
At what time (GMT) on what date do these polls close ?

Waiting to the last moment to get the last ounce of data ( from Mystic Meg and The Farmers' Almanac ) before posting my wholly scientific prediction.

2359 BST is the last timestamp that counts. i.e. about 4 days, 2 hours and 8 minutes from now.

Viggy

  • New ice
  • Posts: 83
    • View Profile
  • Liked: 51
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #13 on: June 12, 2018, 11:03:26 PM »
Chalk it up to me being an eternal pessimist but here goes literally nothing -
JAXA - New record, Very High (id like to vote High but the fact that there isn't a 1 bin out on the lower end leads me to think I should just commit to it)
NSIDC - 2.75 - 3.25, High (Given that we are 2 prediction bands below the previous low, I'll hedge with a sensible High bet here)

Win (Lose) big or go home!

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #14 on: June 16, 2018, 12:53:37 PM »
The June polls are all closed. This is the last day for challenge entries. Anyone waiting for the last little bit of data to guide them has about 12 hours left.

gerontocrat

  • Multi-year ice
  • Posts: 22166
    • View Profile
  • Liked: 5435
  • Likes Given: 70
Re: September Predictions Challenge
« Reply #15 on: June 16, 2018, 02:18:35 PM »
The June polls are all closed. This is the last day for challenge entries. Anyone waiting for the last little bit of data to guide them has about 12 hours left.
I think I went
NSIDC - 4.0 to 4.5
JAXA   -  3.75 to 4.25

My confidence is zero.
"Para a Causa do Povo a Luta Continua!"
"And that's all I'm going to say about that". Forrest Gump
"Damn, I wanted to see what happened next" (Epitaph)

RealityCheck

  • Frazil ice
  • Posts: 174
    • View Profile
  • Liked: 45
  • Likes Given: 6
Re: September Predictions Challenge
« Reply #16 on: June 16, 2018, 04:54:38 PM »
Here goes...
JAXA min 3.75-4.25; high conf
NSIDC av 4.25-4.75; high conf
(High conf due to recently discovered crystal ball, now polished, plus a pinch of 'what the heck' feeling. 😁
Sic transit gloria mundi

Juan C. García

  • Young ice
  • Posts: 3360
    • View Profile
  • Liked: 1282
  • Likes Given: 1138
Re: September Predictions Challenge
« Reply #17 on: June 16, 2018, 05:01:55 PM »
June:
JAXA - 3.75-4.25, Medium
NSDIC - 4.0-4.5, Medium


I voted the same way that jdallen.
Confidence? I think that there is a low confidence for everybody, including me. But let's say that I could expect one bin up, but several bins down.
Which is the best answer to Sep-2012 ASI lost (compared to 1979-2000)?
50% [NSIDC Extent] or
73% [PIOMAS Volume]

Volume is harder to measure than extent, but 3-dimensional space is real, 2D's hide ~50% thickness gone.
-> IPCC/NSIDC trends [based on extent] underestimate the real speed of ASI lost.

uniquorn

  • First-year ice
  • Posts: 5342
    • View Profile
  • Liked: 2279
  • Likes Given: 393
Re: September Predictions Challenge
« Reply #18 on: June 16, 2018, 05:02:55 PM »
JAXA min 3.5-4.0; medium
NSIDC av 3.5-4.0; low

Flatter bottom.

jdallen

  • Young ice
  • Posts: 3433
    • View Profile
  • Liked: 670
  • Likes Given: 254
Re: September Predictions Challenge
« Reply #19 on: June 16, 2018, 07:14:02 PM »
June:
JAXA - 3.75-4.25, Medium
NSDIC - 4.0-4.5, Medium


I voted the same way that jdallen.
Confidence? I think that there is a low confidence for everybody, including me. But let's say that I could expect one bin up, but several bins down.

I figure my choices have about a 50-50 chance of being correct, thus Medium.

And I agree whole heartedly with your expectation about bins.
This space for Rent.

Steven

  • Nilas ice
  • Posts: 1062
    • View Profile
    • Arctic sea ice data and graphs
  • Liked: 555
  • Likes Given: 19
Re: September Predictions Challenge
« Reply #20 on: June 16, 2018, 09:35:55 PM »
JAXA:    3.75-4.25,  Medium
NSIDC:  4.0-4.5,  Medium

This is based on the long term linear trends for September/minimum extent.

I'm not sure what to expect.  On the one hand, NSIDC sea ice area and compactness are pretty high at the moment.  That suggests a mediocre melt season.  On the other hand, warm air advection on the Siberian side of the Arctic has been strong.  So far this mainly affected Kara and Laptev Sea.  It remains to be seen how much it will affect the East Siberian Sea, and whether or not the ice arm in the East Siberian Sea could melt out this year.  Chukchi and Beaufort Sea should have thinner than normal ice according to PIOMAS, but it looks like weather conditions in those regions have been favorable for ice retention so far.

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #21 on: June 17, 2018, 01:19:12 AM »
June entries are now closed. Looks like an interesting mix of confidence.

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #22 on: June 20, 2018, 08:22:23 AM »
After the fact, I know, but I think the scoring system is a little prejudicial in giving people with a High or Medium confidence significantly better point rewards, eg as good a score when one bin out as people with a Low confidence would get for the correct bin only...

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #23 on: June 20, 2018, 09:32:38 AM »
After the fact, I know, but I think the scoring system is a little prejudicial in giving people with a High or Medium confidence significantly better point rewards, eg as good a score when one bin out as people with a Low confidence would get for the correct bin only...
But significantly higher penalties if they're 10 bins out. The game's about avoiding penalties as much as it's about scoring points.

You have to get you average in positive territory, before you worry about how high you can go. I suspect most people with high confidence in June will end up on negative points come September, while most people with low confidence will be slightly positive and have a head start next year.

(I went medium confidence, shooting for slightly higher points at a calculated risk of taking negatives instead. In July I'll probably go high confidence at a greater reward but even greater risk)
« Last Edit: June 20, 2018, 09:51:06 AM by Brigantine »

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #24 on: June 20, 2018, 07:44:26 PM »
After the fact, I know, but I think the scoring system is a little prejudicial in giving people with a High or Medium confidence significantly better point rewards, eg as good a score when one bin out as people with a Low confidence would get for the correct bin only...

This is by intent. The lower the confidence level, the further away from the correct bin the rewards crossover. The challenge is intended to reward both being accurate, and having a realistic appreciation of your level of accuracy, and this is a necessary part of that design.

In practice there aren't enough independent polls to really sort entrants by skill, and Napoleon's dictum applies, its better to be lucky than good.  ;)

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #25 on: June 23, 2018, 05:32:36 PM »
SIPN median is 4.6, perhaps a touch higher than entrants here, but not much in it.

(I'd rate Rob Dekker's SIPN entry as  4.5 - 5.0  M  here, Nico Sun looks like 4.0 - 4.5 M but there are inconsistencies in the confidence measures he quotes, and it might be VL or H, Slater method 4.75 - 5.25 M, UCL model 4.0 - 4.5 L and the model of the Met Office 4.75 - 5.25 L)


Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #26 on: June 23, 2018, 06:58:32 PM »
Quote

This is by intent. The lower the confidence level, the further away from the correct bin the rewards crossover. The challenge is intended to reward both being accurate, and having a realistic appreciation of your level of accuracy, and this is a necessary part of that design.

In practice there aren't enough independent polls to really sort entrants by skill, and Napoleon's dictum applies, its better to be lucky than good.  ;)

I get it, it just seems a bit odd that "Low" confidence gets worse marks when it's one bin wrong than "Medium" or "High". If they all had the same reward for a near miss (either 1 or 2 points, with an equivalent switch in the equivajent penalty slot) I think we would have seen a broader range of confidence levels taken.

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #27 on: June 23, 2018, 07:41:13 PM »


I get it, it just seems a bit odd that "Low" confidence gets worse marks when it's one bin wrong than "Medium" or "High". If they all had the same reward for a near miss (either 1 or 2 points, with an equivalent switch in the equivajent penalty slot) I think we would have seen a broader range of confidence levels taken.

What counts as a near miss is further away and more spread out for low confidence than for high confidence.

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #28 on: June 24, 2018, 01:44:25 AM »
I guess the point is that for low confidence, 1 bin out is still a 'hit' and not a near miss at all, and it is slightly jarring that the reward has already dropped by half. Whereas the higher levels of confidence have rewards that look more like low-res bell curves. (even though they're not curved).

But I can accept the point that we want people to be trying to guess the exact correct bin, even if it has a low probability. Ultimately winning +1 point from a June poll is nothing to sneeze at. If the confidence levels had been described in reverse order, there would be no anchor at 10 points, so +1 point wouldn't have felt like such a raw deal.

Peter Ellis

  • Grease ice
  • Posts: 619
    • View Profile
  • Liked: 33
  • Likes Given: 14
Re: September Predictions Challenge
« Reply #29 on: June 24, 2018, 01:12:08 PM »
Maybe it helps to reframe it.  Think of it like betting on a roulette wheel.  Each of us gets a hundred tokens.  Distribute those across the various bins, and at the end of the day, your score is however many tokens you put in the correct bin.

High confidence means you stack all your tokens in one bin.  If you're right, you score 100, otherwise you get nothing.

Medium confidence means you put 50 in one of the bins, and the rest in the adjacent bins.  If you're dead on - well you still only score 50, because you weren't really sure about it.  If you weren't quite dead on, you get to pick up 25 points or so because you had at least a few tokens in the right bin.

Low confidence means distributing your tokens widely.  Sure, you'll pick up a few points no matter what the result is, but only a few because you basically pulled the numbers out of your fundament.

Or, put yet another way, saying you have "high confidence" in one bin is the same as saying you're certain it won't be in any of the other bins - so why would you expect to score points for them?

Peter Ellis

  • Grease ice
  • Posts: 619
    • View Profile
  • Liked: 33
  • Likes Given: 14
Re: September Predictions Challenge
« Reply #30 on: June 24, 2018, 01:13:27 PM »
(Side note - next year it might be interesting to try the "distribute 100 tokens" approach since that lets people give a more nuanced description of their confidence intervals, which could be non-normal, asymmetric or even bimodal.)

oren

  • Moderator
  • First-year ice
  • Posts: 9993
    • View Profile
  • Liked: 3674
  • Likes Given: 4248
Re: September Predictions Challenge
« Reply #31 on: June 24, 2018, 06:08:58 PM »
Well explained and a good idea Peter.

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #32 on: June 25, 2018, 04:36:19 PM »
(Side note - next year it might be interesting to try the "distribute 100 tokens" approach since that lets people give a more nuanced description of their confidence intervals, which could be non-normal, asymmetric or even bimodal.)

However, a reward function that rewarded this sort of entry for getting its distribution right would not be simple. This simple reward of +1 per token in the right bin rewards overconfidence, since whatever the actual uncertainty, putting every token on the mode is the optimum play.

If you have a strongly non-normal distribution of your expectations, such that all the other options have negative expectations, thats what my VL confidence option is for. If the arctic behaviour actually fits your expectations, its the winning play.

This is actually a 10-token game, with restrictions on how you can place them, and the penalties for not having a token in the winning bin are how I deal with penalising the overconfidence that this type of structure would otherwise reward.

 

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #33 on: June 26, 2018, 09:00:19 AM »
Only a 10-token game for medium and high confidence, though. For low confidence it's an 8-token game (if we ignore the away-from-mode negative penalties).

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #34 on: June 26, 2018, 09:07:08 AM »
Not quite, because the there's a 0.5 Mm2 range of final values that land you in the correct bin, while each bin further away only covers 0.25Mm2 on each side. So it's more like 20 for VH, 16 for High, 14 for Medium, 10 for Low and 2 for VL.

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #35 on: June 26, 2018, 10:35:48 PM »
Ok... but let's also consider the penalties now (up to 3 bins out). And let's also reduce the effective value of each benefit or penalty by 25% of the points/penalties available for each step from the modal value (ie multiply by 0.75 for 1 step out, 0.5 for 2 steps out, and 0.25 for 3 steps out) as the further they are from your modal guess, the less likely you consider them to be.

That makes the net benefit available  -10 for VH, 10 for H, 11.5 for M, 7 for L, and 2 for VL. (According to my mental arithmetic, anyway).

The numbers will shift slightly depending on how you model this, but I'm not surprised most people opted for M.

EDIT: considering these instead as predicted mean points return according to a model of guessing as above where you're right 4 times in ten, wrong by 1 bin 3 times in ten, wrong by two bins 2 times in ten, and wrong by 3 bins 1 time in ten, this gives the average predicted points return of each level of confidence at:
VH: -2
H: 2
M: 2.3
L: 1.4
VL: 0.4

Again, according to my mental arithmetic, which is feeling a tad strained just now.
« Last Edit: June 26, 2018, 10:48:28 PM by Paddy »

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #36 on: June 26, 2018, 10:48:36 PM »
By choosing to go 3 bins out, you're implicitly judging this from the perspective of someone with medium confidence.

How about judge it from the perspective of someone with quite low confidence, and look 9 bins out:
(weights: 1, 0.9, 0.8 etc. to 0.1 9 bins out.)

-70 for VH, -21.2 for H, -2.6 for M, +3.4 for L, +2.0 for VL

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #37 on: June 26, 2018, 11:53:30 PM »
Overall, the different confidence levels would pay off best at the following ranges, by this model:

VH: If this model goes 0 bins out
H: If this model goes 1 bin out (weighted at 1, 0.5) or 2 bins out (weighted at 1, 0.67, 0.33)
M: 3 bins out to X bins out (and I cannot be bothered to solve for X, but I'd guess it's around the 5 mark)
Etc

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #38 on: June 27, 2018, 02:08:56 AM »
(Side note - next year it might be interesting to try the "distribute 100 tokens" approach since that lets people give a more nuanced description of their confidence intervals, which could be non-normal, asymmetric or even bimodal.)
However, a reward function that rewarded this sort of entry for getting its distribution right would not be simple.
I'd like to suggest this reward function: (TLDR: logs)

If the final result is in a bin where you placed X tokens, you win A*log(X)-C

Where A and C are constants set to scale the range of outcomes from perfect skill = +10, and an honest zero skill = ±0.
A = 10/log[number of bins], C = A*log(100)-10
Assuming you use 25 non-overlapping 0.25Mm2 bins, from <0.25 up to 6.00+, and using natural logs... A = 3.1067, C = 4.3068


If you were so sure the result wouldn't be in a particular bin, that you put 0.00 tokens there, and then it really happens... technically you should be completely discredited and should end up on minus infinity. But without allowing fractional tokens, we can be generous and call it -10 - as though you had placed 0.16 tokens there.

Example scoring: Final result lands in a bin where you placed...
0 tokens -> -10.00 points
1 token --> -4.31
2 tokens -> -2.15
3 tokens -> -0.89
4 tokens -> ±0.00 points (100 tokens spread evenly across 25 bins will get you here, i.e. honest zero skill)
5 tokens -> +0.69
6 tokens -> +1.26
7 tokens -> +1.74
10 tokens -> +2.85
11 tokens -> +3.14
20 tokens -> +5.00 points
21 tokens -> +5.15
50 tokens -> +7.85
51 tokens -> +7.91
90 tokens -> +9.67
91 tokens -> +9.71
99 tokens -> +9.97
100 tokens +10.00 points

For each token you add to a bin up to matching the probability, you expect to gain at least ~0.03 points, but stacking extra tokens in a bin beyond its probability earns you less than ~0.03 for each token.


Having said that, I like the game as it is currently (nice and simple), and do not endorse changing it  :D
« Last Edit: June 27, 2018, 02:25:36 AM by Brigantine »

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #39 on: June 28, 2018, 02:34:30 PM »
I hoped these sorts of reward calculations would happen, and I'm a little disappointed that the predictomat didn't do them.

Nearly July, who is still feeling they made the right choice? When I saw the entries coming in I thought I might have pitched too low a confidence at medium, but I'm feeling that it might have been too high. Melt in the later part of June has not only been slow, but slower than I thought possible.

There's going to be a rule change for July if the poll structure stays the same, I didn't look closely enough at the scales and I need to clarify what happens with portmanteau and end of range bins. (they'll be specifically excluded, and the 0.5 wide bin centered on every 0.25 structure will be deemed to continue indefinitely for the purposes of this challenge) I'll write it out explicitly after the July polls are open but please wait till I have posted to say the challenge is open for July entries before posting them.

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #40 on: June 30, 2018, 01:38:19 PM »
Top post editted with details of the July polls. Entries may now be made here.

While I could make my June entry 6 months in advance, for July there are indicators on the current state that I want to see, so I'm not going to be straight off the mark this time. My hunch is up a bin or two and still medium but I'll want PIOMAS to be out and agree with my hunch of where it is before I firm up on that.

Paddy

  • Nilas ice
  • Posts: 1074
    • View Profile
  • Liked: 166
  • Likes Given: 185
Re: September Predictions Challenge
« Reply #41 on: July 01, 2018, 11:03:26 AM »
My predictions:

JAXA - 4.0-4.5, Low
NSDIC - 4.5-5.0, Low

As I've said, "Low" confidence is no way to win points. But I want the positive outcome range to match up to a decent confidence interval, and outliers like 2012 mean that has to be quite wide.

echoughton

  • Frazil ice
  • Posts: 131
    • View Profile
  • Liked: 24
  • Likes Given: 92
Re: September Predictions Challenge
« Reply #42 on: July 01, 2018, 11:49:04 AM »
I would prefer to wait until September to make my bold prediction of SIE extent in September, but I am sticking with my original optimism.
4.50-5-0

EgalSust

  • New ice
  • Posts: 36
    • View Profile
  • Liked: 1
  • Likes Given: 1
Re: September Predictions Challenge
« Reply #43 on: July 01, 2018, 01:56:22 PM »
July
JAXA - 4.25-4.75, High
NSIDC - 4.75-5.25, High

uniquorn

  • First-year ice
  • Posts: 5342
    • View Profile
  • Liked: 2279
  • Likes Given: 393
Re: September Predictions Challenge
« Reply #44 on: July 01, 2018, 09:15:33 PM »
the compilation of entries appears to be incomplete

Brigantine

  • Frazil ice
  • Posts: 345
    • View Profile
  • Liked: 104
  • Likes Given: 345
Re: September Predictions Challenge
« Reply #45 on: July 11, 2018, 08:50:50 AM »
JAXA - 4.00-4.50, Medium
NSIDC - 4.25-4.75, Medium
July entry: (subject to modification in the final days)
JAXA - 4.25-4.75 Medium
NSIDC - 4.50-5.00 Medium

Very tough choosing between medium confidence and high confidence. Where's that predict-o-matic up to re confidence intervals?

Also I like the new rules re end-of-range bins.
« Last Edit: July 12, 2018, 12:24:40 AM by Brigantine »

Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #46 on: July 11, 2018, 12:48:42 PM »
the compilation of entries appears to be incomplete

Several added. It should be complete for June now, if I missed anyone post or message me.


Richard Rathbone

  • Nilas ice
  • Posts: 1901
    • View Profile
  • Liked: 417
  • Likes Given: 25
Re: September Predictions Challenge
« Reply #47 on: July 11, 2018, 01:22:17 PM »
For July polls I like to use momentum indicators, particularly the Schroeder melt pond analysis, but also the June cliffs on PIOMAS volume, and NSIDC area.

May data from Schroeder was 5.3 with confidence given by a 0.5 standard deviation.
June data is 4.7 +- 0.5 but I'm not sure if thats a standard deviation or a confidence range.
June PIOMAS cliff is there but modest. June NSIDC cliff is absent or in the wrong direction but the data where it shoots up is compromised by satellite tests, so I'm just treating it as absent rather than an upwards cliff in June cancelled by a downwards one in July.

I'm picking the 4.5-5.0 bin because 4.7 is in the middle of that and 4.25-4.75 for the minimum because the minimum tends to be about one bin less than the September average. There are quite a few years in the 4.5-5.0 NSIDC pack and a modest cliff in PIOMAS supports this being another one of them.

Assuming a 0.5 standard deviation gives roughly 40% chance of the picked bin being correct, 30% of being 1 out, 15% of being 2 out, 10% of being 3 out and 5% of being 4 or more out.
I make that an expectation of -2.0 points for VH, 1.8 for H, 2.1 for M. VH looks overconfident, but a 0.5 standard deviation leaves H and M pretty closely matched. If I'd had the Schroeder May projection in time I'd definitely have stuck with medium (though my bin choice would have been different). The June projection from Schroeder ought to have a tighter confidence limit so I'm going to guess that the quoted +-0.5 is actually 2 standard deviations and go High.

NSIDC av 4.5-5.0,  High
JAXA min 4.25-4.75, High

Ned W

  • Guest
Re: September Predictions Challenge
« Reply #48 on: July 11, 2018, 04:19:41 PM »
Where's that predict-o-matic up to re confidence intervals?

Thanks for asking.  Unfortunately the predict-o-matic's confidence is never "High".  Here's the latest (for JAXA):

Predicted daily min = 4.71
95% CI for prediction = 3.68 to 5.74

Here are the probabilities for the bins:

New record0%
Under 2.50%
2.5 to 30%
2.75 to 3.250%
3 to 3.51%
3.25 to 3.753%
3.5 to 47%
3.75 to 4.2515%
4 to 4.526%
4.25 to 4.7535%
4.5 to 537%
4.75 to 5.2532%
5 to 5.523%
Over 5.2515%

The probabilities sum to more than 100% because the bins overlap.  The uncertainty is very wide until late July. 

Random_Weather

  • Frazil ice
  • Posts: 102
    • View Profile
  • Liked: 44
  • Likes Given: 1
Re: September Predictions Challenge
« Reply #49 on: July 11, 2018, 04:25:23 PM »
My own is: 4.24 +-0.33 Mio km^2

Will also published in the SIPN July Outlook, for details go here: https://www.docdroid.net/9FEP0oD/model.docx

For Performance: