Support the Arctic Sea Ice Forum and Blog

Author Topic: Maximising asif skills using near real time Mosaic data  (Read 6854 times)

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #150 on: May 19, 2020, 02:25:07 PM »
Several improvements to the webapp

1) Better maps
2) Added new buoys
3) Added function to easily change parameters of the code

uniquorn, i see your point, thats going to be pretty difficult to do though  :-[


Webapp will be kept online for 12-hours. Find it here:

http://52.14.65.105:8501
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

blumenkraft

  • Young ice
  • Posts: 3911
  • Fans of Hans Ø Club - circa 2018
    • View Profile
  • Liked: 1362
  • Likes Given: 2139
Re: Maximising asif skills using near real time Mosaic data
« Reply #151 on: May 19, 2020, 04:14:24 PM »
Amazing, Simon! Well done.
CEO of bitcoin & Antifa

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #152 on: May 20, 2020, 02:51:24 PM »
Amazing, Simon! Well done.

Thanks!

Understandably I have been very careful about keeping track of usage. At the moment, it looks like if i leave the website live indefinitely, I shouldn't incur any charges.


Random looking ip addresses can be off-putting, I get that. For this reason, I just registered a domain name at "www.mosaic-ice.com", which is being processed by amazon. I can then route the IP of the webapp to that domain name and square the project off.

In the meantime, im going to leave it running for 5-days here;




http://18.222.70.206:8501




Id encourage anyone and everyone to check it out, just so I can be sure that multiple users creating their own cache doesnt go over-and-above my free limits.
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

oren

  • First-year ice
  • Posts: 5209
    • View Profile
  • Liked: 1451
  • Likes Given: 1475
Re: Maximising asif skills using near real time Mosaic data
« Reply #153 on: May 20, 2020, 02:59:16 PM »
Simon thanks so much for creating this.
Isn't there some way for you to set limits on usage such that when it gets to the free allotment it stops for the month?
If not, maybe it would help putting the % of quota used on the home screen, so that users will know to avoid using the site when nearing the limit.

oren

  • First-year ice
  • Posts: 5209
    • View Profile
  • Liked: 1451
  • Likes Given: 1475
Re: Maximising asif skills using near real time Mosaic data
« Reply #154 on: May 20, 2020, 03:20:54 PM »
A question. The dropbox enabled easy download of the processed data files. Will it be possible to do the same from the web app site?
I hope I am not bombarding with requests. Still not sure what of all this I will use, just checking the goods...

Edit: I think we have already discussed download, but I haven't found how to on the current site.

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #155 on: May 20, 2020, 03:31:29 PM »
A question. The dropbox enabled easy download of the processed data files. Will it be possible to do the same from the web app site?
I hope I am not bombarding with requests. Still not sure what of all this I will use, just checking the goods...

Edit: I think we have already discussed download, but I haven't found how to on the current site.

I feel amazon deliberately try to lead you down as slope as they dont provide a way to turn off a machine at a cpu-hour threshold.

Thanks for the suggestion about displaying the quota. However, I feel this would potentiate a back-door onto my account, as I would have to write a way of pulling that data to the webapp. I have set a monthly budget of $5, with email alerts at $2.50, so I should be able to stay well on-top of things.

With respect to downloading the datasets. I am sure there must be a way, still working on figuring it out- but its top of my list!
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #156 on: May 20, 2020, 04:06:30 PM »
oren, you should now be able to download csv files straight from the site, let me know how it goes

(I dont own excel so i can only view them as notepad files)
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #157 on: May 20, 2020, 04:57:04 PM »
just downloaded T56. It needs another 12 zeros at the beginning of the 'change_in_thickness(m)' column. Not much of a problem. Site is timing out right now.

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #158 on: May 20, 2020, 05:33:14 PM »
Replaced NaNs with 0s to help with downloading and using csv files.

Rerouting traffic to new website/ domain name (at a very small cost in the region of $0.40 per million lookups):

The port at 8501 is unfortunately a must, and wont be going away any time soon


http://www.mosaic-ice.com:8501/
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

SteveMDFP

  • Nilas ice
  • Posts: 1670
    • View Profile
  • Liked: 330
  • Likes Given: 23
Re: Maximising asif skills using near real time Mosaic data
« Reply #159 on: May 20, 2020, 05:42:45 PM »
The port at 8501 is unfortunately a must, and wont be going away any time soon

http://www.mosaic-ice.com:8501/

"Any port in a storm."

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #160 on: May 20, 2020, 06:44:23 PM »
The port at 8501 is unfortunately a must, and wont be going away any time soon

http://www.mosaic-ice.com:8501/

"Any port in a storm."

10/10, had a giggle
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #161 on: May 26, 2020, 11:02:24 AM »
http://www.mosaic-ice.com:8501/
data currently stops at may21 (needs a reboot occasionally?)
suggestions:
add links to the original files used (temp_proc, ts, TEMP120)
Is there any chance to show all the 6hourly therm1 temperatures? If not a daily average might be better.
Sorry, have no time to look at streamlit at the moment.

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #162 on: May 28, 2020, 10:29:46 PM »
python ice thickness update, may28. Posting here until I'm sure they are reliable.

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #163 on: May 30, 2020, 11:56:57 PM »
Checking Tbuoy thickness
T73 definitely has a problem. The last few rows of T62.
« Last Edit: May 31, 2020, 12:07:16 AM by uniquorn »

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #164 on: June 01, 2020, 03:49:49 PM »
T77 temperatures, heat 30sec and heat 120sec, may10-jun1
melted snow refreezing?

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #165 on: June 01, 2020, 05:32:32 PM »
The t77 chart above shouldn't be showing that much bottom melt. Here is a close look at the ice/ocean thermistor which stays at, or very close to therm130, possibly moving to 129.
Possibly related to the maximum ocean temperature we are using.
The problem may get worse as the temperature gradient levels off. Tricky as there may be super cooled water beneath the ice. edit:I doubt that with warmer air temps.

updated below
« Last Edit: June 01, 2020, 07:52:40 PM by uniquorn »

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #166 on: June 01, 2020, 07:24:26 PM »
The t77 chart above shouldn't be showing that much bottom melt. Here is a close look at the ice/ocean thermistor which stays at, or very close to therm130, possibly moving to 129.
Possibly related to the maximum ocean temperature we are using.
The problem may get worse as the temperature gradient levels off. Tricky as there may be super cooled water beneath the ice. edit:I doubt that with warmer air temps.

Spring (summer?) really is messing with these buoys, I have a feeling the reliability of any measurements will be decreasing exponentially. Just look at their raw data.

By the way I cant edit the website myself (im on the clock for a project these days) but I just found out its imperative that one clears the cache to see the most up-to-date data.

Clearing the cache can be done in the tab on the right of the main page

Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #167 on: June 01, 2020, 08:03:16 PM »
Thanks, streamlit updates with cleared cache. Still checking data but I see the old python version I was using had mean ocean temp of < -2.5C. Editing that to -2.1C gives closer alignment to the bottom melt thermistor (like the streamlit version using -2.2C). I'd like to go lower but there are a few -2.06C deeper down. That drives up ice thickness.
examples below.
Will keep tinkering. maybe <-2.06C ....
« Last Edit: June 02, 2020, 08:50:45 PM by uniquorn »

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #168 on: June 01, 2020, 10:55:39 PM »
Agree completely about the supercooled idea (well not "supercooled")- there is probably a phase-equilibrium across several thermistors at the moment. Possibly supplement with HEAT data? Ill look into it
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #169 on: June 02, 2020, 02:54:00 PM »
Webapp update.

Amazon emailed me suggesting I may be eligible for 300$ account credit based on my activity. Should I qualify ill be able to significantly ramp-up the scope of;

www.mosaic-ice.com:8501

Adding several more features, including the ability to handle more than one dataset at any one time (due to migrating to a faster machine). Fingers crossed.
« Last Edit: June 02, 2020, 06:52:17 PM by SimonF92 »
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #170 on: June 02, 2020, 04:40:53 PM »
Good luck :) as long as you don't block adblockers.
Continuing analysis of T77 here are the temperatures from may18-jun2. I think there was snow between therm39 (possible therm37) and 43 and based on the animation above, I don't think there is bottom melt yet.
Not sure which parameters can be adjusted to match.
click for a gif to get rid of the lines.

blumenkraft

  • Young ice
  • Posts: 3911
  • Fans of Hans Ø Club - circa 2018
    • View Profile
  • Liked: 1362
  • Likes Given: 2139
Re: Maximising asif skills using near real time Mosaic data
« Reply #171 on: June 02, 2020, 04:51:30 PM »
CEO of bitcoin & Antifa

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #172 on: June 02, 2020, 06:52:36 PM »

www.mosaic.ice.com:8501


What's happening here? Is ice.com your domain?

Oops! Fixed


Great figure uniquorn, so much data
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #173 on: June 02, 2020, 08:23:58 PM »
T76 almost makes me want to give up this idea. One of the heaters looks like it's broken and there appears to be an air cavity that has been iced over. Drained before it could freeze?
The estimation seems too low at the moment.  117-41=76, *2=152cm plus maybe 24cm snow(less since the warm spell) click to play
« Last Edit: June 02, 2020, 08:31:53 PM by uniquorn »

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #174 on: June 02, 2020, 10:52:39 PM »
I didnt know this existed, but reassuringly we use somewhat similar methods

https://www.seanoe.org/data/00485/59709/
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

blumenkraft

  • Young ice
  • Posts: 3911
  • Fans of Hans Ø Club - circa 2018
    • View Profile
  • Liked: 1362
  • Likes Given: 2139
Re: Maximising asif skills using near real time Mosaic data
« Reply #175 on: Today at 11:24:18 AM »

Oops! Fixed

Haha, ok! That would have been amazing.  ;D
CEO of bitcoin & Antifa

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #176 on: Today at 02:24:58 PM »
I didnt know this existed, but reassuringly we use somewhat similar methods
https://www.seanoe.org/data/00485/59709/
Winter only. I think our winter results are ok though snow needs a bit more work.
Might have to be more manual for summer. If the buoys last that long.

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #177 on: Today at 03:57:24 PM »
I take it when they do leave the pack they will get scooped back up and put further north again? Iv been watching their relentless and almost unprecedented race toward the fram with bemusement, half of them will be out the ice within a month at this rate
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501

uniquorn

  • Nilas ice
  • Posts: 2127
    • View Profile
  • Liked: 1099
  • Likes Given: 195
Re: Maximising asif skills using near real time Mosaic data
« Reply #178 on: Today at 04:17:08 PM »
There doesn't seem to be an 'afterstory' for our courageous buoys. If any parts do get re-used they are anonymously renumbered. They should 'do a Musk' and go big on reusability. Whoi itp116 (not part of mosaic)already making its way down the greenland coast.
Hopefully they have a new batch ready for deployment further north.

SimonF92

  • Frazil ice
  • Posts: 204
    • View Profile
  • Liked: 85
  • Likes Given: 43
Re: Maximising asif skills using near real time Mosaic data
« Reply #179 on: Today at 04:46:57 PM »
There doesn't seem to be an 'afterstory' for our courageous buoys. If any parts do get re-used they are anonymously renumbered. They should 'do a Musk' and go big on reusability. Whoi itp116 (not part of mosaic)already making its way down the greenland coast.
Hopefully they have a new batch ready for deployment further north.

If all else fails, I will take one and put it in my living room. I think it would be a good conversation starter. Infact, at 5metres it might do better in the garden- my neighbours might think im conducting surveillance and stop secretly feeding my cat
Check out our home-grown ASIF MOSAiC website @ www.mosaic-ice.com:8501