Author Topic: Maximising asif skills using near real time data (Read 44959 times)

blumenkraft · « **Reply #100 on:** April 06, 2020, 02:17:39 PM »

https://github.com/ipython/xkcd-font

Just sayin'.

SimonF92 · « **Reply #101 on:** April 06, 2020, 03:41:31 PM »

Thats pretty awesome blumenkraft, I didnt know such things existed.

Speaking of, here is the python half of our work.

I havent annotated this version, but I will add an annotated one when i get the time

https://github.com/SimonF92/Arctic/blob/master/Final_MOSAiC_Mining.py

blumenkraft · « **Reply #102 on:** April 06, 2020, 04:10:40 PM »

Quote from: SimonF92 on April 06, 2020, 03:41:31 PM

I didnt know such things existed.

Me neither until today!

But now i feel everyone should know about it.

kassy · « **Reply #103 on:** April 06, 2020, 04:38:41 PM »

Some nice looking work! I enjoy watching this arcane magic at work.

PS: Who doesn´t hate bed kerning.

uniquorn · « **Reply #104 on:** April 06, 2020, 04:51:11 PM »

Ran the new script in spyder (I prefer it to the browser). It created a new Buoy_Data directory in the working directory and wrote all the files. Chart dates stop at mar30? ctr

SimonF92 · « **Reply #105 on:** April 07, 2020, 01:36:13 PM »

Quote from: uniquorn on April 06, 2020, 04:51:11 PM

Ran the new script in spyder (I prefer it to the browser). It created a new Buoy_Data directory in the working directory and wrote all the files. Chart dates stop at mar30? ctr

Thanks for checking that uniquorn, im glad it works in Spyder so thats good to know.

The plots go to todays date, its a problem with the X-ticks. I hadnt noticed, so thanks for pointing that out, ill try fix it

uniquorn · « **Reply #106 on:** April 12, 2020, 10:04:49 AM »

These charts are a very quick and easy check on ice thickening, which may be slowing on some buoys, and local temperature but until the time axis is fixed they are best posted here. If we start to see melt I'll run the thermistor temp charts to check our calculations. ctr

SimonF92 · « **Reply #107 on:** April 14, 2020, 12:55:20 PM »

Quote from: uniquorn on April 12, 2020, 10:04:49 AM

These charts are a very quick and easy check on ice thickening, which may be slowing on some buoys, and local temperature but until the time axis is fixed they are best posted here. If we start to see melt I'll run the thermistor temp charts to check our calculations. ctr

Its fixed

I was a really annoying bug to fix, turned out to be because there are duplicates as a result of keeping the TS data per date.

All good now.

Will update the code on github

uniquorn · « **Reply #108 on:** April 19, 2020, 06:57:20 PM »

Tinkered a bit with the charts to check the data, temperature appears to that at 07:00
T56
2020-04-13T01:00:14 -31.25
2020-04-13T07:00:14 -29.06
2020-04-13T13:00:14 -25.44
2020-04-13T19:00:15 -27.12
2020-04-14T01:00:14 -25.31
2020-04-14T07:00:14 -21.62
2020-04-14T13:00:14 -18.62
2020-04-14T19:00:14 -16.62
2020-04-15T01:00:14 -14.06
2020-04-15T07:00:14 -9.31
2020-04-15T13:00:14 -6.88
2020-04-15T19:00:14 -6.12
2020-04-16T01:00:14 -6.38
2020-04-16T07:00:14 -1.88
2020-04-16T13:00:14 -1.25
2020-04-16T19:00:14 -1.5
2020-04-17T01:00:14 -13.31
2020-04-17T07:00:14 -17.31
2020-04-17T13:00:14 -13.31
2020-04-17T19:00:14 -14.88
2020-04-18T01:00:14 -12.38
2020-04-18T07:00:14 -7.25
2020-04-18T13:00:13 -3.12
2020-04-18T19:00:14 -9.19
2020-04-19T01:00:14 -3.31

T63 may have failed. ctr

uniquorn · « **Reply #109 on:** April 22, 2020, 01:44:25 PM »

Quote from: SimonF92 on April 14, 2020, 12:55:20 PM

Will update the code on github

After that warm spell, perhaps the first signs of bottom melt on T72 or just a warming at the ice/ocean interface.

uniquorn · « **Reply #110 on:** April 22, 2020, 02:06:50 PM »

hmm, some event happened on apr13. Must be water over the surface, perhaps a very close lead opened up. Changed the temperature gradient completely.

added rough location of T72 on S1B, apr14. PS is visible as a bright dot, top right. cffr
2020-04-14T10:00:14,84.522186,11.996773

uniquorn · « **Reply #111 on:** April 27, 2020, 07:38:32 PM »

thickness estimate update

oren · « **Reply #112 on:** April 28, 2020, 06:07:28 PM »

So I've just read 111 posts in this amazing thread. Huge kudos to uniquorn and SimonF92 for analyzing and cracking the data of the mass balance buoys. I should have tracked the thread in real-time but my mind was elsewhere.
Before I turn to trying to analyze the data myself using the tools you have created, which surely will not be easy with my limited time and skills, I have one request right off the bat which I hope you can implement easily.
I consider the ice as some sort of capacitor, dampening the air temperature signal and integrating it on its way to the water interface. I can easily understand how the wild swings in the air signal are not immediately apparent when looking at the thickness growth curve. And I think there is an easy way of visualizing what goes on under the hood so to speak.
Add to the standardized thickness/temperature charts posted herein another temperature plot - that of the thermistor found 50cm below the upper ice surface, i.e. 50cm below the air or snow interface. Consider this the "core temp" of the ice. This variable is driven by the air temp above, and drives the thickening below, via a temp gradient through the ice. Later in the season this variable will rise above some threshold (is that -1.8^o?) and will no longer be able to drive thickening, but will enable bottom melt instead.
Ideally I would add another plot, of the thermistor that is 100cm below the upper ice surface, as long as it is at least 20cm above the bottom ice surface (ice-water interface) to avoid noise and meaningless data. This would be the "deep core temp".
Looking at these two plots together with the air temp plot and the thickening rate will hopefully enable a visually intuitive understanding of the processes governing bottom freezing and bottom melt.

Later I hope to calculate season FDDs over the buoy and compare to the theoretical plots of ice thickening.
I also want to examine the effect of thick/thin/zero snow on the air temp to core temp coupling. Theoretically the effect should be quite strong.
I hope to find the date(s) on which thickening stops and bottom melting begins. I hope a rule of thumb will be found to enable estimating this date even when no buoy is available
And of course, to try and find ways to compare the buoy data to PIOMAS, SMOS/SMAP and even to the dubious DMI thickness map.
Big hopes...

uniquorn · « **Reply #113 on:** April 28, 2020, 09:35:52 PM »

I think we still may have a problem with snow. Looking at a recent temperature gradient on T56 on apr11 using a 'free' spreadsheet (LibreOffice Calc) I can see a change in gradient at thermistors 24-30. I think that is ~12cm of snow but the Python generated chart above doesn't show it.
Assuming that ice starts at Therm30 we call therm(core)55 and therm(deepcore)80. Remember 2cm/therm

T56heat not giving a clear indication of snow between Therm24-30 on apr11. It agrees within 2therms at the ice bottom but is 10 therms different to the ice top??

uniquorn · « **Reply #114 on:** April 28, 2020, 09:45:50 PM »

Select the columns time, T1, T30, T55 and T80 while holding down the Ctrl button on the kbd.
Click the insert chart button and select lines only
Deep core is water until december so you might want to shorten the range.

oren · « **Reply #115 on:** April 28, 2020, 10:54:20 PM »

Thanks uniquorn, that's what I wanted. And I use Excel anyway so the learning curve should be manageable. But where do I download said spreadsheet for the various buoys? I probably missed the link or location while making my way down the thread.

uniquorn · « **Reply #116 on:** April 28, 2020, 11:36:23 PM »

uniquorn · « **Reply #117 on:** April 29, 2020, 01:24:30 PM »

T58 heat's match quite well but T56's don't. Will run some more later ctr

uniquorn · « **Reply #118 on:** April 29, 2020, 08:52:25 PM »

T62 matches quite well, T64 doesn't. ctr
Possible file naming error at mosaic/meereis?

uniquorn · « **Reply #119 on:** April 29, 2020, 09:22:22 PM »

T65 and T66 match quite well. T66 may have a faulty heater ctr

uniquorn · « **Reply #120 on:** April 29, 2020, 09:40:47 PM »

T68 and T70 match quite well

uniquorn · « **Reply #121 on:** April 29, 2020, 10:10:29 PM »

T69 and T72 still struggling on. Not sure if that's water on the surface again on T72 yesterday. ctr

oren · « **Reply #122 on:** April 30, 2020, 05:16:58 AM »

From the "stupid questions" corner:
So I've gone as far as installing Python for Windows and have downloaded Simon's file from github. When trying to run the file I get an error message "ModuleNotFoundError: No module named 'pandas'". It appears to be one of (probably) several missing libraries the script is trying to import. Any idea what I've done wrong, or what should I do to remedy the situation?

oren · « **Reply #123 on:** April 30, 2020, 02:01:25 PM »

From the RTFM answers corner, looking at Simon's code:
##Written in Anaconda Python v3.7 Jupyter

Downloading Anaconda Python, hoping for better results.

uniquorn · « **Reply #124 on:** April 30, 2020, 04:11:40 PM »

Probably anaconda will have pandas, if not, these lines were in a previous script

#uncomment and run this first line if you get an error about not having pandas
#! pip install pandas

# denotes a comment, not sure about the !
I'm using Spyder which detected the import and asked if I wanted to install it.

oren · « **Reply #125 on:** April 30, 2020, 05:10:34 PM »

Anaconda fixed the problem and ran the script successfully. Will attempt the next steps tonight using the walkthroughs upthread.

uniquorn · « **Reply #126 on:** May 02, 2020, 12:03:51 PM »

Recent location of all Tbuoys. We may have lost T72. ctffr

uniquorn · « **Reply #127 on:** May 02, 2020, 07:10:48 PM »

There is one newish Mbuoy deployed on feb16 that has data in the same format as the Tbuoys. However the HEAT file records the actual temperature rather than the difference so it's tricky (for me) to add it to the py file.

I did manage to add 4 of the new Tbuoys by adding the url's to the lists and editing the start date to apr15. click to run

uniquorn · « **Reply #128 on:** May 03, 2020, 01:37:22 PM »

Modifying the basic spreadsheet formula to improve calculations when air or snow temperatures are above -2.1C
From upthread

Quote

The kind people at libreoffice.org have implemented COUNTIF which allows us to count the number of cells in a row with a lower temperature than our ice:ocean interface. I'm going to set that at -2.1 and let the wordsmiths on this forum discuss in detail what it should be.

Cell formula will be =COUNTIF(D2:II2,"<-2.1")
We omit column IJ. I think it is some sort of end of line check.

Now subtract Tas=40
Cell formula will be =COUNTIF(D2:II2,"<-2.1")-40

multiply by 2 because the thermistors are 2cm apart.
Cell formula will be =(COUNTIF(D2:II2,"<-2.1")-40)*2

Label your estimate column, select the cell beneath it and click F2 on the keyboard.
Paste in the formula and click Enter on the kbd

The modified formula is

=COUNTIF(AN2:II2,"<-2.1")*2
where AN2 is the column for the thermistor nearest the snow:ice interface.

This prevents air or snow temperatures above -2.1C being included in the 'ocean' countif
I think SimonF92 covers this in the Py method by identifying these temperatures as 'not ice' using stdev
AN2 is currently valid for T76

uniquorn · « **Reply #129 on:** May 13, 2020, 07:10:42 PM »

Found some deployment info that may help verify ice and snow thickness calcs.
In particular initial ice thickness
T60=6.1m
T61=7.05m
Probably a bit late for checking against the older deployments

SimonF92 · « **Reply #130 on:** May 14, 2020, 06:00:20 PM »

uniquorn got me thinking........

Im pretty sure I could deploy our entire code in a webapp.

This would then be an interactive website that lets users look at individual buoys etc using the figures we developed.

https://www.streamlit.io/

It will be such a minimal app (memory and storage wise) that I wonder if there is a way to host it on googles webapp-engine for free. Could be pretty cool. Will look into it further

Also, some of their measurements are 6+metres
Is that believable?

SimonF92 · « **Reply #131 on:** May 14, 2020, 07:15:28 PM »

Sorry oren, it's my fault the code isnt readable, this is something I really should put more time into.

Its not an RTFM issue as I haven't been very helpful.

Normally a requirements.txt file should be included on Github, which is a one-liner with all the required installs

oren · « **Reply #132 on:** May 15, 2020, 04:47:13 AM »

Thanks Simon. I haven't managed to get around to continuing my project yet, I hope it materializes at some point as there is a great wealth of data here.

Regarding the initial deployment I wonder that it does not state the thermistor number that is at the ice-air interface or the snow-air-interface or whatever. It seems to be something that is crucial to record when deploying.
The 7m thickness is hardly believable especially with a 5m chain, but I haven't investigated the details.

Continuing on the issue of thermistor numbering at the interfaces - the way I understand it the python code is intended to discover this. My question is whether it has given the same results throughout the season. If so, would it be possible to post here the resulting thermistor number for each buoy at the ice-snow interface?
If I understand correctly, this would enable analyzing the csv files without the need to run the python code.

uniquorn · « **Reply #133 on:** May 15, 2020, 11:06:37 AM »

I think the 'fortress' ~1km distance from PS may be over 6m thick. I can't find the reference though.

Quote

On 28 September the first researchers from Polarstern set foot on the floe, which had long been a preferred candidate thanks to the promising analyses of the satellite data. On the radar images produced by the satellites, the dark, nearly oval floe stood out thanks to a large, bright region in its northern section. This clearly set it apart from all of the other potential floes, which were consistently dark in the radar images. In the meantime, the experts have dubbed this region ‘the fortress’: made up of highly compressed, several-meter-thick ice,

webapp is a good idea but probably won't run directly from the forum

SimonF92 · « **Reply #134 on:** May 15, 2020, 11:09:36 AM »

Oren i maintain all of the outputs from the code at

bit.ly/MOSAiC_ASIF_Mining

There you will find the thermistor numbers you are looking for along with everything else, in csv format

Tldr,
The thermistor numbers for the ice/snow (air) boundary are essentially stable over the season, at least since their deployment in October- there have been no top-melt temperatures yet

SimonF92 · « **Reply #135 on:** May 16, 2020, 01:23:05 PM »

Its going pretty well, here is the website- all of this runs on the website itself and not on my actual python environment.

Still about 50hours worth of interactivity to add, but im surprised at how well it seems to be going.

The best part is i can use a free amazon-EC2 to host the server. Im pretty sure those machines have 4Gb ram so it will be relatively slow but beggars cant be choosers

uniquorn · « **Reply #136 on:** May 16, 2020, 09:01:50 PM »

I'm finding this presentation easier to work with (figsize=(5,10)) . Its easier to identify bottom melt.
Note 2 upticks on T62, T64, T65 and T76.

The taller chart would look nice to the right of the 'select buoy' buttons
btw I expect you are aware you are showing T70 with T56 selected...

oren · « **Reply #137 on:** May 17, 2020, 12:06:31 PM »

Thanks a lot. Looking through the website, here are the thermistor numbers for the top of the ice, which are indeed constant through the winter:
T56: 38
T58: 31 (with a rare 32)
T62: 47
T63: 29
T64: 45
T65: 32
T66: 46
T68: 44
T70: 40
T72: 34 (went to 22 in the last day before failure)

I find these to be useful for anyone who wants to analyze the original csv files that contain the temps for each thermistor, so I'm putting them here.

SimonF92 · « **Reply #138 on:** May 17, 2020, 02:00:57 PM »

Update on the webapp and tasks based on what uniquorn and oren have said;

1) on the webapp provide a download feature for the analysed csv files
2) add an x-axis "zoom" function for the plots
3) provide a map of buoys based on their coordinates using Pydeck

If anyone has any other suggestions I will be happy to do it

PS its not too difficult to use streamlit, if anyone wants to have a play around with the webapp ill post the code to my github and provide a link

oren · « **Reply #139 on:** May 17, 2020, 02:54:39 PM »

Simon, thanks for your efforts, two questions:
* Is there another link? The one I used goes to a dropbox without the features shown in the screenshot.
* I notice uniquorn is showing data from other buoys such as T74. Can these get on the site too?

SimonF92 · « **Reply #140 on:** May 17, 2020, 03:30:55 PM »

Quote from: oren on May 17, 2020, 02:54:39 PM

Simon, thanks for your efforts, two questions:
* Is there another link? The one I used goes to a dropbox without the features shown in the screenshot.
* I notice uniquorn is showing data from other buoys such as T74. Can these get on the site too?

The screenshots I am sharing now are from a webapp I am currently developing. At the moment that webapp runs on a local host so you wont be able to visit it, once it is nearing the end of its development I will run it from an EC2 and make the link available. The dropbox link you have visited is just a place to store the data (for now).

uniquorn kindly sent me the urls for the other buoys, so it is a very simple case of just copy and pasting them into the code- this can be done any time new buoys come online

SimonF92 · « **Reply #141 on:** May 17, 2020, 03:48:09 PM »

-Added date-zooming and y-axis stretching for uniquorn.
-Also added caching for the computationally expensive processing part.

Im not doing all of this selflessly, I have an ulterior motive. I was just told that my lab-based PhD now needs to have a "computer chapter" for the time we have missed due to covid. Im going to build a webapp for analysing genetic data, as my chapter- it sure seems a lot more interesting than writing a review.

Anything I do here is really a learning process for me.

Therefore, if anyone has any ideas about what could be interesting to see, ill write it in.

oren · « **Reply #142 on:** May 18, 2020, 04:30:12 AM »

A question. I started playing with the data for buoy T66, looking at the various files:
dT120 file from Simon's dropbox
Temp file showing each thermistor
Heat030 file showing each thermistor
Heat120 file showing each thermistor
The PNG files shown on meereisportal

Using various eyeballing and charting methods and looking at Dec 1st, I can "see" that the air-snow interface is at thermistor 42, while the snow-ice interface is at 48 or 49.
This seems to fit with various posts upthread, for example post #70.
However the dT120 file and post #73 say both interfaces are at 46.
I'm trying to understand the difference in methodology that leads to this result, and what is the "correct" value to use. I know I should have asked this question a month ago, but...

SimonF92 · « **Reply #143 on:** May 18, 2020, 12:20:55 PM »

oren, thanks for the questions, its good to have them because I want people to understand and agree with what we are doing.

uniquorn and myself have been discussing by messages and exchanging bits of code for a while trying to get this figured out.

Here is th_ice written out in full.

Th_snow uses somewhat similar method, but youve actually given me an idea for Th_snow that im going to try now. Let me see if i can get the bug with T66 you pointed out sorted too.

//////

by the way, its actually possible to manually assign th_ice and th_snow on a per buoy basis, but that would take the fun away when top-melt starts

oren · « **Reply #144 on:** May 18, 2020, 04:20:52 PM »

Thanks again.
So T66 suffers from a specific bug? My bad luck. It hadn't occur to me to play with the others until I figured out where I was going wrong.

I would actually recommend to make a consensus "decision" using the automated algorithms that were developed, along with manual validation, to identify the TH_ice for each buoy, using the long set of wintertime data, and before top melt comes along to mess things up. The original Th_ice of each buoy is the single most important piece of the analysis IMHO because all the rest of the conclusions hinge on its correct identification. And in physical reality it is an installation-dependent constant, so its final identification should be feasible.
I will (hopefully) go through the data of the various buoys, and compare to posts #73, #70 and my own post above that summarizes the "dropbox" analysis, and will try to come up with manual validation of the results for Th_ice.

SimonF92 · « **Reply #145 on:** May 18, 2020, 05:18:25 PM »

I completely agree. Infact, im going to provide a link to the webapp now because itll be by far the best way for a few people to get a look and check these things. I would ask that not too many people visit this server until i get the kinks of hosting it ironed out.

Its currently running on 1cpu with 1gig of RAM, on an AWS EC2, completely detached from my own machine in every way.

I will be taking it down 2-hours from now to check various things and also change the domain name to something pretty

http://3.19.26.242:8501

oren · « **Reply #146 on:** May 18, 2020, 06:04:34 PM »

I recommend one small change that will enhance usability of the table.
Either shorten the titles, for example ts_latitude (deg) can be lat(deg) , or allow word wrap of the titles over several rows. The goal should be that the column width will fit the width of the data in the column. This will result in much narrower columns such that lots more data can be seen at a glance without scrolling - including my favorite Th_ice.

SimonF92 · « **Reply #147 on:** May 18, 2020, 06:35:22 PM »

Quote from: oren on May 18, 2020, 06:04:34 PM

I recommend one small change that will enhance usability of the table.
Either shorten the titles, for example ts_latitude (deg) can be lat(deg) , or allow word wrap of the titles over several rows. The goal should be that the column width will fit the width of the data in the column. This will result in much narrower columns such that lots more data can be seen at a glance without scrolling - including my favorite Th_ice.

Added toggle for "key metrics"
(to the offline dev version)

uniquorn · « **Reply #148 on:** May 18, 2020, 09:21:01 PM »

Quote from: uniquorn on April 29, 2020, 09:22:22 PM

T65 and T66 match quite well. T66 may have a faulty heater ctr

noted upthread

webapp is great, location not running yet I assume.... or suddenly it is

Probably should give latest date rather than 'current coordinates'. Latest coordinates would do.

One other thing. Tbuoys record temperatures 4 times/day. Not sure which is the best to present on the chart. An average? Currently it is the 0700 temp. T58 shown

uniquorn · « **Reply #149 on:** May 19, 2020, 10:45:56 AM »

T61 location, reportedly 7.05m thick, for any floe spotters out there.
map rotated 45deg

News:

Author Topic: Maximising asif skills using near real time data (Read 44959 times)

blumenkraft

blumenkraft