Arctic Sea Ice : Forum

AGW in general => Developers Corner => Topic started by: uniquorn on March 03, 2020, 11:06:18 PM

Title: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 03, 2020, 11:06:18 PM
This is probably a once in a decade opportunity to use the mosaic project to gather almost unprecedented near real time arctic data. Perhaps you are content to sit back and watch a model. Maybe a view from a satellite. Maybe just wait for someone else to do it. But why? There are over 70 active buoys in the arctic today(Mar3). Why aren't we analysing them?? If we are. Why aren't we sharing??
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 03, 2020, 11:13:03 PM
Step1. This thread goes probably nowhere unless someone posts a Polarstern (the biggest buoy) temperature chart. Simple spreadsheet skills and probably some data preparation required.
data here (https://www.sailwx.info/shiptrack/shipposition.phtml?call=DBLK), scroll to the bottom of the page and click on track history
Please don't be shy :) 5 charts is better than 0
If converting text to csv is too tricky. Try T62 (https://data.meereisportal.de/download/buoys/0000T62_300234068706290_TEMP_proc.csv) from meereisportal.de using T1 as the surface temperature.
Open source spreadsheet (https://www.libreoffice.org/download/download/)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: psymmo7 on March 04, 2020, 12:37:03 AM
I've done a "dump all track history" which dumps everything back to 2016 and am in the process of extracting MOSAIC relevant data and converting it to a spread sheet. Date and time aren't delimited so needs a bit of hands on work to get it fully analysable. Will probably take me a couple of days. Will post again when done.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 12:47:26 AM
thank you. From oct2019 will correlate with mosaic.

Ideally one asif member could monitor one each of the 16 Tbuoys (https://data.meereisportal.de/gallery/index_new.php?active-tab1=method&buoytype=TB&region=all&buoystate=active&expedition=MOSAiC&submit3=display&lang=en_US&active-tab2=buoy)  which measure thickness

and melt not being funny but you know how asif is massively more interested during the melt season
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 11:38:22 AM
Quote
Date and time aren't delimited
Possible workaround
Copy and paste into notepad (or text editor with no formatting)
Edit > Replace "(double quotes) with blank space to delete them
Save as test.csv
Open with spreadsheet (libreoffice calc, excel....) with space as separator

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 02:09:43 PM
So you downloaded the 'free' open source LibreOffice and installed it. Downloaded the T62 file. Opened it with calc, selecting comma as the separator and you got this.
A datetime, lat, lon and 241 thermistor temperatures.

Take a moment to locate the 'Insert Chart' button, top right.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 02:17:26 PM
Select row1 by clicking on the row number on the left
Scroll down to the bottom row with data
Hold down the control key on the keyboard and select the bottom row number (510 today)
Click the chart button
Enjoy looking at this morning's data.
Play with the chart settings

Note: This quick chart will include the lat/lon data. Exlude those cells as you gain experience 
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 03:14:38 PM
Select columns A and D for a near surface temperature chart
Play with the chart settings.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 04, 2020, 03:36:28 PM
Basic stuff, but what we'd really like to know is how thick the ice/snow is.
What we need is a formula to calculate the number of thermistors between the gradient changes.

TBuoy download list
https://data.meereisportal.de/download/buoys/2019T56_300234065176750_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T57_300234065177750_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T58_300234065171790_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T59_300234065170760_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T61_300234065768480_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T62_300234068706290_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T62_300234068706290_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T63_300234068709320_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T64_300234068701300_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T65_300234068705730_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T66_300234068706330_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T67_300234068704730_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T68_300234068708330_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T69_300234068700320_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T70_300234068705280_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T72_300234068700290_TEMP_proc.csv
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 05, 2020, 01:26:49 PM
Reposting these animated Tbuoy charts from the Mosaic thread we can approximately identify which  thermistor is closest to air/snow interface. During 3mths the air/snow thermistor number (Tas) doesn't change much*. So an approximation of ice/snow thickness can be made by counting the number of thermistors above the ice/ocean temperature and subtracting Tas

We normally take ocean temperature under ice to be -1.8C but looking at a random section of T62 data at the ice/ocean interface it is clearly more complicated.



*except for T69 which has experienced some kind of trauma.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 05, 2020, 01:52:19 PM
Latest surface temps on T62 are nice and cold so it should be easy to find the air/snow thermistor.
It's probably T39 but let's say T40 to be sure. Checking further up the data it looks like we may have 4cm more snow since November. Hopefully some bright enthusiast will deal with that later.
So Tas=40
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 05, 2020, 02:17:24 PM
The kind people at libreoffice.org have implemented COUNTIF which allows us to count the number of cells in a row with a lower temperature than our ice/ocean interface. I'm going to set that at -2.1 and let the wordsmiths on this forum discuss in detail what it should be.

Cell formula will be   =COUNTIF(D2:II2,"<-2.1")
We omit column IJ. I think it is some sort of end of line check.

Now subtract Tas=40
Cell formula will be   =COUNTIF(D2:II2,"<-2.1")-40

multiply by 2 because the thermistors are 2cm apart.
Cell formula will be   =(COUNTIF(D2:II2,"<-2.1")-40)*2

Label your estimate column, select the cell beneath it and click F2 on the keyboard.
Paste in the formula and click Enter on the kbd

edit: I found some -2.06 ocean temperatures on thermistors T203 and T207 (in fact, all over the place) so the ice/ocean interface temperature has been dropped to -2.1C   Interesting to see if they turn up on other Tbuoys
Tip: Select cell B2 > View > Freeze rows and columns
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 05, 2020, 02:42:14 PM
Now we need to populate the whole column.
Select your new formula cell.
Click on the small black square at the bottom right of the cell and drag it down.
When you release the mouse button the cells will be populated with the correct formula.
Ensure that all the cells to the bottom row of the data are populated

Now select columnA
Press the control key on the keyboard and select your new column (IL in my case)
Click the 'Insert Chart' button as before.
Enjoy your SIT estimate
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 05, 2020, 02:47:36 PM
Daily Tbuoy location chart is here (https://data.meereisportal.de/maps/buoys/X/exped_map_MOSAiC_T.png)

Last note: The gradient change for snow on T62 is pretty clearly at around thermistor50 at the moment so we are looking at ~20cm of snow.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 12, 2020, 05:41:38 PM
I am a big fan of this idea and I have pretty deep Python knowledge so when it comes to parsing data or automating this in some way I am happy to help.

I have to ask though, when the temperature difference between ice and air normalises to 0 in the coming months, do you think it is possible to account for this in your analysis? Would it be better to choose a thermistor where the temperature no longer deviates between hour-to-hour recordings as the point at which there is "ice". Im assuming ambient air temp will be much more variable than the temperature within the ice, over short time periods.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 12, 2020, 07:37:57 PM
Quote
choose a thermistor where the temperature no longer deviates between hour-to-hour
I'm not sure there is one. The method above may be adequate for estimating the nearest ice/water thermistor for thickening and bottom melt until surface melt begins. Perhaps once the surface snow is gone the nearest air/ice thermistor will become clearer and easier to locate mathematically with temperature difference.
If you know how to calculate an average gradient change over several rows that may help.

I don't know what the surface numbers will look like as air temperatures rise but the T56 data from nov11-22 may give us an idea
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 16, 2020, 10:42:59 AM
Quote
choose a thermistor where the temperature no longer deviates between hour-to-hour
I'm not sure there is one. The method above may be adequate for estimating the nearest ice/water thermistor for thickening and bottom melt until surface melt begins. Perhaps once the surface snow is gone the nearest air/ice thermistor will become clearer and easier to locate mathematically with temperature difference.
If you know how to calculate an average gradient change over several rows that may help.

I don't know what the surface numbers will look like as air temperatures rise but the T56 data from nov11-22 may give us an idea

Thanks for the advice Uniquorn, I will look into this.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 16, 2020, 02:01:21 PM
I plotted the standard-deviation of temperature over a 10-hour period for (most recent data) for T56.

In the 2nd  plot I plotted the change in stdev between adjacent thermistors

At the moment I think its pretty obvious from both plots the air(snow):ice boundary is around T25. In theory this way should overcome limitations which might begin to exist when there is less of a temperature difference between the ice and the air.

If you think this makes sense I will begin to work on a simple algorithm.

Hopefully we are not talking to cross-purposes at the moment!
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 16, 2020, 02:15:46 PM
Algorithm would be written from the centre of that U-curve to the temp flattening (yellow)

Might be better to have an estimation between the margins of that U
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 16, 2020, 02:16:38 PM
Ignore the labels  >:(

X and Z are swapped
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 16, 2020, 04:46:45 PM
I plotted the standard-deviation of temperature over a 10-hour period for (most recent data) for T56.
<>
Hopefully we are not talking to cross-purposes at the moment!
That looks a lot more robust than my method. Do you mean 10 rows? 1 every 6hrs.
I think we are aiming at the same thing, only the presentation will be different. I plan to use the thermistor numbers to show thickness on the drift animation
drift update, click to run. (might have less time than I thought)
removed-will update later
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 16, 2020, 05:24:57 PM
Beautiful animation.

Yes, 10 rows, my apologies.

Are you doing this in R?

Ill try write a function for thickness in R's syntax if so.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 16, 2020, 06:28:35 PM
Yes, R. I had to 'learn' it to hack macid's original code. Taking on python as well is a bit much.
I had a look at T64, the southernmost Tbuoy to check the ice is still thickening (at around -2°C)
Therm139 to Therm141, so ~4cm (at the bottom) over 16days.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 16, 2020, 08:44:18 PM
Algorithm would be written from the centre of that U-curve to the temp flattening (yellow)
Might be better to have an estimation between the margins of that U
I suspect the U-curve is the snow layer so we should see that disappear narrow? as it melts. This might be of interest to those who like to discuss snow depth. The two margins would then be air/snow, snow/ice (ultimately becoming air/ice) though I'd need to see a time series to see how the std deviation curve moves to comment more. If identifying those margins is too problematic I'll be happy with an ice/snow thickness.
stdev and delta charts are different sizes, here scaled in an attempt to match them

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 17, 2020, 10:59:12 AM
Here they are matched, im working on a time series at the moment
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 17, 2020, 02:31:15 PM
Here they are matched, im working on a time series at the moment
Great. The u-curve on the delta looks very much like the snow layer and the change at ~120 on the std deviation looks like the bottom of the ice. Hopefully confirmed by a time series. How tricky is the algorithm?

Meanwhile, a quick update on thickening.
removed
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 17, 2020, 03:22:58 PM
Here is a time series of November, individual figures are made over 60-hour sampling periods.

A few of these I am not happy with, a few of them I am happy with.

If you think it best to just keep this simply about temperatures and not stdevs then I can understand that.

Let me know!


Ill put in an mp4 for anyone who likes that kind of thing, it might help to visualise what I can see on my end.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on March 17, 2020, 03:32:48 PM
Thanks a lot, Simon. Great work!

Allow me a critique though. Perhaps it's only me, but i can't see where these points are in the 3D space. One solution to fix this maybe would be to add shadows to them (if this is even possible).
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 17, 2020, 03:35:56 PM
blumenkraft you beat me to it while i was changing the size of this animation to help people with low bandwidth  ;D

I agree its quite unclear unless you see it rendered in 3d, I will work on a way to animation the timeseries in 3d
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on March 17, 2020, 03:36:59 PM
OMG, very cool!  :D
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 17, 2020, 05:00:11 PM
This is a current estimation of t56's data, based on the code I just wrote.

I am going to fully annotate the code and put it up here.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 18, 2020, 05:05:06 PM
testing temperature chart with SimonF92 generated provisional thickness estimate
temporary post as it is large
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 18, 2020, 06:16:47 PM
Im posting the main bulk working here.

It would be great to spot any errors in this at the moment and not later on. If anyone disagrees with  any of the lines I will be happy to change it.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 18, 2020, 06:58:42 PM
Thanks.
Is this an ice or an ice+snow thickness estimate? I'm still working my way through the coding.
Do you get the same thickness chart as above with the wild swings at the end? If so, what could cause them? I ran the temperature charts to see if they could explain it. Perhaps it is the temperature gradient flattening out. If so that may be a problem as temperatures rise.
edit:latest fomo (https://follow.mosaic-expedition.org/) may help identifying snow layer
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 18, 2020, 09:55:44 PM
T56 method comparison
eyeballing the numbers Therm-snow/ice=30 for the spreadsheet method
Spreadsheet method is less complicated, std dev+ is more mathematically justifiable
Is the difference down to snow depth?

Quote
Nag to the 17: This is probably a once in a decade opportunity to use the mosaic project to gather almost unprecedented near real time arctic data. Perhaps you are content to sit back and watch a model. Maybe a view from a satellite. Maybe just wait for someone else to do it. But why? There are over 70 active buoys in the arctic today(Mar18). Why aren't we analysing them?? If we are. Why aren't we sharing??
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 19, 2020, 10:26:56 AM
 Spreadsheet method is definitely better currently in my opinion but I am concerned that when temps rise it will be more difficult to pick out boundary thermistiors based on temperature gradients alone.

Thanks for making that plot, its pretty clear to me that the python method needs work.

Could the python method be underestimating when there is no snow and overestimating when there is a lot?

//edit//

Ill try to add an estimate of snow depth based on the U width, will send you the code :)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 19, 2020, 03:23:16 PM
Comparison of the new code (thickness_and_snow_as_timeseries.py) with the basic method. Pretty close agreement on ice thickness. We have no way of verifying snow depth but looking at other buoys may show us if we are on the right track

added yesterday's S1 (http://bslmagb.nerc-bas.ac.uk/iwsviewer/?image=DataPolarview/111_S1jpeg2000_202003/S1B_EW_GRDM_1SDH_20200318T083945_FECE_N_1.8bit.jp2) of the area
nearest coords, roughly middle of the crop
2020-03-18T07:00:12,86.704520,12.369056
2020-03-18T09:00:13,86.696373,12.381268
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 19, 2020, 04:17:45 PM
Snow depth for T56 to T63,

.....excluding the buoys weve already figured arent working properly based on their 0 temps throughout the column.

I think you were spot on when you said that U was snow depth


FYI this is from feb 21st to present
Green is T60
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 19, 2020, 05:43:13 PM
Incase anyone is wondering what we are talking about with this model.

I have to be conservative with the "U" shape because currently the code iterates to provide a reading every 6-hours, 500 iterations per buoy, 14 buoys: over 10,000 measurements inclusive of both ice and snow means lots of room for errors and over-estimations.

Sorry if the dimensions of the mp4 are a bit poor, i have scaled it down again to take up less size.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 20, 2020, 01:37:13 PM
Here are the buoys charted from SimonF92 data. It's unlikely the snow layer is so variable. Probably due to the method and changing temperature. Highly likely that it is adequate for this thread, especially as the snow will be gone soon enough.
T69 may still have some value so I left it in
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 20, 2020, 05:49:34 PM
Here are all the buoys. Its clear at least one is broken and one died. Its possible the code bugs at certain points too, but i think we have made pretty decent progress this week.

I have applied very gentle smoothing to the data,.

This data is from mid-november to present, measurements are made every 6 hours.

The buoys are on the move, so it might be reasonable to expect some variability. I am personally quite curious about the spread in thickness between the buoys
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on March 20, 2020, 06:13:57 PM
So, those downward spikes are measurement errors, no?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 20, 2020, 07:12:21 PM
 :-X
So, those downward spikes are measurement errors, no?

T57, t59 and t61 were removed as they were reporting ~0 degrees values throughout the thermistor column.

 Though i havent personally checked, im pretty sure the values for t69 would be similar at those wild swings and is most likely erroneous data from hardware issues. Just left it in for discussion purposes- i havent personally checked.

Some of the other swings are probably bad code!
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 20, 2020, 08:11:32 PM
T69 had a trauma of some kind as noted upthread. The other drops don't show up on reply#9 (https://forum.arctic-sea-ice.net/index.php/topic,3013.msg252562.html#msg252562) animation showing bottom thickening so are probably related to the method used to detect the nearest snow/ice thermistor.
I think the 'warm snap' around feb20 may be the cause.
10 rows perhaps too short - it might be easy to edit such well documented code.... :)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 20, 2020, 09:24:22 PM
T69 had a trauma of some kind as noted upthread. The other drops don't show up on reply#9 (https://forum.arctic-sea-ice.net/index.php/topic,3013.msg252562.html#msg252562) animation showing bottom thickening so are probably related to the method used to detect the nearest snow/ice thermistor.
I think the 'warm snap' around feb20 may be the cause.
10 rows perhaps too short - it might be easy to edit such well documented code.... :)

Definitely what caused it.

If we had a large training dataset I could use machine learning to estimate snow-depth, but as-is it is, this whole "U-shape" concept is bugging the code- I take responsibility for chasing that idea

Im offline for the weekend but on Monday I will write another version of the algorithm and scale back the complexity- just focus on ice.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 20, 2020, 09:38:59 PM
I think the U shape is good. If you have time I'd like to see how it moves with time. I might be able to chart it in R if not.
(https://forum.arctic-sea-ice.net/index.php?action=dlattach;topic=3013.0;attach=145079;image)
I experimented with changing the number of rows from 10
5 rows was worse
20 rows removed the large drops but the algo is struggling to detect the ice/snow boundary so ice goes up when snow goes down.
at 50 rows other problems crept in.

So far it is good for detecting snow+ice thickness
I'll play with this line in the meantime
#select the region with large stdev snow
        dfU=dfU.where(dfU.stdev_delta_mean<-0.25)

Quote
If we had a large training dataset
rofl. This is probably the biggest it's ever gonna get.

adding the Tice/water, Tsnow/ice numbers etc to the csv might help debugging, maybe I can do that?!?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 22, 2020, 10:27:41 AM
Overnight thoughts: As temperatures rise to close to zero I think we will be looking at a very flat curve across the thermistor chain. More experienced minds than ours will have grappled with the problem of identifying the 'ice-air-water interfaces' and offered this heat method as a solution.
Perhaps stdev and delta will be more reliable when run on these files
https://journals.ametsoc.org/doi/pdf/10.1175/JTECH-D-13-00058.1       2013
Quote
5. Heated operation mode
Theory of operation
The novel feature of this new IMB buoy is the development of chains that can be operated in what can be described as either a ‘‘hot-wire anemometer’’ mode or a needle-probe thermal conductivity mode. Hot-wire anemometry is a standard technique in experimental fluid dynamics and has been widely described (e.g., Perry 1982;La Barbara and Vogel 1976). Essentially a temperature-sensing element in a moving fluid is heated to above ambient temperature, and the amount of heat required to maintain a constant sensor temperature is measured. The amount of heat required depends on both the thermal characteristics of the surrounding fluid and its flow velocity. In principle, the output of the Maxim DS28EA00sensor device can be refined for the estimation of both these quantities, thus allowing determination of the position of the ice–air–water interfaces and quantification of flow speeds. The sensitivity of the present design is not yet capable of the quantification of flow speeds


https://data.meereisportal.de/download/buoys/2019T56_300234065176750_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T57_300234065177750_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T58_300234065171790_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T59_300234065170760_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2020T60_300234066299840_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2020T61_300234065768480_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T62_300234068706290_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T63_300234068709320_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T64_300234068701300_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T65_300234068705730_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T66_300234068706330_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T67_300234068704730_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T68_300234068708330_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T69_300234068700320_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T70_300234068705280_HEAT120_proc.csv
https://data.meereisportal.de/download/buoys/2019T72_300234068700290_HEAT120_proc.csv

There is also a  HEAT030 file for each buoy (edit the url). Maybe one is good for ice-water and one for ice-snow etc.
We've just had 4T of manure delivered so no coding today. Running an animation instead. Tbuoy locations, mar22, click twice for full res
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 23, 2020, 12:57:49 PM
Very interesting, ill write a new version based on HEAT and we can cross reference the data with the old Thermistor data.

I suspect once again that snow might throw a spanner in the works.

Will post full working and send you the code to have a tinker with
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 23, 2020, 01:04:20 PM
The cool thing is, the batch wrapper and parser will not need any tweaking at all- just the algorithm. So any new ideas can be implemented efficiently
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 23, 2020, 01:33:24 PM
I hoped that would be the case. Still shovelling here.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 23, 2020, 02:35:19 PM
Im writing it based on their dT30 data from here, should be very easy to do and resistant to seasonal temperature changes:

https://data.meereisportal.de/download/buoys/2019T56_lastprofile.png

As you said, its why they included these parameters in the first place

//edit//
scratch that, ill do it on the dT120 instead, even more obvious
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 23, 2020, 04:07:52 PM
If it's not too much trouble, please include T1 in the final csv. It will make it easier to run temp vs thickness analysis
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 23, 2020, 06:50:47 PM
New code coming to you. I feel like this one might underestimate, but it is based on dT120 which has a very distinct and clear ice boundary.

The question is, IS it underestimating or it is that thickness?

Their data here,

https://data.meereisportal.de/gallery/index_new.php?lang=en_US&active-tab1=method&active-tab2=buoy&singlemap&buoyname=2019T56

//Edit//

 having read about it I have to say that I think its the real boundary
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 24, 2020, 02:55:25 PM
So we are getting closer, the ice:snow boundary is now very clear.

Can you see the bump labelled green? Thats something were discussing. That could either be a sharp melting event, or a bug.

As the air temperatures do not rise to facilitate this melt, is probably a bug.

The best way to tell is to slice those dates off the dataframe. If those bumps move with the slice, to a different date, then it is definitely a bug.

If the bumps disappear, then it is not a bug.


SUMMARY: ice thickness is blue-red lines
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 24, 2020, 03:03:03 PM
It wasnt a bug.

Any thoughts? Influx of warm water facilitating bottom melt?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 24, 2020, 08:21:03 PM
Looking only at T64 where the chart shows 3 clear drops in estimated thickness. The temperature ani doesn't show any bottom melt events. There are, however, some higher air temperatures that flatten the curve that may occur at roughly the same time as the 3drops.
Not exactly a bug but perhaps an error in the method.
Some thermistors are recording persistently higher temperatures. Perhaps there was a problem during calibration
Was using Spyder but jupyter charts are good


Still need to see a time series of the stdev(10rows) charts to see it they are messing about during temperature rises
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 24, 2020, 09:21:06 PM
Its interesting you picked up on t64, i was also looking at it closely.

I used their proprietary plots to check ours. It seems there are major fluxes of heat (disruption) into the column at the same time we detect "melting"- its possible this is some kind of ocean current?

I dont think this can be overcome at our end at it is data- based.

https://data.meereisportal.de/gallery/index_new.php?lang=en_US&active-tab1=method&active-tab2=buoy&singlemap&buoyname=2019T64

Sorry for the low resolution
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 24, 2020, 10:57:49 PM
Those fluxes are in the atmosphere. I overlayed T64 thermistor1 onto daily temperature. Maybe our algo should detect air/snow first. There seems to be a defined line there, then snow/ice, then ice/water. Possibly only using mean heat temp change.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 25, 2020, 12:11:33 AM
 Agreed- problems with the code and the temp anomalies during these "warm" events. Will figure something out.

Lovely overlay by the way, when this is tied up id love to get a look at some of your code/methods.

Its also on my agenda to do stdev timeseries, though that adds a new dimension to the data im a bit naive about (think of a csv file where each cell has depth) will try to get it working.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 25, 2020, 11:03:58 AM
I was thinking of turning the 126 t64heat stdev images into an animation, just to get a feel for the movement.
Overlay was done in gimp. Drag both images in, measure the width of the charts in pixels, scale then set white to transparent.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 25, 2020, 02:08:55 PM
I will have a bash, I think your skillset is better for animations than mine for sure, id love to see how you use gganimate in R.

I am working on this algo at the moment. Its going to have to be very well written, the changes are subtle but the boundary is there.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 25, 2020, 06:53:38 PM
Ok so its definitely getting a bit more complicated but hopefully what im doing is logical.

The idea above would not work as its just too difficult to resolve. Instead, I am mapping the thermistors

RED: at the -5degree isotherm. (not perfectly mapped yet, currently is between -4 and -6 degrees)

GREEN : our perceived ice:ocean interface

BLUE :  a savitsky-golay fit on the red -5degree isotherm. As warm transient air temps penetrate the ice, they raise our RED isotherm (which is a data quirk and not real, the ice does not have bottom melt during this), which then disrupts our GREEN line. I will correct the GREEN against the delta (change) between RED and BLUE.

Will see how it looks. Im not going to correct until im sure iv got the best savgol method done
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 25, 2020, 09:28:47 PM
got these on the way to making the animation. Using some of your py code with this below               
        #try to save plot
        plt.plot(df1.loc['mean_t'])
        plt.ylabel('mean')
        plt.savefig("meantst5.png", dpi=400)
        #plt.plot(df1.loc['stdev'])
        #plt.ylabel('std')
        #plt.savefig("stdtst5.png", dpi=400)


At least 2 stdev stand out in what I think is the snow layer. I'm not sure what mean_t is yet but it looks the most useful.
click for full res

Check here (https://github.com/macidR/MOSAiC-Buoys-2019/blob/master/mosaic_buoys.R) for original macidR gganimate code and these nice 3d buoy presentations. https://rpubs.com/macid
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 25, 2020, 11:47:11 PM
got these on the way to making the animation. Using some of your py code with this below               
        #try to save plot
        plt.plot(df1.loc['mean_t'])
        plt.ylabel('mean')
        plt.savefig("meantst5.png", dpi=400)
        #plt.plot(df1.loc['stdev'])
        #plt.ylabel('std')
        #plt.savefig("stdtst5.png", dpi=400)


At least 2 stdev stand out in what I think is the snow layer. I'm not sure what mean_t is yet but it looks the most useful.
click for full res

Check here (https://github.com/macidR/MOSAiC-Buoys-2019/blob/master/mosaic_buoys.R) for original macidR gganimate code and these nice 3d buoy presentations. https://rpubs.com/macid

Nice plots and thanks for the link. Didnt know you could use the loc function in plt, thats cool. The algos snowy and thicky (partially) are based on the stdev plot.

Mean_t is the mean temperature at that measurement point (each per 10-single measurements) for each thermistor. Id be careful of using mean_t as we expect that to become very similar to the ice temperature in the coming weeks.

Still working on this smoothing, its driving me mad
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 26, 2020, 11:16:17 AM
I should have included
import matplotlib.pyplot as plt

Quote
Mean_t is the mean temperature at that measurement point (each per 10-single measurements) for each thermistor.
That's what I assumed, but why are the values from 0-5?
mean difference?

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 26, 2020, 03:47:41 PM
The values are between 0 and 5 as HEAT is not real temp, its some fancy thing they do to measure the heat capacity of that thermistor.

uniquorn what do you think of this?

We have an expression in our lab "cant see the forest for the trees". I have become so bogged down in trying to iron out the quirks in the data recently I lost sight of the big picture. Here is the end result? How would you feel about using these in your thickness maps?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 26, 2020, 03:49:40 PM
I have actually been trying to get it working in python but it looks terrible compared to what you had
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 26, 2020, 04:24:05 PM
I get it. It's the mean temp rise over 120s. In which case it should still be valid as temps rise.

That's not so bad but could be misleading without dates. edit: many lat/lon's are bad on the temp_proc files, it's better to use the _ts file for location.
If those thicknesses compare closely enough with the basic method I'd be happy to use them though as mentioned previously I'm not too keen on smoothing.

added heat file ani for reference
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 27, 2020, 03:32:07 PM
I get it. It's the mean temp rise over 120s. In which case it should still be valid as temps rise.

That's not so bad but could be misleading without dates. edit: many lat/lon's are bad on the temp_proc files, it's better to use the _ts file for location.
If those thicknesses compare closely enough with the basic method I'd be happy to use them though as mentioned previously I'm not too keen on smoothing.

added heat file ani for reference

I completely agree, I am not a fan of smoothing under any circumstances.

In that case ill make a "basic method" script for use now and keep tweaking away at this over the coming weeks in the background.




///////

this is the basic method?

"""
Cell formula will be   =COUNTIF(D2:II2,"<-2.1")
We omit column IJ. I think it is some sort of end of line check.

Now subtract Tas=40
Cell formula will be   =COUNTIF(D2:II2,"<-2.1")-40

multiply by 2 because the thermistors are 2cm apart.
Cell formula will be   =(COUNTIF(D2:II2,"<-2.1")-40)*2

"""
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 28, 2020, 11:36:51 PM
Yes, except 40 only applies to T62. Need to identify the Therm number for the others.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 29, 2020, 01:06:04 PM
Estimated thermistor numbers based on last entry in heat120 files (chart attached, cffr)
buoy   Th(snow)   Th(ice)
T56   24   35
T58   18   34
T62   40   51
T63   18   32
T64   24   47
T65   12   35
T66   42   49
T68   34   46
T70   32   42
T72   28   36

T69   25   35

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 29, 2020, 03:41:21 PM
Rough charts based on the data above. Some different curves.
Note that the heat120 method appears to shift the air-snow and snow-ice thermistor no.s upwards, possibly due to heat rising. I haven't corrected for this. Even so, that doesn't explain 0 or negative starting position unless data was sent before (or during) buoy installation. hey ho, it's an estimate.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 29, 2020, 05:11:16 PM
Rough charts based on the data above. Some different curves.
Note that the heat120 method appears to shift the air-snow and snow-ice thermistor no.s upwards, possibly due to heat rising. I haven't corrected for this. Even so, that doesn't explain 0 or negative starting position unless data was sent before (or during) buoy installation. hey ho, it's an estimate.

Im actually working on a "basic method" script at the moment.

Within that script is the algo for detecting Th(snow) and Th(ice) thermistors .They look similar to your estimates, though not identical, ill send you it within 30mins.

New points in the code;

-basic method implemented
-lat and lon now taken from TS buoys, corrected for sampling period (to 1 per day)

Im also working on an additional program that lets you manually enter  Th(snow) and Th(ice) thermistors, when you wish to plot. In some ways I think the human eye might actually be better at this than an algorithm


//////

FYI all the code is pretty horrible at the moment, im reading a book called

 "Clean Code, A Handbook of Agile Software Craftmanship"

Once were happy, ill be tightening everything up very strictly
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 29, 2020, 05:31:48 PM
Iv added some new outputs to the console for you also;

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 29, 2020, 05:43:51 PM
Thinking about those rough charts, it would be easier to compare them if they all started on the same date. I think the shallow gradient was during the warm spell in October, some of the Tbuoy's data don't start till late oct/nov. Of course, if the ice is thin, it thickens quicker.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 29, 2020, 05:48:19 PM
Thinking about those rough charts, it would be easier to compare them if they all started on the same date. I think the shallow gradient was during the warm spell in October, some of the Tbuoy's data don't start till late oct/nov. Of course, if the ice is thin, it thickens quicker.

Good point? I can slice it at a date if you think that would work?

December 1st?


//////

scratch that, you can change it to any date you wish within the code
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on March 29, 2020, 07:28:10 PM
ok. I added thickness*100 to the charts and got this
and I think I'm mixing Th numbers with thickness
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 29, 2020, 08:11:02 PM
ok. I added thickness*100 to the charts and got this
and I think I'm mixing Th numbers with thickness

I dont think you are, the easiest way to change to thickness from m to cm is to change:

 #ice thickness is therefore the number of these thermistors
        thickness=(len(therms)*2)/100

at line 254 on github to:
 
 #ice thickness is therefore the number of these thermistors
        thickness=(len(therms)*2)

and nothing else
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on March 29, 2020, 08:16:58 PM
My output on the action above
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 01, 2020, 03:08:22 PM
Comparison of ice thermistors from the two methods.
Buoy   Th(ice)   Th(ice)SF
T56   35   38
T58   34   31
T62   51   47
T63   32   29
T64   47   45
T65   35   32
T66   49   46
T68   46   44
T70   42   40
T72   36   33

I'll run an ani with the SF ice therms.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 01, 2020, 04:15:47 PM
Here is a very quick way of checking my thermistor method, just stick it in its own cell and it will do its thing


////////////////////////////////////////////////////////////////////


import pandas as pd

class buoys:
   
    def __init__(self,url):
       
        self.url=url
        self.df=pd.read_csv(self.url)
       
    def thice(self):
       
       
       
        df_HEAT=self.df
        ##########
        #possible integer to change below
        ##########
        df_HEAT=df_HEAT.tail(10)
        df_HEAT=df_HEAT.drop(['time', 'latitude (deg)','longitude (deg)'], axis=1)       
        df_HEAT.loc['stdev'] = df_HEAT.std()
        df_HEAT.loc['mean_t'] = df_HEAT.mean()       
        df_HEAT=df_HEAT.tail(2)
        df_HEAT=df_HEAT.T
        df_HEAT.reset_index(level=0, inplace=True)
        df_HEAT["Temp_Change"]=df_HEAT.mean_t.diff()
        ##########
        #possible integer to change below
        ##########
        df_ocean=df_HEAT.tail(50)
        core_temperature=df_ocean.mean_t.mean()
        df_HEAT=df_HEAT.head(-5)
        df_HEAT["Deltacore"]=df_HEAT.mean_t-core_temperature
        ##########
        #possible integer to change below
        ##########
        thermistors_not_in_ice=df_HEAT.where(df_HEAT.Deltacore>1)
        thermistors_not_in_ice=thermistors_not_in_ice.dropna()
        Th_ice= thermistors_not_in_ice.tail(1)
        Th_ice=list(Th_ice.index.values)
        Th_ice=int(Th_ice[0])
       
        self.Th_ice=Th_ice
        return(self.Th_ice)
       
       
T56=buoys("https://data.meereisportal.de/download/buoys/2019T56_300234065176750_HEAT120_proc.csv")
T58=buoys("https://data.meereisportal.de/download/buoys/2019T58_300234065171790_HEAT120_proc.csv")
T62=buoys("https://data.meereisportal.de/download/buoys/2019T62_300234068706290_HEAT120_proc.csv")
T63=buoys("https://data.meereisportal.de/download/buoys/2019T63_300234068709320_HEAT120_proc.csv")
T64=buoys("https://data.meereisportal.de/download/buoys/2019T64_300234068701300_HEAT120_proc.csv")
T65=buoys("https://data.meereisportal.de/download/buoys/2019T65_300234068705730_HEAT120_proc.csv")
T66=buoys("https://data.meereisportal.de/download/buoys/2019T66_300234068706330_HEAT120_proc.csv")
T68=buoys("https://data.meereisportal.de/download/buoys/2019T68_300234068708330_HEAT120_proc.csv")
T70=buoys("https://data.meereisportal.de/download/buoys/2019T70_300234068705280_HEAT120_proc.csv")
T72=buoys("https://data.meereisportal.de/download/buoys/2019T72_300234068700290_HEAT120_proc.csv")

print(T56.thice())
print(T58.thice())
print(T62.thice())
print(T63.thice())
print(T64.thice())
print(T65.thice())
print(T66.thice())
print(T68.thice())
print(T70.thice())
print(T72.thice())
   
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 01, 2020, 04:18:13 PM
Ps, technically my thermistors should be +1 on its value because it is theoretically the first thermistor ABOVE the ice

Not sure about the 1st one



//////

to fix that, add the line

Th_ice=Th_ice+1

underneath

"Th_ice=int(Th_ice[0])"

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 01, 2020, 07:38:40 PM
Will look at that in a bit. First draft of the ani. My initial impression is that it's a shame to lose the path detail by using only 1frame/day. How about we use all the ts data but replicate the thicknesses.
Note that none of the buoys are over 2m thick using this method.  ctr
I have a feeling they are all going to experience sudden melt at this drift rate.

adjusted a few things ctffr
and added est snow thickness, though I can't do 2 colours at a time.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 01, 2020, 09:18:20 PM
The 2 remaining cryosphereinnovation simb buoys (https://www.cryosphereinnovation.com/386850) are in broad agreement with our numbers. Snow depth perhaps more stable.
piomas (https://forum.arctic-sea-ice.net/index.php/topic,119.msg254994.html#msg254994) indicating our results should be a touch thicker
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 02, 2020, 01:19:42 PM
Looks great, fantastic job on the animation, I agree about the frames, I will retain all ts data "fill in the blanks" for thickness
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 02, 2020, 01:47:03 PM
Another thought, maybe thickness change would be better for path colour retaining thickness as label. I can do that in R though.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 02, 2020, 02:13:10 PM
Another thought, maybe thickness change would be better for path colour retaining thickness as label. I can do that in R though.

you can also change the size of the points based on a quantitative variable such as thickness


//////


youll struggle to do thickness change in R now ive stacked the rows n times (where n is the number of TS measurements per day)

ill add it into the code so it operates before this line
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 02, 2020, 05:27:56 PM
I wrote a script for plotting individual buoys in a "nice" way. Based on the figure you shared above
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on April 02, 2020, 05:39:48 PM
So we have an almost linear growth of ice thickness and changing temperatures don't dramatically change the growth? That's stunning.  :o
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 02, 2020, 06:45:49 PM
So it would seem. Id say im probably less informed than yourself and uniquorn about this, but its maintained across the buoys

Sorry about the legend clipping here, python is being a real pain



FYI that is an animated gif
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 02, 2020, 07:52:36 PM
Easier to see slow down in thickening, probably due to temp, with a 'taller' scale
btw SF, Nice emulation ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 02, 2020, 08:30:00 PM
We deliberately cut the start day to Dec 1st, but when you take out that line the changes in growth become more obvious.

We put that line in because not all the buoys start transmitting on the same day so the data-structure gets messy



///////

please ignore the dates, they are now bugged- code is like a house of cards
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 04, 2020, 11:53:53 AM
20 frames of estimated change in thickness. Still tinkering. ctffr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 04, 2020, 02:50:47 PM
20 frames of estimated change in thickness. Still tinkering. ctffr

Looking good.

Would you consider not using viridis as the colourmap for this plot? There are some diverging colourmaps that might be better for easily spotting whether its loss or gain. Im not sure if they translate directly into R but there is a list here;

https://matplotlib.org/3.1.1/gallery/color/colormap_reference.html


Also, for the temps (which is a great feature) I can give you a once-daily temp dataframe so they dont have flats at each day- something I guess is because there are repeating measurements due to the TS fix.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 04, 2020, 03:11:12 PM
R not happy with discrete colourmaps with the current coding. Please suggest your favourite continuous.
Was lazy with the chart but wanted to fill up the blank space.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 04, 2020, 03:27:47 PM
"icechange <- colourmap(c("red", "blue", "green"), breaks=c(float(max_loss),0,
                     (float(max_gain)
)"

May allow you to make a continuous colourmap of your own choosing. Im pretty bad with R though.

https://www.rdocumentation.org/packages/spatstat/versions/1.63-3/topics/colourmap
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 05, 2020, 07:19:36 PM
Looking at thickness change colours and incorporating some bathy (scale appears to be wrong). ctffr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 06, 2020, 12:38:07 PM
Looking at thickness change colours and incorporating some bathy (scale appears to be wrong). ctffr

starting to look really good

are the buoys moving in frames of 10 measurements-per-movement at the moment?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 06, 2020, 12:58:59 PM
After asking for more rows I went back to 1frame/day. Text repel labels jump about too much at that resolution. If you are happy with the numbers I'll post sthing similar to this on the mosaic thread as a reference. Subsequent posts can then be higher res, shorter time frame, more distance between the labels.
For me, the drift path is important so thanks for the extra coding.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 06, 2020, 01:08:57 PM
No problem, I think it looks great- well done.

Id consider having a "thickness" figure, exactly the same as your "change in thickness" except where the colourmap change is just thickness (you've done it before in viridis). I think the two will complement each other and will provide different benefit to different people in the community
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on April 06, 2020, 02:17:39 PM
https://github.com/ipython/xkcd-font

Just sayin'.  ;D
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 06, 2020, 03:41:31 PM
Thats pretty awesome blumenkraft, I didnt know such things existed.

Speaking of, here is the python half of our work.

I havent annotated this version, but I will add an annotated one when i get the time

https://github.com/SimonF92/Arctic/blob/master/Final_MOSAiC_Mining.py
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on April 06, 2020, 04:10:40 PM
I didnt know such things existed.

Me neither until today!  :D

But now i feel everyone should know about it.  ;)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: kassy on April 06, 2020, 04:38:41 PM
Some nice looking work! I enjoy watching this arcane magic at work.

PS: Who doesn´t hate bed kerning.  ;)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 06, 2020, 04:51:11 PM
Ran the new script in spyder (I prefer it to the browser). It created a new Buoy_Data directory in the working directory and wrote all the files. Chart dates stop at mar30? ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 07, 2020, 01:36:13 PM
Ran the new script in spyder (I prefer it to the browser). It created a new Buoy_Data directory in the working directory and wrote all the files. Chart dates stop at mar30? ctr

Thanks for checking that uniquorn, im glad it works in Spyder so thats good to know.

The plots go to todays date, its a problem with the X-ticks. I hadnt noticed, so thanks for pointing that out, ill try fix it
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 12, 2020, 10:04:49 AM
These charts are a very quick and easy check on ice thickening, which may be slowing on some buoys, and local temperature but until the time axis is fixed they are best posted here. If we start to see melt I'll run the thermistor temp charts to check our calculations. ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on April 14, 2020, 12:55:20 PM
These charts are a very quick and easy check on ice thickening, which may be slowing on some buoys, and local temperature but until the time axis is fixed they are best posted here. If we start to see melt I'll run the thermistor temp charts to check our calculations. ctr

Its fixed :)

I was a really annoying bug to fix, turned out to be because there are duplicates as a result of keeping the TS data per date.

All good now.

Will update the code on github
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 19, 2020, 06:57:20 PM
Tinkered a bit with the charts to check the data, temperature appears to that at 07:00
T56
2020-04-13T01:00:14  -31.25
2020-04-13T07:00:14  -29.06
2020-04-13T13:00:14  -25.44
2020-04-13T19:00:15  -27.12
2020-04-14T01:00:14  -25.31
2020-04-14T07:00:14  -21.62
2020-04-14T13:00:14  -18.62
2020-04-14T19:00:14  -16.62
2020-04-15T01:00:14  -14.06
2020-04-15T07:00:14  -9.31
2020-04-15T13:00:14  -6.88
2020-04-15T19:00:14  -6.12
2020-04-16T01:00:14  -6.38
2020-04-16T07:00:14  -1.88
2020-04-16T13:00:14  -1.25
2020-04-16T19:00:14  -1.5
2020-04-17T01:00:14  -13.31
2020-04-17T07:00:14  -17.31
2020-04-17T13:00:14  -13.31
2020-04-17T19:00:14  -14.88
2020-04-18T01:00:14  -12.38
2020-04-18T07:00:14  -7.25
2020-04-18T13:00:13  -3.12
2020-04-18T19:00:14  -9.19
2020-04-19T01:00:14  -3.31

T63 may have failed. ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 22, 2020, 01:44:25 PM
Will update the code on github
After that warm spell, perhaps the first signs of bottom melt on T72 or just a warming at the ice/ocean interface.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 22, 2020, 02:06:50 PM
hmm, some event happened on apr13. Must be water over the surface, perhaps a very close lead opened up. Changed the temperature gradient completely.

added rough location of T72 on S1B, apr14. PS is visible as a bright dot, top right.  cffr
2020-04-14T10:00:14,84.522186,11.996773
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 27, 2020, 07:38:32 PM
thickness estimate update
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on April 28, 2020, 06:07:28 PM
So I've just read 111 posts in this amazing thread. Huge kudos to uniquorn and SimonF92 for analyzing and cracking the data of the mass balance buoys. I should have tracked the thread in real-time but my mind was elsewhere.
Before I turn to trying to analyze the data myself using the tools you have created, which surely will not be easy with my limited time and skills, I have one request right off the bat which I hope you can implement easily.
I consider the ice as some sort of capacitor, dampening the air temperature signal and integrating it on its way to the water interface. I can easily understand how the wild swings in the air signal are not immediately apparent when looking at the thickness growth curve. And I think there is an easy way of visualizing what goes on under the hood so to speak.
Add to the standardized thickness/temperature charts posted herein another temperature plot - that of the thermistor found 50cm below the upper ice surface, i.e. 50cm below the air or snow interface. Consider this the "core temp" of the ice. This variable is driven by the air temp above, and drives the thickening below, via a temp gradient through the ice. Later in the season this variable will rise above some threshold (is that -1.8o?) and will no longer be able to drive thickening, but will enable bottom melt instead.
Ideally I would add another plot, of the thermistor that is 100cm below the upper ice surface, as long as it is at least 20cm above the bottom ice surface (ice-water interface) to avoid noise and meaningless data. This would be the "deep core temp".
Looking at these two plots together with the air temp plot and the thickening rate will hopefully enable a visually intuitive understanding of the processes governing bottom freezing and bottom melt.

Later I hope to calculate season FDDs over the buoy and compare to the theoretical plots of ice thickening.
I also want to examine the effect of thick/thin/zero snow on the air temp to core temp coupling. Theoretically the effect should be quite strong.
I hope to find the date(s) on which thickening stops and bottom melting begins. I hope a rule of thumb will be found to enable estimating this date even when no buoy is available
And of course, to try and find ways to compare the buoy data to PIOMAS, SMOS/SMAP and even to the dubious DMI thickness map.
Big hopes...
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 28, 2020, 09:35:52 PM
I think we still may have a problem with snow. Looking at a recent temperature gradient on T56 on apr11 using a 'free' spreadsheet (LibreOffice Calc) I can see a change in gradient at thermistors 24-30. I think that is ~12cm of snow but the Python generated chart above doesn't show it.
Assuming that ice starts at Therm30 we call therm(core)55 and therm(deepcore)80. Remember 2cm/therm
 
T56heat not giving a clear indication of snow between Therm24-30 on apr11. It agrees within 2therms at the ice bottom but is 10 therms different to the ice top??
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 28, 2020, 09:45:50 PM
Select the columns time, T1, T30, T55 and T80 while holding down the Ctrl button on the kbd.
Click the insert chart button and select lines only
Deep core is water until december so you might want to shorten the range.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on April 28, 2020, 10:54:20 PM
Thanks uniquorn, that's what I wanted. And I use Excel anyway so the learning curve should be manageable. But where do I download said spreadsheet for the various buoys? I probably missed the link or location while making my way down the thread.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 28, 2020, 11:36:23 PM
https://forum.arctic-sea-ice.net/index.php/topic,3013.msg252432.html#msg252432
or here (https://data.meereisportal.de/gallery/index_new.php?active-tab1=method&buoytype=TB&region=all&buoystate=active&expedition=MOSAiC&submit3=display&lang=en_US&active-tab2=buoy) if they deploy any new ones.
edit: There were 6 new Tbuoys when I checked. Here is the full list though some are not active

https://data.meereisportal.de/download/buoys/2019T56_300234065176750_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T57_300234065177750_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T58_300234065171790_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T59_300234065170760_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T60_300234066299840_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T61_300234065768480_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T62_300234068706290_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T63_300234068709320_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T64_300234068701300_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T65_300234068705730_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T66_300234068706330_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T67_300234068704730_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T68_300234068708330_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T69_300234068700320_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T70_300234068705280_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T71_300234068706760_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2019T72_300234068700290_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T73_300234068328160_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T74_300234068225520_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T75_300234068325170_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T76_300234068528490_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T77_300234068524740_TEMP_proc.csv
https://data.meereisportal.de/download/buoys/2020T79_300234068527600_TEMP_proc.csv
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 29, 2020, 01:24:30 PM
T58 heat's match quite well but T56's don't. Will run some more later  ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 29, 2020, 08:52:25 PM
T62 matches quite well, T64 doesn't.  ctr
Possible file naming error at mosaic/meereis?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 29, 2020, 09:22:22 PM
T65 and T66 match quite well. T66 may have a faulty heater  ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 29, 2020, 09:40:47 PM
T68 and T70 match quite well
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 29, 2020, 10:10:29 PM
T69 and T72 still struggling on.  Not sure if that's water on the surface again on T72 yesterday. ctr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on April 30, 2020, 05:16:58 AM
From the "stupid questions" corner:
So I've gone as far as installing Python for Windows and have downloaded Simon's file from github. When trying to run the file I get an error message "ModuleNotFoundError: No module named 'pandas'". It appears to be one of (probably) several missing libraries the script is trying to import. Any idea what I've done wrong, or what should I do to remedy the situation?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on April 30, 2020, 02:01:25 PM
From the RTFM answers corner, looking at Simon's code:
##Written in Anaconda Python v3.7 Jupyter

Downloading Anaconda Python, hoping for better results.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on April 30, 2020, 04:11:40 PM
Probably anaconda will have pandas, if not, these lines were in a previous script

#uncomment and run this first line if you get an error about not having pandas
#! pip install pandas

# denotes a comment, not sure about the !
I'm using Spyder which detected the import and asked if I wanted to install it.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on April 30, 2020, 05:10:34 PM
Anaconda fixed the problem and ran the script successfully. Will attempt the next steps tonight using the walkthroughs upthread.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 02, 2020, 12:03:51 PM
Recent location of all Tbuoys. We may have lost T72.  ctffr
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 02, 2020, 07:10:48 PM
There is one newish Mbuoy (https://data.meereisportal.de/gallery/index_new.php?active-tab1=method&buoytype=MB&region=all&buoystate=active&expedition=MOSAiC&buoynode=all&submit3=display&lang=en_US&active-tab2=buoy) deployed on feb16 that has data in the same format as the Tbuoys. However the HEAT file records the actual temperature rather than the difference so it's tricky (for me) to add it to the py file.

I did manage to add 4 of the new Tbuoys by adding the url's to the lists and editing the start date to apr15. click to run
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 03, 2020, 01:37:22 PM
Modifying the basic spreadsheet formula to improve calculations when air or snow temperatures are above -2.1C
From upthread (https://forum.arctic-sea-ice.net/index.php/topic,3013.msg252575.html#msg252575)

Quote
The kind people at libreoffice.org have implemented COUNTIF which allows us to count the number of cells in a row with a lower temperature than our ice:ocean interface. I'm going to set that at -2.1 and let the wordsmiths on this forum discuss in detail what it should be.

Cell formula will be   =COUNTIF(D2:II2,"<-2.1")
We omit column IJ. I think it is some sort of end of line check.

Now subtract Tas=40
Cell formula will be   =COUNTIF(D2:II2,"<-2.1")-40

multiply by 2 because the thermistors are 2cm apart.
Cell formula will be   =(COUNTIF(D2:II2,"<-2.1")-40)*2

Label your estimate column, select the cell beneath it and click F2 on the keyboard.
Paste in the formula and click Enter on the kbd
The modified formula is

=COUNTIF(AN2:II2,"<-2.1")*2
where AN2 is the column for the thermistor nearest the snow:ice interface.

This prevents air or snow temperatures above -2.1C being included in the 'ocean' countif
I think SimonF92 covers this in the Py method by identifying these temperatures as 'not ice' using stdev
AN2 is currently valid for T76
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 13, 2020, 07:10:42 PM
Found some deployment info that may help verify ice and snow thickness calcs.
In particular initial ice thickness
T60=6.1m
T61=7.05m
Probably a bit late for checking against the older deployments
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 14, 2020, 06:00:20 PM
uniquorn got me thinking........

Im pretty sure I could deploy our entire code in a webapp.

This would then be an interactive website that lets users look at individual buoys etc using the figures we developed.

https://www.streamlit.io/

It will be such a minimal app (memory and storage wise) that I wonder if there is a way to host it on googles webapp-engine for free. Could be pretty cool. Will look into it further



Also, some of their measurements are 6+metres
Is that believable?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 14, 2020, 07:15:28 PM
Sorry oren, it's my fault the code isnt readable, this is something I really should put more time into.

Its not an RTFM issue as I haven't been very helpful.

Normally a requirements.txt file should be included on Github, which is a one-liner with all the required installs
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 15, 2020, 04:47:13 AM
Thanks Simon. I haven't managed to get around to continuing my project yet, I hope it materializes at some point as there is a great wealth of data here.

Regarding the initial deployment I wonder that it does not state the thermistor number that is at the ice-air interface or the snow-air-interface or whatever. It seems to be something that is crucial to record when deploying.
The 7m thickness is hardly believable especially with a 5m chain, but I haven't investigated the details.

Continuing on the issue of thermistor numbering at the interfaces - the way I understand it the python code is intended to discover this. My question is whether it has given the same results throughout the season. If so, would it be possible to post here the resulting thermistor number for each buoy at the ice-snow interface?
If I understand correctly, this would enable analyzing the csv files without the need to run the python code.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 15, 2020, 11:06:37 AM
I think the 'fortress' ~1km distance from PS may be over 6m thick. I can't find the reference though.
Quote
On 28 September the first researchers from Polarstern set foot on the floe, which had long been a preferred candidate thanks to the promising analyses of the satellite data. On the radar images produced by the satellites, the dark, nearly oval floe stood out thanks to a large, bright region in its northern section. This clearly set it apart from all of the other potential floes, which were consistently dark in the radar images. In the meantime, the experts have dubbed this region ‘the fortress’: made up of highly compressed, several-meter-thick ice,
webapp is a good idea but probably won't run directly from the forum
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 15, 2020, 11:09:36 AM
Oren i maintain all of the outputs from the code at

bit.ly/MOSAiC_ASIF_Mining

There you will find the thermistor numbers you are looking for along with everything else, in csv format



Tldr,
The thermistor numbers for the ice/snow (air) boundary are essentially stable over the season, at least since their deployment in October- there have been no top-melt temperatures yet
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 16, 2020, 01:23:05 PM
Its going pretty well, here is the website- all of this runs on the website itself and not on my actual python environment.

Still about 50hours worth of interactivity to add, but im surprised at how well it seems to be going.

The best part is i can use a free amazon-EC2 to host the server. Im pretty sure those machines have 4Gb ram so it will be relatively slow but beggars cant be choosers :)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 16, 2020, 09:01:50 PM
I'm finding this presentation easier to work with (figsize=(5,10)) . Its easier to identify bottom melt.
Note 2 upticks on T62, T64, T65 and T76.

The taller chart would look nice to the right of the 'select buoy' buttons
btw I expect you are aware you are showing T70 with T56 selected...
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 17, 2020, 12:06:31 PM
Thanks a lot. Looking through the website, here are the thermistor numbers for the top of the ice, which are indeed constant through the winter:
T56: 38
T58: 31 (with a rare 32)
T62: 47
T63: 29
T64: 45
T65: 32
T66: 46
T68: 44
T70: 40
T72: 34 (went to 22 in the last day before failure)

I find these to be useful for anyone who wants to analyze the original csv files that contain the temps for each thermistor, so I'm putting them here.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 17, 2020, 02:00:57 PM
Update on the webapp and tasks based on what uniquorn and oren have said;

1) on the webapp provide a download feature for the analysed csv files
2) add an x-axis "zoom" function for the plots
3) provide a map of buoys based on their coordinates using Pydeck

If anyone has any other suggestions I will be happy to do it



PS its not too difficult to use streamlit, if anyone wants to have a play around with the webapp ill post the code to my github and provide a link
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 17, 2020, 02:54:39 PM
Simon, thanks for your efforts, two questions:
* Is there another link? The one I used goes to a dropbox without the features shown in the screenshot.
* I notice uniquorn is showing data from other buoys such as T74. Can these get on the site too?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 17, 2020, 03:30:55 PM
Simon, thanks for your efforts, two questions:
* Is there another link? The one I used goes to a dropbox without the features shown in the screenshot.
* I notice uniquorn is showing data from other buoys such as T74. Can these get on the site too?

The screenshots I am sharing now are from a webapp I am currently developing. At the moment that webapp runs on a local host so you wont be able to visit it, once it is nearing the end of its development I will run it from an EC2 and make the link available. The dropbox link you have visited is just a place to store the data (for now).

uniquorn kindly sent me the urls for the other buoys, so it is a very simple case of just copy and pasting them into the code- this can be done any time new buoys come online
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 17, 2020, 03:48:09 PM
-Added date-zooming and y-axis stretching for uniquorn.
-Also added caching for the computationally expensive processing part.

Im not doing all of this selflessly, I have an ulterior motive. I was just told that my lab-based PhD now needs to have a "computer chapter" for the time we have missed due to covid. Im going to build a webapp for analysing genetic data, as my chapter- it sure seems a lot more interesting than writing a review.

Anything I do here is really a learning process for me.

Therefore, if anyone has any ideas about what could be interesting to see, ill write it in.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 18, 2020, 04:30:12 AM
A question. I started playing with the data for buoy T66, looking at the various files:
dT120 file from Simon's dropbox
Temp file showing each thermistor
Heat030 file showing each thermistor
Heat120 file showing each thermistor
The PNG files shown on meereisportal

Using various eyeballing and charting methods and looking at Dec 1st, I can "see" that the air-snow interface is at thermistor 42, while the snow-ice interface is at 48 or 49.
This seems to fit with various posts upthread, for example post #70.
However the dT120 file and post #73 say both interfaces are at 46.
I'm trying to understand the difference in methodology that leads to this result, and what is the "correct" value to use. I know I should have asked this question a month ago, but...
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 18, 2020, 12:20:55 PM
oren, thanks for the questions, its good to have them because I want people to understand and agree with what we are doing.

uniquorn and myself have been discussing by messages and exchanging bits of code for a while trying to get this figured out.

Here is th_ice written out in full.

Th_snow uses somewhat similar method, but youve actually given me an idea for Th_snow that im going to try now. Let me see if i can get the bug with T66 you pointed out sorted too.



//////

by the way, its actually possible to manually assign th_ice and th_snow on a per buoy basis, but that would take the fun away when top-melt starts :)

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 18, 2020, 04:20:52 PM
Thanks again.
So T66 suffers from a specific bug? My bad luck. It hadn't occur to me to play with the others until I figured out where I was going wrong.

I would actually recommend to make a consensus "decision" using the automated algorithms that were developed, along with manual validation, to identify the TH_ice for each buoy, using the long set of wintertime data, and before top melt comes along to mess things up. The original Th_ice of each buoy is the single most important piece of the analysis IMHO because all the rest of the conclusions hinge on its correct identification. And in physical reality it is an installation-dependent constant, so its final identification should be feasible.
I will (hopefully) go through the data of the various buoys, and compare to posts #73, #70 and my own post above that summarizes the "dropbox" analysis, and will try to come up with manual validation of the results for Th_ice.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 18, 2020, 05:18:25 PM
I completely agree. Infact, im going to provide a link to the webapp now because itll be by far the best way for a few people to get a look and check these things. I would ask that not too many people visit this server until i get the kinks of hosting it ironed out.

Its currently running on 1cpu with 1gig of RAM, on an AWS EC2, completely detached from my own machine in every way.

I will be taking it down 2-hours from now to check various things and also change the domain name to something pretty

http://3.19.26.242:8501
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 18, 2020, 06:04:34 PM
I recommend one small change that will enhance usability of the table.
Either shorten the titles, for example ts_latitude (deg) can be lat(deg) , or allow word wrap of the titles over several rows. The goal should be that the column width will fit the width of the data in the column. This will result in much narrower columns such that lots more data can be seen at a glance without scrolling - including my favorite Th_ice.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 18, 2020, 06:35:22 PM
I recommend one small change that will enhance usability of the table.
Either shorten the titles, for example ts_latitude (deg) can be lat(deg) , or allow word wrap of the titles over several rows. The goal should be that the column width will fit the width of the data in the column. This will result in much narrower columns such that lots more data can be seen at a glance without scrolling - including my favorite Th_ice.

Added toggle for "key metrics"
(to the offline dev version)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 18, 2020, 09:21:01 PM
T65 and T66 match quite well. T66 may have a faulty heater  ctr
noted upthread

webapp is great, location not running yet I assume.... or suddenly it is :)
Probably should give latest date rather than 'current coordinates'. Latest coordinates would do.

One other thing. Tbuoys record temperatures 4 times/day. Not sure which is the best to present on the chart. An average? Currently it is the 0700 temp. T58 shown
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 19, 2020, 10:45:56 AM
T61 location, reportedly 7.05m thick, for any floe spotters out there.
map rotated 45deg
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 19, 2020, 02:25:07 PM
Several improvements to the webapp

1) Better maps
2) Added new buoys
3) Added function to easily change parameters of the code

uniquorn, i see your point, thats going to be pretty difficult to do though  :-[


Webapp will be kept online for 12-hours. Find it here:

http://52.14.65.105:8501
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on May 19, 2020, 04:14:24 PM
Amazing, Simon! Well done.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 20, 2020, 02:51:24 PM
Amazing, Simon! Well done.

Thanks!

Understandably I have been very careful about keeping track of usage. At the moment, it looks like if i leave the website live indefinitely, I shouldn't incur any charges.


Random looking ip addresses can be off-putting, I get that. For this reason, I just registered a domain name at "www.mosaic-ice.com", which is being processed by amazon. I can then route the IP of the webapp to that domain name and square the project off.

In the meantime, im going to leave it running for 5-days here;




http://18.222.70.206:8501




Id encourage anyone and everyone to check it out, just so I can be sure that multiple users creating their own cache doesnt go over-and-above my free limits.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 20, 2020, 02:59:16 PM
Simon thanks so much for creating this.
Isn't there some way for you to set limits on usage such that when it gets to the free allotment it stops for the month?
If not, maybe it would help putting the % of quota used on the home screen, so that users will know to avoid using the site when nearing the limit.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on May 20, 2020, 03:20:54 PM
A question. The dropbox enabled easy download of the processed data files. Will it be possible to do the same from the web app site?
I hope I am not bombarding with requests. Still not sure what of all this I will use, just checking the goods...

Edit: I think we have already discussed download, but I haven't found how to on the current site.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 20, 2020, 03:31:29 PM
A question. The dropbox enabled easy download of the processed data files. Will it be possible to do the same from the web app site?
I hope I am not bombarding with requests. Still not sure what of all this I will use, just checking the goods...

Edit: I think we have already discussed download, but I haven't found how to on the current site.

I feel amazon deliberately try to lead you down as slope as they dont provide a way to turn off a machine at a cpu-hour threshold.

Thanks for the suggestion about displaying the quota. However, I feel this would potentiate a back-door onto my account, as I would have to write a way of pulling that data to the webapp. I have set a monthly budget of $5, with email alerts at $2.50, so I should be able to stay well on-top of things.

With respect to downloading the datasets. I am sure there must be a way, still working on figuring it out- but its top of my list!
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 20, 2020, 04:06:30 PM
oren, you should now be able to download csv files straight from the site, let me know how it goes

(I dont own excel so i can only view them as notepad files)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 20, 2020, 04:57:04 PM
just downloaded T56. It needs another 12 zeros at the beginning of the 'change_in_thickness(m)' column. Not much of a problem. Site is timing out right now.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 20, 2020, 05:33:14 PM
Replaced NaNs with 0s to help with downloading and using csv files.

Rerouting traffic to new website/ domain name (at a very small cost in the region of $0.40 per million lookups):

The port at 8501 is unfortunately a must, and wont be going away any time soon


http://www.mosaic-ice.com:8501/
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SteveMDFP on May 20, 2020, 05:42:45 PM
The port at 8501 is unfortunately a must, and wont be going away any time soon

http://www.mosaic-ice.com:8501/

"Any port in a storm."
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on May 20, 2020, 06:44:23 PM
The port at 8501 is unfortunately a must, and wont be going away any time soon

http://www.mosaic-ice.com:8501/

"Any port in a storm."

10/10, had a giggle
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 26, 2020, 11:02:24 AM
http://www.mosaic-ice.com:8501/
data currently stops at may21 (needs a reboot occasionally?)
suggestions:
add links to the original files used (temp_proc, ts, TEMP120)
Is there any chance to show all the 6hourly therm1 temperatures? If not a daily average might be better.
Sorry, have no time to look at streamlit at the moment.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 28, 2020, 10:29:46 PM
python ice thickness update, may28. Posting here until I'm sure they are reliable.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on May 30, 2020, 11:56:57 PM
Checking Tbuoy thickness
T73 definitely has a problem. The last few rows of T62.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 01, 2020, 03:49:49 PM
T77 temperatures, heat 30sec and heat 120sec, may10-jun1
melted snow refreezing?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 01, 2020, 05:32:32 PM
The t77 chart above shouldn't be showing that much bottom melt. Here is a close look at the ice/ocean thermistor which stays at, or very close to therm130, possibly moving to 129.
Possibly related to the maximum ocean temperature we are using.
The problem may get worse as the temperature gradient levels off. Tricky as there may be super cooled water beneath the ice. edit:I doubt that with warmer air temps.

updated below
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 01, 2020, 07:24:26 PM
The t77 chart above shouldn't be showing that much bottom melt. Here is a close look at the ice/ocean thermistor which stays at, or very close to therm130, possibly moving to 129.
Possibly related to the maximum ocean temperature we are using.
The problem may get worse as the temperature gradient levels off. Tricky as there may be super cooled water beneath the ice. edit:I doubt that with warmer air temps.

Spring (summer?) really is messing with these buoys, I have a feeling the reliability of any measurements will be decreasing exponentially. Just look at their raw data.

By the way I cant edit the website myself (im on the clock for a project these days) but I just found out its imperative that one clears the cache to see the most up-to-date data.

Clearing the cache can be done in the tab on the right of the main page

Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 01, 2020, 08:03:16 PM
Thanks, streamlit updates with cleared cache. Still checking data but I see the old python version I was using had mean ocean temp of < -2.5C. Editing that to -2.1C gives closer alignment to the bottom melt thermistor (like the streamlit version using -2.2C). I'd like to go lower but there are a few -2.06C deeper down. That drives up ice thickness.
examples below.
Will keep tinkering. maybe <-2.06C ....
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 01, 2020, 10:55:39 PM
Agree completely about the supercooled idea (well not "supercooled")- there is probably a phase-equilibrium across several thermistors at the moment. Possibly supplement with HEAT data? Ill look into it
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 02, 2020, 02:54:00 PM
Webapp update.

Amazon emailed me suggesting I may be eligible for 300$ account credit based on my activity. Should I qualify ill be able to significantly ramp-up the scope of;

www.mosaic-ice.com:8501

Adding several more features, including the ability to handle more than one dataset at any one time (due to migrating to a faster machine). Fingers crossed.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 02, 2020, 04:40:53 PM
Good luck :) as long as you don't block adblockers.
Continuing analysis of T77 here are the temperatures from may18-jun2. I think there was snow between therm39 (possible therm37) and 43 and based on the animation above, I don't think there is bottom melt yet.
Not sure which parameters can be adjusted to match.
click for a gif to get rid of the lines.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on June 02, 2020, 04:51:30 PM

www.mosaic.ice.com:8501


What's happening here? Is ice.com your domain?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 02, 2020, 06:52:36 PM

www.mosaic.ice.com:8501


What's happening here? Is ice.com your domain?

Oops! Fixed


Great figure uniquorn, so much data
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 02, 2020, 08:23:58 PM
T76 almost makes me want to give up this idea. One of the heaters looks like it's broken and there appears to be an air cavity that has been iced over. Drained before it could freeze?
The estimation seems too low at the moment.  117-41=76, *2=152cm plus maybe 24cm snow(less since the warm spell) click to play
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 02, 2020, 10:52:39 PM
I didnt know this existed, but reassuringly we use somewhat similar methods

https://www.seanoe.org/data/00485/59709/
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: blumenkraft on June 03, 2020, 11:24:18 AM

Oops! Fixed

Haha, ok! That would have been amazing.  ;D
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 03, 2020, 02:24:58 PM
I didnt know this existed, but reassuringly we use somewhat similar methods
https://www.seanoe.org/data/00485/59709/
Winter only. I think our winter results are ok though snow needs a bit more work.
Might have to be more manual for summer. If the buoys last that long.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 03, 2020, 03:57:24 PM
I take it when they do leave the pack they will get scooped back up and put further north again? Iv been watching their relentless and almost unprecedented race toward the fram with bemusement, half of them will be out the ice within a month at this rate
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 03, 2020, 04:17:08 PM
There doesn't seem to be an 'afterstory' for our courageous buoys. If any parts do get re-used they are anonymously renumbered. They should 'do a Musk' and go big on reusability. Whoi itp116 (not part of mosaic)already making its way down the greenland coast.
Hopefully they have a new batch ready for deployment further north.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 03, 2020, 04:46:57 PM
There doesn't seem to be an 'afterstory' for our courageous buoys. If any parts do get re-used they are anonymously renumbered. They should 'do a Musk' and go big on reusability. Whoi itp116 (not part of mosaic)already making its way down the greenland coast.
Hopefully they have a new batch ready for deployment further north.

If all else fails, I will take one and put it in my living room. I think it would be a good conversation starter. Infact, at 5metres it might do better in the garden- my neighbours might think im conducting surveillance and stop secretly feeding my cat
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 04, 2020, 10:54:40 PM
Checking T64, the most southerly.
2020-05-10   28   45   144
algo is detecting the snow peaks from heat, pretty good. Arguably the ice should be thicker at th150 but we have to make a cut off temperature.

There is a loss of snow event by may26 22:00. Algo begins to detect it on may30 though the chart shows it before then.
2020-05-29      28   45   141
2020-05-30    33   45   140
snow drops to 2cm, should be closer to 10cm.

I don't see any bottom melt. algo detects 6cm from may24. Experimenting with the old py code I think I see better results with mean ocean temp of -2.05 during summer. Less 'phase-equilibrium' perhaps.
2020-05-24   28   45   144
2020-05-25   28   45   142

Something is wrong near the end because we lose 0.1m. Incomplete day perhaps?
2020-06-03   43   44   140   -0.06   T64   0   -1.8   0.02
2020-06-04   43   44   139   -0.1   T64   0   -1.7   0.02





Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 05, 2020, 03:54:16 PM
Checking T64, the most southerly.
2020-05-10   28   45   144
algo is detecting the snow peaks from heat, pretty good. Arguably the ice should be thicker at th150 but we have to make a cut off temperature.

There is a loss of snow event by may26 22:00. Algo begins to detect it on may30 though the chart shows it before then.
2020-05-29      28   45   141
2020-05-30    33   45   140
snow drops to 2cm, should be closer to 10cm.

I don't see any bottom melt. algo detects 6cm from may24. Experimenting with the old py code I think I see better results with mean ocean temp of -2.05 during summer. Less 'phase-equilibrium' perhaps.
2020-05-24   28   45   144
2020-05-25   28   45   142

Something is wrong near the end because we lose 0.1m. Incomplete day perhaps?
2020-06-03   43   44   140   -0.06   T64   0   -1.8   0.02
2020-06-04   43   44   139   -0.1   T64   0   -1.7   0.02

I get a day off Sunday. Im going to take some time to properly use python to debug the model, will post the comparison results here
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 05, 2020, 03:57:24 PM
PS iv been thinking, the ocean is more saline in the region the buoys are migrating to. This would in theory increase bottom melt, even though to us it would appear not much had changed in terms of ocean temp? Could this be a factor?

I cant find a current figure, but I know you and blumenkraft are pretty keyed in with your maps. Heres where T64 is. Im pretty sure thats close to a sharp salinity gradient

Scrub that i found one :)
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 05, 2020, 06:39:34 PM
whoi itp94 (https://www.whoi.edu/page.do?pid=165636) is our active profiler in the array. It's a touch more saline but presently able to be much a bit colder to match, mixing down to 150m. Besides, if there was bottom melt we'd see it on the animation.
Don't spoil your weekend for it.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 07, 2020, 06:56:46 PM
Ok so it looks like that the recent temperatures are causing the value of the Th(ice) to change, not Th(ocean), causing these spurious "melt" events due to the way I have coded the final figures, will continue to work on it

this is T64
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 07, 2020, 09:44:31 PM
Thanks. If you have time it would be good to see the py code before it goes streamlit
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 08, 2020, 12:04:18 AM
 ::)
Thanks. If you have time it would be good to see the py code before it goes streamlit

No problem, will post a link here when its done
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 09, 2020, 10:27:27 AM
Ok so I have been really working at this, using algos for both HEAT deviation from top and bottom separately, then in combination. The readings are just too sporadic to return meaningful and consistent values. Th(ice) is fluctuating no matter what I try.

So I came up with something else.

How about we SET th(ice) as the winter measurement. Now, when temperatures are more variable and harder to work with, I suggest we use this set value for th(ice), but change it based on the ambient air temperature.

 If ambient air temps are above 0degc for a sustained period, we infer top-melt. Here's the if statement long-hand.

if (ambient air temperature is >0degC for 24hours) then (Th(ice) = Th(ice) - 1)

I need some expert opinions.

1) is this reasonable?
2) How do we calibrate? ie is 24hrs of +1degC enough to cause 2cm of top melt?

In the image, you can actually see the measurement blip that is throwing off our Th(ice), unfortunately there really is not much I can do to correct for this.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on June 09, 2020, 11:03:17 AM
Yes, Th(ice) needs tp be fixed at a stable value derived from winter. This represents a physical attribute of the buoy installation. Since ice does not grow from the top, once installed Th(ice) remains the same. It just need to be discovered once for each buoy, using the various algorithms.
This, combined with limited time invested, has been hindering me from the start. I can't concentrate enough to decide once and for all on this Th(ice)(installed) for each buoy, as each algorithm gives a slightly different number. IMHO the installers should have published this number as part of buoy data, but oh well, never mind.
As for top melt, this can be identified by temperatures below Th(ice)(installed) being equal to air temperatures above that. Air temps fluctuate. Ice is an insulator. If a thermistor is in ice, it will fluctuate less and remain more rangebound than air temp. (The same is even truer for snow). In addition when air temps are extreme in any direction, the ice (and snow) should show a rather smooth gradient. Thus whether each thermistor is exposed to air or not can be calculated from the data, and no need for formulas which really cannot give a true answer as ice melt depends on more than just air temps - sunlight or clouds, wind speed, humidity, drainage, ice salinity, and more.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 09, 2020, 12:16:36 PM
Yes, Th(ice) needs tp be fixed at a stable value derived from winter. This represents a physical attribute of the buoy installation. Since ice does not grow from the top, once installed Th(ice) remains the same. It just need to be discovered once for each buoy, using the various algorithms.
This, combined with limited time invested, has been hindering me from the start. I can't concentrate enough to decide once and for all on this Th(ice)(installed) for each buoy, as each algorithm gives a slightly different number. IMHO the installers should have published this number as part of buoy data, but oh well, never mind.
As for top melt, this can be identified by temperatures below Th(ice)(installed) being equal to air temperatures above that. Air temps fluctuate. Ice is an insulator. If a thermistor is in ice, it will fluctuate less and remain more rangebound than air temp. (The same is even truer for snow). In addition when air temps are extreme in any direction, the ice (and snow) should show a rather smooth gradient. Thus whether each thermistor is exposed to air or not can be calculated from the data, and no need for formulas which really cannot give a true answer as ice melt depends on more than just air temps - sunlight or clouds, wind speed, humidity, drainage, ice salinity, and more.

I agree with what you are saying, human eye for this may actually be best. However, if we do set th(ice) as immutable we reduce dimensionality through the assumption that top-melt does not exist?

At the moment, being exasperated as I am with the recent changes to reliability, I might just make it an immutable value.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on June 09, 2020, 12:34:13 PM
The fixed value should be Th(ice)(installed) or similarly named. Th(ice) should vary when top melt begins, but never fall below (physically above...) Th(ice)(installed). Th(snow) could still make an appearance even during summer.

Thus we have air-snow-ThIcIn-ice-ocean thermistors in winter.
And air-ThIcIn-air-ice-ocean thermistors in summer when there is no snow
Thickness graphs should best be aligned at Th(Ice)(Installed) as they were in winter, thus showing top melt as well as bottom melt visually.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 09, 2020, 12:46:22 PM
The fixed value should be Th(ice)(installed) or similarly named. Th(ice) should vary when top melt begins, but never fall below (physically above...) Th(ice)(installed). Th(snow) could still make an appearance even during summer.

Thus we have air-snow-ThIcIn-ice-ocean thermistors in winter.
And air-ThIcIn-air-ice-ocean thermistors in summer when there is no snow
Thickness graphs should best be aligned at Th(Ice)(Installed) as they were in winter, thus showing top melt as well as bottom melt visually.

ok, ill add a class variable Th(ice)[installed] on a per buoy basis. This will definitely clean up spurious data
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 09, 2020, 01:51:17 PM
Not sure what to suggest. Perhaps it will become clearer once we see some top melt.
Thanks Simon, I hope it's still fun.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 09, 2020, 04:08:07 PM
Python is a joy to write (when I get the chance).

Will distribute a code update soon.

PS Amazon gave me that $300. I'm suspicious it might be an entrapment to get me hooked. Dont look a gift horse in the mouth? A Trojan horse perhaps
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 12, 2020, 10:39:10 AM
Hate to say this, so decided to wait a few days before I did.

I really cant see a way of getting th(ocean) anymore. Uniquorn I believe if you run the code on your end youll see what i mean. The temps at the ocean:ice interface are just too variable now.

That being said, if anyone has any data they would like to see graphed, let me know
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 13, 2020, 08:58:29 PM
It's surprised me how shallow (https://forum.arctic-sea-ice.net/index.php/topic,2906.msg268474.html#msg268474) the gradient is.
Much easier to define the points by eye than by maths.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 13, 2020, 09:34:00 PM
another way of looking at it.
This method makes it easier to spot snow and ice thickness. click to run
air where the lines diverge on the left.
Snow or melt ponds?  T56, T68 and T75 reach +3C close to ice surface.
ice is the thickening band in the middle. The thickness is calculated from the number of thermistors along the x-axis. Multiply by 2.
ocean on the right.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on June 14, 2020, 08:39:31 AM
Isn't T58's data indicative of a deepening meltpond?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 14, 2020, 02:32:24 PM
I'm not sure. If so that is very rapid melt. jun8-14
Or is that what 'phase change' looks like?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on June 14, 2020, 03:02:47 PM
Yeah I didn't notice the numbers. 80cm of top melt in 4 days is impossible. But once the ice reaches 0C I think melt progress should be rapid nonetheless.
I still wonder how 0C can propagate so deeply if the ice is solid, especially considering it's supposed to be somewhat salty. Above my pay grade unfortunately.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 14, 2020, 04:54:04 PM
t58 since oct2, no traumatic events, just thickening. These aren't the melting profiles I was expecting. Maybe jdallen is watching.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 14, 2020, 10:48:41 PM
Yeah I didn't notice the numbers. 80cm of top melt in 4 days is impossible. But once the ice reaches 0C I think melt progress should be rapid nonetheless.
I still wonder how 0C can propagate so deeply if the ice is solid, especially considering it's supposed to be somewhat salty. Above my pay grade unfortunately.
It could be 'flushing'
https://www.cambridge.org/core/journals/annals-of-glaciology/article/on-the-thermodynamics-of-melting-sea-ice-versus-melting-freshwater-ice/B88EBD53A0209295EA0662825C48EBAE/core-reader
Quote
The simulations show clearly that after the onset of surface warming, flushing sets in, which transports cold and salty brine from higher up in the ice to the region close to the ice-ocean interface.
Still frozen, fresher ice.     or something else completely.
Quote
One final notable feature related to the evolution of internal temperature in our experiments is the signature of a localized flushing event (ellipse in Fig. 5a): Around 25 hours after we raised the air temperature, the ice temperature increased strongly and subsequently decreased again in the upper two-thirds of the ice within a few hours. The measured temperature reaches 0°C and even higher. The sudden onset and short duration of this event point towards a localized flushing event, where penetrating comparably fresh meltwater from the surface desalinates a small region of ice quickly. The rapid desalination leads to a short release of latent heat, as a fraction of the percolated water freezes in order to reach thermal equilibrium. Once the desalination ends, much of the percolated freshwater refreezes in the interior of the ice due to heat diffusion from the colder surrounding ice which was not desalinated by the localized flushing event.

 In our set-up, it is likely that the thermistor chain sticking in the ice supported or even induced this localized flushing event due to additional heat conduction that let the ice melt around the chain. After the flushing event the measured temperatures stay above 0°C in the upper 0.02-0.03 m of the ice, which indicates that the thermistors are no longer in contact with the ice, but instead with comparably warm meltwater. Note, however, that a similar flushing event was observed by Pringle and others (2007) in a set-up with only horizontal thermistor strings.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on June 14, 2020, 11:08:54 PM
Very interesting.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 15, 2020, 11:12:06 AM
All the mosaic temp_procs I could find back to back, jun1-15.
Much quicker for me this way.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 15, 2020, 11:27:03 AM
Very interesting, had no idea about "flushing"
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on June 15, 2020, 02:52:39 PM
https://www.essoar.org/pdfjs/10.1002/essoar.10503129.1

platelets and supercooling. A bit late for the freezing season but a good read.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: SimonF92 on June 15, 2020, 06:39:19 PM
https://www.essoar.org/pdfjs/10.1002/essoar.10503129.1

platelets and supercooling. A bit late for the freezing season but a good read.

Normally get a brain-drain reading arctic papers because of the physics, but that was a good read
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on July 03, 2020, 01:51:06 PM
Hate to say this, so decided to wait a few days before I did.
I really cant see a way of getting th(ocean) anymore. Uniquorn I believe if you run the code on your end youll see what i mean. The temps at the ocean:ice interface are just too variable now.
That being said, if anyone has any data they would like to see graphed, let me know
I had another look at the heat120 data. It's possible that as the ocean warms we might still be able to retrieve thickness data from some of the buoys. Perhaps the last few weeks has been a tricky transition period. Of course, they may not last much longer.

T67 floating in open water?
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 10, 2020, 10:11:10 PM
continuing from mosaic thread (https://forum.arctic-sea-ice.net/index.php/topic,2906.msg285771.html#msg285771)
squared up the grid and added some labels. Looking at adding some distances but tracking them is difficult as the delaunay lines move around. It is in the code somewhere though.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 14, 2020, 12:34:11 PM
t78 and t81 heat120 temps
t78 has some odd features
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 15, 2020, 04:32:35 PM
t78 and t81 temp proc
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: oren on September 15, 2020, 06:52:28 PM
Thanks for these updates uniquorn.
It appears that at least for T78 bottom melt has been halted at last, and the ice core temp has equalized and soon hopefully to be inverted.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 15, 2020, 08:15:32 PM
They were probably just spam on the mosaic thread.
@SimonF92 is it possible to take surface temperature into account when using the heat files in the algo?So that it amplifies the stdev in some way.
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 16, 2020, 12:07:02 PM
latest t78 and t81. T78 is an algo nightmare
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 16, 2020, 09:42:30 PM
example of ice/ocean interface
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 19, 2020, 10:42:08 PM
tbuoy update
Title: Re: Maximising asif skills using near real time Mosaic data
Post by: uniquorn on September 23, 2020, 09:43:09 PM
tbuoy update. The quick 'refreeze' on T78 might be the porous layer at the bottom cooling.