Author Topic: Maximising asif skills using near real time data (Read 40470 times)

uniquorn · « **on:** March 03, 2020, 11:06:18 PM »

This is probably a once in a decade opportunity to use the mosaic project to gather almost unprecedented near real time arctic data. Perhaps you are content to sit back and watch a model. Maybe a view from a satellite. Maybe just wait for someone else to do it. But why? There are over 70 active buoys in the arctic today(Mar3). Why aren't we analysing them?? If we are. Why aren't we sharing??

uniquorn · « **Reply #1 on:** March 03, 2020, 11:13:03 PM »

Step1. This thread goes probably nowhere unless someone posts a Polarstern (the biggest buoy) temperature chart. Simple spreadsheet skills and probably some data preparation required.
data here, scroll to the bottom of the page and click on track history
Please don't be shy

5 charts is better than 0
If converting text to csv is too tricky. Try T62 from meereisportal.de using T1 as the surface temperature.
Open source spreadsheet

psymmo7 · « **Reply #2 on:** March 04, 2020, 12:37:03 AM »

I've done a "dump all track history" which dumps everything back to 2016 and am in the process of extracting MOSAIC relevant data and converting it to a spread sheet. Date and time aren't delimited so needs a bit of hands on work to get it fully analysable. Will probably take me a couple of days. Will post again when done.

uniquorn · « **Reply #3 on:** March 04, 2020, 12:47:26 AM »

thank you. From oct2019 will correlate with mosaic.

Ideally one asif member could monitor one each of the 16 Tbuoys which measure thickness

and melt not being funny but you know how asif is massively more interested during the melt season

uniquorn · « **Reply #4 on:** March 04, 2020, 11:38:22 AM »

Quote

Date and time aren't delimited

Possible workaround
Copy and paste into notepad (or text editor with no formatting)
Edit > Replace "(double quotes) with blank space to delete them
Save as test.csv
Open with spreadsheet (libreoffice calc, excel....) with space as separator

uniquorn · « **Reply #5 on:** March 04, 2020, 02:09:43 PM »

So you downloaded the 'free' open source LibreOffice and installed it. Downloaded the T62 file. Opened it with calc, selecting comma as the separator and you got this.
A datetime, lat, lon and 241 thermistor temperatures.

Take a moment to locate the 'Insert Chart' button, top right.

uniquorn · « **Reply #6 on:** March 04, 2020, 02:17:26 PM »

Select row1 by clicking on the row number on the left
Scroll down to the bottom row with data
Hold down the control key on the keyboard and select the bottom row number (510 today)
Click the chart button
Enjoy looking at this morning's data.
Play with the chart settings

Note: This quick chart will include the lat/lon data. Exlude those cells as you gain experience

uniquorn · « **Reply #7 on:** March 04, 2020, 03:14:38 PM »

Select columns A and D for a near surface temperature chart
Play with the chart settings.

uniquorn · « **Reply #8 on:** March 04, 2020, 03:36:28 PM »

uniquorn · « **Reply #9 on:** March 05, 2020, 01:26:49 PM »

Reposting these animated Tbuoy charts from the Mosaic thread we can approximately identify which thermistor is closest to air/snow interface. During 3mths the air/snow thermistor number (Tas) doesn't change much*. So an approximation of ice/snow thickness can be made by counting the number of thermistors above the ice/ocean temperature and subtracting Tas

We normally take ocean temperature under ice to be -1.8C but looking at a random section of T62 data at the ice/ocean interface it is clearly more complicated.

*except for T69 which has experienced some kind of trauma.

uniquorn · « **Reply #10 on:** March 05, 2020, 01:52:19 PM »

Latest surface temps on T62 are nice and cold so it should be easy to find the air/snow thermistor.
It's probably T39 but let's say T40 to be sure. Checking further up the data it looks like we may have 4cm more snow since November. Hopefully some bright enthusiast will deal with that later.
So Tas=40

uniquorn · « **Reply #11 on:** March 05, 2020, 02:17:24 PM »

The kind people at libreoffice.org have implemented COUNTIF which allows us to count the number of cells in a row with a lower temperature than our ice/ocean interface. I'm going to set that at -2.1 and let the wordsmiths on this forum discuss in detail what it should be.

Cell formula will be =COUNTIF(D2:II2,"<-2.1")
We omit column IJ. I think it is some sort of end of line check.

Now subtract Tas=40
Cell formula will be =COUNTIF(D2:II2,"<-2.1")-40

multiply by 2 because the thermistors are 2cm apart.
Cell formula will be =(COUNTIF(D2:II2,"<-2.1")-40)*2

Label your estimate column, select the cell beneath it and click F2 on the keyboard.
Paste in the formula and click Enter on the kbd

edit: I found some -2.06 ocean temperatures on thermistors T203 and T207 (in fact, all over the place) so the ice/ocean interface temperature has been dropped to -2.1C Interesting to see if they turn up on other Tbuoys
Tip: Select cell B2 > View > Freeze rows and columns

uniquorn · « **Reply #12 on:** March 05, 2020, 02:42:14 PM »

Now we need to populate the whole column.
Select your new formula cell.
Click on the small black square at the bottom right of the cell and drag it down.
When you release the mouse button the cells will be populated with the correct formula.
Ensure that all the cells to the bottom row of the data are populated

Now select columnA
Press the control key on the keyboard and select your new column (IL in my case)
Click the 'Insert Chart' button as before.
Enjoy your SIT estimate

uniquorn · « **Reply #13 on:** March 05, 2020, 02:47:36 PM »

Daily Tbuoy location chart is here

Last note: The gradient change for snow on T62 is pretty clearly at around thermistor50 at the moment so we are looking at ~20cm of snow.

SimonF92 · « **Reply #14 on:** March 12, 2020, 05:41:38 PM »

I am a big fan of this idea and I have pretty deep Python knowledge so when it comes to parsing data or automating this in some way I am happy to help.

I have to ask though, when the temperature difference between ice and air normalises to 0 in the coming months, do you think it is possible to account for this in your analysis? Would it be better to choose a thermistor where the temperature no longer deviates between hour-to-hour recordings as the point at which there is "ice". Im assuming ambient air temp will be much more variable than the temperature within the ice, over short time periods.

uniquorn · « **Reply #15 on:** March 12, 2020, 07:37:57 PM »

Quote

choose a thermistor where the temperature no longer deviates between hour-to-hour

I'm not sure there is one. The method above may be adequate for estimating the nearest ice/water thermistor for thickening and bottom melt until surface melt begins. Perhaps once the surface snow is gone the nearest air/ice thermistor will become clearer and easier to locate mathematically with temperature difference.
If you know how to calculate an average gradient change over several rows that may help.

I don't know what the surface numbers will look like as air temperatures rise but the T56 data from nov11-22 may give us an idea

SimonF92 · « **Reply #16 on:** March 16, 2020, 10:42:59 AM »

Quote from: uniquorn on March 12, 2020, 07:37:57 PM

Quote
choose a thermistor where the temperature no longer deviates between hour-to-hour
I'm not sure there is one. The method above may be adequate for estimating the nearest ice/water thermistor for thickening and bottom melt until surface melt begins. Perhaps once the surface snow is gone the nearest air/ice thermistor will become clearer and easier to locate mathematically with temperature difference.
If you know how to calculate an average gradient change over several rows that may help.

I don't know what the surface numbers will look like as air temperatures rise but the T56 data from nov11-22 may give us an idea

Thanks for the advice Uniquorn, I will look into this.

SimonF92 · « **Reply #17 on:** March 16, 2020, 02:01:21 PM »

I plotted the standard-deviation of temperature over a 10-hour period for (most recent data) for T56.

In the 2nd plot I plotted the change in stdev between adjacent thermistors

At the moment I think its pretty obvious from both plots the air(snow):ice boundary is around T25. In theory this way should overcome limitations which might begin to exist when there is less of a temperature difference between the ice and the air.

If you think this makes sense I will begin to work on a simple algorithm.

Hopefully we are not talking to cross-purposes at the moment!

SimonF92 · « **Reply #18 on:** March 16, 2020, 02:15:46 PM »

Algorithm would be written from the centre of that U-curve to the temp flattening (yellow)

Might be better to have an estimation between the margins of that U

SimonF92 · « **Reply #19 on:** March 16, 2020, 02:16:38 PM »

Ignore the labels

X and Z are swapped

uniquorn · « **Reply #20 on:** March 16, 2020, 04:46:45 PM »

Quote from: SimonF92 on March 16, 2020, 02:01:21 PM

I plotted the standard-deviation of temperature over a 10-hour period for (most recent data) for T56.
<>
Hopefully we are not talking to cross-purposes at the moment!

That looks a lot more robust than my method. Do you mean 10 rows? 1 every 6hrs.
I think we are aiming at the same thing, only the presentation will be different. I plan to use the thermistor numbers to show thickness on the drift animation
drift update, click to run. (might have less time than I thought)
removed-will update later

SimonF92 · « **Reply #21 on:** March 16, 2020, 05:24:57 PM »

Beautiful animation.

Yes, 10 rows, my apologies.

Are you doing this in R?

Ill try write a function for thickness in R's syntax if so.

uniquorn · « **Reply #22 on:** March 16, 2020, 06:28:35 PM »

Yes, R. I had to 'learn' it to hack macid's original code. Taking on python as well is a bit much.
I had a look at T64, the southernmost Tbuoy to check the ice is still thickening (at around -2°C)
Therm139 to Therm141, so ~4cm (at the bottom) over 16days.

uniquorn · « **Reply #23 on:** March 16, 2020, 08:44:18 PM »

Quote from: SimonF92 on March 16, 2020, 02:15:46 PM

Algorithm would be written from the centre of that U-curve to the temp flattening (yellow)
Might be better to have an estimation between the margins of that U

I suspect the U-curve is the snow layer so we should see that ~~disappear~~ narrow? as it melts. This might be of interest to those who like to discuss snow depth. The two margins would then be air/snow, snow/ice (ultimately becoming air/ice) though I'd need to see a time series to see how the std deviation curve moves to comment more. If identifying those margins is too problematic I'll be happy with an ice/snow thickness.
stdev and delta charts are different sizes, here scaled in an attempt to match them

SimonF92 · « **Reply #24 on:** March 17, 2020, 10:59:12 AM »

Here they are matched, im working on a time series at the moment

uniquorn · « **Reply #25 on:** March 17, 2020, 02:31:15 PM »

Quote from: SimonF92 on March 17, 2020, 10:59:12 AM

Here they are matched, im working on a time series at the moment

Great. The u-curve on the delta looks very much like the snow layer and the change at ~120 on the std deviation looks like the bottom of the ice. Hopefully confirmed by a time series. How tricky is the algorithm?

Meanwhile, a quick update on thickening.
removed

SimonF92 · « **Reply #26 on:** March 17, 2020, 03:22:58 PM »

Here is a time series of November, individual figures are made over 60-hour sampling periods.

A few of these I am not happy with, a few of them I am happy with.

If you think it best to just keep this simply about temperatures and not stdevs then I can understand that.

Let me know!

Ill put in an mp4 for anyone who likes that kind of thing, it might help to visualise what I can see on my end.

blumenkraft · « **Reply #27 on:** March 17, 2020, 03:32:48 PM »

Thanks a lot, Simon. Great work!

Allow me a critique though. Perhaps it's only me, but i can't see where these points are in the 3D space. One solution to fix this maybe would be to add shadows to them (if this is even possible).

SimonF92 · « **Reply #28 on:** March 17, 2020, 03:35:56 PM »

blumenkraft you beat me to it while i was changing the size of this animation to help people with low bandwidth

I agree its quite unclear unless you see it rendered in 3d, I will work on a way to animation the timeseries in 3d

blumenkraft · « **Reply #29 on:** March 17, 2020, 03:36:59 PM »

OMG, very cool!

SimonF92 · « **Reply #30 on:** March 17, 2020, 05:00:11 PM »

This is a current estimation of t56's data, based on the code I just wrote.

I am going to fully annotate the code and put it up here.

uniquorn · « **Reply #31 on:** March 18, 2020, 05:05:06 PM »

testing temperature chart with SimonF92 generated provisional thickness estimate
temporary post as it is large

SimonF92 · « **Reply #32 on:** March 18, 2020, 06:16:47 PM »

Im posting the main bulk working here.

It would be great to spot any errors in this at the moment and not later on. If anyone disagrees with any of the lines I will be happy to change it.

uniquorn · « **Reply #33 on:** March 18, 2020, 06:58:42 PM »

Thanks.
Is this an ice or an ice+snow thickness estimate? I'm still working my way through the coding.
Do you get the same thickness chart as above with the wild swings at the end? If so, what could cause them? I ran the temperature charts to see if they could explain it. Perhaps it is the temperature gradient flattening out. If so that may be a problem as temperatures rise.
edit:latest fomo may help identifying snow layer

uniquorn · « **Reply #34 on:** March 18, 2020, 09:55:44 PM »

T56 method comparison
eyeballing the numbers Therm-snow/ice=30 for the spreadsheet method
Spreadsheet method is less complicated, std dev+ is more mathematically justifiable
Is the difference down to snow depth?

Quote

Nag to the 17: This is probably a once in a decade opportunity to use the mosaic project to gather almost unprecedented near real time arctic data. Perhaps you are content to sit back and watch a model. Maybe a view from a satellite. Maybe just wait for someone else to do it. But why? There are over 70 active buoys in the arctic today(Mar18). Why aren't we analysing them?? If we are. Why aren't we sharing??

SimonF92 · « **Reply #35 on:** March 19, 2020, 10:26:56 AM »

Spreadsheet method is definitely better currently in my opinion but I am concerned that when temps rise it will be more difficult to pick out boundary thermistiors based on temperature gradients alone.

Thanks for making that plot, its pretty clear to me that the python method needs work.

Could the python method be underestimating when there is no snow and overestimating when there is a lot?

//edit//

Ill try to add an estimate of snow depth based on the U width, will send you the code

uniquorn · « **Reply #36 on:** March 19, 2020, 03:23:16 PM »

Comparison of the new code (thickness_and_snow_as_timeseries.py) with the basic method. Pretty close agreement on ice thickness. We have no way of verifying snow depth but looking at other buoys may show us if we are on the right track

added yesterday's S1 of the area
nearest coords, roughly middle of the crop
2020-03-18T07:00:12,86.704520,12.369056
2020-03-18T09:00:13,86.696373,12.381268

SimonF92 · « **Reply #37 on:** March 19, 2020, 04:17:45 PM »

Snow depth for T56 to T63,

.....excluding the buoys weve already figured arent working properly based on their 0 temps throughout the column.

I think you were spot on when you said that U was snow depth

FYI this is from feb 21st to present
Green is T60

SimonF92 · « **Reply #38 on:** March 19, 2020, 05:43:13 PM »

Incase anyone is wondering what we are talking about with this model.

I have to be conservative with the "U" shape because currently the code iterates to provide a reading every 6-hours, 500 iterations per buoy, 14 buoys: over 10,000 measurements inclusive of both ice and snow means lots of room for errors and over-estimations.

Sorry if the dimensions of the mp4 are a bit poor, i have scaled it down again to take up less size.

uniquorn · « **Reply #39 on:** March 20, 2020, 01:37:13 PM »

Here are the buoys charted from SimonF92 data. It's unlikely the snow layer is so variable. Probably due to the method and changing temperature. Highly likely that it is adequate for this thread, especially as the snow will be gone soon enough.
T69 may still have some value so I left it in

SimonF92 · « **Reply #40 on:** March 20, 2020, 05:49:34 PM »

Here are all the buoys. Its clear at least one is broken and one died. Its possible the code bugs at certain points too, but i think we have made pretty decent progress this week.

I have applied very gentle smoothing to the data,.

This data is from mid-november to present, measurements are made every 6 hours.

The buoys are on the move, so it might be reasonable to expect some variability. I am personally quite curious about the spread in thickness between the buoys

blumenkraft · « **Reply #41 on:** March 20, 2020, 06:13:57 PM »

So, those downward spikes are measurement errors, no?

SimonF92 · « **Reply #42 on:** March 20, 2020, 07:12:21 PM »

Quote from: blumenkraft on March 20, 2020, 06:13:57 PM

So, those downward spikes are measurement errors, no?

T57, t59 and t61 were removed as they were reporting ~0 degrees values throughout the thermistor column.

Though i havent personally checked, im pretty sure the values for t69 would be similar at those wild swings and is most likely erroneous data from hardware issues. Just left it in for discussion purposes- i havent personally checked.

Some of the other swings are probably bad code!

uniquorn · « **Reply #43 on:** March 20, 2020, 08:11:32 PM »

T69 had a trauma of some kind as noted upthread. The other drops don't show up on reply#9 animation showing bottom thickening so are probably related to the method used to detect the nearest snow/ice thermistor.
I think the 'warm snap' around feb20 may be the cause.
10 rows perhaps too short - it might be easy to edit such well documented code....

SimonF92 · « **Reply #44 on:** March 20, 2020, 09:24:22 PM »

Quote from: uniquorn on March 20, 2020, 08:11:32 PM

T69 had a trauma of some kind as noted upthread. The other drops don't show up on reply#9 animation showing bottom thickening so are probably related to the method used to detect the nearest snow/ice thermistor.
I think the 'warm snap' around feb20 may be the cause.
10 rows perhaps too short - it might be easy to edit such well documented code....

Definitely what caused it.

If we had a large training dataset I could use machine learning to estimate snow-depth, but as-is it is, this whole "U-shape" concept is bugging the code- I take responsibility for chasing that idea

Im offline for the weekend but on Monday I will write another version of the algorithm and scale back the complexity- just focus on ice.

uniquorn · « **Reply #45 on:** March 20, 2020, 09:38:59 PM »

I think the U shape is good. If you have time I'd like to see how it moves with time. I might be able to chart it in R if not.

I experimented with changing the number of rows from 10
5 rows was worse
20 rows removed the large drops but the algo is struggling to detect the ice/snow boundary so ice goes up when snow goes down.
at 50 rows other problems crept in.

So far it is good for detecting snow+ice thickness
I'll play with this line in the meantime
#select the region with large stdev snow
dfU=dfU.where(dfU.stdev_delta_mean<-0.25)

Quote

If we had a large training dataset

rofl. This is probably the biggest it's ever gonna get.

adding the Tice/water, Tsnow/ice numbers etc to the csv might help debugging, maybe I can do that?!?

uniquorn · « **Reply #46 on:** March 22, 2020, 10:27:41 AM »

Overnight thoughts: As temperatures rise to close to zero I think we will be looking at a very flat curve across the thermistor chain. More experienced minds than ours will have grappled with the problem of identifying the 'ice-air-water interfaces' and offered this heat method as a solution.
Perhaps stdev and delta will be more reliable when run on these files

Quote from: uniquorn on March 01, 2020, 01:32:15 PM

https://journals.ametsoc.org/doi/pdf/10.1175/JTECH-D-13-00058.1 2013
Quote
5. Heated operation mode
Theory of operation
The novel feature of this new IMB buoy is the development of chains that can be operated in what can be described as either a ‘‘hot-wire anemometer’’ mode or a needle-probe thermal conductivity mode. Hot-wire anemometry is a standard technique in experimental fluid dynamics and has been widely described (e.g., Perry 1982;La Barbara and Vogel 1976). Essentially a temperature-sensing element in a moving fluid is heated to above ambient temperature, and the amount of heat required to maintain a constant sensor temperature is measured. The amount of heat required depends on both the thermal characteristics of the surrounding fluid and its flow velocity. In principle, the output of the Maxim DS28EA00sensor device can be refined for the estimation of both these quantities, thus allowing determination of the position of the ice–air–water interfaces and quantification of flow speeds. The sensitivity of the present design is not yet capable of the quantification of flow speeds

SimonF92 · « **Reply #47 on:** March 23, 2020, 12:57:49 PM »

Very interesting, ill write a new version based on HEAT and we can cross reference the data with the old Thermistor data.

I suspect once again that snow might throw a spanner in the works.

Will post full working and send you the code to have a tinker with

SimonF92 · « **Reply #48 on:** March 23, 2020, 01:04:20 PM »

The cool thing is, the batch wrapper and parser will not need any tweaking at all- just the algorithm. So any new ideas can be implemented efficiently

uniquorn · « **Reply #49 on:** March 23, 2020, 01:33:24 PM »

I hoped that would be the case. Still shovelling here.

News:

Author Topic: Maximising asif skills using near real time data (Read 40470 times)

blumenkraft

blumenkraft

blumenkraft