Support the Arctic Sea Ice Forum and Blog

Author Topic: What the Buoys are telling  (Read 943906 times)

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2000 on: December 24, 2020, 05:38:03 PM »
Regional means per day (1979-2000):
If you are still using pythagoras on LatLon for distance that will affect velocity calcs. Longitude distances get very small closer to 90°N

Ha, thats strange, I just messaged you- I think we are psychically in tune  ::)
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2001 on: December 24, 2020, 05:42:15 PM »
Haversine formula (first)

vs

Pythagoras (second)

The data is raw (unsmoothed). The Haversine data is extremely positively skewed, there are clearly many errors (I am sub-sampling about 7,000,000 data points so it will happen). But importantly its possible to make out seasonal variability in Haversine, which makes me believe it more

Also uniquorn just clearly explained why not to use Pythagoras for this

Interpretations and questions:
Can we make an interpretation?
Can we identify any predictors by looking at drift speed?
Has this already been covered before in publication?

Ill continue to work on smoothing and add 2016-2020 (they changed the data format slightly so I cant batch it)






//////////last file is smoothing applied to Central Arctic Basin haversine velocity
« Last Edit: December 24, 2020, 07:28:37 PM by SimonF92 »
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2002 on: December 26, 2020, 10:50:53 PM »
A possible increase in drift speed after 2012. Thinking about the sampling rate, a daily sample will miss any inertial oscillations and/or tidal movement. One way to avoid that would be to run a program to add 3hr drift and/or velocity to all file rows before calculating the averages. Will certainly give your computer a good work out but you'd only have to do it once.

Here looking at whoi itp113 hoping it doesn't trash its profiler on the shallower Northwind Ridge/Chukchi Plateau

edit: profiler battery looking low.
« Last Edit: December 28, 2020, 09:49:35 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2003 on: December 27, 2020, 10:19:50 PM »
In 1990 iabp7049 went down the Nares strait.
12780 has a gap in the data.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2004 on: December 28, 2020, 03:42:17 PM »
Tbuoy temperature profiles, dec1-28. Not too cold for a couple of days over the festive season at ~87N, peaking at around -1.3C on dec24, up from -36C on dec20.
edited gif to 504px
« Last Edit: December 28, 2020, 03:59:36 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2005 on: December 28, 2020, 08:18:42 PM »
With 1990's data I get less animation problems when presenting buoys from one directory at a time.
Here is the 1991 directory. A great cluster north of FJL

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2006 on: December 28, 2020, 08:29:59 PM »
1992 directory. The fjl cluster pops up in jan, data appears to end on day213.
possible that duplicated december data is causing the problems?

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2007 on: December 28, 2020, 09:04:04 PM »
1993 directory contains a lot of 1994
possibly due to 5318
« Last Edit: December 28, 2020, 09:09:08 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2008 on: December 28, 2020, 09:29:11 PM »
1994 dir. Contains 92-95
check 11252

oren

  • Moderator
  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: What the Buoys are telling
« Reply #2009 on: December 29, 2020, 03:59:46 AM »
check 11252
Quite funny.

Thanks for all these animations uniquorn.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2010 on: December 29, 2020, 12:04:39 PM »
11252 data jumps from yr93day155 to yr94day299.  gganimate interpolates with a straight line
Quote
BuoyID Year Hour   DOY      Lat          Lon
11252   93   12   155.5   72.291   -148.553
11252   93   15   155.625   72.286   -148.552
11252   93   18   155.75   72.285   -148.558
11252   94   15   299.625   74.243   -84.608
11252   94   18   299.75   74.243   -84.605
.

dir1995 It seems there is not so much movement but the data only runs from jan-july. The ani has the same number of frames so probably drift looks slower.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2011 on: December 29, 2020, 12:55:43 PM »
1996 directory
A nice cluster in northern Laptev. Might compare well with mosaic.

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2012 on: December 29, 2020, 01:08:16 PM »
Thanks for the suggestion re tidal movements uniquorn.

Chukchi and Beaufort drift velocity;

Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2013 on: December 29, 2020, 02:12:50 PM »
Curious about high drift in 2011 so jumped forward to take a look. That directory may need some quality control.
Slow animation to help identify bad buoys.
A quick, but possibly bad, way to deal with it might be to remove any buoys with drift above, say, 5km/h. iirc 3.7km/h was the max drift before reaching the Fram. There is also the issue of buoys in open water to consider.

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2014 on: December 29, 2020, 02:37:13 PM »
I filter data at

df = df[df['H_Velocity'] < 50]

So 50km per day (2.08km/h)



There is probably a lot you dont agree with in the methodology (which I completely accept), im going to provide the code to you- the thing is it will take a long long time to run (but you could just run a few years instead of all).

I just finshed adding the 2017-2020 dat files so the initial sweep is complete
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2015 on: December 29, 2020, 03:01:52 PM »
A great contribution :)
Central Arctic is the big one for me and daily will do fine for now. I'll check the older animations close up before attempting to look at 3hr velocity. Inertial oscillations may always have been present at a similar magnitude.

1997 directory
The Laptev cluster making its way to the Fram
26695 and 17987 problematic

oren

  • Moderator
  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: What the Buoys are telling
« Reply #2016 on: December 29, 2020, 04:35:03 PM »
For cases where the data results in interpolations and big jumps, perhaps a good solution would be to clean the data by changing the buoy number for each consecutive section.

For example
11252   93   12   155.5   72.291   -148.553
11252   93   15   155.625   72.286   -148.552
11252   93   18   155.75   72.285   -148.558
112520   94   15   299.625   74.243   -84.608
112520  94   18   299.75   74.243   -84.605

This would need to be done manually, but just once, and then the data files can be shared in one repository (or attached to a forum post). Renumbering can be triggered by big jumps in time as well as big jumps in space.
If you (uniq or Simon) upload a sample file or a link to one I can try to assess how work-intensive this process would be and if it can be partly automated.

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2017 on: December 29, 2020, 05:26:28 PM »
For cases where the data results in interpolations and big jumps, perhaps a good solution would be to clean the data by changing the buoy number for each consecutive section.

For example
11252   93   12   155.5   72.291   -148.553
11252   93   15   155.625   72.286   -148.552
11252   93   18   155.75   72.285   -148.558
112520   94   15   299.625   74.243   -84.608
112520  94   18   299.75   74.243   -84.605

This would need to be done manually, but just once, and then the data files can be shared in one repository (or attached to a forum post). Renumbering can be triggered by big jumps in time as well as big jumps in space.
If you (uniq or Simon) upload a sample file or a link to one I can try to assess how work-intensive this process would be and if it can be partly automated.


oren, here is a link to both the tail of the data (last 1000) and also the full dataset (full one might not load in whatever program youre using)

uniquorn, there is also my jupyter notebook file in there which i used to create the datasets (ipynb, will load in jupyter notebook). Its a mess, but cleaning it for github is going to take me a while

https://www.dropbox.com/sh/oejflia4auee7v6/AADdWKNwsdd8CveteG-1tvyka?dl=0

Here is my method of data prep, im posting a screenshot for people who cant use python but may like to comment
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

Glen Koehler

  • Grease ice
  • Posts: 931
    • View Profile
  • Liked: 738
  • Likes Given: 1413
Re: What the Buoys are telling
« Reply #2018 on: December 29, 2020, 06:13:33 PM »
RE  https://forum.arctic-sea-ice.net/index.php/topic,327.msg296846.html#msg296846

     The change in average CAB drift velocity looks like it could be a major systemic step change.  I bet the MOSAIC team wishes they had known what this chart now shows before deciding where to initially park the Polarstern to start their drift year.  But does the average value come from enough widely spaced buoys to represent the CAB overall, or is it possible that a few peripheral buoys serve as outliers to have an over-weighted influence thus distorting the CAB average? 
« Last Edit: December 30, 2020, 12:12:03 AM by Glen Koehler »
“What is at stake.... Everything, I would say." ~ Julienne Stroeve

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2019 on: December 29, 2020, 08:21:22 PM »
RE  https://forum.arctic-sea-ice.net/index.php/topic,327.msg296846.html#msg296846

     The change in average CAB drift velocity looks like it could be a major systemic step change.  I bet MOSAIC wish they had this chart before deciding where to initially park the Polarstern to start their drift year.  But does the average value come from enough widely spaced buoys to represent the CAB overall, or is it possible that a few peripheral buoys serve as outliers to have an over-weighted influence thus distorting the CAB average?

A really good point, there could be sampling bias. The quickest way to check is to determine that there is no dependence of the CAB 'upswing' due to forcing from either lat or lon coordinates.

It appears that buoys deployed post 2016 were deployed in roughly the same pattern as before. Some assumptions of that regression concern me but just as a visual it looks pretty reasonable. Its interesting that after 2011 there is more variability in the latitude coordinates.

I know this doesnt fully address your comment about outliers, so I am thinking about it.

I did attempt to split the CAB into quadrants but I felt the data lost a bit of power.

Given the observation and your comment, ill split it into quadrants and have another look.
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2020 on: December 29, 2020, 08:59:12 PM »
For cases where the data results in interpolations and big jumps, perhaps a good solution would be to clean the data by changing the buoy number for each consecutive section.

For example
11252   93   12   155.5   72.291   -148.553
11252   93   15   155.625   72.286   -148.552
11252   93   18   155.75   72.285   -148.558
112520   94   15   299.625   74.243   -84.608
112520  94   18   299.75   74.243   -84.605

This would need to be done manually, but just once, and then the data files can be shared in one repository (or attached to a forum post). Renumbering can be triggered by big jumps in time as well as big jumps in space.
If you (uniq or Simon) upload a sample file or a link to one I can try to assess how work-intensive this process would be and if it can be partly automated.
3hourly data is about 330MB uncompressed. I suppose this online show and tell is one way of compiling a list of bad buoys.

@SimonF92. Had a quick look at the dataset creation code but checking it would be a bit of a diversion from the ani project + my py isn't up to it.

Here is the 1998 directory. Laptev cluster almost makes it to the fram, the beaufort cluster started further east in oct1997 and did a further 180° before heading north.

1999 directory
check for 2003 data

2000 dir
1301 takes the Nares strait.
« Last Edit: December 29, 2020, 10:12:05 PM by uniquorn »

Glen Koehler

  • Grease ice
  • Posts: 931
    • View Profile
  • Liked: 738
  • Likes Given: 1413
Re: What the Buoys are telling
« Reply #2021 on: December 30, 2020, 12:21:03 AM »
RE  The buoy travel animations.
    Over in the MOSAIC thread, A-Team questions whether the famed Beaufort Gyre (BG) is really a thing.  He includes a variety of proposed definitions of what the BG pattern supposedly is, implying that there is no real defined pattern, just a bunch of different observations that our human tendency to see patterns imagines into existence as an entity.   

   Image below is from A-Team post at [https://forum.arctic-sea-ice.net/index.php/topic,2906.msg296095.html#msg296095


     If you put all the buoy track animations on top of each other and boil them down to an average motion, do you get a set of summary arrows similar to any of those proposed BG tracks?  For example, does the net buoy travel follow something like the image below?  After watching uniquorn's animations I am more inclined to see what A-Team means by saying there really is not enough of a distinct pattern to justify naming a thing called the "Beaufort Gyre."
« Last Edit: December 30, 2020, 12:42:44 AM by Glen Koehler »
“What is at stake.... Everything, I would say." ~ Julienne Stroeve

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2022 on: December 30, 2020, 12:36:36 PM »
RE  The buoy travel animations.
    Over in the MOSAIC thread, A-Team questions whether the famed Beaufort Gyre (BG) is really a thing.  He includes a variety of proposed definitions of what the BG pattern supposedly is, implying that there is no real defined pattern, just a bunch of different observations that our human tendency to see patterns imagines into existence as an entity.   

   Image below is from A-Team post at [https://forum.arctic-sea-ice.net/index.php/topic,2906.msg296095.html#msg296095


     If you put all the buoy track animations on top of each other and boil them down to an average motion, do you get a set of summary arrows similar to any of those proposed BG tracks?  For example, does the net buoy travel follow something like the image below?  After watching uniquorn's animations I am more inclined to see what A-Team means by saying there really is not enough of a distinct pattern to justify naming a thing called the "Beaufort Gyre."

Something like this, but averages?
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2023 on: December 30, 2020, 04:48:56 PM »
Excellent. All it needs is the arrows ;) That looks a bit like the mosaic trip in yellow

Here are 2001-2004
2001 22206, 19579
2002 5315, 8063
« Last Edit: December 30, 2020, 04:56:16 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2024 on: December 30, 2020, 06:28:29 PM »
2005-2007. 2007 gets a special mention for appearing to almost complete a small gyre in one year.
2006 2415, 9115, 11247,
2007 6620

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2025 on: December 30, 2020, 07:39:00 PM »
Definitely looks like it depends on the year (and where the buoys are)
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2026 on: December 30, 2020, 09:57:23 PM »
yes.
2008-2010. TPD struggling for a while in 2010.
Took the labels out and started limiting dates to 01/01 to 01/01.
« Last Edit: December 30, 2020, 10:17:22 PM by uniquorn »

Glen Koehler

  • Grease ice
  • Posts: 931
    • View Profile
  • Liked: 738
  • Likes Given: 1413
Re: What the Buoys are telling
« Reply #2027 on: December 31, 2020, 04:26:45 AM »
Something like this, but averages?
Definitely looks like it depends on the year (and where the buoys are)
   Yes, those are great!  As uniquorn said, summary arrows on the color blotch summary would be perfect.  The color scaling shows motion in the CAA and Beaufort Sea areas, but if the arrows show little directionality, that doesn't fit there being much "Gyreocity".  My guess is that there will be some directionality in the expected directions, but the rather scattered and low velocities indicated by the large amount of purplish (and thus near zero) coloration is not impressively coherent.  The name "Beaufort Gyre" gives the impression of some vast coherent whirlpool, even if it takes 3 or 4 years to make loop.  The velocity color summary does not look very organized or coherent, or at least not enough to deserve a proper name as if it were some beast prowling the Arctic Ocean.   
     
« Last Edit: December 31, 2020, 05:03:46 AM by Glen Koehler »
“What is at stake.... Everything, I would say." ~ Julienne Stroeve

Glen Koehler

  • Grease ice
  • Posts: 931
    • View Profile
  • Liked: 738
  • Likes Given: 1413
Re: What the Buoys are telling
« Reply #2028 on: December 31, 2020, 05:04:06 AM »
Definitely looks like it depends on the year (and where the buoys are)

     At risk of being a pain, instead of single year summaries for the buoy tracks, is it possible to show 3-year or 4-year cumulative track summaries?  Most of the annual tracks are too short to see if anything like a trip around a gyre results.  2007 and 2009 show the closest match to what a presumed Beaufort Gyre would look like, but the others are just segments.
 
     My visual impression from the excellent work by uniquorn and SimonF92 to show "where the buoy's are"  (~~ doop doo doo-wah!~~  8)), is that the buoy tracks in the aggregate are far too variable to summarize into a consistent pattern like the flushing of a toilet bowl. (I hate to think what would result if my toilet's drainage was as feebly organized as the buoy tracks seem to be).  The Transpolar Drift and Fram Export show up in the buoy travel paths loud and clear, but not the famed BG. 

     The observation that ASI would migrate around in the Pacific side of the ocean for multiple years (or at least it used to before what were ice growth nursery areas became summer melting zones) all the while building thickness across those years can hold true without needing to invent a thing called the Beaufort Gyre to concretize that process into a specific travel pattern that does not exist.  Thus the resulting effect from a Beaufort Gyre type pattern as an ice accumulation process seems like an accurate concept.  But as for the existence of the Gyre itself, I don't see it.  I may have to retract my chiding A-Team about being too fussy about terminology.  I agree with A-Team, either show up with data to define what you mean by the Gyre or stop talking about it like it's a thing.  Because until there are data, there is no there, there. 
 
     It looks to me like the term Beaufort Gyre should be replaced by the less catchy but possibly more accurate term "Pacific sector of the larger Transpolar Drift pattern".  I mean I think there is a loose pattern of ASI moving from the Beaufort towards the Chukchi and then towards the ESS and Laptev and then northward.  But to say that those buoys moving like drunken ants are dancing "The Beaufort Gyre" as if it was the Viennese Waltz, or as if they were the Ohio State University marching band creating elegant choreography on the football field at half-time, credits them with way more organizational skill than they deserve.

     The 2020 track summary does seem to indicate that the MOSAIC expedition planning team just happened to pick the one year to drift with the ice when the 'Fram Export' became the 'Fram Freight Train Express' and so got spit out the other side a lot faster than expected.  Or maybe they picked the first year of the new Arctic ice transport pattern that is the new normal from now into the future (for as long as there is ASI to drift).

     More dumb questions:  In some of the post-2001-2010 animations there are buoy positions in the summer in areas of the ESS/Laptev that seem likely to have melted out by those dates.  You may have explained this elsewhere, but are all the tracks supposed to be ice-embedded buoys?  Or are some of the buoy tracks from units floating in open water and thus subject to an entirely different set of influences from ocean and/or wind currents?  As Oren pointed out, the famed (but fading fast in my eyes) Beaufort Gyre only refers to ice pack movement, right?

     Even dumber perhaps:  What is going on with the tracks that zoom off in a straight line at an absurd speed?  Are there aliens stealing our ice buoys?  In doing a summary, it would be good to filter those buoys out of any average track values. 

     Thanks for all the work you guys have done on this (esp. uniquorn who has been beating this drum for as long as I can remember).   It looks like it must take a lot mouse clicks and squinting at rows of numbers on the screen to create these excellent visualizations that the rest of us get to enjoy and learn from.
« Last Edit: December 31, 2020, 05:57:06 AM by Glen Koehler »
“What is at stake.... Everything, I would say." ~ Julienne Stroeve

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2029 on: December 31, 2020, 10:28:36 AM »
If i tried to comment on the who-what-when-where-why of Arctic currents i probably wouldn't contribute anything of value so i wont!

I just hope I am able to supplement uniquorns tenacious work in this thread by being somewhat of a code-monkey. I am genuinely amazed by how much they do on this subject.

Adding arrows means creating vectors from what is generally erratic motion unless filtered/smoothed, so very difficult- but i have an idea for a work around.

When it comes to averages, i tried initially to use a KMeans machine learning algorithm to compute 1000 centroids (or average locations) for 1979-2020 but it didnt come out nicely due to compression of coordinates toward the pole.There was a clear mathematical problem which wasnt easily fixed even when i changed the parameters.

In terms of erratic lat/lon swings, i mask lat/lons which are +4 absolute zscore from mean and then replace those values with interpolated values- hopefully creating a modicum of error correction
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2030 on: December 31, 2020, 10:55:52 AM »
I think this is a pretty reasonable way to represent direction, it is overwhelmingly clockwise when the gyre can be observed

By the way, im no longer smoothing the data- these are raw coordinates

« Last Edit: December 31, 2020, 11:12:39 AM by SimonF92 »
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2031 on: December 31, 2020, 11:14:53 AM »
Here is that as an avi so people can pause

I apologise about the '91 etc issue- the order is good after 1979

The big streak on 2020 is the Mosaic project :)
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2032 on: December 31, 2020, 01:43:29 PM »
The first thing I tried was loading all the data at once expecting a nice progressive pattern of buoy paths. What I got was a mess of lines. Separation by colouring years differently helped a little but the data needs careful attention. Examining the directories one at a time is relatively clean and reasonably quick. After all, there are only a few people interested enough to watch the animations.
Eventually I will try to put them all together if it looks like it would tell us anything. My interest is more in the detailed movements, inertial oscillations, tides. Osi-saf already does tracking, though not back to 1979.

@SimonF92, conversion to utm.lat/lon might help your centroid problem
« Last Edit: January 01, 2021, 11:24:07 PM by uniquorn »

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2033 on: December 31, 2020, 02:01:42 PM »
Very interesting! So apart from 2007, little buoy drift support for a gyre over the last 41 years? The transpolar drift also seems hard to depict accurately in a single graphic. Note the common anti-cyclonic wind pattern from a large high pressure feature favors ice drift up the Alaskan coast (Beaufort Arm) and down the eurasian side (transpolar drift).

The mp4 format also allows good user control over speed, looping, and back-and-forth within the forum tab. Cropping the ROI greatly reduces file size too. The product below was rotated for compatibility with satellite time series such as Ascat which are 'Greenland down'. Good idea to attach the avi as it is hard to recover good frames after lossy mp4 compression.

https://cloudconvert.com/avi-to-mp4
« Last Edit: December 31, 2020, 02:23:40 PM by A-Team »

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2034 on: December 31, 2020, 07:50:11 PM »
Thanks for the tip re coordinates uniquorn, those zipped text files, how were they zipped? I think they are still compressed when i go to open

I will keep in mind what you said about formats A-team, I am going to correct that dates issue to hopefully finalise something so will update the formats and make Greenland down.
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2035 on: January 01, 2021, 05:37:44 PM »
Quote
hopefully can finalize mid-90's, update the formats and standardize map orientation
Amazing to me how many sites still refuse to cooperate with NSIDC, NASA and AWI/Hamburg/Bremen. These +45, -45, -135 rotations cannot be done as matrix transpositions in the manner of n90 and so alignment inevitably degrades data quality. About 2/3 of the archives are still refusing to provide the underlying numerical netCDFs at all or as open access or in a Panoply-compatible format, though it is just a single unix command line for the source. NOAA-PSL and Climate Reanalyzer won't even use the two agreed-upon projections, polar stereographic 70º and equal area azimuthal EASE2. There are no online tools to interconvert (lacking Geo2D netCDFs); scales and ESPG projection parameters aren't even given.

Climate emergency hasn't been enough to induce cooperation across silos.

The neXtSIM archive currently runs 860 days from 01 Nov 2018 to 07 Jan 2021. As a gif, this runs to 250MB (after cropping down from 2.1GB to the Arctic Basin), still 50x too large for the forum though handled seamlessly at any playback speed on any platform by ImageJ freeware, a thirty second download-install that few here will do. Saved as an avi, the file size drops to 12MB which converts to a forum-friendly 4MB mp4.

The CMEMS neXtSIM archive shows Arctic Ocean sea ice motion very clearly and likely quite accurately given daily satellite assimilation and developed model. Promised, with no firm timeline, is neXtSIM imagery back into the early satellite record.

That will resolve a huge issue for us: continuity of ice feature tracking during the cloudy summer when so many of the visible, infrared and microwave products are incapacitated by moisture artifacts or lack of coverage. Articles generally cite DMSP SSM/I-SSMIS for summer months but that 25x25 km data, at least at NSIDC, is not available anywhere as an image archive.

https://nsidc.org/data/ssmr_ssmi/news.html
https://nsidc.org/ancillary-pages/smmr-ssmi-ssmis-sensors

Below is the entire current record from CMEMS neXtSIM. It provides two complete summers, three Nov-Dec fall freeze-ups, the entire Polarstern year, and seven days of marginally informative forward projection. This is a good resource for characterizing TransPolar Drift variation and Beaufort Arm development (no gyre occurs nor is in formation).

On the technical side, it is difficult to simultaneously optimize display of FYI and MYI. Here the palette squeeze, originally on grayscale, was 0 to 3m. It might be feasible to run the animation request twice, at 0-1m and 1-3m and overlay since we are not interested in actual Topaz4 ice thickness numbers, any more than Ascat sigma-0 brightnesses. As noted previously, the Lincoln Sea is modelled way thicker than it actually is; to accommodate its value would seriously degrade the contrast range of the rest of the ice.
« Last Edit: January 01, 2021, 06:02:44 PM by A-Team »

interstitial

  • Young ice
  • Posts: 2867
    • View Profile
  • Liked: 567
  • Likes Given: 96
Re: What the Buoys are telling
« Reply #2036 on: January 01, 2021, 11:36:05 PM »
I do not follow this thread/subject very closely. From the buoy tracks it doesn't look like there is a gyre. I wonder if their is a current but ice movement can be affected by wind and congestion in addition to currents. The return half circle if it exists has more ice and the ice is thicker and more congested. Just a thought.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2037 on: January 02, 2021, 12:17:45 PM »
Quote
ice movement can be affected by wind and congestion in addition to currents. The return half circle if it exists has more ice and the ice is thicker and more congested
That's correct. The ice is largely driven by surface winds acting via aerodynamic drag on ice roughness, notably pressure ridges and freeboard edges. That goes as the square or even cube of wind speed. Current acts by hydrodynamic drag on keels but these are soon ablated.

Since wind speeds on synoptic scale cyclonic air masses are much higher on the rotating periphery, we see the fastest ice motion there on satellite time series and buoys (eg TPD, Beaufort Arm, Ellesmere to Banks, Lincoln Sea to Fram). The ice is largely stagnant in the interior CAB and, somewhat mysteriously, in the ESS and Laptev inside the islands.

As you say, conservation of mass would cause a huge pile-up as this faster ice slowed, just like on a freeway, not observed.  A sharp boundary arc between stagnant ice and returning gyre would be visible, again observable but not observed. Since the pivotal year 2007, so much open water develops in the southern Chukchi and Beaufort that floes can't make it around, along the lines of Big Block a couple years back.

Arctic oceanography -- water properties at depth -- has been very difficult to conduct because of inhospitable access and seasonally thick ice cover. About all they can get from satellites is altimetry (higher sea level over fresher less dense water bodies). Moorings have been placed but at a density of 1-2 per million sq km; there are 3 in the Beaufort. These do not have sonar modems and must be retrieved annually by ship for download.

The US currently has zero icebreaker capability; Norway stepped in this year and was just barely able to recover data from the SIO 1-3 moorings, one of which surfaced under the ice cover and had to be located by a diver and the ice overhead cut out from above.

https://www.nersc.no/news/all-moorings-beaufort-sea-rescued-under-extreme-conditions
Quote
i don't follow buoy thread very closely
Buoys have proven to be a real can of worms yet they're about all that's out there year-round. They have constant issues with malfunction: damage from ice movement and bears, inexplicable reporting excursions, drift out of the ROI, melting out, bottom dragging of cables and battery failure. IABP does archive their data in nrt but seemingly lacks any budget for curation or annotation.

What you are seeing here from Uniquorn (and SimonF92) is highly original work that is gradually mainstreaming decades of display neglect and lack of data integration. Again, this reflects  institutional siloing and journal insistence on print, ie refusal to display time series other than postage stamp .mov (in separated supplemental, not peer reviewed).

There can be legitimate scientific questions with satellite time series, person A seeing such-and-such but person B seeing au contraire. Most of this can be resolved with arrow and color overlays so both are looking at the same thing. This type of editing is far easier done in avi than mp4.

Vox posted the new MIT study of Arctic Ocean eddies on a different forum two weeks ago; the animation of 2011-14 water circulation at two depths is really worth viewing. The authors find no water gyre in the top 50 meters because of frictional suppression by stratification and the ice bottom. At moderate depths, mesoscale clockwise water circulation is observed.

https://forum.arctic-sea-ice.net/index.php/topic,596.msg295858.html#msg295858
http://www.mit.edu/~mgl/pdf/meneghello2019baroclinic.pdf   open access


« Last Edit: January 02, 2021, 12:30:35 PM by A-Team »

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2038 on: January 02, 2021, 06:43:04 PM »
Uniq et al have been able to effectively color buoy tracks by speed, with direction implicit as tangents to the displacement line. This is a lot easier than calculating velocities from persistent features in Ascat pairs. Buoy tracks are just one dimensional though so it takes a lot of them to get an overall coherent picture of ice movement even if delaunay triangulation is feasible.

OsiSaf provides Arctic-wide velocities, though these are 48-hour vs 1-hour for buoys. neXtSIM provides separate x,y velocity maps over its ice thickness animation without really explaining the axes nor providing any means to combine them into vectors (though this feature has be requested by T. Lavergne).

The two animations below look at the x,y coordinates in a split palette as provided by Cmems/Lobelia. Here the reddish color is positive and the bluish negative. It seems that the Greenwich meridian and north pole may define the x axis, with the y being orthogonal in flattened view.

The second mp4 overlays OsiSaf arrows in an effort to make some sense of the Cmems product as currently provided. The avi for that is attached for those wanting to study individual frames.

The third mp4 tests the hourly setting at the CMEMS gif generator. The 73 frames are run at 20 fps, much faster than the mp4 above. Here they need a menu option for 3,4, 6, and 12 hours.

Technical note: The 'interleave' setting on 'stacks' --> 'tools' in ImageJ is key to interdigitating pairs of stacks, followed by doubling down on montage width framing.
« Last Edit: January 02, 2021, 08:07:06 PM by A-Team »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2039 on: January 02, 2021, 08:42:23 PM »
That's good. Arguably colours work better than arrow length with a moving image, particularly when the arrows overlap. Colours on their own are tricky to interpret though without a direction indicator. The two combined are best.

A quick look at whoi itp121. Here comparing temperature and density profiles from 7m-250m. In particular looking to see if there was a density layer at around 200m that matched temperature. It seems not. Wondering if there was detectable wave motion in the pacific layer.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2040 on: January 02, 2021, 08:52:38 PM »
IABP buoys 2012-2014.

interstitial

  • Young ice
  • Posts: 2867
    • View Profile
  • Liked: 567
  • Likes Given: 96
Re: What the Buoys are telling
« Reply #2041 on: January 03, 2021, 06:40:35 AM »
Thanks A team for the follow up. Thanks to everyone who contributes I know it can take a while to put some of this stuff together.

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2042 on: January 03, 2021, 12:01:41 PM »
Quote
colours work better than arrow length with a moving image, particularly when the arrows overlap. Colours on their own are tricky to interpret though without a direction indicator. The two combined are best.
Right. Those colors are supposed to be the vector magnitudes. In theory we could make those from the ice velocity components at Cmems since what is needed, square root (sum of squares), is just a series of commands available at ImageJ -> Process -> Math that could be applied to tile-ups of the two [0,255] grayscale component outputs.

Gimp provides convenient gridding at any scale, down to every second pixel in Filters -> Render -> Pattern. Only Panoply can draw the arrows with any control over appearance. For that, the velocity component files have to Geo2D objects in netCDF format.

Because of modular arithmetic issues in graphics, vector magnitudes are better done at the numeric matrix level, as it would be from a netCDF in Panoply. It does not appear that Cmems, Lobelia and NeXtSIM have described how the graphics are generated, much less made the underlying data available. That's ok for work-in-progress.

OsiSaf uses a 62x62 km grid to attach the base of its 6x arrows. The display is not entirely satisfactory with arrow width too thick, tips too complicated and piling up and crossing over where velocities are too high. There is no mechanism to remove erroneous outliers, change the 6x displacements, show magnitude zones by color, or even isolate the arrow layer.

If their feature movement pair from AI isn't near a grid point, it has to be interpolated over. The
AI does not necessarily follow the same features the next day. This means ice motion say over a week cannot be calculated accurately by vector addition over successive days. With neXtSIM, it's set up to drape velocity magnitude colors over the thickness display (fig.2) which is kept on track by daily satellite assimilation.

There's an easy trick for adding magnitude colors to Osisaf, reminiscent of difference-of-gaussian sharpening (or low band pass fourier filtering). The display is locally darker in proportion to vector lengths. Thus a fairly high level of gaussian blur, say 30 RLE in the Gimp tool, blurs out arrows into an appropriate grayscale layer that can be colored with a LUT over in ImageJ and lowered in opacity back in Gimp to let an arrow layer show through (fig.1) , thus giving both which as you say is more effective in moving time series (fig.3).
« Last Edit: January 03, 2021, 05:11:49 PM by A-Team »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2043 on: January 03, 2021, 12:07:44 PM »
Ingenious method.

Finishing off the 3hour data with 2015 and 2016.
These animations should be seen as a very rough guide from raw data. In many years the buoys don't appear to continue their paths through december to the following january so this method may be losing some data. Will take a look at the full data directories from 2011-2019 before doing any quality control. If anyone else feels like checking some individual buoy start and end dates, please don't hesitate. https://iabp.apl.uw.edu/Data_Products/BUOY_DATA/

well here is a closer look at IABP1920 from 1979.  Quite a bit slower than mosaic.
« Last Edit: January 03, 2021, 02:17:00 PM by uniquorn »

SimonF92

  • Grease ice
  • Posts: 592
    • View Profile
  • Liked: 210
  • Likes Given: 87
Re: What the Buoys are telling
« Reply #2044 on: January 03, 2021, 04:20:08 PM »
Hats off to both of you for the amazing work

I would love to see an overlay between buoy and satellite data to see how well they correlate. Easier said than done i know :).

Also at the expensive of seeming stupid A-team, how can there be negative velocity in the figure? Is velocity relative/normalised to something?
Bunch of small python Arctic Apps:
https://github.com/SimonF92/Arctic

A-Team

  • Young ice
  • Posts: 2977
    • View Profile
  • Liked: 944
  • Likes Given: 35
Re: What the Buoys are telling
« Reply #2045 on: January 03, 2021, 05:42:51 PM »
Quote
can there be negative velocity in the figure? Is velocity relative/normalised to something?
Picture an ordinary x,y cartesian coordinates centered on the north pole with the y axis the prime meridian and the x axis the 90th. Then the x coordinate will be positive to Eurasian side and negative to the Alaskan. The y coordinate is positive towards Siberia and negative towards Svalbard.  The velocity magnitude (speed) will always be non-negative as a square root. (I'm just guessing as neXtSIM/Cmems/Lobelia don't actually provide any explanation.)

Looking at another example of ice velocity display, NOAA-PSL has rather nicer arrows than OsiSaf but presents 6-hour ice motion in a peculiar projection incompatible with other resources with perhaps too many unrelated colors in an inappropriate paired palette. They should also give some thought to a monospace font and leading zeroes so their dates don't jump all over. The size is quite satisfactory at 1024x1024.

NoAA-PSL's web site is structured so as not to allow direct downloads or even shared links to specific animations but tediously opening a separate browser tab for each of 28 built frame links of png, loading in ImageJ, cropping away or whiting out the copious mispositioned junk, then optionally changing the palette one square at a time, then making the avi and mp4 (as the forum does not support modern png animations).

It isn't feasible to compare this product with other sea ice velocity products because there's no way to rectify the projection without the underlying data. It's a forecast but seemingly without follow-up reanalysis or post-forecast re-initialization from transpired GFS. There is an archive of 155MB tar.gz; I only looked at a smaller version which did not contain ice speed stills or animations. Visually this is a nice example of what works, what doesn't in an ice velocity product

https://psl.noaa.gov/forecasts/seaice/img/figs/fig16/list.jsani
psl.noaa.gov/forecasts/seaice/
« Last Edit: January 03, 2021, 05:50:13 PM by A-Team »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2046 on: January 05, 2021, 12:46:37 PM »
drift track and ~6m-200m temperature and salinity profile contours for selected whoi itp's. The criteria for selection was that the buoys should drift close to or within the BGOS buoy moorings (white circles) and have a reasonably long profile. Some fail quickly. Note that there are a couple of scale changes early on and that often the drift track is longer than the profiler battery life (or other reason for failure). So, as nearly always, this is just a very rough historical guide over a small area in the Beaufort from 2006-2021
White can often be interpreted as warmer than top of the scale (off the chart) but may also be due to missing data. For itp121 white means off the chart. Found a newer version of itp5 temp.
« Last Edit: January 05, 2021, 12:56:09 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2047 on: January 27, 2021, 12:18:05 AM »
Slowly but surely making progress with animations spanning different years. Here is 20200921-20210126, roughly 1week per second.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: What the Buoys are telling
« Reply #2048 on: January 27, 2021, 10:30:05 PM »
Beaufort buoys jan2016-jan2021

oren

  • Moderator
  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: What the Buoys are telling
« Reply #2049 on: January 28, 2021, 01:58:38 PM »
Great animation. While looking at it, the concept of a "Beaufort Gyre" seems more absurd than ever. Sometimes the buoys make a concerted effort to drift clockwise, but most of the time they just go every which way. And even when they do  follow a pattern, it's not the same pattern every time, the directions are different.