Support the Arctic Sea Ice Forum and Blog

Author Topic: Dev library  (Read 8339 times)

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Dev library
« on: September 12, 2019, 11:48:41 AM »
Resizing table for overlays:
satellite           pxl       %      ~km/pxl    ~km^2/pxl
SMOS              397.5    75.815   11.92      5.06
Ascat              524.3   100.000    9.04      6.67
OsiSaf            1075.5   204.711    4.41     13.66
SMOS-SMAP   1146.0   218.577    4.13     14.59
AMSR2           1197.6   228.419    3.96     15.24



uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #1 on: September 12, 2019, 11:59:53 AM »
method for creating graticules from A-Team. 2 examples with mosaic start area.
Quote
tech note: to crop to a lat-lon rectangle on an image lacking a suitable graticule, draw a vertical line through the north pole on a transparent layer copy, then rotate it to provide the lon lines. fix the circle tool to 1:1, find the number of pixels to an outer lat line, reduce the circle appropriately, cut the center, enlarge the circle, cut the inverted selection, autocrop image.

formula πR² * diff in lons * diff in sin(lats)

πR² = 126916372 sq km WGS84
πR² = 127516118 sq km sphere

42,900 sq km WGS84 for two degree lat, 20 degrees long
43,103 sq km sphere  for two degree lat, 20 degrees long

64,350 sq km WGS84 for two degree lat, 30 degrees long
64,654 sq km sphere  for two degree lat, 30 degrees long

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #2 on: September 12, 2019, 10:53:49 PM »
osisaf graticule

MyACIsDying

  • Guest
Re: Dev library
« Reply #3 on: September 23, 2019, 12:08:38 AM »
R code for this animation attached. I'm practicing and much needs to be optimized, this run took about 2 hours..

Code is commented as sensibly as possible and can be run to produce the same result, 35 images, gif making done with ezgif. magick for next time.

requires 1.5GB disk space


uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #4 on: September 26, 2019, 08:30:07 PM »
Good gradient palette

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #5 on: October 06, 2019, 04:47:29 PM »
In the macid spirit of sharing. Here is the octave code for creating the images for the mosaic buoy ani here https://forum.arctic-sea-ice.net/index.php/topic,2906.msg232120.html#msg232120
(rem the first line of the buoy file or it won't open)
« Last Edit: October 06, 2019, 04:57:56 PM by uniquorn »

MyACIsDying

  • Guest
Re: Dev library
« Reply #6 on: October 06, 2019, 10:54:16 PM »
In the spirit of sharing I stole the genius of the best libraries around.. plotly does more than I could have dreamed of and exports to WebGL, takes some time to load but worth it :D

Smallest example: http://rpubs.com/macid/ITP119_TempC
check the rest on http://rpubs.com/macid/, the biggest ones have crashed my browser on occasion, would like to hear how it works for other people. Could optimize with averaging some data points if needed or other 'd0h' moments.

code updated on https://github.com/macidR/arcticPlotting/blob/master/plotly.R much quicker and awesome

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #7 on: October 06, 2019, 11:56:01 PM »
« Last Edit: October 07, 2019, 11:46:43 AM by uniquorn »

MyACIsDying

  • Guest
Re: Dev library
« Reply #8 on: October 07, 2019, 12:15:30 AM »
hmm doesn't sound like you're seeing what I see, only tested with chrome, less then GB mem use & 20 secs loading or so, data download should only be 10MB.

The controls are supposed to let you pan/zoom/orbit the thing

Sample of ITP116 attached (one of the nicest)


uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #9 on: October 07, 2019, 12:27:57 AM »
« Last Edit: October 07, 2019, 11:45:13 AM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #10 on: November 08, 2019, 11:57:50 AM »
lat/lon overlay for osi-saf

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #11 on: November 24, 2019, 07:19:11 PM »
I just know everyone is going to want to try this for themselves ;)

Steps and rationales in placing an S1 image under its buoy time frame:

rotate S1 of 05:46 22.11.19 cw 45º in ImageJ
measure scale between 85-86 at 2712.46 pixels or 813.7 pixels per 0.3 deg of lat
animation scale between 85.6 and 85.9 which is 0.3 deg of lat is 369.6 pixels
consequently the S1 needs to be downscaled by 45.42% to fit under animation frame 77 need to find 85.6 120 on the S1 to know how far to drag it.
given 85.0 and 86.0 corners, it will be 0.6 * 2712.46  = 739.2 pixels down from 85.0
add a blank layer and put a green dot there.
put a big red * over the Polarstern on the image
add frame 77 and final frame 100 overlays and co-slide to 85.6 120
autocrop to get rid of S1 that is not under frame 77.
note PS is 53 pxl left and 6 pxl down relative to buoy P204 on frame 77
for future PS put a magenta star 53 pxl left and 6 pxl down relative to buoy P204 on frame 100
sailwx is saying PS is at 85.7 120.3 on frame 100 which is at 04:30 on 11.23.19
put a orange star 27.4 pxls up from 85,7 (in a blank layer, don’t cover up data).
copy out P204’s route, color it cyan and translocate it from PS f77 to PS f100 for
that gives hypothetical intermediate drift that we can test with RAMMB
ship GPS is only 1 dp so has a large uncertainty orange box
save a new-from-visible png enlarged to ~700
 
repeat for all the frames that have a nearby S1 time and post to forum later
most of the steps are the same for all S1 and
steps vary simply if Uniq changes buoy animation scale and orientation

05:46   22 11 19
04:08   22 11 19
02:30   22 11 19
06:43   21 11 19
03:27   21 11 19
06:02   20 11 19
04:24   20 11 19
07:00   19 11 19
05:21   19 11 19

blumenkraft

  • Guest
Re: Dev library
« Reply #12 on: January 30, 2020, 12:38:54 PM »
CesiumJS - an open source JavaScript library for creating world-class 3D globes and maps with the best possible performance, precision, visual quality, and ease of use

Link >> https://cesium.com/cesiumjs/

sidd

  • First-year ice
  • Posts: 6774
    • View Profile
  • Liked: 1047
  • Likes Given: 0
Re: Dev library
« Reply #13 on: February 02, 2020, 10:14:26 PM »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #14 on: March 11, 2020, 05:09:12 PM »
ascat ready land mask from here
« Last Edit: March 11, 2020, 05:24:33 PM by uniquorn »

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #15 on: June 18, 2020, 03:31:12 PM »
May I ask uniquorn how another such image can be produced>>
A-team melting expectations from amsr2. Too busy to try it at the moment.

Quote
download selected dates and years from UHH server; these begin on 01 Aug 2012
load the dates into Gimp freeware as a stack
crop to region of interest, typically Arctic Ocean and a bit of adjacent seas
check that the non-contiguous colorpicker tool is set to radius 0
pick the deep blue of open water RGB = 0,7,135
grow the selection by 1
set the foreground color to light gray 224,224,224 and replace the blue
invert the selection and fill with black 0,0,0
duplicate image and use average command
color land mask a background color
color never open a foreground color
select land and never-open and invert selection
enhance sometimes-open grays or colorize them with blue
add graticule restricted to never-open
set up 400 pxl legend squares so that they average and colorize correctly
use histogram to compile statistics for each sometimes-open gray
(replace large interior never-open with most recent enhanced ascat)
do not resize, this will cause dithering of key colors

oren

  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: Dev library
« Reply #16 on: June 19, 2020, 09:52:09 AM »
Thank you uniquorn (and A-Team). Far above my abilities unfortunately.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #17 on: July 09, 2020, 09:07:02 PM »
possible method for extracting ice structure from amsr2-uhh during clear weather from A-Team
Quote
july 7 set to ‘difference’ (absolute value) relative to each of

7-6
7-5
7-4
7-3
7-2
7-1

followed 6 times by 'new layer from visible’
followed by tile-up, then Color —> Level to taste especially with middle gamma slider, Color Balance to taste, then moved to imagej for clahe and unsharp mask, saved to avi rather than to gif.
saved as gif

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #18 on: July 22, 2020, 05:47:35 PM »
A-team method in gimp for extrapolating possible melt using hycom thickness
Quote
mark up low values on palette better with grid tool on a transparent layer
vary radius until it picks up 0.5, 0.55, 0.6, 0.65, … 1.0
invert selection
make new layers and fill with neutral color, each time saving painted palette too
make slow animation being sure to start with unaltered. frames are nested by construction.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #19 on: August 11, 2020, 08:21:18 PM »
for reference
The Hycom thickness forecast updates every day and so needs a regular reprocessing, as it's not easy to automate. To empower more people per a suggestion of Oren, our moderator extraordinaire, the steps for doing this in Gimp are provided below. Degree of difficulty 2 in 5, ten minutes with practice.

Steps to extract the 7day forecast, enlarge(433.2%) and rotate (45º cw) it to match scale and Greenland-down orientation of AMSR2 uhh, clean up distracting datestamp and excessive palette:

download, open, change to RGB, unoptimize gif (to remove transparencies)
delete early days leaving top 7 frames (mark earliest frame, hold command-x down below it)
duplicate stack, make reusable palette in new window, deleting >3.0 m (not used) to 45x352
duplicate stack file, crop out month and day, adjust width to match palette (45)
crop stack to 631x631 data circle
rotate 45º cw about center
change from indexed to RGB
crop to region of current interest
enlarge 433.2% to match AMSR2uhh (or new AWI as that size finalizes)
enlarge canvas on palette and date to match new width.
tile thickness, palette and date, adjust view, make new layer from visible
move to imageJ and de-tile to make avi/mp4 or gif

https://www7320.nrlssc.navy.mil/GLBhycomcice1-12/navo/arcticictn_nowcast_anim30d.gif

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #20 on: August 11, 2020, 11:49:39 PM »
for ref
Quote
Oren suggested a while back it would be helpful to have an operational description of how Venn diagrams from image pairs are made. Fortunately all manner of boolean logic operations are baked right into Gimp’s color picker.Steps are simple; the trick is keeping color picker at the right setting operating on the right active layer:

open five dates before Aug 10th as a short stack in Gimp, 10th on top
set mode on each to ‘darken only’ to deprecate cloud cover artifacts
capture the view with with ‘new from visible’
delete the other dates and add Aug 01 from below.
set the color picker to desired concentration range on the embedded palette
crop stack to a non-excessive region of interest that works for both
select open water blue in the Bering 10th, change active layer to 1st, change color picker to ’subtract’, make new layer, name to avoid confusion, fill.
select open water blue on the 1st, change active layer to 10th, change color picker to ’subtract’, make new layer, name and fill with 2nd color.
set the color picker to ‘intersect’ to find open water common to both, make new layer, name and fill with 2nd color.
« Last Edit: August 11, 2020, 11:58:24 PM by uniquorn »

oren

  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: Dev library
« Reply #21 on: August 12, 2020, 04:55:23 AM »
To be clear, it was not my own request, though am happy A-Team is giving these detailed explanations. (Not that I can follow them myself).

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #22 on: November 19, 2022, 12:48:37 AM »
Resizing table for overlays:
satellite           pxl       %      ~km/pxl    ~km^2/pxl
SMOS              397.5    75.815   11.92      5.06
Ascat              524.3   100.000    9.04      6.67
OsiSaf            1075.5   204.711    4.41     13.66
SMOS-SMAP   1146.0   218.577    4.13     14.59
AMSR2           1197.6   228.419    3.96     15.24

awi-v110                     284.912%
                                    (35.099)

oren

  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: Dev library
« Reply #23 on: November 19, 2022, 08:14:30 AM »
So is awi-v110 sized at 3.17 km/pxl?
Just trying to see if I understand correctly.

uniquorn

  • First-year ice
  • Posts: 5117
    • View Profile
  • Liked: 2163
  • Likes Given: 388
Re: Dev library
« Reply #24 on: November 19, 2022, 02:33:42 PM »
the resizing table was copy/pasted from A-Team, I didn't check it, but it's been fine for graphic overlays. I wouldn't rely on the distances in an EV  ;)

For v110 I measured tip of the first Kurilskiy Strait (Okhotsk Sea) to tip of Cornwall,UK
Worldview measured it at 8695km
https://www.movable-type.co.uk/scripts/latlong.html says 8669km.

Using GIMP the pixel distances are
ascat 986.7px     8.79km/px   100%
v110  2815.5px  3.079km/px  285.345%
(last night I measured ascat at 988.2px)




oren

  • First-year ice
  • Posts: 9805
    • View Profile
  • Liked: 3584
  • Likes Given: 3922
Re: Dev library
« Reply #25 on: November 20, 2022, 04:15:14 PM »
Thanks.

morganism

  • Nilas ice
  • Posts: 1691
    • View Profile
  • Liked: 215
  • Likes Given: 124
Re: Dev library
« Reply #26 on: July 29, 2023, 02:42:05 AM »

A new dataset of Arctic images will spur artificial intelligence research
The dataset, being collected as part of a US Coast Guard science mission, will be released open source to help advance naval mission planning and climate change studies.

"Lincoln Laboratory has an excellent relationship with the Coast Guard, especially with the Research and Development Center. Over a decade, we’ve established ties that enabled the deployment of the CRISP system," says Amna Greaves, the CRISP project lead and an assistant leader in the AI Software Architectures and Algorithms Group. "We have strong ties not only because of the USCG veterans working at the laboratory and in our group, but also because our technology missions are complementary. Today it was deploying infrared sensing in the Arctic; tomorrow it could be operating quadruped robot dogs on a fast-response cutter."

The CRISP system comprises a long-wave infrared camera, manufactured by Teledyne FLIR (for forward-looking infrared), that is designed for harsh maritime environments. The camera can stabilize itself during rough seas and image in complete darkness, fog, and glare. It is paired with a GPS-enabled time-synchronized clock and a network video recorder to record both video and still imagery along with GPS-positional data.
(snip)
The CRISP project team plans to publicly release the dataset, anticipated to be about 4 terabytes in size, once the USCG science mission concludes in the fall.

https://news.mit.edu/2023/new-dataset-arctic-images-will-spur-artificial-intelligence-research-0724

morganism

  • Nilas ice
  • Posts: 1691
    • View Profile
  • Liked: 215
  • Likes Given: 124
Re: Dev library
« Reply #27 on: August 12, 2023, 09:48:44 PM »
NASA opens their software library. Here is the image processing page..)

https://software.nasa.gov/software/category/data_and_image_processing

this looks useful...)

 NPO-42082-1
PixelLearn
PixelLearn is a tool for classifying the pixels in scientific image data sets. Based on one or more images on the same grid, the tool uses cutting-edge clustering algorithms to automatically find structures in the image, or to label individual classes and use supervised classification methods to extend the labels to the rest of the image.

morganism

  • Nilas ice
  • Posts: 1691
    • View Profile
  • Liked: 215
  • Likes Given: 124
Re: Dev library
« Reply #28 on: February 22, 2024, 05:58:49 AM »

kassy

  • Moderator
  • First-year ice
  • Posts: 8235
    • View Profile
  • Liked: 2041
  • Likes Given: 1986
Re: Dev library
« Reply #29 on: February 22, 2024, 09:04:17 AM »
It is quite easy to play around with but not all data categories are available. Guess they will be added.
Þetta minnismerki er til vitnis um að við vitum hvað er að gerast og hvað þarf að gera. Aðeins þú veist hvort við gerðum eitthvað.