Support the Arctic Sea Ice Forum and Blog

Author Topic: Creating Animated GIFs  (Read 11338 times)

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Creating Animated GIFs
« on: May 16, 2015, 09:05:10 PM »
Following a request from jdallen, here's a place to discuss how best to go about creating GIF animations (and who knows, maybe some other sorts also).

As mentioned in the original thread, I currently use ImageMagick, but there are other possibilities. See for example this existing (if short) thread on ImageJ.

If I recall correctly A-Team is also quite fond of GIMP. [PS - Here's Laurent's video]



The next question is "which operating system?". Wipneus will no doubt suggest Raspberry Pi flavoured Linux, but that isn't necessarily everybody's cup of tea!

Finally, for the moment at least, which sequence of images should we use for the initial tutorial?
« Last Edit: May 16, 2015, 09:59:13 PM by Jim Hunt »
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

Laurent

  • ASIF Governor
  • Posts: 2525
    • View Profile
Re: Creating Animated GIFs
« Reply #1 on: May 16, 2015, 09:24:09 PM »
That video for Gimp a free software available here : http://www.gimp.org/
How to Create an Animated GIF With GIMP

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Re: Creating Animated GIFs
« Reply #2 on: May 17, 2015, 02:02:22 PM »
Personally I am not a Windoze fan. Hence is these circumstances I often use Cygwin to make Windows N work more like *IX.
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

Neven

  • Administrator
  • ASIF Governor
  • *****
  • Posts: 4218
    • View Profile
    • Arctic Sea Ice Blog
Re: Creating Animated GIFs
« Reply #3 on: May 17, 2015, 03:12:34 PM »
Unfortunately I am tied to Windows (for the software I use for work), so a few years ago I decided to purchase this software to make animated GIFs: Easy GIF Animator. Fulfils my needs very well, as I like my animations to keep a small size so people with lower bandwidth don't have to wait ages to see it appear in a blog post.  But I'm sure there's more and better stuff out there.
Il faut cultiver notre jardin

sidd

  • ASIF Upper Class
  • Posts: 1808
    • View Profile
Re: Creating Animated GIFs
« Reply #4 on: May 17, 2015, 06:41:21 PM »
Has anyone suggested ImageMagick yet ?

http://www.imagemagick.org/Usage/anim_basics/#gif_anim

excellent tutorial, many examples. ImageMagick is open source, and can do many more things than animation. All from the command line ...

sidd

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Re: Creating Animated GIFs
« Reply #5 on: May 17, 2015, 06:56:35 PM »
Has anyone suggested ImageMagick yet ?

Yup. See post #1!

I'm happy on the command line, plus scripts for automation, but maybe others here are not?
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

jdallen

  • ASIF Upper Class
  • Posts: 2442
    • View Profile
Re: Creating Animated GIFs
« Reply #6 on: May 20, 2015, 05:31:10 PM »
Thank you gentlemen.  I shall have to apply some of this directly.  I'm currently browsing through prices for a replacement machine - my primary laptop is six years old and falling behind current software.

Gimp questions will follow.
This space for Rent.

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Re: Creating Animated GIFs
« Reply #7 on: May 23, 2015, 07:04:07 PM »
Espen asked how to install ImageMagick on Windows. To the best of my recollection I just followed the instructions here:

http://www.imagemagick.org/script/binary-releases.php

Download the correct .EXE file, depending on whether your Windows installation is 64 bit or not, then run it.
« Last Edit: May 23, 2015, 07:12:35 PM by Jim Hunt »
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

Espen

  • ASIF Governor
  • Posts: 3078
    • View Profile
Re: Creating Animated GIFs
« Reply #8 on: May 23, 2015, 07:39:13 PM »
Havent got a clue my PC is a HP 8760w I7-processor and 16 GB-ram?
Have a ice day!

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Re: Creating Animated GIFs
« Reply #9 on: May 23, 2015, 07:59:46 PM »
I'm making some assumptions here, but if you go to "Control Panel", then "System", it should tell you whether you're running a 32 bit or 64 bit OS.

The 32 bit version of ImageMagick should run on either, but 64 bit's better if your OS supports it.
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

ChadGreene

  • ASIF Lurker
  • Posts: 8
    • View Profile
    • chadagreene.com
Re: Creating Animated GIFs
« Reply #10 on: September 15, 2015, 10:35:33 PM »
GIFs are pretty easy to make in Matlab.  Here's one I made using the seaice function in Matlab:


A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #11 on: January 18, 2016, 09:50:46 AM »
Here is a javascript alternative to animated gifs that gives the end-viewer quite a bit more control. The problems with gifs are that (1) there's no speed controller or freeze-frame tool it may be set to play too fast or too slow (necessitating download and fixing locally say in gimp), (2) gifs are stuck in 8-bit indexed color (256 per frame) whereas many cryosphere products (such as ice sheet velocity) need a full 24-bit gamut of colors supported by png and jpg, and (3) satellite photos such as Landsat are in 16-bit depth tif which is essential to contrast optimization.

N Holschuh, a grad student at PSU, developed this new interface which is designed to display co-registered layers in any sequence or rate that the end-viewer wants. He provides examples for NEGIS, Jakobshavn, Pine Island, Thwaites and four other glaciers, apparently as part of a glacier atlas project. Each glacier has layers for surface elevation, bed elevation, ice thickness, velocity (both linear and log scales, and a Landsat. Details such as color keys and dual coordinate systems (mercator and lat,lon) are nicely done.

Note this is a layer-viewer rather than a full GIS engine that allows basic math operations to act on pairs of layers. For that, transparency overlays and conventional gif animations, load into gimp or ImageJ. Note an individual layers themselves could each be animations, say a time series of calving front position.

http://nholschuh.com/glaciers.html try an example or two before reading on.

The javascript is basically an alternative to a standard html click-map which goes to a new url depending on where you click on the base image. Here all the layers are cached temporarily on your computer. Upon mouse-over of the controller, the appropriate new layer loads instantly. The effect is seamless. (The controller won't work on the image below because  forum software does not support embedded html so visit the link above to see it in action.)

I've made quite a few changes to simplify the code and enlarge the display and controller to our maximal width of 700 pixels. It turns out to be easier to generate a controller an arbitrary number of layers with text names using a spreadsheet (rather than html tables)., second image below shows the set-up for 7 layers.

If you copy out the text attached below and save as html, it will correctly load Holschuh's layers for Zachariae/79N in northern Greenland in any web browser (provided you are online). To make your own display, replace the links to paths to your own images (which can be urls to forum images or anything on the web).

I've colored the text below to distinguish the overall html container (blue) from the javascript (purple), controller, and image urls that have to be changed.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link href="../general.css" rel="stylesheet" type="text/css" />
<script type="text/javascript">
function MM_preloadImages() { //v3.0
  var d=document; if(d.images){ if(!d.MM_p) d.MM_p=new Array();
    var i,j=d.MM_p.length,a=MM_preloadImages.arguments; for(i=0; i<a.length; i  )
    if (a.indexOf("#")!=0){ d.MM_p[j]=new Image; d.MM_p[j  ].src=a;}}
}
function MM_findObj(n, d) { //v4.01
  var p,i,x;  if(!d) d=document; if((p=n.indexOf("?"))>0&&parent.frames.length) {
    d=parent.frames[n.substring(p 1)].document; n=n.substring(0,p);}
  if(!(x=d[n])&&d.all) x=d.all[n]; for (i=0;!x&&i<d.forms.length;i  ) x=d.forms[n];
  for(i=0;!x&&d.layers&&i<d.layers.length;i  ) x=MM_findObj(n,d.layers.document);
  if(!x && d.getElementById) x=d.getElementById(n); return x;
}
function MM_swapImage() { //v3.0
  var i,j=0,x,a=MM_swapImage.arguments; document.MM_sr=new Array; for(i=0;i<(a.length-2);i =3)
   if ((x=MM_findObj(a))!=null){document.MM_sr[j  ]=x; if(!x.oSrc) x.oSrc=x.src; x.src=a[i 2];}
}
function MM_swapImgRestore() { //v3.0
  var i,x,a=document.MM_sr; for(i=0;a&&i<a.length&&(x=a)&&x.oSrc;i  ) x.src=x.oSrc;
}
</script>

</head>
<body onload="MM_preloadImages('http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_3.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4a.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4b.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_5.png']http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_2.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_3.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4a.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4b.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_5.png'[/url]]http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_1.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_3.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4a.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4b.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_5.png']http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_2.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_3.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4a.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4b.png','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_5.png'[/url][/url])">
<img src="http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_1.png" ID="swapper" width="700" height="903" border="0" usemap="#Map">

<map name="Map" id="Map">
   <area shape="rect" coords="88,813,199,857" href="#1" onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_1.png',1)" />
   <area shape="rect" coords="197,813,310,857" href="#2"  onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_2.png',1)" />
   <area shape="rect" coords="309,814,415,858" href="#3" onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_3.png',1)" />
   <area shape="poly" coords="413,814,526,814,523,837,469,837,469,859,413,859,413,815" href="#4a" onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4a.png',1)" />
   <area shape="rect" coords="470,837,524,859" href="#4b" onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_4b.png',1)" />
   <area shape="rect" coords="522,813,636,858" href="#5" onmouseover="MM_swapImage('swapper','','http://nholschuh.com/Ice_Stream_Images/NEGIS/NEGIS_5.png',1)"/>   
</map>

</body>
</html>
« Last Edit: January 19, 2016, 08:54:50 AM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #12 on: January 19, 2016, 09:09:16 AM »
I'm looking now at comparison of two images via a movable vertical slider. A common example of this is before/after satellite images of a glacier calving. The slider is more effective visually than side-by-side or flickering animation comparisons because the end-user can zero in on a specific region to compare.

Our problem has been getting sliders to display within forum software. Usually the slider is implemented in javascript but there are alternatives, CSS and html5, which may display correctly here. I've collected some links below which contain open source code that will do this; more testing in a bit.

http://lea.verou.me/2014/07/image-comparison-slider-with-pure-css/
http://thenewcode.com/819/A-Before-And-After-Image-Comparison-Slide-Control-in-HTML5
https://codyhouse.co/gem/css-jquery-image-comparison-slider/

http://earthobservatory.nasa.gov/IOTD/view.php?id=86436&eocn=home&eoci=iotd_readmore

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #13 on: January 21, 2016, 01:33:16 PM »
Below, I look at the moderately massive computing requirements to process all the flight sub-segments (GPS waystations) of Operation Icebridge (as served by Cresis) so that all the cross-over intersections with other flights can be called up from a database with all their ancillary data such as bearing, bearing with respect to flow, elevation, ice thickness, bedrock profiles and relevant parts of the radargramss cropped out.

The purpose is to facilitate reconstruct all basal upheavals in Greenland in 3D from all their 2D cross-sections to characterize individual features and additionally determine the total region of affected ice-bedreck interface and overall displacement volumes.

The main product is an online portal that, when queried with a region of interest (point, line, or bounding box), returns all the intersecting (or nearby tracks). The algorithm is very similar to that of satellite image servers such as EarthExplorer, though here the objects are 1D flight lines rather than rectangular satellite scenes.

It’s necessary to work at the level of sub-segments since many flights had curved tracks — working with the whole flight segment means working with a chord shortcircuiting an arced track. The sub-segments do this too but their chords are a vastly closer fit to the curve.

An example suffices to illustrate the overall procedure. In reviewing flight line pdfs near Eqip Sermia, I noticed 19980714_01_017 contained a classical basal deformation some distance inland from the deformation highlighted in Bell 2014. To understand this feature, I needed to pull all other tracks from the Cresis archive that also cut through this deformation.

After a simple paste of the kml file into a grepping spreadsheet (which cleans away extraneous kml terms, rounds data to appropriate precision and creates clean columns), the last 9 way-points are shown below. Only the last 9 are used because they have delimit a basal disruption (based on inspection of the radargram 19980714_01_017), with only way-points 3-6 actually lying within the upheaval. The colums show latitude, longitude west, elevation in meters, flight segment name and order within the line.

This particular flight segment was represented by 41 GPS points nearly equally spaced along the 148 km track, dividing it into 40 segments of 3.7 km width.

70.068805   47.076781   1928   19980714_01_017   9
70.052159   47.15645    1865   19980714_01_017   8
70.033927   47.243766   1748   19980714_01_017   7
70.014753   47.335211   1864   19980714_01_017   6
69.995344   47.425215   1820   19980714_01_017   5
69.975694   47.513974   1694   19980714_01_017   4
69.955976   47.603585   1610   19980714_01_017   3
69.935927   47.696685   1595   19980714_01_017   2
69.91662    47.787071   1557   19980714_01_017   1


To find all other way-segments in the vast Cresis archive of all years in the vicinity of these nine sub-segments could be done ad hoc using visuals in Google Earth. However this becomes very ineffecient in studying hundreds of basal deformations.

It is actually quite easy to collect all the relevent way-points because Cresis provides a master kml file for each of the 21 years for which Greenland data was collected. A click loads that file into Google Earth and copy-paste into a text editor extracts the data. These are effortlessly counted by pasting in Word, replacing commas, and counting the replacements, then halving as three coordinates are separated by two commas. For example, year 2014 had 58 flights and a total of 143,764 way-points; earlier years such as 1998 far fewer, 11 flights and 3,528 way-points.

Overall, the master database of way-stations consists of 654,182 database records (rows) which is quite manageable in Excel though fairly slow for conducting operations:

2014   Greenland   P3/   143764   287528
2013   Greenland   P3/    30391    60782
2012   Greenland   P3/    64036   128072
2011   Greenland   P3/   118421   236842
2011   Greenland   TO/     9310    18620
2010   Greenland   DC8/   41396   82792
2010   Greenland   P3/    40705    81410
2009   Greenland   TO/    10406    20812
2008   Greenland   Grd/   410      820
2008   Greenland   TO/    16736    33472
2007   Greenland   P3/    19773    39546
2006   Greenland   TO/    16102    32204
2005   Greenland   TO/    32428    64856
2003   Greenland   P3/    89149   178298
2002   Greenland   P3/     2745     5490
2001   Greenland   P3/     1694     3388
1999   Greenland   P3/     4955     9910
1998   Greenland   P3/     1764     3528
1997   Greenland   P3/     3528     7056
1996   Greenland   P3/      744     1488
1995   Greenland   P3/     2642     5284
1993   Greenland   P3/     3083     6166
         654182   

However, just to pursue the restricted agenda of finding other flights intersecting the basal deformation (thus providing  different perspectives on its structure), it suffices to collate way-points of just four other kml files (see first image) and collect way-points that lie within a generously defined lat,lon bounding box of the upheaval.

This is easily done by first sorting the database by the latitude column, giving a score of 1 in a new column to appropriate new way-points, then sorting by longitude and doing the same, then summing the two new columns into a third, and doing a final sort for records scoring 2 (which amounts to a logical 'AND' operation).

Next, the qualifying way-points are merged into way-paths, which requires that both ‘feet’ lie within the generous bounding box and that the points be consecutive within their track. After building the respective kml units, these way-paths can be displayed in Google Earth and sorted for those that actually intersect the initial track having the deformation.

The radargrams are collected, contrast-enhanced, rescaled to a common vertical exaggeration, and given a background gridding in gimp according to the number of way-points the image contains. This allows cropping of the radargrams to the bounding box and for intersecting paths, mark-up of the cross-over points on the radargrams.

Overall, the idea is to free up the analyst from endless ad hoc manipulations. All the available data relevant to characterizing each basal deformation in Greenland is at the analyst’s fingertips. While this might not be enough to fully understand the feature, it is a big step forward in hypothesis testing, categorizing, and prioritizing suitable  deformations for additional radar overflights.

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #14 on: January 23, 2016, 11:42:50 AM »
This post looks at slicing and dicing an image into parts and then reconstituting it seamlessly in forum view. The first image does this successfully from the 3 attached tile pieces below (click, hold and drag to see the pieces in the big image) without using an html table. Note the tiles could be separate small gif animations instead of stills.

I've also duplicated this effect within 1x1 and 1x3 forum tables (no longer shown); table controllers cellpadding="0" border="0" cellspacing="0" are not needed in default forum table interpretation.

The small pieces could be saved offsite but online (eg Photobucket) and simply referenced by their urls. This provides a workaround for the 4 image limit of forum software. It could be used for example to display many individual frames of a gif animation. This trick also allows images to be placed inline in association with relevant text rather than merely appended as attachments.

The glaciological idea here is to vertically slice radargrams containing basal deformations in part of the image (especially the ones from curved flight lines) into their GPS way stations and discard slices not relevant to the deformation, make a new Google Earth track from the retained way stations that delineate the overall boundaries of the feature (along the cross-section). It's equally useful for deformation extent mapping to show adjacent track segments that miss the deformation.

Gimp as it stands provides controls over slicing but these are extremely tedious at scale. However 'guides-grid' .scm at the plugin registry, placed in gimp's scripts folder, allows creation of an way station appropriate grid that converts into cutting guides for the guillotine command. That creates a new image for each slice that retains the overall layer structure but it is not so convenient to put the 40 separate files thus generated back into a single image with (40 way stations)*(number of layers per ways tation) total layers.

However once the cutting grids are made, Filters --> Web --> Slice... will systematically name and save all slices of the active layer (as gif, jpg, or png) in an orderly row and column manner into an html table (only the slice image files need to be retained). It is somewhat unfortunate that forum software does not have a storage area for files that only need url pointers but not display as solo attachments. I suspect that they periodically sweep and delete orphaned files (no longer attached to any post), invalidating the url they once had.  .

If the first of the slice files is opened in Gimp, the others can be added in appropriate sequence using File --> Open as Layers ... Slice only operates on the active layer but changing this to the others one layer at a time, then opening these products in that same gimp document, a stack can be built that amounts to a correctly implemented Guillotine command option (sliced with layer retention).

Here each radargram needs a second layer with vertical lines indicating the flight sub-segments between GPS way-stations. Fortunately, these are quite even spaced, presumably because the plane flew at constant speed with readings taken at equal times. This second layer is completely transparent other than one pixel wide vertical waystation lines, meaning the radargram is still easily viewed. These layers should not be flattened because it would wipe out data underneath the grid lines.

The main attribute of this second layer is text values for each way station of lat,lon found in kml files that define the way-station coordinates. Gimp does a poor job importing vector files (eg text) from  the host computer's clipboard -- incomprehensively, even its own internal clipboards don't talk to each other.

There's an undocumented trick that provides a workaround: if all the lat,lon data is copied to the host clipboard and the gimp text tool is clicked on the grid layer as if something were to be manually typed in, a contextual menu pops up that allows importation (paste) of text from an external text editor. If the grid is rotated 90º CW ahead of time and re-rotated 90º CCW after text is situated properly relative to grid lines, the text will read sideways as needed.

A third radargram layer shows the graph of the csv file Cresis provides separately for each radargram. These are apparently generated during the process of manual annotation of ice sheet surface and bottom reflections (colored lines in echo2 radargrams). The fields provided are Lat,Lon,UTC-time,Thick,Elevation,Frame,Surface,Bottom,Quality.

These come in excessive precision that greatly exceeds any conceivable experimental input -- for example, neither bottom nor surface of the ice is not measured to the posted 0.1 mm -- the error is more like 10 m per study of flight crossover points. Thus it is necessary to drop Cresis data into a rounding spreadsheet before working with it.

Of these, surface elevation, bedrock depth and ice thickness are already implicitly graphed in the echo_2 version of the radargram even when cropped laterally to a deformation. Here surface and depth graphs could be usefully deployed as masks (ie flattened onto the radargram itself) to either permanently suppress irrelevant parts of the radargram, constrain action of image enhancement filters to internal ice, or best of all, to provide automated cropping guides above and below (often two thirds of the radargram is wasted space).

Such cropping guided by deformation lat,lon is even more effective in reducing images. That's important because the deformation might later have to be enlarged in bringing it to a standard vertical exaggeration.

Needless to say, to do this on the whole Cresis radargram archive is a fairly massive pre-compute. However it is far more convenient to do this once and for all to allow researchers to get on with the glaciology. The same can be said for horizontal and vertical rescaling to a uniform vertical exaggeration (say 10:1) needed for 3D deformation reconstruction and major aspects of cropping and contrast optimization.



[Edit: quite a few text improvements made above since initially posted.]
« Last Edit: January 24, 2016, 01:34:06 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #15 on: January 24, 2016, 10:03:11 PM »
Animated gifs for long time series presented at data-level resolution suffer file size bloat. One workaround masks out regions that aren't important (which could vary from frame to frame). Masking replaces everything within ots boundary of unimportance with a fixed color -- that compress to almost nothing. 

If an alpha transparency channel is used instead, the background gray in forum software will show through. That's usually ok and definitely better practice than the gif-maker's preference color because the end-viewer upon image download gets control over background transparency.

In the case of radargrams, the actual grayscale images is flanked horizontally by very generous white space and scales. The fix there is cropping, though it won't get rid of scale ticks flattened onto the data. Those can be gotten rid of however by opening the pdf compilation in ImageJ (File -->Import -->Extract Images from PDF...).

In the case of 19980714_01 pdf, the images are at resolution 903 x 888 whereas the same image in the archive  is 1165 x 919 (its framing container) (eg see https://data.cresis.ku.edu/data/rds/1998_Greenland_P3/images/19980714_01/).

Something is really wrong here because the pdf's aspect is 1.02 whereas that of the archival image is 1.27. So one or the other (or likely both) is peculiarly rescaled (irrational different vertical and horizontal multipliers). And how was that rescaling done: nearest neighbor, bilinear, bicubic, or sinc?!?  It matters.

Meanwhile wtf is the actual resolution of the data, is it stored openly or only buried in matlab, and is either of the provided resolutions actually at data resolution? This is a huge issue for mapping basal deformations and their sub-structures.

Apparently Cresis not only assumes visitors have purchased expensive proprietary command-line software (matlab) and gone through its year-long learning curve but also store the data in its completely unreadable file format (.mat) which opens as total gibberish in text editors.

I find this completely unacceptable in publicly funded science. While .mat may be a wonderfully powerful and flexible container, 99 times out of a 100 nothing but ordinary row and column numbers are being stored (as here). Data should never be stored in anything more complicated than is necessary, here plain old cvs. Let the matlab users import cvs.

The airplane's raw 'trace' files (the 1 pixel wide vertical lines that tile up to make a radargram numerical array) are being processed via various matlab command settings to produce the radargram, its unwanted overlapping edges, text overlays, and badly done scales.

It is on this raw numerical array that 'contrast' tools will optimally operate. Here it is critical to understand the perfect 1:1 correspondence between working on a numerical array and working on its bitmap display via Photoshop-type tools.

Many people fail to realize that gimp, PS, ImageJ, ImageMagick etc are not inherently graphical software, they are just  Excel running in the background that produces an imagery front end. Matlab processing an array of numbers is just the poor man's Photoshop. It is imperative to use a graphical front end to exploit synergies with the superb pattern recognition ability of human vision.

Cresis, which is to say matlab, is strictly outputting 8-bit grayscale images. In some software, they open as RGB but in fact there is no color. That's easily proven: ImageJ reads the header and always opens files at their full bit depth. Make  a duplicate image, drop it to 8-bit grayscale, form a stack with the original, subtract the layers and observe from the histogram that it is entirely zeroes (pure black). That means no color, not even subtle color.

Where is this grayscale color coming from? I would think matlab normalizes the raw numerical array of radar return power (traces) by dividing by the biggest number to bring everything into [0,1] and then rescales and rounds to integers in the range [0,255]. That opens directly in ImageJ as a BMP or x,y image which is then losslessly converted to more familiar tif or png.

I have huge concerns about the sub-optimality of this. The data, or at least certain depth ranges of it (ie the basal layers), may actually be of much higher bit depth say 12- or 16-bit. Running with 8-bit would be as crazy as dumbing down a hundred million dollar feature of Landsat-8 to make another Landsat-7.

I will also bet the contrast compression was linear. Huge mistake, it should at a bare minimum set a gamma and optimally be adaptive (as in ImageJ Clahe). Very few users are actually interested in the physical meaning of the absolute radar return strength; this class of users suffers from its retention.

Ice penetrating radar was initially tasked solely with determining depth to bedrock. (Surface elevation is better measured with laser, inSar and optical stereo pairs which are 2D rather than sparse 1D tracks; ice thickness = surface - bedrock.) The goal posts have since moved to internal layer isochrons and basal conditions but archive processing has not moved with it. Bedrock topography was terribly important in older stick-or-slip models but with temperate ice, deformable basal sediment and looser lower ice rheology, that becomes less important to predicting future behavior.

Ultimately, even a greatly improved image of a basal deformation may prove just as mysterious as our current blurry images. The real advantage will come from greatly enhanced ability to map the geographical distribution of anomalous basal conditions (ie get past the dramatic ones to the faint but still critical to sliding and deformation).

I am looking now at Wolfram's Mathematica -- it can open this crappy .mat format and export into plain text cvs. Affordable to anyone (cost: a cup of coffee per month), it is world's apart in mathematical and graphical sophistication from conventional scientific software. It may be worthwhile to dump the whole Cresis archive out of .mat into .cvs as a pre-computed cloud, but it remains to be seen if Wolfram offers a convenient method for bulk processing the whole Cresis archive.

http://www.wolframalpha.com/pro/?source=nav

In terms of gif reduction, the images below asks whether the ice surface and bedrock cvs files can serve to define masks. The answer is yes but something is lost even having 409 points per curve per radargram -- they're still chord collections. While the fit might be improved with splines, it is still not quite as good as the original graphic. However a big benefit comes from dropping back a couple of pixels to define crop boundaries, especially in basal deformations of limited extent.
« Last Edit: January 24, 2016, 10:15:06 PM by A-Team »

Watching_from_Canberra

  • ASIF Lurker
  • Posts: 65
    • View Profile
Re: Creating Animated GIFs
« Reply #16 on: September 10, 2016, 03:36:08 PM »
just testing an animated gif that doesn't seem to animate...
« Last Edit: September 10, 2016, 04:02:10 PM by Watching_from_Canberra »

Jim Hunt

  • ASIF Governor
  • Posts: 3334
    • View Profile
    • The Arctic sea ice Great White Con
Re: Creating Animated GIFs
« Reply #17 on: September 10, 2016, 03:54:11 PM »
just testing an animated gif that doesn't seem to animate...

It doesn't animate for me either. I seem to recall A-Team mentioning that you have to reduce the width to 700 px to keep SMF happy?
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

Watching_from_Canberra

  • ASIF Lurker
  • Posts: 65
    • View Profile
Re: Creating Animated GIFs
« Reply #18 on: September 10, 2016, 04:02:51 PM »
Thanks for the pointer - that seems to be the issue.  I'll go and post it where I first posted it now...

Tigertown

  • ASIF Upper Class
  • Posts: 1634
    • View Profile
Re: Creating Animated GIFs
« Reply #19 on: December 20, 2016, 05:58:52 PM »
If you got something too big to work on here and you don't want to really change it, just make a youtube video out of it and post the link.

Example:

Tigertown

  • ASIF Upper Class
  • Posts: 1634
    • View Profile
Re: Creating Animated GIFs
« Reply #20 on: April 19, 2017, 05:02:49 PM »
For an easy to use site, try
http://gifmaker.me

subgeometer

  • ASIF Citizen
  • Posts: 105
    • View Profile
    • All in the Name of Liberty
Re: Creating Animated GIFs
« Reply #21 on: April 20, 2017, 05:02:56 AM »
If running linux or unix ImageMagick is a ubiquitous tool suite for image processing. On Ubuntu at least it comes as part of the standard install. It runs from the command line so is easily usable in scripts for batch processinga

The main tool is convert

A simple shell command to make a sequence of stills from image_1.jpg to image_n.jpg into a gif is

convert -delay 100 image*.jpg the_movie.gif

or

convert -delay 100 image_1.jpg  image_2.jpg  ... image_n.jpg the_movie.gif

The delay parameter sets framerate as milliseconds per frame(I think)

convert can do many other things besides, a swiss army knife for scaling, compositing, image generation,adding text etc . It's fairly well documented online though its manual is somewhat terse

subgeometer

  • ASIF Citizen
  • Posts: 105
    • View Profile
    • All in the Name of Liberty
Re: Creating Animated GIFs
« Reply #22 on: May 27, 2017, 06:52:36 AM »
I'm starting to write a bit of python code to ease downloading of image sequencies, initially from worldview, but the approach could be applied to other sites that encode params in the url

It's just a function so far which I've been using in the interpreter. Basically you feed it the url worldview's image download function loads and python downloads it and a sequence of previous days(the default is 8 ) and writes them with a series of sequential names(based on a nameroot you can give, and which defaults to "worldview")

When I have another free hour or 2 I'll add argument handling so it can be used as a script

import urllib, urlparse

def get_worldview_sequence(start_url, count=8, outfileroot="worldview"):
    parsed_url=urlparse.urlparse(start_url)
    startquery=urlparse.parse_qs(parsed_url.query)
    #this will cause problems when going over the new year
    #but its quick so we do for now
    startday=int(startquery["TIME"][0])
    while count >0:
        #this really should check that
        #this doesn't get to day 999 of the previous year or other nonsense
        startquery["TIME"][0]=str(startday)
        realquery={}
        for nm in startquery:
            realquery[nm] = startquery[nm][0]
        #fucked up character substaitution
        out_url=urlparse.urlunparse(urlparse.ParseResult(parsed_url.scheme,parsed_url.netloc, parsed_url.path, parsed_url.params, urllib.unquote(urllib.urlencode(realquery)), parsed_url.fragment))
        #out_url= "https://gibs.earthdata.nasa.gov/image-download?"+urllib.urlencode(startquery)
        outname=outfileroot+str(startday)+".jpeg"
        print out_url
        urllib.urlretrieve (out_url, outname)
        count=count-1
        startday=startday-1
« Last Edit: May 28, 2017, 12:55:26 PM by subgeometer »

oren

  • ASIF Upper Class
  • Posts: 1597
    • View Profile
Re: Creating Animated GIFs
« Reply #23 on: August 03, 2017, 01:09:36 PM »
For very easy animated GIFs I use ImageJ, using the "Fiji 64-bit" download for Windows.
Open (ctrl-O) in sequence all the images you want to animate (preferably make sure they are limited to 700 pixel width).
Image>Stacks>Images to Stack.
File>Save As>Animated Gif.
Of course you can play around with some options and a lot of other stuff, this is for the very beginner.

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #24 on: September 15, 2017, 08:47:09 PM »
Panoply makes easy animated gif maps in all common projections, provided that the netCDF file itself being used has the data in multiple time shots.

These are geolocated maps in whatever projection and palette color you've found most effective by rapid experimentation, in our case typically the polar stereographic with a sequential palette for an ordered variable like ice area or a divergent palette for ± change about 0.

Panoply has ~100 projections and a great many palettes (plus it's easy to import more from galleries, grab from an online map say one from @zlabe, or made from scratch using rgb or hsv spreadsheet numbers).

The latter is most easily done by downloading a bona fide .rgb palette from UCAR (header and link shown below), opening it as plain text, and overwriting with your own numbers, followed by 'save as'. That trick defeats the Mac operating system which refuses to let you change .txt to .rgb manually.

ncolors  = 254
;R   G   B
0   22   132
0   24   135
0   26   136
...

Now use the Open file command in Panoply to import the .rgb palette; it immediately becomes active on the top map. The forum software also refuses text files attachments that are not .txt so I've relabelled the attached .rgb file below as .txt; it is the blue-to-white palette used at UH AMSR2 for sea ice concentration. (With this, I can use AMSR2 right up to today and then follow with the ESRL forecast in the same colors.)

That will be the case for the .nc files from ESRL climate forecasts which have numerous 5- and 10-day series at 6 hour intervals, so 20 and 40 frames respectively. If that's too many, set the 'stride' between consecutive frames to skip intervening ones.

Panoply does not make gifs per se, just the .png or .jpg frames. These just need to be loaded up as a stack in ImageJ as Oren explains above or as layers in gimp, then cropped/resized to 700 x 700 pixels max and saved as a forum animation. Panoply also makes nice .mp4 videos but the forum software does not accept that format.

Looking at the opportunities at ESRL on 15 Sep 17, the REB.2017.09.14 netCDF bundle offers seven 40-frame maps, forecasts out to D10:

* snow/ice surface temperature
* air temperature
* snow thickness
* ice thickness
* ice area
* air velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction
* ice velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction

Looking at the REB_plot component at the ESRL archive on 15 Sep 17, some 43 graphics are provided as .png or .gif but without the underlying netCDF Geo2D files (meaning projection and palette aren't easily changed). Some of the .gifs are stills but 22 of them are pre-made running gifs, 20-frame forecasts out to D5.

The MacOS has a wonderful feature for rapid scanning of the 43 graphics: select-all and right-click on Quick Look. This does not open the files in an app like Preview but displays them like a slide show, even animating the animations within such slides. (I use Quick Look with our game cam to flip though the tens of thousands of wildlife shots for keepers.)

The ESRL have issues such as too big to load or too wide to play on the forum, local rather than Arctic Ocean wide coverage, zero used for both data and not-on-ice background (ie confusingly same color, white), extraneous comparisons to GFS forecasts, esoteric meteorology variables, unwanted overlays like contours, and so forth.

Like all Panoply maps, these gifs strictly use undithered palette colors and so these can be selected and changed out. This can be done for all frames at once if the gif is loaded into Gimp, converted to RGB, tiled up by Filmstrip, color replaced, and sliced back out to individual frames.

The REB_plots have uninformative names at the moment, no clues as to date or content:

Arctic1    RASM-ESRL vs GFS ice area and snow depth
Arctic2    RASM-ESRL vs GFS 2m temp and surface pressure
Arctic3    ice thickness thermo + dynamics + snow melt + ice melt
Arctic4    RASM-ESRL vs GFS 850 hPa temp and precip
Arctic5    ice and snow thickness contoured
Arctic9    RASM-ESRL vs GFS 500-1000 hPa thickness and precip
Arctic10   bottom ice growth + lateral melt + snow melt + top ice melt
Arctic11   longwave + solar flux + shortwave + albedo
Arctic12   RASM-ESRL vs GFS surface temp and LWP
Arctic13   RASM-ESRL vs GFS surface pressure + 850 hPa height
Arctic14   RASM-ESRL energy flux + LWP + surface temp + IWP
Arctic15   melt pond fraction + SST + heat flux + wind speed
Arctic16   RASM-ESRL ice speed
Arctic19   RASM-ESRL 500hPa height and wind vectors
Arctic20   310K potential vorticity and Surface theta PVU
Arctic22   RASM-ESRL vs GFS longwave flux + shortwave flux
Arctic23   RASM-ESRL energy flux + LWP + temp + IWP [duplicate of Arctic14?]
Arctic24   RASM-ESRL vs GFS surface wind + energy flux

AlaskaRegion2   RASM-ESRL vs GFS 2m temp and surface pressure
AlaskaRegion4   RASM-ESRL vs GFS 850 hPa temp and precip
AlaskaRegion5   ice and snow thickness
AlaskaRegion12  RASM-ESRL vs GFS surface temp and LWP


The third component of the ESRL archive, of form RASM-ESRL_4NIC_2017-09-13.tar.gz, consists of ten time slices at 24 hour intervals in the form of RASM-ESRL_2017-08-23-00_t024.nc. Each of these consists of 18 Geo2D mappable products (or 16 after sea and ice velocity x,y arrays are combined).

aice       sea ice area fraction
hi         sea ice thickness
hs         surface snow thickness

snow ai    lwe snowfall rate
rain ai    rainfall rate
meltl      lateral ice melt
meltb      basil ice melt
meltt      top ice melt

sss        sea water salinity
sst        sea water temperature
Tair       air temperature
Tsfc       surface temperature where sea ice

strength   compressive strength of sea ice

vocn       northward sea water velocity
uocn       eastward sea water velocity

vvel       northward sea ice velocity
uvel       eastward sea ice velocity   

divu       divergence of sea ice velocity


A single such .nc does not have frames, it consists only of its time frame; to make a gif for say rainfall rate, ten one-frame maps have to be made and saved out as .pngs. From those, loaded in Gimp or ImageJ, a D10 daily gif animation can be made. However there's an issue with consistent scale because 'fit-to-data' will show a different ranges and so different utilization of the common palette.

Here are best-links to palette collections, conversion tools, appropriateness, Panoply manual, ESRL offerings, and Gimp indexed color:

https://www.giss.nasa.gov/tools/panoply/betax75/PanoplyCL_Guide_20161109.pdf

http://soliton.vm.bytemark.co.uk/pub/cpt-city/gmt/tn/GMT_panoply.png.index.html

http://soliton.vm.bytemark.co.uk/pub/cptutils-online/select.html

http://geog.uoregon.edu/datagraphics/color_scales.htm

https://www.giss.nasa.gov/tools/panoply/colorbars/

https://www.geo.uni-bremen.de/Interdynamik/images/stories/pdf/visualizing_netcdf_panoply.pdf

http://www.ncl.ucar.edu/Document/Graphics/color_table_gallery.shtml

http://www.ncl.ucar.edu/Document/Graphics/create_color_table.shtml

https://docs.gimp.org/en/gimp-concepts-palettes.html

https://docs.gimp.org/en/gimp-palette-dialog.html

ftp://ftp1.esrl.noaa.gov/RASM-ESRL/ModelOutput

https://www.esrl.noaa.gov/psd/forecasts/seaice/
« Last Edit: September 16, 2017, 01:54:00 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #25 on: September 16, 2017, 01:48:15 PM »
This is quite an interesting article on climate code and how it should be reported in journals, the observation being that a lot of research is currently irreproducible. Not in the sense that you would get a different result duplicating it but rather there's not enough information provided on the software side that would even allow you to get started replicating.

We run into that all time on these forums: maps come with no underlying data link, no underlying publication describing methods, and data not in formats like netCDF, HDF and  GRIB (all of which Panoply can view) that allow re-projection to a standard Arctic Ocean EASE grid or polar stereographic coordinate system which is necessary for comparing or combining data from different sources.

A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences
Damien Irving
http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-15-00010.1 free full

"At first glance, the only difference between a regular journal article and that of IS2015 is the addition of a short computation section (see “Example computation section” for more information).

That section accompanied the traditional description of data and methods within the article and briefly cited the major software packages used in the research before pointing the reader to three key supplementary items: 1) a more detailed description of the software used, 2) a version controlled and publicly available code repository, and 3) a collection of supplementary log files that captured the data processing steps taken in producing each key result.

The repository was hosted at a popular code sharing website called GitHub, while the detailed software description and log files were hosted at Figshare (Irving 2015a), which is a website where researchers commonly archive the 'long tail of their research (e.g., supplementary figures, code, and data)."

An example of best practices:

The results in this paper were obtained using a number of different software packages. A collection of command line utilities known as the NetCDF Operators (NCO) and Climate Data Operators (CDO) were used to edit the attributes of netCDF files and to perform routine calculations on those files (e.g., the calculation of anomalies and climatologies) respectively. For more complex analysis and visualization, a Python distribution called Anaconda was used.

In addition to the Numerical Python (NumPy; Van Der Walt et al. 2011) and Scientific Python (SciPy) libraries that come installed by default with Anaconda, a Python library called xray was used for reading/writing netCDF files and data analysis. Similarly, in addition to Matplotlib (the default Python plotting library; Hunter 2007), Iris and Cartopy were used to generate many of the figures.

To facilitate the reproducibility of the results presented, an accompanying Figshare repository has been created to document the computational methodology (Irving 2015a). In addition to a more detailed account (i.e., version numbers, release dates, web addresses) of the software packages discussed above, the Figshare repository contains a supplementary file for each figure in the paper, outlining the computational steps performed from initial download of the ERA-Interim data through to the final generation of the plot.

A version controlled repository of the code referred to in those supplementary files can be found at https://github.com/DamienIrving/climate-analysis.
« Last Edit: September 16, 2017, 02:21:37 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #26 on: September 16, 2017, 04:47:46 PM »
Below is the current Panoply palette collection. It can be viewed in Preferences; the collection below tiles up screenshots from there, hopefully faithfully. It takes a click to view at scale; reduction in dimensions is dicey because it might invoke interpolation.

The palette names suggest their origins, eg the CB prefix means imported from C. Brewer's widely used ColorBrewer web page at PSU, UO indicates Univ. Oregon geography dept., GMT means Generic Mapping Tools software from SOEST at the University of Hawaii, and many others originate at JJ Green’s cpt-city web site.

While anyone can quickly hack color tables in a spreadsheet or import them into Panoply as .rgb tex from govt climate sites, journal illustrations, twitter sites, or online product maps, there's a whole art and science to palette design.

The great advantage of Panoply, provided there's a netCDF, GRIB or HDF file associated with the map, is the chance to replace a bad palette with a good one, at the same time getting rid of vexatious contours, arrows, text and lat-lon overlays that interfere with the data and its comparison to other maps of other variables that might be in say an equal area projection. For example, a simple grayscale gradient is better-suited to arithmetic operations such as differencing.

An example application: ESRL's palette on 'icethicknesschange' is really bad. Not only is it inconsistent day to day but none of the palettes make conventional sense. That is, thickness is an ordinary real number that varies ± about 0. White is used for the lowest bins but sometimes gray, yet those are also used for the completely ice-free perimeter. This situation calls for a divergent color scale, eg reds for loss and blues for gains.

Since ESRL is using 17 color bins, the 'panoply_diff_17.act' palette is an immediate better fit. Another option, what the ESRL palette is attempting, is color-pairing. That amounts to taking a couple of ColorBrewer 8- or 9-color diverging palettes with trend logic and intercalating them to get better color separation while retaining logic.

Of the stock palettes in Panoply, interleaving CB_RdYBl_09.cpt and CB_PuOr_09.cpt is a good option. By displaying any .nc file as a map, those palettes can be cleanly captured into Gimp which has drag'n'drop indexed color table rearrangement.

The resulting palette, imported via ImageJ rgb listing into Panoply, can then display the ESRL .nc file properly. That palette is shown in the 4th image below. It could be improved for visual consistency with some minor tweaks.
« Last Edit: September 16, 2017, 05:46:00 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #27 on: September 16, 2017, 07:02:04 PM »
Here is ESRL's experimental forecast for 5-day ice thickness change prior to adjacent alternative palette replacement. It may be better however to use fewer colors, say 2-3 on the thinning side and similarly on the thickening, or simple but methodical red and blue gradients increasing out in intensity.

The frames runs out to Sept 19th from a prediction made on the 14th. The first in the series was made on Aug 22nd for the 27th. The variable background color has been put to a consistent gray. It is hard to tell from these whether thickening or thinning is happening overall; other ESRL products may be better for this (notably just the forward prediction from today or just the D0 directly from the ESRL netCDF via Panoply).
« Last Edit: September 16, 2017, 07:09:27 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #28 on: September 18, 2017, 12:55:49 AM »
We're long been able to make excellent line graphs, given the raw data. Now we're moving on -- slowly --  from copy-pasting other people's maps to being able to make them better ourselves, given the raw data. That involves geolocated data and control of the projection, overlays, and especially the palette.

Some more on palettes most appropriate to climate science maps -- there's been a lot of online discussion lately on perceptually neutral palettes, the horrific rainbow palette, and palettes neutral to common male colorblindness.

However fashionably pc the latter might be, it's not enough. The fact is that a great many other factors influence how a given map is perceived by different people on different devices. Some are viewing on $10 cell phone screens; others have calibrated 27" retinal monitors with an advanced color gamut. And then there are academics still printing out pdfs on inkjet CMYK to uncoated paper to read on the plane. Not to mention journals that won't host time series as animated gifs and Twitter that makes dithered mp4's out of them, destroying individual frames. And at the end of the day, it's largely -- but not entirely -- a matter of individual taste.

A better solution may be simply to provide a link to the netCDF file for the map, allowing readers to quickly swap in their favorite Panoply palette.

A lot of online discussion of palette creation today consists of sharing command-line code snippets (mathlab, python, java, IDL, github, etc) with many participants unaware that Photoshop already did all this and much more 30 years ago, with freeware Gimp successfully emulating almost all of it, for example making any continuous gradient and discretizing it into a dozen bins.

It is vastly more efficient to make palettes from scratch and rapidly vary them (on the target map) with sliders in Gimp. This and simple spreadsheet RGB and HSL builds have resulted in very large galleries of free palettes. Most of them come unannotated as to construction however but it's easy however to assess their uniformity as grayscale (luminance, lightness, and average are shown in some popular modern palettes like parula, viridis, geosoft, coolwarm, and cube-helix below).

Nevertheless, a lot depends on the type of climate map, the 2D statistical distribution of the variables being displayed, and what 'talking points' the map product is supposed to bring out. An interesting concept here, after treating outliers, is 'histogram equalization' that results in more or less equal use of each color bin for maximization of color discrimination.

Is it possible to eliminate the subjectivity in palette choice by having its construction data-driven? For example, if grid nearest neighbor variance is high, then contouring will not be effective or even feasible. If it's low, then a binned palette might be effective; indeed the optimal number of colors might be inferable. Similarly, mean and std deviations might allow rational setting of outlier lumping.

Here are some of the better discussions, online palette tools, and palette collections I've come across. The graphics can be downloaded and the palettes imported into Panoply as described above.

https://earthobservatory.nasa.gov/blogs/elegantfigures/2013/08/06/subtleties-of-color-part-2-of-6/

http://geophysicslabs.com/2016/07/04/histogram-equalization-in-python-and-matplotlib/

http://blogs.mathworks.com/steve/2014/10/20/a-new-colormap-for-matlab-part-2-troubles-with-rainbows/

http://hclwizard.org:64230/hclwizard/

http://www.research.ibm.com/people/l/lloydt/color/color.HTM

http://www.mrao.cam.ac.uk/~dag/CUBEHELIX/cubetry.html

http://www.mrao.cam.ac.uk/~dag/CUBEHELIX/cubewedges.html

https://www.ncl.ucar.edu/Document/Graphics/color_table_gallery.shtml
« Last Edit: September 18, 2017, 03:48:12 AM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #29 on: September 18, 2017, 01:21:52 PM »
Given the data, our forum posters make excellent line graphs, which adds analysis and thus value. We do some original work with satellite imagery from portals like Worldview, Jaxa and Sentinel Playground. However we mostly copy-paste map products (even when they're poorly done) rather than make better ones ourselves.

That involves geolocated data and map generating software providing control of the projection, overlays, and especially the palette. The latter four are well-handled by Panoply. Climate science data is generally stored as netCDF files or convertible.

These are basically just collections of excel spreadsheet stacks but are put into a complex non human-readable binary format because file sizes can be immense and non-desktop systems must read them.

As a practical matter, to make a netCDF file on a desktop requires a CDL file and a code snippet called ncgen. There's currently no online tool that will do this, it requires leaving your desktop OS for command line. That's too bad because the CDL format is easy to read, edit, and make from scratch in a text editor using comma separated data numbers.

Panoply can export any .nc file to CDL but cannot reimport them, ie it implements ncdump but not ncgen. Most of the benefit of export can already be seen in the Source window panel "Show info in expanded form'. The array panel in a map shows the numbers in its grid.

Panoply distinguishes geolocated Geo2D items in a netCDF file from generic 2D files. Both are plottable but only the former map onto the globe via a chosen projection. It's unclear how latitude and longitude data in an .nc file come to be associated with array data to convert it from 2D to Geo2D.

So with editing, creating and importing of palettes and data arrays mastered, we just need ncgen and a Geo2D description to be able to fix poor online maps or make our own from scratch using satellite data or other resources. Poster wipneus has, in effect, a workaround to do this in the complex instance of piomas data.

Note the UH AMSR2 3.125 km archive (Arc_20170912_res3.125_pyres.nc etc) doesn't arrive as Geo2D because the sea_ice_concentration variable is missing an 'attribute' that tells Panoply of the gridding scheme. However the land variable is seen by Panoply as a Geo2D variable that can be plotted on a map. That's because it includes a coordinate 'attribute' that explicitly names the auxiliary 2D latitude and longitude grid variables. The sea_ice_concentration variable is missing that coordinates 'attribute'. (Here 'attribute ' has a technical meaning within the format prescription -- thx to RBS here!)

Here are the most useful links I've found to date on ncgen and CDL formatting. Some of it makes sense but it's fair to infer that many code manual writers were not English lit majors in college.

http://www.unidata.ucar.edu/software/netcdf/docs/examples1.html

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html

http://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/CDL.html

http://www.unidata.ucar.edu/software/netcdf/workshops/2011/index.html

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html

https://badc.nerc.ac.uk/help/formats/netcdf/index.html

http://www.ceda.ac.uk/static/media/uploads/ncas-reading-2015/07_ncgen_ncdump_cdl.pdf
« Last Edit: September 18, 2017, 06:18:20 PM by A-Team »

Dryland

  • ASIF Lurker
  • Posts: 6
    • View Profile
Re: Creating Animated GIFs
« Reply #30 on: September 18, 2017, 05:41:17 PM »
Hi all,
I recently contacted A-Team and offered some programming help for the work he's doing with the ESRL products. I see an opportunity to automate some of the steps he's been doing manually, so that, for example, each time there's a new ESRL drop, a short time later there are new visualizations available to view on the web. 

Toward this end, I've written a Python program that performs these steps each time it's run:

(a) Check one or more FTP sites (such as ftp1.esrl.noaa.gov) for files that are new or changed since the last run.

(b) download those files, and recursively unpack any archives (such as .tar.gz files).

(c) walk the tree of new files and apply a set of rules (which are based on regular expressions, and are read from a config file) to transform those files into a tree of output files. The set of operations is expandable as needed. At the simple end, rules allow ignoring files of no interest, copying files as-is to the output tree, renaming them, and putting them in a different directory structure. The next level is to change image formats, add text to images, create thumbnails, build animated .gifs, and so on, using any of the available tools (I'm using ImageMagick at the moment). And the top tier is to do real data manipulation and analysis, and automate tools like Panoply to create new visualization products.

(d) upload the output files to a web server. I've configured a static website using Amazon S3 as a test bed. The HTML is also automatically generated from the data so it also stays in sync with the latest changes.

My proof-of-concept is hosted at http://climate-analyst.s3-website-us-west-1.amazonaws.com/files.html; all it does at this point is a selective copy of a few ESRL files, wrapped in a bare-bones HTML page for access. I plan to keep this URL alive and will have more to show there as this system evolves.

I'm happy to share ideas and code with any forum members who are interested.

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #31 on: September 18, 2017, 06:32:01 PM »
Awright, Dry! Hopefully you'll get some helpful feedback from the other python people here.

A new command line version, PanoplyCL, is available for Mac and Windows at https://www.giss.nasa.gov/tools/panoply/betax75/ along with a not-so-new manual.

PanoplyCL could be quite important for us in automating the transfer of daily ESRL products into stable links and images that work under constraints of the forum. I have just installed it (after a minor hassle activating Java v8 update144) and saved my first script, which has the extensions .pancl.pcl  It is opened as text below and looks quite straightforward.

Basically it seems a fancy way of setting preferences separately for each map in a collection of Geo2D files so the same parameters can be run to make the next day's map the same way. This amounts to making the frames of an animated gif using the ESRL archive.

Going forward, ESRL + Panoply are probably the best things we have going for comprehensive 2017 freezing season forecasts, though specialized meteorological and oceanographic products may be better at certain narrower tasks.

A collection of ESRL's D0 initial states could serve as an improved seasonal archive (even though not hindcasts or reanalysis); we don't know yet how good they are but they allow easy and precise comparison to competitive products if any should surface that play by climate science rules.

// BETA BETA BETA
// Script parsed by PanoplyCL to create a plot. Usage:
// java -jar PanoplyCL.jar aice_h_in_REB.2017-09-11.pancl.pcl

// Open a dataset.
var ncdata1 = panoply.openDataset ( "/Users/a-team/Downloads/REBS nc/REB.2017-09-11.nc" );

// Select a variable.
var ncvar1 = ncdata1.getVariable ( "aice_h" );

// Create the plot.
var myplot = panoply.createPlot ( "lonlat", ncvar1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Specify plot settings.
myplot.set ( "size-factor", 200 );
myplot.set ( "size-width", 100 );
myplot.set ( "size-height", 50 );

myplot.set ( "title-text", "ice area  (aggregate)" );
myplot.set ( "font-master", "SansSerif" );

myplot.set ( "color-background", "black" );
myplot.set ( "interpolate", true );

myplot.set ( "color-invalids", "rgb(96,96,96)" );

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", false );
myplot.set ( "scale-outlier-shape", "NONE" );
myplot.set ( "scale-outlier-side", "both" );
myplot.set ( "scale-outlier-gap", "thin" );
myplot.set ( "scale-tick-size", 7.0 );
myplot.set ( "scale-label-location", "BELOW" );

myplot.set ( "scale-min", 0.0 );
myplot.set ( "scale-max", 0.9999876618385315 );
myplot.set ( "scale-div-major", 9 );
myplot.set ( "scale-div-minor", 4 );
myplot.set ( "scale-exponent", 0 );
myplot.set ( "scale-log10", false );
myplot.set ( "scale-label-custom", false );
myplot.set ( "scale-tick-format", "%.2f" );

myplot.set ( "scale-minmax-note", true );
myplot.set ( "scale-minmax-format", "Same" );

myplot.set ( "grid-spacing-lon", 15.0 );
myplot.set ( "grid-spacing-lat", 15.0 );
myplot.set ( "grid-label-vis", false );
myplot.set ( "grid-label-size", 6.5 );
myplot.set ( "border-weight", 114 );

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 );
myplot.set ( "proj-xparam-2", true );

myplot.set ( "overlay-1-weight", 75 );
myplot.set ( "overlay-1-name", "Earth_mask.gif" );
myplot.set ( "overlay-1-color", "rgb(204,255,214)" );
myplot.set ( "overlay-1-invert", false );
myplot.set ( "overlay-2-weight", 0 );

myplot.set ( "proj-shading-vis", false );

myplot.set ( "grid-weight", 50 );
myplot.set ( "grid-style", "NONE" );
myplot.set ( "grid-color", "rgb(64,64,64)" );

myplot.set ( "contour-weight", 75 );
myplot.set ( "contour-style", "None" );


// Save plot image to disk.
myplot.saveImage ( "PNG", "aice_h_in_REB.2017-09-11.png" );
« Last Edit: September 18, 2017, 07:23:21 PM by A-Team »

Dryland

  • ASIF Lurker
  • Posts: 6
    • View Profile
Re: Creating Animated GIFs
« Reply #32 on: September 19, 2017, 02:00:45 AM »
I replicated this, and it works for me.

I installed PanoplyCL on linux (Ubuntu 14.04) and also had Java version problems - "UnsupportedClassVersionError, Unsupported major.minor version 52.0". The cure was to install Oracle's JDK 8:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

I then ran your script and got the same image back. Nice!

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #33 on: September 19, 2017, 01:15:12 PM »
ran your script and got the same image back.
That is just awesome!

It was fortunate that I happened to use one of the default Panoply palettes here, CB_Blues.cpt, before exporting the script. If the palette had been set to my favorite for ice area aice_h, namely the custom palette AMSR2.rgb (whose file and easy installation is described above), the script would have stalled out unless you too had put it in your palette collection.

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-colorbar", "AMSR2.rgb" );

While Panoply offers a very nice collection of stock palettes, the reason for AMSR2.rgb is to let ESRL seamlessly pick up from UH AMSR2 high resolution observational data in the same color scheme. That is, the UH data only goes up to the day before whereas ESRL forecasts ten days out from there.

Alternatively, if the UH .nc can be reworked from 2D to Geo2D allowing polar stereographic projection, its palette could more simply be replaced by a similar stock palette in Panoply. The UH archival images fortunately are in standard 'Greenland down' orientation and just need a 39% scale reduction in Gimp to match the scale on the King plot.

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 ); //this sets lowest latitudinal circle displayed.

myplot.set ( "size-factor", 240 ); King size plot setting

I'm using the King 240 scale on Panoply so that it will output .png images with the Arctic Ocean fitting in ~700x700 pixels, so after cropping in Gimp it makes maximal use of what the forum software will display as an animation. I don't want to rescale later in Gimp because interpolation messes with the colors; right now just palette colors are used in the map pixels.

Smaller thumbnail animations lose detail; larger animations will run after a click to a new window but counts show few forum visitors take that extra step. File size is definitely an issue with the 40 time slices in the ESRL data.

Note the image the script makes is only the 1st frame out of 40. That is indicated in the 3rd coordinate slot in the script. In exporting an animation, Panoply runs through all 40 of these, in effect bumping the step the internal script it is running. So externally, we need a super script that runs, bumps the step by 1, runs, ... stops at 40 steps (or however many ESRL has provided).

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 5 of 40
myplot.setVarDimension ( 1, 1, 5 );

Two code snippets -- ncdump and ncgen -- are absolutely critical to working with netCDF file format. these interconvert between machine-readable .nc and human-readable and editable CDL plain text. By opening an .nc file and exporting to CDL, Panoply is providing ncdump already without a command line environment. Oddly it cannot convert its own CDL file back to the .nc that it just made it from, which would amount to ncgen.

The link below gives a friendly explanation of how to get ncgen running on a Mac. It's a lot of trouble to get one menu item, involving a massive download and installation of Xcode, MacPorts, X11 and establishing directory paths. At that point, command lines for ncgen can be run on a local CDL file will make it into a netCDF that Panoply can import: http://mazamascience.com/WorkingWithData/?p=1474

-- Generate a NetCDF file
ncgen -o mydata.nc mydata.cdl

Generate code that will produce a NetCDF file
 – Fortran:
ncgen -f mydata.cdl > mydata.f

 – C:
 ncgen -c mydata.cdl > mydata.c

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #34 on: September 19, 2017, 01:57:48 PM »
To facilitate visual ice thickness comparisons of ESRL to Piomas, it's good to map up ESRL's view in the color palette that wipneus has been using on the thickness forum. There are 16 colors there whose rgb values are readily captured in Gimp from a large-format posted graphic and imported into Panoply.

It's a good idea to number imported palettes, say 11wipiomas and 10AMSR2UH so that they appear in order conveniently at the top of the stack in Panoply alphabetization. To import, save with the .rgb tag from a text editor and Open that file in Panoply.They are easy to delete just by a Cut command.

The animation compares mid-August Piomas as prepared by wipneus, mid-Sept ESRL ice thickness as prepared in Panoply with the same palette, and mid-Sept AMSR2 sea ice concentration from the UH archive..

ncolors= 16
#  r   g   b
0   166   0
25   175   0
50   184   0
83   193   0
116   202   0
151   211   0
189   220   0
230   230   0
231   206   29
233   189   58
234   179   88
236   177   118
237   182   148
239   194   179
241   214   211
242   242   242
« Last Edit: September 19, 2017, 05:39:24 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #35 on: September 19, 2017, 06:22:17 PM »
Gimp (or ImageMagick) will play a key role in pipelining ESRL netCDF products onto the forums. While the ESRL graphics are excellent -- as are the Panoply maps produced from ESRL netCDF files -- they are both too large to display properly on the forum.

That's mainly because the Arctic Ocean is quite lopsided with respect to the North Pole, eg the Bering Strait is much further south than Svalbard, yet forum maps need to be centered on the Pole for comparative purposes.

Note further ESRL plots only a fraction of their variables and those in certain combinations. Other variables no doubt play a role in their model calculations but aren't used for standalone graphics. For these, we can make Panoply maps if there is sufficient interest in the variable, eg rainfall, snow depth or compressive strength of the ice pack.

There's an easy way to in Gimp assemble a master graphic for all the Geo2D files in REBplots, REB_nc, and RASM-ESRL archive bundles. (These make sense to mix together only when the animations would be the same length, eg 20 or 40 frames.) From this master graphic, any combination of variables can be exported out as a gif animation.

The issues here are (1) each variable has its own legend, title and palette (2) some variables have interesting max and min variation by frame (which Panoply provides), (3) the same time stamp is shared by all the variables, (4) vectors from a first variable can float over the map of another variable, and (5) some variables apply only to the ice (thickness), others just to open water, and still others to both. This latter fact (and the ambiguous use of zero) require development of an evolving 6-hour open water mask and mask edge.

The process of making a master graphic is fairly simple. First, Panoply scripts generate maps at the same favorable scale for all the variables. Each variable has a different palette appropriate to what it is supposed to convey. Other script settings may also differ slightly, for example log vs scalar. These output pngs are cropped identically using fixed UL and LR corner coordinates. Next the frames are tiled up into one wide image.

Other maps from other variables are then tiled up too and put into the growing master receiving image (for the given D0 dte) as layers. The same is done for the time scale, the palettes, the scales, the legends and the varying max and min data. Open water mask, boundary outlines, and alpha transparencies are then constructed as yet more layers.

The master image is then ready to have various layer combinations constructed by activation and layer ordering. A new-from-visible layer for each desired output is then sliced back down into forum sized frames. These pngs are then saved to animated gifs at whatever frame rate is effective for them and uploaded to a forum post. While some manual intervention and operator aesthetics may be necessary, automating daily ESRL output up to this master file is quite feasible.

Some examples are cross-posted below:






A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #36 on: September 24, 2017, 12:46:20 AM »
This post summarizes forum-relevant highlights of the PanoplyCL scripting manual, as applied to the 49 Geo2D files in the 3 daily ESRL download folders (REB_plots, REB, and RASM-ESRL_4NIC).

PanoplyCL, while written in Java, uses unrelated Javascript as its scripting language. (Java 8 can internally parse and run Javascript.) Not everyone wants to learn full-on Javascript so the goal of this post is to collect the minimal number of code snippets sufficient for the typical forum use cases or alternatively develop tricks for staying within the more intuitive menu-driven GUI environment.

The scripts are actually saved out of the GUI as say vvel_h_in_REB.2017-09-20.pancl.pcl (ie variable_in_netCDFsourceFile.pcl rather than myscript.pjs as in the draft manual. While this may suggest PanoplyCL (Command Language) is a variant of Javascript, later on conventional Javascript snippets are added, suggesting they run as they would anywhere.

As explained up-forum, someone wanting to make a simple map or multi-frame animation just  experiments with Panoply settings until they generate a satisfactory map. Exporting the CL script allows that map to be regenerated later by anyone running a copy of the script, typically on the same Geo2D line within the same netCDF but equally from the next day (or older archived dates) if the file name is bumped in the first line, eg the script vvel_h_in_REB.2017-09-20.pancl.pcl would draw the same map but from the appropriately dated data with a simple script revision:

var ncdata = panoply.openDataset ( "/Users/Fred/REB.2017-09-20.nc" );
-->
var ncdata = panoply.openDataset ( "/Users/Fred/REB.2017-09-21.nc" );

Had "Save Plot Settings as Preferences" been invoked, this could have been done without delving into script particulars, provides some other map prescription hadn't overwritten it. This suggests that simply allowing Preferences to be saved under a name, along with Preference navigation to saved names (like Plot Size menu) would let the user stay within the GUI. For now, representative maps can be held open and used to overwrite Preferences as needed. However this rapidly becomes unwieldy with 49 Geo2D maps from ESRL alone.

A simple script edit would output the map .png in a different preferred palette by simply stubbing in the new palette name:

myplot.set ( "scale-colorbar", "CB_RdBu.cpt" );
-->
myplot.set ( "scale-colorbar", "bluescale.act" );

To help with palette choice, it might be advantageous to enable scrolling through the available palettes by category. Right now, palettes are sorted alphabetically which can group related palettes but doesn't always. Other sort options are by number of distinct colors, from 'continuous' (256) to binned (typically 8, 9, 10)., or by sequential vs divergent. Once the palette is sorted to user preferences, a keyboard up or down arrow could quickly cycle the map through related palettes, with the option key reversing color order. Right now, there is endless mousing of the awkward popup palette.

The script that the current PanoplyCL beta saves from a time series merely generates the first frame, whereas ESRL provides forecasts as ten days at 24 hr steps, five days at 6 hr steps, or ten days at 6 hr steps. Those 'strides' can be adjusted to larger intervals at the time of saving an animation to make fewer frames (eg first and last) but saving the script won't capture that or the time stamps that the frames need.

However it could if Export Animation... dialog had an additional checkbox for Save Script As. That would save users from delving into the details of how JavaScript loops through variables. This does not generate an error message if the script is mis-applied to a file that has no additional times.

However a common situation in climate science might be air temperatures both by date and elevation (or ocean temperatures by date and depth). It would be unusual for an animation to first circulate through the dates at a set elevation and then through the dates at the other elevations. So the dialog needs simply build out the slice options as radio buttons from the file variables.

However for thumbnail maps, it might be feasible to stack the elevation slices (though this is easily done later in Gimp). Alternately, separate time animations could be output for all the elevation choices by a dialog setting. However there isn't any mechanism in the gif89 specification to synch them on a web page so that they all display the same date as the animations go forward.

In summary the current beta of PanoplyCL is 'animation-unaware' and all the saved scripts look the same except for palette choice, image size, and minor appearance settings. However the PanoplyCL manual provides a fix for that with Javascript examples.

The Javascript add-on deduces the frame count for a given netCDF and loops through it using fixed settings from the rest of the script, saving them out as a folder of pngs. Even better, these pngs have their time stamp as a sub-title. The user can then load them into an ImageJ stack or Gimp layers and export out a forum-ready gif animation.

Note that the directory path to the file has to be provided so the script can find it. On a Mac, right-click on the file, then hold down the Option key: the menu will offer copying of the pathway.

var ncdata = panoply.openDataset ( "/Users/Fred/Downloads/REBS_nc/REB.2017-09-20.nc" );
var ncvar1 = ncdata1.getVariable ( "aice_h" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
// Set to the first frame of 40, the 1 in third coordinate position
myplot.setVarDimension ( 1, 1, 1 );

var tdim = ncvar.getDimension (1);
var tlength = tdim.getLength ( );
for (var i = 3; i <= tlength; ++i)
{
 myplot.setVarDimension ( 1, 1, i );
 myplot.set ( "subtitle-text", "Time step: " + tdim.getStrValue ( i ) );
 var istr = "" + i;
 while (istr.length < 3) { istr = "0" + istr; }
 myplot.saveImage ("PNG", "T_" + istr + ".png");
}

No further examples of more complex scripts are provided but the manual states these could include "if-then constructs, iterative loops, string concatenation, create multiple plots, changing plot dimensions and appearance settings as you create each plot."

In other words, it is quite feasible to run a single script that processes all 49 Geo2D files into their daily png sets. These could use different palettes and be of different sizes and formats.

As a practical matter, someone has to go through each of the Geo2D files, decide which are useful at what scale, whether linear or log scaling is appropriate, whether a sequential or divergent palette makes sense and so on, saving each out as a script. These are then concatenated as the master script. To make changes on a particular Geo2D, it is simply re-run with the new parameters and the old version replaced.

It might be easier to make and edit the master script from a 'mail merge' setup. The  database is simply the matrix of Geo2D x parameters. It merges into a text document that is the script; the concatenation of all the text documents is the master script.

To avoid typos and unacceptable parameter values, that could be done via a web front end that forces menu choices to admissible, like plot size, file name, and most importantly palette.

The manual also provides examples for all the other map situations such as zonal average, generic 2D, combined array plots and so forth. Again, it seems there is little gained over simply getting the display set and then saving to a Preferences to be called up for future use.

"5.5. Running PanoplyCL in a Server Environment. Because PanoplyCL runs headless, it has potential to be used on a web server to create graphics based on input received from a remote user. Although this is indeed a possibility, note that there are limitations."

"PanoplyCL does not run continuously but must be invoked each time that you want it to process a script and create a graphic or graphics. The most time-consuming part is initializing Java graphics handling. Repeatedly calling PanoplyCL to create and save a single plot image each invocation is very inefficient compared to calling it once with a script that saves multiple plot images."
 
For security reasons, it is far better to use a pre-written script that can be altered to use different predetermined options rather than insert any content that may have been provided by the user."

https://www.giss.nasa.gov/tools/panoply/betax75/PanoplyCL_Guide_20161109.pdf

Dryland

  • ASIF Lurker
  • Posts: 6
    • View Profile
Re: Creating Animated GIFs
« Reply #37 on: September 25, 2017, 01:39:24 AM »
A-Team,

You've highlighted some important issues. The question I go back to is, who are the target users, and what do they need? I imagine multiple clusters:

a) Forum visitors who want a graphic they can use to make a point or ask a question, but who aren't interested in spending all day, or doing any sort of scripting. As you say, a menu-driven GUI environment is a must for this audience; but rather than expecting them to tackle the Panoply learning curve (daunting enough even without the scripting aspect), I'd be looking for more canned scenarios where they can specify a few variables using a web form and have a PanoplyCL script generated (more or less by template filling) and run for them, resulting in a hosted image that they can view/download/post. The "web front end" you mention seems to be along these lines.

b) Users who are willing to take the plunge, install Panoply (GUI version) and learn to use it to make more sophisticated graphics in a one-off fashion. You've discussed some of the ways their work can be captured and then automated. I think some expert assistance there would make more sense than expecting these users to have a way to add new automated, daily products to the pipeline themselves, without help.

c) Users who want more control and are willing to do some "casual" scripting mixed with manual operations. I'm not sure these users were done any favor by the choice of Javascript as the scripting language for PanoplyCL. I chose to work in Python for my ESRL automation, for two main reasons: (1) the scientific community in general seems to really like Python - lots of scientists have gotten pretty good at it; and (2) Javascript hides some nasty sharp edges within its casual-friendy surface. Now, it may be that the subset of Javascript that matters for PanoplyCL scripting wouldn't run into those issues (e.g. callbacks). The script above isn't that bad. And some people just love a challenge, and will keep trying until they get what they want. How to help these people? We could allow them to upload scripts to be run on PanoplyCL on a server, but that's risky - not just security as you quoted, but also more innocent problems like scripts that run forever and take up 100% CPU time. It would be better to let them develop on their own machines, then review and adapt their scripts if we want to add them to the automated pipeline.

d) A-Team. :)

For now, I have my hands full with the needs of group (d). Later on, I would like to see some other users raise their hands and say what they want.

Neven

  • Administrator
  • ASIF Governor
  • *****
  • Posts: 4218
    • View Profile
    • Arctic Sea Ice Blog
Re: Creating Animated GIFs
« Reply #38 on: September 25, 2017, 10:03:31 AM »
One of these days I'm going to read through this thread again from start to finish, and will then hopefully fully understand what it is you guys are doing with what, and maybe then I can be a target user who is able to express what he needs.

Sorry for not understanding already. My head is too full. I know you're doing awesome stuff, grade A citizen science.
Il faut cultiver notre jardin

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #39 on: September 25, 2017, 01:06:35 PM »
not fully understanding.
The goal (not quite there yet) is to reduce the complexity down to just a double-click on a file or issuance of a single menu command, staying within familial graphical user interfaces though not supporting cell phones and not requiring command-line coding.

Panoply on ESRL data is just a double-click map now, once it is defaulted to stereographic projection and a rainbow palette. However there's more to it in terms of forum-ready accurate scientific visualization. That's what these scripts are supposed to be doing sight-unseen.

Below is an intriguing Panopoly JS option that would let us do fancier presentations, like the ones we see from NASA's visualization studio. There's a valid question (bonbons vs kale) in whether wandering off into eye candy (which might better engage people in climate change) should substitute for conveyance of unperturbed scientific data (ie gif frames in NSIDC projection).

The PanoplyCL manual has a fairly complicated prescription for putting Arctic Ocean data on a spinning globe as viewed from a hovering spaceship. However, as with a BMW, the complications are all under the hood. After some choices are made, it is all pasted somewhere never to be seen again but ready to run any time on different data.

The following script makes a sequence of maps using orthographic projection which displays the northern hemisphere as it appears from a distance. The center of projection is progressively stepped, effectively rotating the globe, saving an animation frame in half-degree steps. Note that without the sample file in hand, the script won't actually run as is. It would make sense also to import a better background earth map than the ones provided.

It does not provide zoom though the option to save frames to kmz format could allow this in Google Earth. That's discussed in section 3.5.4. PCLPlot.saveAsKmz (str, boolean, boolean, boolean). These files do open up in GoogE centered on the north pole and oriented correctly but as miniatures of far too small a scale. It's unclear whether this is a bug in GoogE or in the Panoply export.

var ncdata = panoply.openDataset ( "/Users/jsmythe/cam_data/b20.T.0410-0419.nc" );
var ncvar = ncdata.getVariable ( "T" );
var myplot = panoply.createPlot ( "lonlat", ncvar );
myplot.setVarDimension ( 1, 1, 1 );
myplot.setVarDimension ( 1, 2, 26 );
// Paste in here the 30-40 lines of myplot.set that specify the plot's appearance.
myplot.set ( "proj-name", "Orthographic" );
var plon = 45.; // longitude of rotation pole
var plat = 60.; // latitude of rotation pole
// Define the rotation angles and matrices
var betaRad = - toRadians ( 90. - plat );
var gammaRad = toRadians ( plon + 180. );
var cosBeta = Math.cos ( betaRad );
var sinBeta = Math.sin ( betaRad );
var cosGamma = Math.cos ( gammaRad );
var sinGamma = Math.sin ( gammaRad );
var rotY = [ [ cosBeta, 0., -sinBeta], [ 0., 1., 0. ], [ sinBeta, 0., cosBeta ] ];
var rotZ = [ [ cosGamma, sinGamma, 0.], [ -sinGamma, cosGamma, 0.], [ 0., 0., 1.] ];
// Rotate the sphere.
for (var i = 1; i <= 720; i++)
{
 var lll = 0.5 * i;
 var lllRad = toRadians ( lll );
 var p0 = [ Math.cos (lllRad), Math.sin (lllRad), 0. ];
 var p1 = [ rotZ[0][0]*p0[0] + rotZ[0][1]*p0[1], rotZ[1][0]*p0[0] + rotZ[1][1]*p0[1], 0];
 var p2 = [ rotY[0][0]*p1[0] + rotY[0][2]*p1[2], p1[1], rotY[2][0]*p1[0] + rotY[2][2]*p1[2] ];
 var lon = toDegrees ( Math.atan2 (p2[1], p2[0]) );
 var lat = toDegrees ( Math.asin (p2[2]) );
 // Apply center of Orthographic map projection.
 myplot.set ( "proj-lon0", lon );
 myplot.set ( "proj-lat0", lat );
 // Save plot image to disk with filename based on iteration index.
 var istr = "" + i;
 while (istr.length < 3) { istr = "0" + istr; }
 myplot.saveImage ("png", "T_" + istr + ".png");
}
// Function that converts a value in radians to degrees.
function toDegrees(value) { return value * 180. / Math.PI; }
// Function that converts a value in degrees to radians.
function toRadians(value) { return value * Math.PI / 180.; }

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #40 on: September 27, 2017, 01:29:20 PM »
Minor news from correspondence: the read/write command-line code snippets for netCDF files are called ncdump and ncgen. The former is implemented already in a sense in Panoply in that an .nc file can be opened and saved out as CDL*, the human (or excel) readable version of netCDF.

Here the * on CDL* means the specific Java library used by Panoply for interacting with netCDF may insert extra curly braces in multi-dimensional data that are not strictly speaking part of the CDL format. In consequence, ncgen may not operate properly on Panoply's export to regenerate the original .nc file.

The Java library itself lacks code to implement ncgen. That would be a massive project to write from scratch; Panoply is essentially a shell that harbors as many pre-written open source Java libraries as possible.

Thus export from Panoply is one-directional; data can be visualized, edited, and processed for statistical attributes (ie for outliers, mean, variance and palette optimization) but not re-imported without addressing curly brackets (if present) and using command-line ncgen.

However users can study the header sections of CDL* files, make one appropriate to their situation, stub in their own data from excel, and if compliant, run ncgen and import into Panoply.

Scripting in PanoplyCL was originally motivated by situations where people needed to be able to go back and re-create plots with modest changes. The classic case was a scientist at Goddard who included 40-50 Panoply plots in a journal paper submission and had to re-do them six months later following referee criticisms. Had scripts been saved, small edits could have made the fixes.

I asked about re-doing the unwieldy palette menu as a simple database with sortable fields like alphabetic, sequential, divergent, perceptual; by number of colors (odd, even, 8,9,10,… 256); or by most recently used.

Something will be done about the palette menu down the road. For now, some not-yet-documented palette navigation is possible by typing (eg R for red). This can be effective, depending on if palette names were set informatively.

I asked about palette scrolling: up and down arrows in the Scale tab, acting on the user’s current sort order, visualized in Panoply's map view. This could help find the best palette as it's difficult to anticipate in advance how a palette will look in an animation.

 A JavaScript that walked through the top ten in palette list (as sorted in advance by the user) would be easy enough, with the output of stills then displayed as layers in a Gimp animation.

Panoply works within 8-bit color (alternatively called indexed color, look-up tables (LUTs), or color tables) but exports out into ordinary 24-bit png files. This makes sense because 256 colors is far more than can be distinguished on most climate science maps; indeed ColorBrewer recommends 8-9 as the upper limit if categories are established. However 'continuous' palettes can provide an attractive sense of smoothness to data display with enough precision to discourage binning down.

NCAR, an atmospheric research center in Boulder CO, provides an extraordinary gallery of 438 color tables classified into 31 categories. There is some overlap with the 100 palettes that come with Panoply but together these make a great start on a palette database with applications to climate science graphics.

Each of the NCAR palettes is shown in action on a set of earth maps (2nd image) that can be expanded to a decent scale and include an Arctic Ocean view (3rd and 4th images, needs clicks).

Indexed color is supported by Gimp but only as a sideline. The dialogs are scattered all over the interface, with editing, re-ordering, and substitution possible but awkward, while the provided set of default color tables is clueless. However it is easy to make and add custom palettes with the excellent gradient tool.

ImageJ takes a more scientific approach to color tables and allows quick scrolling through a well-chosen set of furnished look-ups. It also allows export of numeric rgb tables which is essential in coordinating import to Panoply.

http://www.ncl.ucar.edu/Document/Graphics/color_table_gallery.shtml
« Last Edit: September 27, 2017, 01:55:16 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #41 on: September 28, 2017, 11:48:37 PM »
The NCAR palette gallery contains an impressive 436 palettes classified into 31 groups. However 195 are reduncant, appearing in multiple locations, leaving 241 unique choices. The urls for these palettes, png files of size 500 x 101 pixels, have sensible file names, meaning ImageJ can load them from a list into a stack.

That stack can be exported as an animation into Gimp which allows very rapid scrolling through the possibilities for coloring a given map. They can also be concatenated into a single image. That is convenient for determining which are linear in lightness or in perceptual luminance (CIE color space L).

These give equal perceptual color bins, unlike the notorious rainbow palette. The test is, upon gimp 'desaturate to luminance', whether the palette descends to a monotonic increasing grayscale. Most are immediate rejects and the others can be converted to a numeric list of grayscales in ImageJ and tested with spreadsheet subtraction for negative numbers (non-monotonicity).

The database of NCAR palettes, as processed above, is appended below as a text file that is really csv. A set of unique palettes, in alphabetic order, is also attached as a scrolling list. The original NCAR order is also provided.

Some issues remain as ImageJ cannot export a png animation (see https://en.wikipedia.org/wiki/APNG and https://ezgif.com/help/making-animated-png) and instead struggles with a master color table of 256 that cannot possibly for all the gif layers. Each frame in a GIF specification animation can have it own separate 256-color palette and each color table at NCAR and Panoply is compliant with this, there's no 24-bit color here so a gif animation of a palette gallery should work..

The forum software only supports a handful of old formats, not including animated pngs. These run just fine on web browsers such as Opera, Chrome and Safari. Gimp has a plugin for APNG but it seems somewhat problematic. The situation is better for ImageJ.

The current PanoplyCL will only retrieve locally stored palettes with the 'Open' command in the File menu. Remote urls leading to palettes do not work. Consequently it appears necessary to download all the individual NCAR palettes as individual files. However, one-time check, their site does not allow ftp connections so this is fairly tedious. However, individual palettes of interest are easily loaded up in Panoply as temporary or permanent active palette.
« Last Edit: September 29, 2017, 12:09:44 PM by A-Team »

Dryland

  • ASIF Lurker
  • Posts: 6
    • View Profile
Re: Creating Animated GIFs
« Reply #42 on: September 29, 2017, 04:34:39 AM »
Time for an update from me. Progress can be viewed at http://climate-analyst.s3-website-us-west-1.amazonaws.com. It's still only a proof-of-concept, but a lot has changed behind the scenes.

Under the first link are the most recent REB_plots files from ESRL. I'm running a cron job to launch my program several times a day, so new images will appear each time ESRL updates their FTP site.

The second link has daily images created automatically by downloading the latest REB.(date).nc file(s) from ESRL and running them through PanoplyCL. This is still using the example script that A-Team shared, but there is now a templating engine in place that can rewrite the script before using it, replacing placeholders (such as [dataset] and [output_file]) with the right values for each invocation. This can be used to customize palette choice, scale, dimensions, titles, and all the other attributes that are scriptable. (And any number of different templates can be created for different tasks.)

The same templating engine is used to customize HTML templates to make the site pages. For example, in the template for the panoply.html page, the placeholder "list panoply" appears. (Square brackets omitted to avoid confusing the forum software...) When the real html page is generated, this is replaced by a formatted list of links to all the Panoply-created files that were generated earlier in the run. Everything is generated/uploaded without any manual intervention.

There's a framework in place for invoking command-line programs, so it is easy to add more. So far I have PanoplyCL and ImageMagick support, and will be adding things like Gimp, ncgen, etc. as the need arises.

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #43 on: September 29, 2017, 01:25:54 PM »
Awesome progress! Fascinating to see where all you go with this.

If ever there was a process that needed automating, this would be it. The daily roll-over is time-consuming for me yet it only makes sense to work with 'the latest'. It seems like ESRL outputs have settled down now, though some products still need fixes and more explanatory prose.

The PanoplyCL manual has scripts that cover almost every situation, notably scrolling through a variable parameter to make time series or height/depth slices. These are small add-ons to the script posted above. The core of all these scripts is setting basic display parameters, followed by whatever precious add-ons that do some modest AI.

I haven't been able to get clarification on why, given the current parameters of an open map can be saved to Preferences (overwriting whatever was there), there is no outside saving and reloading of parameter sets.

However if there are say five template maps open with five different displays, it is easy enough to switch back and forth and re-establish their preferences to make new maps with these preferences. Once those template files are closed though, their preference settings are lost.

Even if a CL script is saved out, there is no way to run that script except from command line. If the script just sets parameters, there's no way of loading them in. It does record them though for map replication later.

To me this is getting away from the whole point of Panoply, which is transparency and ease of use. There are users out there who may have great visualization skills but no inclination to coding. This is all about communicating out what may be buried in the data and improving on inept representations of it. Those users have a contribution to make.

If there were only some way of running the existing collection of manual scripts from the GUI menu interface, that would take care of 95% of the use cases and empower a talented citizen-scientist who has been frozen out of the .nc world. Admittedly, at some point there are big advantages to delving into javascript or controlling it remotely as with your master templating engine.

It seems like the choices are (1) enabling more of our 1300 members to make innovative displays with ESRL and other sources, (2) archiving providing just enough to reduce the work involved, or (3) doing everything imaginable for them without their engagement. There's no question that time spent making forum graphics is time not spent on their analysis, interpretation and commentary.

I have not yet delved into problems with the ncdump <--> ncgen roundtrip. The holdup seems to be the curly braces that CDL* inserts into multi-dimensional data. I haven't seen any multi-dimensional examples yet in ESRL so CDL* should be the same as CDL here, and ncgen should work on them.

There is some interest in resolving this so people here can make or edit excel spreadsheets into CDL into nc into Panoply maps. There may be some simple grep that straightens out the curly braces.

I'm still working on automating palette choice, or at least cutting the gallery options down by pulling up a half dozen appropriate ones for the user make the final decision. It is not feasible to run a map through 450 palettes and remember which ones worked. A lot of palettes show up completely unannotated; no description at all of how and why they were constructed.

The 'trending' palette today is even grades of perceptual luminance, that is, adjacent colors are discriminate-able by the human eye and precede evenly in the grayscale sense. However it's almost easier to make these from scratch rather than go through big palette galleries to determine their luminance attributes. On the other hand, with a kazillion people making climate science maps, some really awesome palettes have emerged regardless of underlying design motivation.

The CDL view is key to data-driven palettes, ones where statistical considerations of the data distribution impose restrictions on color numbers and range. Here the Panoply export gives a satisfactory start on that. That takes on another dimension optimizing palettes used in animations where between-frame statistical changes could be the driver. It's not clear though whether at the end of the day this would make any real perceptible improvement.
« Last Edit: September 29, 2017, 01:39:39 PM by A-Team »

A-Team

  • ASIF Upper Class
  • Posts: 1966
    • View Profile
Re: Creating Animated GIFs
« Reply #44 on: October 04, 2017, 11:02:49 PM »
Perceptual palettes offer three main advantages: better color discrimination by viewer, desaturation to grayscale for arithmetic operations between graphics, and proper display as (optionally draped) 3D surfaces. This does not resolve issues with contextual contrast: the 4 bars in fig.1 below are all the same shade, neutral gray 128.

ImageJ --> Analyze --> 3D Surface Plot, donated by KU Barthel, comes with two convenient macros, one that rotates the view around like the spinning globe above and another that works through a stack of images to produce a stack in surface view. http://imagej.net/plugins/surface-plot-3d.html

Drag and drop of files to the Mac dock is currently broken in ImageJ but it works to drag files to the control bar. This is a huge boon because ESRL plots of the same graphic from different dates can be fished out through file name search and loaded as an ImageJ stack which the macro can then convert to a 3D stack. (This requires all other extraneous images to first be closed.)

In Gimp, these images would all open separately. The workaround is tiling them up using the 'Filmmaker' controller (which allows open extraneous windows), slicing them back up, and loading as layers.

Gimp also allows easy analysis of palettes and indeed whole galleries of them. (Few collections note which palettes are perceptual, defined here as having an evenly stepped grayscale for the luminance channel when decomposed from RGB to CIE's Lab color space, or equivalently the desaturate-to-luminance command is used.

It's easy to convert bad palettes to perceptual ones: Color --> Components --> Decompose --> LAB --> replace L channel with grayscale gradient 0,255 --> Color --> Components --> Compose --> RGB. The other channels can be treated as well to further color diversity and discrimination as RGB. This process may or may not retain desirable characteristics of the original palette.

In multiplexing climate science displays, important for displaying related parameters simultaneously, a problem arises from the gif animation format. Gifs are indexed 8-bit color, 256 colors is the maximum allowed per frame (these can be a different 256 in each successive frame however).

If each climate parameter is displayed in its own continuous palette (256 colors), then overlays have to be dithered back down from 512 to 256 and so won't display properly. Here two 7-bit palettes of 128 each would work (or four 6-bit palettes of 64 each). The images and their palettes can then be properly displayed in multiplexed animations.

There's an easy way to whittle down palettes of 256 sequential colors to 128 and lower. First, the starting palette has to be confirmed. An 8-bit palette should come only in 256, 512 or 1024 pixel widths. If not, some colors are wider than others. The color picker should select columns 1 pixel wide x the full height.

Many internet 'palettes' are dithered rubbish with tens of thousands of colors, often with boundary issues and internal demarcation lines. This can arise from saving as lossy jpg rather than superior png.

It's best pass to indexed color set at 256, then return to RGB to crop out a central line of 1 pixel height from these, resize with bicubic to 256 pixel width, resize to 40 pixels high (or more) using no interpolation, and then check with Info --> Colorcube Analysis to see how many colors it actually has.

Next, select all and used Edit --> Stroke Selection to create a border all around the palette using any foreground color not found in the palette. Now change foreground again and turn to Filters --> Render --> Patterns --> Grid to set up a vertical mask. The ones provided below expose only every 2nd, every 3rd, or every 4th color of your palette. Add an alpha channel if not already there; select the masking bar color and delete. Finish up with Image --> Zealous Crop; this strips out all vertical lines of constant color, leaving a palette with an optional 1 pixel border.

These condensed palettes still offer plenty of colors. Indeed the 'spacing' provides better discrimination than the original continuous palette, in addition to solving the multiplexing problem within gif format.

Note Colorbrewer2 and others offer much smaller categorical palettes in the belief that viewers really cannot distinguish between more than 8-12 colors. However using paired, abutted or internally ramped palettes, those numbers can be exceeded. Discrete colors in effect create contours without distinct contour lines. They can also pick out valid patterns in data and create artifactual ones.

Maps in discrete palettes can be scored, ie how many pixels of each color are used in the map
 (presented as percentage or line graph of histogram). By walking through the map's embedded palette first with the contiguous and then with the non-contiguous color picker, these numbers can be captured into a spreadsheet.

Alternatively, ImageJ can process the whole image into text RGB lists from which these same numbers can be extracted from background of boundary, lat-lon grid, land, and legend pixels (which must never use palette colors.)

More "best links" to palette design sites and palette collections:

http://tristen.ca/hcl-picker/#/hlc/12/1.09/D56E60/DAF669

http://tools.medialab.sciences-po.fr/iwanthue/

http://tools.medialab.sciences-po.fr/iwanthue/

http://jsfiddle.net/d6wXV/6/embedded/result/

http://hclwizard.org:64230/hclwizard/

http://www.research.ibm.com/people/l/lloydt/color/color.HTM

http://www.kennethmoreland.com/color-advice/

http://www.kennethmoreland.com/color-maps/

https://datascience.lanl.gov/colormaps.html

http://sciviscolor.org

http://sciviscolor.org/ColorMoves/

http://peterkovesi.com/projects/colourmaps/
« Last Edit: October 04, 2017, 11:25:19 PM by A-Team »