Arctic Sea Ice : Forum

AGW in general => Developers Corner => Topic started by: Jim Hunt on May 16, 2015, 09:05:10 PM

Title: Creating Animated GIFs
Post by: Jim Hunt on May 16, 2015, 09:05:10 PM
Following a request (,1149.msg52024.html#msg52024) from jdallen, here's a place to discuss how best to go about creating GIF animations (and who knows, maybe some other sorts also).

As mentioned in the original thread, I currently use ImageMagick (, but there are other possibilities. See for example this existing (if short) thread (,165.0.html) on ImageJ (

If I recall correctly A-Team is also quite fond of GIMP ( [PS - Here's Laurent's video] (

The next question is "which operating system?". Wipneus will no doubt suggest Raspberry Pi flavoured ( Linux, but that isn't necessarily everybody's cup of tea!

Finally, for the moment at least, which sequence of images should we use for the initial tutorial?
Title: Re: Creating Animated GIFs
Post by: Laurent on May 16, 2015, 09:24:09 PM
That video for Gimp a free software available here : (
How to Create an Animated GIF With GIMP (
Title: Re: Creating Animated GIFs
Post by: Jim Hunt on May 17, 2015, 02:02:22 PM
Personally I am not a Windoze fan. Hence is these circumstances I often use Cygwin ( to make Windows N work more like *IX.
Title: Re: Creating Animated GIFs
Post by: Neven on May 17, 2015, 03:12:34 PM
Unfortunately I am tied to Windows (for the software I use for work), so a few years ago I decided to purchase this software to make animated GIFs: Easy GIF Animator ( Fulfils my needs very well, as I like my animations to keep a small size so people with lower bandwidth don't have to wait ages to see it appear in a blog post.  But I'm sure there's more and better stuff out there.
Title: Re: Creating Animated GIFs
Post by: sidd on May 17, 2015, 06:41:21 PM
Has anyone suggested ImageMagick yet ? (

excellent tutorial, many examples. ImageMagick is open source, and can do many more things than animation. All from the command line ...

Title: Re: Creating Animated GIFs
Post by: Jim Hunt on May 17, 2015, 06:56:35 PM
Has anyone suggested ImageMagick yet ?

Yup. See post #1!

I'm happy on the command line, plus scripts for automation, but maybe others here are not?
Title: Re: Creating Animated GIFs
Post by: jdallen on May 20, 2015, 05:31:10 PM
Thank you gentlemen.  I shall have to apply some of this directly.  I'm currently browsing through prices for a replacement machine - my primary laptop is six years old and falling behind current software.

Gimp questions will follow.
Title: Re: Creating Animated GIFs
Post by: Jim Hunt on May 23, 2015, 07:04:07 PM
Espen asked how to install ImageMagick on Windows. To the best of my recollection I just followed the instructions here: (

Download the correct .EXE file, depending on whether your Windows installation is 64 bit or not, then run it.
Title: Re: Creating Animated GIFs
Post by: Espen on May 23, 2015, 07:39:13 PM
Havent got a clue my PC is a HP 8760w I7-processor and 16 GB-ram?
Title: Re: Creating Animated GIFs
Post by: Jim Hunt on May 23, 2015, 07:59:46 PM
I'm making some assumptions here, but if you go to "Control Panel", then "System", it should tell you whether you're running a 32 bit or 64 bit OS.

The 32 bit version of ImageMagick should run on either, but 64 bit's better if your OS supports it.
Title: Re: Creating Animated GIFs
Post by: ChadGreene on September 15, 2015, 10:35:33 PM
GIFs are pretty easy to make in Matlab.  Here's one I made using the seaice function in Matlab:

Title: Re: Creating Animated GIFs
Post by: A-Team on January 18, 2016, 09:50:46 AM
Here is a javascript alternative to animated gifs that gives the end-viewer quite a bit more control. The problems with gifs are that (1) there's no speed controller or freeze-frame tool it may be set to play too fast or too slow (necessitating download and fixing locally say in gimp), (2) gifs are stuck in 8-bit indexed color (256 per frame) whereas many cryosphere products (such as ice sheet velocity) need a full 24-bit gamut of colors supported by png and jpg, and (3) satellite photos such as Landsat are in 16-bit depth tif which is essential to contrast optimization.

N Holschuh, a grad student at PSU, developed this new interface which is designed to display co-registered layers in any sequence or rate that the end-viewer wants. He provides examples for NEGIS, Jakobshavn, Pine Island, Thwaites and four other glaciers, apparently as part of a glacier atlas project. Each glacier has layers for surface elevation, bed elevation, ice thickness, velocity (both linear and log scales, and a Landsat. Details such as color keys and dual coordinate systems (mercator and lat,lon) are nicely done.

Note this is a layer-viewer rather than a full GIS engine that allows basic math operations to act on pairs of layers. For that, transparency overlays and conventional gif animations, load into gimp or ImageJ. Note an individual layers themselves could each be animations, say a time series of calving front position. ( try an example or two before reading on.

The javascript is basically an alternative to a standard html click-map which goes to a new url depending on where you click on the base image. Here all the layers are cached temporarily on your computer. Upon mouse-over of the controller, the appropriate new layer loads instantly. The effect is seamless. (The controller won't work on the image below because  forum software does not support embedded html so visit the link above to see it in action.)

I've made quite a few changes to simplify the code and enlarge the display and controller to our maximal width of 700 pixels. It turns out to be easier to generate a controller an arbitrary number of layers with text names using a spreadsheet (rather than html tables)., second image below shows the set-up for 7 layers.

If you copy out the text attached below and save as html, it will correctly load Holschuh's layers for Zachariae/79N in northern Greenland in any web browser (provided you are online). To make your own display, replace the links to paths to your own images (which can be urls to forum images or anything on the web).

I've colored the text below to distinguish the overall html container (blue) from the javascript (purple), controller, and image urls that have to be changed.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link href="../general.css" rel="stylesheet" type="text/css" />
<script type="text/javascript">
function MM_preloadImages() { //v3.0
  var d=document; if(d.images){ if(!d.MM_p) d.MM_p=new Array();
    var i,j=d.MM_p.length,a=MM_preloadImages.arguments; for(i=0; i<a.length; i  )
    if (a.indexOf("#")!=0){ d.MM_p[j]=new Image; d.MM_p[j  ].src=a;}}
function MM_findObj(n, d) { //v4.01
  var p,i,x;  if(!d) d=document; if((p=n.indexOf("?"))>0&&parent.frames.length) {
    d=parent.frames[n.substring(p 1)].document; n=n.substring(0,p);}
  if(!(x=d[n])&&d.all) x=d.all[n]; for (i=0;!x&&i<d.forms.length;i  ) x=d.forms[n];
  for(i=0;!x&&d.layers&&i<d.layers.length;i  ) x=MM_findObj(n,d.layers.document);
  if(!x && d.getElementById) x=d.getElementById(n); return x;
function MM_swapImage() { //v3.0
  var i,j=0,x,a=MM_swapImage.arguments; document.MM_sr=new Array; for(i=0;i<(a.length-2);i =3)
   if ((x=MM_findObj(a))!=null){document.MM_sr[j  ]=x; if(!x.oSrc) x.oSrc=x.src; x.src=a[i 2];}
function MM_swapImgRestore() { //v3.0
  var i,x,a=document.MM_sr; for(i=0;a&&i<a.length&&(x=a)&&x.oSrc;i  ) x.src=x.oSrc;

<body onload="MM_preloadImages('','','','' (','[url=','[url=','','','')]','','','','' (','','','')[/url]]','','','','' (','[url=','','','')]','','','','' (','','','')[/url][/url])">
<img src="" ID="swapper" width="700" height="903" border="0" usemap="#Map">

<map name="Map" id="Map">
   <area shape="rect" coords="88,813,199,857" href="#1" onmouseover="MM_swapImage('swapper','','',1 (',1))" />
   <area shape="rect" coords="197,813,310,857" href="#2"  onmouseover="MM_swapImage('swapper','','',1 (',1))" />
   <area shape="rect" coords="309,814,415,858" href="#3" onmouseover="MM_swapImage('swapper','','',1 (',1))" />
   <area shape="poly" coords="413,814,526,814,523,837,469,837,469,859,413,859,413,815" href="#4a" onmouseover="MM_swapImage('swapper','','',1 (',1))" />
   <area shape="rect" coords="470,837,524,859" href="#4b" onmouseover="MM_swapImage('swapper','','',1 (',1))" />
   <area shape="rect" coords="522,813,636,858" href="#5" onmouseover="MM_swapImage('swapper','','',1 (',1))"/>   
Title: Re: Creating Animated GIFs
Post by: A-Team on January 19, 2016, 09:09:16 AM
I'm looking now at comparison of two images via a movable vertical slider. A common example of this is before/after satellite images of a glacier calving. The slider is more effective visually than side-by-side or flickering animation comparisons because the end-user can zero in on a specific region to compare.

Our problem has been getting sliders to display within forum software. Usually the slider is implemented in javascript but there are alternatives, CSS and html5, which may display correctly here. I've collected some links below which contain open source code that will do this; more testing in a bit. ( ( ( (
Title: Re: Creating Animated GIFs
Post by: A-Team on January 21, 2016, 01:33:16 PM
Below, I look at the moderately massive computing requirements to process all the flight sub-segments (GPS waystations) of Operation Icebridge (as served by Cresis) so that all the cross-over intersections with other flights can be called up from a database with all their ancillary data such as bearing, bearing with respect to flow, elevation, ice thickness, bedrock profiles and relevant parts of the radargramss cropped out.

The purpose is to facilitate reconstruct all basal upheavals in Greenland in 3D from all their 2D cross-sections to characterize individual features and additionally determine the total region of affected ice-bedreck interface and overall displacement volumes.

The main product is an online portal that, when queried with a region of interest (point, line, or bounding box), returns all the intersecting (or nearby tracks). The algorithm is very similar to that of satellite image servers such as EarthExplorer, though here the objects are 1D flight lines rather than rectangular satellite scenes.

It’s necessary to work at the level of sub-segments since many flights had curved tracks — working with the whole flight segment means working with a chord shortcircuiting an arced track. The sub-segments do this too but their chords are a vastly closer fit to the curve.

An example suffices to illustrate the overall procedure. In reviewing flight line pdfs near Eqip Sermia, I noticed 19980714_01_017 contained a classical basal deformation some distance inland from the deformation highlighted in Bell 2014. To understand this feature, I needed to pull all other tracks from the Cresis archive that also cut through this deformation.

After a simple paste of the kml file into a grepping spreadsheet (which cleans away extraneous kml terms, rounds data to appropriate precision and creates clean columns), the last 9 way-points are shown below. Only the last 9 are used because they have delimit a basal disruption (based on inspection of the radargram 19980714_01_017), with only way-points 3-6 actually lying within the upheaval. The colums show latitude, longitude west, elevation in meters, flight segment name and order within the line.

This particular flight segment was represented by 41 GPS points nearly equally spaced along the 148 km track, dividing it into 40 segments of 3.7 km width.

70.068805   47.076781   1928   19980714_01_017   9
70.052159   47.15645    1865   19980714_01_017   8
70.033927   47.243766   1748   19980714_01_017   7
70.014753   47.335211   1864   19980714_01_017   6
69.995344   47.425215   1820   19980714_01_017   5
69.975694   47.513974   1694   19980714_01_017   4
69.955976   47.603585   1610   19980714_01_017   3
69.935927   47.696685   1595   19980714_01_017   2
69.91662    47.787071   1557   19980714_01_017   1

To find all other way-segments in the vast Cresis archive of all years in the vicinity of these nine sub-segments could be done ad hoc using visuals in Google Earth. However this becomes very ineffecient in studying hundreds of basal deformations.

It is actually quite easy to collect all the relevent way-points because Cresis provides a master kml file for each of the 21 years for which Greenland data was collected. A click loads that file into Google Earth and copy-paste into a text editor extracts the data. These are effortlessly counted by pasting in Word, replacing commas, and counting the replacements, then halving as three coordinates are separated by two commas. For example, year 2014 had 58 flights and a total of 143,764 way-points; earlier years such as 1998 far fewer, 11 flights and 3,528 way-points.

Overall, the master database of way-stations consists of 654,182 database records (rows) which is quite manageable in Excel though fairly slow for conducting operations:

2014   Greenland   P3/   143764   287528
2013   Greenland   P3/    30391    60782
2012   Greenland   P3/    64036   128072
2011   Greenland   P3/   118421   236842
2011   Greenland   TO/     9310    18620
2010   Greenland   DC8/   41396   82792
2010   Greenland   P3/    40705    81410
2009   Greenland   TO/    10406    20812
2008   Greenland   Grd/   410      820
2008   Greenland   TO/    16736    33472
2007   Greenland   P3/    19773    39546
2006   Greenland   TO/    16102    32204
2005   Greenland   TO/    32428    64856
2003   Greenland   P3/    89149   178298
2002   Greenland   P3/     2745     5490
2001   Greenland   P3/     1694     3388
1999   Greenland   P3/     4955     9910
1998   Greenland   P3/     1764     3528
1997   Greenland   P3/     3528     7056
1996   Greenland   P3/      744     1488
1995   Greenland   P3/     2642     5284
1993   Greenland   P3/     3083     6166

However, just to pursue the restricted agenda of finding other flights intersecting the basal deformation (thus providing  different perspectives on its structure), it suffices to collate way-points of just four other kml files (see first image) and collect way-points that lie within a generously defined lat,lon bounding box of the upheaval.

This is easily done by first sorting the database by the latitude column, giving a score of 1 in a new column to appropriate new way-points, then sorting by longitude and doing the same, then summing the two new columns into a third, and doing a final sort for records scoring 2 (which amounts to a logical 'AND' operation).

Next, the qualifying way-points are merged into way-paths, which requires that both ‘feet’ lie within the generous bounding box and that the points be consecutive within their track. After building the respective kml units, these way-paths can be displayed in Google Earth and sorted for those that actually intersect the initial track having the deformation.

The radargrams are collected, contrast-enhanced, rescaled to a common vertical exaggeration, and given a background gridding in gimp according to the number of way-points the image contains. This allows cropping of the radargrams to the bounding box and for intersecting paths, mark-up of the cross-over points on the radargrams.

Overall, the idea is to free up the analyst from endless ad hoc manipulations. All the available data relevant to characterizing each basal deformation in Greenland is at the analyst’s fingertips. While this might not be enough to fully understand the feature, it is a big step forward in hypothesis testing, categorizing, and prioritizing suitable  deformations for additional radar overflights.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 23, 2016, 11:42:50 AM
This post looks at slicing and dicing an image into parts and then reconstituting it seamlessly in forum view. The first image does this successfully from the 3 attached tile pieces below (click, hold and drag to see the pieces in the big image) without using an html table. Note the tiles could be separate small gif animations instead of stills.

I've also duplicated this effect within 1x1 and 1x3 forum tables (no longer shown); table controllers cellpadding="0" border="0" cellspacing="0" are not needed in default forum table interpretation.

The small pieces could be saved offsite but online (eg Photobucket) and simply referenced by their urls. This provides a workaround for the 4 image limit of forum software. It could be used for example to display many individual frames of a gif animation. This trick also allows images to be placed inline in association with relevant text rather than merely appended as attachments.

The glaciological idea here is to vertically slice radargrams containing basal deformations in part of the image (especially the ones from curved flight lines) into their GPS way stations and discard slices not relevant to the deformation, make a new Google Earth track from the retained way stations that delineate the overall boundaries of the feature (along the cross-section). It's equally useful for deformation extent mapping to show adjacent track segments that miss the deformation.

Gimp as it stands provides controls over slicing but these are extremely tedious at scale. However 'guides-grid' .scm at the plugin registry, placed in gimp's scripts folder, allows creation of an way station appropriate grid that converts into cutting guides for the guillotine command. That creates a new image for each slice that retains the overall layer structure but it is not so convenient to put the 40 separate files thus generated back into a single image with (40 way stations)*(number of layers per ways tation) total layers.

However once the cutting grids are made, Filters --> Web --> Slice... will systematically name and save all slices of the active layer (as gif, jpg, or png) in an orderly row and column manner into an html table (only the slice image files need to be retained). It is somewhat unfortunate that forum software does not have a storage area for files that only need url pointers but not display as solo attachments. I suspect that they periodically sweep and delete orphaned files (no longer attached to any post), invalidating the url they once had.  .

If the first of the slice files is opened in Gimp, the others can be added in appropriate sequence using File --> Open as Layers ... Slice only operates on the active layer but changing this to the others one layer at a time, then opening these products in that same gimp document, a stack can be built that amounts to a correctly implemented Guillotine command option (sliced with layer retention).

Here each radargram needs a second layer with vertical lines indicating the flight sub-segments between GPS way-stations. Fortunately, these are quite even spaced, presumably because the plane flew at constant speed with readings taken at equal times. This second layer is completely transparent other than one pixel wide vertical waystation lines, meaning the radargram is still easily viewed. These layers should not be flattened because it would wipe out data underneath the grid lines.

The main attribute of this second layer is text values for each way station of lat,lon found in kml files that define the way-station coordinates. Gimp does a poor job importing vector files (eg text) from  the host computer's clipboard -- incomprehensively, even its own internal clipboards don't talk to each other.

There's an undocumented trick that provides a workaround: if all the lat,lon data is copied to the host clipboard and the gimp text tool is clicked on the grid layer as if something were to be manually typed in, a contextual menu pops up that allows importation (paste) of text from an external text editor. If the grid is rotated 90º CW ahead of time and re-rotated 90º CCW after text is situated properly relative to grid lines, the text will read sideways as needed.

A third radargram layer shows the graph of the csv file Cresis provides separately for each radargram. These are apparently generated during the process of manual annotation of ice sheet surface and bottom reflections (colored lines in echo2 radargrams). The fields provided are Lat,Lon,UTC-time,Thick,Elevation,Frame,Surface,Bottom,Quality.

These come in excessive precision that greatly exceeds any conceivable experimental input -- for example, neither bottom nor surface of the ice is not measured to the posted 0.1 mm -- the error is more like 10 m per study of flight crossover points. Thus it is necessary to drop Cresis data into a rounding spreadsheet before working with it.

Of these, surface elevation, bedrock depth and ice thickness are already implicitly graphed in the echo_2 version of the radargram even when cropped laterally to a deformation. Here surface and depth graphs could be usefully deployed as masks (ie flattened onto the radargram itself) to either permanently suppress irrelevant parts of the radargram, constrain action of image enhancement filters to internal ice, or best of all, to provide automated cropping guides above and below (often two thirds of the radargram is wasted space).

Such cropping guided by deformation lat,lon is even more effective in reducing images. That's important because the deformation might later have to be enlarged in bringing it to a standard vertical exaggeration.

Needless to say, to do this on the whole Cresis radargram archive is a fairly massive pre-compute. However it is far more convenient to do this once and for all to allow researchers to get on with the glaciology. The same can be said for horizontal and vertical rescaling to a uniform vertical exaggeration (say 10:1) needed for 3D deformation reconstruction and major aspects of cropping and contrast optimization.


[Edit: quite a few text improvements made above since initially posted.]
Title: Re: Creating Animated GIFs
Post by: A-Team on January 24, 2016, 10:03:11 PM
Animated gifs for long time series presented at data-level resolution suffer file size bloat. One workaround masks out regions that aren't important (which could vary from frame to frame). Masking replaces everything within ots boundary of unimportance with a fixed color -- that compress to almost nothing. 

If an alpha transparency channel is used instead, the background gray in forum software will show through. That's usually ok and definitely better practice than the gif-maker's preference color because the end-viewer upon image download gets control over background transparency.

In the case of radargrams, the actual grayscale images is flanked horizontally by very generous white space and scales. The fix there is cropping, though it won't get rid of scale ticks flattened onto the data. Those can be gotten rid of however by opening the pdf compilation in ImageJ (File -->Import -->Extract Images from PDF...).

In the case of 19980714_01 pdf, the images are at resolution 903 x 888 whereas the same image in the archive  is 1165 x 919 (its framing container) (eg see (

Something is really wrong here because the pdf's aspect is 1.02 whereas that of the archival image is 1.27. So one or the other (or likely both) is peculiarly rescaled (irrational different vertical and horizontal multipliers). And how was that rescaling done: nearest neighbor, bilinear, bicubic, or sinc?!?  It matters.

Meanwhile wtf is the actual resolution of the data, is it stored openly or only buried in matlab, and is either of the provided resolutions actually at data resolution? This is a huge issue for mapping basal deformations and their sub-structures.

Apparently Cresis not only assumes visitors have purchased expensive proprietary command-line software (matlab) and gone through its year-long learning curve but also store the data in its completely unreadable file format (.mat) which opens as total gibberish in text editors.

I find this completely unacceptable in publicly funded science. While .mat may be a wonderfully powerful and flexible container, 99 times out of a 100 nothing but ordinary row and column numbers are being stored (as here). Data should never be stored in anything more complicated than is necessary, here plain old cvs. Let the matlab users import cvs.

The airplane's raw 'trace' files (the 1 pixel wide vertical lines that tile up to make a radargram numerical array) are being processed via various matlab command settings to produce the radargram, its unwanted overlapping edges, text overlays, and badly done scales.

It is on this raw numerical array that 'contrast' tools will optimally operate. Here it is critical to understand the perfect 1:1 correspondence between working on a numerical array and working on its bitmap display via Photoshop-type tools.

Many people fail to realize that gimp, PS, ImageJ, ImageMagick etc are not inherently graphical software, they are just  Excel running in the background that produces an imagery front end. Matlab processing an array of numbers is just the poor man's Photoshop. It is imperative to use a graphical front end to exploit synergies with the superb pattern recognition ability of human vision.

Cresis, which is to say matlab, is strictly outputting 8-bit grayscale images. In some software, they open as RGB but in fact there is no color. That's easily proven: ImageJ reads the header and always opens files at their full bit depth. Make  a duplicate image, drop it to 8-bit grayscale, form a stack with the original, subtract the layers and observe from the histogram that it is entirely zeroes (pure black). That means no color, not even subtle color.

Where is this grayscale color coming from? I would think matlab normalizes the raw numerical array of radar return power (traces) by dividing by the biggest number to bring everything into [0,1] and then rescales and rounds to integers in the range [0,255]. That opens directly in ImageJ as a BMP or x,y image which is then losslessly converted to more familiar tif or png.

I have huge concerns about the sub-optimality of this. The data, or at least certain depth ranges of it (ie the basal layers), may actually be of much higher bit depth say 12- or 16-bit. Running with 8-bit would be as crazy as dumbing down a hundred million dollar feature of Landsat-8 to make another Landsat-7.

I will also bet the contrast compression was linear. Huge mistake, it should at a bare minimum set a gamma and optimally be adaptive (as in ImageJ Clahe). Very few users are actually interested in the physical meaning of the absolute radar return strength; this class of users suffers from its retention.

Ice penetrating radar was initially tasked solely with determining depth to bedrock. (Surface elevation is better measured with laser, inSar and optical stereo pairs which are 2D rather than sparse 1D tracks; ice thickness = surface - bedrock.) The goal posts have since moved to internal layer isochrons and basal conditions but archive processing has not moved with it. Bedrock topography was terribly important in older stick-or-slip models but with temperate ice, deformable basal sediment and looser lower ice rheology, that becomes less important to predicting future behavior.

Ultimately, even a greatly improved image of a basal deformation may prove just as mysterious as our current blurry images. The real advantage will come from greatly enhanced ability to map the geographical distribution of anomalous basal conditions (ie get past the dramatic ones to the faint but still critical to sliding and deformation).

I am looking now at Wolfram's Mathematica -- it can open this crappy .mat format and export into plain text cvs. Affordable to anyone (cost: a cup of coffee per month), it is world's apart in mathematical and graphical sophistication from conventional scientific software. It may be worthwhile to dump the whole Cresis archive out of .mat into .cvs as a pre-computed cloud, but it remains to be seen if Wolfram offers a convenient method for bulk processing the whole Cresis archive. (

In terms of gif reduction, the images below asks whether the ice surface and bedrock cvs files can serve to define masks. The answer is yes but something is lost even having 409 points per curve per radargram -- they're still chord collections. While the fit might be improved with splines, it is still not quite as good as the original graphic. However a big benefit comes from dropping back a couple of pixels to define crop boundaries, especially in basal deformations of limited extent.
Title: Re: Creating Animated GIFs
Post by: Watching_from_Canberra on September 10, 2016, 03:36:08 PM
just testing an animated gif that doesn't seem to animate...
Title: Re: Creating Animated GIFs
Post by: Jim Hunt on September 10, 2016, 03:54:11 PM
just testing an animated gif that doesn't seem to animate...

It doesn't animate for me either. I seem to recall A-Team mentioning that you have to reduce the width to 700 px to keep SMF happy?
Title: Re: Creating Animated GIFs
Post by: Watching_from_Canberra on September 10, 2016, 04:02:51 PM
Thanks for the pointer - that seems to be the issue.  I'll go and post it where I first posted it now...
Title: Re: Creating Animated GIFs
Post by: Tigertown on December 20, 2016, 05:58:52 PM
If you got something too big to work on here and you don't want to really change it, just make a youtube video out of it and post the link.

Example: (
Title: Re: Creating Animated GIFs
Post by: Tigertown on April 19, 2017, 05:02:49 PM
For an easy to use site, try
Title: Re: Creating Animated GIFs
Post by: subgeometer on April 20, 2017, 05:02:56 AM
If running linux or unix ImageMagick is a ubiquitous tool suite for image processing. On Ubuntu at least it comes as part of the standard install. It runs from the command line so is easily usable in scripts for batch processinga

The main tool is convert

A simple shell command to make a sequence of stills from image_1.jpg to image_n.jpg into a gif is

convert -delay 100 image*.jpg the_movie.gif


convert -delay 100 image_1.jpg  image_2.jpg  ... image_n.jpg the_movie.gif

The delay parameter sets framerate as milliseconds per frame(I think)

convert can do many other things besides, a swiss army knife for scaling, compositing, image generation,adding text etc . It's fairly well documented online though its manual is somewhat terse
Title: Re: Creating Animated GIFs
Post by: subgeometer on May 27, 2017, 06:52:36 AM
I'm starting to write a bit of python code to ease downloading of image sequencies, initially from worldview, but the approach could be applied to other sites that encode params in the url

It's just a function so far which I've been using in the interpreter. Basically you feed it the url worldview's image download function loads and python downloads it and a sequence of previous days(the default is 8 ) and writes them with a series of sequential names(based on a nameroot you can give, and which defaults to "worldview")

When I have another free hour or 2 I'll add argument handling so it can be used as a script

import urllib, urlparse

def get_worldview_sequence(start_url, count=8, outfileroot="worldview"):
    #this will cause problems when going over the new year
    #but its quick so we do for now
    while count >0:
        #this really should check that
        #this doesn't get to day 999 of the previous year or other nonsense
        for nm in startquery:
            realquery[nm] = startquery[nm][0]
        #fucked up character substaitution
        out_url=urlparse.urlunparse(urlparse.ParseResult(parsed_url.scheme,parsed_url.netloc, parsed_url.path, parsed_url.params, urllib.unquote(urllib.urlencode(realquery)), parsed_url.fragment))
        #out_url= ""+urllib.urlencode(startquery)
        print out_url
        urllib.urlretrieve (out_url, outname)
Title: Re: Creating Animated GIFs
Post by: oren on August 03, 2017, 01:09:36 PM
For very easy animated GIFs I use ImageJ, using the "Fiji 64-bit" download for Windows.
Open (ctrl-O) in sequence all the images you want to animate (preferably make sure they are limited to 700 pixel width).
Image>Stacks>Images to Stack.
File>Save As>Animated Gif.
Of course you can play around with some options and a lot of other stuff, this is for the very beginner.
Title: Re: Creating Animated GIFs
Post by: A-Team on September 15, 2017, 08:47:09 PM
Panoply makes easy animated gif maps in all common projections, provided that the netCDF file itself being used has the data in multiple time shots.

These are geolocated maps in whatever projection and palette color you've found most effective by rapid experimentation, in our case typically the polar stereographic with a sequential palette for an ordered variable like ice area or a divergent palette for ± change about 0.

Panoply has ~100 projections and a great many palettes (plus it's easy to import more from galleries, grab from an online map say one from @zlabe, or made from scratch using rgb or hsv spreadsheet numbers).

The latter is most easily done by downloading a bona fide .rgb palette from UCAR (header and link shown below), opening it as plain text, and overwriting with your own numbers, followed by 'save as'. That trick defeats the Mac operating system which refuses to let you change .txt to .rgb manually.

ncolors  = 254
;R   G   B
0   22   132
0   24   135
0   26   136

Now use the Open file command in Panoply to import the .rgb palette; it immediately becomes active on the top map. The forum software also refuses text files attachments that are not .txt so I've relabelled the attached .rgb file below as .txt; it is the blue-to-white palette used at UH AMSR2 for sea ice concentration. (With this, I can use AMSR2 right up to today and then follow with the ESRL forecast in the same colors.)

That will be the case for the .nc files from ESRL climate forecasts which have numerous 5- and 10-day series at 6 hour intervals, so 20 and 40 frames respectively. If that's too many, set the 'stride' between consecutive frames to skip intervening ones.

Panoply does not make gifs per se, just the .png or .jpg frames. These just need to be loaded up as a stack in ImageJ as Oren explains above or as layers in gimp, then cropped/resized to 700 x 700 pixels max and saved as a forum animation. Panoply also makes nice .mp4 videos but the forum software does not accept that format.

Looking at the opportunities at ESRL on 15 Sep 17, the REB.2017.09.14 netCDF bundle offers seven 40-frame maps, forecasts out to D10:

* snow/ice surface temperature
* air temperature
* snow thickness
* ice thickness
* ice area
* air velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction
* ice velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction

Looking at the REB_plot component at the ESRL archive on 15 Sep 17, some 43 graphics are provided as .png or .gif but without the underlying netCDF Geo2D files (meaning projection and palette aren't easily changed). Some of the .gifs are stills but 22 of them are pre-made running gifs, 20-frame forecasts out to D5.

The MacOS has a wonderful feature for rapid scanning of the 43 graphics: select-all and right-click on Quick Look. This does not open the files in an app like Preview but displays them like a slide show, even animating the animations within such slides. (I use Quick Look with our game cam to flip though the tens of thousands of wildlife shots for keepers.)

The ESRL have issues such as too big to load or too wide to play on the forum, local rather than Arctic Ocean wide coverage, zero used for both data and not-on-ice background (ie confusingly same color, white), extraneous comparisons to GFS forecasts, esoteric meteorology variables, unwanted overlays like contours, and so forth.

Like all Panoply maps, these gifs strictly use undithered palette colors and so these can be selected and changed out. This can be done for all frames at once if the gif is loaded into Gimp, converted to RGB, tiled up by Filmstrip, color replaced, and sliced back out to individual frames.

The REB_plots have uninformative names at the moment, no clues as to date or content:

Arctic1    RASM-ESRL vs GFS ice area and snow depth
Arctic2    RASM-ESRL vs GFS 2m temp and surface pressure
Arctic3    ice thickness thermo + dynamics + snow melt + ice melt
Arctic4    RASM-ESRL vs GFS 850 hPa temp and precip
Arctic5    ice and snow thickness contoured
Arctic9    RASM-ESRL vs GFS 500-1000 hPa thickness and precip
Arctic10   bottom ice growth + lateral melt + snow melt + top ice melt
Arctic11   longwave + solar flux + shortwave + albedo
Arctic12   RASM-ESRL vs GFS surface temp and LWP
Arctic13   RASM-ESRL vs GFS surface pressure + 850 hPa height
Arctic14   RASM-ESRL energy flux + LWP + surface temp + IWP
Arctic15   melt pond fraction + SST + heat flux + wind speed
Arctic16   RASM-ESRL ice speed
Arctic19   RASM-ESRL 500hPa height and wind vectors
Arctic20   310K potential vorticity and Surface theta PVU
Arctic22   RASM-ESRL vs GFS longwave flux + shortwave flux
Arctic23   RASM-ESRL energy flux + LWP + temp + IWP [duplicate of Arctic14?]
Arctic24   RASM-ESRL vs GFS surface wind + energy flux

AlaskaRegion2   RASM-ESRL vs GFS 2m temp and surface pressure
AlaskaRegion4   RASM-ESRL vs GFS 850 hPa temp and precip
AlaskaRegion5   ice and snow thickness
AlaskaRegion12  RASM-ESRL vs GFS surface temp and LWP

The third component of the ESRL archive, of form RASM-ESRL_4NIC_2017-09-13.tar.gz, consists of ten time slices at 24 hour intervals in the form of Each of these consists of 18 Geo2D mappable products (or 16 after sea and ice velocity x,y arrays are combined).

aice       sea ice area fraction
hi         sea ice thickness
hs         surface snow thickness

snow ai    lwe snowfall rate
rain ai    rainfall rate
meltl      lateral ice melt
meltb      basil ice melt
meltt      top ice melt

sss        sea water salinity
sst        sea water temperature
Tair       air temperature
Tsfc       surface temperature where sea ice

strength   compressive strength of sea ice

vocn       northward sea water velocity
uocn       eastward sea water velocity

vvel       northward sea ice velocity
uvel       eastward sea ice velocity   

divu       divergence of sea ice velocity

A single such .nc does not have frames, it consists only of its time frame; to make a gif for say rainfall rate, ten one-frame maps have to be made and saved out as .pngs. From those, loaded in Gimp or ImageJ, a D10 daily gif animation can be made. However there's an issue with consistent scale because 'fit-to-data' will show a different ranges and so different utilization of the common palette.

Here are best-links to palette collections, conversion tools, appropriateness, Panoply manual, ESRL offerings, and Gimp indexed color: ( ( ( ( ( ( ( ( ( ( (http:// (
Title: Re: Creating Animated GIFs
Post by: A-Team on September 16, 2017, 01:48:15 PM
This is quite an interesting article on climate code and how it should be reported in journals, the observation being that a lot of research is currently irreproducible. Not in the sense that you would get a different result duplicating it but rather there's not enough information provided on the software side that would even allow you to get started replicating.

We run into that all time on these forums: maps come with no underlying data link, no underlying publication describing methods, and data not in formats like netCDF, HDF and  GRIB (all of which Panoply can view) that allow re-projection to a standard Arctic Ocean EASE grid or polar stereographic coordinate system which is necessary for comparing or combining data from different sources.

A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences
Damien Irving ( free full

"At first glance, the only difference between a regular journal article and that of IS2015 is the addition of a short computation section (see “Example computation section” for more information).

That section accompanied the traditional description of data and methods within the article and briefly cited the major software packages used in the research before pointing the reader to three key supplementary items: 1) a more detailed description of the software used, 2) a version controlled and publicly available code repository, and 3) a collection of supplementary log files that captured the data processing steps taken in producing each key result.

The repository was hosted at a popular code sharing website called GitHub, while the detailed software description and log files were hosted at Figshare (Irving 2015a), which is a website where researchers commonly archive the 'long tail of their research (e.g., supplementary figures, code, and data)."

An example of best practices:

The results in this paper were obtained using a number of different software packages. A collection of command line utilities known as the NetCDF Operators (NCO) and Climate Data Operators (CDO) were used to edit the attributes of netCDF files and to perform routine calculations on those files (e.g., the calculation of anomalies and climatologies) respectively. For more complex analysis and visualization, a Python distribution called Anaconda was used.

In addition to the Numerical Python (NumPy; Van Der Walt et al. 2011) and Scientific Python (SciPy) libraries that come installed by default with Anaconda, a Python library called xray was used for reading/writing netCDF files and data analysis. Similarly, in addition to Matplotlib (the default Python plotting library; Hunter 2007), Iris and Cartopy were used to generate many of the figures.

To facilitate the reproducibility of the results presented, an accompanying Figshare repository has been created to document the computational methodology (Irving 2015a). In addition to a more detailed account (i.e., version numbers, release dates, web addresses) of the software packages discussed above, the Figshare repository contains a supplementary file for each figure in the paper, outlining the computational steps performed from initial download of the ERA-Interim data through to the final generation of the plot.

A version controlled repository of the code referred to in those supplementary files can be found at (
Title: Re: Creating Animated GIFs
Post by: A-Team on September 16, 2017, 04:47:46 PM
Below is the current Panoply palette collection. It can be viewed in Preferences; the collection below tiles up screenshots from there, hopefully faithfully. It takes a click to view at scale; reduction in dimensions is dicey because it might invoke interpolation.

The palette names suggest their origins, eg the CB prefix means imported from C. Brewer's widely used ColorBrewer web page at PSU, UO indicates Univ. Oregon geography dept., GMT means Generic Mapping Tools software from SOEST at the University of Hawaii, and many others originate at JJ Green’s cpt-city web site.

While anyone can quickly hack color tables in a spreadsheet or import them into Panoply as .rgb tex from govt climate sites, journal illustrations, twitter sites, or online product maps, there's a whole art and science to palette design.

The great advantage of Panoply, provided there's a netCDF, GRIB or HDF file associated with the map, is the chance to replace a bad palette with a good one, at the same time getting rid of vexatious contours, arrows, text and lat-lon overlays that interfere with the data and its comparison to other maps of other variables that might be in say an equal area projection. For example, a simple grayscale gradient is better-suited to arithmetic operations such as differencing.

An example application: ESRL's palette on 'icethicknesschange' is really bad. Not only is it inconsistent day to day but none of the palettes make conventional sense. That is, thickness is an ordinary real number that varies ± about 0. White is used for the lowest bins but sometimes gray, yet those are also used for the completely ice-free perimeter. This situation calls for a divergent color scale, eg reds for loss and blues for gains.

Since ESRL is using 17 color bins, the 'panoply_diff_17.act' palette is an immediate better fit. Another option, what the ESRL palette is attempting, is color-pairing. That amounts to taking a couple of ColorBrewer 8- or 9-color diverging palettes with trend logic and intercalating them to get better color separation while retaining logic.

Of the stock palettes in Panoply, interleaving CB_RdYBl_09.cpt and CB_PuOr_09.cpt is a good option. By displaying any .nc file as a map, those palettes can be cleanly captured into Gimp which has drag'n'drop indexed color table rearrangement.

The resulting palette, imported via ImageJ rgb listing into Panoply, can then display the ESRL .nc file properly. That palette is shown in the 4th image below. It could be improved for visual consistency with some minor tweaks.
Title: Re: Creating Animated GIFs
Post by: A-Team on September 16, 2017, 07:02:04 PM
Here is ESRL's experimental forecast for 5-day ice thickness change prior to adjacent alternative palette replacement. It may be better however to use fewer colors, say 2-3 on the thinning side and similarly on the thickening, or simple but methodical red and blue gradients increasing out in intensity.

The frames runs out to Sept 19th from a prediction made on the 14th. The first in the series was made on Aug 22nd for the 27th. The variable background color has been put to a consistent gray. It is hard to tell from these whether thickening or thinning is happening overall; other ESRL products may be better for this (notably just the forward prediction from today or just the D0 directly from the ESRL netCDF via Panoply).
Title: Re: Creating Animated GIFs
Post by: A-Team on September 18, 2017, 12:55:49 AM
We're long been able to make excellent line graphs, given the raw data. Now we're moving on -- slowly --  from copy-pasting other people's maps to being able to make them better ourselves, given the raw data. That involves geolocated data and control of the projection, overlays, and especially the palette.

Some more on palettes most appropriate to climate science maps -- there's been a lot of online discussion lately on perceptually neutral palettes, the horrific rainbow palette, and palettes neutral to common male colorblindness.

However fashionably pc the latter might be, it's not enough. The fact is that a great many other factors influence how a given map is perceived by different people on different devices. Some are viewing on $10 cell phone screens; others have calibrated 27" retinal monitors with an advanced color gamut. And then there are academics still printing out pdfs on inkjet CMYK to uncoated paper to read on the plane. Not to mention journals that won't host time series as animated gifs and Twitter that makes dithered mp4's out of them, destroying individual frames. And at the end of the day, it's largely -- but not entirely -- a matter of individual taste.

A better solution may be simply to provide a link to the netCDF file for the map, allowing readers to quickly swap in their favorite Panoply palette.

A lot of online discussion of palette creation today consists of sharing command-line code snippets (mathlab, python, java, IDL, github, etc) with many participants unaware that Photoshop already did all this and much more 30 years ago, with freeware Gimp successfully emulating almost all of it, for example making any continuous gradient and discretizing it into a dozen bins.

It is vastly more efficient to make palettes from scratch and rapidly vary them (on the target map) with sliders in Gimp. This and simple spreadsheet RGB and HSL builds have resulted in very large galleries of free palettes. Most of them come unannotated as to construction however but it's easy however to assess their uniformity as grayscale (luminance, lightness, and average are shown in some popular modern palettes like parula, viridis, geosoft, coolwarm, and cube-helix below).

Nevertheless, a lot depends on the type of climate map, the 2D statistical distribution of the variables being displayed, and what 'talking points' the map product is supposed to bring out. An interesting concept here, after treating outliers, is 'histogram equalization' that results in more or less equal use of each color bin for maximization of color discrimination.

Is it possible to eliminate the subjectivity in palette choice by having its construction data-driven? For example, if grid nearest neighbor variance is high, then contouring will not be effective or even feasible. If it's low, then a binned palette might be effective; indeed the optimal number of colors might be inferable. Similarly, mean and std deviations might allow rational setting of outlier lumping.

Here are some of the better discussions, online palette tools, and palette collections I've come across. The graphics can be downloaded and the palettes imported into Panoply as described above. ( ( ( ( ( ( ( (
Title: Re: Creating Animated GIFs
Post by: A-Team on September 18, 2017, 01:21:52 PM
Given the data, our forum posters make excellent line graphs, which adds analysis and thus value. We do some original work with satellite imagery from portals like Worldview, Jaxa and Sentinel Playground. However we mostly copy-paste map products (even when they're poorly done) rather than make better ones ourselves.

That involves geolocated data and map generating software providing control of the projection, overlays, and especially the palette. The latter four are well-handled by Panoply. Climate science data is generally stored as netCDF files or convertible.

These are basically just collections of excel spreadsheet stacks but are put into a complex non human-readable binary format because file sizes can be immense and non-desktop systems must read them.

As a practical matter, to make a netCDF file on a desktop requires a CDL file and a code snippet called ncgen. There's currently no online tool that will do this, it requires leaving your desktop OS for command line. That's too bad because the CDL format is easy to read, edit, and make from scratch in a text editor using comma separated data numbers.

Panoply can export any .nc file to CDL but cannot reimport them, ie it implements ncdump but not ncgen. Most of the benefit of export can already be seen in the Source window panel "Show info in expanded form'. The array panel in a map shows the numbers in its grid.

Panoply distinguishes geolocated Geo2D items in a netCDF file from generic 2D files. Both are plottable but only the former map onto the globe via a chosen projection. It's unclear how latitude and longitude data in an .nc file come to be associated with array data to convert it from 2D to Geo2D.

So with editing, creating and importing of palettes and data arrays mastered, we just need ncgen and a Geo2D description to be able to fix poor online maps or make our own from scratch using satellite data or other resources. Poster wipneus has, in effect, a workaround to do this in the complex instance of piomas data.

Note the UH AMSR2 3.125 km archive ( etc) doesn't arrive as Geo2D because the sea_ice_concentration variable is missing an 'attribute' that tells Panoply of the gridding scheme. However the land variable is seen by Panoply as a Geo2D variable that can be plotted on a map. That's because it includes a coordinate 'attribute' that explicitly names the auxiliary 2D latitude and longitude grid variables. The sea_ice_concentration variable is missing that coordinates 'attribute'. (Here 'attribute ' has a technical meaning within the format prescription -- thx to RBS here!)

Here are the most useful links I've found to date on ncgen and CDL formatting. Some of it makes sense but it's fair to infer that many code manual writers were not English lit majors in college. ( ( ( ( ( ( (
Title: Re: Creating Animated GIFs
Post by: Dryland on September 18, 2017, 05:41:17 PM
Hi all,
I recently contacted A-Team and offered some programming help for the work he's doing with the ESRL products. I see an opportunity to automate some of the steps he's been doing manually, so that, for example, each time there's a new ESRL drop, a short time later there are new visualizations available to view on the web. 

Toward this end, I've written a Python program that performs these steps each time it's run:

(a) Check one or more FTP sites (such as for files that are new or changed since the last run.

(b) download those files, and recursively unpack any archives (such as .tar.gz files).

(c) walk the tree of new files and apply a set of rules (which are based on regular expressions, and are read from a config file) to transform those files into a tree of output files. The set of operations is expandable as needed. At the simple end, rules allow ignoring files of no interest, copying files as-is to the output tree, renaming them, and putting them in a different directory structure. The next level is to change image formats, add text to images, create thumbnails, build animated .gifs, and so on, using any of the available tools (I'm using ImageMagick at the moment). And the top tier is to do real data manipulation and analysis, and automate tools like Panoply to create new visualization products.

(d) upload the output files to a web server. I've configured a static website using Amazon S3 as a test bed. The HTML is also automatically generated from the data so it also stays in sync with the latest changes.

My proof-of-concept is hosted at (; all it does at this point is a selective copy of a few ESRL files, wrapped in a bare-bones HTML page for access. I plan to keep this URL alive and will have more to show there as this system evolves.

I'm happy to share ideas and code with any forum members who are interested.
Title: Re: Creating Animated GIFs
Post by: A-Team on September 18, 2017, 06:32:01 PM
Awright, Dry! Hopefully you'll get some helpful feedback from the other python people here.

A new command line version, PanoplyCL, is available for Mac and Windows at ( along with a not-so-new manual.

PanoplyCL could be quite important for us in automating the transfer of daily ESRL products into stable links and images that work under constraints of the forum. I have just installed it (after a minor hassle activating Java v8 update144) and saved my first script, which has the extensions .pancl.pcl  It is opened as text below and looks quite straightforward.

Basically it seems a fancy way of setting preferences separately for each map in a collection of Geo2D files so the same parameters can be run to make the next day's map the same way. This amounts to making the frames of an animated gif using the ESRL archive.

Going forward, ESRL + Panoply are probably the best things we have going for comprehensive 2017 freezing season forecasts, though specialized meteorological and oceanographic products may be better at certain narrower tasks.

A collection of ESRL's D0 initial states could serve as an improved seasonal archive (even though not hindcasts or reanalysis); we don't know yet how good they are but they allow easy and precise comparison to competitive products if any should surface that play by climate science rules.

// Script parsed by PanoplyCL to create a plot. Usage:
// java -jar PanoplyCL.jar aice_h_in_REB.2017-09-11.pancl.pcl

// Open a dataset.
var ncdata1 = panoply.openDataset ( "/Users/a-team/Downloads/REBS nc/" );

// Select a variable.
var ncvar1 = ncdata1.getVariable ( "aice_h" );

// Create the plot.
var myplot = panoply.createPlot ( "lonlat", ncvar1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Specify plot settings.
myplot.set ( "size-factor", 200 );
myplot.set ( "size-width", 100 );
myplot.set ( "size-height", 50 );

myplot.set ( "title-text", "ice area  (aggregate)" );
myplot.set ( "font-master", "SansSerif" );

myplot.set ( "color-background", "black" );
myplot.set ( "interpolate", true );

myplot.set ( "color-invalids", "rgb(96,96,96)" );

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", false );
myplot.set ( "scale-outlier-shape", "NONE" );
myplot.set ( "scale-outlier-side", "both" );
myplot.set ( "scale-outlier-gap", "thin" );
myplot.set ( "scale-tick-size", 7.0 );
myplot.set ( "scale-label-location", "BELOW" );

myplot.set ( "scale-min", 0.0 );
myplot.set ( "scale-max", 0.9999876618385315 );
myplot.set ( "scale-div-major", 9 );
myplot.set ( "scale-div-minor", 4 );
myplot.set ( "scale-exponent", 0 );
myplot.set ( "scale-log10", false );
myplot.set ( "scale-label-custom", false );
myplot.set ( "scale-tick-format", "%.2f" );

myplot.set ( "scale-minmax-note", true );
myplot.set ( "scale-minmax-format", "Same" );

myplot.set ( "grid-spacing-lon", 15.0 );
myplot.set ( "grid-spacing-lat", 15.0 );
myplot.set ( "grid-label-vis", false );
myplot.set ( "grid-label-size", 6.5 );
myplot.set ( "border-weight", 114 );

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 );
myplot.set ( "proj-xparam-2", true );

myplot.set ( "overlay-1-weight", 75 );
myplot.set ( "overlay-1-name", "Earth_mask.gif" );
myplot.set ( "overlay-1-color", "rgb(204,255,214)" );
myplot.set ( "overlay-1-invert", false );
myplot.set ( "overlay-2-weight", 0 );

myplot.set ( "proj-shading-vis", false );

myplot.set ( "grid-weight", 50 );
myplot.set ( "grid-style", "NONE" );
myplot.set ( "grid-color", "rgb(64,64,64)" );

myplot.set ( "contour-weight", 75 );
myplot.set ( "contour-style", "None" );

// Save plot image to disk.
myplot.saveImage ( "PNG", "aice_h_in_REB.2017-09-11.png" );
Title: Re: Creating Animated GIFs
Post by: Dryland on September 19, 2017, 02:00:45 AM
I replicated this, and it works for me.

I installed PanoplyCL on linux (Ubuntu 14.04) and also had Java version problems - "UnsupportedClassVersionError, Unsupported major.minor version 52.0". The cure was to install Oracle's JDK 8:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

I then ran your script and got the same image back. Nice!
Title: Re: Creating Animated GIFs
Post by: A-Team on September 19, 2017, 01:15:12 PM
ran your script and got the same image back.
That is just awesome!

It was fortunate that I happened to use one of the default Panoply palettes here, CB_Blues.cpt, before exporting the script. If the palette had been set to my favorite for ice area aice_h, namely the custom palette AMSR2.rgb (whose file and easy installation is described above), the script would have stalled out unless you too had put it in your palette collection.

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-colorbar", "AMSR2.rgb" );

While Panoply offers a very nice collection of stock palettes, the reason for AMSR2.rgb is to let ESRL seamlessly pick up from UH AMSR2 high resolution observational data in the same color scheme. That is, the UH data only goes up to the day before whereas ESRL forecasts ten days out from there.

Alternatively, if the UH .nc can be reworked from 2D to Geo2D allowing polar stereographic projection, its palette could more simply be replaced by a similar stock palette in Panoply. The UH archival images fortunately are in standard 'Greenland down' orientation and just need a 39% scale reduction in Gimp to match the scale on the King plot.

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 ); //this sets lowest latitudinal circle displayed.

myplot.set ( "size-factor", 240 ); King size plot setting

I'm using the King 240 scale on Panoply so that it will output .png images with the Arctic Ocean fitting in ~700x700 pixels, so after cropping in Gimp it makes maximal use of what the forum software will display as an animation. I don't want to rescale later in Gimp because interpolation messes with the colors; right now just palette colors are used in the map pixels.

Smaller thumbnail animations lose detail; larger animations will run after a click to a new window but counts show few forum visitors take that extra step. File size is definitely an issue with the 40 time slices in the ESRL data.

Note the image the script makes is only the 1st frame out of 40. That is indicated in the 3rd coordinate slot in the script. In exporting an animation, Panoply runs through all 40 of these, in effect bumping the step the internal script it is running. So externally, we need a super script that runs, bumps the step by 1, runs, ... stops at 40 steps (or however many ESRL has provided).

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 5 of 40
myplot.setVarDimension ( 1, 1, 5 );

Two code snippets -- ncdump and ncgen -- are absolutely critical to working with netCDF file format. these interconvert between machine-readable .nc and human-readable and editable CDL plain text. By opening an .nc file and exporting to CDL, Panoply is providing ncdump already without a command line environment. Oddly it cannot convert its own CDL file back to the .nc that it just made it from, which would amount to ncgen.

The link below gives a friendly explanation of how to get ncgen running on a Mac. It's a lot of trouble to get one menu item, involving a massive download and installation of Xcode, MacPorts, X11 and establishing directory paths. At that point, command lines for ncgen can be run on a local CDL file will make it into a netCDF that Panoply can import: (

-- Generate a NetCDF file
ncgen -o mydata.cdl

Generate code that will produce a NetCDF file
 – Fortran:
ncgen -f mydata.cdl > mydata.f

 – C:
 ncgen -c mydata.cdl > mydata.c
Title: Re: Creating Animated GIFs
Post by: A-Team on September 19, 2017, 01:57:48 PM
To facilitate visual ice thickness comparisons of ESRL to Piomas, it's good to map up ESRL's view in the color palette that wipneus has been using on the thickness forum. There are 16 colors there whose rgb values are readily captured in Gimp from a large-format posted graphic and imported into Panoply.

It's a good idea to number imported palettes, say 11wipiomas and 10AMSR2UH so that they appear in order conveniently at the top of the stack in Panoply alphabetization. To import, save with the .rgb tag from a text editor and Open that file in Panoply.They are easy to delete just by a Cut command.

The animation compares mid-August Piomas as prepared by wipneus, mid-Sept ESRL ice thickness as prepared in Panoply with the same palette, and mid-Sept AMSR2 sea ice concentration from the UH archive..

ncolors= 16
#  r   g   b
0   166   0
25   175   0
50   184   0
83   193   0
116   202   0
151   211   0
189   220   0
230   230   0
231   206   29
233   189   58
234   179   88
236   177   118
237   182   148
239   194   179
241   214   211
242   242   242
Title: Re: Creating Animated GIFs
Post by: A-Team on September 19, 2017, 06:22:17 PM
Gimp (or ImageMagick) will play a key role in pipelining ESRL netCDF products onto the forums. While the ESRL graphics are excellent -- as are the Panoply maps produced from ESRL netCDF files -- they are both too large to display properly on the forum.

That's mainly because the Arctic Ocean is quite lopsided with respect to the North Pole, eg the Bering Strait is much further south than Svalbard, yet forum maps need to be centered on the Pole for comparative purposes.

Note further ESRL plots only a fraction of their variables and those in certain combinations. Other variables no doubt play a role in their model calculations but aren't used for standalone graphics. For these, we can make Panoply maps if there is sufficient interest in the variable, eg rainfall, snow depth or compressive strength of the ice pack.

There's an easy way to in Gimp assemble a master graphic for all the Geo2D files in REBplots, REB_nc, and RASM-ESRL archive bundles. (These make sense to mix together only when the animations would be the same length, eg 20 or 40 frames.) From this master graphic, any combination of variables can be exported out as a gif animation.

The issues here are (1) each variable has its own legend, title and palette (2) some variables have interesting max and min variation by frame (which Panoply provides), (3) the same time stamp is shared by all the variables, (4) vectors from a first variable can float over the map of another variable, and (5) some variables apply only to the ice (thickness), others just to open water, and still others to both. This latter fact (and the ambiguous use of zero) require development of an evolving 6-hour open water mask and mask edge.

The process of making a master graphic is fairly simple. First, Panoply scripts generate maps at the same favorable scale for all the variables. Each variable has a different palette appropriate to what it is supposed to convey. Other script settings may also differ slightly, for example log vs scalar. These output pngs are cropped identically using fixed UL and LR corner coordinates. Next the frames are tiled up into one wide image.

Other maps from other variables are then tiled up too and put into the growing master receiving image (for the given D0 dte) as layers. The same is done for the time scale, the palettes, the scales, the legends and the varying max and min data. Open water mask, boundary outlines, and alpha transparencies are then constructed as yet more layers.

The master image is then ready to have various layer combinations constructed by activation and layer ordering. A new-from-visible layer for each desired output is then sliced back down into forum sized frames. These pngs are then saved to animated gifs at whatever frame rate is effective for them and uploaded to a forum post. While some manual intervention and operator aesthetics may be necessary, automating daily ESRL output up to this master file is quite feasible.

Some examples are cross-posted below:



Title: Re: Creating Animated GIFs
Post by: A-Team on September 24, 2017, 12:46:20 AM
This post summarizes forum-relevant highlights of the PanoplyCL scripting manual, as applied to the 49 Geo2D files in the 3 daily ESRL download folders (REB_plots, REB, and RASM-ESRL_4NIC).

PanoplyCL, while written in Java, uses unrelated Javascript as its scripting language. (Java 8 can internally parse and run Javascript.) Not everyone wants to learn full-on Javascript so the goal of this post is to collect the minimal number of code snippets sufficient for the typical forum use cases or alternatively develop tricks for staying within the more intuitive menu-driven GUI environment.

The scripts are actually saved out of the GUI as say vvel_h_in_REB.2017-09-20.pancl.pcl (ie variable_in_netCDFsourceFile.pcl rather than myscript.pjs as in the draft manual. While this may suggest PanoplyCL (Command Language) is a variant of Javascript, later on conventional Javascript snippets are added, suggesting they run as they would anywhere.

As explained up-forum, someone wanting to make a simple map or multi-frame animation just  experiments with Panoply settings until they generate a satisfactory map. Exporting the CL script allows that map to be regenerated later by anyone running a copy of the script, typically on the same Geo2D line within the same netCDF but equally from the next day (or older archived dates) if the file name is bumped in the first line, eg the script vvel_h_in_REB.2017-09-20.pancl.pcl would draw the same map but from the appropriately dated data with a simple script revision:

var ncdata = panoply.openDataset ( "/Users/Fred/" );
var ncdata = panoply.openDataset ( "/Users/Fred/" );

Had "Save Plot Settings as Preferences" been invoked, this could have been done without delving into script particulars, provides some other map prescription hadn't overwritten it. This suggests that simply allowing Preferences to be saved under a name, along with Preference navigation to saved names (like Plot Size menu) would let the user stay within the GUI. For now, representative maps can be held open and used to overwrite Preferences as needed. However this rapidly becomes unwieldy with 49 Geo2D maps from ESRL alone.

A simple script edit would output the map .png in a different preferred palette by simply stubbing in the new palette name:

myplot.set ( "scale-colorbar", "CB_RdBu.cpt" );
myplot.set ( "scale-colorbar", "bluescale.act" );

To help with palette choice, it might be advantageous to enable scrolling through the available palettes by category. Right now, palettes are sorted alphabetically which can group related palettes but doesn't always. Other sort options are by number of distinct colors, from 'continuous' (256) to binned (typically 8, 9, 10)., or by sequential vs divergent. Once the palette is sorted to user preferences, a keyboard up or down arrow could quickly cycle the map through related palettes, with the option key reversing color order. Right now, there is endless mousing of the awkward popup palette.

The script that the current PanoplyCL beta saves from a time series merely generates the first frame, whereas ESRL provides forecasts as ten days at 24 hr steps, five days at 6 hr steps, or ten days at 6 hr steps. Those 'strides' can be adjusted to larger intervals at the time of saving an animation to make fewer frames (eg first and last) but saving the script won't capture that or the time stamps that the frames need.

However it could if Export Animation... dialog had an additional checkbox for Save Script As. That would save users from delving into the details of how JavaScript loops through variables. This does not generate an error message if the script is mis-applied to a file that has no additional times.

However a common situation in climate science might be air temperatures both by date and elevation (or ocean temperatures by date and depth). It would be unusual for an animation to first circulate through the dates at a set elevation and then through the dates at the other elevations. So the dialog needs simply build out the slice options as radio buttons from the file variables.

However for thumbnail maps, it might be feasible to stack the elevation slices (though this is easily done later in Gimp). Alternately, separate time animations could be output for all the elevation choices by a dialog setting. However there isn't any mechanism in the gif89 specification to synch them on a web page so that they all display the same date as the animations go forward.

In summary the current beta of PanoplyCL is 'animation-unaware' and all the saved scripts look the same except for palette choice, image size, and minor appearance settings. However the PanoplyCL manual provides a fix for that with Javascript examples.

The Javascript add-on deduces the frame count for a given netCDF and loops through it using fixed settings from the rest of the script, saving them out as a folder of pngs. Even better, these pngs have their time stamp as a sub-title. The user can then load them into an ImageJ stack or Gimp layers and export out a forum-ready gif animation.

Note that the directory path to the file has to be provided so the script can find it. On a Mac, right-click on the file, then hold down the Option key: the menu will offer copying of the pathway.

var ncdata = panoply.openDataset ( "/Users/Fred/Downloads/REBS_nc/" );
var ncvar1 = ncdata1.getVariable ( "aice_h" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
// Set to the first frame of 40, the 1 in third coordinate position
myplot.setVarDimension ( 1, 1, 1 );

var tdim = ncvar.getDimension (1);
var tlength = tdim.getLength ( );
for (var i = 3; i <= tlength; ++i)
 myplot.setVarDimension ( 1, 1, i );
 myplot.set ( "subtitle-text", "Time step: " + tdim.getStrValue ( i ) );
 var istr = "" + i;
 while (istr.length < 3) { istr = "0" + istr; }
 myplot.saveImage ("PNG", "T_" + istr + ".png");

No further examples of more complex scripts are provided but the manual states these could include "if-then constructs, iterative loops, string concatenation, create multiple plots, changing plot dimensions and appearance settings as you create each plot."

In other words, it is quite feasible to run a single script that processes all 49 Geo2D files into their daily png sets. These could use different palettes and be of different sizes and formats.

As a practical matter, someone has to go through each of the Geo2D files, decide which are useful at what scale, whether linear or log scaling is appropriate, whether a sequential or divergent palette makes sense and so on, saving each out as a script. These are then concatenated as the master script. To make changes on a particular Geo2D, it is simply re-run with the new parameters and the old version replaced.

It might be easier to make and edit the master script from a 'mail merge' setup. The  database is simply the matrix of Geo2D x parameters. It merges into a text document that is the script; the concatenation of all the text documents is the master script.

To avoid typos and unacceptable parameter values, that could be done via a web front end that forces menu choices to admissible, like plot size, file name, and most importantly palette.

The manual also provides examples for all the other map situations such as zonal average, generic 2D, combined array plots and so forth. Again, it seems there is little gained over simply getting the display set and then saving to a Preferences to be called up for future use.

"5.5. Running PanoplyCL in a Server Environment. Because PanoplyCL runs headless, it has potential to be used on a web server to create graphics based on input received from a remote user. Although this is indeed a possibility, note that there are limitations."

"PanoplyCL does not run continuously but must be invoked each time that you want it to process a script and create a graphic or graphics. The most time-consuming part is initializing Java graphics handling. Repeatedly calling PanoplyCL to create and save a single plot image each invocation is very inefficient compared to calling it once with a script that saves multiple plot images."
For security reasons, it is far better to use a pre-written script that can be altered to use different predetermined options rather than insert any content that may have been provided by the user." (
Title: Re: Creating Animated GIFs
Post by: Dryland on September 25, 2017, 01:39:24 AM

You've highlighted some important issues. The question I go back to is, who are the target users, and what do they need? I imagine multiple clusters:

a) Forum visitors who want a graphic they can use to make a point or ask a question, but who aren't interested in spending all day, or doing any sort of scripting. As you say, a menu-driven GUI environment is a must for this audience; but rather than expecting them to tackle the Panoply learning curve (daunting enough even without the scripting aspect), I'd be looking for more canned scenarios where they can specify a few variables using a web form and have a PanoplyCL script generated (more or less by template filling) and run for them, resulting in a hosted image that they can view/download/post. The "web front end" you mention seems to be along these lines.

b) Users who are willing to take the plunge, install Panoply (GUI version) and learn to use it to make more sophisticated graphics in a one-off fashion. You've discussed some of the ways their work can be captured and then automated. I think some expert assistance there would make more sense than expecting these users to have a way to add new automated, daily products to the pipeline themselves, without help.

c) Users who want more control and are willing to do some "casual" scripting mixed with manual operations. I'm not sure these users were done any favor by the choice of Javascript as the scripting language for PanoplyCL. I chose to work in Python for my ESRL automation, for two main reasons: (1) the scientific community in general seems to really like Python - lots of scientists have gotten pretty good at it; and (2) Javascript hides some nasty sharp edges within its casual-friendy surface. Now, it may be that the subset of Javascript that matters for PanoplyCL scripting wouldn't run into those issues (e.g. callbacks). The script above isn't that bad. And some people just love a challenge, and will keep trying until they get what they want. How to help these people? We could allow them to upload scripts to be run on PanoplyCL on a server, but that's risky - not just security as you quoted, but also more innocent problems like scripts that run forever and take up 100% CPU time. It would be better to let them develop on their own machines, then review and adapt their scripts if we want to add them to the automated pipeline.

d) A-Team. :)

For now, I have my hands full with the needs of group (d). Later on, I would like to see some other users raise their hands and say what they want.
Title: Re: Creating Animated GIFs
Post by: Neven on September 25, 2017, 10:03:31 AM
One of these days I'm going to read through this thread again from start to finish, and will then hopefully fully understand what it is you guys are doing with what, and maybe then I can be a target user who is able to express what he needs.

Sorry for not understanding already. My head is too full. I know you're doing awesome stuff, grade A citizen science.
Title: Re: Creating Animated GIFs
Post by: A-Team on September 25, 2017, 01:06:35 PM
not fully understanding.
The goal (not quite there yet) is to reduce the complexity down to just a double-click on a file or issuance of a single menu command, staying within familial graphical user interfaces though not supporting cell phones and not requiring command-line coding.

Panoply on ESRL data is just a double-click map now, once it is defaulted to stereographic projection and a rainbow palette. However there's more to it in terms of forum-ready accurate scientific visualization. That's what these scripts are supposed to be doing sight-unseen.

Below is an intriguing Panopoly JS option that would let us do fancier presentations, like the ones we see from NASA's visualization studio. There's a valid question (bonbons vs kale) in whether wandering off into eye candy (which might better engage people in climate change) should substitute for conveyance of unperturbed scientific data (ie gif frames in NSIDC projection).

The PanoplyCL manual has a fairly complicated prescription for putting Arctic Ocean data on a spinning globe as viewed from a hovering spaceship. However, as with a BMW, the complications are all under the hood. After some choices are made, it is all pasted somewhere never to be seen again but ready to run any time on different data.

The following script makes a sequence of maps using orthographic projection which displays the northern hemisphere as it appears from a distance. The center of projection is progressively stepped, effectively rotating the globe, saving an animation frame in half-degree steps. Note that without the sample file in hand, the script won't actually run as is. It would make sense also to import a better background earth map than the ones provided.

It does not provide zoom though the option to save frames to kmz format could allow this in Google Earth. That's discussed in section 3.5.4. PCLPlot.saveAsKmz (str, boolean, boolean, boolean). These files do open up in GoogE centered on the north pole and oriented correctly but as miniatures of far too small a scale. It's unclear whether this is a bug in GoogE or in the Panoply export.

var ncdata = panoply.openDataset ( "/Users/jsmythe/cam_data/" );
var ncvar = ncdata.getVariable ( "T" );
var myplot = panoply.createPlot ( "lonlat", ncvar );
myplot.setVarDimension ( 1, 1, 1 );
myplot.setVarDimension ( 1, 2, 26 );
// Paste in here the 30-40 lines of myplot.set that specify the plot's appearance.
myplot.set ( "proj-name", "Orthographic" );
var plon = 45.; // longitude of rotation pole
var plat = 60.; // latitude of rotation pole
// Define the rotation angles and matrices
var betaRad = - toRadians ( 90. - plat );
var gammaRad = toRadians ( plon + 180. );
var cosBeta = Math.cos ( betaRad );
var sinBeta = Math.sin ( betaRad );
var cosGamma = Math.cos ( gammaRad );
var sinGamma = Math.sin ( gammaRad );
var rotY = [ [ cosBeta, 0., -sinBeta], [ 0., 1., 0. ], [ sinBeta, 0., cosBeta ] ];
var rotZ = [ [ cosGamma, sinGamma, 0.], [ -sinGamma, cosGamma, 0.], [ 0., 0., 1.] ];
// Rotate the sphere.
for (var i = 1; i <= 720; i++)
 var lll = 0.5 * i;
 var lllRad = toRadians ( lll );
 var p0 = [ Math.cos (lllRad), Math.sin (lllRad), 0. ];
 var p1 = [ rotZ[0][0]*p0[0] + rotZ[0][1]*p0[1], rotZ[1][0]*p0[0] + rotZ[1][1]*p0[1], 0];
 var p2 = [ rotY[0][0]*p1[0] + rotY[0][2]*p1[2], p1[1], rotY[2][0]*p1[0] + rotY[2][2]*p1[2] ];
 var lon = toDegrees ( Math.atan2 (p2[1], p2[0]) );
 var lat = toDegrees ( Math.asin (p2[2]) );
 // Apply center of Orthographic map projection.
 myplot.set ( "proj-lon0", lon );
 myplot.set ( "proj-lat0", lat );
 // Save plot image to disk with filename based on iteration index.
 var istr = "" + i;
 while (istr.length < 3) { istr = "0" + istr; }
 myplot.saveImage ("png", "T_" + istr + ".png");
// Function that converts a value in radians to degrees.
function toDegrees(value) { return value * 180. / Math.PI; }
// Function that converts a value in degrees to radians.
function toRadians(value) { return value * Math.PI / 180.; }
Title: Re: Creating Animated GIFs
Post by: A-Team on September 27, 2017, 01:29:20 PM
Minor news from correspondence: the read/write command-line code snippets for netCDF files are called ncdump and ncgen. The former is implemented already in a sense in Panoply in that an .nc file can be opened and saved out as CDL*, the human (or excel) readable version of netCDF.

Here the * on CDL* means the specific Java library used by Panoply for interacting with netCDF may insert extra curly braces in multi-dimensional data that are not strictly speaking part of the CDL format. In consequence, ncgen may not operate properly on Panoply's export to regenerate the original .nc file.

The Java library itself lacks code to implement ncgen. That would be a massive project to write from scratch; Panoply is essentially a shell that harbors as many pre-written open source Java libraries as possible.

Thus export from Panoply is one-directional; data can be visualized, edited, and processed for statistical attributes (ie for outliers, mean, variance and palette optimization) but not re-imported without addressing curly brackets (if present) and using command-line ncgen.

However users can study the header sections of CDL* files, make one appropriate to their situation, stub in their own data from excel, and if compliant, run ncgen and import into Panoply.

Scripting in PanoplyCL was originally motivated by situations where people needed to be able to go back and re-create plots with modest changes. The classic case was a scientist at Goddard who included 40-50 Panoply plots in a journal paper submission and had to re-do them six months later following referee criticisms. Had scripts been saved, small edits could have made the fixes.

I asked about re-doing the unwieldy palette menu as a simple database with sortable fields like alphabetic, sequential, divergent, perceptual; by number of colors (odd, even, 8,9,10,… 256); or by most recently used.

Something will be done about the palette menu down the road. For now, some not-yet-documented palette navigation is possible by typing (eg R for red). This can be effective, depending on if palette names were set informatively.

I asked about palette scrolling: up and down arrows in the Scale tab, acting on the user’s current sort order, visualized in Panoply's map view. This could help find the best palette as it's difficult to anticipate in advance how a palette will look in an animation.

 A JavaScript that walked through the top ten in palette list (as sorted in advance by the user) would be easy enough, with the output of stills then displayed as layers in a Gimp animation.

Panoply works within 8-bit color (alternatively called indexed color, look-up tables (LUTs), or color tables) but exports out into ordinary 24-bit png files. This makes sense because 256 colors is far more than can be distinguished on most climate science maps; indeed ColorBrewer recommends 8-9 as the upper limit if categories are established. However 'continuous' palettes can provide an attractive sense of smoothness to data display with enough precision to discourage binning down.

NCAR, an atmospheric research center in Boulder CO, provides an extraordinary gallery of 438 color tables classified into 31 categories. There is some overlap with the 100 palettes that come with Panoply but together these make a great start on a palette database with applications to climate science graphics.

Each of the NCAR palettes is shown in action on a set of earth maps (2nd image) that can be expanded to a decent scale and include an Arctic Ocean view (3rd and 4th images, needs clicks).

Indexed color is supported by Gimp but only as a sideline. The dialogs are scattered all over the interface, with editing, re-ordering, and substitution possible but awkward, while the provided set of default color tables is clueless. However it is easy to make and add custom palettes with the excellent gradient tool.

ImageJ takes a more scientific approach to color tables and allows quick scrolling through a well-chosen set of furnished look-ups. It also allows export of numeric rgb tables which is essential in coordinating import to Panoply. (
Title: Re: Creating Animated GIFs
Post by: A-Team on September 28, 2017, 11:48:37 PM
The NCAR palette gallery contains an impressive 436 palettes classified into 31 groups. However 195 are reduncant, appearing in multiple locations, leaving 241 unique choices. The urls for these palettes, png files of size 500 x 101 pixels, have sensible file names, meaning ImageJ can load them from a list into a stack.

That stack can be exported as an animation into Gimp which allows very rapid scrolling through the possibilities for coloring a given map. They can also be concatenated into a single image. That is convenient for determining which are linear in lightness or in perceptual luminance (CIE color space L).

These give equal perceptual color bins, unlike the notorious rainbow palette. The test is, upon gimp 'desaturate to luminance', whether the palette descends to a monotonic increasing grayscale. Most are immediate rejects and the others can be converted to a numeric list of grayscales in ImageJ and tested with spreadsheet subtraction for negative numbers (non-monotonicity).

The database of NCAR palettes, as processed above, is appended below as a text file that is really csv. A set of unique palettes, in alphabetic order, is also attached as a scrolling list. The original NCAR order is also provided.

Some issues remain as ImageJ cannot export a png animation (see and and instead struggles with a master color table of 256 that cannot possibly for all the gif layers. Each frame in a GIF specification animation can have it own separate 256-color palette and each color table at NCAR and Panoply is compliant with this, there's no 24-bit color here so a gif animation of a palette gallery should work..

The forum software only supports a handful of old formats, not including animated pngs. These run just fine on web browsers such as Opera, Chrome and Safari. Gimp has a plugin for APNG but it seems somewhat problematic. The situation is better for ImageJ.

The current PanoplyCL will only retrieve locally stored palettes with the 'Open' command in the File menu. Remote urls leading to palettes do not work. Consequently it appears necessary to download all the individual NCAR palettes as individual files. However, one-time check, their site does not allow ftp connections so this is fairly tedious. However, individual palettes of interest are easily loaded up in Panoply as temporary or permanent active palette.
Title: Re: Creating Animated GIFs
Post by: Dryland on September 29, 2017, 04:34:39 AM
Time for an update from me. Progress can be viewed at ( It's still only a proof-of-concept, but a lot has changed behind the scenes.

Under the first link are the most recent REB_plots files from ESRL. I'm running a cron job to launch my program several times a day, so new images will appear each time ESRL updates their FTP site.

The second link has daily images created automatically by downloading the latest REB.(date).nc file(s) from ESRL and running them through PanoplyCL. This is still using the example script that A-Team shared, but there is now a templating engine in place that can rewrite the script before using it, replacing placeholders (such as [dataset] and [output_file]) with the right values for each invocation. This can be used to customize palette choice, scale, dimensions, titles, and all the other attributes that are scriptable. (And any number of different templates can be created for different tasks.)

The same templating engine is used to customize HTML templates to make the site pages. For example, in the template for the panoply.html page, the placeholder "list panoply" appears. (Square brackets omitted to avoid confusing the forum software...) When the real html page is generated, this is replaced by a formatted list of links to all the Panoply-created files that were generated earlier in the run. Everything is generated/uploaded without any manual intervention.

There's a framework in place for invoking command-line programs, so it is easy to add more. So far I have PanoplyCL and ImageMagick support, and will be adding things like Gimp, ncgen, etc. as the need arises.
Title: Re: Creating Animated GIFs
Post by: A-Team on September 29, 2017, 01:25:54 PM
Awesome progress! Fascinating to see where all you go with this.

If ever there was a process that needed automating, this would be it. The daily roll-over is time-consuming for me yet it only makes sense to work with 'the latest'. It seems like ESRL outputs have settled down now, though some products still need fixes and more explanatory prose.

The PanoplyCL manual has scripts that cover almost every situation, notably scrolling through a variable parameter to make time series or height/depth slices. These are small add-ons to the script posted above. The core of all these scripts is setting basic display parameters, followed by whatever precious add-ons that do some modest AI.

I haven't been able to get clarification on why, given the current parameters of an open map can be saved to Preferences (overwriting whatever was there), there is no outside saving and reloading of parameter sets.

However if there are say five template maps open with five different displays, it is easy enough to switch back and forth and re-establish their preferences to make new maps with these preferences. Once those template files are closed though, their preference settings are lost.

Even if a CL script is saved out, there is no way to run that script except from command line. If the script just sets parameters, there's no way of loading them in. It does record them though for map replication later.

To me this is getting away from the whole point of Panoply, which is transparency and ease of use. There are users out there who may have great visualization skills but no inclination to coding. This is all about communicating out what may be buried in the data and improving on inept representations of it. Those users have a contribution to make.

If there were only some way of running the existing collection of manual scripts from the GUI menu interface, that would take care of 95% of the use cases and empower a talented citizen-scientist who has been frozen out of the .nc world. Admittedly, at some point there are big advantages to delving into javascript or controlling it remotely as with your master templating engine.

It seems like the choices are (1) enabling more of our 1300 members to make innovative displays with ESRL and other sources, (2) archiving providing just enough to reduce the work involved, or (3) doing everything imaginable for them without their engagement. There's no question that time spent making forum graphics is time not spent on their analysis, interpretation and commentary.

I have not yet delved into problems with the ncdump <--> ncgen roundtrip. The holdup seems to be the curly braces that CDL* inserts into multi-dimensional data. I haven't seen any multi-dimensional examples yet in ESRL so CDL* should be the same as CDL here, and ncgen should work on them.

There is some interest in resolving this so people here can make or edit excel spreadsheets into CDL into nc into Panoply maps. There may be some simple grep that straightens out the curly braces.

I'm still working on automating palette choice, or at least cutting the gallery options down by pulling up a half dozen appropriate ones for the user make the final decision. It is not feasible to run a map through 450 palettes and remember which ones worked. A lot of palettes show up completely unannotated; no description at all of how and why they were constructed.

The 'trending' palette today is even grades of perceptual luminance, that is, adjacent colors are discriminate-able by the human eye and precede evenly in the grayscale sense. However it's almost easier to make these from scratch rather than go through big palette galleries to determine their luminance attributes. On the other hand, with a kazillion people making climate science maps, some really awesome palettes have emerged regardless of underlying design motivation.

The CDL view is key to data-driven palettes, ones where statistical considerations of the data distribution impose restrictions on color numbers and range. Here the Panoply export gives a satisfactory start on that. That takes on another dimension optimizing palettes used in animations where between-frame statistical changes could be the driver. It's not clear though whether at the end of the day this would make any real perceptible improvement.
Title: Re: Creating Animated GIFs
Post by: A-Team on October 04, 2017, 11:02:49 PM
Perceptual palettes offer three main advantages: better color discrimination by viewer, desaturation to grayscale for arithmetic operations between graphics, and proper display as (optionally draped) 3D surfaces. This does not resolve issues with contextual contrast: the 4 bars in fig.1 below are all the same shade, neutral gray 128.

ImageJ --> Analyze --> 3D Surface Plot, donated by KU Barthel, comes with two convenient macros, one that rotates the view around like the spinning globe above and another that works through a stack of images to produce a stack in surface view. (

Drag and drop of files to the Mac dock is currently broken in ImageJ but it works to drag files to the control bar. This is a huge boon because ESRL plots of the same graphic from different dates can be fished out through file name search and loaded as an ImageJ stack which the macro can then convert to a 3D stack. (This requires all other extraneous images to first be closed.)

In Gimp, these images would all open separately. The workaround is tiling them up using the 'Filmmaker' controller (which allows open extraneous windows), slicing them back up, and loading as layers.

Gimp also allows easy analysis of palettes and indeed whole galleries of them. (Few collections note which palettes are perceptual, defined here as having an evenly stepped grayscale for the luminance channel when decomposed from RGB to CIE's Lab color space, or equivalently the desaturate-to-luminance command is used.

It's easy to convert bad palettes to perceptual ones: Color --> Components --> Decompose --> LAB --> replace L channel with grayscale gradient 0,255 --> Color --> Components --> Compose --> RGB. The other channels can be treated as well to further color diversity and discrimination as RGB. This process may or may not retain desirable characteristics of the original palette.

In multiplexing climate science displays, important for displaying related parameters simultaneously, a problem arises from the gif animation format. Gifs are indexed 8-bit color, 256 colors is the maximum allowed per frame (these can be a different 256 in each successive frame however).

If each climate parameter is displayed in its own continuous palette (256 colors), then overlays have to be dithered back down from 512 to 256 and so won't display properly. Here two 7-bit palettes of 128 each would work (or four 6-bit palettes of 64 each). The images and their palettes can then be properly displayed in multiplexed animations.

There's an easy way to whittle down palettes of 256 sequential colors to 128 and lower. First, the starting palette has to be confirmed. An 8-bit palette should come only in 256, 512 or 1024 pixel widths. If not, some colors are wider than others. The color picker should select columns 1 pixel wide x the full height.

Many internet 'palettes' are dithered rubbish with tens of thousands of colors, often with boundary issues and internal demarcation lines. This can arise from saving as lossy jpg rather than superior png.

It's best pass to indexed color set at 256, then return to RGB to crop out a central line of 1 pixel height from these, resize with bicubic to 256 pixel width, resize to 40 pixels high (or more) using no interpolation, and then check with Info --> Colorcube Analysis to see how many colors it actually has.

Next, select all and used Edit --> Stroke Selection to create a border all around the palette using any foreground color not found in the palette. Now change foreground again and turn to Filters --> Render --> Patterns --> Grid to set up a vertical mask. The ones provided below expose only every 2nd, every 3rd, or every 4th color of your palette. Add an alpha channel if not already there; select the masking bar color and delete. Finish up with Image --> Zealous Crop; this strips out all vertical lines of constant color, leaving a palette with an optional 1 pixel border.

These condensed palettes still offer plenty of colors. Indeed the 'spacing' provides better discrimination than the original continuous palette, in addition to solving the multiplexing problem within gif format.

Note Colorbrewer2 and others offer much smaller categorical palettes in the belief that viewers really cannot distinguish between more than 8-12 colors. However using paired, abutted or internally ramped palettes, those numbers can be exceeded. Discrete colors in effect create contours without distinct contour lines. They can also pick out valid patterns in data and create artifactual ones.

Maps in discrete palettes can be scored, ie how many pixels of each color are used in the map
 (presented as percentage or line graph of histogram). By walking through the map's embedded palette first with the contiguous and then with the non-contiguous color picker, these numbers can be captured into a spreadsheet.

Alternatively, ImageJ can process the whole image into text RGB lists from which these same numbers can be extracted from background of boundary, lat-lon grid, land, and legend pixels (which must never use palette colors.)

More "best links" to palette design sites and palette collections: ( ( ( ( ( ( ( ( ( ( ( (
Title: Re: Creating Animated GIFs
Post by: A-Team on January 07, 2018, 09:24:35 AM
I've noticed some confusion concerning how to combine data series according to arithmetic, statistical or logical rules.

This can be done in Panoply provided two maps have been made and are open by clicking on the 'Combine Plot' icon. The options are then given as shown below in left column. (Here 'merge' refers to over-writing NaN with data when available.)

It is not possible to then combine a third map with the result of combining of the first two. That would require exporting the first product and then re-importing it as a new netCDF file. However Panoply only emulates 'ncdump', not 'ncgen' so the latter would have to be done in a command line tool.

It is also fairly tedious to combine say a month of two series of daily maps as 'Combine Plot' would have to be repeated 30 times. It may be possible though to script this in PanoplyCL version of Panoply.

A better option in some ways is to select the grayscale palette for both maps, save them out, and combine in ImageJ or Gimp. All Panoply palettes are 8-bit [0,255], in effect color lookup tables or CLUTs so map color can be added back later.

Note too that Panoply uses bilinear for interpolation to make larger size plots, ie not bicubic or sinc. So if you are exceeding the intrinsic scale of the data, your maps sizes will be be rescaled in this way (unless interpolation has been shut off in Preferences).

The number of colors never exceeds 256 however. Panoply does not produce a dithered product: the colors in the associated map palette are exactly those of the map.

Once the two maps have been exported as grayscales, they can be combined in ImageJ (middle column) and/or Gimp (right column). This makes sense if the grayscale value represents a simple numerical data parameter. Data cannot be represented at higher precision than 8 binary 'decimal points'; there is no 16, 24 or 32-bit precision available within Panoply; only ImageJ supports this.

For example, Gimp will conveniently averages a whole month or year of data when given 30 or 365 layers of grayscales. The product is a new grayscale where each pixel represents the numerical average of the stack of pixels above it. That can be re-colored using any of the thousands of climate science palettes online using its CLUT in 'indexed color' (gimp) or lookup table (ImageJ).

It is easy to pass Panoply images back and forth between Gimp and ImageJ. The key trick for combining more than one pair of maps, ie combining time series (animations) is to horizontally tile up all the layers into a single image in gimp. After completing the math or logic operation as one step on a pair of tiles and applying the CLUT, the tiles are sliced back into single frames and reloaded as layers for saving out as animation.

Most of the operations work pointwise, at the level of a single pixel. They do not consider the values of neighboring pixels (2D local information). However some of the ImageJ commands will consider a 3x3 or larger neighborhood, eg a median filter.

The most important of these is the adaptive contrast tool (menu item CLAHE). For satellite imagery
the actual observed corrected value is critical for the original quantitative design purpose, say radar return brightness for salinity. However ice and snow scenes tend to have very little contrast. So if your interest is primarily in visual effective display, it may be better to use non-linear contrast stretching (some form of grayscale histogram equalization). Example: ASCAT imagery can be enhance to show ice pack motion far better than the original.

A second topic in combining Panoply maps is illustrated below. This involves mask replacement. The main application is to pairs of maps which show ice and open water properties, respectively. If the first netCDF file shows a property only defined over the icepack, uses a second color for open water, and a third color for land, it can be combined with a second netCDF file showing a different property (usually for the same date) only defined over open water.

This only works because Panoply does not dither its output maps. The 'select by color' global tool in Gimp can then select each of the complementary sets in the 3+way partition. By 'deleting' ice to transparency in the upper open water layer, water and ice layers can be combined into a new layer using 'new from visible'. I sometimes replace land with the background blue-gray of the forum which gives a further effect of transparency, only showing data.

The only issue arises if a gif animation is being made. The number of colors there is limited to 256. So if two complex palettes are used initially, this limit will be exceeded and the animation ruined. However it is easy to cut the initial palettes down to 128 colors each and import them into Panoply.

The example shows sea water and snow/ice surface temperature from two ESRL netCDFs using a single palette. This works ok visually even though the temperature ranges are different and the ice has a natural reddish boundary demarcating the two. It would have been possible to do additional like snow depth on land but visual complexity may begin to outweigh the benefits (of being easier to understand data presented together rather than in a pair of images or pair of unsynchable animations).
Title: Re: Creating Animated GIFs
Post by: A-Team on January 08, 2018, 06:02:43 PM
The main mistake seen in Panoply-type map products has to do with setting the range. The impulse is to have the palette represent the full data range (the 'fit to data' button in Panoply scale tab).

However the full range often has noise, outliers and/or long thin tails, for example Arctic ice thickness today is overwhelmingly in the 0-2 m thickness range but a few 5-6 m thick floes still hang in there.

In this case 'fit to data' will make poor use of the color palette. Too many palette colors are wasted on thickness bins that are scarcely used and so barely visible in the product; not enough palette colors are left to resolve thinner ice classes.

The first approach to rectify the display is a 'histogram equalization' command that non-linearly stretches the image (and embedded palette) to create an optimal match between value frequencies and the number of colors available to represent them.

Panoply does not offer this, just a monotonic log10 virtual adjustment of data which helps in the ice thickness instance but not very broadly.

Panoply does not require that 'fit to data' be used. Instead the lower and upper ranges can be set manually. This has two effects: linear stretching of the palette to just the new range and lumping of outliers to the single colors of the left and right scale triangles.

These values cannot be saved as Plot Preferences. That wouldn't make sense because the next map made might be salinity or surface temperature so thickness ranges would be generate nonsense.

However, exporting as PanoplyCL script will save those settings. Thus in the common situation making say a 100-day animation of a single Geo2D file within a set of 100 daily netCDF packages.

The script can be revised so that it scrolls through the list of local file paths saving out each png as an animation frame. Indeed it could scroll through a choice of range settings too.

//how lower and upper scale settings appear in a saved PanoplyCL script
myplot.set ( "scale-min", 0.5 );
myplot.set ( "scale-max", 1.5);

The animation below scrolls around various range settings of the same UH SMOS sea ice thickness file for 03 Jan 2018. Watch how the ice display changes as the min and max range numbers are varied.

Depending on what the range of most interest is, the appropriate setting will effectively highlight the chosen ice classes (while suppressing less interesting ones as single colors that can be picked and replaced later in Gimp).

The gif version has been reduced to forum constraints and so is slightly dithered by sinc rescaling; it is now running with a click but without a new tab. Exporting this gif as Movie mp4 reduces file size by 4:1
Title: Re: Creating Animated GIFs
Post by: A-Team on January 09, 2018, 04:34:56 PM
Here I am just exploring why the forum software is not running animations any more. It seems to be size dependent as you can see 100, 300, 500, and 599 pixel widths are working but not the usual 600 or 700. These load fully and display in a new tab but that is a big step backwards from previous forum settings. However a new version 600b is now playing so perhaps it is a fault in loading. However 601, 650 and 699 are loading but not playing.

These are tiny 250 k files of just 3 frames so it is not a size issue.

So what got changed at the admin level??? Actually for purposes of the Arctic 750 x 700 would be a significant improvement over the older 700 x 700 because of the intrinsic proportions of the Arctic Ocean in 'Greenland down' NSIDC position and the native resolutions of major data sources. (However other areas like Antarctica might not especially benefit.)
Title: Re: Creating Animated GIFs
Post by: A-Team on January 09, 2018, 10:36:58 PM
Below I look at whether RASM ice thinness ten-day forecasts can, after some adjustment, serve as UH SMOS forecasts (or provide it with pole hole infills). The animation compares 26 Dec 17 for the two ice thinness products. These are carefully adjusted to the same upper and lower range of ice thinnesses.

It can be seen immediately that the agreement between the two is quite poor and varies regionally. Thus it isn't possible even in the grayscale domain to adjust endpoints, stretch contrast, and adjust midpoint gamma to bring the RASM into a good match with SMOS.

This is probably because SMOS is 'directly' observable and optimized for thin ice measurement whereas RASM is largely just driven from one day to the next by a physical model off a questionable initial calibration, though both are constrained by AMSR2 to the same ice pack edges.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 10, 2018, 11:45:46 AM
Panoply provides 9 main plot sizes that are described in rather odd units. These can be further interpolated in 10 'size' increments. The table below looks at pixel dimensions of the resulting saved-out stereographic circle png images.

Those are computed by opening each size graphic in Gimp, adding an alpha channel, whiting out text and color key, selecting the contiguous white boundary, deleting the selection, followed by "autocrop image" to cleanly delete transparent rows and columns to obtain the minimal square containing the image.

In terms of 700 x 700 pixel limits, the best fit is Panoply size 190 which will give a circle of 686 x 686 pixels. Each 10 units of Panoply 'size' adds 36 pixels to width and height.

There is an issue here with interpolation. The data comes on a grid of a certain resolution (shown in the netCDF header). That resolution might be fairly meagre. Panoply strictly uses linear interpolation, never bicubic, to get up to the requested map size on the requested map projection. There's an option to shut off Panoply interpolation, use a smaller Plot size comparable to data resolution, and then bump up to final map size later with Gimp bicubic or sinc. However that will wreck the correspondence between palette colors and map colors and not allow pixel counting of bins. Consequently it's probably better to precede as described here.

At this width, the maximal size of the palette key with no-gap triangular wings ('Bar Width' parameter) is 50%. This will have a height of 46 pixels (including scale numbers) regardless of its width setting. The width changes as shown in the second table. A wider key allows for more internal subdivisions. That can create conflicts when say a discrete palette with 9 colors is chosen, those can sometimes be resolved by adjusting the range min/max settings.

Since only 14 pixels of height are left from the 700 available, the scale key needs to be lifted into the circle, leaving 686 x 640 of data. This is seldom a problem for the Arctic Ocean centering on the north pole since it is asymmetric, meaning only lower Greenland is covered up by the scale.

Technical notes: the image below shows the circle and key only. The image is really rectangular with transparencies filled by the forum color. The trick here is to take a small screenshot of the two alternating forum colors, fill an extra layer with the appropriate one, and then blend it into the image and rendered text with the 'darken only' layer command in gimp.

In computer graphic manipulations, it is highly desirable to avoid  'micro-management' of image elements, ie squinting at the screen and nudging elements into place. Here the key can be cleanly excised as described above, then added as a floating layer. Here gimp will center the layer but only put it at the top. However by using a vertical flip, it can be put cleanly on the bottom and merged down into the receiving circle image.

1082 x 1082      max.png  size 300
 866 x 866      king.png  size 240
 722 x 722    sjumbo.png  size 200
 686 x 686   sjumbo-.png  size 190
 650 x 650     jumbo.png  size 180
 578 x 578    xlarge.png  size 160
 506 x 506     large.png  size 140
 434 x 434  standard.png  size 120
 362 x 362     small.png  size 100
 290 x 290    xsmall.png  size  80

25%  344 pixels
30%  412 pixels
35%  482 pixels
40%  550  pixels
45%  619 pixels
50%  686 pixels
Title: Re: Creating Animated GIFs
Post by: A-Team on January 14, 2018, 05:26:30 PM
It's been claimed that no one ever looks at 99.999% of all archived satellite scenes, even the cloud-free ones. It's also true that 99% of the web sites archiving single daily satellite records never put together a simple time series movie that could give visitors a quick preview of what was in the collection.

For example, NOAA's ASCAT archive of 9,040 gifs in a single directory does see too many visitors who want to 'download them all' with wget or plugin. However only 1820 of these show the Arctic. Still, at 547 KB each, that is 995 MB. A movie size would be much much smaller, depending on quality retained, rescaling, restricting to even-numbered or just weeks, and codecs used.

It's very much worth doing but would it be distributable?

The ASCAT time series for 2012-18 provide an excellent direct record of ice pack motion, though weather artifacts in individual images can be quite distracting. The gif below looks at using UH AMSR2 double-masking of land and water, leaving only ASCAT ice (which benefits greatly from normalizing and adaptive contrast enhancements).

The land mask is a one-time stationary product, made here from the 6.25 km AMSR2. That does not have a Geo2D file in its netCDF so it cannot be redisplayed in Panoply without its lat-lon lines. Here the Gimp color picker was used, along with single pixel 'grow selection' to replace the distracting red with a bland land mask color. Open water varies day to day but can be color-picked once for the whole time series when the AMSR2 are tiled. This has to be done at the original resolution (ie prior to rescaling to fit ASCAt's scale).

It is easy to escalate the double mask to a tiled layer floating above the ASCAT tile and apply after the latter is enhanced in ImageJ (which mostly lacks alpha channel masking). Gimp cannot de-tile beyond 100 frames due to a limitation in a plugin nor save out as a video so those operations have to be finalized in ImageJ.

Having done each year from 2010 to 2018 separately (which works better seasonally rather than per calendar years) to keep intermediate files small, it is easy to combine these within ImageJ either end to end as a continuous six year roll, or gridded into say a 2x3 rectangle which affords simultaneous display of the same date of each year.

This would make a gif gigantic file because of 365 frames x 6 years x 700 pxl x 650 pxl is a lot to display, even if reduced from RGB to grayscale and gif-differenced to compress. Using movie codecs, file size would become manageable, though display on say QuickTime would need prior testing. It's not clear if or what the forum would show.

Movies can also be extended by adding clips but that would only benefit making a long-term roll; there is no juxtapositioning option. Quicktime Pro 7.6.6 can do a lot more but while AAPL still distributes the Pro software(, it no longer sells the enabling key though these are sometimes offered free online or sold on ebay.
.. edit, layer, change all kinds of metadata, add and delete tracks, QT 10 can do less.
.. has the controls outside the movie, 10 has it covering a part of the movie.
.. can add effects
.. can open a wider range of codecs,  QT 10 has to convert a lot of formats
.. has excellent A/V tools to adjust the video brightness, color, contrast, tint, playback speed, audio volume, audio balance, bass, treble, pitch shift, and playback.

2010 vs 10/11
2011 vs 11/12
2012 vs 12/13
2013 vs 13/14
2014 vs 14/15
2015 vs 15/16
2016 vs 16/17
2017 vs 17/18 Jan
2018 Jan only

To crop image batches identically (to the Arctic Ocean plus a bit of the Fram), use 320x350 as the lower crop on ASCAT and 375x375 as upper crop. After reducing to 8-bit grayscale, 365 images of that size require 48.7 MB or 170 MB if rescaled to 700x700. Since ASCAT is available in the same format back to 2010, those numbers have to multiplied by 8 to reach 31 Dec 2017. That is well within the RAM memory limits of ImageJ (and my Mac) but some menu operations might be 'challenged' allowing no room for error. Hence rescaling is best deferred to the very end of the process.

The 9-year gif for today below is 3.7 MB; movies don't work well when so few frames are available. The still png showing all 9 years has 2010 in upper left and ends in 2018 in the lower right. The goal here is to animate this still image into a good quality movie, preferable at native ASCAT resolution (375x3 --> 1125x1125 (or even better to 2250x2250) to allow all years to play simultaneously in contrast-enhanced mode, possibly with masked and false color versions.

Note 2018 is not unusual in having thick CAA floes having rounded the Beaufort bend and stringing out up into the Chukchi (where they will melt out next summer): six of the 9 years show this same pattern for this Jan date.

Note the widespread confusion between NSIDC's averaged monthly motion at each point (vector sum of daily motion) and actual monthly floe trajectories which are line integrals of the daily displacement of the floe from its current position. It's not possible to get at the latter using the former.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 14, 2018, 06:10:50 PM
A larger improved version of that last multi-year Beaufort-Chukchi comparison file, needs a click.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 15, 2018, 12:25:34 PM
Just testing a preliminary four year array of ASCAT. 121 days from Sept minimum until 13 Jan. Forum software is accepting the .mov uploads but not displaying them. Believe further admin enabling is needed as was done for mp4 and youtube.

Back to converting mov to mp4 (which displays for some but not many people). It sometimes takes a reload to get the actual image going; on the first go-round it only shows the controller. But it is going nicely now. It cannot remember my preferences (loop, play, controller off) and always goes back to default (controller on).

It is fairly wide though at 750x750 which is the natural scale for ASCAT's Arctic Ocean images. The dates need to be offset a bit to the right so that they are more readable over the snowy white background of Greenland. Four years takes about 3 MB so the full eight year record is manageable saving at 'normal' quality. Probably best as 2x4 array. It looks like 8 years at 365 days is too large but 8 years at 182 or 121 would both come in under 10 MB which the forum has accepted in the past. NSIDC uses 52 days per year or 1 date out of 7 for sea ice age animations.
Title: Re: Creating Animated GIFs
Post by: gerontocrat on January 15, 2018, 01:07:20 PM

My PC with Windows 10 plays it (.mp4) OK when just clicking the  play button.

Both .mov and .mp4 download OK and clicking on them gets them to play in the windows 10 utility - "films and tv". Also viewable in other programs (see below).

Not a clue as to whether this is any help.

Title: Re: Creating Animated GIFs
Post by: A-Team on January 15, 2018, 03:48:44 PM
Yes, very helpful. Between windows and mac, it should work for most people here. The key seems to be at my end, processing the .mov saved out from ImageJ to .mp4 using a modern free online converter like

Now I'm wishing there was some way to specify a delay before it begins another loop. One could not have it loop at all but press again on the start controller each time. I don't think it would work to add repeats of the final frame of the parent 65 MB gif animation because the codecs would probably see nothing changing and suppress them I suppose credits or ads or annotations or voice-over or music could be added but I'm not going there at this point.

More experimentation on the early stages first. The new attached has been downscaled to 650x650. that seems to max out the room available in forum width. The second version, ICA indexed color, does well on thicker ice but the green needs to be replaced. The process has also wiped out the day numbers.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 17, 2018, 06:58:17 PM
User's Guide to a new Panoply Time Series Scripting Tool

Introduction Climate science data is mostly stored and distributed as netCDF (.nc) files. Those bundles multiple data files that need to stay together. These are almost always geolocated, each data point being tied to its latitude and longitude (plus ocean depth or atmospheric height if applicable).

The geolocated files within a netCDF bundle are called Geo2D files; they are the only ones that Panoply can use to display the data on a map projection. Typically each project creates a new netCDF bundle each day as new satellite imagery is processed by product pipelines and models, though some like bathymetry are one-off fixed resources. 

It is quite difficult to un-bundle netCDFs and re-assemble a time series of a particular Geo2D file as a single new netCDF. The tool described here provides virtual un-bundling, all that is need in most situations.

Panoply is easy-to-use free software used to make maps that display climate data. Some 52 parameters control the map's appearance, notably the map projection, its center and horizon, data range displayed, and color palette.

After some interactive experimentation, choices can be saved out ('Export CL Script') as a human-readable text (file type .pcl) that can regenerate the map using the PanoplyCL command line tool. The map can in effect be edited at the level of the script by changing parameter settings. The tool here automates that process on a large scale to produce a succession of related maps, usually for an animated time series.

A Panoply script contains 59 lines of which 7 are optional explanatory comments, 45 control map and legend appearance, and 5 provide text boxes that appear outside the map itself.

upper title      myplot.set ( "subtitle-text", "titleText" );
lower subtitle   myplot.set ( "title-text", "subtitleText" );
label for scale  myplot.set ( "scale-label-text", "scaleUnits" );
left footnote    myplot.set ( "footnote-left-text", "leftFootnote" );
right footnote   myplot.set ( "footnote-right-text", "rightFootnote" );

These latter have limited controls on font, size, placement and character length but can nonetheless be filled with date appropriate to each frame, such as date, time, data range restrictions and comparative or summary statistics.

Text is rendered (dithered, not retained as vector layer) irrevocably onto the map periphery unlike map colors which are exactly those of the scale legend. Text can be harvested later into a small consolidated box for better control over final placement, for example over an unused portion of the map.

Panoply places an additional line of text at the bottom showing data range extremes; this can be turned off but not otherwise altered. In particular, it cannot show palette squeezes (range restrictions). The 'fit to data' button in the scale menu is not represented in the script as a central footnote control but as the true/false choice below. It lies on the same line as the left and right footnotes.

bottom data range   myplot.set ( "scale-minmax-note", true );
lower data range    myplot.set ( "scale-min", 0.99999 );
upper data range    myplot.set ( "scale-max", 1.99999 );

The initial script can be modified by 'mail merging' in a list that increments parameter placeholder settings in some useful way, for example over a date range.

The tool described here concatenates all these variant scripts into a single text file that, when run as a large PanoplyCL script, generates a separate output map (.png image) for each line in the list, each according to its parameter settings. These pngs make up the frames of the subsequent animation.

In the example, if the list pointed to 365 netCDF files, a year's worth of daily images would be created. These images, uploaded into ImageJ free software, can be saved out as a .gif animation or .mp4 video.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 27, 2018, 04:10:02 PM
This section goes through real-world examples illustrating how the mail merge tool automates the production of climate science data animations by preparing scripts that PanoplyCL can run to generate the maps that make up the frames.

For brevity, only the relevant lines of a PanoplyCL script are shown. Much longer lists (many hundreds) would normally be used instead of the 4-5 shown in the examples (which could easily be generated manually).

You can best follow along by replicating the examples which requires that the respective netCDF files plus Panoply, PanoplyCL and their manuals have been downloaded, installed, and assimilated.

Example 0 This example just shows how the merge tool can generate a list of download urls for ASCAT daily imagery. Since those are available from 2010 on, leading to 2948 files totaling 2.4 GB up to mid-January 2018. Every 5th day however might give simplified but completely adequate depiction of ice motion. The desired dates are easily built by fill-down in a spreadsheet since ASCAT uses day-number instead of day-month.

Because ftp isn't properly enabled at this host site, the files are best retrieved by 'download it all' or 'bulk url opener' web browser plugins. About 1 in 50 ASCAT (or AMSR2) days are satellite malfunctions, giving 404 duds instead of pngs; these can be avoided based on smaller file size.

tool template:[year][dayNum].sir.gif

mail merge input values:

output list of urls

Example 1a The netCDF bundle here from ARDEM just provides ocean depth nearing the Bering Strait. It consists of 1D latitude x and 1D longitude y files showing the ocean domain mapped plus a single Geo2D file z providing geolocated bathymetry (and land DEM). Using these xyz coordinates relative to the earth's surface and current sea level, Panoply can draw a map using colors to show depth.

This netCDF could, but does not, provide a time coordinate for past and/or future sea levels that would allow Panoply to generate an animation of inundation and recession during the late Pleistocene/Holocene. However we can use the 'mail merge' tool and auxilary data to create these.

It all begins with map experimentation. Here, that shows that while the continental shelf is quite wide in the East Siberian Sea it falls off rapidly to great depth at about 150 meters. Further, land elevations in the netCDF aren't of interest in this context, though they do indicate river drainages.

This means the default 'fit to data' range settings of -9677 and 5965 (units aren't specified) are far too broad to make good use of the color palette so they must be set. The Arctic Ocean reaches its greatest depth of 5669 m in the Fram at the Malloy Deep, the Litke trench attains 5,449 m, the average depth is ~1,000 m and 60% is less than 200 m.

A simple spreadsheet bumping depth in steps of 5 meters out to the edge of the shelf will show the flooding of Beringia since the Last Glacial Maximum. The key placeholders in the PanoplyCL script are called "scale-min" and "scale-max" but directory file paths and so forth are also important, as are useful names for the output graphic frames. (More could be done in the title and subtitle text boxes.) The placeholders are put in brackets so the mail merge tools knows what to put in where as it goes through its little csv database:
tool template:
var ncdata1 = panoply.openDataset ( "[FilePath]" );
var ncvar1 = ncdata1.getVariable ( "[Geo2D]" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
myplot.set ( "scale-min", [depth] );
myplot.set ( "scale-max", [seaLevel]);
myplot.saveImage ( "PNG", "[pngName].png" );

mail merge input values:
FilePath,Geo2D,depth,seaLevel, pngName
/Users/,z,-150,0,Beringia at -150m
/Users/,z,-145,0,Beringia at -145m
/Users/,z,-140,0,Beringia at -140m
/Users/,z,-0,0,Beringia -0m

output PanoplyCL script
var ncdata1 = panoply.openDataset ( "/Users/" );
var ncvar1 = ncdata1.getVariable ( "z" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
myplot.set ( "scale-min", -150 );
myplot.set ( "scale-max", 0);
myplot.saveImage ( "PNG", "Beringia at -150m.png" );

var ncdata1 = panoply.openDataset ( "/Users/" );
var ncvar1 = ncdata1.getVariable ( "z" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
myplot.set ( "scale-min", -145 );
myplot.set ( "scale-max", 0);
myplot.saveImage ( "PNG", "Beringia at -145m.png" );

var ncdata1 = panoply.openDataset ( "/Users/" );
var ncvar1 = ncdata1.getVariable ( "z" );
var myplot = panoply.createPlot ( "lonlat", ncvar1 );
myplot.set ( "scale-min", -140 );
myplot.set ( "scale-max", 0);
myplot.saveImage ( "PNG", "Beringia at -140m.png" );
Example 1b
This is just a minor variation on the previous example that runs through 20,000 years of (sea level,year) pairs manually extracted from a careful paleo-reconstruction graph hosted on wikipedia. Because the rate of sea level rise accelerated during the early Holocene and then flattened out whereas the time interval between frames stays constant, a jerkier animation results than the one above.

Here the primary innovation is placing the varying data within the scale caption of Panoply. This can save quite a bit of 'post-production' custom work in Gimp because vector text and raster imagery don't size well together. Note the dates can't be presented with commas as in 20,000 because those are reserved for field separators though there's an easy workaround using two fields.

tool template:
myplot.set ( "scale-label-custom", true );
myplot.set ( "scale-label-text", "[SCALE CAPTION]" );

mail merge input values:
-126.0 m at 20 000 years BP
-123.3 m at 19 000 years BP
-120.7 m at 18 000 years BP

output PanoplyCL script
myplot.set ( "scale-label-custom", true );
myplot.set ( "scale-label-text", "-126.0 m at 20 000 years BP" );

myplot.set ( "scale-label-custom", true );
myplot.set ( "scale-label-text", "-123.3 m at 19 000 years BP" );

myplot.set ( "scale-label-custom", true );
myplot.set ( "scale-label-text", "-120.7 m at 18 000 years BP" );,416.msg139761.html#msg139761

Example 2 The case study here refers to Uni Hamburg's more complex Arctic Ocean ice thickness netCDF bundle. These are updated daily and archived online without registration barriers. Each day provides 10 Geo2D files, notably SMOS ice thickness, bulk ice salinity, land mask, snow surface temperature, total uncertainty, etc.

Imagine now only being interested in the snow surface temperature Geo2D. The archive here dates back to 15 Oct 2010 but takes a melt season break each year between day 228 and day 301. That's 1408 netCDF bundles at 22.34 MB for 31.4 GB of download of which perhaps 8% is project relevant.

UH provides a convenient wget file for command-line mode. A subset of that, say weekly, can be made from that template following instructions in Example 0.

The settings below fix the map projection to stereographic centered on North Pole in standard 'Greenland down' position with the Arctic Circle as horizon. This allows your map to be rescaled to match maps you might find online for which no netCDF file is provided.

// Open a dataset.
var ncdata1 = panoply.openDataset ( "Downloads/SMOS Hamburg nc/" );

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-xparam-1", 23.6 );

You can adjust these numbers in the friendly Panoply interface; the four lines below show up in the exported CL script. Two other common settings emphasize the Bering Straits (to show conditions in the Chukchi, Beaufort, and East Siberian Sea) or the Fram Strait/Svalbard/Severnaya Zemlya area.

Suppose now you viewed the -45.0 as a placeholder. Making a list that increments this by one degree 360 times, the new mail merge tool will output a concatentated script with the effect of displaying ice data on a rotating globe (restricted to the Arctic).

If the path to the file on your hard drive and the date associated with the netCDFs are also varied, the display will increment the data by a day for each increment of rotation. In other words, construct a 3 row, 365 column table in a spreadsheet, save as comma separated variables (.csv format), and paste into the tool.

var ncdata1 = panoply.openDataset ( "[PlaceholderPath] [PlaceholderDate]" );

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-lon0", [PlaceholderLong] );
myplot.set ( "proj-xparam-1", 23.6 );

Example 3. It can become tedious testing dozens of palettes trying to find the one that best displays a particular data set. The automation option generates a slide show of several dozen potential palettes by varying an initial map. According to the PanoplyCl manual, a list of what it currently has in stock will be given with this command: printColorTableList ( ) void

Alternatively, ImageJ will let you try palette collections: open your Panoply map in any palette, then drag any internet palette (.lut) onto the tool bar to exchange color tables, over all the frames in a stack if in montage view. That way, a new palette (to be used just once maybe) need not be added to the Panoply set which is already cluttered. allows quick trials of 196 palettes

tool template:
myplot.set ( "scale-colorbar", "[palette].[colorTable]" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

mail merge input values:

output PanoplyCL script
myplot.set ( "scale-colorbar", "CB_RdYlGn.cpt" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

myplot.set ( "scale-colorbar", "NEO_modis_sst_45.act" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

myplot.set ( "scale-colorbar", "NEO_ceres_insol.lut" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

myplot.set ( "scale-colorbar", "NEO_trmm_rainfall.rgb" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

myplot.set ( "scale-colorbar", "bathymetry.gct" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", true );
myplot.set ( "scale-outlier-gap", "THIN" );
myplot.set ( "scale-label-location", "ABOVE" );

Example 4 It is not so easy to harvest and re-position text from Panoply's 6 text input boxes once that text has been rendered onto the map background. However it is straightforward to consolidate and reposition text over unused portions of the map.

This is key, in conjunction with plot size 190, in arriving at a final product that fits within the forum's 700x700 pixel size constraints yet retains maximal map resolution. (In other words, the color palette and text lines ordinarily take space away from what's available to the map.)

The trick is to make a one-row montage of the frames, duplicate it, select a line of text across the entire montage, delete its complement to transparency, position it as a new layer over the map montage, then set its Gimp mode to 'darken only'. Since nothing can get darker than black text, it will pasted through; since nothing is less white than pure white, the white will get lost out to whatever lies underneath on the map.

Repeat the design composition with the other lines of text (and the scale color legend). Some of these lines may benefit from vertical resizing (ie, uncoupled from horizonal). Then, rather than flatten or merge, make 'new layer from visible'. This top layer can then be re-sliced in Gimp (100 slices max) or ImageJ back into animation frames.

The animation shows the gain: the map circle is originally 686x686 pixels but reducing the initial Panoply map to forum maximum reduces the 'content circle' to 520x520. This means only 57% as many pixels remain available to display data.

For a given PanoplyCL 'plot size', the text layers will be in predictable positions. Consequently the process can be automated.

... to be revised/continued.
Title: Re: Creating Animated GIFs
Post by: A-Team on January 30, 2018, 04:29:58 PM
Just some remarks on optimally processing ASCAT polar imagery to depict ice pack movement and making long time series, as restricted by forum file sizes and display width.

First note that the process by which the scatterometer instrument on the satellite makes these daily images from its orbital swaths is quite complicated but doesn't concern us as daily images in the archive are readily interpretable in terms of brightness contrasts between Arctic islands, floes, stable features, open water and so on. 

Second, regardless of the physical meaning of the calibrated scattering values, the images may download as color png images but are actually one channel 8-bit grayscales. These make such lopsided use of the 256 available grays on the Arctic ice scenes that they greatly benefit from basic contrast stretching.

For consistency, this is best done simultaneously for all frames on a montage, bringing them into the full range of grays distinguishable by the eye. Here [10,245] is better than [0,255] because extreme values are wasted and the slightly narrowed gamut leaves room for extra colors, here 20 that might be reserved for tracking lines while staying within the 256 that a gif animation is capable of.

Modest gains may be had masking out black satellite data holes and bright land masses such as Greenland as they distort the histogram being stretched.

Third, ASCAT circles of the Arctic have a limited intrinsic size of 1154 x 1154 pixels which extend out to a 45º horizon (ie Lake Michigan) and so don't offer that many pixels for the Arctic Ocean per se. An enveloping rectangle large enough to capture the AO and some of the Bering Sea, Fram, Nares, Barents, CAA channels and Siberian islands has dimensions of about 400 x 345 = 138,000 pixels.

Not counting pixels in the land mask, the relevant display reduces to 89,147 pixels or 64.6% of the total to depict the 9 million sq km of Arctic Ocean proper. Thus a given pixel represents about 10 km x 10 km of sea ice.

The basic ASCAT ocean crop will enlarge to 700 x 606 for forum purposes. However special topical areas like Fram export don't need to show the whole ocean. The question is, how much can small regions be enlarged without pixellating. Here ImageJ provides bilinear and bicubic interpolation; the latter considers all eight neighbors in a 3 x 3 box around an initial pixel and thus may be slightly preferable.

Fourth, regional contrast adjustment can bring out local features that lack sufficient contrast. It wouldn't be possible to enhance these with a global contrast stretch because what worked in one place in the image might work poorly in another. The ImageJ tool called CLAHE can do this at various levels on variously sized boxes. There's relatively little to be gained here by masking.

Color tables are inconsistently implemented in graphics software. Gifs become unportable between them. For this reason, after making a time series in ImageJ and false-coloring it, the gif should be montaged out linearly and copied over to Gimp where it can be re-sliced back to a gif there.

Here, gifs are better because they preserve individual days as separate frames. However file sizes become way too large. Movies such a mp4 greatly reduce file size but degrade the data internally with extreme codecs. However it still works very convincingly to the eye.

The mp4 below shows original ASCAT images over globally and regionally corrected contrast -- there's a vast improvement in feature recognizability. This instrument can see a lot better into the ice than it's given credit for.
Title: Re: Creating Animated GIFs
Post by: Dryland on February 10, 2018, 01:08:18 AM
It's time for another update on the software project and website I've been working on in collaboration with A-Team. The software now has a name, which is 'floe'. You can see the current state of affairs at (temporary URL as I'm still making major changes.) The whole site is generated automatically with a single command, and will do whatever work is necessary to bring it up to date with the latest source data.

The data source, for now, is the 2017 RASM-ESRL FTP archive. They went on hiatus in late December, but they are supposed to start posting updates again on 2/14 (with who-knows-what changes to the products published). An immediate goal is to start incorporating other datasets, like SMOS (, or others that people here might suggest.

As floe has evolved, it's taken two main directions:

1. as a read-only repository of interesting* graphics, both current and historical, that are either unavailable or hard to access elsewhere. (*interestingness is under construction, while I've been focused on getting the platform working)

2. as an interactive web tool allowing anyone to create their own graphics from public data sources, without having to install anything or become experts on every detail of the process.

#1 is embodied in the site linked above. The floe program downloads 49 GB of 2017 data files from the ESRL FTP site, and crunches them into 292 HTML files and 13 GB of images. It takes about an hour and 40 minutes (on a fairly low-end linux box in the Amazon cloud) to run all of 2017 from scratch. Once caught up, handling each new day's data will take a few minutes.

The tabs on top show different ways that floe can add value. "ESRL Images" simply re-hosts RASM-ESRL products that are normally buried in compressed archive files, makes thumbnails, applies friendly names, organizes by date, and so on.

"New Products" uses the same ESRL animations, but makes new animations by disassembling them into individual frames, and recombining the frames into new animations that show both history and forecast frames. This is only a proof of concept and much more interesting things can be done.

"Panoply Experiments" goes further into the custom realm, and shows that we can run PanoplyCL on the server against .nc files published by ESRL, with a custom template script that we fill in with any parameters we choose, to generate new images. (The next step will be to combine these daily images into entirely new animations.)

That's the "static repository" side. I've also done some work on the "interactive web tool" side, which isn't publicly available yet (it's protected by a login screen so that bots don't find it and see how high they can run up my AWS bill). I built a "mail merge" (template filling) tool that A-Team has been discussing in this thread - the next step will be to actually run the resulting PanoplyCL script on the server on demand. This can be taken much further - both as tools for experts, and as other tools that will allow anyone to build custom graphics by choosing options on a form.

So this has evolved into quite a large project, and I'm definitely going to continue and see where it leads, and what this community can think of to do with it.
Title: Re: Creating Animated GIFs
Post by: Neven on February 10, 2018, 11:34:52 PM
I had a look today and it already looks awesome!
Title: Re: Creating Animated GIFs
Post by: A-Team on February 11, 2018, 03:10:25 AM
Very impressive. This is the way to go, an over-arching architectural enabling vision rather than endless ad hoc graphics and futile recommendations that thousands of people to go off on difficult learning curves that they don't have time or interest for, despite having considerable end-user talent if they could only get there.

Meanwhile I am still delving into the human perceptual side of optimal presentation of scientific information, the idea being that numerical time series are basically really boring but if communicated effectively can suggest significant hypotheses of my favorite kind (known in advance to be true) that may still have to be ground out later by some conventional objective process  -- because it's not good enough to say hey look at this, obviously such and such is happening -- but at least it will be time well spent.

We are sort of stuck in a single time zone, a day goes by, there is nothing we can do to speed it up or slow it down. However a lot of processes in nature are taking place at vastly slower paces. By a simple compression of frame rate, we can bring these home, seeing them in a way not possible in ordinary human experience (eg Chasing Ice).

In my view, that could synergize with moving off into more effective color spaces -- the data may just be shades of gray (rods) but we don't have to look at it that way (cones). There's more than meets the eye to the retina -- amacrine and bipolar cells post-process the data before it even hits an integrative ganglion, providing all sorts of no-brainer hard-wired features such as pattern recognition and motion sensing. The coding dna for that has been under development since we diverged from jellyfish in the early Cambrian.

It looks to me like project-specific automated design of color scheme, while not the ultimate optimum, could probably pick the low hanging fruit. These would be based first on the global histogram (overall usage of the various grays, means, std deviations, variances, distribution etc), secondly on how these grays are statistically distributed in the plane, and thirdly on how they are changing in a time series stack. Of course, jpeg and mpeg have been there, done that in terms of designing compressioon, the eg standing for experts group.

At a practical level it suffices to fit a color lookup table to the graphic. Two examples are shown below. These show a linear grayscale of 256 values and what each one of them is going to be replaced with by going over to indexed color.

For unknown reasons, the two below are peculiarly effective in presenting (post-enhancement) Ascat ice motion imagery. Here it has to do with the bimodal peak in the sea ice roughness histogram, FYI vs MYI, and persistence for months or even years as cohesive units aka recognizable features, despite the viscoelastic properties of Arctic sea ice. Or rather, it's all about exploiting the latter.

For now, I am just looking for a digital assistant that will help me screen thousands of these until I see either an end point in improvement or get some idea of how to design a good one from scratch, for any project, given its data distribution, from an algorithm. Ideally the data hosters themselves would provide several as part of their archive.
Title: Re: Creating Animated GIFs
Post by: Dryland on February 11, 2018, 04:02:49 AM
Thank you Neven and A-Team!

A-Team, for automating the color scheme, maybe it would be useful to show the image, allow the user to select a sub-region of special interest (and/or difficulty in seeing what's going on in that particular spot) and tell the system to optimize for that region. You would want a weight parameter to tell it how much to favor that area at the expense of the rest of the image. Of course, first we would have to get the hands-off automated version working, which is a good challenge in itself. It seems similar to making a good topo map - the contours become less useful in areas where they're either too close together, or too far apart.

Do any of the free tools you use offer histogram and statistics extraction from image files? That needs to go on my to-do list.
Title: Re: Creating Animated GIFs
Post by: gerontocrat on February 11, 2018, 09:53:00 AM
Meanwhile I am still delving into the human perceptual side of optimal presentation of scientific information, the idea being that numerical time series are basically really boring .....

The use of graphics versus tables has a long history. Florence Nightingale went as a nurse to the war between Russia and Great Britain in the Crimea in the mid-1850's. Apart from basically inventing the modern profession of nursing, she was appalled by the mortality caused by basic poor hygiene and sanitary conditions in the hospitals and the army camps.

After the disasters of the Crimean war, Florence Nightingale returned to become a passionate campaigner for improvements in the health of the British army.

She developed the visual presentation of information, including the pie chart, first developed by William Playfair in 1801. Nightingale also used statistical graphics in reports to Parliament, realising this was the most effective way of bringing data to life.

I guess that makes you and dryland part of a long and honourable tradition.
Title: Re: Creating Animated GIFs
Post by: A-Team on February 11, 2018, 12:37:19 PM
1801 origin of pie chart.
Interesting, so disruptive technology goes back a ways. I was so excited to get a 480 pixel B/W monitor in 1997. Color meant mailing cmyk plates off to Korea for printing. Today, senior PIs reading printed pdfs en route to meetings, each frame of animation (if any) as text. So it only penetrates to a limited extent.

for automating the color scheme, maybe it would be useful to show the image, allow the user to select a sub-region of special interest and weight the system optimization for that region.
That get mixed in with crop, rescale and masking. For example Ascat can maximally take a 2.5x enlargement. Since the whole AO + some Bering + some Barents takes 380 pixels width, that is too wide for the forum not to mention exploding file size for say a year of mp4. So normally the user would specify corners for a roi crop.

Now the roi itself might only occupy half the rectangular crop, for example Chukchi with extraneous statistics from surrounding land. Most of the tools allow restriction per instructions from a companion image (usually a binary mask). That mask need not be all-or-none though, it can be any grayscale weighting (8-bit) in gimp.

So for Ascat, suppose the user might want to optimize for darker areas (smoother FYI) at the expense of brighter (roughed-up MYI). The mask there would come from a major gaussian blur of an image copy, maybe posterization of that to say 4-bit, attached.

More simply, just blow up contrast remapping at the low end. Below, after thresholding (which could be done uniformly across a whole series or frame-specifically). A lot of extra detail is revealed. However there is only so much water that can be squeezed from a turnip.

One thing I've noticed though in reading academic papers in image enhancement: it is all rigorously optimal [citation] from an information-theoretic standpoint [citation] but the final images are worse than some duffer gets on a flip-phone.

first we would have to get the hands-off automated version working ...similar to topo map - contours less useful where they're either tooclosetogether or too   far    apart.

Right. Same with city names. There is software now for that, dealing with zoom.

Do any of the free tools you use offer histogram and statistics extraction from image files?

ImageJ has graphics-specific offerings in that dept, nothing that has thrilled me. A lot of people would go with R statistics software and process raw numeric, say from ncdump out of Panoply.
Title: Re: Creating Animated GIFs
Post by: Dryland on February 11, 2018, 05:53:27 PM
gerontocrat, thanks for the Florence Nightingale story - I had no idea.

In both substance and style, her chart reminded me of the famous Charles Joseph Minard graphic of just a few years later (1869) showing the decimation of Napoleon's army. The wikipedia article ( ( says perhaps a bit hyperbolically, "Modern information scientists say the illustration may be the best statistical graphic ever drawn".

When I was a freshman in engineering school, they waved this under our noses as an example of the power of graphical presentation. I wonder if they still do that.
Title: Re: Creating Animated GIFs
Post by: uniquorn on February 21, 2018, 08:19:36 PM
Floe looks like it is going to be an excellent tool. Thank you for the hard work so far.
Title: Re: Creating Animated GIFs
Post by: A-Team on February 24, 2018, 10:57:13 AM
While waiting for the ESRL archive to build up again, not being convinced products like sea ice thickness are backwards compatible with 2016-17 values, I have been making some cross-silo products.

That is, most of our data source sites specialize in a fairly narrow daily product like bulk ice salinity and stop with that whereas the whole point of netCDF and Panoply is a seamless data integration framework. So this is a good niche for us, where value can be added and floe's automation is needed worse than ever.

The UH SMOS is a good one for cross-silo demo-ing. It has all the ancillary files such as error grid and land mask, some of the products are supplemental to ESRL, and everything is compliant. It is really a nuisance to spew out longer animations manually so floe/panoplyCL will be a big deal just with sites like this.

There is some really excellent coding going on here at They've been very decent about letting me do a lot of online converting of gifs into forum movies. This is mission-critical for us because file size limits what we can do with gif animations from floe. That is, gif is better in the scientific sense with its individual frames but its compression scheme is lousy.

At some point we need to look at remedying site sources that are producing non-compliant or defective netCDFs. Most commonly this involves lat lon associated with but not integrated into gridded data so a 2D but not a Geo2D file shows up in Panoply. This would either harranguing the site hosts or involve drilling into ncgen and re-hosting the archive properly formatted.

I've also come to realize that a lot of plain satellite imagery falls into the defective netCDF category. In effect what they've done is discard the intermediate gridded data and offered a Panoply map in a fixed Arctic projection. The PanoplyCL scaling parameters aren't provided but usually the map is polar stereographic at some multiple of -45º off the Greenwich meridian with a guessable horizon and pole.

For example Ascat is a one-channel 8-bit grayscale in Greenland-down orientation looking down at the north pole with 45ºN as the horizon. It comes without its land or open water masks but those are readily made with UH AMSR2 which though truncated rectangularly attains the same horizon and otherwise matches except for a scale factor.

The Ascat image amounts to a 350 x 300 rectangular pixel array for the Arctic Ocean. It's easy to overlay lat lon polar coordinates and so in effect assign a lat lon to each pixel. So the "inverse netCDF problem" is resolved by stubbing in these Ascat [0,255] numbers for the data values of any master Geo2D file adjusted to this scale. Once the image is in Panoply, it can be reprojected like any other netCDF and integrated with them arithmetically or as partial overlays.

While ESRL provides little or no explanation of what they are doing or why, the post-hiatus archive has gone from REB_plots to REB2_plots for their front-facing gifs, still without providing the underlying netCDF files needed to draw them directly in Panoply.

They did not fix any of the old bugs that I could see, such as different image sizes in the 5-day before and after thicknesses, or crazy complexity in snow contours over ice thickness, or lack of ± explanation of keys.

In their new file names, UAF stands for University of Alaska Fairbanks. They have some very strong comprehensive products for the Bering, Beaufort, Chukchi and ESAS but tend to truncate them arbitrarily at the Canadian border. So it's kinda narrow-minded, omitting say the Barents, but fortunately the region they do cover is critically important. I don't know what the 4 means in 4UAF_ATM and 'atmosphere' for ATM doesn't really describe the air-ice-sea contents of the nc files.

NIC is the military's National Ice Center whose mission is to provide "global to tactical scale ice and snow products, ice forecasting, and environmental intelligence services for the US government". It involves the navy, coast guard and NOAA. Their analysts manually curate sea ice edge products but don't use the same categories as the Canadian Ice Service. I'm not of the opinion that expert annotation is competitive with machine-learning classification over the long haul.

It looks to me like ESRL ditched their previous sea ice thickness basis and reset to the January 2018 CryoSat averaged observational thickness. ESRL will then thicken and compact it from there using their physics model. Ice and snow thickness are by far the most difficult products. Everyone gets a reset to zero at the annual fall maximum of open water that takes in all the peripheral seas and a good bit of the Arctic Ocean but errors still build and build in multi-year ice.   272 MB   2/13/18, 8:05:00 PM   177 MB   2/16/18, 8:29:00 PM
RASM-ESRL_4NIC_2018-02-16.tar.gz   41.5 MB   2/16/18, 8:22:00 PM
Title: Re: Creating Animated GIFs
Post by: uniquorn on February 26, 2018, 10:57:34 AM
This amounts to land-masked framen of (R,G.B) = (dayn - dayn-1, dayn - dayn-2, dayn - dayn-3) - (50,50,50)) after contrast renormalization, adaptive histogram equalization and bicubic rescaling
Sorry that didn't paste too well.  'Ascat interferometry' looks really useful.
Is there any chance of a slightly more step by step description of this process? I'd like to try to compare this year to previous years.
Title: Re: Creating Animated GIFs
Post by: A-Team on February 26, 2018, 12:47:07 PM
slightly more step by step description of this process? I'd like to try to compare this year to previous years.
That'd be a good project. We're look to automate this either in the ImageJ scripting or macro language so daily updates becomes feasible.

This takes more steps than I had remembered. First, a nuisance download as their ftp is broken. After that, mostly ImageJ commands. But try it first with a minimal set of 4 Ascats, eg Mon-Wed, Mon-Fri, Mon-Sun = R,G.B before getting into 109 pixel files.

drag 150 winter days into ImageJ after cleaning file names

convert to grayscale to reduce file size (Image --> Type --> 8-bit)

stack them into a single file (Image --> Stack --> Images to Stack)

crop down to Arctic Ocean region of interest (Image --> Crop)

tile them into a row (Image --> Stack --> Make Montage... label opp with file names)

adjust bulk contrast (Image --> Adjust... --> Brightness/Contrast)

adjust local contrast (Process --> Enhance Local Contrast (CLAHE) set to 63,256,2.20)

duplicate three times, cutting off newest dates (Image --> Duplicate...)

add blank frames at end to make the rows the same length (Image --> Stack --> Add Frame)

combine the three rows into a stack (Image --> Stack --> Tools --> Combine vertically x2 )

so far, 3 rows, 150 columns with each column having 3 grayscale frames that become an rgb frame

subtract the current date row from each of the three date-staggered rows and and add 128 neutral gray to avoid negative color numbers  [Gimp: set layer mode to grain extract, subtract layers pairwise, capture, Colors --> Components --> RGB]

maybe adjust hue and subtract a constant to get rid of some of the neutral gray (Process --> Math --> Subtract ... 50)

slice the rgb back into a stack (Image --> Stack --> Tools --> Montage to Stack ...)

save it out as movie (Plugins --> Input/Ouput --> Save as Movie: 10 fps avi raw excellent)

open .avi in Quicktime, close, convert to .mov to reduce file size

open and covert .mov (or original avi)to mp4

load mp4 to forum

(The land mask is made separately from AMSR2 by selection all 100 of the concentration colors, deleting to transparency, inverting to everything else and filling with black. The land mask is duplicated 150 times and tiled into a layer to go over the day difference layers. I attached an Ascat-ready mask below, just crop it down in parallel to your original cropping of the Ascat stack.)

Here is one with two-day offsets. It has more interferometric color because there is more motion over two days than the one-day offsets used on the freeze forum, plus I drew it out with hue tweaking.

Gimp also does a good bump map to gussy up plain vanilla Ascat motion (no differencing)
Title: Re: Creating Animated GIFs
Post by: uniquorn on February 26, 2018, 04:08:56 PM
Thank you A-team. Will give it a go.
Title: Re: Creating Animated GIFs
Post by: uniquorn on March 10, 2018, 07:56:54 PM
testing, forgot about the mask and struggled with ffmpeg conversion from avi to mov so went with gif for now
Title: Re: Creating Animated GIFs
Post by: uniquorn on March 15, 2018, 04:58:53 PM
2010-2011, 3day difference. Might try it with the daily amsr2 mask.
Posted a scaled down gif as mov doesn't play
Title: Re: Creating Animated GIFs
Post by: uniquorn on April 29, 2018, 02:47:05 PM
I made quite a few errors with the attempts above, mostly things like not checking that the whole stack was modified and layers not being aligned properly in gimp.
This one is for the current melting season. Might be a bit extreme with the brightness/contrast.
I need to find an easy way to add the mask and had to miss out recent frames with missing data.
Title: Re: Creating Animated GIFs
Post by: uniquorn on June 29, 2018, 12:19:16 AM
testing licecap with windy
quick and easy with imagej, but sprites too quick. Have to reset delay and let the last capture run longer.
Title: Re: Creating Animated GIFs
Post by: A-Team on June 29, 2018, 01:32:11 AM
Nullschool wind and mslp effort below. I made 20 separate 1 sec gifs of 10 frames each, advancing date 3 hrs in between, then loaded into ImageJ and concatenated. It is necessary to load the gifs in order or the sprites will run backwards! ImageJ has a nice duplication-of-selection feature that allowed quick appending of date and data in 24 pixels at the bottom of main content. Data would be better on top because the stupid controller tends to cover it up.

Then sliced into frames and saved as mp4. I tried the 2 sec pause-and-advance method earlier but couldn't keep the number of frames constant. The animation struggles with nullschool complex colors, gif format is limited to 256. Also it jerks too much between dates. Still, a bit of animation catches the sprite action. windy would actually be better as it has both gfs and ecmwf.
Title: Re: Creating Animated GIFs
Post by: be cause on July 22, 2018, 12:38:46 AM
Help .. is there any way to stop large animations loading when I visit the main pages ? For example the current melting season page has 2 animations over 10000kb that start loading every time the page is visited .. I cannot afford this .. they take time to load and they cost me money whither I want to view them or not . Most animations used to be click to view now most just run and run .. until my data allowance runs out .. Help !  b.c.
Title: Re: Creating Animated GIFs
Post by: Neven on July 22, 2018, 11:47:01 AM
I'm sorry to hear this, be cause. I'll see if I can change something in the admin section.

I always try to keep images as small as possible, simply to save some energy, but with all the bandwidth speeds nowadays (glass fibre and so on), it hardly seems to matter any longer.
Title: Re: Creating Animated GIFs
Post by: johnm33 on July 23, 2018, 04:33:49 PM
testing   (
[added] now to figure out how to make it last
Title: Re: Creating Animated GIFs
Post by: uniquorn on August 02, 2018, 01:12:55 PM
testing 701px gif
Title: Re: Creating Animated GIFs
Post by: A-Team on August 03, 2018, 12:22:20 AM
just testing to see how forum handles gif transparency (fills in with zero black if >700,otherwise the ambient of the two bluish gray backgrounds) ... then testing whether these mp4 can be forced to run at 700 rather than 720 (yes, though it struggles to load today). The mp4 shows the advancing front north of Svalbard, 62 days to August 1st.
Title: Re: Creating Animated GIFs
Post by: A-Team on August 03, 2018, 12:46:38 AM
Given that we have 1442 members, it would be better if the 2-3 people on phone pay-per-MB plans would re-set their profiles so fewer posts are shown per page, say 5 instead of 50 (forum software says 'messages' when it means 'posts').  That way older posts don't have to re-load. This would be better than going to the lowest common denominator of the 1442 internet accesses which is probably someone in a remote location still on a rotary phone dial-up connection.

It is a bad idea to force everyone else's animations not to run by going to 701 pixels etc. Very few people will click on through, as shown by the counter. Often there is not indication in the first static thumbnail frame that the animation would be all that interesting.

The real problem is that very few people understand the concept of cropping their images down to the relevant areas. Like that person posted that 13 MB file the other day of which 1.3 MB was needed to show the Arctic. Also, many people are not resizing images down to forum max of 700. Again, they need to look at the menus -- all graphics software including cell phone offers crop and resize ... or you can do it free and fast online.
Title: Re: Creating Animated GIFs
Post by: be cause on August 03, 2018, 01:52:49 AM
thanks A-team .. shall try 25 for now .. good soloution .. b.c.
Title: Re: Creating Animated GIFs
Post by: Neven on August 03, 2018, 11:45:31 PM
Thanks from me too, A-Team. I didn't know how to solve this without impacting the majority's experience.
Title: Re: Creating Animated GIFs
Post by: uniquorn on September 14, 2018, 12:09:26 PM
small ascat test
Title: Re: Creating Animated GIFs
Post by: uniquorn on September 15, 2018, 09:14:26 PM
Testing interferometry on amsr2-uhh
better with clahe, probably should have done that first. yes, the grey is uniform with clahe first.
Title: Re: Creating Animated GIFs
Post by: uniquorn on October 08, 2018, 03:45:53 PM
bilinear scaling test
Title: Re: Creating Animated GIFs
Post by: uniquorn on November 12, 2018, 10:46:10 PM
ascat interferometry, 1 day difference, with mask, 2018, day293-315
match width with land mask
contrast/brightness, clahe, unsharp mask before montage.
Title: Re: Creating Animated GIFs
Post by: uniquorn on November 22, 2018, 09:49:37 PM
amsr2-uhh, with overlaid ice edges of 2016 in red and 2017 in yellow (shows up well against blue but not too good over white)
edit:chose a neutral grey for 100% ice color - too much at once though

not too bad for a close up view

using ImageJ, Process>find edges. Image>color>split channels. Keep blue channel. Adjust color balance to choose color (must be a better way) make montage and save.
using gimp, layer>transparency>set color to alpha, merge down, repeat and save
back to imagej. image>stack>tools>montage to stack. crop, scale, repeat final frames.

Title: Re: Creating Animated GIFs
Post by: uniquorn on May 01, 2019, 12:06:56 PM
Hopefully, here are all the steps to creating an ascat mp4.
1. Download the images.
2. Install Fiji (ImageJ with more plugins)
3. Drag images into Fiji after cleaning file names
4. convert to grayscale to reduce file size (Image --> Type --> 8-bit)
5. stack them into a single file (Image --> Stack --> Images to Stack)
Set animation speed (Image --> Stack --> Animation --> Options)
6. crop down to Arctic Ocean region of interest (Image --> Crop)
7. label images (Image --> Stack --> Label) preview first.
8. adjust bulk contrast (in winter 41,255 is good) apply to all images
9. tile them into a row (Image --> Stack --> Make Montage)
10. adjust local contrast (Process --> Enhance Local Contrast (CLAHE) set to 63,256,2.20)from A-team. Suggest anything up to 127,256,8.3.
11. Convert row back to stack (Image --> Stack --> Tools --> Montage to Stack)
12. Add duplicate end frames (Image --> Stack --> Tools -->Stack Sorter -->Duplicate)
Enjoy animation.
Converting to mp4. Frame rates under 16 don't convert well for me. Sleepy may have better suggestion than this
Download and install ffmpeg
1. Duplicate stack (Image --> Duplicate)
2. Interleave to double frames (Image --> Stack --> Tools --> Interleave)
3. Set animation speed to 16
4. Save as gif
5. Convert to mp4 (ffmpeg -i ascat1.gif -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -level 3 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" ascat1.mp4
Enjoy video

A-Team describes adding a mask and a mask image upthread.
Title: Re: Creating Animated GIFs
Post by: Tor Bejnar on May 01, 2019, 06:37:07 PM
Someone posted recently (elsewhere), which I've been using. 

I have Snagit on my computer, and have made SnaiIt videos by successively clicking on several windows with aligned images (such as DMI images of Nares Strait on different days).  I then grab PNG 'stills' from the video (using Snagit editor), add text or shapes, save them, then download into Gifmaker.  I've always changed the number of colors from 256 to 25 or 50 to reduce file size.
Title: Re: Creating Animated GIFs
Post by: uniquorn on May 17, 2019, 11:19:42 PM
Testing amsr2uhh heavy contrast over mercator salinity 34m
Title: Re: Creating Animated GIFs
Post by: johnm33 on May 18, 2019, 11:00:51 AM
That's an interesting combination, shows up the power of the surge in from the Pacific really well. Great stuff.
Title: Re: Creating Animated GIFs
Post by: uniquorn on July 05, 2019, 12:36:49 PM
must run that combination again sometime.

Rammb rgb compose from 3gifs step by step:
download Fiji (imagej) from, install and run.
drag the file of the first gif into the small interface to open it.
make a montage from image/stacks/make montage (enter frame no. and columns=1,scale=1)
rename to m1 from image/rename
repeat with the other two gifs renaming as m2 and m3
make the three montages into a stack from image/stack/images to stack
set stack to 8bit from image/type/8bit
compose as rgb from image/color/stack to rgb
That leaves us with an rgb montage which we turn back into a stack using
image/stack/tools/montage to stack (enter the same frame no. and columns=1)
set frame rate from image/stacks/animation options
file/save as/gif

optionally add repeat final frames from
image/stacks/tools/stack sorter
process/enhance local contrast

Recommend saving often as it's possible to lose everything with one careless click :)
Title: Re: Creating Animated GIFs
Post by: uniquorn on July 05, 2019, 02:54:10 PM
here focusing on open sea area from,2649.msg210938.html#msg210938 though I'm not sure that much can be discerned from such high contrast settings. Difficult to distinguish between what may be sst and thin cloud.
Title: Re: Creating Animated GIFs
Post by: JayW on July 06, 2019, 04:06:53 AM

The how to is posted here,1259.msg211015.html#msg211015
Let's continue there so as not to disrupt this thread. Looking forward to the next test.

Here are three short gif around Peterman glacier if you want to give it another go, I figure the landcover there is quite varied, so perhaps something will be interesting.  I will work on learning imagej as time allows.  I tried to keep the files smaller, don't know how to package them in a more efficient manner.

Title: Re: Creating Animated GIFs
Post by: b_lumenkraft on July 06, 2019, 08:46:14 AM
This was an interesting spot yesterday. I posted about it in the Petermann thread:

A rather big melt event at Petermann today.

Tomorrows Sentinel shots should show a bluening.

This is the M8 band via RAMMB-SLIDER. Dark grey indicates unstructured snow (aka water), and brightness indicates intact snow structure (aka freshly fallen snow).

Correction: On second thought, this might be caused by rain!

I'm pretty sure it rained there because we obviously had precipitation. On the ice sheet, it fell as snow, but closer to Nares, the temperatures are pretty high atm.
Title: Re: Creating Animated GIFs
Post by: JayW on July 06, 2019, 12:27:49 PM
This was an interesting spot yesterday. I posted about it in the Petermann thread:

I'm pretty sure it rained there because we obviously had precipitation. On the ice sheet, it fell as snow, but closer to Nares, the temperatures are pretty high atm.

Your post is what made me choose that area.  :)
Title: Re: Creating Animated GIFs
Post by: b_lumenkraft on July 06, 2019, 01:03:22 PM
Hah! :)
Title: Re: Creating Animated GIFs
Post by: uniquorn on July 06, 2019, 02:11:28 PM
Imagej default rgb and cropped high contrast (clahe 127,4.1)
Is that a plane flying over? No, probably swath edges.
Title: Re: Creating Animated GIFs
Post by: JayW on July 06, 2019, 03:29:38 PM
Imagej default rgb and cropped high contrast (clahe 127,4.1)
Is that a plane flying over? No, probably swath edges.

I think that looks nice as a snow/land cover. 
I got tripped up a bit somewhere, I got them into an RGB stack, but it wasn't a gif. Operator error I'm sure.  I'll be practicing with imagej as time allows, I have two small kids running around so it gets tough.   I think I'll stick with screenshots over downloaded images, I am finding the screenshots seem preserve more detail.  I'm gong to revisit the SSTs with a different band, focused on the Chukchi. 

Thanks again, I kinda dig the high contrast. 
Title: Re: Creating Animated GIFs
Post by: uniquorn on July 06, 2019, 03:44:13 PM
Let me know if you keep getting stuck in the same place. It's easy to miss out a step when writing instructions. Adjusting contrast before rgb compose may have a more striking effect, but only you know what you're aiming for :)
Title: Re: Creating Animated GIFs
Post by: petm on July 22, 2019, 01:47:22 AM
Following up on:,2591.msg214848.html#msg214848 . Not sure if this is the best thread for this, since I actually just use EZGif's website to make the GIFs once I've created the images, but it can always be moved if appropriate...

Repeating the summary from the thread above:
I do it using the imageJ API (Java code), basically by: for each day, reading in the 5 preceding days' images, converting each pixel color to a concentration value based on the Bremen key, taking the median for each pixel, then creating a new image. (Actually, the algorithm is easily changed -- 5 day median is quite conservative.)

The code is a bit of a hack, so I'll just post some snippets:

0. Make a key to convert colors to concentrations. The key was made by using the ImageJ GUI to manually pick indices corresponding to each color in the key of one particular image. (ImageJ LUT objects store pixel values as a 1d array; indices refer to that array.)
int[] concBuckets = {10, 20, 30, 40, 50, 60, 70, 80, 85, 90, 95, 99,100};
int[] indices = {14, 84, 92,104,110,105, 80,100,124,170, 79,177,174}; 
for (int i=0; i<concBuckets.length; i++) {
      this.keyImage.getProcessor().getLut().getRGB(indices), concBuckets);

For each day do all of the following steps:
for (int n=0; n < this.nDays; n++) { ... }

1. Download the needed number of preceding images (e.g. five, if making a trailing 5-day median) from their URLs or get from cache if already downloaded. URLs are similar to (substitute the dates):
ImagePlus image = null;
String filename = this.getFilename(date, bremenVersion);
File file = new File(this.cacheDir + filename);
if (file.exists()) {
            System.out.println("Loading image from file: " + this.cacheDir + filename);
            image = new ImagePlus(this.cacheDir + filename);
} else {
            String url = this.getUrl(date, filename);
            System.out.println("Downloading image from web: " + url);
            image = new ImagePlus(url);

2. For each preceding image, convert the colors to concentrations using the key.
int[] conc = new int[this.len];
LUT lut = ip.getLut();
byte[] pixels = (byte[]) ip.getPixels();
for (int y=0; y < ip.getHeight(); y++) {
            for (int x=0; x < ip.getWidth(); x++) {
                int i = x + y * ip.getWidth();
                int rgb = lut.getRGB(pixels);
                if (this.rgbToConc.containsKey(rgb)){
                    conc = Integer.parseInt(this.rgbToConc.get(rgb).toString());
                } else {
                    conc = Integer.MAX_VALUE;

3. Use this set of concentrations to calculate the output values; e.g. for median (i.e. quartile = 2):
for (int i=0; i<this.len; i++) {
            int filteredC = this.concentrations.get(0);
            int[] values = new int[this.concentrations.size()];
            for (int z=0; z<values.length; ++z) {
                values[z] = this.concentrations.get(z);
            if (values.length % 2 == 0) {
                throw new RuntimeException("Can't use quartile for even number of ice concentrations.");
            } else {
                int q1 = values.length/4;   // for values.length == 5, q1 == 1 (floor of 1.25)
                filteredC = values[q1*quartile];
            this.filteredConcs = filteredC;

4. Finally, convert the concentrations back into RGB color values and use that as the pixel array to generate a new image.

5. Now I have a set of images, each being the (e.g.) 5-day trailing median. I just use EZGif online to convert them to an animation. To use a different algorithm, just change step 3.

Title: Re: Creating Animated GIFs
Post by: uniquorn on July 22, 2019, 10:41:18 PM
thanks petm, useful stuff
Title: Re: Creating Animated GIFs
Post by: Sterks on July 25, 2019, 09:46:01 PM
Following up on:,2591.msg214848.html#msg214848 . Not sure if this is the best thread for this, since I actually just use EZGif's website to make the GIFs once I've created the images, but it can always be moved if appropriate...

Repeating the summary from the thread above:
I do it using the imageJ API (Java code), basically by: for each day, reading in the 5 preceding days' images, converting each pixel color to a concentration value based on the Bremen key, taking the median for each pixel, then creating a new image. (Actually, the algorithm is easily changed -- 5 day median is quite conservative.)

The code is a bit of a hack, so I'll just post some snippets:

I don’t think I have the Java skills to achieve this in a reasonable time. Thank you anyway and hope you post more filtered gifs when you find the time. I really enjoy them, I don’t think five day average is conservative, actually it leaves the shortest characteristic time that makes sense. Weather patterns change with that pace usually, not faster.
Title: Re: Creating Animated GIFs
Post by: petm on August 08, 2019, 01:50:40 AM
Oof, it turns out I made an error in calculating the index for getting median values from a sorted array. The calculation was only correct for 5-day medians (zero-based index 2). For 3-day medians it was returning the minimum (index 0; should be index 1).

However, this may have been a fortunate error, as it turns out that I actually like the minimum algorithm. It's not conservative, but if it's true that the biggest artifacts are clouds, then this does the best job of eliminating them. On the other hand, maybe it overestimates the amount of melt too much?

Attached is a gif that cycles through each index for today's result. Index 0 is minimum, 2 is median, 4 is maximum, and 1 and 3 are intermediate (something like Q1 and Q3).

Click to animate.

Any comments?
Title: Re: Creating Animated GIFs
Post by: uniquorn on August 08, 2019, 02:03:59 AM
Any comments?
Run all 5 over the last 5years and anecdotally compare them to available clear worldview days. Then satisfy yourself which is best. I'll be very grateful :)
Title: Re: Creating Animated GIFs
Post by: petm on August 08, 2019, 02:05:09 AM
Any comments?
Run all 5 over the last 5years and anecdotally compare them to available clear worldview days. Then satisfy yourself which is best. I'll be very grateful :)

Good idea. I'd be interested to see as well. It'll have to wait for the weekend though...
Title: Re: Creating Animated GIFs
Post by: uniquorn on August 08, 2019, 02:10:28 AM
well, maybe not 5yrs
Title: Re: Creating Animated GIFs
Post by: petm on August 08, 2019, 02:13:30 AM
lol  :D

I'll find some good examples and see if there's a clear trend.
Title: Re: Creating Animated GIFs
Post by: El Cid on August 08, 2019, 08:06:50 AM
If you find out which one is best, please tell us about it and you could also post it on the main thread as it is important and educative
Title: Re: Creating Animated GIFs
Post by: uniquorn on August 08, 2019, 12:08:04 PM
Thinking about it overnight, a median has its uses but I prefer the detail of the daily. Weather artifacts are easy to see in an animation and add further information.
Title: Re: Creating Animated GIFs
Post by: uniquorn on September 05, 2019, 11:51:26 PM
noaa file download for windows, try winhttrack and a text file list

mac and linux should probably use wget