Please support this Forum and Neven's Blog

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - A-Team

Pages: [1] 2 3 ... 37
Very little change is foreseen by ESRL in ice thickness averaged over latitude over the next ten days. This latitudinal averaging capability of Panoply is probably of most interest to variable like temperature, vis-a-vis less nuanced products that lump everything over 80ºN into one bin.

Consequences / Re: Hurricane season 2017
« on: Today at 12:19:17 AM »
These wind field maps are an interesting use of color for magnitude and meteo widgets instead of vector arrows. Note the route of the plane in lower left as it transects can be inferred though not the times. The satellite IR overlay mentioned in bottom left seems to be missing. There must be a need for regular observational wind speeds input into tracking models that isn't being met by doppler radar or sonar.

Here is the original animation of Piomas data 31 Aug to 15 Sept below the ESRL ice thickness out to the 18th in the Piomas palette followed by ESRL ice thickness in a more effective continuous ColorBrewer2 palette, reversed CB_RdYlBl.cpt. See explanation over in the Dev Corner animation section for scripting these out of REB .nc files at the ESRL archive.

The ice thickness forecast is shown from the 18th out to the 28th in 6 hour increments in the CB palette.

Developers Corner / Re: Creating Animated GIFs
« on: September 19, 2017, 06:22:17 PM »
Gimp (or ImageMagick) will play a key role in pipelining ESRL netCDF products onto the forums. While the ESRL graphics are excellent -- as are the Panoply maps produced from ESRL netCDF files -- they are both too large to display properly on the forum.

That's mainly because the Arctic Ocean is quite lopsided with respect to the North Pole, eg the Bering Strait is much further south than Svalbard, yet forum maps need to be centered on the Pole for comparative purposes.

Note further ESRL plots only a fraction of their variables and those in certain combinations. Other variables no doubt play a role in their model calculations but aren't used for standalone graphics. For these, we can make Panoply maps if there is sufficient interest in the variable, eg rainfall, snow depth or compressive strength of the ice pack.

There's an easy way to in Gimp assemble a master graphic for all the Geo2D files in REBplots, REB_nc, and RASM-ESRL archive bundles. (These make sense to mix together only when the animations would be the same length, eg 20 or 40 frames.) From this master graphic, any combination of variables can be exported out as a gif animation.

The issues here are (1) each variable has its own legend, title and palette (2) some variables have interesting max and min variation by frame (which Panoply provides), (3) the same time stamp is shared by all the variables, (4) vectors from a first variable can float over the map of another variable, and (5) some variables apply only to the ice (thickness), others just to open water, and still others to both. This latter fact (and the ambiguous use of zero) require development of an evolving 6-hour open water mask and mask edge.

The process of making a master graphic is fairly simple. First, Panoply scripts generate maps at the same favorable scale for all the variables. Each variable has a different palette appropriate to what it is supposed to convey. Other script settings may also differ slightly, for example log vs scalar. These output pngs are cropped identically using fixed UL and LR corner coordinates. Next the frames are tiled up into one wide image.

Other maps from other variables are then tiled up too and put into the growing master receiving image (for the given D0 dte) as layers. The same is done for the time scale, the palettes, the scales, the legends and the varying max and min data. Open water mask, boundary outlines, and alpha transparencies are then constructed as yet more layers.

The master image is then ready to have various layer combinations constructed by activation and layer ordering. A new-from-visible layer for each desired output is then sliced back down into forum sized frames. These pngs are then saved to animated gifs at whatever frame rate is effective for them and uploaded to a forum post. While some manual intervention and operator aesthetics may be necessary, automating daily ESRL output up to this master file is quite feasible.

Some examples are cross-posted below:

Consequences / Re: Hurricane season 2017
« on: September 19, 2017, 02:56:52 PM »
In all fairness, the NHC kept discussed the possibility of rapid intensification for Maria and said it would not be surprising at all.
Agree. My sense also is that near-real time hurricane prediction products have been doing rather well. If they've already picked the low-hanging fruit and some a bit higher, it's questionable though if much improvement will result from throwing tons more money at models or computer time.

For example, it seems that the lowest central pressure is critical but that cannot be obtained by remote sensing and requires periodic manual dropsondes from airplanes. Harvey went straight at the tip of Florida, the junction of the mainland with the Key West road, so predicting east or west coast was as difficult as saying which way a pencil balanced on its tip is going to fall.

With Maria, all the talk yesterday pm about landfall in Puerto Rico yet today it is projected to glance off, going to the north with the highest winds in the NE corner well off the island. That's a subtle but critical flaw in track prediction (assuming the GFS or ECMWF happens) because Maria would have very much have weakened over land.

Developers Corner / Re: Creating Animated GIFs
« on: September 19, 2017, 01:57:48 PM »
To facilitate visual ice thickness comparisons of ESRL to Piomas, it's good to map up ESRL's view in the color palette that wipneus has been using on the thickness forum. There are 16 colors there whose rgb values are readily captured in Gimp from a large-format posted graphic and imported into Panoply.

It's a good idea to number imported palettes, say 11wipiomas and 10AMSR2UH so that they appear in order conveniently at the top of the stack in Panoply alphabetization. To import, save with the .rgb tag from a text editor and Open that file in Panoply.They are easy to delete just by a Cut command.

The animation compares mid-August Piomas as prepared by wipneus, mid-Sept ESRL ice thickness as prepared in Panoply with the same palette, and mid-Sept AMSR2 sea ice concentration from the UH archive..

ncolors= 16
#  r   g   b
0   166   0
25   175   0
50   184   0
83   193   0
116   202   0
151   211   0
189   220   0
230   230   0
231   206   29
233   189   58
234   179   88
236   177   118
237   182   148
239   194   179
241   214   211
242   242   242

Developers Corner / Re: Creating Animated GIFs
« on: September 19, 2017, 01:15:12 PM »
ran your script and got the same image back.

That is just awesome!

It was fortunate that I happened to use one of the default Panoply palettes here, CB_Blues.cpt, before exporting the script. If the palette had been set to my favorite for ice area aice_h, namely the custom palette AMSR2.rgb (whose file and easy installation is described above), the script would have stalled out unless you too had put it in your palette collection.

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-colorbar", "AMSR2.rgb" );

While Panoply offers a very nice collection of stock palettes, the reason for AMSR2.rgb is to let ESRL seamlessly pick up from UH AMSR2 high resolution observational data in the same color scheme. That is, the UH data only goes up to the day before whereas ESRL forecasts ten days out from there.

Alternatively, if the UH .nc can be reworked from 2D to Geo2D allowing polar stereographic projection, its palette could more simply be replaced by a similar stock palette in Panoply. The UH archival images fortunately are in standard 'Greenland down' orientation and just need a 39% scale reduction in Gimp to match the scale on the King plot.

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 ); //this sets lowest latitudinal circle displayed.

myplot.set ( "size-factor", 240 ); King size plot setting

I'm using the King 240 scale on Panoply so that it will output .png images with the Arctic Ocean fitting in ~700x700 pixels, so after cropping in Gimp it makes maximal use of what the forum software will display as an animation. I don't want to rescale later in Gimp because interpolation messes with the colors; right now just palette colors are used in the map pixels.

Smaller thumbnail animations lose detail; larger animations will run after a click to a new window but counts show few forum visitors take that extra step. File size is definitely an issue with the 40 time slices in the ESRL data.

Note the image the script makes is only the 1st frame out of 40. That is indicated in the 3rd coordinate slot in the script. In exporting an animation, Panoply runs through all 40 of these, in effect bumping the step the internal script it is running. So externally, we need a super script that runs, bumps the step by 1, runs, ... stops at 40 steps (or however many ESRL has provided).

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 5 of 40
myplot.setVarDimension ( 1, 1, 5 );

Two code snippets -- ncdump and ncgen -- are absolutely critical to working with netCDF file format. these interconvert between machine-readable .nc and human-readable and editable CDL plain text. By opening an .nc file and exporting to CDL, Panoply is providing ncdump already without a command line environment. Oddly it cannot convert its own CDL file back to the .nc that it just made it from, which would amount to ncgen.

The link below gives a friendly explanation of how to get ncgen running on a Mac. It's a lot of trouble to get one menu item, involving a massive download and installation of Xcode, MacPorts, X11 and establishing directory paths. At that point, command lines for ncgen can be run on a local CDL file will make it into a netCDF that Panoply can import:

-- Generate a NetCDF file
ncgen -o mydata.cdl

Generate code that will produce a NetCDF file
 – Fortran:
ncgen -f mydata.cdl > mydata.f

 – C:
 ncgen -c mydata.cdl > mydata.c

Arctic sea ice / Re: The 2017 melting season
« on: September 19, 2017, 12:11:00 AM »
Not too much impact is foreseen from the impending low pressure over the next 10 days in the ESRL wind over ice forecasts. The three animations, which seem fairly complex, can actually be updated daily using Dryland's archive parsing, an imported palette from UH AMSR2, Panoply scripting, and Gimp automation (BIMP or python-fu) as described today over at Developer's Corner (or at least that's the goal).

Consequences / Re: Hurricane season 2017
« on: September 18, 2017, 07:30:17 PM »
Just want to echo others to say I really appreciate what you are doing here, Sig. It takes quite a bit of time to chase down the best hurricane links on my own and cycle through them regularly so the forum collection of breaking news is much more convenient. 

J Masters and B Henson are also putting out some fine once-a-day articles on these hurricanes, the most recent being:

Developers Corner / Re: Creating Animated GIFs
« on: September 18, 2017, 06:32:01 PM »
Awright, Dry! Hopefully you'll get some helpful feedback from the other python people here.

A new command line version, PanoplyCL, is available for Mac and Windows at along with a not-so-new manual.

PanoplyCL could be quite important for us in automating the transfer of daily ESRL products into stable links and images that work under constraints of the forum. I have just installed it (after a minor hassle activating Java v8 update144) and saved my first script, which has the extensions .pancl.pcl  It is opened as text below and looks quite straightforward.

Basically it seems a fancy way of setting preferences separately for each map in a collection of Geo2D files so the same parameters can be run to make the next day's map the same way. This amounts to making the frames of an animated gif using the ESRL archive.

Going forward, ESRL + Panoply are probably the best things we have going for comprehensive 2017 freezing season forecasts, though specialized meteorological and oceanographic products may be better at certain narrower tasks.

A collection of ESRL's D0 initial states could serve as an improved seasonal archive (even though not hindcasts or reanalysis); we don't know yet how good they are but they allow easy and precise comparison to competitive products if any should surface that play by climate science rules.

// Script parsed by PanoplyCL to create a plot. Usage:
// java -jar PanoplyCL.jar aice_h_in_REB.2017-09-11.pancl.pcl

// Open a dataset.
var ncdata1 = panoply.openDataset ( "/Users/a-team/Downloads/REBS nc/" );

// Select a variable.
var ncvar1 = ncdata1.getVariable ( "aice_h" );

// Create the plot.
var myplot = panoply.createPlot ( "lonlat", ncvar1 );

// Variable #1 (aice_h), dim 1 (time) -- Set to step 1 of 40
myplot.setVarDimension ( 1, 1, 1 );

// Specify plot settings.
myplot.set ( "size-factor", 200 );
myplot.set ( "size-width", 100 );
myplot.set ( "size-height", 50 );

myplot.set ( "title-text", "ice area  (aggregate)" );
myplot.set ( "font-master", "SansSerif" );

myplot.set ( "color-background", "black" );
myplot.set ( "interpolate", true );

myplot.set ( "color-invalids", "rgb(96,96,96)" );

myplot.set ( "scale-colorbar", "CB_Blues.cpt" );
myplot.set ( "scale-width", 40 );
myplot.set ( "scale-reverse", false );
myplot.set ( "scale-outlier-shape", "NONE" );
myplot.set ( "scale-outlier-side", "both" );
myplot.set ( "scale-outlier-gap", "thin" );
myplot.set ( "scale-tick-size", 7.0 );
myplot.set ( "scale-label-location", "BELOW" );

myplot.set ( "scale-min", 0.0 );
myplot.set ( "scale-max", 0.9999876618385315 );
myplot.set ( "scale-div-major", 9 );
myplot.set ( "scale-div-minor", 4 );
myplot.set ( "scale-exponent", 0 );
myplot.set ( "scale-log10", false );
myplot.set ( "scale-label-custom", false );
myplot.set ( "scale-tick-format", "%.2f" );

myplot.set ( "scale-minmax-note", true );
myplot.set ( "scale-minmax-format", "Same" );

myplot.set ( "grid-spacing-lon", 15.0 );
myplot.set ( "grid-spacing-lat", 15.0 );
myplot.set ( "grid-label-vis", false );
myplot.set ( "grid-label-size", 6.5 );
myplot.set ( "border-weight", 114 );

myplot.set ( "proj-name", "Stereographic" );
myplot.set ( "proj-lon0", -45.0 );
myplot.set ( "proj-lat0", 90.0 );
myplot.set ( "proj-xparam-1", 19.5 );
myplot.set ( "proj-xparam-2", true );

myplot.set ( "overlay-1-weight", 75 );
myplot.set ( "overlay-1-name", "Earth_mask.gif" );
myplot.set ( "overlay-1-color", "rgb(204,255,214)" );
myplot.set ( "overlay-1-invert", false );
myplot.set ( "overlay-2-weight", 0 );

myplot.set ( "proj-shading-vis", false );

myplot.set ( "grid-weight", 50 );
myplot.set ( "grid-style", "NONE" );
myplot.set ( "grid-color", "rgb(64,64,64)" );

myplot.set ( "contour-weight", 75 );
myplot.set ( "contour-style", "None" );

// Save plot image to disk.
myplot.saveImage ( "PNG", "aice_h_in_REB.2017-09-11.png" );

Developers Corner / Re: Creating Animated GIFs
« on: September 18, 2017, 01:21:52 PM »
Given the data, our forum posters make excellent line graphs, which adds analysis and thus value. We do some original work with satellite imagery from portals like Worldview, Jaxa and Sentinel Playground. However we mostly copy-paste map products (even when they're poorly done) rather than make better ones ourselves.

That involves geolocated data and map generating software providing control of the projection, overlays, and especially the palette. The latter four are well-handled by Panoply. Climate science data is generally stored as netCDF files or convertible.

These are basically just collections of excel spreadsheet stacks but are put into a complex non human-readable binary format because file sizes can be immense and non-desktop systems must read them.

As a practical matter, to make a netCDF file on a desktop requires a CDL file and a code snippet called ncgen. There's currently no online tool that will do this, it requires leaving your desktop OS for command line. That's too bad because the CDL format is easy to read, edit, and make from scratch in a text editor using comma separated data numbers.

Panoply can export any .nc file to CDL but cannot reimport them, ie it implements ncdump but not ncgen. Most of the benefit of export can already be seen in the Source window panel "Show info in expanded form'. The array panel in a map shows the numbers in its grid.

Panoply distinguishes geolocated Geo2D items in a netCDF file from generic 2D files. Both are plottable but only the former map onto the globe via a chosen projection. It's unclear how latitude and longitude data in an .nc file come to be associated with array data to convert it from 2D to Geo2D.

So with editing, creating and importing of palettes and data arrays mastered, we just need ncgen and a Geo2D description to be able to fix poor online maps or make our own from scratch using satellite data or other resources. Poster wipneus has, in effect, a workaround to do this in the complex instance of piomas data.

Note the UH AMSR2 3.125 km archive ( etc) doesn't arrive as Geo2D because the sea_ice_concentration variable is missing an 'attribute' that tells Panoply of the gridding scheme. However the land variable is seen by Panoply as a Geo2D variable that can be plotted on a map. That's because it includes a coordinate 'attribute' that explicitly names the auxiliary 2D latitude and longitude grid variables. The sea_ice_concentration variable is missing that coordinates 'attribute'. (Here 'attribute ' has a technical meaning within the format prescription -- thx to RBS here!)

Here are the most useful links I've found to date on ncgen and CDL formatting. Some of it makes sense but it's fair to infer that many code manual writers were not English lit majors in college.

Developers Corner / Re: Creating Animated GIFs
« on: September 18, 2017, 12:55:49 AM »
We're long been able to make excellent line graphs, given the raw data. Now we're moving on -- slowly --  from copy-pasting other people's maps to being able to make them better ourselves, given the raw data. That involves geolocated data and control of the projection, overlays, and especially the palette.

Some more on palettes most appropriate to climate science maps -- there's been a lot of online discussion lately on perceptually neutral palettes, the horrific rainbow palette, and palettes neutral to common male colorblindness.

However fashionably pc the latter might be, it's not enough. The fact is that a great many other factors influence how a given map is perceived by different people on different devices. Some are viewing on $10 cell phone screens; others have calibrated 27" retinal monitors with an advanced color gamut. And then there are academics still printing out pdfs on inkjet CMYK to uncoated paper to read on the plane. Not to mention journals that won't host time series as animated gifs and Twitter that makes dithered mp4's out of them, destroying individual frames. And at the end of the day, it's largely -- but not entirely -- a matter of individual taste.

A better solution may be simply to provide a link to the netCDF file for the map, allowing readers to quickly swap in their favorite Panoply palette.

A lot of online discussion of palette creation today consists of sharing command-line code snippets (mathlab, python, java, IDL, github, etc) with many participants unaware that Photoshop already did all this and much more 30 years ago, with freeware Gimp successfully emulating almost all of it, for example making any continuous gradient and discretizing it into a dozen bins.

It is vastly more efficient to make palettes from scratch and rapidly vary them (on the target map) with sliders in Gimp. This and simple spreadsheet RGB and HSL builds have resulted in very large galleries of free palettes. Most of them come unannotated as to construction however but it's easy however to assess their uniformity as grayscale (luminance, lightness, and average are shown in some popular modern palettes like parula, viridis, geosoft, coolwarm, and cube-helix below).

Nevertheless, a lot depends on the type of climate map, the 2D statistical distribution of the variables being displayed, and what 'talking points' the map product is supposed to bring out. An interesting concept here, after treating outliers, is 'histogram equalization' that results in more or less equal use of each color bin for maximization of color discrimination.

Is it possible to eliminate the subjectivity in palette choice by having its construction data-driven? For example, if grid nearest neighbor variance is high, then contouring will not be effective or even feasible. If it's low, then a binned palette might be effective; indeed the optimal number of colors might be inferable. Similarly, mean and std deviations might allow rational setting of outlier lumping.

Here are some of the better discussions, online palette tools, and palette collections I've come across. The graphics can be downloaded and the palettes imported into Panoply as described above.

Arctic sea ice / Re: The 2017 melting season
« on: September 16, 2017, 09:39:01 PM »
GFS forecast 975hPa moved forward to 19th
RASM-ESRL sees that hitting a bit differently in nine GFS comparisons, notably somewhat stronger wind ('arctic24' in Reb_plot archive) late on the 18th. From the forecast melt, it seems like warmer water might get stirred up in the ESS-Laptev area.

NSIDC, back on Sept 6th so not yet calling a minimum (which for 3 km AMRS2 open water was on Sept 12th), notes "the ice edge in the Beaufort Sea is extremely far north. In parts of this region, the ice edge is farther north than at any time since the satellite record began in 1979."

They don't give distances to nearest land in km though that can be taken as 111 x degrees of latitude to shore (or more simply as a radial gradient out from the North Pole shown over continental shelf bathymetry, bottom). Walruses dive no deeper than 80 m to feed; there's next to nothing of that near the ice edge.

Arctic1    RASM-ESRL vs GFS ice area and snow depth
Arctic2    RASM-ESRL vs GFS 2m temp and surface pressure
Arctic4    RASM-ESRL vs GFS 850 hPa temp and precip
Arctic9    RASM-ESRL vs GFS 500-1000 hPa thickness and precip
Arctic12   RASM-ESRL vs GFS surface temp and LWP
Arctic13   RASM-ESRL vs GFS surface pressure + 850 hPa height
Arctic19   RASM-ESRL 500hPa height and wind vectors
Arctic22   RASM-ESRL vs GFS longwave flux + shortwave flux
Arctic24   RASM-ESRL vs GFS surface wind + energy flux

Arctic sea ice / Re: Home brew AMSR2 extent & area calculation
« on: September 16, 2017, 08:27:56 PM »
last frame shows an interesting web of cracks in the MY ice north of the islands

There was a nice Sentinel-1 radar swath of that area the other day, a 1km resolution convenient mosaic at R Saldo's

That Beaufort tongue is quite persistent; the ESRL melt picture for Sept 15 shows ongoing melt there, mostly from the bottom. The wind (vector overlay, 4th image) starts to pick up on the 18th as expected from the predicted low pressure cyclonic event.

Developers Corner / Re: Creating Animated GIFs
« on: September 16, 2017, 07:02:04 PM »
Here is ESRL's experimental forecast for 5-day ice thickness change prior to adjacent alternative palette replacement. It may be better however to use fewer colors, say 2-3 on the thinning side and similarly on the thickening, or simple but methodical red and blue gradients increasing out in intensity.

The frames runs out to Sept 19th from a prediction made on the 14th. The first in the series was made on Aug 22nd for the 27th. The variable background color has been put to a consistent gray. It is hard to tell from these whether thickening or thinning is happening overall; other ESRL products may be better for this (notably just the forward prediction from today or just the D0 directly from the ESRL netCDF via Panoply).

Developers Corner / Re: Creating Animated GIFs
« on: September 16, 2017, 04:47:46 PM »
Below is the current Panoply palette collection. It can be viewed in Preferences; the collection below tiles up screenshots from there, hopefully faithfully. It takes a click to view at scale; reduction in dimensions is dicey because it might invoke interpolation.

The palette names suggest their origins, eg the CB prefix means imported from C. Brewer's widely used ColorBrewer web page at PSU, UO indicates Univ. Oregon geography dept., GMT means Generic Mapping Tools software from SOEST at the University of Hawaii, and many others originate at JJ Green’s cpt-city web site.

While anyone can quickly hack color tables in a spreadsheet or import them into Panoply as .rgb tex from govt climate sites, journal illustrations, twitter sites, or online product maps, there's a whole art and science to palette design.

The great advantage of Panoply, provided there's a netCDF, GRIB or HDF file associated with the map, is the chance to replace a bad palette with a good one, at the same time getting rid of vexatious contours, arrows, text and lat-lon overlays that interfere with the data and its comparison to other maps of other variables that might be in say an equal area projection. For example, a simple grayscale gradient is better-suited to arithmetic operations such as differencing.

An example application: ESRL's palette on 'icethicknesschange' is really bad. Not only is it inconsistent day to day but none of the palettes make conventional sense. That is, thickness is an ordinary real number that varies ± about 0. White is used for the lowest bins but sometimes gray, yet those are also used for the completely ice-free perimeter. This situation calls for a divergent color scale, eg reds for loss and blues for gains.

Since ESRL is using 17 color bins, the 'panoply_diff_17.act' palette is an immediate better fit. Another option, what the ESRL palette is attempting, is color-pairing. That amounts to taking a couple of ColorBrewer 8- or 9-color diverging palettes with trend logic and intercalating them to get better color separation while retaining logic.

Of the stock palettes in Panoply, interleaving CB_RdYBl_09.cpt and CB_PuOr_09.cpt is a good option. By displaying any .nc file as a map, those palettes can be cleanly captured into Gimp which has drag'n'drop indexed color table rearrangement.

The resulting palette, imported via ImageJ rgb listing into Panoply, can then display the ESRL .nc file properly. That palette is shown in the 4th image below. It could be improved for visual consistency with some minor tweaks.

Developers Corner / Re: Creating Animated GIFs
« on: September 16, 2017, 01:48:15 PM »
This is quite an interesting article on climate code and how it should be reported in journals, the observation being that a lot of research is currently irreproducible. Not in the sense that you would get a different result duplicating it but rather there's not enough information provided on the software side that would even allow you to get started replicating.

We run into that all time on these forums: maps come with no underlying data link, no underlying publication describing methods, and data not in formats like netCDF, HDF and  GRIB (all of which Panoply can view) that allow re-projection to a standard Arctic Ocean EASE grid or polar stereographic coordinate system which is necessary for comparing or combining data from different sources.

A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences
Damien Irving free full

"At first glance, the only difference between a regular journal article and that of IS2015 is the addition of a short computation section (see “Example computation section” for more information).

That section accompanied the traditional description of data and methods within the article and briefly cited the major software packages used in the research before pointing the reader to three key supplementary items: 1) a more detailed description of the software used, 2) a version controlled and publicly available code repository, and 3) a collection of supplementary log files that captured the data processing steps taken in producing each key result.

The repository was hosted at a popular code sharing website called GitHub, while the detailed software description and log files were hosted at Figshare (Irving 2015a), which is a website where researchers commonly archive the 'long tail of their research (e.g., supplementary figures, code, and data)."

An example of best practices:

The results in this paper were obtained using a number of different software packages. A collection of command line utilities known as the NetCDF Operators (NCO) and Climate Data Operators (CDO) were used to edit the attributes of netCDF files and to perform routine calculations on those files (e.g., the calculation of anomalies and climatologies) respectively. For more complex analysis and visualization, a Python distribution called Anaconda was used.

In addition to the Numerical Python (NumPy; Van Der Walt et al. 2011) and Scientific Python (SciPy) libraries that come installed by default with Anaconda, a Python library called xray was used for reading/writing netCDF files and data analysis. Similarly, in addition to Matplotlib (the default Python plotting library; Hunter 2007), Iris and Cartopy were used to generate many of the figures.

To facilitate the reproducibility of the results presented, an accompanying Figshare repository has been created to document the computational methodology (Irving 2015a). In addition to a more detailed account (i.e., version numbers, release dates, web addresses) of the software packages discussed above, the Figshare repository contains a supplementary file for each figure in the paper, outlining the computational steps performed from initial download of the ERA-Interim data through to the final generation of the plot.

A version controlled repository of the code referred to in those supplementary files can be found at

Arctic sea ice / Re: The 2017 melting season
« on: September 15, 2017, 11:44:56 PM »
5days out  GFS is predicting 978 hPa
Zack posted a nice animation of Arctic temperatures for Oct 2016 (ERA5, 2-m air temperature). Check out the surges of warmth through the Bering Strait and Barents areas.

ESRL is back at it after a computer glitch knocking out Sept 12th. They're not seeing a whole lot of action coming in the next ten days. The differencing map shows some lineations in ice thickness that don't fully correspond to anything that noticeable in Sentinel-1 images.

The latest on working with ESRL in Panoply is pulled together over at the Developers Corner. (The real developer over at Nasa-Goddard has been very helpful explaining features via email.) Notably, the full file structure is provided in annotated form and some palette trickery explained. Choice of palette makes a very big difference in bringing out features; this can be explored very rapidly in Panoply.,1259.msg129050.html#msg129050

There's a host of potentially useful forecast products that we've yet to do much with, in addition to the daily gifs ESRL offers on their web page.

snow ai    snowfall rate
rain ai    rainfall rate
hi_h      ice thickness
hs_h     snow thickness

strength   compressive strength of sea ice
divu       divergence of sea ice velocity
Tsfc       surface temperature where sea ice

vocn       northward sea water velocity
uocn       eastward sea water velocity
vvel       northward sea ice velocity
uvel       eastward sea ice velocity   

meltl      lateral ice melt
meltb      basil ice melt
meltt      top ice melt

Developers Corner / Re: Creating Animated GIFs
« on: September 15, 2017, 08:47:09 PM »
Panoply makes easy animated gif maps in all common projections, provided that the netCDF file itself being used has the data in multiple time shots.

These are geolocated maps in whatever projection and palette color you've found most effective by rapid experimentation, in our case typically the polar stereographic with a sequential palette for an ordered variable like ice area or a divergent palette for ± change about 0.

Panoply has ~100 projections and a great many palettes (plus it's easy to import more from galleries, grab from an online map say one from @zlabe, or made from scratch using rgb or hsv spreadsheet numbers).

The latter is most easily done by downloading a bona fide .rgb palette from UCAR (header and link shown below), opening it as plain text, and overwriting with your own numbers, followed by 'save as'. That trick defeats the Mac operating system which refuses to let you change .txt to .rgb manually.

ncolors  = 254
;R   G   B
0   22   132
0   24   135
0   26   136

Now use the Open file command in Panoply to import the .rgb palette; it immediately becomes active on the top map. The forum software also refuses text files attachments that are not .txt so I've relabelled the attached .rgb file below as .txt; it is the blue-to-white palette used at UH AMSR2 for sea ice concentration. (With this, I can use AMSR2 right up to today and then follow with the ESRL forecast in the same colors.)

That will be the case for the .nc files from ESRL climate forecasts which have numerous 5- and 10-day series at 6 hour intervals, so 20 and 40 frames respectively. If that's too many, set the 'stride' between consecutive frames to skip intervening ones.

Panoply does not make gifs per se, just the .png or .jpg frames. These just need to be loaded up as a stack in ImageJ as Oren explains above or as layers in gimp, then cropped/resized to 700 x 700 pixels max and saved as a forum animation. Panoply also makes nice .mp4 videos but the forum software does not accept that format.

Looking at the opportunities at ESRL on 15 Sep 17, the REB.2017.09.14 netCDF bundle offers seven 40-frame maps, forecasts out to D10:

* snow/ice surface temperature
* air temperature
* snow thickness
* ice thickness
* ice area
* air velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction
* ice velocity: x and y Geo2D files need to be opened separately, combine plots combined as sq root for magnitude, add vector for direction

Looking at the REB_plot component at the ESRL archive on 15 Sep 17, some 43 graphics are provided as .png or .gif but without the underlying netCDF Geo2D files (meaning projection and palette aren't easily changed). Some of the .gifs are stills but 22 of them are pre-made running gifs, 20-frame forecasts out to D5.

The MacOS has a wonderful feature for rapid scanning of the 43 graphics: select-all and right-click on Quick Look. This does not open the files in an app like Preview but displays them like a slide show, even animating the animations within such slides. (I use Quick Look with our game cam to flip though the tens of thousands of wildlife shots for keepers.)

The ESRL have issues such as too big to load or too wide to play on the forum, local rather than Arctic Ocean wide coverage, zero used for both data and not-on-ice background (ie confusingly same color, white), extraneous comparisons to GFS forecasts, esoteric meteorology variables, unwanted overlays like contours, and so forth.

Like all Panoply maps, these gifs strictly use undithered palette colors and so these can be selected and changed out. This can be done for all frames at once if the gif is loaded into Gimp, converted to RGB, tiled up by Filmstrip, color replaced, and sliced back out to individual frames.

The REB_plots have uninformative names at the moment, no clues as to date or content:

Arctic1    RASM-ESRL vs GFS ice area and snow depth
Arctic2    RASM-ESRL vs GFS 2m temp and surface pressure
Arctic3    ice thickness thermo + dynamics + snow melt + ice melt
Arctic4    RASM-ESRL vs GFS 850 hPa temp and precip
Arctic5    ice and snow thickness contoured
Arctic9    RASM-ESRL vs GFS 500-1000 hPa thickness and precip
Arctic10   bottom ice growth + lateral melt + snow melt + top ice melt
Arctic11   longwave + solar flux + shortwave + albedo
Arctic12   RASM-ESRL vs GFS surface temp and LWP
Arctic13   RASM-ESRL vs GFS surface pressure + 850 hPa height
Arctic14   RASM-ESRL energy flux + LWP + surface temp + IWP
Arctic15   melt pond fraction + SST + heat flux + wind speed
Arctic16   RASM-ESRL ice speed
Arctic19   RASM-ESRL 500hPa height and wind vectors
Arctic20   310K potential vorticity and Surface theta PVU
Arctic22   RASM-ESRL vs GFS longwave flux + shortwave flux
Arctic23   RASM-ESRL energy flux + LWP + temp + IWP [duplicate of Arctic14?]
Arctic24   RASM-ESRL vs GFS surface wind + energy flux

AlaskaRegion2   RASM-ESRL vs GFS 2m temp and surface pressure
AlaskaRegion4   RASM-ESRL vs GFS 850 hPa temp and precip
AlaskaRegion5   ice and snow thickness
AlaskaRegion12  RASM-ESRL vs GFS surface temp and LWP

The third component of the ESRL archive, of form RASM-ESRL_4NIC_2017-09-13.tar.gz, consists of ten time slices at 24 hour intervals in the form of Each of these consists of 18 Geo2D mappable products (or 16 after sea and ice velocity x,y arrays are combined).

aice       sea ice area fraction
hi         sea ice thickness
hs         surface snow thickness

snow ai    lwe snowfall rate
rain ai    rainfall rate
meltl      lateral ice melt
meltb      basil ice melt
meltt      top ice melt

sss        sea water salinity
sst        sea water temperature
Tair       air temperature
Tsfc       surface temperature where sea ice

strength   compressive strength of sea ice

vocn       northward sea water velocity
uocn       eastward sea water velocity

vvel       northward sea ice velocity
uvel       eastward sea ice velocity   

divu       divergence of sea ice velocity

A single such .nc does not have frames, it consists only of its time frame; to make a gif for say rainfall rate, ten one-frame maps have to be made and saved out as .pngs. From those, loaded in Gimp or ImageJ, a D10 daily gif animation can be made. However there's an issue with consistent scale because 'fit-to-data' will show a different ranges and so different utilization of the common palette.

Here are best-links to palette collections, conversion tools, appropriateness, Panoply manual, ESRL offerings, and Gimp indexed color:

Arctic sea ice / Re: The 2017 melting season
« on: September 15, 2017, 01:49:47 PM »
Here are the percentages of open water on Sept 14th for each of the last six years, based on 3.125 km resolution UH AMSR2.

Relative to what is possible (with Arctic Ocean as defined in the first frame as Inner Basin + upper Barents), the melt season peak on average is over halfway (51.4%) to a 'blue ocean' event.

Indeed, much of the Arctic Ocean is seasonally open already in yearly terms. Regions like the Chukchi experience a 'blue sea' event every year that lasts most of the year.

The lower image shows the six year average sea ice concentration for Sept 14th with 2017 outlined in yellow and 2012 in black.

Arctic sea ice / Re: The 2017 melting season
« on: September 14, 2017, 07:13:11 PM »
supporting data on polar bear decline?

Sure do, wish I didn't. One thing that confuses people is adult bear counts vs recruitment, ie females having enough food and denning opportunities to bear enough adult replacement cubs that survive to reproductive age at weight. Off-topic here, be good to pursue in depth on a separate forum.

Try looking at the primary peer-reviewed papers by field biologists (as well as field journals), forget the Koch's twitter page (Crawford), the nonsense around goose eggs, the ABC special situation, and environmental charities pitching false hope to donors. The habitat situation for both bears and walruses is hopeless, don't kid yourself.

You can best find them at Pubmed among its 27,000,000 abstracts. Svalbard also tracks polar bears. The Siberian populations, the Russians may or may not be able to track them in remote locations.

Back on topic, here is the what the bears and walrus are up against quantitatively in the Chukchi, Beaufort, and Barents: way too much open water for way too much of the year. We've shown open water over bathymetry (ie walrus food diving depth) many times up-forum. Snow depth is also up-forum (10 cm doesn't work out for dens, that's why they head for certain Svalbard islands).,230.msg128925.html#msg128925 rest of post

Arctic sea ice / Re: IJIS
« on: September 14, 2017, 05:20:20 PM »
This time of year, it is quite easy to track the area of open water at the 3.125 km resolution provided by UH AMSR2 via a single mouse click in any image processing software and histogram read-out.

There's a lot to recommend this over opaque legacy algorithms at far poorer resolution. The archive goes back to August 2012 so the seasonality of open water could readily be quantitated for the last five years.

'Binary thinking' -- blue ocean or not blue ocean -- just doesn't tell the climate impact story. Which is happening already.

While this is a rather flat year for picking the date of maximal open water which in any event has little comparative significance year-on-year, so far that is Sept 12th. According to both ESRL and Hycom (13th to 19th shown), another day or two of more open water is still to come.

The first image shows the mask used to exclude (yellow boundary) highly variable ice in the Fram, Nares, and CAA channels that would otherwise distort measurement of open water in the Arctic Ocean proper. Pixel counts serve quite accurately here to measure relative area because there's little day-to-day latitudinal variability.

The bottom animation fixes the Sept 12 maximum (right panel) and compares it to Sept 1-21 (2nd from right), then takes the difference (3rd from right) or just subtracts (left panel). Differencing the 12th with itself gives the two black panes at the small pause. The later frames show the Hycom forecast of maximal open water may be the 12th itself.

Arctic sea ice / Re: The 2017 melting season
« on: September 14, 2017, 03:09:10 PM »
Polar bears are flourishing.
Polar bears have taken a massive hit the last few years, notably in the Beaufort (fetus resorption), Chukchi (drowning), Hudson Bay (starving) and Barents (unable to reach denning sites). Inbreeding with brown bears? Selective advantages for life on ice become maladaptive on land meaning those genes will be rapidly lost, there's no turning back. Walruses though are far worse off already.

Arctic sea ice is a lagging indicator.
Might try "Arctic amplification" on google scholar before posting more nonsense. Arctic sea ice is the leading indicator of climate change. That's why we're here. As its current seasonal absence trends to imminent effective disappearance, it will have massive mid-latitude -- and indeed -- global consequences for climate.

Meanwhile, the zone north of the Barents is shape-shifting, translating, compactifying and melting. The first animation shows the moving 0-25% sea ice concentration line along with 4 floes over a static 13 Sep 17. The second shows open water along the Barents line; note the unexplained doughnut hole in the Laptev still persists but the opening towards the coast varies.

Consequences / Re: Hurricane season 2017
« on: September 13, 2017, 06:56:51 PM »
a heatwave in the American Southwest that kills 150,000 could be useful in [[waking people up]]
Well, that would be the homeless in Phoenix or Yuma, or a weeklong power failure affecting gas pumps (as nearby Flagstaff is 7000'). Everyone else has AC or a swamp cooler: home, car, parking lot, work, stores. It's like a Martian colony down here already.

Global mean surface temperature isn't applicable to the Southwest, it's one of the all-time bad IPCC jokes. It's been running +7ºF down here since March. Not a drop in the second half of the rainy season (below).

Irma + Harvey together will not bring a blip to CDC mortality statistics. The numbers are dwarfed by just the weekly murder rates in Miami + Jacksonville + Houston. In terms of cotton-tops (elderly) in nursing homes, they weren't going to live much longer on average anyway, so by CDC's reckoning (mortality weighted by life expectancy), it might take 40 of them to add up to one toddler. There's also been discussion on how to score Alzheimer warehouses.

Despite the hurricanes being primarily property damage, the info-wars may still have went well. The message did get out, despite an excellent stream of tweets from the Koch's hurricane guy R Maue, that climate change did contribute to frequency and intensity of these tropical storms. And a lot of the public here goes by their gut feeling, which worked to our advantage for once.

Arctic sea ice / Re: The 2017 melting season
« on: September 13, 2017, 04:08:08 AM »
The snow thickness time series below runs fro 15 Aug to 11 Sept at one frame per day using archived D0 initial states, then continues on as a forecast to Sep; 20th in 6 hour increments. This was fairly difficult to make given how the archive is organized, interface bugs in Panoply, and need for add-ons to basal Gimp but typifies value-added products that might suit forum interests better than the forecast-only perspective.

ESRL has not written up any documentation for this difficult-to-validate product so it is unclear to what extent it is satellite-based, a straight meteorological forecast, or coupled weather + ice + ocean. It may not consider snow drifts, melt, or rain-on-snow, in which case snow thickness per se would not yield thermal insulation or albedo. They have separate products for those though, as well as melt ponds.

We went through the summer believing snow cover was thicker and more persistent but without any concrete notion of how much the snow actually out there, what its condition was, and where it was distributed. ESRL may provide a sense of measurement accuracy down the road but for now it’s safe to say the snow is very thin, nowhere exceeding 20 cm. Note the storms that appear to sweep in from the south from the Svalbard area.

Arctic sea ice / Re: The 2017 melting season
« on: September 11, 2017, 11:15:48 PM »
ESRL is terrible. I know of much, much better models (forget names). They’re described in peer-reviewed journals (don’t recall cites, paywalled so haven’t read) but that data (proprietary format, floppy disk by request) shows how bad the ESRL fall 2016 validation really is (can’t really give specifics, no time).
Thanks for sharing.

Actually ESRL did not even begin their archive until nine months later, on July 27th of 2017. ESRL did not go ‘whole Arctic’ until the 22nd of August 2017. The web site was first opened on August 23rd. ESRL products, as labelled in 20 pt font on all 37 daily graphics, are experimental. And that product mix is still being revised daily into September.

Only as the upcoming refreezing season progresses will sufficient product accumulate for evaluation, after which the much-published PI will publish another journal publication. For now, let’s not confuse validation with placeholder (lorem ipsum dolor...)

In my view, there is only do, or not do.

In physics, if no experimentally testable prediction is coming out the door, you’re not doing. For climate science, forecasting means an .nc data file and its graphic stably posted without delay or registration barriers to the open internet. It does not mean twitter media or a dead link to little endian Word compilation in a non-peer reviewed supplement of an outdated paywalled journal article. At this time, only ESRL and Hycom are predicting the Arctic Ocean ice; only ESRL provides the data files.

ESRL is about forecasting out to D5 (and for some observables, out to D10); Hycom goes to D7. At this time, both offer archives but neither posts hindcasts or reanalysis. To make a good forecast, ESRL needs a decent D0, the initial state. So there are two separate validation issues: how good is the D0 and, given a spot-on D0, how well does their coupled air-ice-ocean model move it forward to D5? However the nature of the Arctic ice is such that neither is easily tested:

We’ve agreed to agree (with some holdouts) that nobody has the slightest idea what the ice thickness really is, within 50% error, across the Arctic Ocean on a given day. We’re not on solid ground when 2m of observed “buttery ice” at the North Pole on 05 Aug 17 is held equivalent to 2m brittle ice in the Lincoln Sea (show me the Arctic-wide buttery ice map). There is a week of accurate but limited 2017 swath data off Alert but so far it’s unpublished.

In 2019, the Polarstern will spend a winter collecting real ice thicknesses by helicopters 35 km to the sides of its drift. That data may be an uncomfortable fit to certain model products followed on the forums (the ones that make testable predictions, not all do).

Bottom melt? Nobody is down there in scuba gear; can predictions be assessed from a few broken-down buoys and moorings in wrong places? Rain gauges? Not a single one for an area the size of Europe. We go by droplets and soggy snow on a couple of web cams. That's a budget issue though, not string theory.

So while ESRL has gone all-in on ice thickness prediction, the best part really is their release of all the contributing components to that calculation (e.g. ice-to-ocean thermal flux daily graphic and its underlying .nc data). There’s no obligation to use all their inputs or computational pipeline. Anyone can stub in an improved ingredient if they have it, or use it for validation.

As examples, the UH AMSR2 3.125 km sea ice concentration is a far better product than some of the 25 km resolution competition. It’s backed by a careful journal article; the underlying .nc data accompanies the daily image; there’s no mickey-mouse registration barrier or ftp passwords. ESRL uses AMSR2 but not this one. So for a value-added hybrid concentration, the UH can be used as initial state until it runs out (day before) and ESRL can pick it up and take it forward to D5.

The time series shown below takes 20 archived ESRL initial states for ice thickness (which go up to today, Sept 10th) and pushes that out to the expected change in ice thickness on Sept 15th, D5. The archive is a bit awkward to use for this but a volunteer here is restructuring it in the auto-cloud.

The D5 image below has pixel counts next each palette tile. Taking the dot product of count and thickness bin to estimate volume change, it emerges that Sept 15 is a wash, relative to the 10th. There's slightly more melt than freeze but the difference is minor compared to error issues. This has been a very flat minimum and ESRL sees that continuing to mid-month.

Consequences / Re: Hurricane season 2017
« on: September 10, 2017, 11:48:42 PM »
Yet to hear anyone express solidarity with Marcos Island yacht owners, damage must be vast.

Consequences / Re: Hurricane season 2017
« on: September 10, 2017, 10:53:53 PM »
Marcos Island .... 6-9 feet underwater, 17,134 housing units, median sales price $525,000 based on 181 home sales, could run to $8 billion not counting infrastructure damage.

Arctic sea ice / Re: 2017 sea ice area and extent data
« on: September 10, 2017, 09:41:32 PM »
Looking at our highest resolution sea ice concentration, the 3.125 km resolution UH AMSR2, and simply counting pixels of open water (blue, respectively <20% concentration green) by date, suggests we not yet reached the fall minimum for either of these measures. The graph shows these counts normalized to the lowest count on 14 Aug 17; the raw data is attached as a csv text file. The animation is done to the <20% regions.

The forum / Re: https broken
« on: September 10, 2017, 02:59:20 PM »
allow https run in a much smoother way. forum now has it's own official ssl certificate, so this will also help a lot!
Will this help people find us at Google Search? Seems like GS finds the images and finds the texts but doesn't put them together properly.

Also wondering about linking in youtubes, if this will help. 


Arctic sea ice / Re: The 2017 melting season
« on: September 09, 2017, 03:13:40 PM »
in 3-5 days, that refreeze is stronger now then melting

The best quantitative information we have on that -- the ESRL forecasts out to Sept 13th -- says no.

In the 20 frames below, 6882 pixels of bottom freezing areas are shown, almost entirely in the lower bins, but 96104 pixels of bottom melt, a ratio of 1:14. The second animation just lumps freeze (to red) and melt (to blue) categories for easier visualization.

The ESRL data is readily available online, refreshed daily, both as an attractive web page and as easy-to-plot grid data archive. To date, only 3 of our 1310 members have looked at it.

Despite its sophistication, the model may not give spot-on perfect results (eg no attempt is made to resolve ±0). However it's hopefully more accurate than feelings, hunches, intuitions, wishful thinking, Old Arctic trendings, clay tablets or guesswork. If someone has a better source of near-real time physical data, please include the link to it so others may evaluate it. see Arctic10.gif Panoply: double-click on any Geo2D for an instant map

Consequences / Re: Hurricane season 2017
« on: September 09, 2017, 04:06:11 AM »
Dvorak ADT (3rd column) suggests late intensification:

Consequences / Re: Hurricane season 2017
« on: September 08, 2017, 08:15:27 PM »
Rather nice 4x4 animated tile-up of all the Goes-16 satellite channels that are the source of all the Irma imagery (along with VIIRS on the NOAA/NASA Suomi NPP satellite).

They are still obsessing over labelling every last image "non-operational" which the public does NOT need to hear now or ever. It merely means Goes-16 instruments are fully operational but perhaps still undergoing calibrations and positioning. Very similar language is used to "commission" a new sailboat which just means testing its systems for a few months close to harbor before crossing an ocean.

I dunno about "looting". In marine salvage, if the ship and its equipment or cargo have been abandoned, it's up for grabs under international law if you can bring it ashore. There is terrible poverty in the Caribbean, terrible disparities in wealth between the "former" slaves who make up 95+% of the residents, their resort hotel masters, and private mansion owners like Trump who don't spend even a week a year in their $19,000,000 beachfront palacios. (It's the same here in Tucson where I live!)

Arctic sea ice / Re: The 2017 melting season
« on: September 08, 2017, 06:41:29 PM »
Here is a sequence from the 3.125 km resolution UH AMSR2 sea ice concentration for the last 26 days. The 100% level is grayed out as nothing within is resolvable. However the crazy little floe whose voyage in the current, wind and tides is marked with a red star, shows specific ice features can be followed under favorable circumstances.

Hycom shows what the next week might bring: more garlic press plus odd goings-on near the Fram (arrows). Very little of the very thickest ice will be left by fall the way things are going.

ESRL shows ice thickness for the Beaufort thumb for the next 120 hours in 6 hour increments. It's underlying grid has resolution of 384 x 432, which is significantly higher than Piomas' 360 x 120 and  Hycom's (box of 100 grid points, 4th animation, from 365 day ave of motion vectors). If the ice here moves further in direction of the Alaskan coast (south is down in 3rd animation), it will find itself floating in much warmer waters.

Consequences / Re: Hurricane season 2017
« on: September 08, 2017, 12:56:12 AM »
Not to put too fine a point on it at a time of hectic breaking news, but the Andrew/Irma pair above were not rescaled nor oriented correctly. Best practice is to check work at large scale as a rapid-fire two-frame gif. Andrew is dropped onto the Irma layer by 'lighten only' which only works because the two are spatially displaced in the source images and Irma is darker underneath.

Arctic sea ice / Re: Latest PIOMAS update (September update)
« on: September 07, 2017, 12:06:18 PM »
That gives ESRL a very large improvement in resolution, 384x432 = 165,888 vs 360x120 = 43,200 is 3.84 times as many grid cells. This explains in part why Piomas graphics are so horrible.

Wip's excellent efforts (where are APL's?) can only put lipstick on a pig because the data is way off the mark: concentric rings of monotonically thinning ice on the 31 August are inconsistent with what we know (a lot) about ice age distribution classes and thickness. Their distribution is quite complex; they're in concentric bands off the CAA only to 0th order.

But we knew that much just from it being colder there. With a one-parameter fit (slope), the concentric rings can be generated and intersected with the ice pack shape from AMSR. Since a half meter error on two meter ice is acceptable on this forum, we're just about home with a photoshop hack.

If the Arctic Ocean proper is 8 million sq km, then ESRL cells are ~48 sq km or about 7 km on a side whereas Piomas cells are 185 sq km or about 24 km on a side. This matters relative to intrinsic scales of ice features such as floe size, ridging, ice edge and lead openings, dispersion, compaction, and localized interaction with air temperatures, melt, snow cover and rain.

It is a pity but the fact is, groups or individuals (like J Hansen after retirement) lacking adequate computer resources aren't going to be in the game, any more than they are in hurricane track prediction.

Equally unfortunate for Piomas is the unavoidable importance of graphics in effectively communicating results in climate science beyond a tiny clique of numerical analysts. See @zlabe to get an idea of what great graphics can do.

Along these lines, it's worth noting that ESRL is able to put out daily forecasts up to 10 days out in 6 hour increments (40 frame animations, bundles) whereas Piomas is hindcasting once a month several days after the fact, and only then because people here bug them. ESRL is thus readily integrable in near-real time in the GIS sense -- by anybody -- with many other independent .nc Arctic resources such as the 3.125 km UH AMSR2.

Panoply is not really "specialist software". It is a small, intuitive menu-driven bit of NASA freeware that's been available since Dec 2002 in everyone's operating system, updated regularly to the present. Nothing could be easier for viewing data arrays or exporting them (or subsets) as csv tables to excel. For people who do not want to look at command-line or big-endian (99% of forum members?), this is about the only option to unpack .nc files and get a quick map view of the data -- just double-click on any line labelled Geo2D. How hard is that -- and what is the alternative?

If it is so easy to get Piomas out of its funky coordinate system, why then do we have a separate 286-post forum explaining the intricate struggles to get it re-cast as something that can be compared to say Cryosat?

The PIOMAS grid is rendered on the 0.5km grid using polygons to approximate the curvilinear grid. This results in a maximum error of around 1km at the centre of the polygon faces. The result of this error is that each grid cell loses a small area to the grid cell in the row below and gains slightly less area from the grid cell in the row above. There is a similar error at the grid cell corners because the latitude & longitude co-ordinates for the PIOMAS grid are only provided to an accuracy of two decimal places.

There's free full text of Maslowski 2012 at ResearchGate, just google the doi:10.1029/2011JC007257. It's been cited subsequently 50 times. His academic papers overall have gotten a respectable 2,460 journal cites, about a third of mine.

Arctic sea ice / Re: The 2017 melting season
« on: September 06, 2017, 11:33:46 PM »
The overly complex bottom animation experiments with one of many possible options for 'adding value' via Panoply to the ESRL set of daily 5 day forecasts. It forecasts ice movement and air velocity 10 days ahead with a faint overlay of ice thickness.

Both ESRL and Panoply have widespread ambiguity issues with the same white 255 being used in too many places in palette tiles and as open water background. Incidentally, they changed that around quite a bit over night, renaming all the files more informatively, eliminating some and adding others.

ESRL is a very complex project but is probably the best thing going in terms of both forecasts and reanalysis of the coming refreeze season. It has a whole raft of data that we have not gotten into such as melt ponds, solar energy fluxes, upwelling longwave radiation, albedo, water and ice surface temperatures and so forth.

Panoply, the open-source all-platform software for .nc files such as ESRL, is as easy to use as a web browser so hopefully more people will dip into it. The palette controllers are very similar to Worldview, giving rather differently appearances to maps according to how it squeezes and truncates.

ESRL has quite a different, more nuanced take on current ice thicknesses than Piomas and as usual there is insufficient observational evidence to distinguish between them. Although it's fair to say that Cryosat differs wildly from Piomas in percentage thickness (2nd image below, cross-post of Michael's graphic), we don't yet have an experimental comparison for ESRL as it is too new.

ESRL archives three file bundles every day. The previous post looked at REB_plots which are the products such as 5 day forecasts used on their front web page.

The second bundle, eg, contains 7 plotable Geo2D files and animations that go out 10 days at 6 hour intervals for 40 frames. These are not offered on the ESRL web page. The map resolution or number of grid cells in the rectangular data array i x j appears to be 384 x 432 = 16588.

Here are the geolocated Geo2D files that can be animated. Note the two sets of velocity vectors would normally be combined into magnitude color and direction arrows:

aice_h   ice area  (aggregate)
hi_h      ice thickness
hs_h     snow thickness
Tair_h   air temperature
Tsfc_h   snow/ice surface temperature
uvel_h   ice velocity (x)
vvel_h    ice velocity (y)
uatm_h   atm velocity (x)
vatm_h   atm velocity (y)

Arctic sea ice / Re: Latest PIOMAS update (September update)
« on: September 06, 2017, 07:44:15 PM »
A comparison of Piomas to ice thickness at the new NOAA-RASM-ESRL-PSD site ( shown). It's not clear that a netCDF file exists for Piomas thickness, they seem still into fortran mode. This makes quantitative comparisons fairly difficult. Maybe that's the idea (?).

Here's a cross-post to Michael's careful comparison of Piomas to Cryosat; the comparison of ESRL to the experimental data from that satellite is not yet feasible.

None of the many thickness products can reproduce the 05 Aug 17 observations of "buttery" ice (ie not brittle) by a Russian icebreaker at the north pole. The transition between brittle and ductile ice is discussed here:

Arctic sea ice / Re: The 2017 melting season
« on: September 05, 2017, 09:19:25 PM »
This ESRL site is putting out so many excellent new graphics daily that it is a challenge to assimilate them on the forum. The question is, can we add value re-working raw netCDF file combinations in Panoply, or has ESRL already done everything worthwhile? (Answer: not yet but they are improving the site almost every day.)

NOAA-ESRL-PSD has an institutional focus on the Chukchi and Beaufort, presumably because of their adjacency to the US. While the PI's real interests are likely in climate change, the project funding rationale presumably revolves around maritime safety, drilling platforms, coastal erosion, and futuristic naval conflicts.

Indeed, model quality greatly improves on the navy's site (Hycom inset, first animation vs ESRL's version of the Beaufort thumb; the former which the oft-maligned NavGem and has not been updated since 2016. The eight comparison graphics with GFS models imply a very big improvement there as well.

Today's 5-day net melt forecast (top + bottom + sides) remains melt everywhere, no net refreezing growth anywhere in the Arctic Ocean. In the 2nd animation, keep an eye on the palette colors used for freezing -- there is no use of them yet in the 20 forecast time frames. That doesn't mean ice isn't thickening in places due to other considerations (upper left corner), nor does it mean the 5-day direction of change is not positive in places (3rd animation).

The following little database provides the current content of ESRL's elegant web page graphics, the files found in their REBplot archives. We've only explored 7 of the 31 offerings so far (x column);
a 2x2 entry indicates 4 distinct but related displays per frame. Note 8 of these are comparisons of RASM-ESRL to lower quality GFS models. Twenty png's means an animation showing 4 frames per day x five days; elsewhere ESRL is going out to day 10 so 40 frames. front page graphics navigator download archive

1   figAR2   AlaskaRegion2   20 png   2x2      RASM-ESRL vs GFS 2m temp and surface pressure
2   figAR4   AlaskaRegion4   20 png   2x2   -   RASM-ESRL vs GFS 850 hPa temp and precip
3   figAR5   AlaskaRegion5   20 png   1x1   x   ice and snow thickness
4   figAR12   AlaskaRegion12   20 png   2x2      RASM-ESRL vs GFS surface temp and LWP
5   fig0 fig6.gif   Arctic 0   1 png   1x1   x   RASM-ESRL ice edge evolution
6   fig0 fig7.gif   Arctic 1   1 png   1x1      snow depth 5 day change
7   fig0 fig8.gif   Arctic 2   1 png   1x1   x   ice thickness 5 day change
8   fig0 fig17.gif   Arctic 3   1 png   1x1   x   initial ice thickness
9   fig0 fig18.gif   Arctic 4   1 png   1x1   x   ice thickness day 5
10   fig1    Arctic1   20 png   2x2      ice area and snow depth RASM-ESRL vs GFS
11   fig2    Arctic2   20 png   2x2      RASM-ESRL vs GFS 2m temp and surface pressure
12   fig3    Arctic3   20 png   2x2   x   ice thickness thermo + dynamics + snow melt + ice melt
13   fig4    Arctic4   20 png   2x2      RASM-ESRL vs GFS 850 hPa temp and precip
14   fig5    Arctic5   20 png   1x1      ice and snow thickness contoured
15   fig9    Arctic9   20 png   2x2      RASM-ESRL vs GFS 500-1000 hPa thickness and precip
16   fig10    Arctic10   20 png   2x2      bottom ice growth + lateral melt + snow melt + top ice melt
17   fig11    Arctic11   20 png   2x2      longwave - solar flux + shortwave + albedo
18   fig12    Arctic12   20 png   2x2      RASM-ESRL vs GFS surface temp and LWP
19   fig13    Arctic13   20 png   2x2      RASM-ESRL vs GFS surface pressure + 850 hPa height
20   fig14    Arctic14   20 png   2x2      RASM-ESRL energy flux + LWP + surface temp + IWP
21   fig15    Arctic15   20 png   2x2      melt pond fraction + SST + heat flux + wind speed
22   fig16    Arctic16   20 png   1x1   x   RASM-ESRL ice speed
23   fig19    Arctic19   20 png   1x1      RASM-ESRL500hPa height and wind vectors
24   fig20    Arctic20   20 png   1x1   *    310K potential vorticity and Surface theta PVU
25   Meteograms   Beaufort   3 png   1x1      meteogram + cross section + text
26   Meteograms   Tiksi   3 png   1x2      meteogram + cross section + text
27   Meteograms   Oliktok   3 png   1x3      meteogram + cross section + text
28   Meteograms   Eureka   3 png   1x4      meteogram + cross section + text
29   Meteograms   Barrow   3 png   1x5      meteogram + cross section + text
30   Meteograms   Alert   3 png   1x6      meteogram + cross section + text
31   Meteograms   BeringStraits   3 png   1x7      meteogram + cross section + text

Consequences / Re: Hurricane season 2017
« on: September 05, 2017, 12:29:34 AM »
Talk about horrifying projections
Indeed. The projection is enlarged slightly below. The eye appears at the junction of Key Largo and and mainland. The Homestead air force base is 6 miles ENE of town.

Arctic sea ice / Re: The 2017 melting season
« on: September 05, 2017, 12:06:52 AM »
Nice presentation of trajectory, almost looks like a hurricane forecast! @zlabe posted a beautiful extent anomaly graphic today, saying no recovery this year relative to climatology...

Arctic sea ice / Re: The 2017 melting season
« on: September 04, 2017, 10:14:22 PM »
expect a far more gradual increase in in thickness  away from the ice edge, like the  Hycom image
Good question, Niall. ESRL does provide sections on validation and forecast skill assessment that might offer more details. JimH notes up-forum they are using v5.1 whereas Hycom is on v4.6 of ACFNS. A version change history might help but it won't yield a quantitative description of improvements applicable to this specific instance. The ESRL model is much more up to date overall.

Running the ESRL file through Panoply can change the output visual appearance in many ways, for example choice of palette as continuous or banded, whether the palette is fit to data or user-specified range, whether interpolation is on or off, how contouring is set, map projection choice and parameters, how image size is set and so forth.

'Interpolation off' is supposed to show actual grid cell resolution. It is easy to lose track of this in all the pretty flowing colors of a bicubic'ed 40-frame animation or smoothed MP4 video. (In Hycom ice motion, we've seen up-forum that averaging those long black vectors over 30 days reveals a surprisingly coarse grid base.)

Panoply does allow drilling into the netCDF numeric array to copy-paste the underlying grid cell data, here for this surprisingly persistent thumb of ice off Banks Island (bights are embayments, not so applicable). That would be the literal model output, graphics are downstream.

Earlier in the year, there was a magnetometer overflight in this region that can measure the ice thickness quite accurately (as the difference between sea salinity and lidar surface elevation). These are not flown at regular intervals as far as I know. In short, there's no current observational data against which to test the models.

The animation below shows a couple of Panoply views of the thumb from and their official version from REB_plots.2017-09-03 fig.17 (which is done in some unknown software used at Noaa). I could not accurately re-project or re-scale these to match the Hycom as its map parameters aren't specified. Note Panoply can save as kmz which could provide a common meeting ground on Google Earth (though it is very buggy and has not been updated in years).

Overall, ESRL thickness nearly stalls out by 13 Sep 17 though it is perhaps less than on the 3rd (bottom animation, Panoply zonal ice thickness forecast aggregated by latitude, which is like a Hovmöller diagram, only animated). The yellow line is the 3rd, the last green is the 13th. The areas under the curve would be volume if these were equal-area pixels. As is, the 13th has 99.4% as many 'volume' pixels as the 3rd. That difference is no doubt dwarfed by model error.

Arctic sea ice / Re: The 2017 melting season
« on: September 04, 2017, 07:47:38 PM »
The first animation shows "snow/ice surface temperatures" from today to Sept 13th, from today's ESRL. The DMI 80ºN number has limited value in describing conditions but there's a multi-year record that could equally be relevant or misleading for melt seasons in the 'new Arctic'.

The second shows ice thickness predicted for the same date range: the ice pack morphs to new shapes with little net change in volume.

Arctic sea ice / Re: The 2017 melting season
« on: September 04, 2017, 12:46:47 AM »
The first animation shows ESRL-PSD's forecasted snow thickness (orange) in six hour increments for the next ten days overlying forecasted sea ice thickness (blue). The second shows their ice thickness by itself.

The third shows contours for the thicker snow classes over the same ice thickness series. The fourth shows the various arithmetic means over this period, again with contours of thicker mean snow over mean ice thickness.. This time of year, there's little incoming solar energy; depending on its melt history, thicker snow might insulate the ice underneath from the cold.

These are produced from ESRL's netCDF Geo2D files, as visualized with Panoply, as post-processed in Gimp, via thousands of discarded intermediate files. It might barely be possible to keep up with ESRL's daily output depending on which of the 37 products in which repurposed combinations prove most informative here.

Arctic sea ice / Re: The 2017 melting season
« on: September 03, 2017, 08:04:11 PM »
The first animation shows how the location of open water has changed relative to its position as fixed on 13 Aug 17. To keep the colors scheme simple, non-zero sea ice concentrations are blacked out,

The second shows the snow thickness forecast from ESRL,, via Panoply in 40 time increments out to Sept 11th. Note the snow is typically ankle-deep and at most mid-calf; the forecast has it thinning. While this seems a decent product, the ESRL archive doesn't start up until mid-August which leaves us without data for spring and summer.

The thicknesses could be summed etc in Panoply but more informative is where thick snow lies over thin ice etc; 9 bin palettes for both gives 81 thickness combinations for each animation frame. I'll add the ice thickness counterpart animation in a bit.

The netCDF sea ice concentration file from UH AMSR2 hasn't worked out yet: it provides concentrations in a generic raster '2D' file rather than as the geolocated 'Geo2D' that Panoply needs,  although UH provides linked '1D' linked lat,lon files.

The role of snow changes with the seasons, from being a reflective surface in early summer inhibiting melt pond formation to being an insulating blanket in winter keeping frigid Arctic air away from the ice.

However 'snow' doesn't really describe the range of variability of physical properties for a snow pack. The Sami language of northern Scandinavia has some 300 words for snow types and Eskimo-Inuit over a hundred. There's been a lot of back-and-forth on this since F Boaz's writings in 1911; for an update, see:

Neither ESRL or anyone else is offering a rain product. That's unfortunate because we know for certain that it rained at the North Pole on 05 Aug 17, that lightning detection products for the Beaufort implies summer rains there too, and more warm vapor swept in this fall from the North Atlantic. The issue here is both latent heat release from the rain and associated humidity and major optical and thermal changes in the properties of the surviving snow.

Arctic sea ice / Re: The 2017 melting season
« on: September 02, 2017, 07:54:01 PM »
Here is a hybrid of ESRL zonal ice location (aggregation by latitude, REB plot comparing 6 hour intervals of 22 Aug (purple line) and 01 Sep (blue)  with three weeks of UH AMSR2 ending the same date and 30 days of Hycom ending 09 Sep 17.

The ESRL products are still in design flux; they don't move from their Alaska focus to whole Arctic Ocean until Aug 22nd in their archive. In the ten day animation of ice thickness below, the palette itself is in day to day flux. Since ESRL's focus is D5 prediction, it's not likely that they will fix archival plots.

Consequences / Re: Hurricane season 2017
« on: September 02, 2017, 02:04:22 PM »
Flooding on the western edge of Addicks Reservoir
Meanwhile, mopping up some misinformation up-forum using new information from the Harris County flood control district twitter site

Addicks Reservoir is currently releasing 7000 cfs and Barker 6300 cfs which will continue for another couple of weeks before cutting back to 4000 (if no further rain). Elsewhere newspapers had given these as 4000 + 4000 as conduit spill design maximums. Evidently not.

The emergency spillways (armored sections of berm ends) are tapered downwards. This means the north end of the north Addicks spillway overflows first; if the reservoir rises further, overflow moves up higher on the taper. The other spillway, being at higher elevation all along its taper, never comes into play. (The Addicks watershed had about 35" of rain, not 60".) They expect a slow end to Addicks north end spilling by Saturday Sept. 2nd.

The effect of draining the reservoirs is to keep Buffalo Bayou too high. Many houses that are flooded now will stay flooded for another two weeks. If they had not aggressively drained the reservoirs, up-reservoir flooding would have been prolonged.
As of 8/31 they've estimated an 136,000 flooded structures, roughly about 10 percent of registered HCAD structures. (Harris County Appraisal District lists all homes for property tax purposes). The previous flooding record here was 70,000 homes.

Consequences / Re: Hurricane season 2017
« on: September 01, 2017, 10:27:37 PM »
Whoa, wet and muddy Houston aftermath on Sentinel-2AB playground. The overtopped berm on Addicks has a white rectangular service building at its end.

Guardian on Friday 1 Sep 17 14.31 EDT: The Texas department of public safety said more than 185,000 homes were damaged and 9,000 destroyed, estimates which are likely to rise once receding waters give authorities access to heavily populated suburbs.

With swollen rivers and reservoirs still risking potentially deadly flooding, the Red Cross said the number of people in shelters across the region had increased to 42,000.

Arctic sea ice / Re: The 2017 melting season
« on: September 01, 2017, 08:32:58 PM »

No problem. I should like to thank a very generous donor in Greece for a new computer -- the fastest Mac ever built (no hard drive etc) -- which makes more complex animations possible, the Evans and Sperl Foundations for extensive logistical support, countless others for all the open source code and initial imagery, and of course Neven and the regulars for years of most excellent hosting and posting.

This new ESRL site, while experimental, appears a real breakthrough for both forecast and reanalysis products. We need to somehow assimilate what they are doing to stay in the  game for near-real time.

It does not work just to gesture at their url and say it shows such-and-such because they overwrite the online graphics every day. It's not possible to link to individual gifs because they're in a bundled -up archive compressed. Nor is copy-and-paste off the web page effective because their imagery is over-sized.

Yet we'd like to be like wikipedia on these forums and have every posted claim linked to a supporting source lol.

Quite a few people here are now doing respectable stills and animations though cropping, contrast, and resizing skills remain elusive. It's possible but not easy to restructure ESRL products to 700x700 pixel forum size with full retention of information. Panoply can potentially do this better without adding another layer of complexity -- provided it comes to you with configured defaults.

So for now we need to look at exactly which of the 37 ESRL products are most useful going into the refreeze season and how to get a dozen or more members reprocessing up to forum standards. It may be feasible to mirror the ESRL site in a Panoply post-processed sense which would the bar enough to let everyone in.

Right now the ESRL archive is up to 10.1 GB with Aug 31st upload taking the form of three file types summing to 330 MB unpacked; the netCDF can be opened in Panoply remotely.

RASM-ESRL_4NIC_2017-08-31.tar.gz   37.4 MB   232 MB
REB_plots.2017-07-27.tar.gz 60.8 MB

Panoply itself offers not a single word towards a v4.8 help manual despite a 2002 initial release but there's a 2012 German tutorial that works through your choice of Pangaea data set that illustrates Panoply's capabilities. version 1.5 (ancient)

Below is the same ESRL sea ice motion as above re-animated using different Panoply features and options. This is a vast improvement over Hycom's forecast motion both in scientific accuracy and graphical communication of it.

Consequences / Re: Hurricane season 2017
« on: September 01, 2017, 07:19:16 PM »
Wow look at this! You can actually see the flood waters flowing out of Houston and into The Gulf of Mexico from space!
Sigh, another tv journalist tweeting away without providing a timestamp, credit or source for the satellite material. That means everyone else must spend hours chasing down Landsats or Sentinel-1ABs trying to find the cloud-free scene at high resolution.

It turns out to be a Modis Terra 250m available at WorldView (momentarily down).  Also seen on GOES16 Not good enough, we need the Sentinels here.

There'll be a big bathtub ring of mud defining the high water mark. Since houses in Houston are almost always higher than their lawns, if the street flooded but the lawn is still green, that house did not flood. Ditto for cars on that driveway. So this is a very unusual situation wherein oblique aerial photos are not needed for a damage assessment.

The flooded cars are probably totaled from the perspective of the insurance company, meaning the title gets automatically changed to 'salvage' and later to 'rebuilt' if someone can get them running again. These will be wholesaled at Denver auctions and redistributed all over the US with 'clean' titles and wiped carfax histories to unwary buyers.

It is far too early for the White House to talk about the extent of damage -- 100k homes is just a wild guess. The real issue may be road beds, bridges, sewer and water lines. Costs to fix those are astronomic. However these these strong, self-reliant, white evangelical trump-voting (every single Gulf Coast county!) coastal Americans will reject anything that smacks of a gov't handout (?) as they have always done (?).

In the early days of Harvey, it was reported literally hundreds of times that GOES-16 was "not operational", even as the scientist or reporter was posting imagery from an obviously fully functioning satellite.

The geostationary GOES-16 was actually launched in November 2016 and became operational shortly thereafter (for those not needing the solar instruments, quantitative calibration or final East positioning). Its imagery packets are beamed down compressed but not encrypted and anyone underneath can download and unpack them at their home. That's operational from my perspective.

The advanced baseline imager is the primary weather instrument on GOES-16. It has sixteen spectral bands: two visible, four near-infrared and ten infrared. It's at higher bit depth and 4x better   ground resolution than previous satellites in the series. Being 35,786 km up, it cannot take high resolution ground photos like Landsat or Sentinel.

The GOES-16 Geostationary Lightning Mapper images intra-cloud lightning 24/7 of severe storms even when high cirrus clouds obscure underlying convection from the ABI. Flash rate correlates with increased storm intensity. The  telescopic CCD camera is tuned to 777.4 nm with a spatial resolution of 8 km, capturing 500 frames per second.

Pages: [1] 2 3 ... 37