Please support this Forum and Neven's Blog

Author Topic: Mapping GeoCoded Data Sets  (Read 775 times)

CognitiveBias

  • ASIF Lurker
  • Posts: 72
    • View Profile
Mapping GeoCoded Data Sets
« on: February 03, 2017, 11:02:44 PM »
Since ECMWF has an up to 2 month lag on posting this data set I'm looking for a geo coded temperature data set that's up to date.


You may wish to take a look at:

ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/surface/

Perhaps we should take this conversation over to the Developer's Corner?


Thanks Jim, That gives me Jan 2017 to play with. 

I was also pointed to Tealights thread on Albedo warming potential.  I love the data presentation he's doing with the color coded map.  Can anyone give pointers to getting started with something like that?

Jim Hunt

  • ASIF Governor
  • Posts: 2990
    • View Profile
    • The Arctic sea ice Great White Con
Re: Mapping GeoCoded Data Sets
« Reply #1 on: February 03, 2017, 11:43:06 PM »
I'd hazard a guess that you're after something a bit more flexible, but for viewing geocoded data sets I start off with Panoply, which allows you to easily create animations:

https://www.giss.nasa.gov/tools/panoply/

A-Team recommends ImageJ if additional processing is required, but I've never tried it:

http://forum.arctic-sea-ice.net/index.php/topic,165.0.html

Perhaps Tealight will pop in here and reveal his secrets?
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

CognitiveBias

  • ASIF Lurker
  • Posts: 72
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #2 on: February 04, 2017, 01:12:23 AM »
Thanks again.  I will check those out. 

You are correct that I'm not looking for a professional developer level experience.  I just started playing with the historic temperature data to see for myself how the changes are geographically distributed.  I did get the data I was after, but my initial attempts at visualizing the results were underwhelming.

A localized FDD map, updated hopefully daily seems like an interesting next challenge.  Anyway, it beats Soduku.




Tealight

  • ASIF Citizen
  • Posts: 236
    • View Profile
    • CryosphereComputing
Re: Mapping GeoCoded Data Sets
« Reply #3 on: February 04, 2017, 12:28:25 PM »

You may wish to take a look at:

ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/surface/

Perhaps we should take this conversation over to the Developer's Corner?


I had a look at the NOAA reanalysis map and think it's terrible to create a FDD map for the Arctic. You would need transform the mercator projection into a polar sterographic projection. There probably are some libaries which help you to do it, but it will be extra work.

For all of my work I use Python 3 with the Anaconda distribution. It includes essential libaries like numpy and matplotlib to create maps/graphs. I learned the basics of Python in around one month and the rest is googling your needs because 90% of your problems were already solved by someone else.

CognitiveBias

  • ASIF Lurker
  • Posts: 72
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #4 on: February 04, 2017, 02:07:26 PM »
Thanks for the info Tealight.

I'm not sure why the Mercator projection would come into play.  I already have a geo coded data set of temperature data from ECMWF.  I'm really just trying to get that data mapped onto a polar map.  Then animated for good measure.  Your albedo anomaly animations seem the perfect way to display this type data.

Am I misunderstanding something wrt the NOAA reanalysis map?


   

Jim Hunt

  • ASIF Governor
  • Posts: 2990
    • View Profile
    • The Arctic sea ice Great White Con
Re: Mapping GeoCoded Data Sets
« Reply #5 on: February 04, 2017, 09:04:31 PM »
For all of my work I use Python 3


Wipneus also uses Python for most (if not all?) of his work. See also:

http://forum.arctic-sea-ice.net/index.php/topic,782.0.html
Reality is merely an illusion, albeit a very persistent one - Albert Einstein

Blizzard92

  • ASIF Lurker
  • Posts: 34
    • View Profile
    • Personal Research Website
Re: Mapping GeoCoded Data Sets
« Reply #6 on: February 04, 2017, 09:31:20 PM »
Hi everyone!

I am really interested in Arctic visualizations/data and would be happy to help anyone else interested on how to get started. I primarily use Python 2.7 (soon switching to 3 - but no big deal either way).

What would be helpful... for instance: a blog tutorial on reading and plotting map data (such as from reanalysis)?
UC Irvine - Earth System Science Ph.D. Student
Cornell University - Atmospheric Sciences B.Sc.

Twitter: @ZLabe
Website: http://sites.uci.edu/zlabe/

Blizzard92

  • ASIF Lurker
  • Posts: 34
    • View Profile
    • Personal Research Website
Re: Mapping GeoCoded Data Sets
« Reply #7 on: February 04, 2017, 09:35:11 PM »
This may be of interest for some people - this was a course I helped teach at UC Irvine last summer on using Python to visualize Arctic sea ice data.

https://github.com/strongh/DIRECT-STEM-climate-workshop

Click on "notebooks" to find the Python code (via Jupyter notebooks). For instance, here is one on Arctic amplification in CMIP5: https://github.com/strongh/DIRECT-STEM-climate-workshop/blob/master/notebooks/ArcticAmplification.ipynb. Jupyter notebooks are useful because they are interactive via code followed by output/text/instruction.
UC Irvine - Earth System Science Ph.D. Student
Cornell University - Atmospheric Sciences B.Sc.

Twitter: @ZLabe
Website: http://sites.uci.edu/zlabe/

CognitiveBias

  • ASIF Lurker
  • Posts: 72
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #8 on: February 05, 2017, 12:00:22 AM »
Thanks guys... I appreciate the attention a great deal.  I will look thru the provided content when I get another block of time.

I have gotten the Panoply piece working somewhat.  It displays netCDF content without issue, and also provides any number of map projections.  Getting an Arctic ocean display from the ECMWF data sets was easy enough.

netCDF seems to be the lingua franca for Panoply.  I have read in netCDF data from ECMWF, and processed it in a relational database to get the FDD data.  Just another step to output netCDF representing the geo coded FDD actuals and anomalies. 

Panoply allows export of KMZ for use with google earth, which I've not yet explored.  It also allows export of the time series to a .MOV.  I quickly generated a 30MB looking at low res daily temps.   I'm not sure this is the right playback format due to size, so keeping my eyes open. 

All in all, relative to expectations, some decent early success.  Thanks for all the help.

Using this localized FDD concept I can see the collapse of freezing potential near FJL and serious degradation for hundreds of miles poleward from the Bering Strait.  I think the right visualization of this would be powerful.

slow wing

  • ASIF Middle Class
  • Posts: 505
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #9 on: February 12, 2017, 01:44:42 AM »
Thanks for the interest in a map of FDD and for looking into coding it up. I am looking forward to seeing these plots and - thanks Blizzard for the links - there is an outside chance I might try to learn Python to try something myself, but likely not realistic.


Would like to raise one issue of definition: when is the first day to use for calculating FDD(x) at each point, x (shorthand for a two dimensional coordinate)?


Use a single, universal date for the whole map, i_start? This seems a poor solution to me. The problem with that is that the colder places will begin freezing before the warmer ones. The warmer places would still be having days above the reference temperature, Tref (either 0 or -1.8 degrees C, not sure that has been decided?) while the cooler places begin accumulating FDDs.

  A better solution imo would be to store a separate start date, i_start(x) appropriate for each point x on your 2-dimensional map grid.

Further, I would propose a suitable definition for i_start(x) as the start day that maximises the FDD at that point x.

That is, the FDD at point x on day n, FDD(n,x) is determined by stepping over days, j, and finding the maximum for:


FDD(n,x) = max. for i_start(x) of, sum[j=i_start(x),n] (Tref-T(j,x)).


The day i_start(x) only needs to be determined once for each x - just step through the days up to any day n well into the freezing season and record the value of i_start(x) that maximises it.


Has anyone encountered this as an issue? Does the above proposed solution make any sense?
« Last Edit: February 12, 2017, 01:56:28 AM by slow wing »

CognitiveBias

  • ASIF Lurker
  • Posts: 72
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #10 on: February 16, 2017, 11:34:15 PM »
slow wing,
  Sorry I missed the post.  I was winging it, for the most part.  I used Sept 1 each year as a start.  FDD was never allowed to go negative.  However, FDD can you up or down in any given day.   I have not written the output netCDF format required to visualize this new data set using panoply.

I did get distracted looking at some other data and satellite images.  Given a bit of interest I can definitely get back into the project.

-CB

slow wing

  • ASIF Middle Class
  • Posts: 505
    • View Profile
Re: Mapping GeoCoded Data Sets
« Reply #11 on: February 26, 2017, 11:58:06 PM »
Hi CognitiveBias

   Yep, that should also be fine. I might have been getting ahead of myself thinking about a problem that doesn't matter - your option of using Sept 1 everywhere is simpler and it would be great to get a first look at the distributions!