[ncl-talk] Important announcement regarding the future of NCL

Puneet Sharma puneet.988 at gmail.com
Fri Feb 8 02:05:11 MST 2019 

Hello all

Since I extensively use python for analysis of data generated from model
simulations and satellite data, I would like to share some of the strong
points of python.

1. Handling of netcdf, hdf and grib data is easy. There are packages such
as pyhdf, netcdf4-python and pygrib. Also for .h5 files there is h5py
package. CSV data handling is also available.

2. I use the python ecosystem called CDAT developed at LLNL (
http://cdat.llnl.gov/) . It has packages such as cdms2 to easily handle
netcdf data I/O and its metadata. It is capable of handling large data sets
of size of the order of GBs. It also has packages such as cdutil and
genutil for performing various statistical analysis of time series data
from simulations and observations. Since the data usually has masks and
fill values, the functions and procedure available in these packages
automatically take care of that. Compared with NCL, CDAT seamlessly handles
masking of data using the MV2 package. All of these packages have numpy as
their base which is a python package for manipulating numerical data. I
have struggled with NCL for masking data which lead me to switch to python.

3. Regridding of data and handling of various grids and interpolation
routines which are available in NCL are also available CDAT.

4. String manipulation, data subsetting is very intuitive in python (CDAT).
Moreover python code is highly readable since it requires forced
indentation which keeps the code clean and readable.

5. The exceptional plotting libraries available in python such as
matplotlib, seaborn, bokeh etc have an edge over NCL since very few lines
of code can produce complex plots in publication format.

6. Python has intuitive data structures like dictionary, lists and
functions like zip, enumerate for iteration which are not available in NCL

The only place where I found python to be lagging was the spatial plotting
of gridded data. Compared to matplotlib-basemap plotting tool, NCL gives
far more clearer and visually appealing spatial plots. But that issue has
already been solved through PyNGL package.

I would further like to emphasise that together with the points mentioned
above, there are many use cases in python which are not available in NCL
such as

1. Interoperability with languages such as C, Java, R, FORTRAN since there
are wrappers readily available for these languages.

2. Handling url requests for downloading data from different servers.

3. Handling of categorical data using the pandas package.

4. It is easy to call contemporary statistical functions and packages
written in R through rpy2 package.

I feel that python can become one single platform for performing different
tasks related to atmospheric and climate data handling encompassing various
tasks which are not yet available in NCL.

I would be happy to clarify any questions.

Regards


On Fri, Feb 8, 2019 at 9:27 AM Barry Lynn <barry.h.lynn at gmail.com> wrote:

> Hello
>
> Now that we read about the obvious advantages (logic and ease of i/o) of
> NCL, is it possible for someone to list why someone would want to switch to
> python?  What does python have for the scientist that NCL does not?
>
> Right now, I read netcdf files, ascii files, csv files, and I made graphs
> and maps, as well as write data using Fortran based programs (WRAPIT).  I
> find this all relatively easy to do (I do it, so it can't be too hard, it's
> just a lot to know!).
>
> Why haven't I switched to Python already.  It sounds like the wave of the
> future!  What's in the wave?
>
> Barry
>
> On Fri, Feb 8, 2019 at 4:05 AM Colin Zarzycki <czarzycki at psu.edu> wrote:
>
>> Like Carl, I have assumed this inevitability, although I have also
>> scientifically "grown up" with NCL over the past decade+ and have many
>> hundreds (probably thousands!) of hours invested in using NCL to answer a
>> vast array of scientific questions. While I think the community's march
>> towards Python has some underappreciated warts (I spent a couple hours
>> updating/rolling back/softlinking mismatched libraries on my personal
>> machine after simply running conda update last week), there is no denying
>> that its growth across, not only Earth, but the broader data sciences and
>> rising stature in university curricula makes it likely the preeminent tool
>> of the next generation of researchers. For better or for worse, adoption
>> and growth of languages are driven by the size/engagement of their userbase
>> instead of optimal practicality for each individual.
>>
>> A couple thoughts after reading the pivot roadmap...
>>
>> A.) Based on my "toe-in-the-water geoscience Python scripting" the last
>> few years, I actually think one of the weaker points in (right now) is that
>> I have yet to find a workflow for weather/climate data I/O where opening,
>> massaging, and writing netCDF datasets (and their corresponding variables,
>> dimensions, etc.) comes with the same ease as in NCL. Some of that is
>> naturally due to tremendous work that has been undertaken in the past to
>> support multiple formats behind the curtain, but this also includes NCL's
>> amazingly useful calendar routines, coordinate subscripting, simple
>> handling of attributes, ability to trivially output a new *.nc file in 3
>> lines, etc. As the roadmap notes on pg. 13-14, currently a host of somewhat
>> fractured Python packages that can lack clean, easy-to-understand syntax
>> are used by the community and (to me) this is a large gap that has remained
>> unfilled even with the proliferation of Python support over the last few
>> years.
>>
>> Therefore, I would argue additional examples* regarding side by side
>> snippets of "writing Python for NCLers" with a specific emphasis on I/O
>> would be incredibly helpful moving forward, particularly with respect to
>> functionality NCL has historically "wrapped" to make life easier (for me,
>> trading off ease-of-use for a little abstraction/processing inefficiency is
>> OK, especially if it means I have time to grab a cup of coffee).
>>
>> B.) Hopefully I am interpreting the goal of NCOMP correctly, but I am
>> also glad to see there is effort to port NCL's functions that are
>> highly-specific to the atmospheric sciences (ex: aforementioned calendar
>> routines, climatological analysis, vertical coordinate interpolation,
>> etc.). This functionality is the workhorse when, for example, I can take
>> data straight from a native model grid and have clean, interpolated,
>> post-processed variables "on my desk" in a couple minutes with ~10 lines of
>> code.
>>
>> -Colin
>>
>> *the first draft/examples on the NCL site from Karin Meier-Fleischer are
>> already incredibly useful!
>>
>> On Thu, Feb 7, 2019 at 4:25 PM Adam Phillips <asphilli at ucar.edu> wrote:
>>
>>> Hi all,
>>> Not everyone got my original post so I am resending this to the cgd-etal
>>> list. My apologies for the duplicate email.
>>>
>>> See Mary Haley's email below for information about the future of NCL.
>>> Quick summary: NCL is *not* going away, NCL graphics will continue to
>>> be developed, and the support structure remains intact. If you have any
>>> questions, feel free to contact myself, Nan Rosenbloom, or Laura Landrum,
>>> as all of us serve on the NCL Advisory Board.
>>> Best,
>>> Adam
>>>
>>> ---------- Forwarded message ---------
>>> From: Mary Haley <haley at ucar.edu>
>>> Date: Wed, Feb 6, 2019 at 6:24 AM
>>> Subject: [ncl-talk] Important announcement regarding the future of NCL
>>> To: Ncl-talk <ncl-talk at ucar.edu>
>>>
>>>
>>> Dear NCL Users,
>>>
>>> This letter is in regard to the future of NCL, following NCAR's decision
>>> to move to Python as the scripting language of choice for future
>>> visualization and analysis software development. Note that this decision
>>> targets new development, leaving existing NCL functionality intact.
>>>
>>> NCAR is committed to supporting data analysis software for atmospheric,
>>> oceanic, and climate science research. However, decreases in budgets and
>>> staff, coupled with the enormous functionality that Python brings to the
>>> earth sciences, has made it difficult to justify continuing new development
>>> on NCL. Python has seen rapid adoption by the earth science community and
>>> duplicates much of NCL's functionality, while adding critical features that
>>> NCL doesn't offer.
>>>
>>> Based on recommendations from NSF, CISL and NCL advisory panels, the
>>> results of the NCL survey, and months of evaluating different strategies
>>> for the future development and support of NCL, NCAR has arrived at these
>>> major decisions, effective immediately:
>>>
>>>    - Python will be adopted as the scripting language platform for
>>>    future visualization and analysis development.
>>>    - NCL's core language and file I/O will be placed into maintenance
>>>    mode.
>>>    - NCL's graphics will have continued development through PyNGL
>>>    <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487407101&sdata=M92NtzKIYOk6B31Me1UB%2BKRGqVPlskQqj2Rz6hBqswo%3D&reserved=0>
>>>    ***.
>>>    - NCL's unique and critical computational routines will be ported to
>>>    an as-yet-to-be-named Python package.
>>>    - PyNIO
>>>    <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu%2FNio.shtml&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487407101&sdata=PBIVL%2BUZ4iXvEiOpZSi%2B3fg5%2F%2FZEyU2ZF%2FZzX2oaPPQ%3D&reserved=0>
>>>    *** will be placed into maintenance mode.
>>>    - Development will continue on WRF-Python
>>>    <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwrf-python.readthedocs.io%2Fen%2Flatest%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487417106&sdata=4AUCC128c90a5ZjLQdg0ttaqhQjiikW84wmwibCXW0s%3D&reserved=0>
>>>    ***.
>>>    - All software, including NCL and PyNIO, will be moved to a more
>>>    open development software platform to allow for continued community
>>>    development.
>>>
>>> *** PyNIO, PyNGL, and WRF-Python are Python modules built on top of NCL
>>> libraries, and are developed and supported by the NCL team.
>>>
>>>
>>> NCAR recognizes the significance of these changes. It will take time for
>>> NCL users to transition to Python, and some users may not want to make the
>>> switch at all. As such, we want to stress that NCL is not going away. NCL
>>> users will be able to download NCL and execute their scripts for the
>>> foreseeable future.
>>>
>>> To help users who want to begin transitioning their graphical NCL
>>> scripts to PyNGL right away, Karin Meier-Fleischer of DKRZ has written a
>>> first draft of an "NCL-to-Python Transition Guide
>>> <http:///www.ncl.ucar.edu/Document/Manuals/NCL_to_Python/Transition_Guide_NCL_PyNGL.pdf>"
>>> accompanied by a suite of NCL and Python examples
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FApplications%2FNCL_to_Python%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487417106&sdata=OUUTrGkALC3USzYa8GP3gSCkTFm72p0ZK7Yz4luKqYA%3D&reserved=0>.
>>> Additionally, we will soon begin converting a subset of the NCL application
>>> examples to Python, using PyNGL
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487427111&sdata=hr3gpDV65YDMYJzWPYEuu3bHnpz6cBa2hL2%2Fr80%2BYXo%3D&reserved=0>
>>> and matplotlib
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmatplotlib.org&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487427111&sdata=RauXYSJZZsxQh7cDSPtss0Woz2PHcMgVBxMwE7qAmi4%3D&reserved=0>,
>>> and will continue to answer questions on the ncl-talk
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487437120&sdata=5NS69uehgGN8K8agtBo9KyXgrFw2fl1RoysYMTuF4SE%3D&reserved=0>
>>> email list, but scaling back in order to start helping with Python
>>> questions.
>>>
>>> For a detailed report and roadmap on the "pivot to Python" decision and
>>> transition plan, please read the "NCL and the Pivot to Python:
>>> Discussion and Roadmap
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FPivot_to_Python%2FNCL_Pivot_to_Python_Report_and_Roadmap.pdf&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487437120&sdata=PlSPR%2BHTiPLXaRtuZq60KHzZ%2FqdZD0Ux89Zh76GN5ps%3D&reserved=0>"
>>> report, which can be found on a special page we created containing other
>>> supporting documents
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FPivot_to_Python%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487447130&sdata=46CI9mvV%2FLNWvIGGD492vddz0PItL4trdPUXk15lR8E%3D&reserved=0>
>>> .
>>>
>>> The NCL team welcomes your input on this decision. We also want to know
>>> if there are other ways we can help ease the transition to Python and
>>> encourage users to become more active contributors through open
>>> development. Please use this GitHub issue
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNCAR%2Fncl%2Fissues%2F64&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487447130&sdata=f6slOq7pKScfUdv2hiW5u5BTXTIDTMQ8e6xluqmTeXA%3D&reserved=0>
>>> to submit questions or comments so we can keep the discussion public.
>>>
>>> *NCL Team:*
>>> John Clyne (acting group head)
>>> Rick Brownrigg
>>> Mary Haley
>>> Kevin Hallock
>>> Bill Ladwig
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=x3LkBTRWrJRHPb0NChym0No6QWBE2gQ5jims0UyUCRA%3D&reserved=0>
>>>
>>>
>>> --
>>> Adam Phillips
>>> Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
>>> www.cgd.ucar.edu/staff/asphilli/
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=1nLuGaRf5AggSZyznMgb9QlSGyntziTbTw1v%2Bt8MEV8%3D&reserved=0>
>>> 303-497-1726
>>>
>>>
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=kCdxy%2FQRpsQirNd4%2Fwt9RTXdtYgyYpr51HllJPA3cw4%3D&reserved=0>
>>>
>>>
>>> --
>>> Adam Phillips
>>> Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
>>> www.cgd.ucar.edu/staff/asphilli/
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487467140&sdata=rOw%2F9v%2Ba1l7c6AseFcLB3DIHEH%2BcNchZdD4UKCaVa6o%3D&reserved=0>
>>> 303-497-1726
>>>
>>>
>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487467140&sdata=77bg13k%2BE1VGDTHk26IBVqIXQTju1hawg3pDFYJg2xs%3D&reserved=0>
>>>
>>
>>
>> --
>> Colin M. Zarzycki, Ph.D.
>> Assistant Professor of Meteorology and Climate Dynamics
>> Department of Meteorology and Atmospheric Science
>> Pennsylvania State University
>> Web: http://www.colinzarzycki.com | Office: +1 (814) 863-6888
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> --
> Barry H. Lynn, Ph.D
> Senior Associate Scientist, Lecturer,
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>


-- 
Puneet Sharma
Ph.D. Research Scholar
Centre for Atmospheric Sciences
IIT Delhi, Hauz Khas
Contact No. 9891582124

https://www.puneetks.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190208/b5ab1b2e/attachment.html>

More information about the ncl-talk mailing list