[ncl-talk] Important announcement regarding the future of NCL

Fri Feb 8 07:48:57 MST 2019

All of those tasks are trivially possible in Python, which is not a bespoke
enough language to require a dedicated site with detailed explanation of
every part of the quite unique language.
Having sunk many hours into trying to fix issues in IFTRAN deep within NCL
(which has to be transpiled to Fortran before being compiled to NCL), I am
greatly looking forward to a Python future.
Keep up the good work guys.
Cheers,
Matthew

On Fri, 8 Feb 2019, 8:00 pm Barry Lynn <barry.h.lynn at gmail.com wrote:

> Hello Puneet:
>
> Thank you for that helpful summary.
>
> There are critical features that I (and others use) in NCL that would be
> "showstoppers" if not present.
>
> For example,
> 1) the calendar functions, inverse and regular.
> 2) The ability to subset strings and obtain dates for example from the
> strings.
> 3) The ability to use attributes to obtain times (e.g., from the GFS data).
> 4)  The ability to call shell commands from within NCL.
> 5) The ability to parse variables in a list, e.g., CSV files read as
> single lines and then parsed into separate variables.
> 6) The ability to "WRAPIT" Fortran programs.
> 7) The ability to subset arrays and conform them.
>
> Perhaps the biggest obstacle is learning how to do all of these things in
> Python, let alone making maps, overlays, panels.
>
> It seems to me that it will take a while to create Python equivalent
> examples as exist online with explicit detail on the NCL pages.
>
> Barry
>
>
> On Fri, Feb 8, 2019 at 11:05 AM Puneet Sharma <puneet.988 at gmail.com>
> wrote:
>
>> Hello all
>>
>> Since I extensively use python for analysis of data generated from model
>> simulations and satellite data, I would like to share some of the strong
>> points of python.
>>
>> 1. Handling of netcdf, hdf and grib data is easy. There are packages such
>> as pyhdf, netcdf4-python and pygrib. Also for .h5 files there is h5py
>> package. CSV data handling is also available.
>>
>> 2. I use the python ecosystem called CDAT developed at LLNL (
>> http://cdat.llnl.gov/) . It has packages such as cdms2 to easily handle
>> netcdf data I/O and its metadata. It is capable of handling large data sets
>> of size of the order of GBs. It also has packages such as cdutil and
>> genutil for performing various statistical analysis of time series data
>> from simulations and observations. Since the data usually has masks and
>> fill values, the functions and procedure available in these packages
>> automatically take care of that. Compared with NCL, CDAT seamlessly handles
>> masking of data using the MV2 package. All of these packages have numpy as
>> their base which is a python package for manipulating numerical data. I
>> have struggled with NCL for masking data which lead me to switch to python.
>>
>> 3. Regridding of data and handling of various grids and interpolation
>> routines which are available in NCL are also available CDAT.
>>
>> 4. String manipulation, data subsetting is very intuitive in python
>> (CDAT). Moreover python code is highly readable since it requires forced
>> indentation which keeps the code clean and readable.
>>
>> 5. The exceptional plotting libraries available in python such as
>> matplotlib, seaborn, bokeh etc have an edge over NCL since very few lines
>> of code can produce complex plots in publication format.
>>
>> 6. Python has intuitive data structures like dictionary, lists and
>> functions like zip, enumerate for iteration which are not available in NCL
>>
>> The only place where I found python to be lagging was the spatial
>> plotting of gridded data. Compared to matplotlib-basemap plotting tool, NCL
>> gives far more clearer and visually appealing spatial plots. But that issue
>> has already been solved through PyNGL package.
>>
>> I would further like to emphasise that together with the points mentioned
>> above, there are many use cases in python which are not available in NCL
>> such as
>>
>> 1. Interoperability with languages such as C, Java, R, FORTRAN since
>> there are wrappers readily available for these languages.
>>
>> 2. Handling url requests for downloading data from different servers.
>>
>> 3. Handling of categorical data using the pandas package.
>>
>> 4. It is easy to call contemporary statistical functions and packages
>> written in R through rpy2 package.
>>
>> I feel that python can become one single platform for performing
>> different tasks related to atmospheric and climate data handling
>> encompassing various tasks which are not yet available in NCL.
>>
>> I would be happy to clarify any questions.
>>
>> Regards
>>
>>
>> On Fri, Feb 8, 2019 at 9:27 AM Barry Lynn <barry.h.lynn at gmail.com> wrote:
>>
>>> Hello
>>>
>>> Now that we read about the obvious advantages (logic and ease of i/o) of
>>> NCL, is it possible for someone to list why someone would want to switch to
>>> python?  What does python have for the scientist that NCL does not?
>>>
>>> Right now, I read netcdf files, ascii files, csv files, and I made
>>> graphs and maps, as well as write data using Fortran based programs
>>> (WRAPIT).  I find this all relatively easy to do (I do it, so it can't be
>>> too hard, it's just a lot to know!).
>>>
>>> Why haven't I switched to Python already.  It sounds like the wave of
>>> the future!  What's in the wave?
>>>
>>> Barry
>>>
>>> On Fri, Feb 8, 2019 at 4:05 AM Colin Zarzycki <czarzycki at psu.edu> wrote:
>>>
>>>> Like Carl, I have assumed this inevitability, although I have also
>>>> scientifically "grown up" with NCL over the past decade+ and have many
>>>> hundreds (probably thousands!) of hours invested in using NCL to answer a
>>>> vast array of scientific questions. While I think the community's march
>>>> towards Python has some underappreciated warts (I spent a couple hours
>>>> updating/rolling back/softlinking mismatched libraries on my personal
>>>> machine after simply running conda update last week), there is no denying
>>>> that its growth across, not only Earth, but the broader data sciences and
>>>> rising stature in university curricula makes it likely the preeminent tool
>>>> of the next generation of researchers. For better or for worse, adoption
>>>> and growth of languages are driven by the size/engagement of their userbase
>>>> instead of optimal practicality for each individual.
>>>>
>>>> A couple thoughts after reading the pivot roadmap...
>>>>
>>>> A.) Based on my "toe-in-the-water geoscience Python scripting" the last
>>>> few years, I actually think one of the weaker points in (right now) is that
>>>> I have yet to find a workflow for weather/climate data I/O where opening,
>>>> massaging, and writing netCDF datasets (and their corresponding variables,
>>>> dimensions, etc.) comes with the same ease as in NCL. Some of that is
>>>> naturally due to tremendous work that has been undertaken in the past to
>>>> support multiple formats behind the curtain, but this also includes NCL's
>>>> amazingly useful calendar routines, coordinate subscripting, simple
>>>> handling of attributes, ability to trivially output a new *.nc file in 3
>>>> lines, etc. As the roadmap notes on pg. 13-14, currently a host of somewhat
>>>> fractured Python packages that can lack clean, easy-to-understand syntax
>>>> are used by the community and (to me) this is a large gap that has remained
>>>> unfilled even with the proliferation of Python support over the last few
>>>> years.
>>>>
>>>> Therefore, I would argue additional examples* regarding side by side
>>>> snippets of "writing Python for NCLers" with a specific emphasis on I/O
>>>> would be incredibly helpful moving forward, particularly with respect to
>>>> functionality NCL has historically "wrapped" to make life easier (for me,
>>>> trading off ease-of-use for a little abstraction/processing inefficiency is
>>>> OK, especially if it means I have time to grab a cup of coffee).
>>>>
>>>> B.) Hopefully I am interpreting the goal of NCOMP correctly, but I am
>>>> also glad to see there is effort to port NCL's functions that are
>>>> highly-specific to the atmospheric sciences (ex: aforementioned calendar
>>>> routines, climatological analysis, vertical coordinate interpolation,
>>>> etc.). This functionality is the workhorse when, for example, I can take
>>>> data straight from a native model grid and have clean, interpolated,
>>>> post-processed variables "on my desk" in a couple minutes with ~10 lines of
>>>> code.
>>>>
>>>> -Colin
>>>>
>>>> *the first draft/examples on the NCL site from Karin Meier-Fleischer
>>>> are already incredibly useful!
>>>>
>>>> On Thu, Feb 7, 2019 at 4:25 PM Adam Phillips <asphilli at ucar.edu> wrote:
>>>>
>>>>> Hi all,
>>>>> Not everyone got my original post so I am resending this to the
>>>>> cgd-etal list. My apologies for the duplicate email.
>>>>>
>>>>> See Mary Haley's email below for information about the future of NCL.
>>>>> Quick summary: NCL is *not* going away, NCL graphics will continue to
>>>>> be developed, and the support structure remains intact. If you have any
>>>>> questions, feel free to contact myself, Nan Rosenbloom, or Laura Landrum,
>>>>> as all of us serve on the NCL Advisory Board.
>>>>> Best,
>>>>> Adam
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: Mary Haley <haley at ucar.edu>
>>>>> Date: Wed, Feb 6, 2019 at 6:24 AM
>>>>> Subject: [ncl-talk] Important announcement regarding the future of NCL
>>>>> To: Ncl-talk <ncl-talk at ucar.edu>
>>>>>
>>>>>
>>>>> Dear NCL Users,
>>>>>
>>>>> This letter is in regard to the future of NCL, following NCAR's
>>>>> decision to move to Python as the scripting language of choice for future
>>>>> visualization and analysis software development. Note that this decision
>>>>> targets new development, leaving existing NCL functionality intact.
>>>>>
>>>>> NCAR is committed to supporting data analysis software for
>>>>> atmospheric, oceanic, and climate science research. However, decreases in
>>>>> budgets and staff, coupled with the enormous functionality that Python
>>>>> brings to the earth sciences, has made it difficult to justify continuing
>>>>> new development on NCL. Python has seen rapid adoption by the earth science
>>>>> community and duplicates much of NCL's functionality, while adding critical
>>>>> features that NCL doesn't offer.
>>>>>
>>>>> Based on recommendations from NSF, CISL and NCL advisory panels, the
>>>>> results of the NCL survey, and months of evaluating different strategies
>>>>> for the future development and support of NCL, NCAR has arrived at these
>>>>> major decisions, effective immediately:
>>>>>
>>>>>    - Python will be adopted as the scripting language platform for
>>>>>    future visualization and analysis development.
>>>>>    - NCL's core language and file I/O will be placed into maintenance
>>>>>    mode.
>>>>>    - NCL's graphics will have continued development through PyNGL
>>>>>    <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487407101&sdata=M92NtzKIYOk6B31Me1UB%2BKRGqVPlskQqj2Rz6hBqswo%3D&reserved=0>
>>>>>    ***.
>>>>>    - NCL's unique and critical computational routines will be ported
>>>>>    to an as-yet-to-be-named Python package.
>>>>>    - PyNIO
>>>>>    <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu%2FNio.shtml&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487407101&sdata=PBIVL%2BUZ4iXvEiOpZSi%2B3fg5%2F%2FZEyU2ZF%2FZzX2oaPPQ%3D&reserved=0>
>>>>>    *** will be placed into maintenance mode.
>>>>>    - Development will continue on WRF-Python
>>>>>    <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwrf-python.readthedocs.io%2Fen%2Flatest%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487417106&sdata=4AUCC128c90a5ZjLQdg0ttaqhQjiikW84wmwibCXW0s%3D&reserved=0>
>>>>>    ***.
>>>>>    - All software, including NCL and PyNIO, will be moved to a more
>>>>>    open development software platform to allow for continued community
>>>>>    development.
>>>>>
>>>>> *** PyNIO, PyNGL, and WRF-Python are Python modules built on top of
>>>>> NCL libraries, and are developed and supported by the NCL team.
>>>>>
>>>>>
>>>>> NCAR recognizes the significance of these changes. It will take time
>>>>> for NCL users to transition to Python, and some users may not want to make
>>>>> the switch at all. As such, we want to stress that NCL is not going away.
>>>>> NCL users will be able to download NCL and execute their scripts for the
>>>>> foreseeable future.
>>>>>
>>>>> To help users who want to begin transitioning their graphical NCL
>>>>> scripts to PyNGL right away, Karin Meier-Fleischer of DKRZ has written a
>>>>> first draft of an "NCL-to-Python Transition Guide
>>>>> <http:///www.ncl.ucar.edu/Document/Manuals/NCL_to_Python/Transition_Guide_NCL_PyNGL.pdf>"
>>>>> accompanied by a suite of NCL and Python examples
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FApplications%2FNCL_to_Python%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487417106&sdata=OUUTrGkALC3USzYa8GP3gSCkTFm72p0ZK7Yz4luKqYA%3D&reserved=0>.
>>>>> Additionally, we will soon begin converting a subset of the NCL application
>>>>> examples to Python, using PyNGL
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pyngl.ucar.edu%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487427111&sdata=hr3gpDV65YDMYJzWPYEuu3bHnpz6cBa2hL2%2Fr80%2BYXo%3D&reserved=0>
>>>>> and matplotlib
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmatplotlib.org&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487427111&sdata=RauXYSJZZsxQh7cDSPtss0Woz2PHcMgVBxMwE7qAmi4%3D&reserved=0>,
>>>>> and will continue to answer questions on the ncl-talk
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487437120&sdata=5NS69uehgGN8K8agtBo9KyXgrFw2fl1RoysYMTuF4SE%3D&reserved=0>
>>>>> email list, but scaling back in order to start helping with Python
>>>>> questions.
>>>>>
>>>>> For a detailed report and roadmap on the "pivot to Python" decision
>>>>> and transition plan, please read the "NCL and the Pivot to Python:
>>>>> Discussion and Roadmap
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FPivot_to_Python%2FNCL_Pivot_to_Python_Report_and_Roadmap.pdf&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487437120&sdata=PlSPR%2BHTiPLXaRtuZq60KHzZ%2FqdZD0Ux89Zh76GN5ps%3D&reserved=0>"
>>>>> report, which can be found on a special page we created containing other
>>>>> supporting documents
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FPivot_to_Python%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487447130&sdata=46CI9mvV%2FLNWvIGGD492vddz0PItL4trdPUXk15lR8E%3D&reserved=0>
>>>>> .
>>>>>
>>>>> The NCL team welcomes your input on this decision. We also want to
>>>>> know if there are other ways we can help ease the transition to Python and
>>>>> encourage users to become more active contributors through open
>>>>> development. Please use this GitHub issue
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNCAR%2Fncl%2Fissues%2F64&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487447130&sdata=f6slOq7pKScfUdv2hiW5u5BTXTIDTMQ8e6xluqmTeXA%3D&reserved=0>
>>>>> to submit questions or comments so we can keep the discussion public.
>>>>>
>>>>> *NCL Team:*
>>>>> John Clyne (acting group head)
>>>>> Rick Brownrigg
>>>>> Mary Haley
>>>>> Kevin Hallock
>>>>> Bill Ladwig
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=x3LkBTRWrJRHPb0NChym0No6QWBE2gQ5jims0UyUCRA%3D&reserved=0>
>>>>>
>>>>>
>>>>> --
>>>>> Adam Phillips
>>>>> Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
>>>>> www.cgd.ucar.edu/staff/asphilli/
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=1nLuGaRf5AggSZyznMgb9QlSGyntziTbTw1v%2Bt8MEV8%3D&reserved=0>
>>>>> 303-497-1726
>>>>>
>>>>>
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487457135&sdata=kCdxy%2FQRpsQirNd4%2Fwt9RTXdtYgyYpr51HllJPA3cw4%3D&reserved=0>
>>>>>
>>>>>
>>>>> --
>>>>> Adam Phillips
>>>>> Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
>>>>> www.cgd.ucar.edu/staff/asphilli/
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli%2F&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487467140&sdata=rOw%2F9v%2Ba1l7c6AseFcLB3DIHEH%2BcNchZdD4UKCaVa6o%3D&reserved=0>
>>>>> 303-497-1726
>>>>>
>>>>>
>>>>> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.ucar.edu%2Fstaff%2Fasphilli&data=02%7C01%7Ccmz5202%40psu.edu%7C475b847b8a1a4321b2a808d68d42d347%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636851715487467140&sdata=77bg13k%2BE1VGDTHk26IBVqIXQTju1hawg3pDFYJg2xs%3D&reserved=0>
>>>>>
>>>>
>>>>
>>>> --
>>>> Colin M. Zarzycki, Ph.D.
>>>> Assistant Professor of Meteorology and Climate Dynamics
>>>> Department of Meteorology and Atmospheric Science
>>>> Pennsylvania State University
>>>> Web: http://www.colinzarzycki.com | Office: +1 (814) 863-6888
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>
>>>
>>> --
>>> Barry H. Lynn, Ph.D
>>> Senior Associate Scientist, Lecturer,
>>> The Institute of the Earth Science,
>>> The Hebrew University of Jerusalem,
>>> Givat Ram, Jerusalem 91904, Israel
>>> Tel: 972 547 231 170
>>> Fax: (972)-25662581
>>>
>>> C.E.O, Weather It Is, LTD
>>> Weather and Climate Focus
>>> http://weather-it-is.com
>>> Jerusalem, Israel
>>> Local: 02 930 9525
>>> Cell: 054 7 231 170
>>> Int-IS: x972 2 930 9525
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>>
>> --
>> Puneet Sharma
>> Ph.D. Research Scholar
>> Centre for Atmospheric Sciences
>> IIT Delhi, Hauz Khas
>> Contact No. 9891582124
>>
>> https://www.puneetks.com
>>
>
>
> --
> Barry H. Lynn, Ph.D
> Senior Associate Scientist, Lecturer,
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190208/ce4f2dd2/attachment-0001.html>