[ncl-talk] Important announcement regarding the future of NCL

Thu Feb 7 11:52:35 MST 2019

Hi Barry,
Just to add to what Toni said: You will be able to continue to run your
existing NCL scripts, there will be support through github and ncl-talk
(albeit there may be less responses from the developers), and NCL graphics
(in concert with pyNGL) will continue to be developed. I think some points
were left unsaid in the original letter, as future staffing and future
budgets are unknown.
Adam

On Thu, Feb 7, 2019 at 5:46 AM Toni Klemm <toni-klemm at tamu.edu> wrote:

> Barry et al.,
>
> I don’t think those programs have to be rewritten. My understanding is the
> NCL version on your system will keep working, it just won’t get future
> updates from the NCL team, and maybe less user support. Like many, I have
> dozens of NCL scripts, but they will keep working. For the future though,
> NCL users might be smart to transitioning to R or Python. FOR R and R
> Studio there are NCL packages, basically add-ons that allow you to process
> netCDF data in R. The basics of Python you can learn for example through Software
> Carpentry <https://software-carpentry.org>.
>
> I hope that helps.
>
> Toni
>
>
> *Toni Klemm**, Ph.D.*
> Postdoctoral Research Associate
> Department of Ecosystem Science and Management
> College of Agriculture and Life Sciences
> Texas A&M University, College Station, TX
> Contributor to the Early Career Climate Forum <https://www.eccforum.org/>
> www.toni-klemm.de | @toniklemm <https://twitter.com/ToniKlemm>
>
>
>
>
>
>
> On Feb 6, 2019, at 10:06 PM, Barry Lynn <barry.h.lynn at gmail.com> wrote:
>
> Hi Mary:
>
> Thanks.
>
> A most pertinent question: how hard will it be for someone who has worked
> hard to "know" NCL to transition to Python.
>
> Also, keep in mind that I (and others) have written 10s of programs in NC,
> and these would need to be rewritten.
>
> Barry
>
> On Thu, Feb 7, 2019 at 12:59 AM Mary Haley <haley at ucar.edu> wrote:
>
>> Hi Barry,
>>
>> I encourage folks to read the report, as it covers in detail why the
>> decision to transition to Python, and what Python brings to the table:
>>
>>
>> http://www.ncl.ucar.edu/Document/Pivot_to_Python/NCL_Pivot_to_Python_Report_and_Roadmap.pdf
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncl.ucar.edu_Document_Pivot-5Fto-5FPython_NCL-5FPivot-5Fto-5FPython-5FReport-5Fand-5FRoadmap.pdf&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=yz-Fa4nJPdDM4mRUn0H5NfqdFWgASciTiMBeSdNjHSg&e=>
>>
>> There's a section "Why Python" (starts on page 5) that explains some of
>> the reasoning behind this decision.
>>
>> Here's the pertinent part of that section:
>>
>> *Why Python?*
>> Python has gained widespread acceptance by universities and research
>> organizations around the world and is being adopted as the programming
>> language of choice for scientific computing. This is evidenced by several
>> factors: 1) the availability of quality scientific Python modules via the
>> SciPy ecosystem, 2) the continued and growing popularity of the annual
>> SciPy conference, now in its 17th year, 3) the availability of books on
>> Python for scientists, and 4) the increasing number of scientific graduate
>> students who are learning Python in college as an open source alternative
>> to other non-free software like IDL and MATLAB. In September 2018—for the
>> first time in history—Python entered the TIOBE index top 3 (www.tiobe.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tiobe.com&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=NPg_3kgLJfrAEc8Q3bF6ZufzwtozXd-5FCPtL3LKB9g&e=>),
>> a measure of popularity of programming languages based on search engine
>> results.
>>
>> Python has picked up rapid steam in the geoscientific community as well.
>> For the last eight years the American Meteorological Society Annual Meeting
>> has hosted a popular and well-attended symposium on the “Advances in
>> Modeling and Analysis Using Python”. NCAR is a major partner in the Pangeo (
>> pangeo.io
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__pangeo.io&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=_BI6R6g4EqWUuigwl0IZHXnwHXFSOz47FSLhJSZdYMI&e=>)
>> community, an NSF EarthCube funded effort that provides an “open source
>> scientific Python ecosystem for ocean / atmosphere / land / climate
>> science” and is focused on providing tools and support for handling
>> petabyte-scale datasets on HPC and cloud platforms. There are hundreds of
>> scientific Python modules that provide domain-specific functionality for
>> reading/writing data, computational analyses, and visualization. The
>> benefit of these individual packages is that they are usually specialized
>> for a specific domain or class of problems, thus filling a critical need
>> that a more general-purpose language cannot.
>>
>> The Python language itself provides rich language features that NCL does
>> not have, including optional arguments, a robust interactive interface,
>> generators, exception handling, and built-in debugging and testing. The
>> Python community has a rapidly growing base of scientific software
>> developers that are able to address the growing needs of the geoscientific
>> community much faster than we can in the areas of scalability, interfaces
>> to other languages like R for statistical calculations, and support for a
>> wider range of complex data formats. By replacing the NCL language with the
>> Python language, the NCL user base will instantly gain access to these
>> features, and we will be able to benefit from the already vibrant and
>> active open development Python community. Python itself has been open
>> developed since October 2000.
>> Last but not least, it is becoming harder to hire developers who want to
>> work on a programming language with a narrow focus, versus a highly visible
>> and mainstream language like Python.
>>
>> I hope this addresses your questions.
>>
>> --Mary
>>
>>
>> On Wed, Feb 6, 2019 at 7:29 AM Barry Lynn <barry.h.lynn at gmail.com> wrote:
>>
>>> Hello Mary:
>>>
>>> Could you please help us understand what critical features are missing
>>> from NCL but present in python so that we better understand why we should
>>> switch.
>>>
>>> Thank you,
>>>
>>> Barry
>>>
>>> On Wed, Feb 6, 2019 at 3:23 PM Mary Haley <haley at ucar.edu> wrote:
>>>
>>>> Dear NCL Users,
>>>>
>>>> This letter is in regard to the future of NCL, following NCAR's
>>>> decision to move to Python as the scripting language of choice for future
>>>> visualization and analysis software development. Note that this decision
>>>> targets new development, leaving existing NCL functionality intact.
>>>>
>>>> NCAR is committed to supporting data analysis software for atmospheric,
>>>> oceanic, and climate science research. However, decreases in budgets and
>>>> staff, coupled with the enormous functionality that Python brings to the
>>>> earth sciences, has made it difficult to justify continuing new development
>>>> on NCL. Python has seen rapid adoption by the earth science community and
>>>> duplicates much of NCL's functionality, while adding critical features that
>>>> NCL doesn't offer.
>>>>
>>>> Based on recommendations from NSF, CISL and NCL advisory panels, the
>>>> results of the NCL survey, and months of evaluating different strategies
>>>> for the future development and support of NCL, NCAR has arrived at these
>>>> major decisions, effective immediately:
>>>>
>>>>    - Python will be adopted as the scripting language platform for
>>>>    future visualization and analysis development.
>>>>    - NCL's core language and file I/O will be placed into maintenance
>>>>    mode.
>>>>    - NCL's graphics will have continued development through PyNGL
>>>>    <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pyngl.ucar.edu&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=XjUKpVjTosG1QXg8tPanvc1VWz8lgMBhRmHlr54D4ww&e=>
>>>>    ***.
>>>>    - NCL's unique and critical computational routines will be ported
>>>>    to an as-yet-to-be-named Python package.
>>>>    - PyNIO
>>>>    <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pyngl.ucar.edu_Nio.shtml&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=eYYQhhwmjnYoMGlTw_DrlkxnLRKaZf9qt4qZ7oIl4b0&e=>
>>>>    *** will be placed into maintenance mode.
>>>>    - Development will continue on WRF-Python
>>>>    <https://urldefense.proofpoint.com/v2/url?u=https-3A__wrf-2Dpython.readthedocs.io_en_latest_&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=SRZaRV6rql3QcJHAIFeRtPaJNotCWR_YFuywSlAxDAk&e=>
>>>>    ***.
>>>>    - All software, including NCL and PyNIO, will be moved to a more
>>>>    open development software platform to allow for continued community
>>>>    development.
>>>>
>>>> *** PyNIO, PyNGL, and WRF-Python are Python modules built on top of
>>>> NCL libraries, and are developed and supported by the NCL team.
>>>>
>>>>
>>>> NCAR recognizes the significance of these changes. It will take time
>>>> for NCL users to transition to Python, and some users may not want to make
>>>> the switch at all. As such, we want to stress that NCL is not going away.
>>>> NCL users will be able to download NCL and execute their scripts for the
>>>> foreseeable future.
>>>>
>>>> To help users who want to begin transitioning their graphical NCL
>>>> scripts to PyNGL right away, Karin Meier-Fleischer of DKRZ has written a
>>>> first draft of an "NCL-to-Python Transition Guide
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A___www.ncl.ucar.edu_Document_Manuals_NCL-5Fto-5FPython_Transition-5FGuide-5FNCL-5FPyNGL.pdf&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=a3sboqqRFVThkjcuPudM_0tLy_Bo0eDIIlgK5OASFwo&e=>"
>>>> accompanied by a suite of NCL and Python examples
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncl.ucar.edu_Applications_NCL-5Fto-5FPython_&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=LEBpgpbmvF5EKsuZVq-Oik-e3WTT26MmT6Zy0FCeb40&e=>.
>>>> Additionally, we will soon begin converting a subset of the NCL application
>>>> examples to Python, using PyNGL
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pyngl.ucar.edu_&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=VZDbL9EjlrCHncOZb_tT6Q8ifLAfXDt5XR1_thUOj9M&e=>
>>>>  and matplotlib
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__matplotlib.org&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=houOTRZi2bLqejxNKntfvEpoJj3nsFfjc_keFU-o4NE&e=>,
>>>> and will continue to answer questions on the ncl-talk
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=TEqpggwoOULSGTXA0ktYGBrVl79eAsBtpqWNPVyiGe4&e=>
>>>>  email list, but scaling back in order to start helping with Python
>>>> questions.
>>>>
>>>> For a detailed report and roadmap on the "pivot to Python" decision and
>>>> transition plan, please read the "NCL and the Pivot to Python:
>>>> Discussion and Roadmap
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncl.ucar.edu_Document_Pivot-5Fto-5FPython_NCL-5FPivot-5Fto-5FPython-5FReport-5Fand-5FRoadmap.pdf&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=yz-Fa4nJPdDM4mRUn0H5NfqdFWgASciTiMBeSdNjHSg&e=>"
>>>> report, which can be found on a special page we created containing other
>>>> supporting documents
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncl.ucar.edu_Document_Pivot-5Fto-5FPython_&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=SRVKQ5isbDnyHvL4wzQeQEpJthMV6M3e0w01qJF9S50&e=>
>>>> .
>>>>
>>>> The NCL team welcomes your input on this decision. We also want to know
>>>> if there are other ways we can help ease the transition to Python and
>>>> encourage users to become more active contributors through open
>>>> development. Please use this GitHub issue
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_NCAR_ncl_issues_64&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=2dmElBi9C2oq4us6lvx7AJ7C9I8qbH8wiV1PzKs9u3k&e=>
>>>>  to submit questions or comments so we can keep the discussion public.
>>>>
>>>> *NCL Team:*
>>>> John Clyne (acting group head)
>>>> Rick Brownrigg
>>>> Mary Haley
>>>> Kevin Hallock
>>>> Bill Ladwig
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=TEqpggwoOULSGTXA0ktYGBrVl79eAsBtpqWNPVyiGe4&e=>
>>>>
>>>
>>>
>>> --
>>> Barry H. Lynn, Ph.D
>>> Senior Associate Scientist, Lecturer,
>>> The Institute of the Earth Science,
>>> The Hebrew University of Jerusalem,
>>> Givat Ram, Jerusalem 91904, Israel
>>> Tel: 972 547 231 170
>>> Fax: (972)-25662581
>>>
>>> C.E.O, Weather It Is, LTD
>>> Weather and Climate Focus
>>> http://weather-it-is.com
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__weather-2Dit-2Dis.com&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=Uv0CFIV_YJOKuyMuPo3lRqFEZpZiEHfHOPwrZRcPADU&e=>
>>> Jerusalem, Israel
>>> Local: 02 930 9525
>>> Cell: 054 7 231 170
>>> Int-IS: x972 2 930 9525
>>>
>>>
>
> --
> Barry H. Lynn, Ph.D
> Senior Associate Scientist, Lecturer,
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__weather-2Dit-2Dis.com&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=iOo39tnO8aeORpG5noJIhwsDxjOcD1HnwHFp0MTvde4&s=Uv0CFIV_YJOKuyMuPo3lRqFEZpZiEHfHOPwrZRcPADU&e=>
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Adam Phillips
Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
www.cgd.ucar.edu/staff/asphilli/   303-497-1726

<http://www.cgd.ucar.edu/staff/asphilli>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190207/d558bde6/attachment.html>