[ncl-talk] Important announcement regarding the future of NCL

Sat Feb 9 09:04:12 MST 2019

Dear NCL team:

                          My question about the transition to Python is that to what extent, 
               the present data processing/visualizing strategies offered by NCL will be inherited 
               by the PyNGL and a yet-to-be-named Python Package. These well-selected and high-quality strategies
               are unique for earth science and have been familiarized by a great lot of users. It seems not likely that the 
               existing Python plotting packages (e.g., matplotlib) can easily compete. At least for now, it seems to me
               that PyNGL still lags behind of the latest NCL interpreter in terms of resources control.
--
NCL's graphics will have continued development through PyNGL*.
NCL's unique and critical computational routines will be ported to an as-yet-to-be-named Python package.
 --
Best Regard
Cheung

At 2019-02-09 00:53:38, "Mary Haley" <haley at ucar.edu> wrote:

Hi Gus,

I had started to respond to each of your individual points in your original email, but realized I will likely be doing this all day long if I do this for every email that comes in, and it may not have the desired effect.

Please know that all these comments are being read by NCAR upper management and they are taking serious note of them.

I was asked to address one of your comments below:

With all due respect to my friend Mary, the notion proposed in the announcement email that
"NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future is hard to believe.

Once the Linux distributions update the system and other libraries that NCL uses, the NCL executables will stop working. And this won't take much long. Compilation from source will become even harder than it is today, because if the original code is no longer maintained, incompatibilities with the system and other libraries will creep in, and won't have how to be resolved.

I'm not an expert on industry trends, but in my simple view, as long as there are C and Fortran compilers and UNIX systems, NCL should continue to build on those systems. I've had enough experience with these things that I can usually get stuff up and running in a few hours if there's a hiccup. There are also other folks who are experts in this and have helped us tackle NCL builds in the past. I don't see UNIX systems going away right away, so IMHO, NCL should continue to exist and work right along with them.

I personally have a vested interest in making sure NCL lives on for awhile, because I use it at home quite extensively. And yes, I am trying to pivot to Python myself on some things that I do at home, but if I need to do something quickly that involves graphics, then I may fall back to NCL because it is what I know best right now.

Kevin Hallock has created a nice script as part of our conda-based installation that uses conda to install all the required dependencies that NCL has, and then builds NCL from source code. We haven't made this script public yet, but this script will greatly simplify the NCL build for the average user.

Thanks for taking the time to provide your feedback.

--Mary

On Thu, Feb 7, 2019 at 5:17 PM Gus Correa <gus at ldeo.columbia.edu> wrote:

Hi Carl, List

I am not against progress (let alone evolution!) but the Python hype folks have to present arguments that are more
sound than just "everybody is using it".
That is fashion, that is not software design.

NCL is a very well designed, well implemented, concise, and user friendly data analysis and visualization language and package,

besides being well documented and supported.
That is what really counts from the user standpoint.

Can the explosive myriad of Python modules, with all their undocumented and incompatible versions,
claim the same?

Such hype and the same explosion of modules,

also happened to Perl years back.
It put the language in a fast rotating centrifuge, and Perl nearly died from it.

Can the NCL developers guarantee that this transition to Python will keep the tight consistency,

clear user interface, and beloved language features that NCL has, once it starts depending
on the Python module diffuse "ecosystem"?

***

You get Python 2.7 with most Linux distributions today. It is called simply python. 

To get Python 3 you need to install it separately, and it will be called ... well ... python3, because the underlying
OS (Linux) underpinnings depend deeply on python (2.7) to work, and python 2.7 is incompatible with python3 (3.4, 3.6 or whatever).
Now, installing Python 3 this way doesn't guarantee that you will have the latest greatest Python packages.
Then people go for Anaconda.com to get anaconda, miniconda, etc.
And there the versions abound, new ones coming by the day, with little regard for backward compatibility, documentation,

consistency, correctness, etc.
Did they agree on how to represent complex numbers already or not? On all packages?

Do we need to put ourselves (and science-oriented users that are not professional programmers or system administrators)

in this situation to continue to use NCL?

A language is hard to learn, and once learned it is treasured by those who become fluent.
Can you imagine if somebody decides that we all should switch from English to Mandarin Chinese,
because the majority rules?
Would you be happy?

***

With all due respect to my friend Mary, the notion proposed in the announcement email that

"NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future."
is hard to believe.

Once the Linux distributions update the system and other libraries that NCL uses, the NCL executables will stop working.
And this won't take much long. Compilation from source will become even harder than it is today,

because if the original code is no longer maintained, incompatibilities with the system and other libraries will creep in,
and won't have how to be resolved.

***

Likewise, how can this transition be put forward because:

"Last but not least, it is becoming harder to hire developers who want to work on a programming language with a narrow focus, versus a highly visible and mainstream language like Python."

This simply puts the developers first, and the users last!
How can a project manager or managing group establish this type of priority?

I can't believe this is part of the argument supporting this transition.
I can't believe the programmer job market has such a surplus of job offers with the good salaries, benefits, and opportunities of growth

offered by NCAR, that the job candidates to NCAR jobs are in a position to dictate the projects they want to work

on even before they take the job.
Years back there was a big wave of programmers taking jobs to fix the Y2K problem and other things and to

program in COBOL language.
Geez, how cheesy and outdated! ... a Python cool kid would say (if he only knew what COBOL is).

Yet the jobs were taken, the work was done, and proceeded.
It is not the language that dictates the quality of programming, it is the programmer quality that dictates it.

Actually, COBOL is alive and kicking:
https://devops.com/the-beauty-of-the-cobol-programming-language-v2/

NCL is not worse than COBOL.

It also deserves better.

***

Thank you,

Gus Correa

On Thu, Feb 7, 2019 at 3:08 PM Carl Schreck <cjschrec at ncsu.edu> wrote:

Just to chime in... I'm second to none in my love for NCL. It's the only language I use on a daily basis and has been so for more than a decade.

That said, I've known for a long time that python would probably be the last language that I would ever learn. It's the first time since Fortran that scientists, engineers, programmers, and web developers are all using the same language. That confluence will carry it for decades. We can't overstate the value of being able to tap into such a large community of users.

My gratitude for the NCL team is beyond words. Like many of you, my learning curve and conversion to python will be slow and painful. I will greatly miss how seamlessly NCL reads netcdf & grib, and esp. it's { } coordinate subsetting. But as others have said, all my legacy code will work during the transition and beyond.

It's a sad day, but I'm confident that NCAR made the right decision for our community. 

Carl

On Thu, Feb 7, 2019 at 2:45 PM Barry Lynn <barry.h.lynn at gmail.com> wrote:

Hi Gus:

I think that was well said.

I, for one, was quite proud of myself for becoming proficient in NCL -- but especially so because it is so logically designed and so useful.  It is a a very flexible program for  both plotting data and for extracting data from various data sets, and even has a way to run various batch commands from within NCL. In other words, it is a programming (graphics) language that I and others can do really useful scientific and scientific applications. 

I wonder if the decision makers asked how many NCL users want to switch to python!

Barry

On Thu, Feb 7, 2019 at 9:25 PM Gus Correa <gus at ldeo.columbia.edu> wrote:

Hi Mary, hi List and All.

I share Barry's concerns for the same reasons and more,
which I expressed in the NCL survey months ago:

0) Excellence of NCL both as a data processing and data visualization tool.
[Why should we phase out a winner?]

1) Large code base of programs written in NCL by many users.
[Would you phase out Fortran just because it is old-fashioned?

Replace it by something cool and trendy, just because it is cool and trendy?]

2) Has the decision considered that many, if not most, NCL users are not

expert programmers, already invested a lot of time to become proficient in NCL,
and do not have the time or interest to learn another language to do the same things that they already do?
The NCL user main goal is science, not programming.

3) Rapid release of Python modules (often unstable, low QC, often buggy, poorly documented, inconsistent versions, no
commitment to back compatibility, etc),

making it difficult to keep a sane code base, and requiring users to enter the nitty-gritty details of
versioning and version control, instead of concentrating in using the tool for data analysis and science (which is their goal).

4) Possible dependency on Python relases distributed and controlled by a commercial
enterprise (Anaconda.com: anaconda, miniconda and other constrictors).

For example, was the decision years back by UCAR to base the netCDF-4 structure
and functionality on the (commercial entrerpise controlled) HDF-5 framework a wise one?
A few months back the HDF-5 group announced that it will no longer support the

community legacy versions, and concentrate on their commercial latest greatest version
and on the company profitability. 

I probably could list more items, but right now I am really sad to learn that this is NCAR's decision.
In my view a very poor one.

Thank you,
Gus Correa

On Thu, Feb 7, 2019 at 1:53 PM Adam Phillips <asphilli at ucar.edu> wrote:

Hi Barry,
Just to add to what Toni said: You will be able to continue to run your existing NCL scripts, there will be support through github and ncl-talk (albeit there may be less responses from the developers), and NCL graphics (in concert with pyNGL) will continue to be developed. I think some points were left unsaid in the original letter, as future staffing and future budgets are unknown.
Adam 

On Thu, Feb 7, 2019 at 5:46 AM Toni Klemm <toni-klemm at tamu.edu> wrote:

Barry et al.,

I don’t think those programs have to be rewritten. My understanding is the NCL version on your system will keep working, it just won’t get future updates from the NCL team, and maybe less user support. Like many, I have dozens of NCL scripts, but they will keep working. For the future though, NCL users might be smart to transitioning to R or Python. FOR R and R Studio there are NCL packages, basically add-ons that allow you to process netCDF data in R. The basics of Python you can learn for example through Software Carpentry.

I hope that helps.

Toni

Toni Klemm, Ph.D.
Postdoctoral Research Associate
Department of Ecosystem Science and Management
College of Agriculture and Life Sciences
Texas A&M University, College Station, TX
Contributor to the Early Career Climate Forum
www.toni-klemm.de | @toniklemm

On Feb 6, 2019, at 10:06 PM, Barry Lynn <barry.h.lynn at gmail.com> wrote:

Hi Mary:

Thanks.

A most pertinent question: how hard will it be for someone who has worked hard to "know" NCL to transition to Python.

Also, keep in mind that I (and others) have written 10s of programs in NC, and these would need to be rewritten.

Barry

On Thu, Feb 7, 2019 at 12:59 AM Mary Haley <haley at ucar.edu> wrote:

Hi Barry,

I encourage folks to read the report, as it covers in detail why the decision to transition to Python, and what Python brings to the table:

http://www.ncl.ucar.edu/Document/Pivot_to_Python/NCL_Pivot_to_Python_Report_and_Roadmap.pdf

There's a section "Why Python" (starts on page 5) that explains some of the reasoning behind this decision.

Here's the pertinent part of that section:

Why Python?
Python has gained widespread acceptance by universities and research organizations around the world and is being adopted as the programming language of choice for scientific computing. This is evidenced by several factors: 1) the availability of quality scientific Python modules via the SciPy ecosystem, 2) the continued and growing popularity of the annual SciPy conference, now in its 17th year, 3) the availability of books on Python for scientists, and 4) the increasing number of scientific graduate students who are learning Python in college as an open source alternative to other non-free software like IDL and MATLAB. In September 2018—for the first time in history—Python entered the TIOBE index top 3 (www.tiobe.com), a measure of popularity of programming languages based on search engine results.

Python has picked up rapid steam in the geoscientific community as well. For the last eight years the American Meteorological Society Annual Meeting has hosted a popular and well-attended symposium on the “Advances in Modeling and Analysis Using Python”. NCAR is a major partner in the Pangeo (pangeo.io) community, an NSF EarthCube funded effort that provides an “open source scientific Python ecosystem for ocean / atmosphere / land / climate science” and is focused on providing tools and support for handling petabyte-scale datasets on HPC and cloud platforms. There are hundreds of scientific Python modules that provide domain-specific functionality for reading/writing data, computational analyses, and visualization. The benefit of these individual packages is that they are usually specialized for a specific domain or class of problems, thus filling a critical need that a more general-purpose language cannot.

The Python language itself provides rich language features that NCL does not have, including optional arguments, a robust interactive interface, generators, exception handling, and built-in debugging and testing. The Python community has a rapidly growing base of scientific software developers that are able to address the growing needs of the geoscientific community much faster than we can in the areas of scalability, interfaces to other languages like R for statistical calculations, and support for a wider range of complex data formats. By replacing the NCL language with the Python language, the NCL user base will instantly gain access to these features, and we will be able to benefit from the already vibrant and active open development Python community. Python itself has been open developed since October 2000.
Last but not least, it is becoming harder to hire developers who want to work on a programming language with a narrow focus, versus a highly visible and mainstream language like Python.

I hope this addresses your questions.

--Mary

On Wed, Feb 6, 2019 at 7:29 AM Barry Lynn <barry.h.lynn at gmail.com> wrote:

Hello Mary:

Could you please help us understand what critical features are missing from NCL but present in python so that we better understand why we should switch.

Thank you,

Barry

On Wed, Feb 6, 2019 at 3:23 PM Mary Haley <haley at ucar.edu> wrote:

Dear NCL Users,

This letter is in regard to the future of NCL, following NCAR's decision to move to Python as the scripting language of choice for future visualization and analysis software development. Note that this decision targets new development, leaving existing NCL functionality intact.

NCAR is committed to supporting data analysis software for atmospheric, oceanic, and climate science research. However, decreases in budgets and staff, coupled with the enormous functionality that Python brings to the earth sciences, has made it difficult to justify continuing new development on NCL. Python has seen rapid adoption by the earth science community and duplicates much of NCL's functionality, while adding critical features that NCL doesn't offer.

Based on recommendations from NSF, CISL and NCL advisory panels, the results of the NCL survey, and months of evaluating different strategies for the future development and support of NCL, NCAR has arrived at these major decisions, effective immediately:
Python will be adopted as the scripting language platform for future visualization and analysis development.

NCL's core language and file I/O will be placed into maintenance mode.
NCL's graphics will have continued development through PyNGL*.

NCL's unique and critical computational routines will be ported to an as-yet-to-be-named Python package.

PyNIO* will be placed into maintenance mode.

Development will continue on WRF-Python*.
All software, including NCL and PyNIO, will be moved to a more open development software platform to allow for continued community development.
* PyNIO, PyNGL, and WRF-Python are Python modules built on top of NCL libraries, and are developed and supported by the NCL team.

NCAR recognizes the significance of these changes. It will take time for NCL users to transition to Python, and some users may not want to make the switch at all. As such, we want to stress that NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future.

To help users who want to begin transitioning their graphical NCL scripts to PyNGL right away, Karin Meier-Fleischer of DKRZ has written a first draft of an "NCL-to-Python Transition Guide" accompanied by a suite of NCL and Python examples.  Additionally, we will soon begin converting a subset of the NCL application examples to Python, using PyNGL and matplotlib, and will continue to answer questions on the ncl-talk email list, but scaling back in order to start helping with Python questions.

For a detailed report and roadmap on the "pivot to Python" decision and transition plan, please read the "NCL and the Pivot to Python: Discussion and Roadmap" report, which can be found on a special page we created containing other supporting documents.

The NCL team welcomes your input on this decision. We also want to know if there are other ways we can help ease the transition to Python and encourage users to become more active contributors through open development. Please use this GitHub issue to submit questions or comments so we can keep the discussion public.

NCL Team:
John Clyne (acting group head)
Rick Brownrigg
Mary Haley
Kevin Hallock
Bill Ladwig

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-- 

Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science, 
The Hebrew University of Jerusalem, 
Givat Ram, Jerusalem 91904, Israel 

Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525

-- 

Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science, 
The Hebrew University of Jerusalem, 
Givat Ram, Jerusalem 91904, Israel 

Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

--

Adam Phillips

Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR

www.cgd.ucar.edu/staff/asphilli/   303-497-1726

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

--

Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science, 
The Hebrew University of Jerusalem, 
Givat Ram, Jerusalem 91904, Israel 

Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

--

| | Carl J. Schreck III, PhD
Research Scholar
North Carolina State University
North Carolina Institute for Climate Studies (NCICS)
151 Patton Ave, Asheville, NC 28801
e: cjschrec at ncsu.edu
o: +1 828 257 3140
c: +1 828 484 1702
Publications
ncics.org/mjo
CycloneCenter.org
|
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190210/f0f96ef0/attachment.html>