[pyngl-talk] delegation, division of labor, packaging, was: netcdf4-python vs PyNIO

[Tom Roche]

David Brown Wed, 24 Jun 2015 19:09:22 -0600 (rearranged)[1]
> we do understand that PyNIO is considered difficult to install and currently one of our highest priorities is to address this issue. We are in the process of implementing a conda install process for PyNIO and intend to have that ready within a month or so.

More on packaging below.

> Due to our small staff and our focus on NCL, our Python modules have been somewhat neglected in recent years.

I believe this has been noticed :-)

> But our plans now include much more focus on our Python tools. In fact you may have noticed Mary's announcement of a new position where we are looking for someone with Python scientific programming expertise. Also we have been focused recently on updating PyNIO's capabilities with respect to more advanced features of NetCDF4/HDF5. We have good support for groups now and are close to completing support for compound data types, variable length arrays, etc.

> If you are only interested in NetCDF data, then there is not much reason to prefer PyNIO over netcdf4-python. Sasha's points in favor of netcdf4-python are valid and I could add that netcdf4-python has since its inception supported the complete NetCDF4 specification that we are only now putting into PyNIO. [But] PyNIO makes multiple formats available in a consistent fashion that conforms to the NetCDF model.

That being said, one could make an argument (and I will :-)

1. A foundational principle of civilization is division of labor (DoL). It's been moderately successful :-)

2. A (probably *the*) foundational principle of software engineering is abstraction aka delegation aka layering. (ditto)

3. Current economic and political "realities" imply that budgets for "the sort of stuff" listizens in general do, and NCAR/UCAR specifically, are unlikely to improve significantly near-term.

4. CISL's current policy seems to be to develop support for netCDF4/HDF5 that already exists in Unidata's netcdf4-python.

Am I missing something? If not, allow me to assert that a more sensible policy--ceteris paribus[2]--for the allocation of scarce resources would be layering+DoL:

1. PyNIO should delegate basic netCDF4 support to netcdf4-python.

2. PyNIO should focus on supporting additional/non-netCDF formats.

3. PyNIO should adopt netcdf4-python as a dependency.

Granted, it is sometimes the case that "them ceteris just ain't paribus," and I may have just offended political sensibilities hugely. But in the absence of very good reasons to do so, I can't see why PyNIO should put scarce resources into reinventing python support for netCDF4.

> Concerning xray, I agree this is a very promising tool. A point that may not be totally clear from the PyAOS discussion is that this tool relies on a backend IO-module to do the low-level reading of data files.

Yes, layering+DoL is quite a popular idea :-)

> Experimentally, I recently created a PyNIO backend for xray that can be used in place of netcdf4-python. The immediate advantage for an xray user is that you can now access, in a uniform manner, all the PyNIO-supported formats along with NetCDF. Hopefully once it is fully vetted, Steven Hoyer can be persuaded to add it as another alternate backend to the xray code base.

Sounds good!

> our new release with conda-based installation.

and how 'bout conda-packaging NCL[3]? Which leads to a more general political point, perhaps to add to my two points above:

Packaging is important, because it directly affects other important issues like installability and footprint. Unfortunately (as with documentation) there's little glory, and less funding, to be gotten from packaging. It's also hard, but now that Continuum/conda/binstar seem to have done most of the lowest-level gruntwork, It Would Be Great if CISL, Unidata, or some other part of this ecosystem could agree to specialize in this task and assure the code you are all developing gets appropriately packaged.

HTH, Tom Roche <Tom_Roche at pobox.com>

[1]: http://mailman.ucar.edu/pipermail/pyngl-talk/2015-June/000049.html
[2]: https://en.wiktionary.org/wiki/ceteris_paribus
[3]: http://www.ncl.ucar.edu/Support/talk_archives/2014/0884.html Notice some work toward this end @ https://github.com/bjlittle/conda-recipes-ncl