[pyngl-talk] delegation, division of labor, packaging, was: netcdf4-python vs PyNIO
dbrown at ucar.edu
Mon Jun 29 19:02:20 MDT 2015
OK you make some good points. We are all for division of labor and not
reinventing the wheel, etc. And perhaps it might actually be best for
us to incorporate NetCDF4-Python as a sub-component of PyNIO, allowing
us, as you suggest, to concentrate on supporting the other formats. I
am not ruling it out.
The counter-argument is this: the new development is not just for
NetCDF4 but also to expose the HDF5 capabilities of the NIO library.
We do know that there is some capability for handling HDF5 files
through NetCDF4-python, but we also know that it is not complete. We
have example files with variables that do not show up through the
NetCDF4 interface, but are visible with tools such as h5dump, h5py,
and also in the development version of PyNIO. This work is pretty far
along already. It also adds value by incorporating some ideas from
h5py that in my opinion make it easier to work with files that have a
complicated group hierarchy.
I know that HDF5 is becoming an important format for NCL users, and we
want to make it just as accessible using the NetCDF-like model in
However, if we knew that the NetCDF library would be providing more
complete coverage of HDF5 in the near future, it still might make
sense to use netCDF4-python. I am not sure why certain things seem to
be missing. They are not on the list of HDF5 features that NetCDF4
does not intend to support, such as circular references.
When we finish our conda install for PyNIO and PyNGL, it will provide
us with almost everything needed to do an NCL conda install as well.
So I believe that we can make them all available via conda at the same
On Thu, Jun 25, 2015 at 1:47 PM, Tom Roche <Tom_Roche at pobox.com> wrote:
> David Brown Wed, 24 Jun 2015 19:09:22 -0600 (rearranged)
>> we do understand that PyNIO is considered difficult to install and currently one of our highest priorities is to address this issue. We are in the process of implementing a conda install process for PyNIO and intend to have that ready within a month or so.
> More on packaging below.
>> Due to our small staff and our focus on NCL, our Python modules have been somewhat neglected in recent years.
> I believe this has been noticed :-)
>> But our plans now include much more focus on our Python tools. In fact you may have noticed Mary's announcement of a new position where we are looking for someone with Python scientific programming expertise. Also we have been focused recently on updating PyNIO's capabilities with respect to more advanced features of NetCDF4/HDF5. We have good support for groups now and are close to completing support for compound data types, variable length arrays, etc.
>> If you are only interested in NetCDF data, then there is not much reason to prefer PyNIO over netcdf4-python. Sasha's points in favor of netcdf4-python are valid and I could add that netcdf4-python has since its inception supported the complete NetCDF4 specification that we are only now putting into PyNIO. [But] PyNIO makes multiple formats available in a consistent fashion that conforms to the NetCDF model.
> That being said, one could make an argument (and I will :-)
> 1. A foundational principle of civilization is division of labor (DoL). It's been moderately successful :-)
> 2. A (probably *the*) foundational principle of software engineering is abstraction aka delegation aka layering. (ditto)
> 3. Current economic and political "realities" imply that budgets for "the sort of stuff" listizens in general do, and NCAR/UCAR specifically, are unlikely to improve significantly near-term.
> 4. CISL's current policy seems to be to develop support for netCDF4/HDF5 that already exists in Unidata's netcdf4-python.
> Am I missing something? If not, allow me to assert that a more sensible policy--ceteris paribus--for the allocation of scarce resources would be layering+DoL:
> 1. PyNIO should delegate basic netCDF4 support to netcdf4-python.
> 2. PyNIO should focus on supporting additional/non-netCDF formats.
> 3. PyNIO should adopt netcdf4-python as a dependency.
> Granted, it is sometimes the case that "them ceteris just ain't paribus," and I may have just offended political sensibilities hugely. But in the absence of very good reasons to do so, I can't see why PyNIO should put scarce resources into reinventing python support for netCDF4.
>> Concerning xray, I agree this is a very promising tool. A point that may not be totally clear from the PyAOS discussion is that this tool relies on a backend IO-module to do the low-level reading of data files.
> Yes, layering+DoL is quite a popular idea :-)
>> Experimentally, I recently created a PyNIO backend for xray that can be used in place of netcdf4-python. The immediate advantage for an xray user is that you can now access, in a uniform manner, all the PyNIO-supported formats along with NetCDF. Hopefully once it is fully vetted, Steven Hoyer can be persuaded to add it as another alternate backend to the xray code base.
> Sounds good!
>> our new release with conda-based installation.
> and how 'bout conda-packaging NCL? Which leads to a more general political point, perhaps to add to my two points above:
> Packaging is important, because it directly affects other important issues like installability and footprint. Unfortunately (as with documentation) there's little glory, and less funding, to be gotten from packaging. It's also hard, but now that Continuum/conda/binstar seem to have done most of the lowest-level gruntwork, It Would Be Great if CISL, Unidata, or some other part of this ecosystem could agree to specialize in this task and assure the code you are all developing gets appropriately packaged.
> HTH, Tom Roche <Tom_Roche at pobox.com>
> : http://mailman.ucar.edu/pipermail/pyngl-talk/2015-June/000049.html
> : https://en.wiktionary.org/wiki/ceteris_paribus
> : http://www.ncl.ucar.edu/Support/talk_archives/2014/0884.html Notice some work toward this end @ https://github.com/bjlittle/conda-recipes-ncl
> pyngl-talk mailing list
> List instructions, subscriber options, unsubscribe:
More information about the pyngl-talk