[pyngl-talk] PyNIO 1.5.0 Beta vs 1.4.1 - NetCDF Variable Access and Numpy

[Jason Greenlaw - NOAA Affiliate]

Hello,

I recently installed the 1.5.0 beta versions of PyNGL and PyNIO (using
64-bit binaries for CentOS6) and attempted to run some existing code, but
encountered an issue when numpy functions (e.g. numpy.ma.masked_equal())
are called with NioVariable object arguments.

At PyNIO v1.4.1/numpy1.6.1 I was able to do the following:

>>> import Nio
>>> Nio.__version__
'1.4.1'
>>> import numpy
>>> numpy.__version__
'1.6.1'
>>> f = Nio.open_file("glofs.leofs.fields.nowcast.20151229.t12z.nc", "r")
>>> m = f.variables["mask"]
>>> numpy.ma.masked_equal(m, 1.0)
masked_array(data =
 [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 ...,
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],
             mask =
 [[False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]
 ...,
 [False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]],
       fill_value = 1.0)



However at PyNIO 1.5.0 beta/numpy 1.9.2, the numpy function call hangs, and
the process begins consuming memory at an exponential rate until the call
is interrupted.

>>> import Nio
>>> Nio.__version__
'1.5.0-beta'
>>> import numpy
>>> numpy.__version__
'1.9.2'
>>> f = Nio.open_file("glofs.leofs.fields.nowcast.20151229.t12z.nc", "r")
>>> m = f.variables["mask"]
>>> numpy.ma.masked_equal(m, 1.0)
^C^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py",
line 1982, in masked_equal
    output = masked_where(equal(x, value), x, copy=copy)
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py",
line 928, in __call__
    (da, db) = (getdata(a, subok=False), getdata(b, subok=False))
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py",
line 667, in getdata
    data = np.array(a, copy=False, subok=subok)
  File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/Nio.py", line
325, in __getitem__
    ret = get_variable(self.file, self.varname, xsel)
  File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/coordsel.py",
line 60, in get_variable
    ret = file.file.variables[varname][xsel]
KeyboardInterrupt


But if I change the call to use numpy indexing, it works:

>>> m.shape
(24, 81)
>>> numpy.ma.masked_equal(m[:,:], 1.0)
masked_array(data =
 [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 ...,
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],
             mask =
 [[False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]
 ...,
 [False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]],
       fill_value = 1.0)


Was this change in functionality intentional?

The NetCDF files I used are available at:

    ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nos/prod/glofs.20151229/

(directory will change based on date)

Thanks,
Jason
--
Jason Greenlaw
Software Developer, ERT, Inc.
NOAA/NOS/OCS/CSDL
http://nowcoast.noaa.gov
Jason.Greenlaw at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/pyngl-talk/attachments/20151229/2e66d043/attachment.html