[pyngl-talk] PyNIO 1.5.0 Beta vs 1.4.1 - NetCDF Variable Access and Numpy

[Cronk,Heather]

Hi Jason,

I am a PyNIO user, not a developer so I can't speak to any intentionality, but I am more surprised that your code works with version PyNIO version 1.4.1 than that it does't work with the beta. I don't have the old version anymore, but I am curious the output of type(m) with your original code? I was under the impression that the call f.variables["mask"] had always produced a Nio object and not a numpy array. Using the beta version I see this:


m_obj = f.variables["mask"]

print type(m_obj)

>>  <class 'Nio.NioVariable'>

m_contents = f.variables["mask"][:]

print type(m_contents)

>>  <type 'numpy.ndarray'>


What does the corresponding code produce with the 1.4.1?


Thanks!

Heather

From: <pyngl-talk-bounces at ucar.edu<mailto:pyngl-talk-bounces at ucar.edu>> on behalf of Jason Greenlaw - NOAA Affiliate <jason.greenlaw at noaa.gov<mailto:jason.greenlaw at noaa.gov>>
Date: Tuesday, December 29, 2015 at 10:36 AM
To: "pyngl-talk at ucar.edu<mailto:pyngl-talk at ucar.edu>" <pyngl-talk at ucar.edu<mailto:pyngl-talk at ucar.edu>>
Subject: [pyngl-talk] PyNIO 1.5.0 Beta vs 1.4.1 - NetCDF Variable Access and Numpy

Hello,

I recently installed the 1.5.0 beta versions of PyNGL and PyNIO (using 64-bit binaries for CentOS6) and attempted to run some existing code, but encountered an issue when numpy functions (e.g. numpy.ma.masked_equal()) are called with NioVariable object arguments.

At PyNIO v1.4.1/numpy1.6.1 I was able to do the following:

>>> import Nio
>>> Nio.__version__
'1.4.1'
>>> import numpy
>>> numpy.__version__
'1.6.1'
>>> f = Nio.open_file("glofs.leofs.fields.nowcast.20151229.t12z.nc<http://glofs.leofs.fields.nowcast.20151229.t12z.nc>", "r")
>>> m = f.variables["mask"]
>>> numpy.ma.masked_equal(m, 1.0)
masked_array(data =
 [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 ...,
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],
             mask =
 [[False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]
 ...,
 [False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]],
       fill_value = 1.0)


However at PyNIO 1.5.0 beta/numpy 1.9.2, the numpy function call hangs, and the process begins consuming memory at an exponential rate until the call is interrupted.

>>> import Nio
>>> Nio.__version__
'1.5.0-beta'
>>> import numpy
>>> numpy.__version__
'1.9.2'
>>> f = Nio.open_file("glofs.leofs.fields.nowcast.20151229.t12z.nc<http://glofs.leofs.fields.nowcast.20151229.t12z.nc>", "r")
>>> m = f.variables["mask"]
>>> numpy.ma.masked_equal(m, 1.0)
^C^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 1982, in masked_equal
    output = masked_where(equal(x, value), x, copy=copy)
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 928, in __call__
    (da, db) = (getdata(a, subok=False), getdata(b, subok=False))
  File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 667, in getdata
    data = np.array(a, copy=False, subok=subok)
  File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/Nio.py", line 325, in __getitem__
    ret = get_variable(self.file, self.varname, xsel)
  File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/coordsel.py", line 60, in get_variable
    ret = file.file.variables[varname][xsel]
KeyboardInterrupt

But if I change the call to use numpy indexing, it works:

>>> m.shape
(24, 81)
>>> numpy.ma.masked_equal(m[:,:], 1.0)
masked_array(data =
 [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 ...,
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]
 [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],
             mask =
 [[False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]
 ...,
 [False False False ..., False False False]
 [False False False ..., False False False]
 [False False False ..., False False False]],
       fill_value = 1.0)

Was this change in functionality intentional?

The NetCDF files I used are available at:

    ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nos/prod/glofs.20151229/

(directory will change based on date)

Thanks,
Jason
--
Jason Greenlaw
Software Developer, ERT, Inc.
NOAA/NOS/OCS/CSDL
http://nowcoast.noaa.gov
Jason.Greenlaw at noaa.gov<mailto:Jason.Greenlaw at noaa.gov>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/pyngl-talk/attachments/20151229/5ebc9387/attachment-0001.html