<div dir="ltr">Hi Heather,<div><br></div><div>Yes, it is an NioVariable object. Seems that at 1.4.1, NioVariable provided some numpy functionality from the object itself rather than requiring you to extract the numpy array first. </div><div><br></div><div>I am not an expert with PyNIO/numpy and this code was written by someone else, but I was under the impression NioVariable provided access to the arrays via an iterator (i.e. lazy loading), which would be preferable in some cases to loading the entire arrays into memory using numpy indexing. But I could be totally off base there.</div><div><br></div><div>Output is below.</div><div><br></div><div>Thanks,<br>Jason</div><div><br></div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>$ python</div><div>Python 2.7.2 (default, May 7 2012, 16:54:01)</div><div>[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2</div><div>Type "help", "copyright", "credits" or "license" for more information.</div><div>>>> import Nio</div><div>>>> import numpy</div><div>>>> Nio.__version__</div><div>'1.4.1'</div><div>>>> numpy.__version__</div><div>'1.6.1'</div><div>>>> f = Nio.open_file("<a href="http://glofs.leofs.fields.nowcast.20151229.t12z.nc">glofs.leofs.fields.nowcast.20151229.t12z.nc</a>", "r")</div><div>>>> m = f.variables["mask"]</div><div>>>> m</div><div><Nio.NioVariable object at 0x2290050></div><div>>>> print type(m)</div><div><class 'Nio.NioVariable'></div><div>>>> m.shape</div><div>(24, 81)</div><div>>>> m_contents = m[:,:]</div><div>>>> print type(m_contents)</div><div><type 'numpy.ndarray'></div><div>>>> numpy.ma.masked_equal(m, 1.0)</div><div>masked_array(data =</div><div> [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> ...,</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],</div><div> mask =</div><div> [[False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> ...,</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]],</div><div> fill_value = 1.0)</div><div><br></div><div>>>> numpy.ma.masked_equal(m_contents, 1.0)</div><div>masked_array(data =</div><div> [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> ...,</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div><div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],</div><div> mask =</div><div> [[False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> ...,</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]</div><div> [False False False ..., False False False]],</div><div> fill_value = 1.0)</div><div><br></div></div></blockquote></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>--</div>Jason Greenlaw<div>Software Developer, ERT, Inc.<br><div>NOAA/NOS/OCS/CSDL<div><a href="http://nowcoast.noaa.gov" target="_blank">http://nowcoast.noaa.gov</a></div><div><a href="mailto:Jason.Greenlaw@noaa.gov" target="_blank">Jason.Greenlaw@noaa.gov</a></div><div><br></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Dec 29, 2015 at 1:19 PM, Cronk,Heather <span dir="ltr"><<a href="mailto:Heather.Cronk@colostate.edu" target="_blank">Heather.Cronk@colostate.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Hi Jason,</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div><font face="Calibri,sans-serif">I am a PyNIO user, not a developer so I can’t speak to any intentionality, but I am more surprised that your code works with version PyNIO version 1.4.1 than that it does’t work with the beta. I don’t have the old version
anymore, but I am curious the output of type(m) with your original code? I was under the impression that the call </font><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">f.variables[</span><span style="color:rgb(209,47,27);font-family:Menlo;font-size:11px">"mask</span><font color="#d12f1b" face="Menlo" size="2">”</font><font face="Menlo" size="2">] </font><font face="Calibri,sans-serif">had
always produced a Nio object and not a numpy array. Using the beta version I see this:</font></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<p style="margin:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin:0px;font-size:11px;font-family:Menlo">m_obj = f.variables[<span style="color:#d12f1b">"mask"</span>]</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="color:#bb2ca2">print</span> type(m_obj)</p>
<p style="margin:0px;font-size:11px;font-family:Menlo">>> <class 'Nio.NioVariable'></p>
<p style="margin:0px;font-size:11px;font-family:Menlo">m_contents = f.variables[<span style="color:#d12f1b">"mask"</span>][:]</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="color:#bb2ca2">print</span> type(m_contents)</p>
<p style="margin:0px;font-size:11px;font-family:Menlo">>> <type 'numpy.ndarray’></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="font-family:Calibri,sans-serif;font-size:medium">What does the corresponding code produce with the 1.4.1?</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="font-family:Calibri,sans-serif;font-size:medium"><br>
</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="font-family:Calibri,sans-serif;font-size:medium">Thanks!</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><span style="font-family:Calibri,sans-serif;font-size:medium">Heather</span></p>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<span style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:pyngl-talk-bounces@ucar.edu" target="_blank">pyngl-talk-bounces@ucar.edu</a>> on behalf of Jason Greenlaw - NOAA Affiliate <<a href="mailto:jason.greenlaw@noaa.gov" target="_blank">jason.greenlaw@noaa.gov</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, December 29, 2015 at 10:36 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:pyngl-talk@ucar.edu" target="_blank">pyngl-talk@ucar.edu</a>" <<a href="mailto:pyngl-talk@ucar.edu" target="_blank">pyngl-talk@ucar.edu</a>><br>
<span style="font-weight:bold">Subject: </span>[pyngl-talk] PyNIO 1.5.0 Beta vs 1.4.1 - NetCDF Variable Access and Numpy<br>
</div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>I recently installed the 1.5.0 beta versions of PyNGL and PyNIO (using 64-bit binaries for CentOS6) and attempted to run some existing code, but encountered an issue when numpy functions (e.g. numpy.ma.masked_equal()) are called with NioVariable object
arguments.</div>
<div><br>
</div>
<div>At PyNIO v1.4.1/numpy1.6.1 I was able to do the following:</div>
<div><br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>>>> import Nio</div>
<div>
<div>>>> Nio.__version__</div>
<div>'1.4.1'</div>
</div>
<div>>>> import numpy</div>
<div>>>> numpy.__version__</div>
<div>'1.6.1'</div>
<div>>>> f = Nio.open_file("<a href="http://glofs.leofs.fields.nowcast.20151229.t12z.nc" target="_blank">glofs.leofs.fields.nowcast.20151229.t12z.nc</a>", "r")</div>
<div>>>> m = f.variables["mask"]</div>
<div>>>> numpy.ma.masked_equal(m, 1.0)</div>
<div>masked_array(data =</div>
<div> [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> ...,</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],</div>
<div> mask =</div>
<div> [[False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> ...,</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]],</div>
<div> fill_value = 1.0)</div>
</blockquote>
</div>
<div><br>
</div>
<div><br>
</div>
<div>However at PyNIO 1.5.0 beta/numpy 1.9.2, the numpy function call hangs, and the process begins consuming memory at an exponential rate until the call is interrupted.</div>
<div><br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>
<div>>>> import Nio<br>
</div>
<div>>>> Nio.__version__</div>
<div>'1.5.0-beta'</div>
<div>>>> import numpy</div>
<div>>>> numpy.__version__</div>
<div>'1.9.2'</div>
</div>
<div>
<div>>>> f = Nio.open_file("<a href="http://glofs.leofs.fields.nowcast.20151229.t12z.nc" target="_blank">glofs.leofs.fields.nowcast.20151229.t12z.nc</a>", "r")</div>
<div>>>> m = f.variables["mask"]</div>
</div>
<div>
<div>>>> numpy.ma.masked_equal(m, 1.0)</div>
<div>^C^CTraceback (most recent call last):</div>
<div> File "<stdin>", line 1, in <module></div>
<div> File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 1982, in masked_equal</div>
<div> output = masked_where(equal(x, value), x, copy=copy)</div>
<div> File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 928, in __call__</div>
<div> (da, db) = (getdata(a, subok=False), getdata(b, subok=False))</div>
<div> File "/opt/pyngl/python/lib/python2.7/site-packages/numpy/ma/core.py", line 667, in getdata</div>
<div> data = np.array(a, copy=False, subok=subok)</div>
<div> File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/Nio.py", line 325, in __getitem__</div>
<div> ret = get_variable(self.file, self.varname, xsel)</div>
<div> File "/opt/pyngl/python/lib/python2.7/site-packages/PyNIO/coordsel.py", line 60, in get_variable</div>
<div> ret = file.file.variables[varname][xsel]</div>
<div>KeyboardInterrupt</div>
</div>
</blockquote>
<div><br>
</div>
<div>But if I change the call to use numpy indexing, it works:</div>
<br>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>
<div>
<div>>>> m.shape</div>
<div>(24, 81)</div>
</div>
<div>>>> numpy.ma.masked_equal(m[:,:], 1.0)</div>
<div>masked_array(data =</div>
<div> [[0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> ...,</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]</div>
<div> [0.0 0.0 0.0 ..., 0.0 0.0 0.0]],</div>
<div> mask =</div>
<div> [[False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> ...,</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]</div>
<div> [False False False ..., False False False]],</div>
<div> fill_value = 1.0)</div>
</div>
</blockquote>
<br>
</div>
<div>Was this change in functionality intentional? </div>
<div><br>
</div>
<div>The NetCDF files I used are available at:</div>
<div><br>
</div>
<div> <a href="ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nos/prod/glofs.20151229/" target="_blank">
ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nos/prod/glofs.20151229/</a><br>
</div>
<div><br>
</div>
(directory will change based on date)
<div><br>
</div>
<div>Thanks,</div>
<div>Jason</div>
<div>
<div>
<div>
<div dir="ltr">
<div>--</div>
Jason Greenlaw
<div>Software Developer, ERT, Inc.<br>
<div>NOAA/NOS/OCS/CSDL
<div><a href="http://nowcoast.noaa.gov" target="_blank">http://nowcoast.noaa.gov</a></div>
<div><a href="mailto:Jason.Greenlaw@noaa.gov" target="_blank">Jason.Greenlaw@noaa.gov</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></span>
</div>
</blockquote></div><br></div></div>