[Met_help] [rt.rap.ucar.edu #92378] History for MET 8.1 python embedding failing when using grib2 file
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 11 16:25:25 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hey all,
I'm having trouble getting plot_data_plane to work on cheyenne - I am
trying it using these two commands:
plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
>
and
plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py mflux_ana.npy";'
>
the read_GFSv3.py code is as follows:
from __future__ import print_function
> import os
> import sys
> import re
> import numpy as np
> import datetime as dt
> import pygrib
>
> def ivt(input_file):
> grbs = pygrib.open(input_file)
> g = 9.81
> grbs.rewind()
>
> levs = []
> for grb in grbs:
> if 'humidity' in grb.parameterName.lower():
> levs.append(grb.level*100)
> levs=np.array(levs)
> grbs.rewind()
>
> q=[]
> for grb in grbs:
> if 'humidity' in grb.parameterName:
> q.append(grb.values)
> q=np.array(q)
> grbs.rewind()
>
> hgt=[]
> for grb in grbs:
> if 'Geopotential' in grb.parameterName:
> hgt.append(grb.values)
> hgt = np.array(hgt)
> grbs.rewind()
>
> temp=[]
> for grb in grbs:
> if 'Temperature' in grb.parameterName:
> temp.append(grb.values)
> temp = np.array(temp)
> grbs.rewind()
>
> u=[]
> for grb in grbs:
> if 'u-' in grb.parameterName:
> u.append(grb.values)
> u = np.array(u)
> grbs.rewind()
>
> v=[]
> for grb in grbs:
> if 'v-' in grb.parameterName:
> v.append(grb.values)
> v = np.array(v)
> grbs.rewind()
>
> uv = np.sqrt(u**2+v**2)
> mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> mflux = np.sum(mflux,axis=0)
> mflux_avg = np.mean(mflux,axis=0)
> return mflux
>
> if len(sys.argv) == 2:
> input_file = os.path.expandvars(sys.argv[1])
> try:
> # Print some output to verify that this script ran
> print("Input File: " + repr(input_file))
>
> # Read input file
> try:
> data = ivt(input_file)
> except:
> data = np.load(input_file)
> print(type(data))
> # Read and re-orient the data
> met_data = data
>
> print("Data Shape: " + repr(met_data.shape))
> print("Data Type: " + repr(met_data.dtype))
> except NameError:
> print("Trouble reading input file: " + input_file)
> else:
> print("Must specify exactly one input file.")
> sys.exit(1)
> valid = dt.datetime(2016,1,24)
>
> attrs = {
> 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> 'init': valid.strftime("%Y%m%d_%H%M%S"),
> 'lead': '0',
> 'accum': '00',
>
> 'name': 'mflux',
> 'long_name': 'UNKNOWN',
> 'level': '1000_300',
> 'units': 'UNKNOWN',
>
> 'grid': {
> 'name': 'Global 1 Degree',
> 'type' : 'LatLon',
> 'lat_ll' : -90.0,
> 'lon_ll' : 0.0,
> 'delta_lat' : 1.0,
> 'delta_lon' : 1.0,
> 'Nlat' : 181,
> 'Nlon' : 360,
> }
> }
>
>
>
> print("Attributes: " + repr(attrs))
>
The only difference between the two is the grib2 invokes the ivt function,
where pygrib opens the file. When I try using the grib2 file, I get a
segmentation fault (core dumped) error. Using the first command (with the
.npy file) works. In both cases, met_data is a numpy array (which is why I
kept PYTHON_NUMPY). Am I missing a flag or another issue I'm overlooking?
Thanks!
--
*Taylor Mandelbaum*
Research Technician
SoMAS | Stony Brook University
678.468.5789
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: John Halley Gotway
Time: Mon Sep 30 09:48:22 2019
Hi Taylor,
I see you're having trouble with python embedding on Cheyenne. I
logged on
this morning to try to replicate the errors you're getting.
Below are the commands I used to test this:
*cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
*module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule load
met/8.1_pythonncar_pylib *
*wget
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
<ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048>*
*python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
Since you didn't send me the path to where your data lives on
Cheyenne, I
ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your filename
includes "reduced" which I suspect means that some GRIB records have
been
excluded. But the one I have contains the full 586 records.
The first step with python embedding in MET is to make sure the script
runs
fine through python itself. But mine produces a runtime error:
python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
Input File: 'gfs.t00z.pgrb2.1p00.f048'
read_GFSv3.py:56: RuntimeWarning: overflow encountered in square
uv = np.sqrt(u**2+v**2)
Traceback (most recent call last):
File "read_GFSv3.py", line 72, in <module>
data = np.load(input_file)
File
"/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
lin
e 431, in load
"Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048' as a
pickle
Does yours run without error?
*python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
If so, please tell me where these files live.
Thanks,
John
On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT
<met_help at ucar.edu>
wrote:
>
> Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> Transaction: Ticket created by taylor.mandelbaum at stonybrook.edu
> Queue: met_help
> Subject: MET 8.1 python embedding failing when using grib2 file
> Owner: Nobody
> Requestors: taylor.mandelbaum at stonybrook.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
>
> Hey all,
>
> I'm having trouble getting plot_data_plane to work on cheyenne - I
am
> trying it using these two commands:
>
> plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> >
>
> and
>
> plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> mflux_ana.npy";'
> >
>
> the read_GFSv3.py code is as follows:
>
> from __future__ import print_function
> > import os
> > import sys
> > import re
> > import numpy as np
> > import datetime as dt
> > import pygrib
> >
> > def ivt(input_file):
> > grbs = pygrib.open(input_file)
> > g = 9.81
> > grbs.rewind()
> >
> > levs = []
> > for grb in grbs:
> > if 'humidity' in grb.parameterName.lower():
> > levs.append(grb.level*100)
> > levs=np.array(levs)
> > grbs.rewind()
> >
> > q=[]
> > for grb in grbs:
> > if 'humidity' in grb.parameterName:
> > q.append(grb.values)
> > q=np.array(q)
> > grbs.rewind()
> >
> > hgt=[]
> > for grb in grbs:
> > if 'Geopotential' in grb.parameterName:
> > hgt.append(grb.values)
> > hgt = np.array(hgt)
> > grbs.rewind()
> >
> > temp=[]
> > for grb in grbs:
> > if 'Temperature' in grb.parameterName:
> > temp.append(grb.values)
> > temp = np.array(temp)
> > grbs.rewind()
> >
> > u=[]
> > for grb in grbs:
> > if 'u-' in grb.parameterName:
> > u.append(grb.values)
> > u = np.array(u)
> > grbs.rewind()
> >
> > v=[]
> > for grb in grbs:
> > if 'v-' in grb.parameterName:
> > v.append(grb.values)
> > v = np.array(v)
> > grbs.rewind()
> >
> > uv = np.sqrt(u**2+v**2)
> > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > mflux = np.sum(mflux,axis=0)
> > mflux_avg = np.mean(mflux,axis=0)
> > return mflux
> >
> > if len(sys.argv) == 2:
> > input_file = os.path.expandvars(sys.argv[1])
> > try:
> > # Print some output to verify that this script ran
> > print("Input File: " + repr(input_file))
> >
> > # Read input file
> > try:
> > data = ivt(input_file)
> > except:
> > data = np.load(input_file)
> > print(type(data))
> > # Read and re-orient the data
> > met_data = data
> >
> > print("Data Shape: " + repr(met_data.shape))
> > print("Data Type: " + repr(met_data.dtype))
> > except NameError:
> > print("Trouble reading input file: " + input_file)
> > else:
> > print("Must specify exactly one input file.")
> > sys.exit(1)
> > valid = dt.datetime(2016,1,24)
> >
> > attrs = {
> > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > 'lead': '0',
> > 'accum': '00',
> >
> > 'name': 'mflux',
> > 'long_name': 'UNKNOWN',
> > 'level': '1000_300',
> > 'units': 'UNKNOWN',
> >
> > 'grid': {
> > 'name': 'Global 1 Degree',
> > 'type' : 'LatLon',
> > 'lat_ll' : -90.0,
> > 'lon_ll' : 0.0,
> > 'delta_lat' : 1.0,
> > 'delta_lon' : 1.0,
> > 'Nlat' : 181,
> > 'Nlon' : 360,
> > }
> > }
> >
> >
> >
> > print("Attributes: " + repr(attrs))
> >
>
> The only difference between the two is the grib2 invokes the ivt
function,
> where pygrib opens the file. When I try using the grib2 file, I get
a
> segmentation fault (core dumped) error. Using the first command
(with the
> .npy file) works. In both cases, met_data is a numpy array (which is
why I
> kept PYTHON_NUMPY). Am I missing a flag or another issue I'm
overlooking?
>
> Thanks!
> --
> *Taylor Mandelbaum*
> Research Technician
> SoMAS | Stony Brook University
> 678.468.5789
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: Taylor Mandelbaum
Time: Mon Sep 30 09:50:36 2019
Hey John,
Yeah, mine runs through when I try it using just python. My data is on
my
local directory in cheyenne - is there an easy way to give you access
to it?
On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Taylor,
>
> I see you're having trouble with python embedding on Cheyenne. I
logged on
> this morning to try to replicate the errors you're getting.
>
> Below are the commands I used to test this:
>
>
> *cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
>
>
>
> *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> met/8.1_pythonncar_pylib *
> *wget
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> <
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> >*
> *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
>
> Since you didn't send me the path to where your data lives on
Cheyenne, I
> ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your filename
> includes "reduced" which I suspect means that some GRIB records have
been
> excluded. But the one I have contains the full 586 records.
>
> The first step with python embedding in MET is to make sure the
script runs
> fine through python itself. But mine produces a runtime error:
>
> python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
>
> Input File: 'gfs.t00z.pgrb2.1p00.f048'
> read_GFSv3.py:56: RuntimeWarning: overflow encountered in square
> uv = np.sqrt(u**2+v**2)
> Traceback (most recent call last):
> File "read_GFSv3.py", line 72, in <module>
> data = np.load(input_file)
> File
>
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> lin
> e 431, in load
> "Failed to interpret file %s as a pickle" % repr(file))
> IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048' as a
pickle
>
> Does yours run without error?
> *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
>
> If so, please tell me where these files live.
>
> Thanks,
> John
>
> On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > Transaction: Ticket created by taylor.mandelbaum at stonybrook.edu
> > Queue: met_help
> > Subject: MET 8.1 python embedding failing when using grib2
file
> > Owner: Nobody
> > Requestors: taylor.mandelbaum at stonybrook.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> >
> >
> > Hey all,
> >
> > I'm having trouble getting plot_data_plane to work on cheyenne - I
am
> > trying it using these two commands:
> >
> > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > >
> >
> > and
> >
> > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > mflux_ana.npy";'
> > >
> >
> > the read_GFSv3.py code is as follows:
> >
> > from __future__ import print_function
> > > import os
> > > import sys
> > > import re
> > > import numpy as np
> > > import datetime as dt
> > > import pygrib
> > >
> > > def ivt(input_file):
> > > grbs = pygrib.open(input_file)
> > > g = 9.81
> > > grbs.rewind()
> > >
> > > levs = []
> > > for grb in grbs:
> > > if 'humidity' in grb.parameterName.lower():
> > > levs.append(grb.level*100)
> > > levs=np.array(levs)
> > > grbs.rewind()
> > >
> > > q=[]
> > > for grb in grbs:
> > > if 'humidity' in grb.parameterName:
> > > q.append(grb.values)
> > > q=np.array(q)
> > > grbs.rewind()
> > >
> > > hgt=[]
> > > for grb in grbs:
> > > if 'Geopotential' in grb.parameterName:
> > > hgt.append(grb.values)
> > > hgt = np.array(hgt)
> > > grbs.rewind()
> > >
> > > temp=[]
> > > for grb in grbs:
> > > if 'Temperature' in grb.parameterName:
> > > temp.append(grb.values)
> > > temp = np.array(temp)
> > > grbs.rewind()
> > >
> > > u=[]
> > > for grb in grbs:
> > > if 'u-' in grb.parameterName:
> > > u.append(grb.values)
> > > u = np.array(u)
> > > grbs.rewind()
> > >
> > > v=[]
> > > for grb in grbs:
> > > if 'v-' in grb.parameterName:
> > > v.append(grb.values)
> > > v = np.array(v)
> > > grbs.rewind()
> > >
> > > uv = np.sqrt(u**2+v**2)
> > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > mflux = np.sum(mflux,axis=0)
> > > mflux_avg = np.mean(mflux,axis=0)
> > > return mflux
> > >
> > > if len(sys.argv) == 2:
> > > input_file = os.path.expandvars(sys.argv[1])
> > > try:
> > > # Print some output to verify that this script ran
> > > print("Input File: " + repr(input_file))
> > >
> > > # Read input file
> > > try:
> > > data = ivt(input_file)
> > > except:
> > > data = np.load(input_file)
> > > print(type(data))
> > > # Read and re-orient the data
> > > met_data = data
> > >
> > > print("Data Shape: " + repr(met_data.shape))
> > > print("Data Type: " + repr(met_data.dtype))
> > > except NameError:
> > > print("Trouble reading input file: " + input_file)
> > > else:
> > > print("Must specify exactly one input file.")
> > > sys.exit(1)
> > > valid = dt.datetime(2016,1,24)
> > >
> > > attrs = {
> > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > 'lead': '0',
> > > 'accum': '00',
> > >
> > > 'name': 'mflux',
> > > 'long_name': 'UNKNOWN',
> > > 'level': '1000_300',
> > > 'units': 'UNKNOWN',
> > >
> > > 'grid': {
> > > 'name': 'Global 1 Degree',
> > > 'type' : 'LatLon',
> > > 'lat_ll' : -90.0,
> > > 'lon_ll' : 0.0,
> > > 'delta_lat' : 1.0,
> > > 'delta_lon' : 1.0,
> > > 'Nlat' : 181,
> > > 'Nlon' : 360,
> > > }
> > > }
> > >
> > >
> > >
> > > print("Attributes: " + repr(attrs))
> > >
> >
> > The only difference between the two is the grib2 invokes the ivt
> function,
> > where pygrib opens the file. When I try using the grib2 file, I
get a
> > segmentation fault (core dumped) error. Using the first command
(with the
> > .npy file) works. In both cases, met_data is a numpy array (which
is why
> I
> > kept PYTHON_NUMPY). Am I missing a flag or another issue I'm
overlooking?
> >
> > Thanks!
> > --
> > *Taylor Mandelbaum*
> > Research Technician
> > SoMAS | Stony Brook University
> > 678.468.5789
> >
> >
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: John Halley Gotway
Time: Mon Sep 30 10:22:00 2019
Taylor,
OK, I went hunting around and found your working directory in:
/glade/u/home/mandelbaum/test
And I can read data from your directory, so that's fine. Please send
me
the exact python command that you're running on the command line to
confirm
that your script works.
When I run it, it does not run fine for me. The first step in
debugging
this is figuring out exactly what error we need to debug:
*module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule load
met/8.1_pythonncar_pylib python
/glade/u/home/mandelbaum/test/read_GFSv3.py
gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
Trouble reading input file: gfs.t00z.pgrb2.1p00.f048.reduced.grib2
Attributes: {'long_name': 'UNKNOWN', 'init': '20160124_000000',
'valid':
'20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
'Global 1 Degree', 'Nla
t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type': 'LatLon', 'Nlon':
360},
'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units': 'UNKNOWN',
'accum': '00'}
Note that this script is NOT successful in reading the command line
argument file (Trouble reading input file:...). So it falls into the
exception block and reads the analysis file instead.
I'm trying to get to the same state that you are:
(1) Have a command that runs via python on the command line.
(2) When I run that same command through MET, if fails.
Thanks,
John
On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
> Hey John,
>
> Yeah, mine runs through when I try it using just python. My data is
on my
> local directory in cheyenne - is there an easy way to give you
access to
> it?
>
> On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Taylor,
> >
> > I see you're having trouble with python embedding on Cheyenne. I
logged
> on
> > this morning to try to replicate the errors you're getting.
> >
> > Below are the commands I used to test this:
> >
> >
> > *cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> >
> >
> >
> > *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> > met/8.1_pythonncar_pylib *
> > *wget
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > <
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > >*
> > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> >
> > Since you didn't send me the path to where your data lives on
Cheyenne, I
> > ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your
filename
> > includes "reduced" which I suspect means that some GRIB records
have been
> > excluded. But the one I have contains the full 586 records.
> >
> > The first step with python embedding in MET is to make sure the
script
> runs
> > fine through python itself. But mine produces a runtime error:
> >
> > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> >
> > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > read_GFSv3.py:56: RuntimeWarning: overflow encountered in square
> > uv = np.sqrt(u**2+v**2)
> > Traceback (most recent call last):
> > File "read_GFSv3.py", line 72, in <module>
> > data = np.load(input_file)
> > File
> >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > lin
> > e 431, in load
> > "Failed to interpret file %s as a pickle" % repr(file))
> > IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048' as a
pickle
> >
> > Does yours run without error?
> > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> >
> > If so, please tell me where these files live.
> >
> > Thanks,
> > John
> >
> > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > > Transaction: Ticket created by taylor.mandelbaum at stonybrook.edu
> > > Queue: met_help
> > > Subject: MET 8.1 python embedding failing when using grib2
file
> > > Owner: Nobody
> > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> >
> > >
> > >
> > > Hey all,
> > >
> > > I'm having trouble getting plot_data_plane to work on cheyenne -
I am
> > > trying it using these two commands:
> > >
> > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > >
> > >
> > > and
> > >
> > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > mflux_ana.npy";'
> > > >
> > >
> > > the read_GFSv3.py code is as follows:
> > >
> > > from __future__ import print_function
> > > > import os
> > > > import sys
> > > > import re
> > > > import numpy as np
> > > > import datetime as dt
> > > > import pygrib
> > > >
> > > > def ivt(input_file):
> > > > grbs = pygrib.open(input_file)
> > > > g = 9.81
> > > > grbs.rewind()
> > > >
> > > > levs = []
> > > > for grb in grbs:
> > > > if 'humidity' in grb.parameterName.lower():
> > > > levs.append(grb.level*100)
> > > > levs=np.array(levs)
> > > > grbs.rewind()
> > > >
> > > > q=[]
> > > > for grb in grbs:
> > > > if 'humidity' in grb.parameterName:
> > > > q.append(grb.values)
> > > > q=np.array(q)
> > > > grbs.rewind()
> > > >
> > > > hgt=[]
> > > > for grb in grbs:
> > > > if 'Geopotential' in grb.parameterName:
> > > > hgt.append(grb.values)
> > > > hgt = np.array(hgt)
> > > > grbs.rewind()
> > > >
> > > > temp=[]
> > > > for grb in grbs:
> > > > if 'Temperature' in grb.parameterName:
> > > > temp.append(grb.values)
> > > > temp = np.array(temp)
> > > > grbs.rewind()
> > > >
> > > > u=[]
> > > > for grb in grbs:
> > > > if 'u-' in grb.parameterName:
> > > > u.append(grb.values)
> > > > u = np.array(u)
> > > > grbs.rewind()
> > > >
> > > > v=[]
> > > > for grb in grbs:
> > > > if 'v-' in grb.parameterName:
> > > > v.append(grb.values)
> > > > v = np.array(v)
> > > > grbs.rewind()
> > > >
> > > > uv = np.sqrt(u**2+v**2)
> > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > mflux = np.sum(mflux,axis=0)
> > > > mflux_avg = np.mean(mflux,axis=0)
> > > > return mflux
> > > >
> > > > if len(sys.argv) == 2:
> > > > input_file = os.path.expandvars(sys.argv[1])
> > > > try:
> > > > # Print some output to verify that this script ran
> > > > print("Input File: " + repr(input_file))
> > > >
> > > > # Read input file
> > > > try:
> > > > data = ivt(input_file)
> > > > except:
> > > > data = np.load(input_file)
> > > > print(type(data))
> > > > # Read and re-orient the data
> > > > met_data = data
> > > >
> > > > print("Data Shape: " + repr(met_data.shape))
> > > > print("Data Type: " + repr(met_data.dtype))
> > > > except NameError:
> > > > print("Trouble reading input file: " + input_file)
> > > > else:
> > > > print("Must specify exactly one input file.")
> > > > sys.exit(1)
> > > > valid = dt.datetime(2016,1,24)
> > > >
> > > > attrs = {
> > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > 'lead': '0',
> > > > 'accum': '00',
> > > >
> > > > 'name': 'mflux',
> > > > 'long_name': 'UNKNOWN',
> > > > 'level': '1000_300',
> > > > 'units': 'UNKNOWN',
> > > >
> > > > 'grid': {
> > > > 'name': 'Global 1 Degree',
> > > > 'type' : 'LatLon',
> > > > 'lat_ll' : -90.0,
> > > > 'lon_ll' : 0.0,
> > > > 'delta_lat' : 1.0,
> > > > 'delta_lon' : 1.0,
> > > > 'Nlat' : 181,
> > > > 'Nlon' : 360,
> > > > }
> > > > }
> > > >
> > > >
> > > >
> > > > print("Attributes: " + repr(attrs))
> > > >
> > >
> > > The only difference between the two is the grib2 invokes the ivt
> > function,
> > > where pygrib opens the file. When I try using the grib2 file, I
get a
> > > segmentation fault (core dumped) error. Using the first command
(with
> the
> > > .npy file) works. In both cases, met_data is a numpy array
(which is
> why
> > I
> > > kept PYTHON_NUMPY). Am I missing a flag or another issue I'm
> overlooking?
> > >
> > > Thanks!
> > > --
> > > *Taylor Mandelbaum*
> > > Research Technician
> > > SoMAS | Stony Brook University
> > > 678.468.5789
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: Taylor Mandelbaum
Time: Mon Sep 30 10:33:03 2019
Hey John,
try to cd into /fv3/gfs.20160120/00 and run:
python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
That's what I use after running ncar_pylib to initailize the
virtualenv
with all the relevant packages. That should work... let me know if it
doesn't. I'm also running plot_data_plane from that directory.
On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Taylor,
>
> OK, I went hunting around and found your working directory in:
> /glade/u/home/mandelbaum/test
>
> And I can read data from your directory, so that's fine. Please
send me
> the exact python command that you're running on the command line to
confirm
> that your script works.
>
> When I run it, it does not run fine for me. The first step in
debugging
> this is figuring out exactly what error we need to debug:
>
>
>
>
> *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> met/8.1_pythonncar_pylib python
/glade/u/home/mandelbaum/test/read_GFSv3.py
> gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
>
> Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> Trouble reading input file: gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> Attributes: {'long_name': 'UNKNOWN', 'init': '20160124_000000',
'valid':
> '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
> 'Global 1 Degree', 'Nla
> t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type': 'LatLon',
'Nlon': 360},
> 'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units':
'UNKNOWN',
> 'accum': '00'}
>
> Note that this script is NOT successful in reading the command line
> argument file (Trouble reading input file:...). So it falls into
the
> exception block and reads the analysis file instead.
>
> I'm trying to get to the same state that you are:
> (1) Have a command that runs via python on the command line.
> (2) When I run that same command through MET, if fails.
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> >
> > Hey John,
> >
> > Yeah, mine runs through when I try it using just python. My data
is on my
> > local directory in cheyenne - is there an easy way to give you
access to
> > it?
> >
> > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi Taylor,
> > >
> > > I see you're having trouble with python embedding on Cheyenne.
I
> logged
> > on
> > > this morning to try to replicate the errors you're getting.
> > >
> > > Below are the commands I used to test this:
> > >
> > >
> > > *cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > >
> > >
> > >
> > > *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> > > met/8.1_pythonncar_pylib *
> > > *wget
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > <
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > >*
> > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > >
> > > Since you didn't send me the path to where your data lives on
> Cheyenne, I
> > > ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your
filename
> > > includes "reduced" which I suspect means that some GRIB records
have
> been
> > > excluded. But the one I have contains the full 586 records.
> > >
> > > The first step with python embedding in MET is to make sure the
script
> > runs
> > > fine through python itself. But mine produces a runtime error:
> > >
> > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > >
> > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > read_GFSv3.py:56: RuntimeWarning: overflow encountered in square
> > > uv = np.sqrt(u**2+v**2)
> > > Traceback (most recent call last):
> > > File "read_GFSv3.py", line 72, in <module>
> > > data = np.load(input_file)
> > > File
> > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > lin
> > > e 431, in load
> > > "Failed to interpret file %s as a pickle" % repr(file))
> > > IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048' as
a
> pickle
> > >
> > > Does yours run without error?
> > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > >
> > > If so, please tell me where these files live.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > > > Transaction: Ticket created by
taylor.mandelbaum at stonybrook.edu
> > > > Queue: met_help
> > > > Subject: MET 8.1 python embedding failing when using
grib2 file
> > > > Owner: Nobody
> > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > >
> > > >
> > > >
> > > > Hey all,
> > > >
> > > > I'm having trouble getting plot_data_plane to work on cheyenne
- I am
> > > > trying it using these two commands:
> > > >
> > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > >
> > > >
> > > > and
> > > >
> > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > mflux_ana.npy";'
> > > > >
> > > >
> > > > the read_GFSv3.py code is as follows:
> > > >
> > > > from __future__ import print_function
> > > > > import os
> > > > > import sys
> > > > > import re
> > > > > import numpy as np
> > > > > import datetime as dt
> > > > > import pygrib
> > > > >
> > > > > def ivt(input_file):
> > > > > grbs = pygrib.open(input_file)
> > > > > g = 9.81
> > > > > grbs.rewind()
> > > > >
> > > > > levs = []
> > > > > for grb in grbs:
> > > > > if 'humidity' in grb.parameterName.lower():
> > > > > levs.append(grb.level*100)
> > > > > levs=np.array(levs)
> > > > > grbs.rewind()
> > > > >
> > > > > q=[]
> > > > > for grb in grbs:
> > > > > if 'humidity' in grb.parameterName:
> > > > > q.append(grb.values)
> > > > > q=np.array(q)
> > > > > grbs.rewind()
> > > > >
> > > > > hgt=[]
> > > > > for grb in grbs:
> > > > > if 'Geopotential' in grb.parameterName:
> > > > > hgt.append(grb.values)
> > > > > hgt = np.array(hgt)
> > > > > grbs.rewind()
> > > > >
> > > > > temp=[]
> > > > > for grb in grbs:
> > > > > if 'Temperature' in grb.parameterName:
> > > > > temp.append(grb.values)
> > > > > temp = np.array(temp)
> > > > > grbs.rewind()
> > > > >
> > > > > u=[]
> > > > > for grb in grbs:
> > > > > if 'u-' in grb.parameterName:
> > > > > u.append(grb.values)
> > > > > u = np.array(u)
> > > > > grbs.rewind()
> > > > >
> > > > > v=[]
> > > > > for grb in grbs:
> > > > > if 'v-' in grb.parameterName:
> > > > > v.append(grb.values)
> > > > > v = np.array(v)
> > > > > grbs.rewind()
> > > > >
> > > > > uv = np.sqrt(u**2+v**2)
> > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > mflux = np.sum(mflux,axis=0)
> > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > return mflux
> > > > >
> > > > > if len(sys.argv) == 2:
> > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > try:
> > > > > # Print some output to verify that this script ran
> > > > > print("Input File: " + repr(input_file))
> > > > >
> > > > > # Read input file
> > > > > try:
> > > > > data = ivt(input_file)
> > > > > except:
> > > > > data = np.load(input_file)
> > > > > print(type(data))
> > > > > # Read and re-orient the data
> > > > > met_data = data
> > > > >
> > > > > print("Data Shape: " + repr(met_data.shape))
> > > > > print("Data Type: " + repr(met_data.dtype))
> > > > > except NameError:
> > > > > print("Trouble reading input file: " + input_file)
> > > > > else:
> > > > > print("Must specify exactly one input file.")
> > > > > sys.exit(1)
> > > > > valid = dt.datetime(2016,1,24)
> > > > >
> > > > > attrs = {
> > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > 'lead': '0',
> > > > > 'accum': '00',
> > > > >
> > > > > 'name': 'mflux',
> > > > > 'long_name': 'UNKNOWN',
> > > > > 'level': '1000_300',
> > > > > 'units': 'UNKNOWN',
> > > > >
> > > > > 'grid': {
> > > > > 'name': 'Global 1 Degree',
> > > > > 'type' : 'LatLon',
> > > > > 'lat_ll' : -90.0,
> > > > > 'lon_ll' : 0.0,
> > > > > 'delta_lat' : 1.0,
> > > > > 'delta_lon' : 1.0,
> > > > > 'Nlat' : 181,
> > > > > 'Nlon' : 360,
> > > > > }
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > print("Attributes: " + repr(attrs))
> > > > >
> > > >
> > > > The only difference between the two is the grib2 invokes the
ivt
> > > function,
> > > > where pygrib opens the file. When I try using the grib2 file,
I get a
> > > > segmentation fault (core dumped) error. Using the first
command (with
> > the
> > > > .npy file) works. In both cases, met_data is a numpy array
(which is
> > why
> > > I
> > > > kept PYTHON_NUMPY). Am I missing a flag or another issue I'm
> > overlooking?
> > > >
> > > > Thanks!
> > > > --
> > > > *Taylor Mandelbaum*
> > > > Research Technician
> > > > SoMAS | Stony Brook University
> > > > 678.468.5789
> > > >
> > > >
> > >
> > >
> >
> >
>
>
--
*Taylor Mandelbaum*
Research Technician
SoMAS | Stony Brook University
678.468.5789
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: John Halley Gotway
Time: Mon Sep 30 11:25:02 2019
Taylor,
OK, I'm able to replicate the behavior you describe... running the
script
fine with the interpreter but getting a core dump via plot_data_plane.
The
MET source code links to the GRIB2 library, as does the pygrib python
package. This is similar to the behavior for HDF5 on Cheyenne as
well...
getting a segfault from MET when the python script loads the h5py
package.
I suspect there's a conflict with the linker when the python script
loads
packages which require libraries that were also required to build MET.
But unfortunately I don't have a fix. I'll raise this issue for
discussion
at the MET development meeting tomorrow.
Thanks,
John
On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
> Hey John,
>
> try to cd into /fv3/gfs.20160120/00 and run:
>
> python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
>
> That's what I use after running ncar_pylib to initailize the
virtualenv
> with all the relevant packages. That should work... let me know if
it
> doesn't. I'm also running plot_data_plane from that directory.
>
> On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Taylor,
> >
> > OK, I went hunting around and found your working directory in:
> > /glade/u/home/mandelbaum/test
> >
> > And I can read data from your directory, so that's fine. Please
send me
> > the exact python command that you're running on the command line
to
> confirm
> > that your script works.
> >
> > When I run it, it does not run fine for me. The first step in
debugging
> > this is figuring out exactly what error we need to debug:
> >
> >
> >
> >
> > *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> > met/8.1_pythonncar_pylib python
> /glade/u/home/mandelbaum/test/read_GFSv3.py
> > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> >
> > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > Trouble reading input file: gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > Attributes: {'long_name': 'UNKNOWN', 'init': '20160124_000000',
'valid':
> > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > 'Global 1 Degree', 'Nla
> > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type': 'LatLon',
'Nlon':
> 360},
> > 'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units':
'UNKNOWN',
> > 'accum': '00'}
> >
> > Note that this script is NOT successful in reading the command
line
> > argument file (Trouble reading input file:...). So it falls into
the
> > exception block and reads the analysis file instead.
> >
> > I'm trying to get to the same state that you are:
> > (1) Have a command that runs via python on the command line.
> > (2) When I run that same command through MET, if fails.
> >
> > Thanks,
> > John
> >
> > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > >
> > > Hey John,
> > >
> > > Yeah, mine runs through when I try it using just python. My data
is on
> my
> > > local directory in cheyenne - is there an easy way to give you
access
> to
> > > it?
> > >
> > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Hi Taylor,
> > > >
> > > > I see you're having trouble with python embedding on Cheyenne.
I
> > logged
> > > on
> > > > this morning to try to replicate the errors you're getting.
> > > >
> > > > Below are the commands I used to test this:
> > > >
> > > >
> > > > *cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > >
> > > >
> > > >
> > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule load
> > > > met/8.1_pythonncar_pylib *
> > > > *wget
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > <
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > >*
> > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > >
> > > > Since you didn't send me the path to where your data lives on
> > Cheyenne, I
> > > > ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your
filename
> > > > includes "reduced" which I suspect means that some GRIB
records have
> > been
> > > > excluded. But the one I have contains the full 586 records.
> > > >
> > > > The first step with python embedding in MET is to make sure
the
> script
> > > runs
> > > > fine through python itself. But mine produces a runtime
error:
> > > >
> > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > >
> > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > read_GFSv3.py:56: RuntimeWarning: overflow encountered in
square
> > > > uv = np.sqrt(u**2+v**2)
> > > > Traceback (most recent call last):
> > > > File "read_GFSv3.py", line 72, in <module>
> > > > data = np.load(input_file)
> > > > File
> > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > lin
> > > > e 431, in load
> > > > "Failed to interpret file %s as a pickle" % repr(file))
> > > > IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048'
as a
> > pickle
> > > >
> > > > Does yours run without error?
> > > > *python read_GFSv3.py
gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > >
> > > > If so, please tell me where these files live.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > > > > Transaction: Ticket created by
taylor.mandelbaum at stonybrook.edu
> > > > > Queue: met_help
> > > > > Subject: MET 8.1 python embedding failing when using
grib2
> file
> > > > > Owner: Nobody
> > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > >
> > > > >
> > > > >
> > > > > Hey all,
> > > > >
> > > > > I'm having trouble getting plot_data_plane to work on
cheyenne - I
> am
> > > > > trying it using these two commands:
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > >
> > > > >
> > > > > and
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > > mflux_ana.npy";'
> > > > > >
> > > > >
> > > > > the read_GFSv3.py code is as follows:
> > > > >
> > > > > from __future__ import print_function
> > > > > > import os
> > > > > > import sys
> > > > > > import re
> > > > > > import numpy as np
> > > > > > import datetime as dt
> > > > > > import pygrib
> > > > > >
> > > > > > def ivt(input_file):
> > > > > > grbs = pygrib.open(input_file)
> > > > > > g = 9.81
> > > > > > grbs.rewind()
> > > > > >
> > > > > > levs = []
> > > > > > for grb in grbs:
> > > > > > if 'humidity' in grb.parameterName.lower():
> > > > > > levs.append(grb.level*100)
> > > > > > levs=np.array(levs)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > q=[]
> > > > > > for grb in grbs:
> > > > > > if 'humidity' in grb.parameterName:
> > > > > > q.append(grb.values)
> > > > > > q=np.array(q)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > hgt=[]
> > > > > > for grb in grbs:
> > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > hgt.append(grb.values)
> > > > > > hgt = np.array(hgt)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > temp=[]
> > > > > > for grb in grbs:
> > > > > > if 'Temperature' in grb.parameterName:
> > > > > > temp.append(grb.values)
> > > > > > temp = np.array(temp)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > u=[]
> > > > > > for grb in grbs:
> > > > > > if 'u-' in grb.parameterName:
> > > > > > u.append(grb.values)
> > > > > > u = np.array(u)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > v=[]
> > > > > > for grb in grbs:
> > > > > > if 'v-' in grb.parameterName:
> > > > > > v.append(grb.values)
> > > > > > v = np.array(v)
> > > > > > grbs.rewind()
> > > > > >
> > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > return mflux
> > > > > >
> > > > > > if len(sys.argv) == 2:
> > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > try:
> > > > > > # Print some output to verify that this script ran
> > > > > > print("Input File: " + repr(input_file))
> > > > > >
> > > > > > # Read input file
> > > > > > try:
> > > > > > data = ivt(input_file)
> > > > > > except:
> > > > > > data = np.load(input_file)
> > > > > > print(type(data))
> > > > > > # Read and re-orient the data
> > > > > > met_data = data
> > > > > >
> > > > > > print("Data Shape: " + repr(met_data.shape))
> > > > > > print("Data Type: " + repr(met_data.dtype))
> > > > > > except NameError:
> > > > > > print("Trouble reading input file: " + input_file)
> > > > > > else:
> > > > > > print("Must specify exactly one input file.")
> > > > > > sys.exit(1)
> > > > > > valid = dt.datetime(2016,1,24)
> > > > > >
> > > > > > attrs = {
> > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > 'lead': '0',
> > > > > > 'accum': '00',
> > > > > >
> > > > > > 'name': 'mflux',
> > > > > > 'long_name': 'UNKNOWN',
> > > > > > 'level': '1000_300',
> > > > > > 'units': 'UNKNOWN',
> > > > > >
> > > > > > 'grid': {
> > > > > > 'name': 'Global 1 Degree',
> > > > > > 'type' : 'LatLon',
> > > > > > 'lat_ll' : -90.0,
> > > > > > 'lon_ll' : 0.0,
> > > > > > 'delta_lat' : 1.0,
> > > > > > 'delta_lon' : 1.0,
> > > > > > 'Nlat' : 181,
> > > > > > 'Nlon' : 360,
> > > > > > }
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > print("Attributes: " + repr(attrs))
> > > > > >
> > > > >
> > > > > The only difference between the two is the grib2 invokes the
ivt
> > > > function,
> > > > > where pygrib opens the file. When I try using the grib2
file, I
> get a
> > > > > segmentation fault (core dumped) error. Using the first
command
> (with
> > > the
> > > > > .npy file) works. In both cases, met_data is a numpy array
(which
> is
> > > why
> > > > I
> > > > > kept PYTHON_NUMPY). Am I missing a flag or another issue I'm
> > > overlooking?
> > > > >
> > > > > Thanks!
> > > > > --
> > > > > *Taylor Mandelbaum*
> > > > > Research Technician
> > > > > SoMAS | Stony Brook University
> > > > > 678.468.5789
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> *Taylor Mandelbaum*
> Research Technician
> SoMAS | Stony Brook University
> 678.468.5789
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: Taylor Mandelbaum
Time: Mon Sep 30 12:01:39 2019
Thanks John - I'm going to try a workaround that invokes a separate
function which will load in the grib and transform it to a numpy
array. I
don't think it will work if it's being run in the same instance of
python
as read_GFSv3.py through MET, but it can't hurt to try.
On Mon, Sep 30, 2019 at 1:25 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Taylor,
>
> OK, I'm able to replicate the behavior you describe... running the
script
> fine with the interpreter but getting a core dump via
plot_data_plane. The
> MET source code links to the GRIB2 library, as does the pygrib
python
> package. This is similar to the behavior for HDF5 on Cheyenne as
well...
> getting a segfault from MET when the python script loads the h5py
package.
> I suspect there's a conflict with the linker when the python script
loads
> packages which require libraries that were also required to build
MET.
>
> But unfortunately I don't have a fix. I'll raise this issue for
discussion
> at the MET development meeting tomorrow.
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> >
> > Hey John,
> >
> > try to cd into /fv3/gfs.20160120/00 and run:
> >
> > python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> >
> > That's what I use after running ncar_pylib to initailize the
virtualenv
> > with all the relevant packages. That should work... let me know if
it
> > doesn't. I'm also running plot_data_plane from that directory.
> >
> > On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Taylor,
> > >
> > > OK, I went hunting around and found your working directory in:
> > > /glade/u/home/mandelbaum/test
> > >
> > > And I can read data from your directory, so that's fine. Please
send
> me
> > > the exact python command that you're running on the command line
to
> > confirm
> > > that your script works.
> > >
> > > When I run it, it does not run fine for me. The first step in
> debugging
> > > this is figuring out exactly what error we need to debug:
> > >
> > >
> > >
> > >
> > > *module use /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
load
> > > met/8.1_pythonncar_pylib python
> > /glade/u/home/mandelbaum/test/read_GFSv3.py
> > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> > >
> > > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > > Trouble reading input file:
gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > Attributes: {'long_name': 'UNKNOWN', 'init': '20160124_000000',
> 'valid':
> > > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > > 'Global 1 Degree', 'Nla
> > > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type': 'LatLon',
'Nlon':
> > 360},
> > > 'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units':
'UNKNOWN',
> > > 'accum': '00'}
> > >
> > > Note that this script is NOT successful in reading the command
line
> > > argument file (Trouble reading input file:...). So it falls
into the
> > > exception block and reads the analysis file instead.
> > >
> > > I'm trying to get to the same state that you are:
> > > (1) Have a command that runs via python on the command line.
> > > (2) When I run that same command through MET, if fails.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
>
> > > >
> > > > Hey John,
> > > >
> > > > Yeah, mine runs through when I try it using just python. My
data is
> on
> > my
> > > > local directory in cheyenne - is there an easy way to give you
access
> > to
> > > > it?
> > > >
> > > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > > Hi Taylor,
> > > > >
> > > > > I see you're having trouble with python embedding on
Cheyenne. I
> > > logged
> > > > on
> > > > > this morning to try to replicate the errors you're getting.
> > > > >
> > > > > Below are the commands I used to test this:
> > > > >
> > > > >
> > > > > *cd /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > > >
> > > > >
> > > > >
> > > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> load
> > > > > met/8.1_pythonncar_pylib *
> > > > > *wget
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > <
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > >*
> > > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > > >
> > > > > Since you didn't send me the path to where your data lives
on
> > > Cheyenne, I
> > > > > ran wget to pull a similar 1.0 degree GFS GRIB2 file. Your
> filename
> > > > > includes "reduced" which I suspect means that some GRIB
records
> have
> > > been
> > > > > excluded. But the one I have contains the full 586 records.
> > > > >
> > > > > The first step with python embedding in MET is to make sure
the
> > script
> > > > runs
> > > > > fine through python itself. But mine produces a runtime
error:
> > > > >
> > > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > > >
> > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > > read_GFSv3.py:56: RuntimeWarning: overflow encountered in
square
> > > > > uv = np.sqrt(u**2+v**2)
> > > > > Traceback (most recent call last):
> > > > > File "read_GFSv3.py", line 72, in <module>
> > > > > data = np.load(input_file)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > > lin
> > > > > e 431, in load
> > > > > "Failed to interpret file %s as a pickle" % repr(file))
> > > > > IOError: Failed to interpret file 'gfs.t00z.pgrb2.1p00.f048'
as a
> > > pickle
> > > > >
> > > > > Does yours run without error?
> > > > > *python read_GFSv3.py
gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > > >
> > > > > If so, please tell me where these files live.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > > > > > Transaction: Ticket created by
taylor.mandelbaum at stonybrook.edu
> > > > > > Queue: met_help
> > > > > > Subject: MET 8.1 python embedding failing when using
grib2
> > file
> > > > > > Owner: Nobody
> > > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > > >
> > > > > >
> > > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I'm having trouble getting plot_data_plane to work on
cheyenne -
> I
> > am
> > > > > > trying it using these two commands:
> > > > > >
> > > > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > > >
> > > > > >
> > > > > > and
> > > > > >
> > > > > > plot_data_plane PYTHON_NUMPY mflux.ps 'name="read_GFSv3.py
> > > > > > mflux_ana.npy";'
> > > > > > >
> > > > > >
> > > > > > the read_GFSv3.py code is as follows:
> > > > > >
> > > > > > from __future__ import print_function
> > > > > > > import os
> > > > > > > import sys
> > > > > > > import re
> > > > > > > import numpy as np
> > > > > > > import datetime as dt
> > > > > > > import pygrib
> > > > > > >
> > > > > > > def ivt(input_file):
> > > > > > > grbs = pygrib.open(input_file)
> > > > > > > g = 9.81
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > levs = []
> > > > > > > for grb in grbs:
> > > > > > > if 'humidity' in grb.parameterName.lower():
> > > > > > > levs.append(grb.level*100)
> > > > > > > levs=np.array(levs)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > q=[]
> > > > > > > for grb in grbs:
> > > > > > > if 'humidity' in grb.parameterName:
> > > > > > > q.append(grb.values)
> > > > > > > q=np.array(q)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > hgt=[]
> > > > > > > for grb in grbs:
> > > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > > hgt.append(grb.values)
> > > > > > > hgt = np.array(hgt)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > temp=[]
> > > > > > > for grb in grbs:
> > > > > > > if 'Temperature' in grb.parameterName:
> > > > > > > temp.append(grb.values)
> > > > > > > temp = np.array(temp)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > u=[]
> > > > > > > for grb in grbs:
> > > > > > > if 'u-' in grb.parameterName:
> > > > > > > u.append(grb.values)
> > > > > > > u = np.array(u)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > v=[]
> > > > > > > for grb in grbs:
> > > > > > > if 'v-' in grb.parameterName:
> > > > > > > v.append(grb.values)
> > > > > > > v = np.array(v)
> > > > > > > grbs.rewind()
> > > > > > >
> > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > > return mflux
> > > > > > >
> > > > > > > if len(sys.argv) == 2:
> > > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > > try:
> > > > > > > # Print some output to verify that this script
ran
> > > > > > > print("Input File: " + repr(input_file))
> > > > > > >
> > > > > > > # Read input file
> > > > > > > try:
> > > > > > > data = ivt(input_file)
> > > > > > > except:
> > > > > > > data = np.load(input_file)
> > > > > > > print(type(data))
> > > > > > > # Read and re-orient the data
> > > > > > > met_data = data
> > > > > > >
> > > > > > > print("Data Shape: " + repr(met_data.shape))
> > > > > > > print("Data Type: " + repr(met_data.dtype))
> > > > > > > except NameError:
> > > > > > > print("Trouble reading input file: " +
input_file)
> > > > > > > else:
> > > > > > > print("Must specify exactly one input file.")
> > > > > > > sys.exit(1)
> > > > > > > valid = dt.datetime(2016,1,24)
> > > > > > >
> > > > > > > attrs = {
> > > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > 'lead': '0',
> > > > > > > 'accum': '00',
> > > > > > >
> > > > > > > 'name': 'mflux',
> > > > > > > 'long_name': 'UNKNOWN',
> > > > > > > 'level': '1000_300',
> > > > > > > 'units': 'UNKNOWN',
> > > > > > >
> > > > > > > 'grid': {
> > > > > > > 'name': 'Global 1 Degree',
> > > > > > > 'type' : 'LatLon',
> > > > > > > 'lat_ll' : -90.0,
> > > > > > > 'lon_ll' : 0.0,
> > > > > > > 'delta_lat' : 1.0,
> > > > > > > 'delta_lon' : 1.0,
> > > > > > > 'Nlat' : 181,
> > > > > > > 'Nlon' : 360,
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > print("Attributes: " + repr(attrs))
> > > > > > >
> > > > > >
> > > > > > The only difference between the two is the grib2 invokes
the ivt
> > > > > function,
> > > > > > where pygrib opens the file. When I try using the grib2
file, I
> > get a
> > > > > > segmentation fault (core dumped) error. Using the first
command
> > (with
> > > > the
> > > > > > .npy file) works. In both cases, met_data is a numpy array
(which
> > is
> > > > why
> > > > > I
> > > > > > kept PYTHON_NUMPY). Am I missing a flag or another issue
I'm
> > > > overlooking?
> > > > > >
> > > > > > Thanks!
> > > > > > --
> > > > > > *Taylor Mandelbaum*
> > > > > > Research Technician
> > > > > > SoMAS | Stony Brook University
> > > > > > 678.468.5789
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > *Taylor Mandelbaum*
> > Research Technician
> > SoMAS | Stony Brook University
> > 678.468.5789
> >
> >
>
>
--
*Taylor Mandelbaum*
Research Technician
SoMAS | Stony Brook University
678.468.5789
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: John Halley Gotway
Time: Mon Sep 30 15:10:55 2019
Taylor,
OK, glad you have a temporary workaround for now. FYI, I wrote up
this
GitHub issue for MET with 2 specific examples to debug on cheyenne.
https://github.com/NCAR/MET/issues/1205
Thanks,
John
On Mon, Sep 30, 2019 at 12:02 PM Taylor Mandelbaum via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
> Thanks John - I'm going to try a workaround that invokes a separate
> function which will load in the grib and transform it to a numpy
array. I
> don't think it will work if it's being run in the same instance of
python
> as read_GFSv3.py through MET, but it can't hurt to try.
>
> On Mon, Sep 30, 2019 at 1:25 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Taylor,
> >
> > OK, I'm able to replicate the behavior you describe... running the
script
> > fine with the interpreter but getting a core dump via
plot_data_plane.
> The
> > MET source code links to the GRIB2 library, as does the pygrib
python
> > package. This is similar to the behavior for HDF5 on Cheyenne as
well...
> > getting a segfault from MET when the python script loads the h5py
> package.
> > I suspect there's a conflict with the linker when the python
script loads
> > packages which require libraries that were also required to build
MET.
> >
> > But unfortunately I don't have a fix. I'll raise this issue for
> discussion
> > at the MET development meeting tomorrow.
> >
> > Thanks,
> > John
> >
> > On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > >
> > > Hey John,
> > >
> > > try to cd into /fv3/gfs.20160120/00 and run:
> > >
> > > python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > >
> > > That's what I use after running ncar_pylib to initailize the
virtualenv
> > > with all the relevant packages. That should work... let me know
if it
> > > doesn't. I'm also running plot_data_plane from that directory.
> > >
> > > On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Taylor,
> > > >
> > > > OK, I went hunting around and found your working directory in:
> > > > /glade/u/home/mandelbaum/test
> > > >
> > > > And I can read data from your directory, so that's fine.
Please send
> > me
> > > > the exact python command that you're running on the command
line to
> > > confirm
> > > > that your script works.
> > > >
> > > > When I run it, it does not run fine for me. The first step in
> > debugging
> > > > this is figuring out exactly what error we need to debug:
> > > >
> > > >
> > > >
> > > >
> > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule load
> > > > met/8.1_pythonncar_pylib python
> > > /glade/u/home/mandelbaum/test/read_GFSv3.py
> > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> > > >
> > > > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > > > Trouble reading input file:
gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > Attributes: {'long_name': 'UNKNOWN', 'init':
'20160124_000000',
> > 'valid':
> > > > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > > > 'Global 1 Degree', 'Nla
> > > > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type': 'LatLon',
'Nlon':
> > > 360},
> > > > 'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units':
> 'UNKNOWN',
> > > > 'accum': '00'}
> > > >
> > > > Note that this script is NOT successful in reading the command
line
> > > > argument file (Trouble reading input file:...). So it falls
into the
> > > > exception block and reads the analysis file instead.
> > > >
> > > > I'm trying to get to the same state that you are:
> > > > (1) Have a command that runs via python on the command line.
> > > > (2) When I run that same command through MET, if fails.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Yeah, mine runs through when I try it using just python. My
data is
> > on
> > > my
> > > > > local directory in cheyenne - is there an easy way to give
you
> access
> > > to
> > > > > it?
> > > > >
> > > > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Taylor,
> > > > > >
> > > > > > I see you're having trouble with python embedding on
Cheyenne. I
> > > > logged
> > > > > on
> > > > > > this morning to try to replicate the errors you're
getting.
> > > > > >
> > > > > > Below are the commands I used to test this:
> > > > > >
> > > > > >
> > > > > > *cd
/glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > load
> > > > > > met/8.1_pythonncar_pylib *
> > > > > > *wget
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > >*
> > > > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > > > >
> > > > > > Since you didn't send me the path to where your data lives
on
> > > > Cheyenne, I
> > > > > > ran wget to pull a similar 1.0 degree GFS GRIB2 file.
Your
> > filename
> > > > > > includes "reduced" which I suspect means that some GRIB
records
> > have
> > > > been
> > > > > > excluded. But the one I have contains the full 586
records.
> > > > > >
> > > > > > The first step with python embedding in MET is to make
sure the
> > > script
> > > > > runs
> > > > > > fine through python itself. But mine produces a runtime
error:
> > > > > >
> > > > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > > > >
> > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > > > read_GFSv3.py:56: RuntimeWarning: overflow encountered in
square
> > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > Traceback (most recent call last):
> > > > > > File "read_GFSv3.py", line 72, in <module>
> > > > > > data = np.load(input_file)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > > > lin
> > > > > > e 431, in load
> > > > > > "Failed to interpret file %s as a pickle" % repr(file))
> > > > > > IOError: Failed to interpret file
'gfs.t00z.pgrb2.1p00.f048' as a
> > > > pickle
> > > > > >
> > > > > > Does yours run without error?
> > > > > > *python read_GFSv3.py
gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > > > >
> > > > > > If so, please tell me where these files live.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted upon.
> > > > > > > Transaction: Ticket created by
> taylor.mandelbaum at stonybrook.edu
> > > > > > > Queue: met_help
> > > > > > > Subject: MET 8.1 python embedding failing when
using grib2
> > > file
> > > > > > > Owner: Nobody
> > > > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > I'm having trouble getting plot_data_plane to work on
cheyenne
> -
> > I
> > > am
> > > > > > > trying it using these two commands:
> > > > > > >
> > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > > > >
> > > > > > >
> > > > > > > and
> > > > > > >
> > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > mflux_ana.npy";'
> > > > > > > >
> > > > > > >
> > > > > > > the read_GFSv3.py code is as follows:
> > > > > > >
> > > > > > > from __future__ import print_function
> > > > > > > > import os
> > > > > > > > import sys
> > > > > > > > import re
> > > > > > > > import numpy as np
> > > > > > > > import datetime as dt
> > > > > > > > import pygrib
> > > > > > > >
> > > > > > > > def ivt(input_file):
> > > > > > > > grbs = pygrib.open(input_file)
> > > > > > > > g = 9.81
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > levs = []
> > > > > > > > for grb in grbs:
> > > > > > > > if 'humidity' in grb.parameterName.lower():
> > > > > > > > levs.append(grb.level*100)
> > > > > > > > levs=np.array(levs)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > q=[]
> > > > > > > > for grb in grbs:
> > > > > > > > if 'humidity' in grb.parameterName:
> > > > > > > > q.append(grb.values)
> > > > > > > > q=np.array(q)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > hgt=[]
> > > > > > > > for grb in grbs:
> > > > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > > > hgt.append(grb.values)
> > > > > > > > hgt = np.array(hgt)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > temp=[]
> > > > > > > > for grb in grbs:
> > > > > > > > if 'Temperature' in grb.parameterName:
> > > > > > > > temp.append(grb.values)
> > > > > > > > temp = np.array(temp)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > u=[]
> > > > > > > > for grb in grbs:
> > > > > > > > if 'u-' in grb.parameterName:
> > > > > > > > u.append(grb.values)
> > > > > > > > u = np.array(u)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > v=[]
> > > > > > > > for grb in grbs:
> > > > > > > > if 'v-' in grb.parameterName:
> > > > > > > > v.append(grb.values)
> > > > > > > > v = np.array(v)
> > > > > > > > grbs.rewind()
> > > > > > > >
> > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > > > return mflux
> > > > > > > >
> > > > > > > > if len(sys.argv) == 2:
> > > > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > > > try:
> > > > > > > > # Print some output to verify that this script
ran
> > > > > > > > print("Input File: " + repr(input_file))
> > > > > > > >
> > > > > > > > # Read input file
> > > > > > > > try:
> > > > > > > > data = ivt(input_file)
> > > > > > > > except:
> > > > > > > > data = np.load(input_file)
> > > > > > > > print(type(data))
> > > > > > > > # Read and re-orient the data
> > > > > > > > met_data = data
> > > > > > > >
> > > > > > > > print("Data Shape: " + repr(met_data.shape))
> > > > > > > > print("Data Type: " + repr(met_data.dtype))
> > > > > > > > except NameError:
> > > > > > > > print("Trouble reading input file: " +
input_file)
> > > > > > > > else:
> > > > > > > > print("Must specify exactly one input file.")
> > > > > > > > sys.exit(1)
> > > > > > > > valid = dt.datetime(2016,1,24)
> > > > > > > >
> > > > > > > > attrs = {
> > > > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > 'lead': '0',
> > > > > > > > 'accum': '00',
> > > > > > > >
> > > > > > > > 'name': 'mflux',
> > > > > > > > 'long_name': 'UNKNOWN',
> > > > > > > > 'level': '1000_300',
> > > > > > > > 'units': 'UNKNOWN',
> > > > > > > >
> > > > > > > > 'grid': {
> > > > > > > > 'name': 'Global 1 Degree',
> > > > > > > > 'type' : 'LatLon',
> > > > > > > > 'lat_ll' : -90.0,
> > > > > > > > 'lon_ll' : 0.0,
> > > > > > > > 'delta_lat' : 1.0,
> > > > > > > > 'delta_lon' : 1.0,
> > > > > > > > 'Nlat' : 181,
> > > > > > > > 'Nlon' : 360,
> > > > > > > > }
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > print("Attributes: " + repr(attrs))
> > > > > > > >
> > > > > > >
> > > > > > > The only difference between the two is the grib2 invokes
the
> ivt
> > > > > > function,
> > > > > > > where pygrib opens the file. When I try using the grib2
file, I
> > > get a
> > > > > > > segmentation fault (core dumped) error. Using the first
command
> > > (with
> > > > > the
> > > > > > > .npy file) works. In both cases, met_data is a numpy
array
> (which
> > > is
> > > > > why
> > > > > > I
> > > > > > > kept PYTHON_NUMPY). Am I missing a flag or another issue
I'm
> > > > > overlooking?
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --
> > > > > > > *Taylor Mandelbaum*
> > > > > > > Research Technician
> > > > > > > SoMAS | Stony Brook University
> > > > > > > 678.468.5789
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > *Taylor Mandelbaum*
> > > Research Technician
> > > SoMAS | Stony Brook University
> > > 678.468.5789
> > >
> > >
> >
> >
>
> --
> *Taylor Mandelbaum*
> Research Technician
> SoMAS | Stony Brook University
> 678.468.5789
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: George McCabe
Time: Thu Oct 10 19:28:59 2019
Hi Taylor,
We are exploring options to be able to handle python embedding without
running into these conflicts. Did your workaround work? If not, I
wrote a
simple python script that can be used as a workaround. It is called
outside
of METplus to generate a .py file that contains the data and
attributes
that can be called in the python embedding via METplus. The script it
attached. If you were previously setting:
*name="my_script.py filename args"*
in the MET call, you can run
*python write_data.py <OUTPUT_FILENAME> my_script.py filename args*
using a version of python that will allow you to use the modules that
were
failing inside MET. The path specified by <OUTPUT_FILENAME> (must end
with
.py) will be a file that you can pass into MET like this:
name="<OUTPUT_FILENAME>"
Let me know if that works for you or if you have any questions.
Thanks,
George
------------------------------------------------------------
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
------------------------------------------------------------
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
On Mon, Sep 30, 2019 at 3:11 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
> Taylor,
>
> OK, glad you have a temporary workaround for now. FYI, I wrote up
this
> GitHub issue for MET with 2 specific examples to debug on cheyenne.
>
> https://github.com/NCAR/MET/issues/1205
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 12:02 PM Taylor Mandelbaum via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> >
> > Thanks John - I'm going to try a workaround that invokes a
separate
> > function which will load in the grib and transform it to a numpy
array. I
> > don't think it will work if it's being run in the same instance of
python
> > as read_GFSv3.py through MET, but it can't hurt to try.
> >
> > On Mon, Sep 30, 2019 at 1:25 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Taylor,
> > >
> > > OK, I'm able to replicate the behavior you describe... running
the
> script
> > > fine with the interpreter but getting a core dump via
plot_data_plane.
> > The
> > > MET source code links to the GRIB2 library, as does the pygrib
python
> > > package. This is similar to the behavior for HDF5 on Cheyenne
as
> well...
> > > getting a segfault from MET when the python script loads the
h5py
> > package.
> > > I suspect there's a conflict with the linker when the python
script
> loads
> > > packages which require libraries that were also required to
build MET.
> > >
> > > But unfortunately I don't have a fix. I'll raise this issue for
> > discussion
> > > at the MET development meeting tomorrow.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
>
> > > >
> > > > Hey John,
> > > >
> > > > try to cd into /fv3/gfs.20160120/00 and run:
> > > >
> > > > python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > >
> > > > That's what I use after running ncar_pylib to initailize the
> virtualenv
> > > > with all the relevant packages. That should work... let me
know if it
> > > > doesn't. I'm also running plot_data_plane from that directory.
> > > >
> > > > On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Taylor,
> > > > >
> > > > > OK, I went hunting around and found your working directory
in:
> > > > > /glade/u/home/mandelbaum/test
> > > > >
> > > > > And I can read data from your directory, so that's fine.
Please
> send
> > > me
> > > > > the exact python command that you're running on the command
line to
> > > > confirm
> > > > > that your script works.
> > > > >
> > > > > When I run it, it does not run fine for me. The first step
in
> > > debugging
> > > > > this is figuring out exactly what error we need to debug:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> load
> > > > > met/8.1_pythonncar_pylib python
> > > > /glade/u/home/mandelbaum/test/read_GFSv3.py
> > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> > > > >
> > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > > > > Trouble reading input file:
gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > > Attributes: {'long_name': 'UNKNOWN', 'init':
'20160124_000000',
> > > 'valid':
> > > > > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
> 'name':
> > > > > 'Global 1 Degree', 'Nla
> > > > > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type':
'LatLon',
> 'Nlon':
> > > > 360},
> > > > > 'name': 'mflux', 'lead': '0', 'level': '1000_300', 'units':
> > 'UNKNOWN',
> > > > > 'accum': '00'}
> > > > >
> > > > > Note that this script is NOT successful in reading the
command line
> > > > > argument file (Trouble reading input file:...). So it falls
into
> the
> > > > > exception block and reads the analysis file instead.
> > > > >
> > > > > I'm trying to get to the same state that you are:
> > > > > (1) Have a command that runs via python on the command line.
> > > > > (2) When I run that same command through MET, if fails.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > > > > >
> > > > > > Hey John,
> > > > > >
> > > > > > Yeah, mine runs through when I try it using just python.
My data
> is
> > > on
> > > > my
> > > > > > local directory in cheyenne - is there an easy way to give
you
> > access
> > > > to
> > > > > > it?
> > > > > >
> > > > > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Taylor,
> > > > > > >
> > > > > > > I see you're having trouble with python embedding on
> Cheyenne. I
> > > > > logged
> > > > > > on
> > > > > > > this morning to try to replicate the errors you're
getting.
> > > > > > >
> > > > > > > Below are the commands I used to test this:
> > > > > > >
> > > > > > >
> > > > > > > *cd
/glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *module use
> /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > > load
> > > > > > > met/8.1_pythonncar_pylib *
> > > > > > > *wget
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > >*
> > > > > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > > > > >
> > > > > > > Since you didn't send me the path to where your data
lives on
> > > > > Cheyenne, I
> > > > > > > ran wget to pull a similar 1.0 degree GFS GRIB2 file.
Your
> > > filename
> > > > > > > includes "reduced" which I suspect means that some GRIB
records
> > > have
> > > > > been
> > > > > > > excluded. But the one I have contains the full 586
records.
> > > > > > >
> > > > > > > The first step with python embedding in MET is to make
sure the
> > > > script
> > > > > > runs
> > > > > > > fine through python itself. But mine produces a runtime
error:
> > > > > > >
> > > > > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > > > > >
> > > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > > > > read_GFSv3.py:56: RuntimeWarning: overflow encountered
in
> square
> > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > Traceback (most recent call last):
> > > > > > > File "read_GFSv3.py", line 72, in <module>
> > > > > > > data = np.load(input_file)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > > > > lin
> > > > > > > e 431, in load
> > > > > > > "Failed to interpret file %s as a pickle" %
repr(file))
> > > > > > > IOError: Failed to interpret file
'gfs.t00z.pgrb2.1p00.f048'
> as a
> > > > > pickle
> > > > > > >
> > > > > > > Does yours run without error?
> > > > > > > *python read_GFSv3.py
> gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > > > > >
> > > > > > > If so, please tell me where these files live.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted
upon.
> > > > > > > > Transaction: Ticket created by
> > taylor.mandelbaum at stonybrook.edu
> > > > > > > > Queue: met_help
> > > > > > > > Subject: MET 8.1 python embedding failing when
using
> grib2
> > > > file
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > I'm having trouble getting plot_data_plane to work on
> cheyenne
> > -
> > > I
> > > > am
> > > > > > > > trying it using these two commands:
> > > > > > > >
> > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > > > > >
> > > > > > > >
> > > > > > > > and
> > > > > > > >
> > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > > mflux_ana.npy";'
> > > > > > > > >
> > > > > > > >
> > > > > > > > the read_GFSv3.py code is as follows:
> > > > > > > >
> > > > > > > > from __future__ import print_function
> > > > > > > > > import os
> > > > > > > > > import sys
> > > > > > > > > import re
> > > > > > > > > import numpy as np
> > > > > > > > > import datetime as dt
> > > > > > > > > import pygrib
> > > > > > > > >
> > > > > > > > > def ivt(input_file):
> > > > > > > > > grbs = pygrib.open(input_file)
> > > > > > > > > g = 9.81
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > levs = []
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'humidity' in grb.parameterName.lower():
> > > > > > > > > levs.append(grb.level*100)
> > > > > > > > > levs=np.array(levs)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > q=[]
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'humidity' in grb.parameterName:
> > > > > > > > > q.append(grb.values)
> > > > > > > > > q=np.array(q)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > hgt=[]
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > > > > hgt.append(grb.values)
> > > > > > > > > hgt = np.array(hgt)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > temp=[]
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'Temperature' in grb.parameterName:
> > > > > > > > > temp.append(grb.values)
> > > > > > > > > temp = np.array(temp)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > u=[]
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'u-' in grb.parameterName:
> > > > > > > > > u.append(grb.values)
> > > > > > > > > u = np.array(u)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > v=[]
> > > > > > > > > for grb in grbs:
> > > > > > > > > if 'v-' in grb.parameterName:
> > > > > > > > > v.append(grb.values)
> > > > > > > > > v = np.array(v)
> > > > > > > > > grbs.rewind()
> > > > > > > > >
> > > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > > > > return mflux
> > > > > > > > >
> > > > > > > > > if len(sys.argv) == 2:
> > > > > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > > > > try:
> > > > > > > > > # Print some output to verify that this
script ran
> > > > > > > > > print("Input File: " + repr(input_file))
> > > > > > > > >
> > > > > > > > > # Read input file
> > > > > > > > > try:
> > > > > > > > > data = ivt(input_file)
> > > > > > > > > except:
> > > > > > > > > data = np.load(input_file)
> > > > > > > > > print(type(data))
> > > > > > > > > # Read and re-orient the data
> > > > > > > > > met_data = data
> > > > > > > > >
> > > > > > > > > print("Data Shape: " + repr(met_data.shape))
> > > > > > > > > print("Data Type: " + repr(met_data.dtype))
> > > > > > > > > except NameError:
> > > > > > > > > print("Trouble reading input file: " +
input_file)
> > > > > > > > > else:
> > > > > > > > > print("Must specify exactly one input file.")
> > > > > > > > > sys.exit(1)
> > > > > > > > > valid = dt.datetime(2016,1,24)
> > > > > > > > >
> > > > > > > > > attrs = {
> > > > > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > 'lead': '0',
> > > > > > > > > 'accum': '00',
> > > > > > > > >
> > > > > > > > > 'name': 'mflux',
> > > > > > > > > 'long_name': 'UNKNOWN',
> > > > > > > > > 'level': '1000_300',
> > > > > > > > > 'units': 'UNKNOWN',
> > > > > > > > >
> > > > > > > > > 'grid': {
> > > > > > > > > 'name': 'Global 1 Degree',
> > > > > > > > > 'type' : 'LatLon',
> > > > > > > > > 'lat_ll' : -90.0,
> > > > > > > > > 'lon_ll' : 0.0,
> > > > > > > > > 'delta_lat' : 1.0,
> > > > > > > > > 'delta_lon' : 1.0,
> > > > > > > > > 'Nlat' : 181,
> > > > > > > > > 'Nlon' : 360,
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > print("Attributes: " + repr(attrs))
> > > > > > > > >
> > > > > > > >
> > > > > > > > The only difference between the two is the grib2
invokes the
> > ivt
> > > > > > > function,
> > > > > > > > where pygrib opens the file. When I try using the
grib2
> file, I
> > > > get a
> > > > > > > > segmentation fault (core dumped) error. Using the
first
> command
> > > > (with
> > > > > > the
> > > > > > > > .npy file) works. In both cases, met_data is a numpy
array
> > (which
> > > > is
> > > > > > why
> > > > > > > I
> > > > > > > > kept PYTHON_NUMPY). Am I missing a flag or another
issue I'm
> > > > > > overlooking?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --
> > > > > > > > *Taylor Mandelbaum*
> > > > > > > > Research Technician
> > > > > > > > SoMAS | Stony Brook University
> > > > > > > > 678.468.5789
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > *Taylor Mandelbaum*
> > > > Research Technician
> > > > SoMAS | Stony Brook University
> > > > 678.468.5789
> > > >
> > > >
> > >
> > >
> >
> > --
> > *Taylor Mandelbaum*
> > Research Technician
> > SoMAS | Stony Brook University
> > 678.468.5789
> >
> >
>
>
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: Taylor Mandelbaum
Time: Thu Oct 10 19:57:37 2019
Hey George,
That workaround idea is essentially the same as how mine works, I had
the
files converted from grib to npy in a separate call entirely. Thanks
though! I missed the call today because I lost track of time, but I
need to
reach out and see how we can move forward with either MET or METplus
so we
can script in pygrib, xarray, cfgrib, etc. into the python embedding
calls.
If there are issues that I can assist with for that bug, let me know.
On Thu, Oct 10, 2019 at 9:29 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Taylor,
>
> We are exploring options to be able to handle python embedding
without
> running into these conflicts. Did your workaround work? If not, I
wrote a
> simple python script that can be used as a workaround. It is called
outside
> of METplus to generate a .py file that contains the data and
attributes
> that can be called in the python embedding via METplus. The script
it
> attached. If you were previously setting:
>
> *name="my_script.py filename args"*
>
> in the MET call, you can run
>
>
> *python write_data.py <OUTPUT_FILENAME> my_script.py filename args*
>
> using a version of python that will allow you to use the modules
that were
> failing inside MET. The path specified by <OUTPUT_FILENAME> (must
end with
> .py) will be a file that you can pass into MET like this:
>
> name="<OUTPUT_FILENAME>"
>
> Let me know if that works for you or if you have any questions.
>
> Thanks,
> George
>
> ------------------------------------------------------------
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ------------------------------------------------------------
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
> On Mon, Sep 30, 2019 at 3:11 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> >
> > Taylor,
> >
> > OK, glad you have a temporary workaround for now. FYI, I wrote up
this
> > GitHub issue for MET with 2 specific examples to debug on
cheyenne.
> >
> > https://github.com/NCAR/MET/issues/1205
> >
> > Thanks,
> > John
> >
> > On Mon, Sep 30, 2019 at 12:02 PM Taylor Mandelbaum via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > >
> > > Thanks John - I'm going to try a workaround that invokes a
separate
> > > function which will load in the grib and transform it to a numpy
> array. I
> > > don't think it will work if it's being run in the same instance
of
> python
> > > as read_GFSv3.py through MET, but it can't hurt to try.
> > >
> > > On Mon, Sep 30, 2019 at 1:25 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Taylor,
> > > >
> > > > OK, I'm able to replicate the behavior you describe... running
the
> > script
> > > > fine with the interpreter but getting a core dump via
> plot_data_plane.
> > > The
> > > > MET source code links to the GRIB2 library, as does the pygrib
python
> > > > package. This is similar to the behavior for HDF5 on Cheyenne
as
> > well...
> > > > getting a segfault from MET when the python script loads the
h5py
> > > package.
> > > > I suspect there's a conflict with the linker when the python
script
> > loads
> > > > packages which require libraries that were also required to
build
> MET.
> > > >
> > > > But unfortunately I don't have a fix. I'll raise this issue
for
> > > discussion
> > > > at the MET development meeting tomorrow.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > try to cd into /fv3/gfs.20160120/00 and run:
> > > > >
> > > > > python read_GFSv3,py gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > >
> > > > > That's what I use after running ncar_pylib to initailize the
> > virtualenv
> > > > > with all the relevant packages. That should work... let me
know if
> it
> > > > > doesn't. I'm also running plot_data_plane from that
directory.
> > > > >
> > > > > On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Taylor,
> > > > > >
> > > > > > OK, I went hunting around and found your working directory
in:
> > > > > > /glade/u/home/mandelbaum/test
> > > > > >
> > > > > > And I can read data from your directory, so that's fine.
Please
> > send
> > > > me
> > > > > > the exact python command that you're running on the
command line
> to
> > > > > confirm
> > > > > > that your script works.
> > > > > >
> > > > > > When I run it, it does not run fine for me. The first
step in
> > > > debugging
> > > > > > this is figuring out exactly what error we need to debug:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *module use
/glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > load
> > > > > > met/8.1_pythonncar_pylib python
> > > > > /glade/u/home/mandelbaum/test/read_GFSv3.py
> > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> > > > > >
> > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > > > > > Trouble reading input file:
> gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > > > Attributes: {'long_name': 'UNKNOWN', 'init':
'20160124_000000',
> > > > 'valid':
> > > > > > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > 'name':
> > > > > > 'Global 1 Degree', 'Nla
> > > > > > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type':
'LatLon',
> > 'Nlon':
> > > > > 360},
> > > > > > 'name': 'mflux', 'lead': '0', 'level': '1000_300',
'units':
> > > 'UNKNOWN',
> > > > > > 'accum': '00'}
> > > > > >
> > > > > > Note that this script is NOT successful in reading the
command
> line
> > > > > > argument file (Trouble reading input file:...). So it
falls into
> > the
> > > > > > exception block and reads the analysis file instead.
> > > > > >
> > > > > > I'm trying to get to the same state that you are:
> > > > > > (1) Have a command that runs via python on the command
line.
> > > > > > (2) When I run that same command through MET, if fails.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT <
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> >
> > > > > > >
> > > > > > > Hey John,
> > > > > > >
> > > > > > > Yeah, mine runs through when I try it using just python.
My
> data
> > is
> > > > on
> > > > > my
> > > > > > > local directory in cheyenne - is there an easy way to
give you
> > > access
> > > > > to
> > > > > > > it?
> > > > > > >
> > > > > > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Taylor,
> > > > > > > >
> > > > > > > > I see you're having trouble with python embedding on
> > Cheyenne. I
> > > > > > logged
> > > > > > > on
> > > > > > > > this morning to try to replicate the errors you're
getting.
> > > > > > > >
> > > > > > > > Below are the commands I used to test this:
> > > > > > > >
> > > > > > > >
> > > > > > > > *cd
/glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *module use
> > /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > > > load
> > > > > > > > met/8.1_pythonncar_pylib *
> > > > > > > > *wget
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > > >*
> > > > > > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > > > > > >
> > > > > > > > Since you didn't send me the path to where your data
lives on
> > > > > > Cheyenne, I
> > > > > > > > ran wget to pull a similar 1.0 degree GFS GRIB2 file.
Your
> > > > filename
> > > > > > > > includes "reduced" which I suspect means that some
GRIB
> records
> > > > have
> > > > > > been
> > > > > > > > excluded. But the one I have contains the full 586
records.
> > > > > > > >
> > > > > > > > The first step with python embedding in MET is to make
sure
> the
> > > > > script
> > > > > > > runs
> > > > > > > > fine through python itself. But mine produces a
runtime
> error:
> > > > > > > >
> > > > > > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > > > > > >
> > > > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > > > > > read_GFSv3.py:56: RuntimeWarning: overflow encountered
in
> > square
> > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > Traceback (most recent call last):
> > > > > > > > File "read_GFSv3.py", line 72, in <module>
> > > > > > > > data = np.load(input_file)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > > > > > lin
> > > > > > > > e 431, in load
> > > > > > > > "Failed to interpret file %s as a pickle" %
repr(file))
> > > > > > > > IOError: Failed to interpret file
'gfs.t00z.pgrb2.1p00.f048'
> > as a
> > > > > > pickle
> > > > > > > >
> > > > > > > > Does yours run without error?
> > > > > > > > *python read_GFSv3.py
> > gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > > > > > >
> > > > > > > > If so, please tell me where these files live.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> > > taylor.mandelbaum at stonybrook.edu
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: MET 8.1 python embedding failing when
using
> > grib2
> > > > > file
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hey all,
> > > > > > > > >
> > > > > > > > > I'm having trouble getting plot_data_plane to work
on
> > cheyenne
> > > -
> > > > I
> > > > > am
> > > > > > > > > trying it using these two commands:
> > > > > > > > >
> > > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > and
> > > > > > > > >
> > > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
'name="read_GFSv3.py
> > > > > > > > > mflux_ana.npy";'
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > the read_GFSv3.py code is as follows:
> > > > > > > > >
> > > > > > > > > from __future__ import print_function
> > > > > > > > > > import os
> > > > > > > > > > import sys
> > > > > > > > > > import re
> > > > > > > > > > import numpy as np
> > > > > > > > > > import datetime as dt
> > > > > > > > > > import pygrib
> > > > > > > > > >
> > > > > > > > > > def ivt(input_file):
> > > > > > > > > > grbs = pygrib.open(input_file)
> > > > > > > > > > g = 9.81
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > levs = []
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'humidity' in
grb.parameterName.lower():
> > > > > > > > > > levs.append(grb.level*100)
> > > > > > > > > > levs=np.array(levs)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > q=[]
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'humidity' in grb.parameterName:
> > > > > > > > > > q.append(grb.values)
> > > > > > > > > > q=np.array(q)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > hgt=[]
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > > > > > hgt.append(grb.values)
> > > > > > > > > > hgt = np.array(hgt)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > temp=[]
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'Temperature' in grb.parameterName:
> > > > > > > > > > temp.append(grb.values)
> > > > > > > > > > temp = np.array(temp)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > u=[]
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'u-' in grb.parameterName:
> > > > > > > > > > u.append(grb.values)
> > > > > > > > > > u = np.array(u)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > v=[]
> > > > > > > > > > for grb in grbs:
> > > > > > > > > > if 'v-' in grb.parameterName:
> > > > > > > > > > v.append(grb.values)
> > > > > > > > > > v = np.array(v)
> > > > > > > > > > grbs.rewind()
> > > > > > > > > >
> > > > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > > > mflux = q*(1/g)*uv*(np.max(levs)-np.min(levs))
> > > > > > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > > > > > return mflux
> > > > > > > > > >
> > > > > > > > > > if len(sys.argv) == 2:
> > > > > > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > > > > > try:
> > > > > > > > > > # Print some output to verify that this
script
> ran
> > > > > > > > > > print("Input File: " + repr(input_file))
> > > > > > > > > >
> > > > > > > > > > # Read input file
> > > > > > > > > > try:
> > > > > > > > > > data = ivt(input_file)
> > > > > > > > > > except:
> > > > > > > > > > data = np.load(input_file)
> > > > > > > > > > print(type(data))
> > > > > > > > > > # Read and re-orient the data
> > > > > > > > > > met_data = data
> > > > > > > > > >
> > > > > > > > > > print("Data Shape: " +
repr(met_data.shape))
> > > > > > > > > > print("Data Type: " +
repr(met_data.dtype))
> > > > > > > > > > except NameError:
> > > > > > > > > > print("Trouble reading input file: " +
> input_file)
> > > > > > > > > > else:
> > > > > > > > > > print("Must specify exactly one input file.")
> > > > > > > > > > sys.exit(1)
> > > > > > > > > > valid = dt.datetime(2016,1,24)
> > > > > > > > > >
> > > > > > > > > > attrs = {
> > > > > > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > > 'lead': '0',
> > > > > > > > > > 'accum': '00',
> > > > > > > > > >
> > > > > > > > > > 'name': 'mflux',
> > > > > > > > > > 'long_name': 'UNKNOWN',
> > > > > > > > > > 'level': '1000_300',
> > > > > > > > > > 'units': 'UNKNOWN',
> > > > > > > > > >
> > > > > > > > > > 'grid': {
> > > > > > > > > > 'name': 'Global 1 Degree',
> > > > > > > > > > 'type' : 'LatLon',
> > > > > > > > > > 'lat_ll' : -90.0,
> > > > > > > > > > 'lon_ll' : 0.0,
> > > > > > > > > > 'delta_lat' : 1.0,
> > > > > > > > > > 'delta_lon' : 1.0,
> > > > > > > > > > 'Nlat' : 181,
> > > > > > > > > > 'Nlon' : 360,
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > print("Attributes: " + repr(attrs))
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The only difference between the two is the grib2
invokes
> the
> > > ivt
> > > > > > > > function,
> > > > > > > > > where pygrib opens the file. When I try using the
grib2
> > file, I
> > > > > get a
> > > > > > > > > segmentation fault (core dumped) error. Using the
first
> > command
> > > > > (with
> > > > > > > the
> > > > > > > > > .npy file) works. In both cases, met_data is a numpy
array
> > > (which
> > > > > is
> > > > > > > why
> > > > > > > > I
> > > > > > > > > kept PYTHON_NUMPY). Am I missing a flag or another
issue
> I'm
> > > > > > > overlooking?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > --
> > > > > > > > > *Taylor Mandelbaum*
> > > > > > > > > Research Technician
> > > > > > > > > SoMAS | Stony Brook University
> > > > > > > > > 678.468.5789
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > *Taylor Mandelbaum*
> > > > > Research Technician
> > > > > SoMAS | Stony Brook University
> > > > > 678.468.5789
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > *Taylor Mandelbaum*
> > > Research Technician
> > > SoMAS | Stony Brook University
> > > 678.468.5789
> > >
> > >
> >
> >
>
>
--
*Taylor Mandelbaum*
Research Technician
SoMAS | Stony Brook University
678.468.5789
------------------------------------------------
Subject: MET 8.1 python embedding failing when using grib2 file
From: George McCabe
Time: Fri Oct 11 08:31:35 2019
Hi Taylor,
We have been discussing options for resolving this bug for the 9.0
release.
I think the most likely solution will be to call the user's Python
script
using the user's Python environment and capturing the result in a
temporary
pickle file that will be read by the Python interpreter baked into
MET.
This is essentially doing what both of us separately decided to do to
get
around this issue, so it seems like the right choice. Letting the
users use
their own version of Python will allow them to use libraries that may
not
be included in the MET version, so this will allow more flexibility on
their end. Thanks for bringing this issue to our attention so we can
get it
resolved before more people run into the same issue.
------------------------------------------------------------
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
------------------------------------------------------------
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
On Thu, Oct 10, 2019 at 7:58 PM Taylor Mandelbaum via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
>
> Hey George,
>
> That workaround idea is essentially the same as how mine works, I
had the
> files converted from grib to npy in a separate call entirely. Thanks
> though! I missed the call today because I lost track of time, but I
need to
> reach out and see how we can move forward with either MET or METplus
so we
> can script in pygrib, xarray, cfgrib, etc. into the python embedding
calls.
> If there are issues that I can assist with for that bug, let me
know.
>
> On Thu, Oct 10, 2019 at 9:29 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Taylor,
> >
> > We are exploring options to be able to handle python embedding
without
> > running into these conflicts. Did your workaround work? If not, I
wrote a
> > simple python script that can be used as a workaround. It is
called
> outside
> > of METplus to generate a .py file that contains the data and
attributes
> > that can be called in the python embedding via METplus. The script
it
> > attached. If you were previously setting:
> >
> > *name="my_script.py filename args"*
> >
> > in the MET call, you can run
> >
> >
> > *python write_data.py <OUTPUT_FILENAME> my_script.py filename
args*
> >
> > using a version of python that will allow you to use the modules
that
> were
> > failing inside MET. The path specified by <OUTPUT_FILENAME> (must
end
> with
> > .py) will be a file that you can pass into MET like this:
> >
> > name="<OUTPUT_FILENAME>"
> >
> > Let me know if that works for you or if you have any questions.
> >
> > Thanks,
> > George
> >
> > ------------------------------------------------------------
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ------------------------------------------------------------
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
> > On Mon, Sep 30, 2019 at 3:11 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > >
> > > Taylor,
> > >
> > > OK, glad you have a temporary workaround for now. FYI, I wrote
up this
> > > GitHub issue for MET with 2 specific examples to debug on
cheyenne.
> > >
> > > https://github.com/NCAR/MET/issues/1205
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Sep 30, 2019 at 12:02 PM Taylor Mandelbaum via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
>
> > > >
> > > > Thanks John - I'm going to try a workaround that invokes a
separate
> > > > function which will load in the grib and transform it to a
numpy
> > array. I
> > > > don't think it will work if it's being run in the same
instance of
> > python
> > > > as read_GFSv3.py through MET, but it can't hurt to try.
> > > >
> > > > On Mon, Sep 30, 2019 at 1:25 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Taylor,
> > > > >
> > > > > OK, I'm able to replicate the behavior you describe...
running the
> > > script
> > > > > fine with the interpreter but getting a core dump via
> > plot_data_plane.
> > > > The
> > > > > MET source code links to the GRIB2 library, as does the
pygrib
> python
> > > > > package. This is similar to the behavior for HDF5 on
Cheyenne as
> > > well...
> > > > > getting a segfault from MET when the python script loads the
h5py
> > > > package.
> > > > > I suspect there's a conflict with the linker when the python
script
> > > loads
> > > > > packages which require libraries that were also required to
build
> > MET.
> > > > >
> > > > > But unfortunately I don't have a fix. I'll raise this issue
for
> > > > discussion
> > > > > at the MET development meeting tomorrow.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Sep 30, 2019 at 10:33 AM Taylor Mandelbaum via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378 >
> > > > > >
> > > > > > Hey John,
> > > > > >
> > > > > > try to cd into /fv3/gfs.20160120/00 and run:
> > > > > >
> > > > > > python read_GFSv3,py
gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > > >
> > > > > > That's what I use after running ncar_pylib to initailize
the
> > > virtualenv
> > > > > > with all the relevant packages. That should work... let me
know
> if
> > it
> > > > > > doesn't. I'm also running plot_data_plane from that
directory.
> > > > > >
> > > > > > On Mon, Sep 30, 2019 at 12:22 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Taylor,
> > > > > > >
> > > > > > > OK, I went hunting around and found your working
directory in:
> > > > > > > /glade/u/home/mandelbaum/test
> > > > > > >
> > > > > > > And I can read data from your directory, so that's fine.
> Please
> > > send
> > > > > me
> > > > > > > the exact python command that you're running on the
command
> line
> > to
> > > > > > confirm
> > > > > > > that your script works.
> > > > > > >
> > > > > > > When I run it, it does not run fine for me. The first
step in
> > > > > debugging
> > > > > > > this is figuring out exactly what error we need to
debug:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *module use
> /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > > load
> > > > > > > met/8.1_pythonncar_pylib python
> > > > > > /glade/u/home/mandelbaum/test/read_GFSv3.py
> > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2 *
> > > > > > >
> > > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048.reduced.grib2'
> > > > > > > Trouble reading input file:
> > gfs.t00z.pgrb2.1p00.f048.reduced.grib2
> > > > > > > Attributes: {'long_name': 'UNKNOWN', 'init':
'20160124_000000',
> > > > > 'valid':
> > > > > > > '20160124_000000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > > 'name':
> > > > > > > 'Global 1 Degree', 'Nla
> > > > > > > t': 181, 'lat_ll': -90.0, 'delta_lon': 1.0, 'type':
'LatLon',
> > > 'Nlon':
> > > > > > 360},
> > > > > > > 'name': 'mflux', 'lead': '0', 'level': '1000_300',
'units':
> > > > 'UNKNOWN',
> > > > > > > 'accum': '00'}
> > > > > > >
> > > > > > > Note that this script is NOT successful in reading the
command
> > line
> > > > > > > argument file (Trouble reading input file:...). So it
falls
> into
> > > the
> > > > > > > exception block and reads the analysis file instead.
> > > > > > >
> > > > > > > I'm trying to get to the same state that you are:
> > > > > > > (1) Have a command that runs via python on the command
line.
> > > > > > > (2) When I run that same command through MET, if fails.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Sep 30, 2019 at 9:51 AM Taylor Mandelbaum via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > >
> > > > > > > >
> > > > > > > > Hey John,
> > > > > > > >
> > > > > > > > Yeah, mine runs through when I try it using just
python. My
> > data
> > > is
> > > > > on
> > > > > > my
> > > > > > > > local directory in cheyenne - is there an easy way to
give
> you
> > > > access
> > > > > > to
> > > > > > > > it?
> > > > > > > >
> > > > > > > > On Mon, Sep 30, 2019, 11:48 John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Taylor,
> > > > > > > > >
> > > > > > > > > I see you're having trouble with python embedding on
> > > Cheyenne. I
> > > > > > > logged
> > > > > > > > on
> > > > > > > > > this morning to try to replicate the errors you're
getting.
> > > > > > > > >
> > > > > > > > > Below are the commands I used to test this:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *cd
> /glade/p/ral/jntp/MET/MET_Help/mandelbaum_data_20190930*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *module use
> > > /glade/p/ral/jntp/MET/MET_releases/modulefilesmodule
> > > > > load
> > > > > > > > > met/8.1_pythonncar_pylib *
> > > > > > > > > *wget
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20190925/00/gfs.t00z.pgrb2.1p00.f048
> > > > > > > > > >*
> > > > > > > > > *python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048 *
> > > > > > > > >
> > > > > > > > > Since you didn't send me the path to where your data
lives
> on
> > > > > > > Cheyenne, I
> > > > > > > > > ran wget to pull a similar 1.0 degree GFS GRIB2
file. Your
> > > > > filename
> > > > > > > > > includes "reduced" which I suspect means that some
GRIB
> > records
> > > > > have
> > > > > > > been
> > > > > > > > > excluded. But the one I have contains the full 586
> records.
> > > > > > > > >
> > > > > > > > > The first step with python embedding in MET is to
make sure
> > the
> > > > > > script
> > > > > > > > runs
> > > > > > > > > fine through python itself. But mine produces a
runtime
> > error:
> > > > > > > > >
> > > > > > > > > python read_GFSv3.py gfs.t00z.pgrb2.1p00.f048
> > > > > > > > >
> > > > > > > > > Input File: 'gfs.t00z.pgrb2.1p00.f048'
> > > > > > > > > read_GFSv3.py:56: RuntimeWarning: overflow
encountered in
> > > square
> > > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > > Traceback (most recent call last):
> > > > > > > > > File "read_GFSv3.py", line 72, in <module>
> > > > > > > > > data = np.load(input_file)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/gpfs/u/apps/ch/opt/python/2.7.14/intel/17.0.1/pkg-
library/20180329/lib/python2.7/site-packages/numpy/lib/npyio.py",
> > > > > > > > > lin
> > > > > > > > > e 431, in load
> > > > > > > > > "Failed to interpret file %s as a pickle" %
repr(file))
> > > > > > > > > IOError: Failed to interpret file
> 'gfs.t00z.pgrb2.1p00.f048'
> > > as a
> > > > > > > pickle
> > > > > > > > >
> > > > > > > > > Does yours run without error?
> > > > > > > > > *python read_GFSv3.py
> > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2*
> > > > > > > > >
> > > > > > > > > If so, please tell me where these files live.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Fri, Sep 27, 2019 at 10:01 PM Taylor Mandelbaum
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Fri Sep 27 22:01:34 2019: Request 92378 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by
> > > > taylor.mandelbaum at stonybrook.edu
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: MET 8.1 python embedding failing
when using
> > > grib2
> > > > > > file
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: taylor.mandelbaum at stonybrook.edu
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92378
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hey all,
> > > > > > > > > >
> > > > > > > > > > I'm having trouble getting plot_data_plane to work
on
> > > cheyenne
> > > > -
> > > > > I
> > > > > > am
> > > > > > > > > > trying it using these two commands:
> > > > > > > > > >
> > > > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
> 'name="read_GFSv3.py
> > > > > > > > > > > gfs.t00z.pgrb2.1p00.f048.reduced.grib2";'
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > plot_data_plane PYTHON_NUMPY mflux.ps
> 'name="read_GFSv3.py
> > > > > > > > > > mflux_ana.npy";'
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > the read_GFSv3.py code is as follows:
> > > > > > > > > >
> > > > > > > > > > from __future__ import print_function
> > > > > > > > > > > import os
> > > > > > > > > > > import sys
> > > > > > > > > > > import re
> > > > > > > > > > > import numpy as np
> > > > > > > > > > > import datetime as dt
> > > > > > > > > > > import pygrib
> > > > > > > > > > >
> > > > > > > > > > > def ivt(input_file):
> > > > > > > > > > > grbs = pygrib.open(input_file)
> > > > > > > > > > > g = 9.81
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > levs = []
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'humidity' in
grb.parameterName.lower():
> > > > > > > > > > > levs.append(grb.level*100)
> > > > > > > > > > > levs=np.array(levs)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > q=[]
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'humidity' in grb.parameterName:
> > > > > > > > > > > q.append(grb.values)
> > > > > > > > > > > q=np.array(q)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > hgt=[]
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'Geopotential' in grb.parameterName:
> > > > > > > > > > > hgt.append(grb.values)
> > > > > > > > > > > hgt = np.array(hgt)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > temp=[]
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'Temperature' in grb.parameterName:
> > > > > > > > > > > temp.append(grb.values)
> > > > > > > > > > > temp = np.array(temp)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > u=[]
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'u-' in grb.parameterName:
> > > > > > > > > > > u.append(grb.values)
> > > > > > > > > > > u = np.array(u)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > v=[]
> > > > > > > > > > > for grb in grbs:
> > > > > > > > > > > if 'v-' in grb.parameterName:
> > > > > > > > > > > v.append(grb.values)
> > > > > > > > > > > v = np.array(v)
> > > > > > > > > > > grbs.rewind()
> > > > > > > > > > >
> > > > > > > > > > > uv = np.sqrt(u**2+v**2)
> > > > > > > > > > > mflux = q*(1/g)*uv*(np.max(levs)-
np.min(levs))
> > > > > > > > > > > mflux = np.sum(mflux,axis=0)
> > > > > > > > > > > mflux_avg = np.mean(mflux,axis=0)
> > > > > > > > > > > return mflux
> > > > > > > > > > >
> > > > > > > > > > > if len(sys.argv) == 2:
> > > > > > > > > > > input_file = os.path.expandvars(sys.argv[1])
> > > > > > > > > > > try:
> > > > > > > > > > > # Print some output to verify that this
script
> > ran
> > > > > > > > > > > print("Input File: " + repr(input_file))
> > > > > > > > > > >
> > > > > > > > > > > # Read input file
> > > > > > > > > > > try:
> > > > > > > > > > > data = ivt(input_file)
> > > > > > > > > > > except:
> > > > > > > > > > > data = np.load(input_file)
> > > > > > > > > > > print(type(data))
> > > > > > > > > > > # Read and re-orient the data
> > > > > > > > > > > met_data = data
> > > > > > > > > > >
> > > > > > > > > > > print("Data Shape: " +
repr(met_data.shape))
> > > > > > > > > > > print("Data Type: " +
repr(met_data.dtype))
> > > > > > > > > > > except NameError:
> > > > > > > > > > > print("Trouble reading input file: " +
> > input_file)
> > > > > > > > > > > else:
> > > > > > > > > > > print("Must specify exactly one input
file.")
> > > > > > > > > > > sys.exit(1)
> > > > > > > > > > > valid = dt.datetime(2016,1,24)
> > > > > > > > > > >
> > > > > > > > > > > attrs = {
> > > > > > > > > > > 'valid': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > > > 'init': valid.strftime("%Y%m%d_%H%M%S"),
> > > > > > > > > > > 'lead': '0',
> > > > > > > > > > > 'accum': '00',
> > > > > > > > > > >
> > > > > > > > > > > 'name': 'mflux',
> > > > > > > > > > > 'long_name': 'UNKNOWN',
> > > > > > > > > > > 'level': '1000_300',
> > > > > > > > > > > 'units': 'UNKNOWN',
> > > > > > > > > > >
> > > > > > > > > > > 'grid': {
> > > > > > > > > > > 'name': 'Global 1 Degree',
> > > > > > > > > > > 'type' : 'LatLon',
> > > > > > > > > > > 'lat_ll' : -90.0,
> > > > > > > > > > > 'lon_ll' : 0.0,
> > > > > > > > > > > 'delta_lat' : 1.0,
> > > > > > > > > > > 'delta_lon' : 1.0,
> > > > > > > > > > > 'Nlat' : 181,
> > > > > > > > > > > 'Nlon' : 360,
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > print("Attributes: " + repr(attrs))
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The only difference between the two is the grib2
invokes
> > the
> > > > ivt
> > > > > > > > > function,
> > > > > > > > > > where pygrib opens the file. When I try using the
grib2
> > > file, I
> > > > > > get a
> > > > > > > > > > segmentation fault (core dumped) error. Using the
first
> > > command
> > > > > > (with
> > > > > > > > the
> > > > > > > > > > .npy file) works. In both cases, met_data is a
numpy
> array
> > > > (which
> > > > > > is
> > > > > > > > why
> > > > > > > > > I
> > > > > > > > > > kept PYTHON_NUMPY). Am I missing a flag or another
issue
> > I'm
> > > > > > > > overlooking?
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > --
> > > > > > > > > > *Taylor Mandelbaum*
> > > > > > > > > > Research Technician
> > > > > > > > > > SoMAS | Stony Brook University
> > > > > > > > > > 678.468.5789
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > *Taylor Mandelbaum*
> > > > > > Research Technician
> > > > > > SoMAS | Stony Brook University
> > > > > > 678.468.5789
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > *Taylor Mandelbaum*
> > > > Research Technician
> > > > SoMAS | Stony Brook University
> > > > 678.468.5789
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> *Taylor Mandelbaum*
> Research Technician
> SoMAS | Stony Brook University
> 678.468.5789
>
>
------------------------------------------------
More information about the Met_help
mailing list