[Met_help] [rt.rap.ucar.edu #91588] History for Python embedding and pygrib
John Halley Gotway via RT
met_help at ucar.edu
Fri Jun 19 15:19:59 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello MET Help,
There still seems to be an issue using the python embedding feature in
association with pygrib, which I'd like to use to read GRIB data into
python, and then eventually pass data to MET.
There are two ways for you to test:
1) Copy /glade/scratch/schwartz/cloud_vx to /glade/scratch/$USER. Then:
cd /glade/scratch/$USER/cloud_vx
cd bin/MET
./driver_script_with_python_manual.ksh
That should run everything without you having to change anything. Output
of the failed run will be in /glade/scratch/$USER/cloud_vx/metprd
/2017060500/f06/lowCloudFrac/SATCORPS.
If you look in ./driver_script_with_python_manual.ksh , you'll see some
"module load" statements near the top. I've also tried other MET builds
located at /glade/p/ral/jntp/MET/MET_releases, but always get the same seg
fault.
2) Running the full set of scripts in #1 means the MET configuration file
is populated using environmental variables. If you want perhaps a simpler
test, just copy everything in
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
and run MET:
/glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat PYTHON_NUMPY
PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
No matter which way you test, the issue occurs in line 200 of "python_stuff.
py" (in /glade/scratch/$USER/cloud_vx/bin) :
read_var = x.values
The ".values" is a pygrib method. MET doesn't seem to like that
Note that In /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
lowCloudFrac/SATCORPS,
if you just do python python_script_fcst.py, you should see the script runs
without error. Thus, the issue certainly appears to be calling the python
code from within MET.
I hope the instructions makes sense. Please let me know if you have any
questions. I'm happy to help you debug this issue.
Thanks!
Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Python embedding and pygrib
From: John Halley Gotway
Time: Mon Aug 19 17:04:23 2019
Craig,
I wanted to let you know that I've been trying to debug this issue
today.
But unfortunately, I don't have a solution for you yet. Thanks for
providing your test data. I was definitely able to replicate the
error and
agree that the segfault is occurring at the between the two debug
print
statements you added.
Coincidentally, Julie Prestopnik and I have also been working on
python
embedding cheyenne for other reasons.
We're testing the met/8.1_python module build from inside the
ncar_pylib
environment. And in there, I observed that...
(1) Two sample python embedding scripts from the MET website (
https://dtcenter.org/community-code/model-evaluation-tools-met/sample-
analysis-scripts)
do actually run fine (one reading a binary input file and another
reading
NetCDF).
(2) When I edit those python scripts by adding (import h5py) to import
the
HDF5 module, then they segfault when run through MET. Thinking you
may be
having a similar issue, I tested all your imports one-by-one, but none
of
them triggered a segfault. It really is when executing that single
line
you mentioned.
(3) Now I'm wondering if there's somehow a conflict in MET's dependent
libraries and those used by python. Investigating this I did find a
slight
mismatch... python was compiled with intel/17.0.1 while we were
compiling
MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to see if
anything changes.
Just wanted to let you know that we're aware of this issue and looking
into
it!
Thanks,
John
On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> Transaction: Ticket created by schwartz at ucar.edu
> Queue: met_help
> Subject: Python embedding and pygrib
> Owner: Nobody
> Requestors: schwartz at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
>
>
> Hello MET Help,
>
> There still seems to be an issue using the python embedding feature
in
> association with pygrib, which I'd like to use to read GRIB data
into
> python, and then eventually pass data to MET.
>
> There are two ways for you to test:
>
> 1) Copy /glade/scratch/schwartz/cloud_vx to /glade/scratch/$USER.
Then:
>
> cd /glade/scratch/$USER/cloud_vx
> cd bin/MET
> ./driver_script_with_python_manual.ksh
>
> That should run everything without you having to change anything.
Output
> of the failed run will be in /glade/scratch/$USER/cloud_vx/metprd
> /2017060500/f06/lowCloudFrac/SATCORPS.
>
> If you look in ./driver_script_with_python_manual.ksh , you'll see
some
> "module load" statements near the top. I've also tried other MET
builds
> located at /glade/p/ral/jntp/MET/MET_releases, but always get the
same seg
> fault.
>
> 2) Running the full set of scripts in #1 means the MET configuration
file
> is populated using environmental variables. If you want perhaps a
simpler
> test, just copy everything in
>
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> and run MET:
>
> /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
PYTHON_NUMPY
> PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
>
> No matter which way you test, the issue occurs in line 200 of
> "python_stuff.
> py" (in /glade/scratch/$USER/cloud_vx/bin) :
> read_var = x.values
>
> The ".values" is a pygrib method. MET doesn't seem to like that
>
> Note that In /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> lowCloudFrac/SATCORPS,
> if you just do python python_script_fcst.py, you should see the
script runs
> without error. Thus, the issue certainly appears to be calling the
python
> code from within MET.
>
> I hope the instructions makes sense. Please let me know if you have
any
> questions. I'm happy to help you debug this issue.
>
> Thanks!
> Craig
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: Craig Schwartz
Time: Tue Aug 20 11:17:19 2019
Hi John,
Thank you for the update. Only thing I could think of was some
conflict
between MET and python--kind of along the lines of your #3 bullet.
It's
interesting that the code only crashes when accessing a ".values" or
".data" (in pygrib, x.data()[0] === x.values and gives the same
segfault). Accessing other keys (like x.missingValue) doesn't give a
crash. Could there be anything special about ".values" or ".data"
conflicting with MET?
Thanks again for looking into this. Let me know if there's anything I
can do to help.
Best,
Craig
On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Craig,
>
> I wanted to let you know that I've been trying to debug this issue
today.
> But unfortunately, I don't have a solution for you yet. Thanks for
> providing your test data. I was definitely able to replicate the
error and
> agree that the segfault is occurring at the between the two debug
print
> statements you added.
>
> Coincidentally, Julie Prestopnik and I have also been working on
python
> embedding cheyenne for other reasons.
>
> We're testing the met/8.1_python module build from inside the
ncar_pylib
> environment. And in there, I observed that...
>
> (1) Two sample python embedding scripts from the MET website (
>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> )
> do actually run fine (one reading a binary input file and another
reading
> NetCDF).
>
> (2) When I edit those python scripts by adding (import h5py) to
import the
> HDF5 module, then they segfault when run through MET. Thinking you
may be
> having a similar issue, I tested all your imports one-by-one, but
none of
> them triggered a segfault. It really is when executing that single
line
> you mentioned.
>
> (3) Now I'm wondering if there's somehow a conflict in MET's
dependent
> libraries and those used by python. Investigating this I did find a
slight
> mismatch... python was compiled with intel/17.0.1 while we were
compiling
> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to see if
> anything changes.
>
> Just wanted to let you know that we're aware of this issue and
looking into
> it!
>
> Thanks,
> John
>
>
>
> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> > Transaction: Ticket created by schwartz at ucar.edu
> > Queue: met_help
> > Subject: Python embedding and pygrib
> > Owner: Nobody
> > Requestors: schwartz at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> >
> >
> > Hello MET Help,
> >
> > There still seems to be an issue using the python embedding
feature in
> > association with pygrib, which I'd like to use to read GRIB data
into
> > python, and then eventually pass data to MET.
> >
> > There are two ways for you to test:
> >
> > 1) Copy /glade/scratch/schwartz/cloud_vx to /glade/scratch/$USER.
Then:
> >
> > cd /glade/scratch/$USER/cloud_vx
> > cd bin/MET
> > ./driver_script_with_python_manual.ksh
> >
> > That should run everything without you having to change anything.
> Output
> > of the failed run will be in /glade/scratch/$USER/cloud_vx/metprd
> > /2017060500/f06/lowCloudFrac/SATCORPS.
> >
> > If you look in ./driver_script_with_python_manual.ksh , you'll see
some
> > "module load" statements near the top. I've also tried other MET
builds
> > located at /glade/p/ral/jntp/MET/MET_releases, but always get the
same
> seg
> > fault.
> >
> > 2) Running the full set of scripts in #1 means the MET
configuration file
> > is populated using environmental variables. If you want perhaps a
> simpler
> > test, just copy everything in
> >
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > and run MET:
> >
> > /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
PYTHON_NUMPY
> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> >
> > No matter which way you test, the issue occurs in line 200 of
> > "python_stuff.
> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > read_var = x.values
> >
> > The ".values" is a pygrib method. MET doesn't seem to like that
> >
> > Note that In /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > lowCloudFrac/SATCORPS,
> > if you just do python python_script_fcst.py, you should see the
script
> runs
> > without error. Thus, the issue certainly appears to be calling
the
> python
> > code from within MET.
> >
> > I hope the instructions makes sense. Please let me know if you
have any
> > questions. I'm happy to help you debug this issue.
> >
> > Thanks!
> > Craig
> >
> >
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: Craig Schwartz
Time: Thu Sep 05 10:39:10 2019
Hi John,
Have you been able to make any progress on this issue?
Also, wanted to let you know I found a hack around the bug (underlying
problem is not solved). Basically, from the driving ksh script, right
before calling grid_stat, I call the python code, which I changed to
output
a GRIB2 file with one record containing the "met_data" NUMPY array.
Then,
I simply have MET read data from the temporary GRIB2 file I've
created,
thus, bypassing calling the python code from MET. Seems to work.
Craig
On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz <schwartz at ucar.edu>
wrote:
> Hi John,
>
> Thank you for the update. Only thing I could think of was some
conflict
> between MET and python--kind of along the lines of your #3 bullet.
It's
> interesting that the code only crashes when accessing a ".values" or
> ".data" (in pygrib, x.data()[0] === x.values and gives the same
> segfault). Accessing other keys (like x.missingValue) doesn't give
a
> crash. Could there be anything special about ".values" or ".data"
> conflicting with MET?
>
> Thanks again for looking into this. Let me know if there's anything
I
> can do to help.
>
> Best,
> Craig
>
>
> On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
<met_help at ucar.
> edu> wrote:
>
>> Craig,
>>
>> I wanted to let you know that I've been trying to debug this issue
today.
>> But unfortunately, I don't have a solution for you yet. Thanks for
>> providing your test data. I was definitely able to replicate the
error
>> and
>> agree that the segfault is occurring at the between the two debug
print
>> statements you added.
>>
>> Coincidentally, Julie Prestopnik and I have also been working on
python
>> embedding cheyenne for other reasons.
>>
>> We're testing the met/8.1_python module build from inside the
ncar_pylib
>> environment. And in there, I observed that...
>>
>> (1) Two sample python embedding scripts from the MET website (
>>
>> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
>> )
>> do actually run fine (one reading a binary input file and another
reading
>> NetCDF).
>>
>> (2) When I edit those python scripts by adding (import h5py) to
import the
>> HDF5 module, then they segfault when run through MET. Thinking you
may be
>> having a similar issue, I tested all your imports one-by-one, but
none of
>> them triggered a segfault. It really is when executing that single
line
>> you mentioned.
>>
>> (3) Now I'm wondering if there's somehow a conflict in MET's
dependent
>> libraries and those used by python. Investigating this I did find
a
>> slight
>> mismatch... python was compiled with intel/17.0.1 while we were
compiling
>> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to see
if
>> anything changes.
>>
>> Just wanted to let you know that we're aware of this issue and
looking
>> into
>> it!
>>
>> Thanks,
>> John
>>
>>
>>
>> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
>> > Transaction: Ticket created by schwartz at ucar.edu
>> > Queue: met_help
>> > Subject: Python embedding and pygrib
>> > Owner: Nobody
>> > Requestors: schwartz at ucar.edu
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
>> >
>> >
>> > Hello MET Help,
>> >
>> > There still seems to be an issue using the python embedding
feature in
>> > association with pygrib, which I'd like to use to read GRIB data
into
>> > python, and then eventually pass data to MET.
>> >
>> > There are two ways for you to test:
>> >
>> > 1) Copy /glade/scratch/schwartz/cloud_vx to /glade/scratch/$USER.
Then:
>> >
>> > cd /glade/scratch/$USER/cloud_vx
>> > cd bin/MET
>> > ./driver_script_with_python_manual.ksh
>> >
>> > That should run everything without you having to change anything.
>> Output
>> > of the failed run will be in /glade/scratch/$USER/cloud_vx/metprd
>> > /2017060500/f06/lowCloudFrac/SATCORPS.
>> >
>> > If you look in ./driver_script_with_python_manual.ksh , you'll
see some
>> > "module load" statements near the top. I've also tried other MET
builds
>> > located at /glade/p/ral/jntp/MET/MET_releases, but always get the
same
>> seg
>> > fault.
>> >
>> > 2) Running the full set of scripts in #1 means the MET
configuration
>> file
>> > is populated using environmental variables. If you want perhaps
a
>> simpler
>> > test, just copy everything in
>> >
>> >
>>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
>> > and run MET:
>> >
>> > /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
PYTHON_NUMPY
>> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
>> >
>> > No matter which way you test, the issue occurs in line 200 of
>> > "python_stuff.
>> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
>> > read_var = x.values
>> >
>> > The ".values" is a pygrib method. MET doesn't seem to like that
>> >
>> > Note that In /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
>> > lowCloudFrac/SATCORPS,
>> > if you just do python python_script_fcst.py, you should see the
script
>> runs
>> > without error. Thus, the issue certainly appears to be calling
the
>> python
>> > code from within MET.
>> >
>> > I hope the instructions makes sense. Please let me know if you
have any
>> > questions. I'm happy to help you debug this issue.
>> >
>> > Thanks!
>> > Craig
>> >
>> >
>>
>>
------------------------------------------------
Subject: Python embedding and pygrib
From: John Halley Gotway
Time: Thu Oct 10 18:31:49 2019
Craig,
George McCabe, a software engineer here, did some more testing of h5py
and
the pygrib modules on cheyenne and ran into the same troubles you and
I
found.
As a result of this, we're rethinking how the python embedding should
work
for met-9.0. The underlying issue is that MET is calling the python
interpreter that it was compiled against, not whatever version you
happen
to be running in your environment. And the runtime linker can run
into
conflicts with the versions of the python libraries and those same
libraries that MET was compiled against.
Ideally, MET would execute the python scripts with the user's python
environment and not the one when MET was compiled. Our current idea
for
met-9.0 is this:
(1) The user specifies the python command to run, including the path
to
python:
/path/to/python sample.py a b c
(2) MET slightly modifies that call and forks a process to execute the
following:
/path/to/python *write_pickle.py temp.pickle* sample.py a b c
The write_pickle.py script would live in the MET repo. It's a
wrapper
that would execute the user's script with its arguments (sample.py a b
c)
and write the resulting data to a python pickle file (temp.pickle, but
we'd
pick a better temp file name).
(3) Then MET's python interpreter would execute this command:
read_pickle.py temp.pickle
And read the data from that temporary pickle file back in via
memory.
>From the user's perspective, there would be very little change. But
the
details have changed a lot under the covers. The goal is to execute
the
user's python script in their own environment, so that what you get
from
running the python interpreter on the command line is the same as what
you
get from running MET.
Any thoughts or feedback on that?
Thanks,
John
On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
>
> Hi John,
>
> Have you been able to make any progress on this issue?
>
> Also, wanted to let you know I found a hack around the bug
(underlying
> problem is not solved). Basically, from the driving ksh script,
right
> before calling grid_stat, I call the python code, which I changed to
output
> a GRIB2 file with one record containing the "met_data" NUMPY array.
Then,
> I simply have MET read data from the temporary GRIB2 file I've
created,
> thus, bypassing calling the python code from MET. Seems to work.
>
> Craig
>
>
> On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz <schwartz at ucar.edu>
wrote:
>
> > Hi John,
> >
> > Thank you for the update. Only thing I could think of was some
conflict
> > between MET and python--kind of along the lines of your #3 bullet.
It's
> > interesting that the code only crashes when accessing a ".values"
or
> > ".data" (in pygrib, x.data()[0] === x.values and gives the same
> > segfault). Accessing other keys (like x.missingValue) doesn't
give a
> > crash. Could there be anything special about ".values" or ".data"
> > conflicting with MET?
> >
> > Thanks again for looking into this. Let me know if there's
anything I
> > can do to help.
> >
> > Best,
> > Craig
> >
> >
> > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
<met_help at ucar.
> > edu> wrote:
> >
> >> Craig,
> >>
> >> I wanted to let you know that I've been trying to debug this
issue
> today.
> >> But unfortunately, I don't have a solution for you yet. Thanks
for
> >> providing your test data. I was definitely able to replicate the
error
> >> and
> >> agree that the segfault is occurring at the between the two debug
print
> >> statements you added.
> >>
> >> Coincidentally, Julie Prestopnik and I have also been working on
python
> >> embedding cheyenne for other reasons.
> >>
> >> We're testing the met/8.1_python module build from inside the
ncar_pylib
> >> environment. And in there, I observed that...
> >>
> >> (1) Two sample python embedding scripts from the MET website (
> >>
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> >> )
> >> do actually run fine (one reading a binary input file and another
> reading
> >> NetCDF).
> >>
> >> (2) When I edit those python scripts by adding (import h5py) to
import
> the
> >> HDF5 module, then they segfault when run through MET. Thinking
you may
> be
> >> having a similar issue, I tested all your imports one-by-one, but
none
> of
> >> them triggered a segfault. It really is when executing that
single line
> >> you mentioned.
> >>
> >> (3) Now I'm wondering if there's somehow a conflict in MET's
dependent
> >> libraries and those used by python. Investigating this I did
find a
> >> slight
> >> mismatch... python was compiled with intel/17.0.1 while we were
> compiling
> >> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to see
if
> >> anything changes.
> >>
> >> Just wanted to let you know that we're aware of this issue and
looking
> >> into
> >> it!
> >>
> >> Thanks,
> >> John
> >>
> >>
> >>
> >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> >> > Transaction: Ticket created by schwartz at ucar.edu
> >> > Queue: met_help
> >> > Subject: Python embedding and pygrib
> >> > Owner: Nobody
> >> > Requestors: schwartz at ucar.edu
> >> > Status: new
> >> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> >
> >> >
> >> >
> >> > Hello MET Help,
> >> >
> >> > There still seems to be an issue using the python embedding
feature in
> >> > association with pygrib, which I'd like to use to read GRIB
data into
> >> > python, and then eventually pass data to MET.
> >> >
> >> > There are two ways for you to test:
> >> >
> >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
/glade/scratch/$USER.
> Then:
> >> >
> >> > cd /glade/scratch/$USER/cloud_vx
> >> > cd bin/MET
> >> > ./driver_script_with_python_manual.ksh
> >> >
> >> > That should run everything without you having to change
anything.
> >> Output
> >> > of the failed run will be in
/glade/scratch/$USER/cloud_vx/metprd
> >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> >> >
> >> > If you look in ./driver_script_with_python_manual.ksh , you'll
see
> some
> >> > "module load" statements near the top. I've also tried other
MET
> builds
> >> > located at /glade/p/ral/jntp/MET/MET_releases, but always get
the same
> >> seg
> >> > fault.
> >> >
> >> > 2) Running the full set of scripts in #1 means the MET
configuration
> >> file
> >> > is populated using environmental variables. If you want
perhaps a
> >> simpler
> >> > test, just copy everything in
> >> >
> >> >
> >>
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> >> > and run MET:
> >> >
> >> > /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> PYTHON_NUMPY
> >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> >> >
> >> > No matter which way you test, the issue occurs in line 200 of
> >> > "python_stuff.
> >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> >> > read_var = x.values
> >> >
> >> > The ".values" is a pygrib method. MET doesn't seem to like
that
> >> >
> >> > Note that In
/glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> >> > lowCloudFrac/SATCORPS,
> >> > if you just do python python_script_fcst.py, you should see the
script
> >> runs
> >> > without error. Thus, the issue certainly appears to be calling
the
> >> python
> >> > code from within MET.
> >> >
> >> > I hope the instructions makes sense. Please let me know if you
have
> any
> >> > questions. I'm happy to help you debug this issue.
> >> >
> >> > Thanks!
> >> > Craig
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: Craig Schwartz
Time: Mon Oct 14 09:24:21 2019
Hi John,
Thank you for the update. Definitely agree that MET should execute
the
python scripts with the user's python environment.
Your proposed method seems basically the same as my work-around, where
I
just call the python script from the driving ksh script and then write
out
a temporary GRIB file that MET actually reads. But, your proposed
approach
is better, because there's no need for the user to explicitly write
out a
temporary file or call the python script from elsewhere.
I think as long as very little changes from the user perspective, your
method seems like a good idea. Thanks for asking my thoughts.
Best,
Craig
On Thu, Oct 10, 2019 at 6:32 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Craig,
>
> George McCabe, a software engineer here, did some more testing of
h5py and
> the pygrib modules on cheyenne and ran into the same troubles you
and I
> found.
>
> As a result of this, we're rethinking how the python embedding
should work
> for met-9.0. The underlying issue is that MET is calling the python
> interpreter that it was compiled against, not whatever version you
happen
> to be running in your environment. And the runtime linker can run
into
> conflicts with the versions of the python libraries and those same
> libraries that MET was compiled against.
>
> Ideally, MET would execute the python scripts with the user's python
> environment and not the one when MET was compiled. Our current idea
for
> met-9.0 is this:
>
> (1) The user specifies the python command to run, including the path
to
> python:
> /path/to/python sample.py a b c
> (2) MET slightly modifies that call and forks a process to execute
the
> following:
> /path/to/python *write_pickle.py temp.pickle* sample.py a b c
> The write_pickle.py script would live in the MET repo. It's a
wrapper
> that would execute the user's script with its arguments (sample.py a
b c)
> and write the resulting data to a python pickle file (temp.pickle,
but we'd
> pick a better temp file name).
> (3) Then MET's python interpreter would execute this command:
> read_pickle.py temp.pickle
> And read the data from that temporary pickle file back in via
memory.
>
> From the user's perspective, there would be very little change. But
the
> details have changed a lot under the covers. The goal is to execute
the
> user's python script in their own environment, so that what you get
from
> running the python interpreter on the command line is the same as
what you
> get from running MET.
>
> Any thoughts or feedback on that?
>
> Thanks,
> John
>
> On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> >
> > Hi John,
> >
> > Have you been able to make any progress on this issue?
> >
> > Also, wanted to let you know I found a hack around the bug
(underlying
> > problem is not solved). Basically, from the driving ksh script,
right
> > before calling grid_stat, I call the python code, which I changed
to
> output
> > a GRIB2 file with one record containing the "met_data" NUMPY
array.
> Then,
> > I simply have MET read data from the temporary GRIB2 file I've
created,
> > thus, bypassing calling the python code from MET. Seems to work.
> >
> > Craig
> >
> >
> > On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz
<schwartz at ucar.edu>
> wrote:
> >
> > > Hi John,
> > >
> > > Thank you for the update. Only thing I could think of was some
> conflict
> > > between MET and python--kind of along the lines of your #3
bullet.
> It's
> > > interesting that the code only crashes when accessing a
".values" or
> > > ".data" (in pygrib, x.data()[0] === x.values and gives the same
> > > segfault). Accessing other keys (like x.missingValue) doesn't
give a
> > > crash. Could there be anything special about ".values" or
".data"
> > > conflicting with MET?
> > >
> > > Thanks again for looking into this. Let me know if there's
anything I
> > > can do to help.
> > >
> > > Best,
> > > Craig
> > >
> > >
> > > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
> <met_help at ucar.
> > > edu> wrote:
> > >
> > >> Craig,
> > >>
> > >> I wanted to let you know that I've been trying to debug this
issue
> > today.
> > >> But unfortunately, I don't have a solution for you yet. Thanks
for
> > >> providing your test data. I was definitely able to replicate
the
> error
> > >> and
> > >> agree that the segfault is occurring at the between the two
debug
> print
> > >> statements you added.
> > >>
> > >> Coincidentally, Julie Prestopnik and I have also been working
on
> python
> > >> embedding cheyenne for other reasons.
> > >>
> > >> We're testing the met/8.1_python module build from inside the
> ncar_pylib
> > >> environment. And in there, I observed that...
> > >>
> > >> (1) Two sample python embedding scripts from the MET website (
> > >>
> > >>
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > >> )
> > >> do actually run fine (one reading a binary input file and
another
> > reading
> > >> NetCDF).
> > >>
> > >> (2) When I edit those python scripts by adding (import h5py) to
import
> > the
> > >> HDF5 module, then they segfault when run through MET. Thinking
you
> may
> > be
> > >> having a similar issue, I tested all your imports one-by-one,
but none
> > of
> > >> them triggered a segfault. It really is when executing that
single
> line
> > >> you mentioned.
> > >>
> > >> (3) Now I'm wondering if there's somehow a conflict in MET's
dependent
> > >> libraries and those used by python. Investigating this I did
find a
> > >> slight
> > >> mismatch... python was compiled with intel/17.0.1 while we were
> > compiling
> > >> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to
see if
> > >> anything changes.
> > >>
> > >> Just wanted to let you know that we're aware of this issue and
looking
> > >> into
> > >> it!
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >>
> > >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> > met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> > >> > Transaction: Ticket created by schwartz at ucar.edu
> > >> > Queue: met_help
> > >> > Subject: Python embedding and pygrib
> > >> > Owner: Nobody
> > >> > Requestors: schwartz at ucar.edu
> > >> > Status: new
> > >> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> > >
> > >> >
> > >> >
> > >> > Hello MET Help,
> > >> >
> > >> > There still seems to be an issue using the python embedding
feature
> in
> > >> > association with pygrib, which I'd like to use to read GRIB
data
> into
> > >> > python, and then eventually pass data to MET.
> > >> >
> > >> > There are two ways for you to test:
> > >> >
> > >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
/glade/scratch/$USER.
> > Then:
> > >> >
> > >> > cd /glade/scratch/$USER/cloud_vx
> > >> > cd bin/MET
> > >> > ./driver_script_with_python_manual.ksh
> > >> >
> > >> > That should run everything without you having to change
anything.
> > >> Output
> > >> > of the failed run will be in
/glade/scratch/$USER/cloud_vx/metprd
> > >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> > >> >
> > >> > If you look in ./driver_script_with_python_manual.ksh ,
you'll see
> > some
> > >> > "module load" statements near the top. I've also tried other
MET
> > builds
> > >> > located at /glade/p/ral/jntp/MET/MET_releases, but always get
the
> same
> > >> seg
> > >> > fault.
> > >> >
> > >> > 2) Running the full set of scripts in #1 means the MET
configuration
> > >> file
> > >> > is populated using environmental variables. If you want
perhaps a
> > >> simpler
> > >> > test, just copy everything in
> > >> >
> > >> >
> > >>
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > >> > and run MET:
> > >> >
> > >> > /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> > PYTHON_NUMPY
> > >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> > >> >
> > >> > No matter which way you test, the issue occurs in line 200 of
> > >> > "python_stuff.
> > >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > >> > read_var = x.values
> > >> >
> > >> > The ".values" is a pygrib method. MET doesn't seem to like
that
> > >> >
> > >> > Note that In
/glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > >> > lowCloudFrac/SATCORPS,
> > >> > if you just do python python_script_fcst.py, you should see
the
> script
> > >> runs
> > >> > without error. Thus, the issue certainly appears to be
calling the
> > >> python
> > >> > code from within MET.
> > >> >
> > >> > I hope the instructions makes sense. Please let me know if
you have
> > any
> > >> > questions. I'm happy to help you debug this issue.
> > >> >
> > >> > Thanks!
> > >> > Craig
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: John Halley Gotway
Time: Mon Oct 14 16:29:07 2019
Craig,
Great, thanks for the feedback. Glad to hear you've found a work-
around
for the shortcomings we have on Cheyenne. I'll go ahead and resolve
this
ticket now.
Thanks,
John
On Mon, Oct 14, 2019 at 9:24 AM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
>
> Hi John,
>
> Thank you for the update. Definitely agree that MET should execute
the
> python scripts with the user's python environment.
>
> Your proposed method seems basically the same as my work-around,
where I
> just call the python script from the driving ksh script and then
write out
> a temporary GRIB file that MET actually reads. But, your proposed
approach
> is better, because there's no need for the user to explicitly write
out a
> temporary file or call the python script from elsewhere.
>
> I think as long as very little changes from the user perspective,
your
> method seems like a good idea. Thanks for asking my thoughts.
>
> Best,
> Craig
>
>
> On Thu, Oct 10, 2019 at 6:32 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Craig,
> >
> > George McCabe, a software engineer here, did some more testing of
h5py
> and
> > the pygrib modules on cheyenne and ran into the same troubles you
and I
> > found.
> >
> > As a result of this, we're rethinking how the python embedding
should
> work
> > for met-9.0. The underlying issue is that MET is calling the
python
> > interpreter that it was compiled against, not whatever version you
happen
> > to be running in your environment. And the runtime linker can run
into
> > conflicts with the versions of the python libraries and those same
> > libraries that MET was compiled against.
> >
> > Ideally, MET would execute the python scripts with the user's
python
> > environment and not the one when MET was compiled. Our current
idea for
> > met-9.0 is this:
> >
> > (1) The user specifies the python command to run, including the
path to
> > python:
> > /path/to/python sample.py a b c
> > (2) MET slightly modifies that call and forks a process to execute
the
> > following:
> > /path/to/python *write_pickle.py temp.pickle* sample.py a b c
> > The write_pickle.py script would live in the MET repo. It's a
wrapper
> > that would execute the user's script with its arguments (sample.py
a b c)
> > and write the resulting data to a python pickle file (temp.pickle,
but
> we'd
> > pick a better temp file name).
> > (3) Then MET's python interpreter would execute this command:
> > read_pickle.py temp.pickle
> > And read the data from that temporary pickle file back in via
memory.
> >
> > From the user's perspective, there would be very little change.
But the
> > details have changed a lot under the covers. The goal is to
execute the
> > user's python script in their own environment, so that what you
get from
> > running the python interpreter on the command line is the same as
what
> you
> > get from running MET.
> >
> > Any thoughts or feedback on that?
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> > >
> > > Hi John,
> > >
> > > Have you been able to make any progress on this issue?
> > >
> > > Also, wanted to let you know I found a hack around the bug
(underlying
> > > problem is not solved). Basically, from the driving ksh script,
right
> > > before calling grid_stat, I call the python code, which I
changed to
> > output
> > > a GRIB2 file with one record containing the "met_data" NUMPY
array.
> > Then,
> > > I simply have MET read data from the temporary GRIB2 file I've
created,
> > > thus, bypassing calling the python code from MET. Seems to
work.
> > >
> > > Craig
> > >
> > >
> > > On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz
<schwartz at ucar.edu>
> > wrote:
> > >
> > > > Hi John,
> > > >
> > > > Thank you for the update. Only thing I could think of was
some
> > conflict
> > > > between MET and python--kind of along the lines of your #3
bullet.
> > It's
> > > > interesting that the code only crashes when accessing a
".values" or
> > > > ".data" (in pygrib, x.data()[0] === x.values and gives the
same
> > > > segfault). Accessing other keys (like x.missingValue) doesn't
give a
> > > > crash. Could there be anything special about ".values" or
".data"
> > > > conflicting with MET?
> > > >
> > > > Thanks again for looking into this. Let me know if there's
anything
> I
> > > > can do to help.
> > > >
> > > > Best,
> > > > Craig
> > > >
> > > >
> > > > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
> > <met_help at ucar.
> > > > edu> wrote:
> > > >
> > > >> Craig,
> > > >>
> > > >> I wanted to let you know that I've been trying to debug this
issue
> > > today.
> > > >> But unfortunately, I don't have a solution for you yet.
Thanks for
> > > >> providing your test data. I was definitely able to replicate
the
> > error
> > > >> and
> > > >> agree that the segfault is occurring at the between the two
debug
> > print
> > > >> statements you added.
> > > >>
> > > >> Coincidentally, Julie Prestopnik and I have also been working
on
> > python
> > > >> embedding cheyenne for other reasons.
> > > >>
> > > >> We're testing the met/8.1_python module build from inside the
> > ncar_pylib
> > > >> environment. And in there, I observed that...
> > > >>
> > > >> (1) Two sample python embedding scripts from the MET website
(
> > > >>
> > > >>
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > > >> )
> > > >> do actually run fine (one reading a binary input file and
another
> > > reading
> > > >> NetCDF).
> > > >>
> > > >> (2) When I edit those python scripts by adding (import h5py)
to
> import
> > > the
> > > >> HDF5 module, then they segfault when run through MET.
Thinking you
> > may
> > > be
> > > >> having a similar issue, I tested all your imports one-by-one,
but
> none
> > > of
> > > >> them triggered a segfault. It really is when executing that
single
> > line
> > > >> you mentioned.
> > > >>
> > > >> (3) Now I'm wondering if there's somehow a conflict in MET's
> dependent
> > > >> libraries and those used by python. Investigating this I did
find a
> > > >> slight
> > > >> mismatch... python was compiled with intel/17.0.1 while we
were
> > > compiling
> > > >> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1 to
see
> if
> > > >> anything changes.
> > > >>
> > > >> Just wanted to let you know that we're aware of this issue
and
> looking
> > > >> into
> > > >> it!
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> > > met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> > > >> > Transaction: Ticket created by schwartz at ucar.edu
> > > >> > Queue: met_help
> > > >> > Subject: Python embedding and pygrib
> > > >> > Owner: Nobody
> > > >> > Requestors: schwartz at ucar.edu
> > > >> > Status: new
> > > >> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> > > >
> > > >> >
> > > >> >
> > > >> > Hello MET Help,
> > > >> >
> > > >> > There still seems to be an issue using the python embedding
> feature
> > in
> > > >> > association with pygrib, which I'd like to use to read GRIB
data
> > into
> > > >> > python, and then eventually pass data to MET.
> > > >> >
> > > >> > There are two ways for you to test:
> > > >> >
> > > >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
/glade/scratch/$USER.
> > > Then:
> > > >> >
> > > >> > cd /glade/scratch/$USER/cloud_vx
> > > >> > cd bin/MET
> > > >> > ./driver_script_with_python_manual.ksh
> > > >> >
> > > >> > That should run everything without you having to change
anything.
> > > >> Output
> > > >> > of the failed run will be in
/glade/scratch/$USER/cloud_vx/metprd
> > > >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> > > >> >
> > > >> > If you look in ./driver_script_with_python_manual.ksh ,
you'll see
> > > some
> > > >> > "module load" statements near the top. I've also tried
other MET
> > > builds
> > > >> > located at /glade/p/ral/jntp/MET/MET_releases, but always
get the
> > same
> > > >> seg
> > > >> > fault.
> > > >> >
> > > >> > 2) Running the full set of scripts in #1 means the MET
> configuration
> > > >> file
> > > >> > is populated using environmental variables. If you want
perhaps a
> > > >> simpler
> > > >> > test, just copy everything in
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > > >> > and run MET:
> > > >> >
> > > >> > /glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> > > PYTHON_NUMPY
> > > >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> > > >> >
> > > >> > No matter which way you test, the issue occurs in line 200
of
> > > >> > "python_stuff.
> > > >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > > >> > read_var = x.values
> > > >> >
> > > >> > The ".values" is a pygrib method. MET doesn't seem to like
that
> > > >> >
> > > >> > Note that In
/glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > > >> > lowCloudFrac/SATCORPS,
> > > >> > if you just do python python_script_fcst.py, you should see
the
> > script
> > > >> runs
> > > >> > without error. Thus, the issue certainly appears to be
calling
> the
> > > >> python
> > > >> > code from within MET.
> > > >> >
> > > >> > I hope the instructions makes sense. Please let me know if
you
> have
> > > any
> > > >> > questions. I'm happy to help you debug this issue.
> > > >> >
> > > >> > Thanks!
> > > >> > Craig
> > > >> >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: Craig Schwartz
Time: Fri Jun 19 14:00:19 2020
Hi John,
I know that python embedding was re-engineered under-the-hood for
METv9.0,
but when I'm using the MET version here
(/glade/p/ral/jntp/MET/MET_releases/9.0)
I'm still having the same problem as before (above thread) when
loading the
pygrib module. Same failure at the same place.
Do I need to be using a different MET executable/version (like MET
v9.0.2)?
Thanks,
Craig
On Mon, Oct 14, 2019 at 4:29 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Craig,
>
> Great, thanks for the feedback. Glad to hear you've found a work-
around
> for the shortcomings we have on Cheyenne. I'll go ahead and resolve
this
> ticket now.
>
> Thanks,
> John
>
> On Mon, Oct 14, 2019 at 9:24 AM Craig Schwartz via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> >
> > Hi John,
> >
> > Thank you for the update. Definitely agree that MET should
execute the
> > python scripts with the user's python environment.
> >
> > Your proposed method seems basically the same as my work-around,
where I
> > just call the python script from the driving ksh script and then
write
> out
> > a temporary GRIB file that MET actually reads. But, your proposed
> approach
> > is better, because there's no need for the user to explicitly
write out a
> > temporary file or call the python script from elsewhere.
> >
> > I think as long as very little changes from the user perspective,
your
> > method seems like a good idea. Thanks for asking my thoughts.
> >
> > Best,
> > Craig
> >
> >
> > On Thu, Oct 10, 2019 at 6:32 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Craig,
> > >
> > > George McCabe, a software engineer here, did some more testing
of h5py
> > and
> > > the pygrib modules on cheyenne and ran into the same troubles
you and I
> > > found.
> > >
> > > As a result of this, we're rethinking how the python embedding
should
> > work
> > > for met-9.0. The underlying issue is that MET is calling the
python
> > > interpreter that it was compiled against, not whatever version
you
> happen
> > > to be running in your environment. And the runtime linker can
run into
> > > conflicts with the versions of the python libraries and those
same
> > > libraries that MET was compiled against.
> > >
> > > Ideally, MET would execute the python scripts with the user's
python
> > > environment and not the one when MET was compiled. Our current
idea
> for
> > > met-9.0 is this:
> > >
> > > (1) The user specifies the python command to run, including the
path to
> > > python:
> > > /path/to/python sample.py a b c
> > > (2) MET slightly modifies that call and forks a process to
execute the
> > > following:
> > > /path/to/python *write_pickle.py temp.pickle* sample.py a b c
> > > The write_pickle.py script would live in the MET repo. It's a
> wrapper
> > > that would execute the user's script with its arguments
(sample.py a b
> c)
> > > and write the resulting data to a python pickle file
(temp.pickle, but
> > we'd
> > > pick a better temp file name).
> > > (3) Then MET's python interpreter would execute this command:
> > > read_pickle.py temp.pickle
> > > And read the data from that temporary pickle file back in via
> memory.
> > >
> > > From the user's perspective, there would be very little change.
But
> the
> > > details have changed a lot under the covers. The goal is to
execute
> the
> > > user's python script in their own environment, so that what you
get
> from
> > > running the python interpreter on the command line is the same
as what
> > you
> > > get from running MET.
> > >
> > > Any thoughts or feedback on that?
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
>
> > > >
> > > > Hi John,
> > > >
> > > > Have you been able to make any progress on this issue?
> > > >
> > > > Also, wanted to let you know I found a hack around the bug
> (underlying
> > > > problem is not solved). Basically, from the driving ksh
script,
> right
> > > > before calling grid_stat, I call the python code, which I
changed to
> > > output
> > > > a GRIB2 file with one record containing the "met_data" NUMPY
array.
> > > Then,
> > > > I simply have MET read data from the temporary GRIB2 file I've
> created,
> > > > thus, bypassing calling the python code from MET. Seems to
work.
> > > >
> > > > Craig
> > > >
> > > >
> > > > On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz
<schwartz at ucar.edu>
> > > wrote:
> > > >
> > > > > Hi John,
> > > > >
> > > > > Thank you for the update. Only thing I could think of was
some
> > > conflict
> > > > > between MET and python--kind of along the lines of your #3
bullet.
> > > It's
> > > > > interesting that the code only crashes when accessing a
".values"
> or
> > > > > ".data" (in pygrib, x.data()[0] === x.values and gives the
same
> > > > > segfault). Accessing other keys (like x.missingValue)
doesn't
> give a
> > > > > crash. Could there be anything special about ".values" or
".data"
> > > > > conflicting with MET?
> > > > >
> > > > > Thanks again for looking into this. Let me know if there's
> anything
> > I
> > > > > can do to help.
> > > > >
> > > > > Best,
> > > > > Craig
> > > > >
> > > > >
> > > > > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
> > > <met_help at ucar.
> > > > > edu> wrote:
> > > > >
> > > > >> Craig,
> > > > >>
> > > > >> I wanted to let you know that I've been trying to debug
this issue
> > > > today.
> > > > >> But unfortunately, I don't have a solution for you yet.
Thanks
> for
> > > > >> providing your test data. I was definitely able to
replicate the
> > > error
> > > > >> and
> > > > >> agree that the segfault is occurring at the between the two
debug
> > > print
> > > > >> statements you added.
> > > > >>
> > > > >> Coincidentally, Julie Prestopnik and I have also been
working on
> > > python
> > > > >> embedding cheyenne for other reasons.
> > > > >>
> > > > >> We're testing the met/8.1_python module build from inside
the
> > > ncar_pylib
> > > > >> environment. And in there, I observed that...
> > > > >>
> > > > >> (1) Two sample python embedding scripts from the MET
website (
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > > > >> )
> > > > >> do actually run fine (one reading a binary input file and
another
> > > > reading
> > > > >> NetCDF).
> > > > >>
> > > > >> (2) When I edit those python scripts by adding (import
h5py) to
> > import
> > > > the
> > > > >> HDF5 module, then they segfault when run through MET.
Thinking
> you
> > > may
> > > > be
> > > > >> having a similar issue, I tested all your imports one-by-
one, but
> > none
> > > > of
> > > > >> them triggered a segfault. It really is when executing
that
> single
> > > line
> > > > >> you mentioned.
> > > > >>
> > > > >> (3) Now I'm wondering if there's somehow a conflict in
MET's
> > dependent
> > > > >> libraries and those used by python. Investigating this I
did
> find a
> > > > >> slight
> > > > >> mismatch... python was compiled with intel/17.0.1 while we
were
> > > > compiling
> > > > >> MET with intel/18.0.5. So I'm recompiling MET with 17.0.1
to see
> > if
> > > > >> anything changes.
> > > > >>
> > > > >> Just wanted to let you know that we're aware of this issue
and
> > looking
> > > > >> into
> > > > >> it!
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> > > > met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> > > > >> > Transaction: Ticket created by schwartz at ucar.edu
> > > > >> > Queue: met_help
> > > > >> > Subject: Python embedding and pygrib
> > > > >> > Owner: Nobody
> > > > >> > Requestors: schwartz at ucar.edu
> > > > >> > Status: new
> > > > >> > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> > > > >
> > > > >> >
> > > > >> >
> > > > >> > Hello MET Help,
> > > > >> >
> > > > >> > There still seems to be an issue using the python
embedding
> > feature
> > > in
> > > > >> > association with pygrib, which I'd like to use to read
GRIB data
> > > into
> > > > >> > python, and then eventually pass data to MET.
> > > > >> >
> > > > >> > There are two ways for you to test:
> > > > >> >
> > > > >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
> /glade/scratch/$USER.
> > > > Then:
> > > > >> >
> > > > >> > cd /glade/scratch/$USER/cloud_vx
> > > > >> > cd bin/MET
> > > > >> > ./driver_script_with_python_manual.ksh
> > > > >> >
> > > > >> > That should run everything without you having to change
> anything.
> > > > >> Output
> > > > >> > of the failed run will be in
> /glade/scratch/$USER/cloud_vx/metprd
> > > > >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> > > > >> >
> > > > >> > If you look in ./driver_script_with_python_manual.ksh ,
you'll
> see
> > > > some
> > > > >> > "module load" statements near the top. I've also tried
other
> MET
> > > > builds
> > > > >> > located at /glade/p/ral/jntp/MET/MET_releases, but always
get
> the
> > > same
> > > > >> seg
> > > > >> > fault.
> > > > >> >
> > > > >> > 2) Running the full set of scripts in #1 means the MET
> > configuration
> > > > >> file
> > > > >> > is populated using environmental variables. If you want
> perhaps a
> > > > >> simpler
> > > > >> > test, just copy everything in
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > > > >> > and run MET:
> > > > >> >
> > > > >> >
/glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> > > > PYTHON_NUMPY
> > > > >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> > > > >> >
> > > > >> > No matter which way you test, the issue occurs in line
200 of
> > > > >> > "python_stuff.
> > > > >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > > > >> > read_var = x.values
> > > > >> >
> > > > >> > The ".values" is a pygrib method. MET doesn't seem to
like that
> > > > >> >
> > > > >> > Note that In
> /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > > > >> > lowCloudFrac/SATCORPS,
> > > > >> > if you just do python python_script_fcst.py, you should
see the
> > > script
> > > > >> runs
> > > > >> > without error. Thus, the issue certainly appears to be
calling
> > the
> > > > >> python
> > > > >> > code from within MET.
> > > > >> >
> > > > >> > I hope the instructions makes sense. Please let me know
if you
> > have
> > > > any
> > > > >> > questions. I'm happy to help you debug this issue.
> > > > >> >
> > > > >> > Thanks!
> > > > >> > Craig
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: John Halley Gotway
Time: Fri Jun 19 14:32:14 2020
No, you don't need to use a different executable. However, you do need
to
set the MET_PYTHON_EXE environment variable. That triggers the logic
to
execute the script with the user's version of python rather than the
one
defined at compilation time.
Please look on page 472 of the user's guide for a more detailed
explanation:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v9.0.2.pdf
Hope that does the trick!
John
On Fri, Jun 19, 2020 at 2:00 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
>
> Hi John,
>
> I know that python embedding was re-engineered under-the-hood for
METv9.0,
> but when I'm using the MET version here
> (/glade/p/ral/jntp/MET/MET_releases/9.0)
> I'm still having the same problem as before (above thread) when
loading the
> pygrib module. Same failure at the same place.
>
> Do I need to be using a different MET executable/version (like MET
v9.0.2)?
>
> Thanks,
> Craig
>
>
>
>
>
> On Mon, Oct 14, 2019 at 4:29 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Craig,
> >
> > Great, thanks for the feedback. Glad to hear you've found a work-
around
> > for the shortcomings we have on Cheyenne. I'll go ahead and
resolve this
> > ticket now.
> >
> > Thanks,
> > John
> >
> > On Mon, Oct 14, 2019 at 9:24 AM Craig Schwartz via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> > >
> > > Hi John,
> > >
> > > Thank you for the update. Definitely agree that MET should
execute the
> > > python scripts with the user's python environment.
> > >
> > > Your proposed method seems basically the same as my work-around,
where
> I
> > > just call the python script from the driving ksh script and then
write
> > out
> > > a temporary GRIB file that MET actually reads. But, your
proposed
> > approach
> > > is better, because there's no need for the user to explicitly
write
> out a
> > > temporary file or call the python script from elsewhere.
> > >
> > > I think as long as very little changes from the user
perspective, your
> > > method seems like a good idea. Thanks for asking my thoughts.
> > >
> > > Best,
> > > Craig
> > >
> > >
> > > On Thu, Oct 10, 2019 at 6:32 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Craig,
> > > >
> > > > George McCabe, a software engineer here, did some more testing
of
> h5py
> > > and
> > > > the pygrib modules on cheyenne and ran into the same troubles
you
> and I
> > > > found.
> > > >
> > > > As a result of this, we're rethinking how the python embedding
should
> > > work
> > > > for met-9.0. The underlying issue is that MET is calling the
python
> > > > interpreter that it was compiled against, not whatever version
you
> > happen
> > > > to be running in your environment. And the runtime linker can
run
> into
> > > > conflicts with the versions of the python libraries and those
same
> > > > libraries that MET was compiled against.
> > > >
> > > > Ideally, MET would execute the python scripts with the user's
python
> > > > environment and not the one when MET was compiled. Our
current idea
> > for
> > > > met-9.0 is this:
> > > >
> > > > (1) The user specifies the python command to run, including
the path
> to
> > > > python:
> > > > /path/to/python sample.py a b c
> > > > (2) MET slightly modifies that call and forks a process to
execute
> the
> > > > following:
> > > > /path/to/python *write_pickle.py temp.pickle* sample.py a b
c
> > > > The write_pickle.py script would live in the MET repo. It's
a
> > wrapper
> > > > that would execute the user's script with its arguments
(sample.py a
> b
> > c)
> > > > and write the resulting data to a python pickle file
(temp.pickle,
> but
> > > we'd
> > > > pick a better temp file name).
> > > > (3) Then MET's python interpreter would execute this command:
> > > > read_pickle.py temp.pickle
> > > > And read the data from that temporary pickle file back in
via
> > memory.
> > > >
> > > > From the user's perspective, there would be very little
change. But
> > the
> > > > details have changed a lot under the covers. The goal is to
execute
> > the
> > > > user's python script in their own environment, so that what
you get
> > from
> > > > running the python interpreter on the command line is the same
as
> what
> > > you
> > > > get from running MET.
> > > >
> > > > Any thoughts or feedback on that?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Have you been able to make any progress on this issue?
> > > > >
> > > > > Also, wanted to let you know I found a hack around the bug
> > (underlying
> > > > > problem is not solved). Basically, from the driving ksh
script,
> > right
> > > > > before calling grid_stat, I call the python code, which I
changed
> to
> > > > output
> > > > > a GRIB2 file with one record containing the "met_data" NUMPY
array.
> > > > Then,
> > > > > I simply have MET read data from the temporary GRIB2 file
I've
> > created,
> > > > > thus, bypassing calling the python code from MET. Seems to
work.
> > > > >
> > > > > Craig
> > > > >
> > > > >
> > > > > On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz
<schwartz at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thank you for the update. Only thing I could think of was
some
> > > > conflict
> > > > > > between MET and python--kind of along the lines of your #3
> bullet.
> > > > It's
> > > > > > interesting that the code only crashes when accessing a
".values"
> > or
> > > > > > ".data" (in pygrib, x.data()[0] === x.values and gives
the same
> > > > > > segfault). Accessing other keys (like x.missingValue)
doesn't
> > give a
> > > > > > crash. Could there be anything special about ".values" or
> ".data"
> > > > > > conflicting with MET?
> > > > > >
> > > > > > Thanks again for looking into this. Let me know if
there's
> > anything
> > > I
> > > > > > can do to help.
> > > > > >
> > > > > > Best,
> > > > > > Craig
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via RT
> > > > <met_help at ucar.
> > > > > > edu> wrote:
> > > > > >
> > > > > >> Craig,
> > > > > >>
> > > > > >> I wanted to let you know that I've been trying to debug
this
> issue
> > > > > today.
> > > > > >> But unfortunately, I don't have a solution for you yet.
Thanks
> > for
> > > > > >> providing your test data. I was definitely able to
replicate
> the
> > > > error
> > > > > >> and
> > > > > >> agree that the segfault is occurring at the between the
two
> debug
> > > > print
> > > > > >> statements you added.
> > > > > >>
> > > > > >> Coincidentally, Julie Prestopnik and I have also been
working on
> > > > python
> > > > > >> embedding cheyenne for other reasons.
> > > > > >>
> > > > > >> We're testing the met/8.1_python module build from inside
the
> > > > ncar_pylib
> > > > > >> environment. And in there, I observed that...
> > > > > >>
> > > > > >> (1) Two sample python embedding scripts from the MET
website (
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > > > > >> )
> > > > > >> do actually run fine (one reading a binary input file and
> another
> > > > > reading
> > > > > >> NetCDF).
> > > > > >>
> > > > > >> (2) When I edit those python scripts by adding (import
h5py) to
> > > import
> > > > > the
> > > > > >> HDF5 module, then they segfault when run through MET.
Thinking
> > you
> > > > may
> > > > > be
> > > > > >> having a similar issue, I tested all your imports one-by-
one,
> but
> > > none
> > > > > of
> > > > > >> them triggered a segfault. It really is when executing
that
> > single
> > > > line
> > > > > >> you mentioned.
> > > > > >>
> > > > > >> (3) Now I'm wondering if there's somehow a conflict in
MET's
> > > dependent
> > > > > >> libraries and those used by python. Investigating this I
did
> > find a
> > > > > >> slight
> > > > > >> mismatch... python was compiled with intel/17.0.1 while
we were
> > > > > compiling
> > > > > >> MET with intel/18.0.5. So I'm recompiling MET with
17.0.1 to
> see
> > > if
> > > > > >> anything changes.
> > > > > >>
> > > > > >> Just wanted to let you know that we're aware of this
issue and
> > > looking
> > > > > >> into
> > > > > >> it!
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> > > > > met_help at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted upon.
> > > > > >> > Transaction: Ticket created by schwartz at ucar.edu
> > > > > >> > Queue: met_help
> > > > > >> > Subject: Python embedding and pygrib
> > > > > >> > Owner: Nobody
> > > > > >> > Requestors: schwartz at ucar.edu
> > > > > >> > Status: new
> > > > > >> > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> > > > > >
> > > > > >> >
> > > > > >> >
> > > > > >> > Hello MET Help,
> > > > > >> >
> > > > > >> > There still seems to be an issue using the python
embedding
> > > feature
> > > > in
> > > > > >> > association with pygrib, which I'd like to use to read
GRIB
> data
> > > > into
> > > > > >> > python, and then eventually pass data to MET.
> > > > > >> >
> > > > > >> > There are two ways for you to test:
> > > > > >> >
> > > > > >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
> > /glade/scratch/$USER.
> > > > > Then:
> > > > > >> >
> > > > > >> > cd /glade/scratch/$USER/cloud_vx
> > > > > >> > cd bin/MET
> > > > > >> > ./driver_script_with_python_manual.ksh
> > > > > >> >
> > > > > >> > That should run everything without you having to change
> > anything.
> > > > > >> Output
> > > > > >> > of the failed run will be in
> > /glade/scratch/$USER/cloud_vx/metprd
> > > > > >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> > > > > >> >
> > > > > >> > If you look in ./driver_script_with_python_manual.ksh ,
you'll
> > see
> > > > > some
> > > > > >> > "module load" statements near the top. I've also tried
other
> > MET
> > > > > builds
> > > > > >> > located at /glade/p/ral/jntp/MET/MET_releases, but
always get
> > the
> > > > same
> > > > > >> seg
> > > > > >> > fault.
> > > > > >> >
> > > > > >> > 2) Running the full set of scripts in #1 means the MET
> > > configuration
> > > > > >> file
> > > > > >> > is populated using environmental variables. If you
want
> > perhaps a
> > > > > >> simpler
> > > > > >> > test, just copy everything in
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > > > > >> > and run MET:
> > > > > >> >
> > > > > >> >
/glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> > > > > PYTHON_NUMPY
> > > > > >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> > > > > >> >
> > > > > >> > No matter which way you test, the issue occurs in line
200 of
> > > > > >> > "python_stuff.
> > > > > >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > > > > >> > read_var = x.values
> > > > > >> >
> > > > > >> > The ".values" is a pygrib method. MET doesn't seem to
like
> that
> > > > > >> >
> > > > > >> > Note that In
> > /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > > > > >> > lowCloudFrac/SATCORPS,
> > > > > >> > if you just do python python_script_fcst.py, you should
see
> the
> > > > script
> > > > > >> runs
> > > > > >> > without error. Thus, the issue certainly appears to be
> calling
> > > the
> > > > > >> python
> > > > > >> > code from within MET.
> > > > > >> >
> > > > > >> > I hope the instructions makes sense. Please let me
know if
> you
> > > have
> > > > > any
> > > > > >> > questions. I'm happy to help you debug this issue.
> > > > > >> >
> > > > > >> > Thanks!
> > > > > >> > Craig
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Python embedding and pygrib
From: Craig Schwartz
Time: Fri Jun 19 15:07:04 2020
Ahh, yes, I knew that actually. Thank you for the reminder.
That does indeed seem to have done the trick.
Thanks,
Craig
On Fri, Jun 19, 2020 at 2:32 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> No, you don't need to use a different executable. However, you do
need to
> set the MET_PYTHON_EXE environment variable. That triggers the logic
to
> execute the script with the user's version of python rather than the
one
> defined at compilation time.
>
> Please look on page 472 of the user's guide for a more detailed
> explanation:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.2.pdf
>
> Hope that does the trick!
>
> John
>
> On Fri, Jun 19, 2020 at 2:00 PM Craig Schwartz via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> >
> > Hi John,
> >
> > I know that python embedding was re-engineered under-the-hood for
> METv9.0,
> > but when I'm using the MET version here
> > (/glade/p/ral/jntp/MET/MET_releases/9.0)
> > I'm still having the same problem as before (above thread) when
loading
> the
> > pygrib module. Same failure at the same place.
> >
> > Do I need to be using a different MET executable/version (like MET
> v9.0.2)?
> >
> > Thanks,
> > Craig
> >
> >
> >
> >
> >
> > On Mon, Oct 14, 2019 at 4:29 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Craig,
> > >
> > > Great, thanks for the feedback. Glad to hear you've found a
> work-around
> > > for the shortcomings we have on Cheyenne. I'll go ahead and
resolve
> this
> > > ticket now.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Oct 14, 2019 at 9:24 AM Craig Schwartz via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
>
> > > >
> > > > Hi John,
> > > >
> > > > Thank you for the update. Definitely agree that MET should
execute
> the
> > > > python scripts with the user's python environment.
> > > >
> > > > Your proposed method seems basically the same as my work-
around,
> where
> > I
> > > > just call the python script from the driving ksh script and
then
> write
> > > out
> > > > a temporary GRIB file that MET actually reads. But, your
proposed
> > > approach
> > > > is better, because there's no need for the user to explicitly
write
> > out a
> > > > temporary file or call the python script from elsewhere.
> > > >
> > > > I think as long as very little changes from the user
perspective,
> your
> > > > method seems like a good idea. Thanks for asking my thoughts.
> > > >
> > > > Best,
> > > > Craig
> > > >
> > > >
> > > > On Thu, Oct 10, 2019 at 6:32 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Craig,
> > > > >
> > > > > George McCabe, a software engineer here, did some more
testing of
> > h5py
> > > > and
> > > > > the pygrib modules on cheyenne and ran into the same
troubles you
> > and I
> > > > > found.
> > > > >
> > > > > As a result of this, we're rethinking how the python
embedding
> should
> > > > work
> > > > > for met-9.0. The underlying issue is that MET is calling
the
> python
> > > > > interpreter that it was compiled against, not whatever
version you
> > > happen
> > > > > to be running in your environment. And the runtime linker
can run
> > into
> > > > > conflicts with the versions of the python libraries and
those same
> > > > > libraries that MET was compiled against.
> > > > >
> > > > > Ideally, MET would execute the python scripts with the
user's
> python
> > > > > environment and not the one when MET was compiled. Our
current
> idea
> > > for
> > > > > met-9.0 is this:
> > > > >
> > > > > (1) The user specifies the python command to run, including
the
> path
> > to
> > > > > python:
> > > > > /path/to/python sample.py a b c
> > > > > (2) MET slightly modifies that call and forks a process to
execute
> > the
> > > > > following:
> > > > > /path/to/python *write_pickle.py temp.pickle* sample.py a
b c
> > > > > The write_pickle.py script would live in the MET repo.
It's a
> > > wrapper
> > > > > that would execute the user's script with its arguments
(sample.py
> a
> > b
> > > c)
> > > > > and write the resulting data to a python pickle file
(temp.pickle,
> > but
> > > > we'd
> > > > > pick a better temp file name).
> > > > > (3) Then MET's python interpreter would execute this
command:
> > > > > read_pickle.py temp.pickle
> > > > > And read the data from that temporary pickle file back in
via
> > > memory.
> > > > >
> > > > > From the user's perspective, there would be very little
change.
> But
> > > the
> > > > > details have changed a lot under the covers. The goal is to
> execute
> > > the
> > > > > user's python script in their own environment, so that what
you get
> > > from
> > > > > running the python interpreter on the command line is the
same as
> > what
> > > > you
> > > > > get from running MET.
> > > > >
> > > > > Any thoughts or feedback on that?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Sep 5, 2019 at 10:39 AM Craig Schwartz via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Have you been able to make any progress on this issue?
> > > > > >
> > > > > > Also, wanted to let you know I found a hack around the bug
> > > (underlying
> > > > > > problem is not solved). Basically, from the driving ksh
script,
> > > right
> > > > > > before calling grid_stat, I call the python code, which I
changed
> > to
> > > > > output
> > > > > > a GRIB2 file with one record containing the "met_data"
NUMPY
> array.
> > > > > Then,
> > > > > > I simply have MET read data from the temporary GRIB2 file
I've
> > > created,
> > > > > > thus, bypassing calling the python code from MET. Seems
to work.
> > > > > >
> > > > > > Craig
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 20, 2019 at 11:17 AM Craig Schwartz <
> schwartz at ucar.edu
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Thank you for the update. Only thing I could think of
was some
> > > > > conflict
> > > > > > > between MET and python--kind of along the lines of your
#3
> > bullet.
> > > > > It's
> > > > > > > interesting that the code only crashes when accessing a
> ".values"
> > > or
> > > > > > > ".data" (in pygrib, x.data()[0] === x.values and gives
the
> same
> > > > > > > segfault). Accessing other keys (like x.missingValue)
doesn't
> > > give a
> > > > > > > crash. Could there be anything special about ".values"
or
> > ".data"
> > > > > > > conflicting with MET?
> > > > > > >
> > > > > > > Thanks again for looking into this. Let me know if
there's
> > > anything
> > > > I
> > > > > > > can do to help.
> > > > > > >
> > > > > > > Best,
> > > > > > > Craig
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Aug 19, 2019 at 5:04 PM John Halley Gotway via
RT
> > > > > <met_help at ucar.
> > > > > > > edu> wrote:
> > > > > > >
> > > > > > >> Craig,
> > > > > > >>
> > > > > > >> I wanted to let you know that I've been trying to debug
this
> > issue
> > > > > > today.
> > > > > > >> But unfortunately, I don't have a solution for you yet.
> Thanks
> > > for
> > > > > > >> providing your test data. I was definitely able to
replicate
> > the
> > > > > error
> > > > > > >> and
> > > > > > >> agree that the segfault is occurring at the between the
two
> > debug
> > > > > print
> > > > > > >> statements you added.
> > > > > > >>
> > > > > > >> Coincidentally, Julie Prestopnik and I have also been
working
> on
> > > > > python
> > > > > > >> embedding cheyenne for other reasons.
> > > > > > >>
> > > > > > >> We're testing the met/8.1_python module build from
inside the
> > > > > ncar_pylib
> > > > > > >> environment. And in there, I observed that...
> > > > > > >>
> > > > > > >> (1) Two sample python embedding scripts from the MET
website (
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > > > > > >> )
> > > > > > >> do actually run fine (one reading a binary input file
and
> > another
> > > > > > reading
> > > > > > >> NetCDF).
> > > > > > >>
> > > > > > >> (2) When I edit those python scripts by adding (import
h5py)
> to
> > > > import
> > > > > > the
> > > > > > >> HDF5 module, then they segfault when run through MET.
> Thinking
> > > you
> > > > > may
> > > > > > be
> > > > > > >> having a similar issue, I tested all your imports one-
by-one,
> > but
> > > > none
> > > > > > of
> > > > > > >> them triggered a segfault. It really is when executing
that
> > > single
> > > > > line
> > > > > > >> you mentioned.
> > > > > > >>
> > > > > > >> (3) Now I'm wondering if there's somehow a conflict in
MET's
> > > > dependent
> > > > > > >> libraries and those used by python. Investigating this
I did
> > > find a
> > > > > > >> slight
> > > > > > >> mismatch... python was compiled with intel/17.0.1 while
we
> were
> > > > > > compiling
> > > > > > >> MET with intel/18.0.5. So I'm recompiling MET with
17.0.1 to
> > see
> > > > if
> > > > > > >> anything changes.
> > > > > > >>
> > > > > > >> Just wanted to let you know that we're aware of this
issue and
> > > > looking
> > > > > > >> into
> > > > > > >> it!
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Aug 19, 2019 at 1:04 PM Craig Schwartz via RT <
> > > > > > met_help at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > Mon Aug 19 13:04:06 2019: Request 91588 was acted
upon.
> > > > > > >> > Transaction: Ticket created by schwartz at ucar.edu
> > > > > > >> > Queue: met_help
> > > > > > >> > Subject: Python embedding and pygrib
> > > > > > >> > Owner: Nobody
> > > > > > >> > Requestors: schwartz at ucar.edu
> > > > > > >> > Status: new
> > > > > > >> > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91588
> > > > > > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Hello MET Help,
> > > > > > >> >
> > > > > > >> > There still seems to be an issue using the python
embedding
> > > > feature
> > > > > in
> > > > > > >> > association with pygrib, which I'd like to use to
read GRIB
> > data
> > > > > into
> > > > > > >> > python, and then eventually pass data to MET.
> > > > > > >> >
> > > > > > >> > There are two ways for you to test:
> > > > > > >> >
> > > > > > >> > 1) Copy /glade/scratch/schwartz/cloud_vx to
> > > /glade/scratch/$USER.
> > > > > > Then:
> > > > > > >> >
> > > > > > >> > cd /glade/scratch/$USER/cloud_vx
> > > > > > >> > cd bin/MET
> > > > > > >> > ./driver_script_with_python_manual.ksh
> > > > > > >> >
> > > > > > >> > That should run everything without you having to
change
> > > anything.
> > > > > > >> Output
> > > > > > >> > of the failed run will be in
> > > /glade/scratch/$USER/cloud_vx/metprd
> > > > > > >> > /2017060500/f06/lowCloudFrac/SATCORPS.
> > > > > > >> >
> > > > > > >> > If you look in ./driver_script_with_python_manual.ksh
,
> you'll
> > > see
> > > > > > some
> > > > > > >> > "module load" statements near the top. I've also
tried
> other
> > > MET
> > > > > > builds
> > > > > > >> > located at /glade/p/ral/jntp/MET/MET_releases, but
always
> get
> > > the
> > > > > same
> > > > > > >> seg
> > > > > > >> > fault.
> > > > > > >> >
> > > > > > >> > 2) Running the full set of scripts in #1 means the
MET
> > > > configuration
> > > > > > >> file
> > > > > > >> > is populated using environmental variables. If you
want
> > > perhaps a
> > > > > > >> simpler
> > > > > > >> > test, just copy everything in
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/scratch/schwartz/cloud_vx/metprd/2017060500/f06/lowCloudFrac/SATCORPSsomewhere
> > > > > > >> > and run MET:
> > > > > > >> >
> > > > > > >> >
/glade/p/ral/jntp/MET/MET_releases/8.1_python/bin/grid_stat
> > > > > > PYTHON_NUMPY
> > > > > > >> > PYTHON_NUMPY ./GridStatConfig_all -outdir . -v 2
> > > > > > >> >
> > > > > > >> > No matter which way you test, the issue occurs in
line 200
> of
> > > > > > >> > "python_stuff.
> > > > > > >> > py" (in /glade/scratch/$USER/cloud_vx/bin) :
> > > > > > >> > read_var = x.values
> > > > > > >> >
> > > > > > >> > The ".values" is a pygrib method. MET doesn't seem
to like
> > that
> > > > > > >> >
> > > > > > >> > Note that In
> > > /glade/scratch/$USER/cloud_vx/metprd/2017060500/f06/
> > > > > > >> > lowCloudFrac/SATCORPS,
> > > > > > >> > if you just do python python_script_fcst.py, you
should see
> > the
> > > > > script
> > > > > > >> runs
> > > > > > >> > without error. Thus, the issue certainly appears to
be
> > calling
> > > > the
> > > > > > >> python
> > > > > > >> > code from within MET.
> > > > > > >> >
> > > > > > >> > I hope the instructions makes sense. Please let me
know if
> > you
> > > > have
> > > > > > any
> > > > > > >> > questions. I'm happy to help you debug this issue.
> > > > > > >> >
> > > > > > >> > Thanks!
> > > > > > >> > Craig
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list