[Met_help] [rt.rap.ucar.edu #99830] History for MET module load issues in python environment
John Halley Gotway via RT
met_help at ucar.edu
Mon Jul 12 11:30:11 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good afternoon,
I have a python script that reads in observed precipitation to pass to
MET/METplus (and includes the attribute dictionary). Running the python
script by itself works with no errors, so I decided to include this script
to run the "plot_data_plane" tool:
/opt/MET/90/bin/plot_data_plane PYTHON_NUMPY obstestout.ps
'name="/export/cpc-lw7-jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
/export/cpc-lw7-jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
precip_07d_20210420.nc precip";'
'Precip' -- in this case -- is the variable I want to plot and is included
as a 'sys' argument for the python script. When I run this, the python
script runs (the output prints the contents of a few variables). It then
prints the following error:
Traceback (most recent call last):
File "/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py", line 10,
in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
This is interesting, as numpy is needed to run the python script -- which
looks like it runs successfully. But passing this script from python to
this MET tool fails because it cannot find this numpy library. I should
mention that initially, I was receiving errors regarding the netCDF4
library, but when I set the MET_PYTHON_EXE environment variable to the
conda install of python, this error went away.
Perhaps I need to re-configure MET? I'm using a newly mounted version of
MET/METplus originally setup by the WPC, so I'm not sure if this is needed
or not. It seems like it may just be an issue with my conda environment and
the correct specification of environment variables. I'm running MET version
9.0.1 and METplus version 3.0.1 on the cpc-lw7 machine.
Thanks!
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET module load issues in python environment
From: John Halley Gotway
Time: Mon May 10 10:27:16 2021
Hi Justin,
I see that you're having trouble with the python embedding in MET.
Here's the short answer... please try setting an environment variable
named
MET_PYTHON_EXE. Assuming python is in your path, it could be as simple
as:
export MET_PYTHON_EXE=`which python3`
On some systems, "python" points to python version 2 while "python3"
points
to python version 3. We want python3 in MET version 9.0+ so that's why
I
included it in the command above. Please try rerunning plot_data_plane
with
the environment variable set.
For a description of how that setting works, please see this link:
https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#met-
python-exe
Thanks,
John Halley Gotway
On Mon, May 10, 2021 at 10:07 AM Justin Hicks - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Mon May 10 10:06:48 2021: Request 99830 was acted upon.
> Transaction: Ticket created by justin.hicks at noaa.gov
> Queue: met_help
> Subject: MET module load issues in python environment
> Owner: Nobody
> Requestors: justin.hicks at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99830 >
>
>
> Good afternoon,
>
> I have a python script that reads in observed precipitation to pass
to
> MET/METplus (and includes the attribute dictionary). Running the
python
> script by itself works with no errors, so I decided to include this
script
> to run the "plot_data_plane" tool:
>
> /opt/MET/90/bin/plot_data_plane PYTHON_NUMPY obstestout.ps
>
> 'name="/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
>
> /export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
> precip_07d_20210420.nc precip";'
>
> 'Precip' -- in this case -- is the variable I want to plot and is
included
> as a 'sys' argument for the python script. When I run this, the
python
> script runs (the output prints the contents of a few variables). It
then
> prints the following error:
>
> Traceback (most recent call last):
> File "/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py",
line 10,
> in <module>
> import numpy as np
> ModuleNotFoundError: No module named 'numpy'
>
> This is interesting, as numpy is needed to run the python script --
which
> looks like it runs successfully. But passing this script from python
to
> this MET tool fails because it cannot find this numpy library. I
should
> mention that initially, I was receiving errors regarding the netCDF4
> library, but when I set the MET_PYTHON_EXE environment variable to
the
> conda install of python, this error went away.
>
> Perhaps I need to re-configure MET? I'm using a newly mounted
version of
> MET/METplus originally setup by the WPC, so I'm not sure if this is
needed
> or not. It seems like it may just be an issue with my conda
environment and
> the correct specification of environment variables. I'm running MET
version
> 9.0.1 and METplus version 3.0.1 on the cpc-lw7 machine.
>
> Thanks!
>
>
------------------------------------------------
Subject: MET module load issues in python environment
From: Justin Hicks - NOAA Affiliate
Time: Mon May 10 11:01:37 2021
Hi John,
I tried this previously and it helps with recognizing the netCDF4
library.
I still receive the following error upon running the plot_data_plane
tool:
File "/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py", line
10, in
<module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
WARNING:
WARNING: python_dataplane() -> an error occurred importing module
"read_pickle_dataplane"
WARNING:
ERROR :
ERROR : plot_data_plane -> trouble getting field
"name="/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
precip_07d_20210420.nc precip";" from file "PYTHON_NUMPY"
ERROR :
The python script I am running imports the following libraries in
order:
netCDF4, numpy, datetime, os, sys, and math. Setting this environment
variable takes care of netCDF4, but fails to import numpy. 'Which
python3'
points to the conda install of python3, which is able to import numpy
without issue when I run python3 at the command line.
On Mon, May 10, 2021 at 12:27 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Justin,
>
> I see that you're having trouble with the python embedding in MET.
>
> Here's the short answer... please try setting an environment
variable named
> MET_PYTHON_EXE. Assuming python is in your path, it could be as
simple as:
> export MET_PYTHON_EXE=`which python3`
>
> On some systems, "python" points to python version 2 while "python3"
points
> to python version 3. We want python3 in MET version 9.0+ so that's
why I
> included it in the command above. Please try rerunning
plot_data_plane with
> the environment variable set.
>
> For a description of how that setting works, please see this link:
>
> https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#met-
python-exe
>
> Thanks,
> John Halley Gotway
>
> On Mon, May 10, 2021 at 10:07 AM Justin Hicks - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 10 10:06:48 2021: Request 99830 was acted upon.
> > Transaction: Ticket created by justin.hicks at noaa.gov
> > Queue: met_help
> > Subject: MET module load issues in python environment
> > Owner: Nobody
> > Requestors: justin.hicks at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99830 >
> >
> >
> > Good afternoon,
> >
> > I have a python script that reads in observed precipitation to
pass to
> > MET/METplus (and includes the attribute dictionary). Running the
python
> > script by itself works with no errors, so I decided to include
this
> script
> > to run the "plot_data_plane" tool:
> >
> > /opt/MET/90/bin/plot_data_plane PYTHON_NUMPY obstestout.ps
> >
> >
> 'name="/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
> >
> >
> /export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
> > precip_07d_20210420.nc precip";'
> >
> > 'Precip' -- in this case -- is the variable I want to plot and is
> included
> > as a 'sys' argument for the python script. When I run this, the
python
> > script runs (the output prints the contents of a few variables).
It then
> > prints the following error:
> >
> > Traceback (most recent call last):
> > File "/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py",
line
> 10,
> > in <module>
> > import numpy as np
> > ModuleNotFoundError: No module named 'numpy'
> >
> > This is interesting, as numpy is needed to run the python script
-- which
> > looks like it runs successfully. But passing this script from
python to
> > this MET tool fails because it cannot find this numpy library. I
should
> > mention that initially, I was receiving errors regarding the
netCDF4
> > library, but when I set the MET_PYTHON_EXE environment variable to
the
> > conda install of python, this error went away.
> >
> > Perhaps I need to re-configure MET? I'm using a newly mounted
version of
> > MET/METplus originally setup by the WPC, so I'm not sure if this
is
> needed
> > or not. It seems like it may just be an issue with my conda
environment
> and
> > the correct specification of environment variables. I'm running
MET
> version
> > 9.0.1 and METplus version 3.0.1 on the cpc-lw7 machine.
> >
> > Thanks!
> >
> >
>
>
------------------------------------------------
Subject: MET module load issues in python environment
From: John Halley Gotway
Time: Mon May 10 11:43:08 2021
Justin,
Ah OK, thanks for clarifying the issue. As described in Appendix F of
the
User's Guide, python-embedding is disabled by default. It is turned on
at
compilation time by configuring with the "--enable-python" option. And
the
MET_PYTHON_CC and MET_PYTHON_LD point to the python libraries that
should
be compiled into MET.
However, in MET version 9.1, that instance of python requires that the
NumPy and pickle library be available.
My theory is that MET was compiled against a version of python for
which
NumPy is NOT available. If that's the case, we'll need to update the
compilation environment to point to a different version of python and
recompile.
We could test this theory, by figuring out which version of python was
used
for compilation. I see you're running /opt/MET/90/bin/plot_data_plane.
When
MET is configured, it creates a log file named "config.log" with some
good
info in it. Please try running:
find /opt/MET/90 -name "config.log"
If you're able to find it, we want to see how MET_PYTHON_CC and
MET_PYTHON_LD were set there:
grep MET_PYTHON_ config.log
For example, on my laptop, I have:
MET_PYTHON_CC='-
I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8'
MET_PYTHON_LD='-
L/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/config-
3.8-darwin
-lpython3.8 -ldl -framework CoreFoundation'
And the corresponding python executable is in the bin directory:
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3
Running that on the command line, importing numpy works fine. If the
"import numpy as np" command fails with that version of python, then
that's
where the problem lies.
You'll either need to recompile MET pointing to a different version of
Python, or perhaps easier, request that whomever has the right
permissions
install numpy and pickle for that python instance.
Thanks,
John
On Mon, May 10, 2021 at 11:02 AM Justin Hicks - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99830 >
>
> Hi John,
>
> I tried this previously and it helps with recognizing the netCDF4
library.
> I still receive the following error upon running the plot_data_plane
tool:
>
> File "/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py", line
10, in
> <module>
> import numpy as np
> ModuleNotFoundError: No module named 'numpy'
> WARNING:
> WARNING: python_dataplane() -> an error occurred importing module
> "read_pickle_dataplane"
> WARNING:
> ERROR :
> ERROR : plot_data_plane -> trouble getting field
>
> "name="/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
>
> /export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
> precip_07d_20210420.nc precip";" from file "PYTHON_NUMPY"
> ERROR :
>
> The python script I am running imports the following libraries in
order:
> netCDF4, numpy, datetime, os, sys, and math. Setting this
environment
> variable takes care of netCDF4, but fails to import numpy. 'Which
python3'
> points to the conda install of python3, which is able to import
numpy
> without issue when I run python3 at the command line.
>
>
>
>
> On Mon, May 10, 2021 at 12:27 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hi Justin,
> >
> > I see that you're having trouble with the python embedding in MET.
> >
> > Here's the short answer... please try setting an environment
variable
> named
> > MET_PYTHON_EXE. Assuming python is in your path, it could be as
simple
> as:
> > export MET_PYTHON_EXE=`which python3`
> >
> > On some systems, "python" points to python version 2 while
"python3"
> points
> > to python version 3. We want python3 in MET version 9.0+ so that's
why I
> > included it in the command above. Please try rerunning
plot_data_plane
> with
> > the environment variable set.
> >
> > For a description of how that setting works, please see this link:
> >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#met-
python-exe
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Mon, May 10, 2021 at 10:07 AM Justin Hicks - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon May 10 10:06:48 2021: Request 99830 was acted upon.
> > > Transaction: Ticket created by justin.hicks at noaa.gov
> > > Queue: met_help
> > > Subject: MET module load issues in python environment
> > > Owner: Nobody
> > > Requestors: justin.hicks at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99830
> >
> > >
> > >
> > > Good afternoon,
> > >
> > > I have a python script that reads in observed precipitation to
pass to
> > > MET/METplus (and includes the attribute dictionary). Running the
python
> > > script by itself works with no errors, so I decided to include
this
> > script
> > > to run the "plot_data_plane" tool:
> > >
> > > /opt/MET/90/bin/plot_data_plane PYTHON_NUMPY obstestout.ps
> > >
> > >
> >
> 'name="/export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/precip_07d_TRMM.py
> > >
> > >
> >
> /export/cpc-lw7-
jhicks/jhicks/CPC_metplus/Int_Desk_Forecasts/Africa/Week1/Observations/
> > > precip_07d_20210420.nc precip";'
> > >
> > > 'Precip' -- in this case -- is the variable I want to plot and
is
> > included
> > > as a 'sys' argument for the python script. When I run this, the
python
> > > script runs (the output prints the contents of a few variables).
It
> then
> > > prints the following error:
> > >
> > > Traceback (most recent call last):
> > > File
"/opt/MET/90/share/met/wrappers/read_pickle_dataplane.py", line
> > 10,
> > > in <module>
> > > import numpy as np
> > > ModuleNotFoundError: No module named 'numpy'
> > >
> > > This is interesting, as numpy is needed to run the python script
--
> which
> > > looks like it runs successfully. But passing this script from
python to
> > > this MET tool fails because it cannot find this numpy library. I
should
> > > mention that initially, I was receiving errors regarding the
netCDF4
> > > library, but when I set the MET_PYTHON_EXE environment variable
to the
> > > conda install of python, this error went away.
> > >
> > > Perhaps I need to re-configure MET? I'm using a newly mounted
version
> of
> > > MET/METplus originally setup by the WPC, so I'm not sure if this
is
> > needed
> > > or not. It seems like it may just be an issue with my conda
environment
> > and
> > > the correct specification of environment variables. I'm running
MET
> > version
> > > 9.0.1 and METplus version 3.0.1 on the cpc-lw7 machine.
> > >
> > > Thanks!
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list