[Met_help] [rt.rap.ucar.edu #99147] History for question on python embedding in MET

John Halley Gotway via RT met_help at ucar.edu
Thu Apr 22 13:28:37 MDT 2021


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

I'm attempting to build a python script to read in a netcdf 
(non-CF-compliant) dataset and create a Pandas dataframe to be used by 
ascii2nc, using the example in the MET distribution 
(read_ascii_point.py) as the pattern for the script.

The problem is that although the pandas dataframe I create has the 
required 11 columns and matching data types, and is converted to a list 
using

point_data=point_data.values.tolist()

when I try to use the script with ascii2nc using the -python flag, it 
errors out like so:

ERROR  :
ERROR  : pyobject_as_string (PyObject *) -> bad object type
ERROR  :

When I googled on this, I ran into a similar issue with python embedding 
script for TC dropsonde data
https://github.com/dtcenter/METplus/issues/734#issuecomment-745654393

But there's no hint in the github issue as to what was done to correct 
it.  Can you help?

-- 
Dr. Todd Spindler
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., #2118
College Park, MD 20740
Todd.Spindler at noaa.gov
301-683-3757





----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: question on python embedding in MET
From: David Fillmore
Time: Fri Mar 19 12:27:55 2021

Hi Todd -
Thanks for sending info on this issue.
We are making some changes to how the python embedding works for the
10.0.0
release coming up in a few weeks,
and want to be sure this is not a persistent problem.
I want to check that each entry in your resulting list is of the
correct
type.
If you are OK to share your python script (your version of
read_ascii_point.py) please send as an attachment ...
Also, can you try to rerun ascii2nc with -v 4 to see of we get any
more
detail on the specific error?
thanks,
David

On Fri, Mar 19, 2021 at 11:25 AM Todd Spindler via RT
<met_help at ucar.edu>
wrote:

>
> Fri Mar 19 11:25:04 2021: Request 99147 was acted upon.
> Transaction: Ticket created by todd.spindler at noaa.gov
>        Queue: met_help
>      Subject: question on python embedding in MET
>        Owner: Nobody
>   Requestors: todd.spindler at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99147 >
>
>
> Hi,
>
> I'm attempting to build a python script to read in a netcdf
> (non-CF-compliant) dataset and create a Pandas dataframe to be used
by
> ascii2nc, using the example in the MET distribution
> (read_ascii_point.py) as the pattern for the script.
>
> The problem is that although the pandas dataframe I create has the
> required 11 columns and matching data types, and is converted to a
list
> using
>
> point_data=point_data.values.tolist()
>
> when I try to use the script with ascii2nc using the -python flag,
it
> errors out like so:
>
> ERROR  :
> ERROR  : pyobject_as_string (PyObject *) -> bad object type
> ERROR  :
>
> When I googled on this, I ran into a similar issue with python
embedding
> script for TC dropsonde data
> https://github.com/dtcenter/METplus/issues/734#issuecomment-
745654393
>
> But there's no hint in the github issue as to what was done to
correct
> it.  Can you help?
>
> --
> Dr. Todd Spindler
> IMSG at NOAA/NWS/NCEP/EMC
> 5830 University Research Ct., #2118
> College Park, MD 20740
> Todd.Spindler at noaa.gov
> 301-683-3757
>
>
>
>
>

------------------------------------------------
Subject: question on python embedding in MET
From: Todd Spindler
Time: Fri Mar 19 13:09:11 2021

Hi David,

I've attached the script I'm using, and here's the link to the netcdf
file it reads:
ftp://polar.ncep.noaa.gov/rtofs/for_DTC/class4_20210309_HYCOM_RTOFS_2.0_profile.nc

I tried your -v 4 trick and got this:

 > ascii2nc -v 4 -format python "read_godae_point_notab.py
class4_20210309_HYCOM_RTOFS_2.0_profile.nc" sample_godae_obs.nc
DEBUG 1: Config File:
/contrib/met/10.0.0-beta3/share/met/config/Ascii2NcConfig_default
DEBUG 3: Running user's python script (read_godae_point_notab.py).
/scratch2/NCEPDEV/marine/Todd.Spindler/save/anaconda-
personal/envs/xesmf/lib/python3.7/site-packages/dask/config.py:168:
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe. Please read
https://msg.pyyaml.org/load
for full details.
   data = yaml.load(f.read()) or {}
Data Length:    660
Data Type:      <class 'list'>
ERROR  :
ERROR  : pyobject_as_string (PyObject *) -> bad object type
ERROR  :

Here's the setup (I'm running on the NOAA Hera system):

 > module list

Currently Loaded Modules:
   1) intel/18.0.5.274   3) nco/4.9.1        5) wgrib2/2.0.8   7)
metplus/4.0.0-beta3
   2) met/10.0.0-beta3   4) wgrib/1.8.1.0b   6) R/3.5.0        8)
anaconda-xesmf/1.0.0

The anaconda-xesmf is my own anaconda.  I've attached the list of
packages it contains.

Some background information on the task I'm attempting.  The dataset
contains a set of ARGO drifter ocean temperature and salinity
profiles,
which have been matched up with profiles extracted from our Global
RTOFS
ocean model temperature and salinity 3D fields from that day.  I had
to
make a number of guesses in trying to fit the data against the 11-
column
format of the ascii2nc routine.  i set elevation to 0.0 and used the
depth parameter from the data for the height field (units of meters,
increasing downwards), but as an initial run I only selected the first
depth as a "height" value.  I don't know if the obs_type codes in the
data set match up with anything in MET or if it matters, but I just
saved it as a string value.  Also, I converted any NaN values in the
data to the string "NA" prior to converting the Pandas dataframe to a
list.

I have managed to write out the Pandas dataframe to an ASCII file and
read it back in with ascii2nc without issue, and plotted the obs
locations without problem, which I hope is an indication that the data
isn't the issue here.   I know that your read_ascii_point.py script
uses
a simple pandas.read_csv() to load the data, and I had to read in the
data using xarray.load_dataset() and then manipulate the data into the
dataframe layout.  The resulting list appears, at least superficially,
to be the same as the list created in read_ascii_point.py, but
obviously
I'm missing something.  Also, the dask warning from the ascii2nc run
is
something that xarray has been saying for all of my work for a long
time
now, and doesn't seem to cause any problems.

I hope this isn't too confusing.  Thanks for looking into this.

--
Dr. Todd Spindler
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., #2118
College Park, MD 20740
Todd.Spindler at noaa.gov
301-683-3757

On 3/19/21 2:27 PM, David Fillmore via RT wrote:
> Hi Todd -
> Thanks for sending info on this issue.
> We are making some changes to how the python embedding works for the
10.0.0
> release coming up in a few weeks,
> and want to be sure this is not a persistent problem.
> I want to check that each entry in your resulting list is of the
correct
> type.
> If you are OK to share your python script (your version of
> read_ascii_point.py) please send as an attachment ...
> Also, can you try to rerun ascii2nc with -v 4 to see of we get any
more
> detail on the specific error?
> thanks,
> David
>
> On Fri, Mar 19, 2021 at 11:25 AM Todd Spindler via RT
<met_help at ucar.edu>
> wrote:
>
>> Fri Mar 19 11:25:04 2021: Request 99147 was acted upon.
>> Transaction: Ticket created by todd.spindler at noaa.gov
>>         Queue: met_help
>>       Subject: question on python embedding in MET
>>         Owner: Nobody
>>    Requestors: todd.spindler at noaa.gov
>>        Status: new
>>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99147 >
>>
>>
>> Hi,
>>
>> I'm attempting to build a python script to read in a netcdf
>> (non-CF-compliant) dataset and create a Pandas dataframe to be used
by
>> ascii2nc, using the example in the MET distribution
>> (read_ascii_point.py) as the pattern for the script.
>>
>> The problem is that although the pandas dataframe I create has the
>> required 11 columns and matching data types, and is converted to a
list
>> using
>>
>> point_data=point_data.values.tolist()
>>
>> when I try to use the script with ascii2nc using the -python flag,
it
>> errors out like so:
>>
>> ERROR  :
>> ERROR  : pyobject_as_string (PyObject *) -> bad object type
>> ERROR  :
>>
>> When I googled on this, I ran into a similar issue with python
embedding
>> script for TC dropsonde data
>> https://github.com/dtcenter/METplus/issues/734#issuecomment-
745654393
>>
>> But there's no hint in the github issue as to what was done to
correct
>> it.  Can you help?
>>
>> --
>> Dr. Todd Spindler
>> IMSG at NOAA/NWS/NCEP/EMC
>> 5830 University Research Ct., #2118
>> College Park, MD 20740
>> Todd.Spindler at noaa.gov
>> 301-683-3757
>>
>>
>>
>>
>>


------------------------------------------------
Subject: question on python embedding in MET
From: Todd Spindler
Time: Fri Mar 19 13:09:11 2021

# packages in environment at
/scratch2/NCEPDEV/marine/Todd.Spindler/save/anaconda-
personal/envs/xesmf:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
atomicwrites              1.4.0                      py_0
attrs                     20.3.0             pyhd3eb1b0_0
backcall                  0.2.0                      py_0
blas                      1.0                         mkl
bokeh                     2.2.3                    py37_0
brotlipy                  0.7.0           py37h27cfd23_1003
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2020.12.8            h06a4308_0
cartopy                   0.17.0           py37hbb7e04d_1
certifi                   2020.12.5        py37h06a4308_0
cffi                      1.14.4           py37h261ae71_0
cfgrib                    0.9.6.2                    py_0    conda-
forge
cftime                    1.3.0            py37h6323ea4_0
chardet                   4.0.0           py37h06a4308_1003
click                     7.1.2                      py_0
cloudpickle               1.6.0                      py_0
configobj                 5.0.6                    py37_1
cryptography              3.3.1            py37h3c74f83_0
curl                      7.71.1               hbc83047_1
cycler                    0.10.0                   py37_0
cytoolz                   0.11.0           py37h7b6447c_0
dask                      1.1.4                    py37_1
dask-core                 1.1.4                    py37_1
dbus                      1.13.18              hb2f20db_0
decorator                 4.4.2                      py_0
distributed               1.28.1                   py37_0
eccodes                   2.12.3               h4fa793d_1    conda-
forge
esmf                      7.1.0             h9eb252b_1005    conda-
forge
esmpy                     7.1.0            py37h5ca1d4c_3    conda-
forge
expat                     2.2.10               he6710b0_2
fontconfig                2.13.0               h9420a91_0
freetype                  2.10.4               h5ab3b9f_0
future                    0.18.2                   py37_1
geos                      3.7.1                he6710b0_0
glib                      2.66.1               h92f7085_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb31296c_0
hdf4                      4.2.13               h3ca952b_2
hdf5                      1.10.4               hb1b8bf9_0
heapdict                  1.0.1                      py_0
icu                       58.2                 he6710b0_3
idna                      2.10                       py_0
imageio                   2.9.0                      py_0
importlib-metadata        2.0.0                      py_1
importlib_metadata        2.0.0                         1
intel-openmp              2020.2                      254
ipdb                      0.13.3             pyh9f0ad1d_0    conda-
forge
ipykernel                 5.3.4            py37h5ca1d4c_0
ipyparallel               6.2.4                    py37_0
ipython                   7.9.0            py37h39e3cac_0
ipython_genutils          0.2.0              pyhd3eb1b0_1
jasper                    1.900.1              hd497a04_4
jedi                      0.18.0           py37h06a4308_0
jinja2                    2.11.2                     py_0
joblib                    1.0.0              pyhd3eb1b0_0
jpeg                      9d                   h36c2ea0_0    conda-
forge
jupyter_client            6.1.7                      py_0
jupyter_core              4.7.0            py37h06a4308_0
kiwisolver                1.3.0            py37h2531618_0
krb5                      1.18.2               h173b8e3_0
lcms2                     2.11                 h396b838_0
ld_impl_linux-64          2.33.1               h53a641e_7
libaec                    1.0.4                he6710b0_1
libcurl                   7.71.1               h20c2e04_1
libedit                   3.1.20191231         h14c3975_1
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libnetcdf                 4.6.2             hbdf4f91_1001    conda-
forge
libpng                    1.6.37               hbc83047_0
libsodium                 1.0.18               h7b6447c_0
libssh2                   1.9.0                h1ba5d50_1
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_1
libuuid                   1.0.3                h1bed415_2
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.10               hb55368b_3
libxslt                   1.1.34               hc22bd24_0
locket                    0.2.0                    py37_1
lxml                      4.6.2            py37h9120a33_0
lz4-c                     1.9.2                heb0550a_3
markupsafe                1.1.1            py37h14c3975_1
matplotlib                3.3.2                h06a4308_0
matplotlib-base           3.3.2            py37h817c723_0
mkl                       2020.2                      256
mkl-service               2.3.0            py37he8ac12f_0
mkl_fft                   1.2.0            py37h23d657b_0
mkl_random                1.1.1            py37h0573a6f_0
more-itertools            8.6.0              pyhd3eb1b0_0
mpi                       1.0                       mpich
mpich                     3.2.1                hf39692e_8
msgpack-python            1.0.1            py37hff7bd54_0
ncurses                   6.2                  he6710b0_1
netcdf-fortran            4.4.5             hea25ff8_1000    conda-
forge
netcdf4                   1.4.2            py37h808af73_0
networkx                  2.5                        py_0
numpy                     1.19.2           py37h54aff64_0
numpy-base                1.19.2           py37hfa32c7d_0
olefile                   0.46                     py37_0
openssl                   1.1.1i               h27cfd23_0
owslib                    0.18.0                     py_0
packaging                 20.8               pyhd3eb1b0_0
pandas                    1.1.3            py37h9fdb41a_2    conda-
forge
parso                     0.7.0                      py_0
partd                     1.1.0                      py_0
pathlib2                  2.3.5            py37h06a4308_2
pcre                      8.44                 he6710b0_0
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    8.0.1            py37he98fc37_0
pip                       19.3.1                   py37_0
pluggy                    0.13.1                   py37_0
proj4                     5.2.0                he6710b0_1
prompt_toolkit            2.0.10                     py_0
psutil                    5.7.2            py37h7b6447c_0
ptyprocess                0.6.0              pyhd3eb1b0_2
py                        1.10.0             pyhd3eb1b0_0
pycparser                 2.20                       py_2
pyepsg                    0.4.0                    py37_0
pygments                  2.7.3              pyhd3eb1b0_0
pykdtree                  1.3.1            py37hdd07704_2
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7                      py_0
pyproj                    1.9.6            py37h14380d9_0
pyqt                      5.9.2            py37h05f1152_2
pyresample                1.16.0           py37h9fdb41a_1    conda-
forge
pyshp                     2.1.2                      py_0
pysocks                   1.7.1                    py37_1
pytest                    5.2.4                    py37_1
python                    3.7.9                h7579374_0
python-dateutil           2.8.1                      py_0
python_abi                3.7                     1_cp37m    conda-
forge
pytz                      2020.5             pyhd3eb1b0_0
pywavelets                1.1.1            py37h7b6447c_2
pyyaml                    5.3.1            py37h7b6447c_1
pyzmq                     20.0.0           py37h2531618_1
qt                        5.9.7                h5867ecd_1
readline                  8.0                  h7b6447c_0
requests                  2.25.1             pyhd3eb1b0_0
scikit-image              0.15.0           py37he6710b0_0
scikit-learn              0.21.3           py37hd81dba3_0
scipy                     1.5.2            py37h0b6359f_0
seawater                  3.3.4                      py_1    conda-
forge
setuptools                51.0.0           py37h06a4308_2
shapely                   1.6.4            py37h86c5351_0
sip                       4.19.8           py37hf484d3e_0
six                       1.15.0           py37h06a4308_0
sortedcontainers          2.3.0              pyhd3eb1b0_0
sqlite                    3.33.0               h62c20be_0
tabulate                  0.8.7                    py37_0
tblib                     1.7.0                      py_0
tk                        8.6.10               hbc83047_0
toolz                     0.11.1                     py_0
tornado                   6.1              py37h27cfd23_0
traitlets                 5.0.5                      py_0
typing                    3.7.4.3                  py37_0
typing_extensions         3.7.4.3                    py_0
urllib3                   1.26.2             pyhd3eb1b0_0
wcwidth                   0.2.5                      py_0
wheel                     0.36.2             pyhd3eb1b0_0
xarray                    0.16.2             pyhd3eb1b0_0
xesmf                     0.3.0                      py_0    conda-
forge
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h7b6447c_0
zeromq                    4.3.3                he6710b0_3
zict                      2.0.0                      py_0
zipp                      3.4.0              pyhd3eb1b0_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

------------------------------------------------


More information about the Met_help mailing list