[Met_help] [rt.rap.ucar.edu #100204] History for Testing h5py in Python embedding for "direct" hdf5 readin
John Halley Gotway via RT
met_help at ucar.edu
Mon Jul 12 11:36:46 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
We are currently testing the functionality reading an hdf5 file "directly"
into plot_data_plane using python embedding. Our python script utilizes
python lib h5py to read in an h5 file and extract a single field and return
said field in exactly the same way we were doing it when we were reading in
IEEE files. In short, the ending matrix we are passing to met_data is the
same np.array. When testing this out with plot_data_plane, we get the
following error pasted below (I've highlighted an hdf5 mismatch error at the
end). My question is why is there a hdf5 version conflict if the hdf5 file
handling is being done by python and not met? Any help would be
appreciated:
[tsu at lorenz h5_work]$ plot_data_plane PYTHON_NUMPY test.ps
'name="./read_NRL_hdf5.py
data/spoutData_T0119L060_slfull_quad_2020040300_000081_one.h5 airtmp
zht_0002.0";'
DEBUG 1: Opening data file: PYTHON_NUMPY
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application
continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
'HDF5_DISABLE_VERSION_CHECK' environment variable is set to 1, application
will
continue at your own risk.
Headers are 1.10.4, library is 1.8.18
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
-------------------
HDF5 Version: 1.8.18
Configured on: Fri Jan 29 12:52:51 PST 2021
Configured by: ramos at lorenz.nrlmry.navy.mil
Configure mode: production
Host system: x86_64-unknown-linux-gnu
Uname information: Linux lorenz.nrlmry.navy.mil
2.6.32-754.35.1.el6.x86_64 #1 SMP Wed Sep 16 06:48:01 EDT 2020 x86_64 x86_64
x86_64 GNU/Linux
Byte sex: little-endian
Libraries: static, shared
Installation point:
/software/depot/met-8.1a-patched/external_libs
Compiling Options:
------------------
Compilation Mode: production
C Compiler: /usr/bin/gcc ( gcc (GCC) 4.4.7 20120313 )
CFLAGS:
H5_CFLAGS: -std=c99 -pedantic -Wall -Wextra -Wundef
-Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align
-Wwrite-strings -Wconversion -Waggregate-return -Wstrict-prototypes
-Wmissing-prototypes -Wmissing-declarations -Wredundant-decls
-Wnested-externs -Winline -Wno-long-long -Wfloat-equal
-Wmissing-format-attribute -Wmissing-noreturn -Wpacked
-Wdisabled-optimization -Wformat=2 -Wunreachable-code -Wendif-labels
-Wdeclaration-after-statement -Wold-style-definition -Winvalid-pch
-Wvariadic-macros -Wnonnull -Winit-self -Wmissing-include-dirs
-Wswitch-default -Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations
-Wc++-compat -Wstrict-overflow -Wlogical-op -Wlarger-than=2048 -Wvla
-Wsync-nand -Wframe-larger-than=16384 -Wpacked-bitfield-compat -O3
AM_CFLAGS:
CPPFLAGS:
-I/software/depot/met-8.1a-patched/external_libs/include
H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L
-DNDEBUG -UH5_DEBUG_API
AM_CPPFLAGS:
-I/software/depot/met-8.1a-patched/external_libs/lib/include
Shared C Library: yes
Static C Library: yes
Statically Linked Executables: no
LDFLAGS:
-L/software/depot/met-8.1a-patched/external_libs/lib
H5_LDFLAGS:
AM_LDFLAGS:
-L/software/depot/met-8.1a-patched/external_libs/lib/lib
Extra libraries: -lrt -lz -ldl -lm
Archiver: ar
Ranlib: ranlib
Debugged Packages:
API Tracing: no
Languages:
----------
Fortran: no
C++: no
Features:
---------
Parallel HDF5: no
High Level library: yes
Threadsafety: no
Default API Mapping: v18
With Deprecated Public Symbols: yes
I/O filters (external): deflate(zlib)
MPE: no
Direct VFD: no
dmalloc: no
Clear file buffers before write: yes
Using memory checker: no
Function Stack Tracing: no
Strict File Format Checks: no
Optimization Instrumentation: no
/software/depot/Anaconda2-2019.07/lib/python2.7/site-packages/h5py/__init__.
py:75: UserWarning: h5py is running against HDF5 1.8.18 when it was built
against 1.10.4, this may cause problems
'{0}.{1}.{2}'.format(*version.hdf5_built_version_tuple)
WARNING:
WARNING: python_dataplane() -> an error occurred importing module
"/./read_NRL_hdf5.py"
WARNING:
ERROR :
ERROR : plot_data_plane -> trouble getting field "name="./read_NRL_hdf5.py
data/spoutData_T0119L060_slfull_quad_2020040300_000081_one.h5 airtmp
zht_0002.0";" from file "PYTHON_NUMPY"
ERROR :
Justin Tsu
Marine Meteorology Division
Data Assimilation Branch, Code 7531
Mesoscale Modeling Branch, Code 7533
Building 704 Room 212
Naval Research Laboratory, Monterey
7 Grace Hopper Avenue
Monterey, Ca 93943-5502
Ph. (831) 656-4111
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Testing h5py in Python embedding for "direct" hdf5 readin
From: John Opatz
Time: Fri Jun 11 15:46:30 2021
Hi Justin,
Thank you for your inquiry. I'm going to direct your email to John HG,
who
should be able to help you sort out the hdf5 version you're
experiencing.
-John O.
On Fri, Jun 11, 2021 at 3:07 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jun 11 15:06:20 2021: Request 100204 was acted upon.
> Transaction: Ticket created by justin.tsu at nrlmry.navy.mil
> Queue: met_help
> Subject: Testing h5py in Python embedding for "direct" hdf5
readin
> Owner: Nobody
> Requestors: justin.tsu at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100204 >
>
>
> Hello,
>
>
>
> We are currently testing the functionality reading an hdf5 file
"directly"
> into plot_data_plane using python embedding. Our python script
utilizes
> python lib h5py to read in an h5 file and extract a single field and
return
> said field in exactly the same way we were doing it when we were
reading in
> IEEE files. In short, the ending matrix we are passing to met_data
is the
> same np.array. When testing this out with plot_data_plane, we get
the
> following error pasted below (I've highlighted an hdf5 mismatch
error at
> the
> end). My question is why is there a hdf5 version conflict if the
hdf5 file
> handling is being done by python and not met? Any help would be
> appreciated:
>
>
>
> [tsu at lorenz h5_work]$ plot_data_plane PYTHON_NUMPY test.ps
> 'name="./read_NRL_hdf5.py
> data/spoutData_T0119L060_slfull_quad_2020040300_000081_one.h5 airtmp
> zht_0002.0";'
>
> DEBUG 1: Opening data file: PYTHON_NUMPY
>
> Warning! ***HDF5 library version mismatched error***
>
> The HDF5 header files used to compile this application do not match
>
> the version used by the HDF5 library to which this application is
linked.
>
> Data corruption or segmentation faults may occur if the application
> continues.
>
> This can happen when an application was compiled by one version of
HDF5 but
>
> linked with a different version of static or shared HDF5 library.
>
> You should recompile the application or check your shared library
related
>
> settings such as 'LD_LIBRARY_PATH'.
>
> 'HDF5_DISABLE_VERSION_CHECK' environment variable is set to 1,
application
> will
>
> continue at your own risk.
>
> Headers are 1.10.4, library is 1.8.18
>
> SUMMARY OF THE HDF5 CONFIGURATION
>
> =================================
>
>
>
> General Information:
>
> -------------------
>
> HDF5 Version: 1.8.18
>
> Configured on: Fri Jan 29 12:52:51 PST 2021
>
> Configured by: ramos at lorenz.nrlmry.navy.mil
>
> Configure mode: production
>
> Host system: x86_64-unknown-linux-gnu
>
> Uname information: Linux lorenz.nrlmry.navy.mil
> 2.6.32-754.35.1.el6.x86_64 #1 SMP Wed Sep 16 06:48:01 EDT 2020
x86_64
> x86_64
> x86_64 GNU/Linux
>
> Byte sex: little-endian
>
> Libraries: static, shared
>
> Installation point:
> /software/depot/met-8.1a-patched/external_libs
>
>
>
> Compiling Options:
>
> ------------------
>
> Compilation Mode: production
>
> C Compiler: /usr/bin/gcc ( gcc (GCC) 4.4.7
20120313 )
>
> CFLAGS:
>
> H5_CFLAGS: -std=c99 -pedantic -Wall -Wextra
-Wundef
> -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-
align
> -Wwrite-strings -Wconversion -Waggregate-return -Wstrict-prototypes
> -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls
> -Wnested-externs -Winline -Wno-long-long -Wfloat-equal
> -Wmissing-format-attribute -Wmissing-noreturn -Wpacked
> -Wdisabled-optimization -Wformat=2 -Wunreachable-code -Wendif-labels
> -Wdeclaration-after-statement -Wold-style-definition -Winvalid-pch
> -Wvariadic-macros -Wnonnull -Winit-self -Wmissing-include-dirs
> -Wswitch-default -Wswitch-enum -Wunused-macros -Wunsafe-loop-
optimizations
> -Wc++-compat -Wstrict-overflow -Wlogical-op -Wlarger-than=2048 -Wvla
> -Wsync-nand -Wframe-larger-than=16384 -Wpacked-bitfield-compat -O3
>
> AM_CFLAGS:
>
> CPPFLAGS:
> -I/software/depot/met-8.1a-patched/external_libs/include
>
> H5_CPPFLAGS: -D_GNU_SOURCE
-D_POSIX_C_SOURCE=200112L
> -DNDEBUG -UH5_DEBUG_API
>
> AM_CPPFLAGS:
> -I/software/depot/met-8.1a-patched/external_libs/lib/include
>
> Shared C Library: yes
>
> Static C Library: yes
>
> Statically Linked Executables: no
>
> LDFLAGS:
> -L/software/depot/met-8.1a-patched/external_libs/lib
>
> H5_LDFLAGS:
>
> AM_LDFLAGS:
> -L/software/depot/met-8.1a-patched/external_libs/lib/lib
>
> Extra libraries: -lrt -lz -ldl -lm
>
> Archiver: ar
>
> Ranlib: ranlib
>
> Debugged Packages:
>
> API Tracing: no
>
>
>
> Languages:
>
> ----------
>
> Fortran: no
>
>
>
> C++: no
>
>
>
> Features:
>
> ---------
>
> Parallel HDF5: no
>
> High Level library: yes
>
> Threadsafety: no
>
> Default API Mapping: v18
>
> With Deprecated Public Symbols: yes
>
> I/O filters (external): deflate(zlib)
>
> MPE: no
>
> Direct VFD: no
>
> dmalloc: no
>
> Clear file buffers before write: yes
>
> Using memory checker: no
>
> Function Stack Tracing: no
>
> Strict File Format Checks: no
>
> Optimization Instrumentation: no
>
>
> /software/depot/Anaconda2-2019.07/lib/python2.7/site-
packages/h5py/__init__.
> py:75: UserWarning: h5py is running against HDF5 1.8.18 when it was
built
> against 1.10.4, this may cause problems
>
> '{0}.{1}.{2}'.format(*version.hdf5_built_version_tuple)
>
> WARNING:
>
> WARNING: python_dataplane() -> an error occurred importing module
> "/./read_NRL_hdf5.py"
>
> WARNING:
>
> ERROR :
>
> ERROR : plot_data_plane -> trouble getting field
"name="./read_NRL_hdf5.py
> data/spoutData_T0119L060_slfull_quad_2020040300_000081_one.h5 airtmp
> zht_0002.0";" from file "PYTHON_NUMPY"
>
> ERROR :
>
>
>
> Justin Tsu
>
> Marine Meteorology Division
>
> Data Assimilation Branch, Code 7531
>
> Mesoscale Modeling Branch, Code 7533
>
> Building 704 Room 212
>
> Naval Research Laboratory, Monterey
>
> 7 Grace Hopper Avenue
>
> Monterey, Ca 93943-5502
>
>
>
> Ph. (831) 656-4111
>
>
>
>
>
--
John Opatz
Associate Scientist III
NCAR RAL and DTC
Boulder, Colorado
+1 303-497-8305
------------------------------------------------
Subject: Testing h5py in Python embedding for "direct" hdf5 readin
From: John Halley Gotway
Time: Fri Jun 11 16:18:01 2021
Justin,
Yes, there's a bit of version-itis going on here. The NetCDF 4 library
is
built upon HDF5 and so is the h5py python library. Those are the two
versions of that library that are in conflict. We did run into this
problem
in the past, and that was the motivation for adding support for the
MET_PYTHON_EXE environment variable in MET version 9.0:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0#notes
You can read a description of how that works here:
https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#met-
python-exe
The purpose is to handle environment conflicts such as this. So
instead of
MET running the script with it's version of python, it calls the
user's
specified version to Python to run their script and write that to a
temp
file. And in MET version 10.0.0, we switched from writing temp files
in the
Python pickle format to using NetCDF or ascii files. We ran into more
version-itis with the pickle version.
However, I see that you're running MET version 8.1. Unfortunately, I
don't
have a good solution to recommend for MET version 8.1.
If possible, I'd recommend trying this using MET version 10.0.0 and
setting
the MET_PYTHON_EXE environment variable to point to the instance of
python3
you'd like to run.
Thanks,
John HG
On Fri, Jun 11, 2021 at 3:47 PM John Opatz via RT <met_help at ucar.edu>
wrote:
>
> Fri Jun 11 15:46:52 2021: Request 100204 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by jopatz
> Queue: met_help
> Subject: Testing h5py in Python embedding for "direct" hdf5
readin
> Owner: johnhg
> Requestors: justin.tsu at nrlmry.navy.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100204 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Testing h5py in Python embedding for "direct" hdf5 readin
From: Tsu, Mr. Justin
Time: Fri Jun 11 16:49:41 2021
Thanks John,
I will work with my IT to get MET 10.0.0 on our systems and retest.
Justin
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, June 11, 2021 3:18 PM
To: Tsu, Mr. Justin <justin.tsu at nrlmry.navy.mil>
Cc: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #100204] Testing h5py in Python
embedding for
"direct" hdf5 readin
Justin,
Yes, there's a bit of version-itis going on here. The NetCDF 4 library
is
built upon HDF5 and so is the h5py python library. Those are the two
versions
of that library that are in conflict. We did run into this problem in
the
past, and that was the motivation for adding support for the
MET_PYTHON_EXE
environment variable in MET version 9.0:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0#notes
You can read a description of how that works here:
https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#met-
python-exe
The purpose is to handle environment conflicts such as this. So
instead of MET
running the script with it's version of python, it calls the user's
specified
version to Python to run their script and write that to a temp file.
And in
MET version 10.0.0, we switched from writing temp files in the Python
pickle
format to using NetCDF or ascii files. We ran into more version-itis
with the
pickle version.
However, I see that you're running MET version 8.1. Unfortunately, I
don't
have a good solution to recommend for MET version 8.1.
If possible, I'd recommend trying this using MET version 10.0.0 and
setting
the MET_PYTHON_EXE environment variable to point to the instance of
python3
you'd like to run.
Thanks,
John HG
On Fri, Jun 11, 2021 at 3:47 PM John Opatz via RT <met_help at ucar.edu>
wrote:
>
> Fri Jun 11 15:46:52 2021: Request 100204 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by jopatz
> Queue: met_help
> Subject: Testing h5py in Python embedding for "direct" hdf5
readin
> Owner: johnhg
> Requestors: justin.tsu at nrlmry.navy.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100204
> >
>
>
> This transaction appears to have no content
>
------------------------------------------------
More information about the Met_help
mailing list