[Met_help] [rt.rap.ucar.edu #97175] History for Need help compiling MET-9.1 with python
Julie Prestopnik via RT
met_help at ucar.edu
Mon Oct 26 13:48:41 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET help,
I am trying to build MET v9.1.20200810 with the '--enable-python' option
in hope of installing METplus next. However, it fails to build halfway
at the ensemble_stat section. Specifically, the linker fails with many
undefined references to python functions (please see attached make
output). I confirmed met will build just fine if I remove the
'--enable-python' configure option.
For background, I am using python-3.8.5 installed in a "p38" environment
with Anaconda-4.9.0 on ubuntu-18.04.5. This environment contains basic
packages (scipy,matplotlib,cartopy,etc). As per the user guide, I have
exported the following environment variables that likely have an impact
on my compile error:
echo $MET_PYTHON_CC
-I/home/scott/anaconda3/envs/p38/include/python3.8
-I/home/scott/anaconda3/envs/p38/include/python3.8 -Wno-unused-result
-Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize -fPIC
-fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe -isystem
/home/scott/anaconda3/envs/p38/include
-fdebug-prefix-map=/tmp/build/80754af9/python_1599203911753/work=/usr/local/src/conda/python-3.8.5
-fdebug-prefix-map=/home/scott/anaconda3/envs/p38=/usr/local/src/conda-prefix
-fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto
-DNDEBUG -fwrapv -O3 -Wall
echo $MET_PYTHON_LD
-L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-linux-gnu
-L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil -lrt
-lm -lm
echo $LIBS
-lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
-lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lcrypto -lssl
If I randomly search a few these undefined reference function names, I
find them in various *.h file in the
/home/scott/anaconda3/envs/p38/include/python3.8 directory. Similarly, I
can find many of these in the
/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a
library file. I believe both of these paths are provided to the linker
so I am a bit confused what is going on here.
Do you have any other ideas I can try to get this working? Do you have
any feedback for Anaconda users or recommendations of versions of python
you have found compatible? Honestly, I'm unsure where the problem lies.
Any help would be appreciated.
Thanks,
Scott
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Wed Oct 21 15:33:28 2020
Hi Scott.
I see that you are having difficulty building METv9.1 with python
embedding
enabled.
Thanks for all of this helpful information. Could you please try
setting
MET_PYTHON_CC to the following?
-I/home/scott/anaconda3/envs/p38/include/python3.8
Also, it looks like MET_PYTHON_LD is missing the -lpython3.8m, so
please
try adding that before "-lcrypt".
Then, please run configure again and then try running make again.
Please
let me know how it goes. If it is unsuccessful, please send the new
make.log file along with the config.log file as well. Thanks and good
luck!
Julie
On Wed, Oct 21, 2020 at 3:15 PM Scott Rabenhorst via RT
<met_help at ucar.edu>
wrote:
>
> Wed Oct 21 15:15:25 2020: Request 97175 was acted upon.
> Transaction: Ticket created by scott.rabenhorst at gmail.com
> Queue: met_help
> Subject: Need help compiling MET-9.1 with python
> Owner: Nobody
> Requestors: scott.rabenhorst at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>
>
> Hi MET help,
>
> I am trying to build MET v9.1.20200810 with the '--enable-python'
option
> in hope of installing METplus next. However, it fails to build
halfway
> at the ensemble_stat section. Specifically, the linker fails with
many
> undefined references to python functions (please see attached make
> output). I confirmed met will build just fine if I remove the
> '--enable-python' configure option.
>
> For background, I am using python-3.8.5 installed in a "p38"
environment
> with Anaconda-4.9.0 on ubuntu-18.04.5. This environment contains
basic
> packages (scipy,matplotlib,cartopy,etc). As per the user guide, I
have
> exported the following environment variables that likely have an
impact
> on my compile error:
>
> echo $MET_PYTHON_CC
> -I/home/scott/anaconda3/envs/p38/include/python3.8
> -I/home/scott/anaconda3/envs/p38/include/python3.8 -Wno-unused-
result
> -Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize -fPIC
> -fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe
-isystem
> /home/scott/anaconda3/envs/p38/include
> -fdebug-prefix-
map=/tmp/build/80754af9/python_1599203911753/work=/usr/local/src/conda/python-
3.8.5
>
> -fdebug-prefix-
map=/home/scott/anaconda3/envs/p38=/usr/local/src/conda-prefix
>
> -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto
> -DNDEBUG -fwrapv -O3 -Wall
>
> echo $MET_PYTHON_LD
> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
> -lm -lm
>
> echo $LIBS
> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lcrypto -lssl
>
> If I randomly search a few these undefined reference function names,
I
> find them in various *.h file in the
> /home/scott/anaconda3/envs/p38/include/python3.8 directory.
Similarly, I
> can find many of these in the
> /home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a
>
> library file. I believe both of these paths are provided to the
linker
> so I am a bit confused what is going on here.
>
> Do you have any other ideas I can try to get this working? Do you
have
> any feedback for Anaconda users or recommendations of versions of
python
> you have found compatible? Honestly, I'm unsure where the problem
lies.
> Any help would be appreciated.
>
> Thanks,
> Scott
>
>
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Scott Rabenhorst
Time: Thu Oct 22 08:01:46 2020
Hi Julie,
Thank you so much for your fast reply. That was a good suggestion. It
seemed to get me further along adding -lpython3.8 to LIBS and the path
to LD_LIBRARY_PATH.
Now I am getting a build error something like this:
...
Making install in ensemble_stat
make[4]: Entering directory
'/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
/usr/bin/g++-7 -g -L../../../../src/basic/vx_cal
-L../../../../src/basic/vx_config -L../../../../src/basic/vx_log
-L../../../../src/basic/vx_math -L../../../../src/basic/vx_util
-L../../../../src/libcode/vx_afm
-L../../../../src/libcode/vx_analysis_util
-L../../../../src/libcode/vx_color -L../../../../src/libcode/vx_data2d
-L../../../../src/libcode/vx_data2d_factory
-L../../../../src/libcode/vx_data2d_grib
-L../../../../src/libcode/vx_data2d_grib2
-L../../../../src/libcode/vx_data2d_nccf
-L../../../../src/libcode/vx_data2d_nc_met
-L../../../../src/libcode/vx_data2d_nc_pinterp
-L../../../../src/libcode/vx_data2d_python
-L../../../../src/libcode/vx_python3_utils
-L../../../../src/libcode/vx_geodesy -L../../../../src/libcode/vx_gis
-L../../../../src/libcode/vx_gnomon -L../../../../src/libcode/vx_grid
-L../../../../src/libcode/vx_gsl_prob -L../../../../src/libcode/vx_nav
-L../../../../src/libcode/vx_nc_obs
-L../../../../src/libcode/vx_nc_util
-L../../../../src/libcode/vx_pb_util
-L../../../../src/libcode/vx_plot_util -L../../../../src/libcode/vx_ps
-L../../../../src/libcode/vx_pxm -L../../../../src/libcode/vx_render
-L../../../../src/libcode/vx_regrid
-L../../../../src/libcode/vx_shapedata
-L../../../../src/libcode/vx_solar
-L../../../../src/libcode/vx_statistics
-L../../../../src/libcode/vx_stat_out
-L../../../../src/libcode/vx_summary
-L../../../../src/libcode/vx_time_series
-L../../../../src/libcode/vx_series_data
-L../../../../src/libcode/vx_tc_util -L/opt/app/libs_gnu-9.3.0_v02/lib
-L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
-L/opt/app/libs_gnu-9.3.0_v02/lib
-L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
-L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
-lm -lm -L/opt/app/libs_gnu-9.3.0_v02/lib
-L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
-L/opt/app/libs_gnu-9.3.0_v02/lib
-L/opt/app/libs_gnu-9.3.0_v02/lib/cairo
-L/opt/app/libs_gnu-9.3.0_v02/lib -o ensemble_stat
ensemble_stat-ensemble_stat.o ensemble_stat-ensemble_stat_conf_info.o
-lvx_stat_out -lvx_statistics -lvx_shapedata -lvx_gsl_prob
-lvx_analysis_util -lvx_data2d_factory -lvx_data2d_nc_met
-lvx_data2d_grib -lvx_data2d_grib2 -lgrib2c -ljasper -lpng -lz
-lvx_data2d_nc_pinterp -lvx_data2d_python -lvx_python3_utils
-L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
-L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
-lm -lm -lvx_data2d_nccf -lvx_data2d -lvx_nc_obs -lvx_nc_util
-lvx_regrid -lvx_grid -lvx_config -lvx_cal -lvx_util -lvx_math
-lvx_color -lvx_log -lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas
-lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
-lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lpython3.8 -lcrypto
-lssl
lto1: fatal error: bytecode stream in file
‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-linux-
gnu/libpython3.8.a’
generated with LTO version 6.0 instead of the expected 6.2
compilation terminated.
lto-wrapper: fatal error: /usr/bin/g++-7 returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
Makefile:426: recipe for target 'ensemble_stat' failed
make[4]: *** [ensemble_stat] Error 1
make[4]: Leaving directory
'/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
Makefile:379: recipe for target 'install-recursive' failed
...
This error looks like an issue caused because Anaconda uses its own
internal compiler version and not any of the ones on my system. Have
any
of your users encountered this issue before? Do you happen to know if
there are any easy solutions? I tried passing:
export CFLAGS="-fno-lto ${CFLAGS}"
export LDFLAGS="$(CFLAGS) ${LDFLAGS}"
but that didn't seem to make any difference. Please let me know if you
have any thoughts.
Thanks,
Scott
On 10/21/20 5:33 PM, Julie Prestopnik via RT wrote:
> Hi Scott.
>
> I see that you are having difficulty building METv9.1 with python
embedding
> enabled.
>
> Thanks for all of this helpful information. Could you please try
setting
> MET_PYTHON_CC to the following?
>
> -I/home/scott/anaconda3/envs/p38/include/python3.8
>
> Also, it looks like MET_PYTHON_LD is missing the -lpython3.8m, so
please
> try adding that before "-lcrypt".
>
> Then, please run configure again and then try running make again.
Please
> let me know how it goes. If it is unsuccessful, please send the new
> make.log file along with the config.log file as well. Thanks and
good luck!
>
> Julie
>
> On Wed, Oct 21, 2020 at 3:15 PM Scott Rabenhorst via RT
<met_help at ucar.edu>
> wrote:
>
>> Wed Oct 21 15:15:25 2020: Request 97175 was acted upon.
>> Transaction: Ticket created by scott.rabenhorst at gmail.com
>> Queue: met_help
>> Subject: Need help compiling MET-9.1 with python
>> Owner: Nobody
>> Requestors: scott.rabenhorst at gmail.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>>
>>
>> Hi MET help,
>>
>> I am trying to build MET v9.1.20200810 with the '--enable-python'
option
>> in hope of installing METplus next. However, it fails to build
halfway
>> at the ensemble_stat section. Specifically, the linker fails with
many
>> undefined references to python functions (please see attached make
>> output). I confirmed met will build just fine if I remove the
>> '--enable-python' configure option.
>>
>> For background, I am using python-3.8.5 installed in a "p38"
environment
>> with Anaconda-4.9.0 on ubuntu-18.04.5. This environment contains
basic
>> packages (scipy,matplotlib,cartopy,etc). As per the user guide, I
have
>> exported the following environment variables that likely have an
impact
>> on my compile error:
>>
>> echo $MET_PYTHON_CC
>> -I/home/scott/anaconda3/envs/p38/include/python3.8
>> -I/home/scott/anaconda3/envs/p38/include/python3.8 -Wno-unused-
result
>> -Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize -fPIC
>> -fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe
-isystem
>> /home/scott/anaconda3/envs/p38/include
>> -fdebug-prefix-
map=/tmp/build/80754af9/python_1599203911753/work=/usr/local/src/conda/python-
3.8.5
>>
>> -fdebug-prefix-
map=/home/scott/anaconda3/envs/p38=/usr/local/src/conda-prefix
>>
>> -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto
>> -DNDEBUG -fwrapv -O3 -Wall
>>
>> echo $MET_PYTHON_LD
>> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
>> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
>> -lm -lm
>>
>> echo $LIBS
>> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
>> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lcrypto -lssl
>>
>> If I randomly search a few these undefined reference function
names, I
>> find them in various *.h file in the
>> /home/scott/anaconda3/envs/p38/include/python3.8 directory.
Similarly, I
>> can find many of these in the
>> /home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a
>>
>> library file. I believe both of these paths are provided to the
linker
>> so I am a bit confused what is going on here.
>>
>> Do you have any other ideas I can try to get this working? Do you
have
>> any feedback for Anaconda users or recommendations of versions of
python
>> you have found compatible? Honestly, I'm unsure where the problem
lies.
>> Any help would be appreciated.
>>
>> Thanks,
>> Scott
>>
>>
>>
>>
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Thu Oct 22 08:42:15 2020
Hi Scott.
I'm glad that you were able to get further.
Could you please send me the output of the following two commands?
- ls
/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython*
- ls /home/scott/anaconda3/envs/p38/lib/libpython*
Could you please also send me your config.log file and your make.log
file?
Those will give me more information.
Thanks!
Julie
On Thu, Oct 22, 2020 at 8:01 AM Scott Rabenhorst via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>
> Hi Julie,
>
> Thank you so much for your fast reply. That was a good suggestion.
It
> seemed to get me further along adding -lpython3.8 to LIBS and the
path
> to LD_LIBRARY_PATH.
>
> Now I am getting a build error something like this:
>
> ...
> Making install in ensemble_stat
> make[4]: Entering directory
> '/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
> /usr/bin/g++-7 -g -L../../../../src/basic/vx_cal
> -L../../../../src/basic/vx_config -L../../../../src/basic/vx_log
> -L../../../../src/basic/vx_math -L../../../../src/basic/vx_util
> -L../../../../src/libcode/vx_afm
> -L../../../../src/libcode/vx_analysis_util
> -L../../../../src/libcode/vx_color
-L../../../../src/libcode/vx_data2d
> -L../../../../src/libcode/vx_data2d_factory
> -L../../../../src/libcode/vx_data2d_grib
> -L../../../../src/libcode/vx_data2d_grib2
> -L../../../../src/libcode/vx_data2d_nccf
> -L../../../../src/libcode/vx_data2d_nc_met
> -L../../../../src/libcode/vx_data2d_nc_pinterp
> -L../../../../src/libcode/vx_data2d_python
> -L../../../../src/libcode/vx_python3_utils
> -L../../../../src/libcode/vx_geodesy
-L../../../../src/libcode/vx_gis
> -L../../../../src/libcode/vx_gnomon
-L../../../../src/libcode/vx_grid
> -L../../../../src/libcode/vx_gsl_prob
-L../../../../src/libcode/vx_nav
> -L../../../../src/libcode/vx_nc_obs
-L../../../../src/libcode/vx_nc_util
> -L../../../../src/libcode/vx_pb_util
> -L../../../../src/libcode/vx_plot_util
-L../../../../src/libcode/vx_ps
> -L../../../../src/libcode/vx_pxm -L../../../../src/libcode/vx_render
> -L../../../../src/libcode/vx_regrid
> -L../../../../src/libcode/vx_shapedata
> -L../../../../src/libcode/vx_solar
> -L../../../../src/libcode/vx_statistics
> -L../../../../src/libcode/vx_stat_out
> -L../../../../src/libcode/vx_summary
> -L../../../../src/libcode/vx_time_series
> -L../../../../src/libcode/vx_series_data
> -L../../../../src/libcode/vx_tc_util -L/opt/app/libs_gnu-
9.3.0_v02/lib
> -L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
> -L/opt/app/libs_gnu-9.3.0_v02/lib
> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
> -lm -lm -L/opt/app/libs_gnu-9.3.0_v02/lib
> -L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
> -L/opt/app/libs_gnu-9.3.0_v02/lib
> -L/opt/app/libs_gnu-9.3.0_v02/lib/cairo
> -L/opt/app/libs_gnu-9.3.0_v02/lib -o ensemble_stat
> ensemble_stat-ensemble_stat.o ensemble_stat-
ensemble_stat_conf_info.o
> -lvx_stat_out -lvx_statistics -lvx_shapedata -lvx_gsl_prob
> -lvx_analysis_util -lvx_data2d_factory -lvx_data2d_nc_met
> -lvx_data2d_grib -lvx_data2d_grib2 -lgrib2c -ljasper -lpng -lz
> -lvx_data2d_nc_pinterp -lvx_data2d_python -lvx_python3_utils
> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
> -lm -lm -lvx_data2d_nccf -lvx_data2d -lvx_nc_obs -lvx_nc_util
> -lvx_regrid -lvx_grid -lvx_config -lvx_cal -lvx_util -lvx_math
> -lvx_color -lvx_log -lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas
> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lpython3.8
-lcrypto -lssl
> lto1: fatal error: bytecode stream in file
> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>
> generated with LTO version 6.0 instead of the expected 6.2
> compilation terminated.
> lto-wrapper: fatal error: /usr/bin/g++-7 returned 1 exit status
> compilation terminated.
> /usr/bin/ld: error: lto-wrapper failed
> collect2: error: ld returned 1 exit status
> Makefile:426: recipe for target 'ensemble_stat' failed
> make[4]: *** [ensemble_stat] Error 1
> make[4]: Leaving directory
> '/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
> Makefile:379: recipe for target 'install-recursive' failed
> ...
>
> This error looks like an issue caused because Anaconda uses its own
> internal compiler version and not any of the ones on my system. Have
any
> of your users encountered this issue before? Do you happen to know
if
> there are any easy solutions? I tried passing:
>
> export CFLAGS="-fno-lto ${CFLAGS}"
> export LDFLAGS="$(CFLAGS) ${LDFLAGS}"
>
> but that didn't seem to make any difference. Please let me know if
you
> have any thoughts.
>
> Thanks,
> Scott
>
>
> On 10/21/20 5:33 PM, Julie Prestopnik via RT wrote:
> > Hi Scott.
> >
> > I see that you are having difficulty building METv9.1 with python
> embedding
> > enabled.
> >
> > Thanks for all of this helpful information. Could you please try
setting
> > MET_PYTHON_CC to the following?
> >
> > -I/home/scott/anaconda3/envs/p38/include/python3.8
> >
> > Also, it looks like MET_PYTHON_LD is missing the -lpython3.8m, so
please
> > try adding that before "-lcrypt".
> >
> > Then, please run configure again and then try running make again.
Please
> > let me know how it goes. If it is unsuccessful, please send the
new
> > make.log file along with the config.log file as well. Thanks and
good
> luck!
> >
> > Julie
> >
> > On Wed, Oct 21, 2020 at 3:15 PM Scott Rabenhorst via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >> Wed Oct 21 15:15:25 2020: Request 97175 was acted upon.
> >> Transaction: Ticket created by scott.rabenhorst at gmail.com
> >> Queue: met_help
> >> Subject: Need help compiling MET-9.1 with python
> >> Owner: Nobody
> >> Requestors: scott.rabenhorst at gmail.com
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175
> >
> >>
> >>
> >> Hi MET help,
> >>
> >> I am trying to build MET v9.1.20200810 with the '--enable-python'
option
> >> in hope of installing METplus next. However, it fails to build
halfway
> >> at the ensemble_stat section. Specifically, the linker fails with
many
> >> undefined references to python functions (please see attached
make
> >> output). I confirmed met will build just fine if I remove the
> >> '--enable-python' configure option.
> >>
> >> For background, I am using python-3.8.5 installed in a "p38"
environment
> >> with Anaconda-4.9.0 on ubuntu-18.04.5. This environment contains
basic
> >> packages (scipy,matplotlib,cartopy,etc). As per the user guide, I
have
> >> exported the following environment variables that likely have an
impact
> >> on my compile error:
> >>
> >> echo $MET_PYTHON_CC
> >> -I/home/scott/anaconda3/envs/p38/include/python3.8
> >> -I/home/scott/anaconda3/envs/p38/include/python3.8 -Wno-unused-
result
> >> -Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize
-fPIC
> >> -fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe
-isystem
> >> /home/scott/anaconda3/envs/p38/include
> >>
> -fdebug-prefix-
map=/tmp/build/80754af9/python_1599203911753/work=/usr/local/src/conda/python-
3.8.5
> >>
> >>
> -fdebug-prefix-
map=/home/scott/anaconda3/envs/p38=/usr/local/src/conda-prefix
> >>
> >> -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto
> >> -DNDEBUG -fwrapv -O3 -Wall
> >>
> >> echo $MET_PYTHON_LD
> >>
> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
> >> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl
-lutil -lrt
> >> -lm -lm
> >>
> >> echo $LIBS
> >> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
> >> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lcrypto -lssl
> >>
> >> If I randomly search a few these undefined reference function
names, I
> >> find them in various *.h file in the
> >> /home/scott/anaconda3/envs/p38/include/python3.8 directory.
Similarly, I
> >> can find many of these in the
> >>
> /home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a
> >>
> >> library file. I believe both of these paths are provided to the
linker
> >> so I am a bit confused what is going on here.
> >>
> >> Do you have any other ideas I can try to get this working? Do you
have
> >> any feedback for Anaconda users or recommendations of versions of
python
> >> you have found compatible? Honestly, I'm unsure where the problem
lies.
> >> Any help would be appreciated.
> >>
> >> Thanks,
> >> Scott
> >>
> >>
> >>
> >>
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Scott Rabenhorst
Time: Thu Oct 22 12:43:53 2020
Hi Julie,
Thanks for continuing to look into this. Here is the information you
requested:
1)
ls -1
/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-linux-
gnu/libpython*
/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-linux-
gnu/libpython3.8.a
2)
ls -1 /home/scott/anaconda3/envs/p38/lib/libpython*
/home/scott/anaconda3/envs/p38/lib/libpython3.8.a
/home/scott/anaconda3/envs/p38/lib/libpython3.8.nolto.a
/home/scott/anaconda3/envs/p38/lib/libpython3.8.so
/home/scott/anaconda3/envs/p38/lib/libpython3.8.so.1.0
/home/scott/anaconda3/envs/p38/lib/libpython3.so
3)
I attached the log files from my met build as a tar file.
4) These are the env vars I set beforehand:
export CC=/usr/bin/gcc-7
export CXX=/usr/bin/g++-7
export FC=/usr/bin/gfortran-7
export F77=/usr/bin/gfortran-7
export F90=/usr/bin/gfortran-7
export LDFLAGS="-L${LIB_DIR}/lib"
export CPPFLAGS="-I${LIB_DIR}/include"
export LIBS="-lpthread -lm"
export CFLAGS=" -pthread"
export MET_NETCDF=${LIB_DIR}
export MET_BUFRLIB=${LIB_DIR}/lib
export MET_GSL=${LIB_DIR}
export MET_GRIB2C=${LIB_DIR}
export MET_HDF=${LIB_DIR}
export MET_HDF5=${LIB_DIR}
export MET_HDFEOS=${LIB_DIR}
export MET_CAIROINC=${LIB_DIR}/include/cairo
export MET_CAIROLIB=${LIB_DIR}/lib/cairo
export MET_FREETYPEINC=${LIB_DIR}/include/freetype2
export MET_FREETYPELIB=${LIB_DIR}/lib
export PYCONF=/home/scott/anaconda3/envs/p38/bin/python3-config
export MET_PYTHON_CC=`$PYCONF --cflags`
export MET_PYTHON_LD=`$PYCONF --ldflags`
export
LD_LIBRARY_PATH="/home/scott/anaconda3/envs/p38/lib/python3.8/config-
3.8-x86_64-linux-gnu:${LD_LIBRARY_PATH}"
export LIBS="-lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4
-lnetcdf -lgsl -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread
-lpython3.8 -lcrypto -lssl"
So the native compilers on Ubuntu machine are:
gcc version 7.5.0
gcc version 8.4.0
gcc version 9.3.0 (default)
However, python uses internal compiler GCC 7.3.0. I think this is the
issue with the LTO. On my desktop I have more flexibility to install
packages, but on HPC I do not - I have to pretty much go with what
they
have. I've been looking into ways of forcing Anaconda to build things
with the system's compilers because I think that is the only way in
the
long run. Or maybe you have a clever solution to get met to play
nicely
with anaconda/python ;-)
Please let me know if any other information would be helpful.
Thanks,
Scott
On 10/22/20 10:42 AM, Julie Prestopnik via RT wrote:
> Hi Scott.
>
> I'm glad that you were able to get further.
>
> Could you please send me the output of the following two commands?
>
> - ls
> /home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython*
> - ls /home/scott/anaconda3/envs/p38/lib/libpython*
>
> Could you please also send me your config.log file and your make.log
file?
> Those will give me more information.
>
> Thanks!
>
> Julie
>
> On Thu, Oct 22, 2020 at 8:01 AM Scott Rabenhorst via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>>
>> Hi Julie,
>>
>> Thank you so much for your fast reply. That was a good suggestion.
It
>> seemed to get me further along adding -lpython3.8 to LIBS and the
path
>> to LD_LIBRARY_PATH.
>>
>> Now I am getting a build error something like this:
>>
>> ...
>> Making install in ensemble_stat
>> make[4]: Entering directory
>> '/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
>> /usr/bin/g++-7 -g -L../../../../src/basic/vx_cal
>> -L../../../../src/basic/vx_config -L../../../../src/basic/vx_log
>> -L../../../../src/basic/vx_math -L../../../../src/basic/vx_util
>> -L../../../../src/libcode/vx_afm
>> -L../../../../src/libcode/vx_analysis_util
>> -L../../../../src/libcode/vx_color
-L../../../../src/libcode/vx_data2d
>> -L../../../../src/libcode/vx_data2d_factory
>> -L../../../../src/libcode/vx_data2d_grib
>> -L../../../../src/libcode/vx_data2d_grib2
>> -L../../../../src/libcode/vx_data2d_nccf
>> -L../../../../src/libcode/vx_data2d_nc_met
>> -L../../../../src/libcode/vx_data2d_nc_pinterp
>> -L../../../../src/libcode/vx_data2d_python
>> -L../../../../src/libcode/vx_python3_utils
>> -L../../../../src/libcode/vx_geodesy
-L../../../../src/libcode/vx_gis
>> -L../../../../src/libcode/vx_gnomon
-L../../../../src/libcode/vx_grid
>> -L../../../../src/libcode/vx_gsl_prob
-L../../../../src/libcode/vx_nav
>> -L../../../../src/libcode/vx_nc_obs
-L../../../../src/libcode/vx_nc_util
>> -L../../../../src/libcode/vx_pb_util
>> -L../../../../src/libcode/vx_plot_util
-L../../../../src/libcode/vx_ps
>> -L../../../../src/libcode/vx_pxm
-L../../../../src/libcode/vx_render
>> -L../../../../src/libcode/vx_regrid
>> -L../../../../src/libcode/vx_shapedata
>> -L../../../../src/libcode/vx_solar
>> -L../../../../src/libcode/vx_statistics
>> -L../../../../src/libcode/vx_stat_out
>> -L../../../../src/libcode/vx_summary
>> -L../../../../src/libcode/vx_time_series
>> -L../../../../src/libcode/vx_series_data
>> -L../../../../src/libcode/vx_tc_util -L/opt/app/libs_gnu-
9.3.0_v02/lib
>> -L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
>> -L/opt/app/libs_gnu-9.3.0_v02/lib
>> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
>> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
>> -lm -lm -L/opt/app/libs_gnu-9.3.0_v02/lib
>> -L/opt/app/libs_gnu-9.3.0_v02/lib -L/opt/app/libs_gnu-9.3.0_v02/lib
>> -L/opt/app/libs_gnu-9.3.0_v02/lib
>> -L/opt/app/libs_gnu-9.3.0_v02/lib/cairo
>> -L/opt/app/libs_gnu-9.3.0_v02/lib -o ensemble_stat
>> ensemble_stat-ensemble_stat.o ensemble_stat-
ensemble_stat_conf_info.o
>> -lvx_stat_out -lvx_statistics -lvx_shapedata -lvx_gsl_prob
>> -lvx_analysis_util -lvx_data2d_factory -lvx_data2d_nc_met
>> -lvx_data2d_grib -lvx_data2d_grib2 -lgrib2c -ljasper -lpng -lz
>> -lvx_data2d_nc_pinterp -lvx_data2d_python -lvx_python3_utils
>> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
>> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl -lutil
-lrt
>> -lm -lm -lvx_data2d_nccf -lvx_data2d -lvx_nc_obs -lvx_nc_util
>> -lvx_regrid -lvx_grid -lvx_config -lvx_cal -lvx_util -lvx_math
>> -lvx_color -lvx_log -lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas
>> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
>> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lpython3.8
-lcrypto -lssl
>> lto1: fatal error: bytecode stream in file
>> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>>
>> generated with LTO version 6.0 instead of the expected 6.2
>> compilation terminated.
>> lto-wrapper: fatal error: /usr/bin/g++-7 returned 1 exit status
>> compilation terminated.
>> /usr/bin/ld: error: lto-wrapper failed
>> collect2: error: ld returned 1 exit status
>> Makefile:426: recipe for target 'ensemble_stat' failed
>> make[4]: *** [ensemble_stat] Error 1
>> make[4]: Leaving directory
>> '/opt/app/met/9.1.20200810/gnu-9.3.0/src/tools/core/ensemble_stat'
>> Makefile:379: recipe for target 'install-recursive' failed
>> ...
>>
>> This error looks like an issue caused because Anaconda uses its own
>> internal compiler version and not any of the ones on my system.
Have any
>> of your users encountered this issue before? Do you happen to know
if
>> there are any easy solutions? I tried passing:
>>
>> export CFLAGS="-fno-lto ${CFLAGS}"
>> export LDFLAGS="$(CFLAGS) ${LDFLAGS}"
>>
>> but that didn't seem to make any difference. Please let me know if
you
>> have any thoughts.
>>
>> Thanks,
>> Scott
>>
>>
>> On 10/21/20 5:33 PM, Julie Prestopnik via RT wrote:
>>> Hi Scott.
>>>
>>> I see that you are having difficulty building METv9.1 with python
>> embedding
>>> enabled.
>>>
>>> Thanks for all of this helpful information. Could you please try
setting
>>> MET_PYTHON_CC to the following?
>>>
>>> -I/home/scott/anaconda3/envs/p38/include/python3.8
>>>
>>> Also, it looks like MET_PYTHON_LD is missing the -lpython3.8m, so
please
>>> try adding that before "-lcrypt".
>>>
>>> Then, please run configure again and then try running make again.
Please
>>> let me know how it goes. If it is unsuccessful, please send the
new
>>> make.log file along with the config.log file as well. Thanks and
good
>> luck!
>>> Julie
>>>
>>> On Wed, Oct 21, 2020 at 3:15 PM Scott Rabenhorst via RT <
>> met_help at ucar.edu>
>>> wrote:
>>>
>>>> Wed Oct 21 15:15:25 2020: Request 97175 was acted upon.
>>>> Transaction: Ticket created by scott.rabenhorst at gmail.com
>>>> Queue: met_help
>>>> Subject: Need help compiling MET-9.1 with python
>>>> Owner: Nobody
>>>> Requestors: scott.rabenhorst at gmail.com
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175
>>>>
>>>> Hi MET help,
>>>>
>>>> I am trying to build MET v9.1.20200810 with the '--enable-python'
option
>>>> in hope of installing METplus next. However, it fails to build
halfway
>>>> at the ensemble_stat section. Specifically, the linker fails with
many
>>>> undefined references to python functions (please see attached
make
>>>> output). I confirmed met will build just fine if I remove the
>>>> '--enable-python' configure option.
>>>>
>>>> For background, I am using python-3.8.5 installed in a "p38"
environment
>>>> with Anaconda-4.9.0 on ubuntu-18.04.5. This environment contains
basic
>>>> packages (scipy,matplotlib,cartopy,etc). As per the user guide, I
have
>>>> exported the following environment variables that likely have an
impact
>>>> on my compile error:
>>>>
>>>> echo $MET_PYTHON_CC
>>>> -I/home/scott/anaconda3/envs/p38/include/python3.8
>>>> -I/home/scott/anaconda3/envs/p38/include/python3.8 -Wno-unused-
result
>>>> -Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize
-fPIC
>>>> -fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe
-isystem
>>>> /home/scott/anaconda3/envs/p38/include
>>>>
>> -fdebug-prefix-
map=/tmp/build/80754af9/python_1599203911753/work=/usr/local/src/conda/python-
3.8.5
>>>>
>> -fdebug-prefix-
map=/home/scott/anaconda3/envs/p38=/usr/local/src/conda-prefix
>>>> -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto
>>>> -DNDEBUG -fwrapv -O3 -Wall
>>>>
>>>> echo $MET_PYTHON_LD
>>>>
>> -L/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu
>>>> -L/home/scott/anaconda3/envs/p38/lib -lcrypt -lpthread -ldl
-lutil -lrt
>>>> -lm -lm
>>>>
>>>> echo $LIBS
>>>> -lhdf5_hl -lhdf5 -lmfhdf -ldf -lm -lnetcdf_c++4 -lnetcdf -lgsl
>>>> -lgslcblas -lz -lsz -ljpeg -lcurl -ldl -lpthread -lcrypto -lssl
>>>>
>>>> If I randomly search a few these undefined reference function
names, I
>>>> find them in various *.h file in the
>>>> /home/scott/anaconda3/envs/p38/include/python3.8 directory.
Similarly, I
>>>> can find many of these in the
>>>>
>> /home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a
>>>> library file. I believe both of these paths are provided to the
linker
>>>> so I am a bit confused what is going on here.
>>>>
>>>> Do you have any other ideas I can try to get this working? Do you
have
>>>> any feedback for Anaconda users or recommendations of versions of
python
>>>> you have found compatible? Honestly, I'm unsure where the problem
lies.
>>>> Any help would be appreciated.
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>>
>>>>
>>>>
>>
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Thu Oct 22 13:04:34 2020
Hi Scott.
I have seen this problem:
> lto1: fatal error: bytecode stream in file
> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
> generated with LTO version\
> 6.0 instead of the expected 6.2
>
once before and I believe, as you suspected, that it has to do with
linking
against code that was compiled with an older version of the compiler.
I see in your config.log file that MET is being compiled with "gcc
version
7.5.0". Do you have control of your Anaconda environment? If so, are
you
able to update to Python compiled with 7.5.0 instead of 7.3.0? If
not, are
there other versions of Python installed on your system that you could
potentially use instead?
What HPCs do you have access to? It is possible we have already
installed
a version of METv9.1 on it. Here is a link to our existing builds:
https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
Julie
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #97175] Need help compiling MET-9.1 with python
From: Scott Rabenhorst
Time: Thu Oct 22 13:29:17 2020
Hi Julie,
Yes, I have access to my own anaconda and can install any packages
there, both own my desktop and HPC. I'm having a hard time finding a
way
to get the same gcc version in both as they don't offer the same
versions using the default packages - frustrating. I'm sure there is a
way but not as simple as I was hoping for.
Regarding the existing builds, I don't have access to the machines
listed in your link. I'm with NASA and we are trying to get this on
our
HPC environment to explore additional tools for evaluating GEOS FP. We
have met built there, but not with metplus and other software.
I'll keep looking for was to get the same version in both the system
and
python - I hope this will solve the problem.
Thanks,
Scott
On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
> Hi Scott.
>
> I have seen this problem:
>
>> lto1: fatal error: bytecode stream in file
>> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>> generated with LTO version\
>> 6.0 instead of the expected 6.2
>>
> once before and I believe, as you suspected, that it has to do with
linking
> against code that was compiled with an older version of the
compiler.
>
> I see in your config.log file that MET is being compiled with "gcc
version
> 7.5.0". Do you have control of your Anaconda environment? If so,
are you
> able to update to Python compiled with 7.5.0 instead of 7.3.0? If
not, are
> there other versions of Python installed on your system that you
could
> potentially use instead?
>
> What HPCs do you have access to? It is possible we have already
installed
> a version of METv9.1 on it. Here is a link to our existing builds:
> https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
>
> Julie
>
>
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Thu Oct 22 15:46:24 2020
I'm sorry to hear that getting compiler issues resolved in anaconda
has not
been as simple as you'd hoped. While I have built MET with Python
embedding using an anaconda installation, I'm not incredibly familiar
with
anaconda and haven't done anything like that before. I wish I could
be of
more assistance.
I'll keep looking for how to get the same version in both the system
and
> python - I hope this will solve the problem.
>
I do suspect this will solve the problem. Please let us know if you
have
any other questions or encounter any problems.
Julie
On Thu, Oct 22, 2020 at 1:29 PM Scott Rabenhorst via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>
> Hi Julie,
>
> Yes, I have access to my own anaconda and can install any packages
> there, both own my desktop and HPC. I'm having a hard time finding a
way
> to get the same gcc version in both as they don't offer the same
> versions using the default packages - frustrating. I'm sure there is
a
> way but not as simple as I was hoping for.
>
> Regarding the existing builds, I don't have access to the machines
> listed in your link. I'm with NASA and we are trying to get this on
our
> HPC environment to explore additional tools for evaluating GEOS FP.
We
> have met built there, but not with metplus and other software.
>
> I'll keep looking for was to get the same version in both the system
and
> python - I hope this will solve the problem.
>
> Thanks,
> Scott
>
>
> On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
> > Hi Scott.
> >
> > I have seen this problem:
> >
> >> lto1: fatal error: bytecode stream in file
> >>
> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
> >> generated with LTO version\
> >> 6.0 instead of the expected 6.2
> >>
> > once before and I believe, as you suspected, that it has to do
with
> linking
> > against code that was compiled with an older version of the
compiler.
> >
> > I see in your config.log file that MET is being compiled with "gcc
> version
> > 7.5.0". Do you have control of your Anaconda environment? If so,
are
> you
> > able to update to Python compiled with 7.5.0 instead of 7.3.0? If
not,
> are
> > there other versions of Python installed on your system that you
could
> > potentially use instead?
> >
> > What HPCs do you have access to? It is possible we have already
> installed
> > a version of METv9.1 on it. Here is a link to our existing
builds:
> > https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
> >
> > Julie
> >
> >
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Scott Rabenhorst
Time: Mon Oct 26 11:26:01 2020
Hi Julie,
The good news is I installed the gcc version that Anaconda uses, and
it
looks like everything built and installed just fine. That issue looks
resolved. However, now there is a new problem. Met tests fail on pb2nc
with a core dump. I have attached my logs for both met and bufrlib
11.3.0. The first line in the bufr logs show the build commands. Would
you mind taking a look and seeing if there is anything that should
have
been compiled differently? Any ideas?
Thanks,
Scott
On 10/22/20 3:29 PM, scott rabenhorst wrote:
> Hi Julie,
>
> Yes, I have access to my own anaconda and can install any packages
> there, both own my desktop and HPC. I'm having a hard time finding a
> way to get the same gcc version in both as they don't offer the same
> versions using the default packages - frustrating. I'm sure there is
a
> way but not as simple as I was hoping for.
>
> Regarding the existing builds, I don't have access to the machines
> listed in your link. I'm with NASA and we are trying to get this on
> our HPC environment to explore additional tools for evaluating GEOS
> FP. We have met built there, but not with metplus and other
software.
>
> I'll keep looking for was to get the same version in both the system
> and python - I hope this will solve the problem.
>
> Thanks,
> Scott
>
>
> On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
>> Hi Scott.
>>
>> I have seen this problem:
>>
>>> lto1: fatal error: bytecode stream in file
>>> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>>>
>>> generated with LTO version\
>>> 6.0 instead of the expected 6.2
>>>
>> once before and I believe, as you suspected, that it has to do with
>> linking
>> against code that was compiled with an older version of the
compiler.
>>
>> I see in your config.log file that MET is being compiled with "gcc
>> version
>> 7.5.0". Do you have control of your Anaconda environment? If so,
>> are you
>> able to update to Python compiled with 7.5.0 instead of 7.3.0? If
>> not, are
>> there other versions of Python installed on your system that you
could
>> potentially use instead?
>>
>> What HPCs do you have access to? It is possible we have already
>> installed
>> a version of METv9.1 on it. Here is a link to our existing builds:
>> https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
>>
>> Julie
>>
>>
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Mon Oct 26 12:08:32 2020
Hi Scott.
I'm so glad you were able to get the needed gcc version working in
Anaconda, although I'm sorry to hear that you encountered another
problem
when compiling MET.
Looking in the logs that you sent, I see in the MET config.log:
> CC='/opt/compilers/gcc-7.3.0/bin/gcc'
but in bufr.CC.log it looks like a different version of gcc,
/usr/bin/gcc,
may be being used:
> $CC -c -DUNDERSCORE `./getdefflags_C.sh` *.c
>
+++ ./getdefflags_C.sh
> ++ /usr/bin/gcc -c -DUNDERSCORE -DDYNAMIC_ALLOCATION -DMAXNC=600
-DMXNAF=3
> arallocc.c ardllocc.c bort_exit.c ccbfl.c cmpia.c cmpstia1.c
cmpstia2.c
> cobfl.c cpmstabs.c crbmg.c cread.c cwbmg.c dlloctbf.c icvidx.c
inittbf.c
> nummtb.c rbytes.c restd.c sorttbf.c srchtbf.c strtbfe.c stseq.c
wrdesc.c
Perhaps check your values for CC and FC and ensure that they are
pointing
to the desired compilers (e.g. /opt/compilers/gcc-7.3.0/bin/gcc and
/opt/compilers/gcc-7.3.0/bin/gfortran), then recompile BUFRLIB and
then
reconfigure and recompile MET. Please give that a try and let me know
how
it goes.
Thanks!
Julie
On Mon, Oct 26, 2020 at 11:26 AM Scott Rabenhorst via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>
> Hi Julie,
>
> The good news is I installed the gcc version that Anaconda uses, and
it
> looks like everything built and installed just fine. That issue
looks
> resolved. However, now there is a new problem. Met tests fail on
pb2nc
> with a core dump. I have attached my logs for both met and bufrlib
> 11.3.0. The first line in the bufr logs show the build commands.
Would
> you mind taking a look and seeing if there is anything that should
have
> been compiled differently? Any ideas?
>
> Thanks,
> Scott
>
>
> On 10/22/20 3:29 PM, scott rabenhorst wrote:
> > Hi Julie,
> >
> > Yes, I have access to my own anaconda and can install any packages
> > there, both own my desktop and HPC. I'm having a hard time finding
a
> > way to get the same gcc version in both as they don't offer the
same
> > versions using the default packages - frustrating. I'm sure there
is a
> > way but not as simple as I was hoping for.
> >
> > Regarding the existing builds, I don't have access to the machines
> > listed in your link. I'm with NASA and we are trying to get this
on
> > our HPC environment to explore additional tools for evaluating
GEOS
> > FP. We have met built there, but not with metplus and other
software.
> >
> > I'll keep looking for was to get the same version in both the
system
> > and python - I hope this will solve the problem.
> >
> > Thanks,
> > Scott
> >
> >
> > On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
> >> Hi Scott.
> >>
> >> I have seen this problem:
> >>
> >>> lto1: fatal error: bytecode stream in file
> >>>
> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>
> >>>
> >>> generated with LTO version\
> >>> 6.0 instead of the expected 6.2
> >>>
> >> once before and I believe, as you suspected, that it has to do
with
> >> linking
> >> against code that was compiled with an older version of the
compiler.
> >>
> >> I see in your config.log file that MET is being compiled with
"gcc
> >> version
> >> 7.5.0". Do you have control of your Anaconda environment? If
so,
> >> are you
> >> able to update to Python compiled with 7.5.0 instead of 7.3.0? If
> >> not, are
> >> there other versions of Python installed on your system that you
could
> >> potentially use instead?
> >>
> >> What HPCs do you have access to? It is possible we have already
> >> installed
> >> a version of METv9.1 on it. Here is a link to our existing
builds:
> >> https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
> >>
> >> Julie
> >>
> >>
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #97175] Need help compiling MET-9.1 with python
From: Scott Rabenhorst
Time: Mon Oct 26 13:32:28 2020
Thanks, Julie. That was it. Good suggestion. Everything build and
tested
okay now - yeah! Got to that line:
*** Finished Testing the Model Evaluation Tools Project ***
...now on to installing METplus. Thanks for your help!
~Scott
On 10/26/20 2:08 PM, Julie Prestopnik via RT wrote:
> Hi Scott.
>
> I'm so glad you were able to get the needed gcc version working in
> Anaconda, although I'm sorry to hear that you encountered another
problem
> when compiling MET.
>
> Looking in the logs that you sent, I see in the MET config.log:
>
>> CC='/opt/compilers/gcc-7.3.0/bin/gcc'
>
> but in bufr.CC.log it looks like a different version of gcc,
/usr/bin/gcc,
> may be being used:
>
>> $CC -c -DUNDERSCORE `./getdefflags_C.sh` *.c
>>
> +++ ./getdefflags_C.sh
>> ++ /usr/bin/gcc -c -DUNDERSCORE -DDYNAMIC_ALLOCATION -DMAXNC=600
-DMXNAF=3
>> arallocc.c ardllocc.c bort_exit.c ccbfl.c cmpia.c cmpstia1.c
cmpstia2.c
>> cobfl.c cpmstabs.c crbmg.c cread.c cwbmg.c dlloctbf.c icvidx.c
inittbf.c
>> nummtb.c rbytes.c restd.c sorttbf.c srchtbf.c strtbfe.c stseq.c
wrdesc.c
>
> Perhaps check your values for CC and FC and ensure that they are
pointing
> to the desired compilers (e.g. /opt/compilers/gcc-7.3.0/bin/gcc and
> /opt/compilers/gcc-7.3.0/bin/gfortran), then recompile BUFRLIB and
then
> reconfigure and recompile MET. Please give that a try and let me
know how
> it goes.
>
> Thanks!
>
> Julie
>
>
> On Mon, Oct 26, 2020 at 11:26 AM Scott Rabenhorst via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>>
>> Hi Julie,
>>
>> The good news is I installed the gcc version that Anaconda uses,
and it
>> looks like everything built and installed just fine. That issue
looks
>> resolved. However, now there is a new problem. Met tests fail on
pb2nc
>> with a core dump. I have attached my logs for both met and bufrlib
>> 11.3.0. The first line in the bufr logs show the build commands.
Would
>> you mind taking a look and seeing if there is anything that should
have
>> been compiled differently? Any ideas?
>>
>> Thanks,
>> Scott
>>
>>
>> On 10/22/20 3:29 PM, scott rabenhorst wrote:
>>> Hi Julie,
>>>
>>> Yes, I have access to my own anaconda and can install any packages
>>> there, both own my desktop and HPC. I'm having a hard time finding
a
>>> way to get the same gcc version in both as they don't offer the
same
>>> versions using the default packages - frustrating. I'm sure there
is a
>>> way but not as simple as I was hoping for.
>>>
>>> Regarding the existing builds, I don't have access to the machines
>>> listed in your link. I'm with NASA and we are trying to get this
on
>>> our HPC environment to explore additional tools for evaluating
GEOS
>>> FP. We have met built there, but not with metplus and other
software.
>>>
>>> I'll keep looking for was to get the same version in both the
system
>>> and python - I hope this will solve the problem.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>> On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
>>>> Hi Scott.
>>>>
>>>> I have seen this problem:
>>>>
>>>>> lto1: fatal error: bytecode stream in file
>>>>>
>> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
>>
>>>>> generated with LTO version\
>>>>> 6.0 instead of the expected 6.2
>>>>>
>>>> once before and I believe, as you suspected, that it has to do
with
>>>> linking
>>>> against code that was compiled with an older version of the
compiler.
>>>>
>>>> I see in your config.log file that MET is being compiled with
"gcc
>>>> version
>>>> 7.5.0". Do you have control of your Anaconda environment? If
so,
>>>> are you
>>>> able to update to Python compiled with 7.5.0 instead of 7.3.0? If
>>>> not, are
>>>> there other versions of Python installed on your system that you
could
>>>> potentially use instead?
>>>>
>>>> What HPCs do you have access to? It is possible we have already
>>>> installed
>>>> a version of METv9.1 on it. Here is a link to our existing
builds:
>>>> https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
>>>>
>>>> Julie
>>>>
>>>>
>>
------------------------------------------------
Subject: Need help compiling MET-9.1 with python
From: Julie Prestopnik
Time: Mon Oct 26 13:47:50 2020
Great, Scott! Thanks for letting me know. I'll go ahead and close
this
ticket. Please open a new one if you have any questions or issues
while
installing METplus (or about anything else).
Julie
On Mon, Oct 26, 2020 at 1:33 PM Scott Rabenhorst via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
>
> Thanks, Julie. That was it. Good suggestion. Everything build and
tested
> okay now - yeah! Got to that line:
>
> *** Finished Testing the Model Evaluation Tools Project ***
>
> ...now on to installing METplus. Thanks for your help!
>
> ~Scott
>
>
> On 10/26/20 2:08 PM, Julie Prestopnik via RT wrote:
> > Hi Scott.
> >
> > I'm so glad you were able to get the needed gcc version working in
> > Anaconda, although I'm sorry to hear that you encountered another
problem
> > when compiling MET.
> >
> > Looking in the logs that you sent, I see in the MET config.log:
> >
> >> CC='/opt/compilers/gcc-7.3.0/bin/gcc'
> >
> > but in bufr.CC.log it looks like a different version of gcc,
> /usr/bin/gcc,
> > may be being used:
> >
> >> $CC -c -DUNDERSCORE `./getdefflags_C.sh` *.c
> >>
> > +++ ./getdefflags_C.sh
> >> ++ /usr/bin/gcc -c -DUNDERSCORE -DDYNAMIC_ALLOCATION -DMAXNC=600
> -DMXNAF=3
> >> arallocc.c ardllocc.c bort_exit.c ccbfl.c cmpia.c cmpstia1.c
cmpstia2.c
> >> cobfl.c cpmstabs.c crbmg.c cread.c cwbmg.c dlloctbf.c icvidx.c
inittbf.c
> >> nummtb.c rbytes.c restd.c sorttbf.c srchtbf.c strtbfe.c stseq.c
wrdesc.c
> >
> > Perhaps check your values for CC and FC and ensure that they are
pointing
> > to the desired compilers (e.g. /opt/compilers/gcc-7.3.0/bin/gcc
and
> > /opt/compilers/gcc-7.3.0/bin/gfortran), then recompile BUFRLIB and
then
> > reconfigure and recompile MET. Please give that a try and let me
know
> how
> > it goes.
> >
> > Thanks!
> >
> > Julie
> >
> >
> > On Mon, Oct 26, 2020 at 11:26 AM Scott Rabenhorst via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97175 >
> >>
> >> Hi Julie,
> >>
> >> The good news is I installed the gcc version that Anaconda uses,
and it
> >> looks like everything built and installed just fine. That issue
looks
> >> resolved. However, now there is a new problem. Met tests fail on
pb2nc
> >> with a core dump. I have attached my logs for both met and
bufrlib
> >> 11.3.0. The first line in the bufr logs show the build commands.
Would
> >> you mind taking a look and seeing if there is anything that
should have
> >> been compiled differently? Any ideas?
> >>
> >> Thanks,
> >> Scott
> >>
> >>
> >> On 10/22/20 3:29 PM, scott rabenhorst wrote:
> >>> Hi Julie,
> >>>
> >>> Yes, I have access to my own anaconda and can install any
packages
> >>> there, both own my desktop and HPC. I'm having a hard time
finding a
> >>> way to get the same gcc version in both as they don't offer the
same
> >>> versions using the default packages - frustrating. I'm sure
there is a
> >>> way but not as simple as I was hoping for.
> >>>
> >>> Regarding the existing builds, I don't have access to the
machines
> >>> listed in your link. I'm with NASA and we are trying to get this
on
> >>> our HPC environment to explore additional tools for evaluating
GEOS
> >>> FP. We have met built there, but not with metplus and other
software.
> >>>
> >>> I'll keep looking for was to get the same version in both the
system
> >>> and python - I hope this will solve the problem.
> >>>
> >>> Thanks,
> >>> Scott
> >>>
> >>>
> >>> On 10/22/20 3:04 PM, Julie Prestopnik via RT wrote:
> >>>> Hi Scott.
> >>>>
> >>>> I have seen this problem:
> >>>>
> >>>>> lto1: fatal error: bytecode stream in file
> >>>>>
> >>
> ‘/home/scott/anaconda3/envs/p38/lib/python3.8/config-3.8-x86_64-
linux-gnu/libpython3.8.a’
> >>
> >>>>> generated with LTO version\
> >>>>> 6.0 instead of the expected 6.2
> >>>>>
> >>>> once before and I believe, as you suspected, that it has to do
with
> >>>> linking
> >>>> against code that was compiled with an older version of the
compiler.
> >>>>
> >>>> I see in your config.log file that MET is being compiled with
"gcc
> >>>> version
> >>>> 7.5.0". Do you have control of your Anaconda environment? If
so,
> >>>> are you
> >>>> able to update to Python compiled with 7.5.0 instead of 7.3.0?
If
> >>>> not, are
> >>>> there other versions of Python installed on your system that
you could
> >>>> potentially use instead?
> >>>>
> >>>> What HPCs do you have access to? It is possible we have
already
> >>>> installed
> >>>> a version of METv9.1 on it. Here is a link to our existing
builds:
> >>>>
> https://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
> >>>>
> >>>> Julie
> >>>>
> >>>>
> >>
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list