[Met_help] [rt.rap.ucar.edu #88781] History for MODE failure

John Halley Gotway via RT met_help at ucar.edu
Wed Jul 10 17:01:08 MDT 2019 

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello,

We have managed to get a failure again when using the BOTH option for merging. It only happens on selected files and we don't understand why. Here is the output.  We've also attached the output that is produced for the other merging options: engine, thresh and none. All of them work fine for this pair of input files. [Note the forecast is a rebadged analysis field as we don't have any forecast data at the moment so we're creating a persistence forecast, which gives a warning. It does mean that the output file names are also wrong but it's not a problem at the moment.]

The script and config file are also attached.

Regards and thanks in advance,
Marion & Rachel
==============================================================================================================
vld303:/project/hive/scripts > run_mode_BIO_wrapper_by_day.sh
mode /scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-CPWC_Forecast_24_2018-07-12.nc /project/hive/input/model/BIO/MetO-NWS-BIO-dm-CPWC_Analysis_2018-07-12.nc /project/hive//MET/MODEConfig_BIO -outdir /project/hive/output/mode/BIO/fake_forecasts_mergeboth ...
DEBUG 1: Default Config File: /project/ukmo/meteval/met-8.0/share/met/config/MODEConfig_default
DEBUG 1: Match Config File: /project/hive//config/MODEConfig_BIO_quilt_mergeboth
DEBUG 1: Merge Config File: /project/hive//config/MODEConfig_BIO_quilt_mergeboth
WARNING:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention time unit "hours since  00:00:00"
WARNING:
WARNING:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention time unit "hours since  00:00:00"
WARNING:
DEBUG 1: Forecast File: /scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-CPWC_Forecast_24_2018-07-12.nc
DEBUG 1: Observation File: /project/hive/input/model/BIO/MetO-NWS-BIO-dm-CPWC_Analysis_2018-07-12.nc
WARNING:
WARNING: setup_fcst_obs_data() -> Forecast and observation valid times do not match 20180711_120000 != 20180712_120000 for CHL(0,0,*,*) versus CHL(0,0,*,*).
WARNING:
DEBUG 1: Forecast Field: CHL at 0,0,*,*
DEBUG 1: Observation Field: CHL at 0,0,*,*
DEBUG 2: Processing masking regions.
DEBUG 2: Identifying objects in the forecast and observation fields...
DEBUG 2: Computing contingency table statistics...
DEBUG 2: Identified: 35 forecast objects and 37 observation objects.
DEBUG 2: Performing merging (BOTH) in the forecast field.
DEBUG 2: Performing merging (BOTH) in the observation field.
DEBUG 2: Remaining: 35 forecast objects and 37 observation objects.
DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and observation fields.
*** Error in `mode': free(): invalid next size (normal): 0x0000000001430cb0 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x81489)[0x2b8f3b36a489]
mode[0x4a3d0b]
mode[0x4a3b82]
mode[0x4a4614]
mode[0x4a468e]
mode[0x49531b]
mode[0x493df4]
mode[0x423026]
mode[0x411dc6]
mode[0x411c43]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b8f3b30b3d5]
mode[0x411b09]
======= Memory map: ========
00400000-007dc000 r-xp 00000000 00:33 42920288                           /net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
009db000-00abd000 rw-p 003db000 00:33 42920288                           /net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
00abd000-00ae5000 rw-p 00000000 00:00 0
01404000-07924000 rw-p 00000000 00:00 0                                  [heap]
2b8f3985f000-2b8f39881000 r-xp 00000000 fd:00 8389987                    /usr/lib64/ld-2.17.so
2b8f39881000-2b8f39884000 rw-p 00000000 00:00 0
2b8f398a1000-2b8f39a12000 rw-p 00000000 00:00 0
2b8f39a80000-2b8f39a81000 r--p 00021000 fd:00 8389987                    /usr/lib64/ld-2.17.so
2b8f39a81000-2b8f39a82000 rw-p 00022000 fd:00 8389987                    /usr/lib64/ld-2.17.so
2b8f39a82000-2b8f39a83000 rw-p 00000000 00:00 0
2b8f39a83000-2b8f39a85000 r-xp 00000000 fd:00 8390030                    /usr/lib64/libutil-2.17.so
2b8f39a85000-2b8f39c84000 ---p 00002000 fd:00 8390030                    /usr/lib64/libutil-2.17.so
2b8f39c84000-2b8f39c85000 r--p 00001000 fd:00 8390030                    /usr/lib64/libutil-2.17.so
2b8f39c85000-2b8f39c86000 rw-p 00002000 fd:00 8390030                    /usr/lib64/libutil-2.17.so
2b8f39c86000-2b8f39ccf000 r-xp 00000000 00:33 30659164                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ccf000-2b8f39ecf000 ---p 00049000 00:33 30659164                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ecf000-2b8f39ed3000 rw-p 00049000 00:33 30659164                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ed3000-2b8f3a02c000 r-xp 00000000 00:33 18140739                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a02c000-2b8f3a22b000 ---p 00159000 00:33 18140739                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a22b000-2b8f3a241000 rw-p 00158000 00:33 18140739                   /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a241000-2b8f3a24c000 rw-p 00000000 00:00 0
2b8f3a24c000-2b8f3a461000 r-xp 00000000 fd:00 9064793                    /usr/lib64/libgsl.so.0.16.0
2b8f3a461000-2b8f3a661000 ---p 00215000 fd:00 9064793                    /usr/lib64/libgsl.so.0.16.0
2b8f3a661000-2b8f3a664000 r--p 00215000 fd:00 9064793                    /usr/lib64/libgsl.so.0.16.0
2b8f3a664000-2b8f3a674000 rw-p 00218000 fd:00 9064793                    /usr/lib64/libgsl.so.0.16.0
2b8f3a674000-2b8f3a675000 rw-p 00000000 00:00 0
2b8f3a675000-2b8f3a6b1000 r-xp 00000000 fd:00 9064795                    /usr/lib64/libgslcblas.so.0.0.0
2b8f3a6b1000-2b8f3a8b0000 ---p 0003c000 fd:00 9064795                    /usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b0000-2b8f3a8b1000 r--p 0003b000 fd:00 9064795                    /usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b1000-2b8f3a8b2000 rw-p 0003c000 fd:00 9064795                    /usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b2000-2b8f3a8c9000 r-xp 00000000 00:33 2616435                    /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3a8c9000-2b8f3aac9000 ---p 00017000 00:33 2616435                    /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3aac9000-2b8f3aaca000 rw-p 00017000 00:33 2616435                    /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3aaca000-2b8f3abb3000 r-xp 00000000 fd:00 8954912                    /usr/lib64/libstdc++.so.6.0.19
2b8f3abb3000-2b8f3adb2000 ---p 000e9000 fd:00 8954912                    /usr/lib64/libstdc++.so.6.0.19
2b8f3adb2000-2b8f3adba000 r--p 000e8000 fd:00 8954912                    /usr/lib64/libstdc++.so.6.0.19
2b8f3adba000-2b8f3adbc000 rw-p 000f0000 fd:00 8954912                    /usr/lib64/libstdc++.so.6.0.19
2b8f3adbc000-2b8f3add1000 rw-p 00000000 00:00 0
2b8f3add1000-2b8f3aed2000 r-xp 00000000 fd:00 8390004                    /usr/lib64/libm-2.17.so
2b8f3aed2000-2b8f3b0d1000 ---p 00101000 fd:00 8390004                    /usr/lib64/libm-2.17.so
2b8f3b0d1000-2b8f3b0d2000 r--p 00100000 fd:00 8390004                    /usr/lib64/libm-2.17.so
2b8f3b0d2000-2b8f3b0d3000 rw-p 00101000 fd:00 8390004                    /usr/lib64/libm-2.17.so
2b8f3b0d3000-2b8f3b0e8000 r-xp 00000000 fd:00 10593334                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b0e8000-2b8f3b2e7000 ---p 00015000 fd:00 10593334                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e7000-2b8f3b2e8000 r--p 00014000 fd:00 10593334                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e8000-2b8f3b2e9000 rw-p 00015000 fd:00 10593334                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e9000-2b8f3b4ab000 r-xp 00000000 fd:00 8389996                    /usr/lib64/libc-2.17.so
2b8f3b4ab000-2b8f3b6ab000 ---p 001c2000 fd:00 8389996                    /usr/lib64/libc-2.17.so
2b8f3b6ab000-2b8f3b6af000 r--p 001c2000 fd:00 8389996                    /usr/lib64/libc-2.17.so
2b8f3b6af000-2b8f3b6b1000 rw-p 001c6000 fd:00 8389996                    /usr/lib64/libc-2.17.so
2b8f3b6b1000-2b8f3b6b6000 rw-p 00000000 00:00 0
2b8f3b6b6000-2b8f3b6b8000 r-xp 00000000 fd:00 8390002                    /usr/lib64/libdl-2.17.so
2b8f3b6b8000-2b8f3b8b8000 ---p 00002000 fd:00 8390002                    /usr/lib64/libdl-2.17.so
2b8f3b8b8000-2b8f3b8b9000 r--p 00002000 fd:00 8390002                    /usr/lib64/libdl-2.17.so
2b8f3b8b9000-2b8f3b8ba000 rw-p 00003000 fd:00 8390002                    /usr/lib64/libdl-2.17.so
2b8f3b8ba000-2b8f3b8d1000 r-xp 00000000 fd:00 8390022                    /usr/lib64/libpthread-2.17.so
2b8f3b8d1000-2b8f3bad0000 ---p 00017000 fd:00 8390022                    /usr/lib64/libpthread-2.17.so
2b8f3bad0000-2b8f3bad1000 r--p 00016000 fd:00 8390022                    /usr/lib64/libpthread-2.17.so
2b8f3bad1000-2b8f3bad2000 rw-p 00017000 fd:00 8390022                    /usr/lib64/libpthread-2.17.so
2b8f3bad2000-2b8f3bad6000 rw-p 00000000 00:00 0
2b8f3bad6000-2b8f3baf6000 r-xp 00000000 00:33 18140735                   /net/project/ukmo/rhel6/meteval/lib/libhdf5_hl.so.100.1.0run_mode_BIO_wrapper_by_day.sh: line 39: 54415 Aborted                 mode ${FCSTDIR}/MetO-NWS-BIO-dm-CPWC_Forecast_24_${YYYY}-${MM}-${DD}.nc ${OBSDIR}/MetO-NWS-BIO-dm-CPWC_Analysis_${YYYY}-${MM}-${DD}.nc ${RUNDIR}/config/MODEConfig_BIO_quilt_mergeboth -outdir ${OUTDIR}

--
Dr Marion Mittermaier     Manager: Model diagnostics and novel verification

Met Office   FitzRoy Road   Exeter   EX1 3PB   United Kingdom
Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604  Fax: +44 (0)1392 885681
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>  http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>

http://www.metoffice.gov.uk/research/people/marion-mittermaier

I am co-chair of the WMO WWRP Joint Working Group on Forecast Verification Research (JWGFVR) http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: FW: [rt.rap.ucar.edu #88781] AutoReply: MODE failure
From: marion.mittermaier at metoffice.gov.uk
Time: Thu Mar 28 08:12:45 2019

Hello,

just wondering if anyone has been able to look at this ticket we
raised a while ago? It is hindering project progress.

Last time we spoke you said you couldn't replicate the failure but
weren't sure. It would be good to know if that is the case.

Many thanks
Marion

-----Original Message-----
From: met_help at ucar.edu via RT <met_help at ucar.edu>
Sent: 05 February 2019 09:33
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: [rt.rap.ucar.edu #88781] AutoReply: MODE failure

Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding:
	"MODE failure",
a summary of which appears below.

There is no need to reply to this message right now.  Your ticket has
been assigned an ID of [rt.rap.ucar.edu #88781].

Please include the string:

         [rt.rap.ucar.edu #88781]

in the subject line of all future correspondence about this issue. To
do so, you may reply to this message.

For more information, please see:

MET Online Tutorial:
   https://www.dtcenter.org/met/users/support/online_tutorial/index.php

MET Users Guide:
   https://www.dtcenter.org/met/users/docs/overview.php

MET FAQs:
   https://www.dtcenter.org/met/users/support/faqs/index.php

MET-Help Email Archive:
   http://mailman.ucar.edu/pipermail/met_help

                        Thank you,
                        met_help at ucar.edu

-------------------------------------------------------------------------
Hello,

We have managed to get a failure again when using the BOTH option for
merging. It only happens on selected files and we don't understand
why. Here is the output.  We've also attached the output that is
produced for the other merging options: engine, thresh and none. All
of them work fine for this pair of input files. [Note the forecast is
a rebadged analysis field as we don't have any forecast data at the
moment so we're creating a persistence forecast, which gives a
warning. It does mean that the output file names are also wrong but
it's not a problem at the moment.]

The script and config file are also attached.

Regards and thanks in advance,
Marion & Rachel
==============================================================================================================
vld303:/project/hive/scripts > run_mode_BIO_wrapper_by_day.sh mode
/scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-
CPWC_Forecast_24_2018-07-12.nc /project/hive/input/model/BIO/MetO-NWS-
BIO-dm-CPWC_Analysis_2018-07-12.nc /project/hive//MET/MODEConfig_BIO
-outdir /project/hive/output/mode/BIO/fake_forecasts_mergeboth ...
DEBUG 1: Default Config File: /project/ukmo/meteval/met-
8.0/share/met/config/MODEConfig_default
DEBUG 1: Match Config File:
/project/hive//config/MODEConfig_BIO_quilt_mergeboth
DEBUG 1: Merge Config File:
/project/hive//config/MODEConfig_BIO_quilt_mergeboth
WARNING:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time unit "hours since  00:00:00"
WARNING:
WARNING:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time unit "hours since  00:00:00"
WARNING:
DEBUG 1: Forecast File:
/scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-
CPWC_Forecast_24_2018-07-12.nc
DEBUG 1: Observation File: /project/hive/input/model/BIO/MetO-NWS-BIO-
dm-CPWC_Analysis_2018-07-12.nc
WARNING:
WARNING: setup_fcst_obs_data() -> Forecast and observation valid times
do not match 20180711_120000 != 20180712_120000 for CHL(0,0,*,*)
versus CHL(0,0,*,*).
WARNING:
DEBUG 1: Forecast Field: CHL at 0,0,*,*
DEBUG 1: Observation Field: CHL at 0,0,*,* DEBUG 2: Processing masking
regions.
DEBUG 2: Identifying objects in the forecast and observation fields...
DEBUG 2: Computing contingency table statistics...
DEBUG 2: Identified: 35 forecast objects and 37 observation objects.
DEBUG 2: Performing merging (BOTH) in the forecast field.
DEBUG 2: Performing merging (BOTH) in the observation field.
DEBUG 2: Remaining: 35 forecast objects and 37 observation objects.
DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and
observation fields.
*** Error in `mode': free(): invalid next size (normal):
0x0000000001430cb0 *** ======= Backtrace: =========
/usr/lib64/libc.so.6(+0x81489)[0x2b8f3b36a489]
mode[0x4a3d0b]
mode[0x4a3b82]
mode[0x4a4614]
mode[0x4a468e]
mode[0x49531b]
mode[0x493df4]
mode[0x423026]
mode[0x411dc6]
mode[0x411c43]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b8f3b30b3d5]
mode[0x411b09]
======= Memory map: ========
00400000-007dc000 r-xp 00000000 00:33 42920288
/net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
009db000-00abd000 rw-p 003db000 00:33 42920288
/net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
00abd000-00ae5000 rw-p 00000000 00:00 0
01404000-07924000 rw-p 00000000 00:00 0
[heap]
2b8f3985f000-2b8f39881000 r-xp 00000000 fd:00 8389987
/usr/lib64/ld-2.17.so
2b8f39881000-2b8f39884000 rw-p 00000000 00:00 0
2b8f398a1000-2b8f39a12000 rw-p 00000000 00:00 0
2b8f39a80000-2b8f39a81000 r--p 00021000 fd:00 8389987
/usr/lib64/ld-2.17.so
2b8f39a81000-2b8f39a82000 rw-p 00022000 fd:00 8389987
/usr/lib64/ld-2.17.so
2b8f39a82000-2b8f39a83000 rw-p 00000000 00:00 0
2b8f39a83000-2b8f39a85000 r-xp 00000000 fd:00 8390030
/usr/lib64/libutil-2.17.so
2b8f39a85000-2b8f39c84000 ---p 00002000 fd:00 8390030
/usr/lib64/libutil-2.17.so
2b8f39c84000-2b8f39c85000 r--p 00001000 fd:00 8390030
/usr/lib64/libutil-2.17.so
2b8f39c85000-2b8f39c86000 rw-p 00002000 fd:00 8390030
/usr/lib64/libutil-2.17.so
2b8f39c86000-2b8f39ccf000 r-xp 00000000 00:33 30659164
/net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ccf000-2b8f39ecf000 ---p 00049000 00:33 30659164
/net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ecf000-2b8f39ed3000 rw-p 00049000 00:33 30659164
/net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
2b8f39ed3000-2b8f3a02c000 r-xp 00000000 00:33 18140739
/net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a02c000-2b8f3a22b000 ---p 00159000 00:33 18140739
/net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a22b000-2b8f3a241000 rw-p 00158000 00:33 18140739
/net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
2b8f3a241000-2b8f3a24c000 rw-p 00000000 00:00 0
2b8f3a24c000-2b8f3a461000 r-xp 00000000 fd:00 9064793
/usr/lib64/libgsl.so.0.16.0
2b8f3a461000-2b8f3a661000 ---p 00215000 fd:00 9064793
/usr/lib64/libgsl.so.0.16.0
2b8f3a661000-2b8f3a664000 r--p 00215000 fd:00 9064793
/usr/lib64/libgsl.so.0.16.0
2b8f3a664000-2b8f3a674000 rw-p 00218000 fd:00 9064793
/usr/lib64/libgsl.so.0.16.0
2b8f3a674000-2b8f3a675000 rw-p 00000000 00:00 0
2b8f3a675000-2b8f3a6b1000 r-xp 00000000 fd:00 9064795
/usr/lib64/libgslcblas.so.0.0.0
2b8f3a6b1000-2b8f3a8b0000 ---p 0003c000 fd:00 9064795
/usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b0000-2b8f3a8b1000 r--p 0003b000 fd:00 9064795
/usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b1000-2b8f3a8b2000 rw-p 0003c000 fd:00 9064795
/usr/lib64/libgslcblas.so.0.0.0
2b8f3a8b2000-2b8f3a8c9000 r-xp 00000000 00:33 2616435
/net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3a8c9000-2b8f3aac9000 ---p 00017000 00:33 2616435
/net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3aac9000-2b8f3aaca000 rw-p 00017000 00:33 2616435
/net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
2b8f3aaca000-2b8f3abb3000 r-xp 00000000 fd:00 8954912
/usr/lib64/libstdc++.so.6.0.19
2b8f3abb3000-2b8f3adb2000 ---p 000e9000 fd:00 8954912
/usr/lib64/libstdc++.so.6.0.19
2b8f3adb2000-2b8f3adba000 r--p 000e8000 fd:00 8954912
/usr/lib64/libstdc++.so.6.0.19
2b8f3adba000-2b8f3adbc000 rw-p 000f0000 fd:00 8954912
/usr/lib64/libstdc++.so.6.0.19
2b8f3adbc000-2b8f3add1000 rw-p 00000000 00:00 0
2b8f3add1000-2b8f3aed2000 r-xp 00000000 fd:00 8390004
/usr/lib64/libm-2.17.so
2b8f3aed2000-2b8f3b0d1000 ---p 00101000 fd:00 8390004
/usr/lib64/libm-2.17.so
2b8f3b0d1000-2b8f3b0d2000 r--p 00100000 fd:00 8390004
/usr/lib64/libm-2.17.so
2b8f3b0d2000-2b8f3b0d3000 rw-p 00101000 fd:00 8390004
/usr/lib64/libm-2.17.so
2b8f3b0d3000-2b8f3b0e8000 r-xp 00000000 fd:00 10593334
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b0e8000-2b8f3b2e7000 ---p 00015000 fd:00 10593334
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e7000-2b8f3b2e8000 r--p 00014000 fd:00 10593334
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e8000-2b8f3b2e9000 rw-p 00015000 fd:00 10593334
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b8f3b2e9000-2b8f3b4ab000 r-xp 00000000 fd:00 8389996
/usr/lib64/libc-2.17.so
2b8f3b4ab000-2b8f3b6ab000 ---p 001c2000 fd:00 8389996
/usr/lib64/libc-2.17.so
2b8f3b6ab000-2b8f3b6af000 r--p 001c2000 fd:00 8389996
/usr/lib64/libc-2.17.so
2b8f3b6af000-2b8f3b6b1000 rw-p 001c6000 fd:00 8389996
/usr/lib64/libc-2.17.so
2b8f3b6b1000-2b8f3b6b6000 rw-p 00000000 00:00 0
2b8f3b6b6000-2b8f3b6b8000 r-xp 00000000 fd:00 8390002
/usr/lib64/libdl-2.17.so
2b8f3b6b8000-2b8f3b8b8000 ---p 00002000 fd:00 8390002
/usr/lib64/libdl-2.17.so
2b8f3b8b8000-2b8f3b8b9000 r--p 00002000 fd:00 8390002
/usr/lib64/libdl-2.17.so
2b8f3b8b9000-2b8f3b8ba000 rw-p 00003000 fd:00 8390002
/usr/lib64/libdl-2.17.so
2b8f3b8ba000-2b8f3b8d1000 r-xp 00000000 fd:00 8390022
/usr/lib64/libpthread-2.17.so
2b8f3b8d1000-2b8f3bad0000 ---p 00017000 fd:00 8390022
/usr/lib64/libpthread-2.17.so
2b8f3bad0000-2b8f3bad1000 r--p 00016000 fd:00 8390022
/usr/lib64/libpthread-2.17.so
2b8f3bad1000-2b8f3bad2000 rw-p 00017000 fd:00 8390022
/usr/lib64/libpthread-2.17.so
2b8f3bad2000-2b8f3bad6000 rw-p 00000000 00:00 0
2b8f3bad6000-2b8f3baf6000 r-xp 00000000 00:33 18140735
/net/project/ukmo/rhel6/meteval/lib/libhdf5_hl.so.100.1.0run_mode_BIO_wrapper_by_day.sh:
line 39: 54415 Aborted                 mode ${FCSTDIR}/MetO-NWS-BIO-
dm-CPWC_Forecast_24_${YYYY}-${MM}-${DD}.nc ${OBSDIR}/MetO-NWS-BIO-dm-
CPWC_Analysis_${YYYY}-${MM}-${DD}.nc
${RUNDIR}/config/MODEConfig_BIO_quilt_mergeboth -outdir ${OUTDIR}

--
Dr Marion Mittermaier     Manager: Model diagnostics and novel
verification

Met Office   FitzRoy Road   Exeter   EX1 3PB   United Kingdom
Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604  Fax: +44 (0)1392
885681
E-mail:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>

http://www.metoffice.gov.uk/research/people/marion-mittermaier

I am co-chair of the WMO WWRP Joint Working Group on Forecast
Verification Research (JWGFVR)
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html





------------------------------------------------
Subject: MODE failure
From: Randy Bullock
Time: Thu Mar 28 11:06:03 2019

Hi Marion -

I hadn't looked at this in a while (we've been busy putting out other
fires) so I copied all the associated files for this issue to a new
area
and started from scratch.  That must've been the right way to go
because I
can now replicate your segfault, and, though it took me most of the
morning, I believe I have found the part of the MODE code that is
causing
the error.  Yay!

I know you've been waiting for this for a while, so it's strongly
tempting
for me to just slam out some code in a shoot-from-the-hip sort of way
to
patch this, but the problem is in a complicated part of the code and I
think I should talk this over with some of the other developers and
get
their thoughts on how best to proceed.  Unintended consequences are
the
bane of the software engineer's existence.

But anyway ... Progress!   And hopefully a fix sometime next week.

Take care.

Randy


On Thu, Mar 28, 2019 at 8:12 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88781 >
>
> Hello,
>
> just wondering if anyone has been able to look at this ticket we
raised a
> while ago? It is hindering project progress.
>
> Last time we spoke you said you couldn't replicate the failure but
weren't
> sure. It would be good to know if that is the case.
>
> Many thanks
> Marion
>
> -----Original Message-----
> From: met_help at ucar.edu via RT <met_help at ucar.edu>
> Sent: 05 February 2019 09:33
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: [rt.rap.ucar.edu #88781] AutoReply: MODE failure
>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
>         "MODE failure",
> a summary of which appears below.
>
> There is no need to reply to this message right now.  Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #88781].
>
> Please include the string:
>
>          [rt.rap.ucar.edu #88781]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
>    https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
>    https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
>    http://mailman.ucar.edu/pipermail/met_help
>
>                         Thank you,
>                         met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> Hello,
>
> We have managed to get a failure again when using the BOTH option
for
> merging. It only happens on selected files and we don't understand
why.
> Here is the output.  We've also attached the output that is produced
for
> the other merging options: engine, thresh and none. All of them work
fine
> for this pair of input files. [Note the forecast is a rebadged
analysis
> field as we don't have any forecast data at the moment so we're
creating a
> persistence forecast, which gives a warning. It does mean that the
output
> file names are also wrong but it's not a problem at the moment.]
>
> The script and config file are also attached.
>
> Regards and thanks in advance,
> Marion & Rachel
>
>
==============================================================================================================
> vld303:/project/hive/scripts > run_mode_BIO_wrapper_by_day.sh mode
> /scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-
CPWC_Forecast_24_
> 2018-07-12.nc /project/hive/input/model/BIO/MetO-NWS-BIO-dm-
CPWC_Analysis_
> 2018-07-12.nc /project/hive//MET/MODEConfig_BIO -outdir
> /project/hive/output/mode/BIO/fake_forecasts_mergeboth ...
> DEBUG 1: Default Config File:
> /project/ukmo/meteval/met-8.0/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File:
> /project/hive//config/MODEConfig_BIO_quilt_mergeboth
> DEBUG 1: Merge Config File:
> /project/hive//config/MODEConfig_BIO_quilt_mergeboth
> WARNING:
> WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
> unit "hours since  00:00:00"
> WARNING:
> WARNING:
> WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
> unit "hours since  00:00:00"
> WARNING:
> DEBUG 1: Forecast File:
> /scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-
CPWC_Forecast_24_
> 2018-07-12.nc
> DEBUG 1: Observation File:
> /project/hive/input/model/BIO/MetO-NWS-BIO-dm-CPWC_Analysis_2018-07-
12.nc
> WARNING:
> WARNING: setup_fcst_obs_data() -> Forecast and observation valid
times do
> not match 20180711_120000 != 20180712_120000 for CHL(0,0,*,*) versus
> CHL(0,0,*,*).
> WARNING:
> DEBUG 1: Forecast Field: CHL at 0,0,*,*
> DEBUG 1: Observation Field: CHL at 0,0,*,* DEBUG 2: Processing
masking
> regions.
> DEBUG 2: Identifying objects in the forecast and observation
fields...
> DEBUG 2: Computing contingency table statistics...
> DEBUG 2: Identified: 35 forecast objects and 37 observation objects.
> DEBUG 2: Performing merging (BOTH) in the forecast field.
> DEBUG 2: Performing merging (BOTH) in the observation field.
> DEBUG 2: Remaining: 35 forecast objects and 37 observation objects.
> DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and
> observation fields.
> *** Error in `mode': free(): invalid next size (normal):
> 0x0000000001430cb0 *** ======= Backtrace: =========
> /usr/lib64/libc.so.6(+0x81489)[0x2b8f3b36a489]
> mode[0x4a3d0b]
> mode[0x4a3b82]
> mode[0x4a4614]
> mode[0x4a468e]
> mode[0x49531b]
> mode[0x493df4]
> mode[0x423026]
> mode[0x411dc6]
> mode[0x411c43]
> /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b8f3b30b3d5]
> mode[0x411b09]
> ======= Memory map: ========
> 00400000-007dc000 r-xp 00000000 00:33 42920288
>  /net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
> 009db000-00abd000 rw-p 003db000 00:33 42920288
>  /net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
> 00abd000-00ae5000 rw-p 00000000 00:00 0
> 01404000-07924000 rw-p 00000000 00:00 0
> [heap]
> 2b8f3985f000-2b8f39881000 r-xp 00000000 fd:00 8389987
> /usr/lib64/ld-2.17.so
> 2b8f39881000-2b8f39884000 rw-p 00000000 00:00 0
> 2b8f398a1000-2b8f39a12000 rw-p 00000000 00:00 0
> 2b8f39a80000-2b8f39a81000 r--p 00021000 fd:00 8389987
> /usr/lib64/ld-2.17.so
> 2b8f39a81000-2b8f39a82000 rw-p 00022000 fd:00 8389987
> /usr/lib64/ld-2.17.so
> 2b8f39a82000-2b8f39a83000 rw-p 00000000 00:00 0
> 2b8f39a83000-2b8f39a85000 r-xp 00000000 fd:00 8390030
> /usr/lib64/libutil-2.17.so
> 2b8f39a85000-2b8f39c84000 ---p 00002000 fd:00 8390030
> /usr/lib64/libutil-2.17.so
> 2b8f39c84000-2b8f39c85000 r--p 00001000 fd:00 8390030
> /usr/lib64/libutil-2.17.so
> 2b8f39c85000-2b8f39c86000 rw-p 00002000 fd:00 8390030
> /usr/lib64/libutil-2.17.so
> 2b8f39c86000-2b8f39ccf000 r-xp 00000000 00:33 30659164
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
> 2b8f39ccf000-2b8f39ecf000 ---p 00049000 00:33 30659164
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
> 2b8f39ecf000-2b8f39ed3000 rw-p 00049000 00:33 30659164
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf_c++4.so.1.0.3
> 2b8f39ed3000-2b8f3a02c000 r-xp 00000000 00:33 18140739
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
> 2b8f3a02c000-2b8f3a22b000 ---p 00159000 00:33 18140739
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
> 2b8f3a22b000-2b8f3a241000 rw-p 00158000 00:33 18140739
>  /net/project/ukmo/rhel6/meteval/lib/libnetcdf.so.13.1.1
> 2b8f3a241000-2b8f3a24c000 rw-p 00000000 00:00 0
> 2b8f3a24c000-2b8f3a461000 r-xp 00000000 fd:00 9064793
> /usr/lib64/libgsl.so.0.16.0
> 2b8f3a461000-2b8f3a661000 ---p 00215000 fd:00 9064793
> /usr/lib64/libgsl.so.0.16.0
> 2b8f3a661000-2b8f3a664000 r--p 00215000 fd:00 9064793
> /usr/lib64/libgsl.so.0.16.0
> 2b8f3a664000-2b8f3a674000 rw-p 00218000 fd:00 9064793
> /usr/lib64/libgsl.so.0.16.0
> 2b8f3a674000-2b8f3a675000 rw-p 00000000 00:00 0
> 2b8f3a675000-2b8f3a6b1000 r-xp 00000000 fd:00 9064795
> /usr/lib64/libgslcblas.so.0.0.0
> 2b8f3a6b1000-2b8f3a8b0000 ---p 0003c000 fd:00 9064795
> /usr/lib64/libgslcblas.so.0.0.0
> 2b8f3a8b0000-2b8f3a8b1000 r--p 0003b000 fd:00 9064795
> /usr/lib64/libgslcblas.so.0.0.0
> 2b8f3a8b1000-2b8f3a8b2000 rw-p 0003c000 fd:00 9064795
> /usr/lib64/libgslcblas.so.0.0.0
> 2b8f3a8b2000-2b8f3a8c9000 r-xp 00000000 00:33 2616435
> /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
> 2b8f3a8c9000-2b8f3aac9000 ---p 00017000 00:33 2616435
> /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
> 2b8f3aac9000-2b8f3aaca000 rw-p 00017000 00:33 2616435
> /net/project/ukmo/rhel6/meteval/lib/libz.so.1.2.11
> 2b8f3aaca000-2b8f3abb3000 r-xp 00000000 fd:00 8954912
> /usr/lib64/libstdc++.so.6.0.19
> 2b8f3abb3000-2b8f3adb2000 ---p 000e9000 fd:00 8954912
> /usr/lib64/libstdc++.so.6.0.19
> 2b8f3adb2000-2b8f3adba000 r--p 000e8000 fd:00 8954912
> /usr/lib64/libstdc++.so.6.0.19
> 2b8f3adba000-2b8f3adbc000 rw-p 000f0000 fd:00 8954912
> /usr/lib64/libstdc++.so.6.0.19
> 2b8f3adbc000-2b8f3add1000 rw-p 00000000 00:00 0
> 2b8f3add1000-2b8f3aed2000 r-xp 00000000 fd:00 8390004
> /usr/lib64/libm-2.17.so
> 2b8f3aed2000-2b8f3b0d1000 ---p 00101000 fd:00 8390004
> /usr/lib64/libm-2.17.so
> 2b8f3b0d1000-2b8f3b0d2000 r--p 00100000 fd:00 8390004
> /usr/lib64/libm-2.17.so
> 2b8f3b0d2000-2b8f3b0d3000 rw-p 00101000 fd:00 8390004
> /usr/lib64/libm-2.17.so
> 2b8f3b0d3000-2b8f3b0e8000 r-xp 00000000 fd:00 10593334
>  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 2b8f3b0e8000-2b8f3b2e7000 ---p 00015000 fd:00 10593334
>  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 2b8f3b2e7000-2b8f3b2e8000 r--p 00014000 fd:00 10593334
>  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 2b8f3b2e8000-2b8f3b2e9000 rw-p 00015000 fd:00 10593334
>  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 2b8f3b2e9000-2b8f3b4ab000 r-xp 00000000 fd:00 8389996
> /usr/lib64/libc-2.17.so
> 2b8f3b4ab000-2b8f3b6ab000 ---p 001c2000 fd:00 8389996
> /usr/lib64/libc-2.17.so
> 2b8f3b6ab000-2b8f3b6af000 r--p 001c2000 fd:00 8389996
> /usr/lib64/libc-2.17.so
> 2b8f3b6af000-2b8f3b6b1000 rw-p 001c6000 fd:00 8389996
> /usr/lib64/libc-2.17.so
> 2b8f3b6b1000-2b8f3b6b6000 rw-p 00000000 00:00 0
> 2b8f3b6b6000-2b8f3b6b8000 r-xp 00000000 fd:00 8390002
> /usr/lib64/libdl-2.17.so
> 2b8f3b6b8000-2b8f3b8b8000 ---p 00002000 fd:00 8390002
> /usr/lib64/libdl-2.17.so
> 2b8f3b8b8000-2b8f3b8b9000 r--p 00002000 fd:00 8390002
> /usr/lib64/libdl-2.17.so
> 2b8f3b8b9000-2b8f3b8ba000 rw-p 00003000 fd:00 8390002
> /usr/lib64/libdl-2.17.so
> 2b8f3b8ba000-2b8f3b8d1000 r-xp 00000000 fd:00 8390022
> /usr/lib64/libpthread-2.17.so
> 2b8f3b8d1000-2b8f3bad0000 ---p 00017000 fd:00 8390022
> /usr/lib64/libpthread-2.17.so
> 2b8f3bad0000-2b8f3bad1000 r--p 00016000 fd:00 8390022
> /usr/lib64/libpthread-2.17.so
> 2b8f3bad1000-2b8f3bad2000 rw-p 00017000 fd:00 8390022
> /usr/lib64/libpthread-2.17.so
> 2b8f3bad2000-2b8f3bad6000 rw-p 00000000 00:00 0
> 2b8f3bad6000-2b8f3baf6000 r-xp 00000000 00:33 18140735
>  /net/project/ukmo/rhel6/meteval/lib/
> libhdf5_hl.so.100.1.0run_mode_BIO_wrapper_by_day.sh: line 39: 54415
> Aborted                 mode
> ${FCSTDIR}/MetO-NWS-BIO-dm-CPWC_Forecast_24_${YYYY}-${MM}-${DD}.nc
> ${OBSDIR}/MetO-NWS-BIO-dm-CPWC_Analysis_${YYYY}-${MM}-${DD}.nc
> ${RUNDIR}/config/MODEConfig_BIO_quilt_mergeboth -outdir ${OUTDIR}
>
> --
> Dr Marion Mittermaier     Manager: Model diagnostics and novel
verification
>
> Met Office   FitzRoy Road   Exeter   EX1 3PB   United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604  Fax: +44 (0)1392
885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>  http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
Verification
> Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
>
>
>

------------------------------------------------

More information about the Met_help mailing list