[Met_help] [rt.rap.ucar.edu #88127] History for RE: GPM file read failure in pcp_combine part 2

John Halley Gotway via RT met_help at ucar.edu
Tue Jul 9 12:07:13 MDT 2019 

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi John,

I’ve finally got around to testing this with an updated MET8.0 build.
Unfortunately, I still get a Memory fault error.
$ plot_data_plane -v 4 PYTHON_NUMPY plot.ps \
   'name="read_IMERG.py 3B-HHR-L.MS.MRG.3IMERG.20180928-S233000-E235959.1410.V05B.RT-H5 IRprecipitation";'
DEBUG 1: Opening data file: PYTHON_NUMPY
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Python_Numpy".
Memory fault(coredump)

However, with a different raw file I get a different error:
$ plot_data_plane PYTHON_NUMPY plot.ps \
  'name="read_IMERG.py 3B-HHR-L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5 IRprecipitation;"'
ERROR  :
ERROR  : yyerror() -> syntax error in file "/tmp/config_14067_0_.temp"
ERROR  :
ERROR  :    line   = 2
ERROR  :
ERROR  :    column = 1
ERROR  :
ERROR  :    text   = "(nul)"
ERROR  :
ERROR  :
ERROR  : (nul)
ERROR  : (nul)
ERROR  :

Does this provide any extra clues? If you want me to pass you the data file, let me know.

I’ve been told that the MET8.0 build was rebuilt with the latest patches.
Is there an easy way to check that the patches have been applied and work properly?

Kind Regards,
Rachel

Dr Rachel North, Research Scientist
Met Office, FitzRoy Road, Exeter,  EX1 3PB, United Kingdom
Tel: +44 (0)1392 884332  Fax: +44 (0)1392 885681
rachel.north at metoffice.gov.uk http://www.metoffice.gov.uk
From: Mittermaier, Marion
Sent: 11 October 2018 08:07
To: John Halley Gotway <johnhg at ucar.edu>; North, Rachel <rachel.north at metoffice.gov.uk>
Cc: met_help at ucar.edu
Subject: RE: GPM file read failure in pcp_combine part 2

Thanks John.

I’ve added Rachel to the list.

If we can’t read the GPM data directly then I guess we should resolve the core dump from the netcdf files I also sent. They’re hopefully more likely to be resolvable. One of the files I sent was created by reading a bunch of the hdf5 files into IRIS, adding them up, and writing out a netcdf file. All the netcdf files we have created using IRIS have caused MET to fail so there is something in the metadata that MET doesn’t like.

Marion
From: John Halley Gotway <johnhg at ucar.edu<mailto:johnhg at ucar.edu>>
Sent: 10 October 2018 22:27
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
Cc: met_help at ucar.edu<mailto:met_help at ucar.edu>
Subject: Re: GPM file read failure in pcp_combine part 2

Hi Marion,

I wanted to let you know that I've been working on this data this week.  We've been ironing out a few wrinkles in the MET calling Python arena this week.

I'm using the attached script for testing.  It reads the data fine when I run it through python on the command line:
   python read_IMERG.py 3B-HHR-L.MS.MRG.3IMERG.20180928-S233000-E235959.1410.V05B.RT-H5 IRprecipitation

However, when I run the script through MET, it aborts:
   plot_data_plane PYTHON_NUMPY plot.ps<http://plot.ps> \
   'name="read_IMERG.py 3B-HHR-L.MS.MRG.3IMERG.20180928-S233000-E235959.1410.V05B.RT-H5 IRprecipitation";'

And it prints a warning about a mismatch between the versions of the HDF5 header and libraries:
Headers are 1.10.2, library is 1.8.14

The instance of MET I'm running links against HDF5 since NetCDF4 is built on top of HDF5.  My python script loads the h5py module.  And that's where this version discrepancy occurs... MET is compiled using HDF5 version 1.8.14 while (presumably) the python HDF5 module is compiled using 1.10.2.

So that's where I'm stuck.  Thought I'd see if you have any better luck running this script than it do.

Thanks,
John


On Fri, Oct 5, 2018 at 7:25 AM Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>> wrote:
Hi,

The files we download from NASA are in hdf5 format. I’ve attached a raw file, which is half-hourly. We process these into hourly and other lengths using IRIS (Met Office Python library available on GitHub). We believe IRIS produces CF-compliant netcdf files.

It would be nice to use pcp_combine on the 30 min hdf5 files to cut out the intermediate step.

Regards
Marion

--
Dr Marion Mittermaier       Manager: Model Diagnostics and Novel Verification Methods
Met Office FitzRoy Rd Exeter EX1 3PB United Kingdom
Tel +44 (0)1392 884830
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
http://www.metoffice.gov.uk<https://mail.cmpd1.metoffice.gov.uk/owa/redir.aspx?C=AKHoDb1piEyUp0_4-dOB-Dx8qHdiINAIaqQAFV6wmm8Xr4GUlN7JVn45tRMjN0WAQ09F11kk8qY.&URL=http%3a%2f%2fwww.metoffice.gov.uk>

http://www.metoffice.gov.uk/research/people/marion-mittermaier<https://mail.cmpd1.metoffice.gov.uk/owa/redir.aspx?C=AKHoDb1piEyUp0_4-dOB-Dx8qHdiINAIaqQAFV6wmm8Xr4GUlN7JVn45tRMjN0WAQ09F11kk8qY.&URL=http%3a%2f%2fwww.metoffice.gov.uk%2fresearch%2fpeople%2fmarion-mittermaier>



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: RE: GPM file read failure in pcp_combine part 2
From: John Halley Gotway
Time: Wed Dec 12 09:35:06 2018

Rachel,

Good news, this is a simple syntax error.  Thanks for sending the
command
you used:

*plot_data_plane PYTHON_NUMPY plot.ps <http://plot.ps> \*

*  'name="read_IMERG.py
3B-HHR-L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5
IRprecipitation;"'*


The string inside the double quotes should be terminated with a
semi-colon.  You have *;* followed by *"*, but those need to be
swapped,
like this...


*plot_data_plane PYTHON_NUMPY plot.ps <http://plot.ps> \*

*  'name="read_IMERG.py
3B-HHR-L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5
IRprecipitation";'*


Please let me know if that yields better results.


Thanks,

John

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88127] RE: GPM file read failure in pcp_combine part 2
From: North, Rachel
Time: Wed Dec 12 09:44:01 2018

Thanks for spotting that John,

Changing the syntax means that both GPM files now yield a consistent
result, but that result is a memory fault error message.

$ plot_data_plane PYTHON_NUMPY plot.ps
'name="/home/h02/fryl/VERIFICATION/MET/read_IMERG.py 3B-HHR-
L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5
IRprecipitation";'
DEBUG 1: Opening data file: PYTHON_NUMPY
Memory fault(coredump)

Regards,
Rachel

Dr Rachel North, Research Scientist
Met Office, FitzRoy Road, Exeter,  EX1 3PB, United Kingdom
Tel: +44 (0)1392 884332  Fax: +44 (0)1392 885681
rachel.north at metoffice.gov.uk http://www.metoffice.gov.uk
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 12 December 2018 16:35
To: North, Rachel <rachel.north at metoffice.gov.uk>
Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #88127] RE: GPM file read failure in
pcp_combine part 2

Rachel,

Good news, this is a simple syntax error.  Thanks for sending the
command you used:

*plot_data_plane PYTHON_NUMPY plot.ps <http://plot.ps> \*

*  'name="read_IMERG.py
3B-HHR-L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5
IRprecipitation;"'*


The string inside the double quotes should be terminated with a semi-
colon.  You have *;* followed by *"*, but those need to be swapped,
like this...


*plot_data_plane PYTHON_NUMPY plot.ps <http://plot.ps> \*

*  'name="read_IMERG.py
3B-HHR-L.MS.MRG.3IMERG.20181020-S100000-E102959.0600.V05B.RT-H5
IRprecipitation";'*


Please let me know if that yields better results.


Thanks,

John




------------------------------------------------

More information about the Met_help mailing list