[Met_help] [rt.rap.ucar.edu #87015] History for MET5.2 question: Dealing with projected inputs
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:06:48 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET-HELP:
Sorry to bug you a question, but I am trying to use MODE and MTD with some model inputs that have unusual map projections (i.e. the "poles" not being where they are usually; or "rotated pole" in the way we called them here). MODE, MTD and plot_data_plane do not seem wants to read them even after xy dependent latitude and longitude information are added to the netcdf file. It gives errors like:
ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc" not a valid data file
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out projection from information in netCDF file.
In the past, we got around this problem by regridding the data to regular lat-lon; due to scientific and technical reasons, we wish to avoid that. Are there any solutions to directly work with the projected data?
Thank you and Best Regards,
Steven
=============================================================
Steven C Chan, Visiting Scientist from Newcastle University
Met Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB United Kingdom
Tel: +44 (0)1392 886153
Email: steven.chan at metoffice.gov.uk Website: www.metoffice.gov.uk
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: John Halley Gotway
Time: Tue Sep 18 09:35:27 2018
Steven,
I see that MET is not able to handle the projection on which your data
lives. We are currently working on the next version of MET, met-8.0.
We've passed the code freeze date and are now in testing. One of the
features we added was support for rotated lat/lon grids. Based on
your
description, it sounds like that's what you have. met-8.0 is
currently
able to read rotated lat/lon data from GRIB2 files, but we've been
looking
for other rotated lat/lon projection examples. However, I suspect we
may
run into other technical issues with your data. Do your NetCDF files
follow the CF (climate-forecast) convention, which MET supports?
Another enhancement in met-8.0 is support for calling python scripts
to
read gridded data. If MET isn't able to read the particular flavor of
NetCDF you have, you could write a python script to read the data,
populate
a NumPy array, and dictionary of metadata. When you run MODE, instead
of
telling it to read a NetCDF input file, you can tell it to execute a
python
script. Then it reads the gridded field of data from that python
script in
memory. We hope that this feature will make it possible to run MET on
a
wider variety of data formats.
Can you please send us a sample rotated lat/lon NetCDF file so that we
can
take a look? Here's how you can post data to our anonymous ftp site:
https://dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John Halley Gotway
On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> Transaction: Ticket created by steven.chan at metoffice.gov.uk
> Queue: met_help
> Subject: MET5.2 question: Dealing with projected inputs
> Owner: Nobody
> Requestors: steven.chan at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
>
> Dear MET-HELP:
>
> Sorry to bug you a question, but I am trying to use MODE and MTD
with some
> model inputs that have unusual map projections (i.e. the "poles" not
being
> where they are usually; or "rotated pole" in the way we called them
here).
> MODE, MTD and plot_data_plane do not seem wants to read them even
after xy
> dependent latitude and longitude information are added to the netcdf
file.
> It gives errors like:
>
> ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc" not a
> valid data file
> ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> from information in netCDF file.
>
> In the past, we got around this problem by regridding the data to
regular
> lat-lon; due to scientific and technical reasons, we wish to avoid
that.
> Are there any solutions to directly work with the projected data?
>
> Thank you and Best Regards,
> Steven
>
> =============================================================
> Steven C Chan, Visiting Scientist from Newcastle University
> Met Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> Kingdom
> Tel: +44 (0)1392 886153
> Email: steven.chan at metoffice.gov.uk Website: www.metoffice.gov.uk
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Tue Sep 18 09:52:08 2018
Dear John:
Thank you very much for your reply. Yes, it is rotated lat-lon grid,
and the files are created supposedly according to CF convention using
Met Office python libraries; hence reading the data with the help with
Python will be extremely helpful function.
I will need to confirm with line manager if it is possible to send a
data sample as Met Office has a lot of rules in sharing data. If that
is not possible, I may be able to I can get you fictitious data with
the right coordinate information. I will get back to you with this
before the end of this week (line manager is out of town right now).
Best Regards,
Steven Chan
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 18 September 2018 16:35
To: Chan, Steven <steven.chan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
I see that MET is not able to handle the projection on which your data
lives. We are currently working on the next version of MET, met-8.0.
We've passed the code freeze date and are now in testing. One of the
features we added was support for rotated lat/lon grids. Based on
your description, it sounds like that's what you have. met-8.0 is
currently able to read rotated lat/lon data from GRIB2 files, but
we've been looking for other rotated lat/lon projection examples.
However, I suspect we may run into other technical issues with your
data. Do your NetCDF files follow the CF (climate-forecast)
convention, which MET supports?
Another enhancement in met-8.0 is support for calling python scripts
to read gridded data. If MET isn't able to read the particular flavor
of NetCDF you have, you could write a python script to read the data,
populate a NumPy array, and dictionary of metadata. When you run
MODE, instead of telling it to read a NetCDF input file, you can tell
it to execute a python script. Then it reads the gridded field of
data from that python script in memory. We hope that this feature
will make it possible to run MET on a wider variety of data formats.
Can you please send us a sample rotated lat/lon NetCDF file so that we
can take a look? Here's how you can post data to our anonymous ftp
site:
https://dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John Halley Gotway
On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> Transaction: Ticket created by steven.chan at metoffice.gov.uk
> Queue: met_help
> Subject: MET5.2 question: Dealing with projected inputs
> Owner: Nobody
> Requestors: steven.chan at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> >
>
>
> Dear MET-HELP:
>
> Sorry to bug you a question, but I am trying to use MODE and MTD
with
> some model inputs that have unusual map projections (i.e. the
"poles"
> not being where they are usually; or "rotated pole" in the way we
called them here).
> MODE, MTD and plot_data_plane do not seem wants to read them even
> after xy dependent latitude and longitude information are added to
the netcdf file.
> It gives errors like:
>
> ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc" not a
> valid data file ERROR : NcCfFile::read_netcdf_grid() -> Couldn't
> figure out projection from information in netCDF file.
>
> In the past, we got around this problem by regridding the data to
> regular lat-lon; due to scientific and technical reasons, we wish to
avoid that.
> Are there any solutions to directly work with the projected data?
>
> Thank you and Best Regards,
> Steven
>
> =============================================================
> Steven C Chan, Visiting Scientist from Newcastle University Met
Office
> Hadley Centre FitzRoy Road Exeter Devon EX1 3PB United Kingdom
> Tel: +44 (0)1392 886153
> Email: steven.chan at metoffice.gov.uk Website: www.metoffice.gov.uk
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: John Halley Gotway
Time: Tue Sep 18 11:27:15 2018
Steven,
OK, please let us know. The development version for met-8.0 doesn't
currently have support for parsing rotated lat/lon grids from CF-
compliant
NetCDF files since we don't have any examples. The GRIB2 example we
have
is from the US Air Force's usage of UK Met Office unified model...
which
they call GALWEM.
If we can get sample data soon enough, hopefully we can get it added
prior
to the met-8.0 release.
Thanks,
John
On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> Thank you very much for your reply. Yes, it is rotated lat-lon grid,
and
> the files are created supposedly according to CF convention using
Met
> Office python libraries; hence reading the data with the help with
Python
> will be extremely helpful function.
>
> I will need to confirm with line manager if it is possible to send a
data
> sample as Met Office has a lot of rules in sharing data. If that is
not
> possible, I may be able to I can get you fictitious data with the
right
> coordinate information. I will get back to you with this before the
end of
> this week (line manager is out of town right now).
>
> Best Regards,
> Steven Chan
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 18 September 2018 16:35
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> I see that MET is not able to handle the projection on which your
data
> lives. We are currently working on the next version of MET, met-
8.0.
> We've passed the code freeze date and are now in testing. One of
the
> features we added was support for rotated lat/lon grids. Based on
your
> description, it sounds like that's what you have. met-8.0 is
currently
> able to read rotated lat/lon data from GRIB2 files, but we've been
looking
> for other rotated lat/lon projection examples. However, I suspect
we may
> run into other technical issues with your data. Do your NetCDF
files
> follow the CF (climate-forecast) convention, which MET supports?
>
> Another enhancement in met-8.0 is support for calling python scripts
to
> read gridded data. If MET isn't able to read the particular flavor
of
> NetCDF you have, you could write a python script to read the data,
populate
> a NumPy array, and dictionary of metadata. When you run MODE,
instead of
> telling it to read a NetCDF input file, you can tell it to execute a
python
> script. Then it reads the gridded field of data from that python
script in
> memory. We hope that this feature will make it possible to run MET
on a
> wider variety of data formats.
>
> Can you please send us a sample rotated lat/lon NetCDF file so that
we can
> take a look? Here's how you can post data to our anonymous ftp
site:
> https://dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John Halley Gotway
>
>
>
> On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > Queue: met_help
> > Subject: MET5.2 question: Dealing with projected inputs
> > Owner: Nobody
> > Requestors: steven.chan at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > >
> >
> >
> > Dear MET-HELP:
> >
> > Sorry to bug you a question, but I am trying to use MODE and MTD
with
> > some model inputs that have unusual map projections (i.e. the
"poles"
> > not being where they are usually; or "rotated pole" in the way we
called
> them here).
> > MODE, MTD and plot_data_plane do not seem wants to read them even
> > after xy dependent latitude and longitude information are added to
the
> netcdf file.
> > It gives errors like:
> >
> > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc" not
a
> > valid data file ERROR : NcCfFile::read_netcdf_grid() -> Couldn't
> > figure out projection from information in netCDF file.
> >
> > In the past, we got around this problem by regridding the data to
> > regular lat-lon; due to scientific and technical reasons, we wish
to
> avoid that.
> > Are there any solutions to directly work with the projected data?
> >
> > Thank you and Best Regards,
> > Steven
> >
> > =============================================================
> > Steven C Chan, Visiting Scientist from Newcastle University Met
Office
> > Hadley Centre FitzRoy Road Exeter Devon EX1 3PB United
Kingdom
> > Tel: +44 (0)1392 886153
> > Email: steven.chan at metoffice.gov.uk Website: www.metoffice.gov.uk
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Thu Sep 20 03:11:39 2018
Dear John:
Line manager said it is fine. I am just waiting for Sys Admin to
actually make that possible through the Met Office firewall (a
technical issue). I will try to get that resolved before the end of
this week.
Thank you
Steven
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 18 September 2018 18:27
To: Chan, Steven <steven.chan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
OK, please let us know. The development version for met-8.0 doesn't
currently have support for parsing rotated lat/lon grids from CF-
compliant NetCDF files since we don't have any examples. The GRIB2
example we have is from the US Air Force's usage of UK Met Office
unified model... which they call GALWEM.
If we can get sample data soon enough, hopefully we can get it added
prior to the met-8.0 release.
Thanks,
John
On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> Thank you very much for your reply. Yes, it is rotated lat-lon grid,
> and the files are created supposedly according to CF convention
using
> Met Office python libraries; hence reading the data with the help
with
> Python will be extremely helpful function.
>
> I will need to confirm with line manager if it is possible to send a
> data sample as Met Office has a lot of rules in sharing data. If
that
> is not possible, I may be able to I can get you fictitious data with
> the right coordinate information. I will get back to you with this
> before the end of this week (line manager is out of town right now).
>
> Best Regards,
> Steven Chan
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 18 September 2018 16:35
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> I see that MET is not able to handle the projection on which your
data
> lives. We are currently working on the next version of MET, met-
8.0.
> We've passed the code freeze date and are now in testing. One of
the
> features we added was support for rotated lat/lon grids. Based on
> your description, it sounds like that's what you have. met-8.0 is
> currently able to read rotated lat/lon data from GRIB2 files, but
> we've been looking for other rotated lat/lon projection examples.
> However, I suspect we may run into other technical issues with your
> data. Do your NetCDF files follow the CF (climate-forecast)
convention, which MET supports?
>
> Another enhancement in met-8.0 is support for calling python scripts
> to read gridded data. If MET isn't able to read the particular
flavor
> of NetCDF you have, you could write a python script to read the
data,
> populate a NumPy array, and dictionary of metadata. When you run
> MODE, instead of telling it to read a NetCDF input file, you can
tell
> it to execute a python script. Then it reads the gridded field of
> data from that python script in memory. We hope that this feature
> will make it possible to run MET on a wider variety of data formats.
>
> Can you please send us a sample rotated lat/lon NetCDF file so that
we
> can take a look? Here's how you can post data to our anonymous ftp
site:
> https://dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John Halley Gotway
>
>
>
> On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > Queue: met_help
> > Subject: MET5.2 question: Dealing with projected inputs
> > Owner: Nobody
> > Requestors: steven.chan at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > >
> >
> >
> > Dear MET-HELP:
> >
> > Sorry to bug you a question, but I am trying to use MODE and MTD
> > with some model inputs that have unusual map projections (i.e. the
"poles"
> > not being where they are usually; or "rotated pole" in the way we
> > called
> them here).
> > MODE, MTD and plot_data_plane do not seem wants to read them even
> > after xy dependent latitude and longitude information are added to
> > the
> netcdf file.
> > It gives errors like:
> >
> > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc" not
a
> > valid data file ERROR : NcCfFile::read_netcdf_grid() -> Couldn't
> > figure out projection from information in netCDF file.
> >
> > In the past, we got around this problem by regridding the data to
> > regular lat-lon; due to scientific and technical reasons, we wish
to
> avoid that.
> > Are there any solutions to directly work with the projected data?
> >
> > Thank you and Best Regards,
> > Steven
> >
> > =============================================================
> > Steven C Chan, Visiting Scientist from Newcastle University Met
> > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB United
> > Kingdom
> > Tel: +44 (0)1392 886153
> > Email: steven.chan at metoffice.gov.uk Website: www.metoffice.gov.uk
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: John Halley Gotway
Time: Thu Sep 20 11:12:13 2018
Steven,
Thanks for the update. We're down to crunch time here getting the
met-8.0
release finalized. Once I get my hands on the sample file, I'll do my
best
to get it working.
Thanks,
John
On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> Line manager said it is fine. I am just waiting for Sys Admin to
actually
> make that possible through the Met Office firewall (a technical
issue). I
> will try to get that resolved before the end of this week.
>
> Thank you
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 18 September 2018 18:27
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> OK, please let us know. The development version for met-8.0 doesn't
> currently have support for parsing rotated lat/lon grids from CF-
compliant
> NetCDF files since we don't have any examples. The GRIB2 example we
have
> is from the US Air Force's usage of UK Met Office unified model...
which
> they call GALWEM.
>
> If we can get sample data soon enough, hopefully we can get it added
prior
> to the met-8.0 release.
>
> Thanks,
> John
>
> On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > and the files are created supposedly according to CF convention
using
> > Met Office python libraries; hence reading the data with the help
with
> > Python will be extremely helpful function.
> >
> > I will need to confirm with line manager if it is possible to send
a
> > data sample as Met Office has a lot of rules in sharing data. If
that
> > is not possible, I may be able to I can get you fictitious data
with
> > the right coordinate information. I will get back to you with this
> > before the end of this week (line manager is out of town right
now).
> >
> > Best Regards,
> > Steven Chan
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 16:35
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > I see that MET is not able to handle the projection on which your
data
> > lives. We are currently working on the next version of MET, met-
8.0.
> > We've passed the code freeze date and are now in testing. One of
the
> > features we added was support for rotated lat/lon grids. Based on
> > your description, it sounds like that's what you have. met-8.0 is
> > currently able to read rotated lat/lon data from GRIB2 files, but
> > we've been looking for other rotated lat/lon projection examples.
> > However, I suspect we may run into other technical issues with
your
> > data. Do your NetCDF files follow the CF (climate-forecast)
convention,
> which MET supports?
> >
> > Another enhancement in met-8.0 is support for calling python
scripts
> > to read gridded data. If MET isn't able to read the particular
flavor
> > of NetCDF you have, you could write a python script to read the
data,
> > populate a NumPy array, and dictionary of metadata. When you run
> > MODE, instead of telling it to read a NetCDF input file, you can
tell
> > it to execute a python script. Then it reads the gridded field of
> > data from that python script in memory. We hope that this feature
> > will make it possible to run MET on a wider variety of data
formats.
> >
> > Can you please send us a sample rotated lat/lon NetCDF file so
that we
> > can take a look? Here's how you can post data to our anonymous
ftp site:
> > https://dtcenter.org/met/users/support/met_help.php#ftp
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > Queue: met_help
> > > Subject: MET5.2 question: Dealing with projected inputs
> > > Owner: Nobody
> > > Requestors: steven.chan at metoffice.gov.uk
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > >
> > >
> > >
> > > Dear MET-HELP:
> > >
> > > Sorry to bug you a question, but I am trying to use MODE and MTD
> > > with some model inputs that have unusual map projections (i.e.
the
> "poles"
> > > not being where they are usually; or "rotated pole" in the way
we
> > > called
> > them here).
> > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > after xy dependent latitude and longitude information are added
to
> > > the
> > netcdf file.
> > > It gives errors like:
> > >
> > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
not a
> > > valid data file ERROR : NcCfFile::read_netcdf_grid() ->
Couldn't
> > > figure out projection from information in netCDF file.
> > >
> > > In the past, we got around this problem by regridding the data
to
> > > regular lat-lon; due to scientific and technical reasons, we
wish to
> > avoid that.
> > > Are there any solutions to directly work with the projected
data?
> > >
> > > Thank you and Best Regards,
> > > Steven
> > >
> > > =============================================================
> > > Steven C Chan, Visiting Scientist from Newcastle University Met
> > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > Kingdom
> > > Tel: +44 (0)1392 886153
> > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Fri Sep 21 03:39:54 2018
Dear John:
I have uploaded two files to the FTP:
/incoming/irap/met_help/StevenChan_data/schan_rotated_pole_with_different_resolution_near_boundary.nc
/incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
They are both rotated pole, but the resolution of the files are
different with one of the files have variable resolution (near the
boundaries).
Please let me know if the files are okay.
Thank you and Best Regards,
Steven
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 20 September 2018 18:12
To: Chan, Steven <steven.chan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
Thanks for the update. We're down to crunch time here getting the
met-8.0 release finalized. Once I get my hands on the sample file,
I'll do my best to get it working.
Thanks,
John
On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> Line manager said it is fine. I am just waiting for Sys Admin to
> actually make that possible through the Met Office firewall (a
> technical issue). I will try to get that resolved before the end of
this week.
>
> Thank you
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 18 September 2018 18:27
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> OK, please let us know. The development version for met-8.0 doesn't
> currently have support for parsing rotated lat/lon grids from
> CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> example we have is from the US Air Force's usage of UK Met Office
> unified model... which they call GALWEM.
>
> If we can get sample data soon enough, hopefully we can get it added
> prior to the met-8.0 release.
>
> Thanks,
> John
>
> On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > and the files are created supposedly according to CF convention
> > using Met Office python libraries; hence reading the data with the
> > help with Python will be extremely helpful function.
> >
> > I will need to confirm with line manager if it is possible to send
a
> > data sample as Met Office has a lot of rules in sharing data. If
> > that is not possible, I may be able to I can get you fictitious
data
> > with the right coordinate information. I will get back to you with
> > this before the end of this week (line manager is out of town
right now).
> >
> > Best Regards,
> > Steven Chan
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 16:35
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > I see that MET is not able to handle the projection on which your
> > data lives. We are currently working on the next version of MET,
met-8.0.
> > We've passed the code freeze date and are now in testing. One of
> > the features we added was support for rotated lat/lon grids.
Based
> > on your description, it sounds like that's what you have. met-8.0
> > is currently able to read rotated lat/lon data from GRIB2 files,
but
> > we've been looking for other rotated lat/lon projection examples.
> > However, I suspect we may run into other technical issues with
your
> > data. Do your NetCDF files follow the CF (climate-forecast)
> > convention,
> which MET supports?
> >
> > Another enhancement in met-8.0 is support for calling python
scripts
> > to read gridded data. If MET isn't able to read the particular
> > flavor of NetCDF you have, you could write a python script to read
> > the data, populate a NumPy array, and dictionary of metadata.
When
> > you run MODE, instead of telling it to read a NetCDF input file,
you
> > can tell it to execute a python script. Then it reads the gridded
> > field of data from that python script in memory. We hope that
this
> > feature will make it possible to run MET on a wider variety of
data formats.
> >
> > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > we can take a look? Here's how you can post data to our anonymous
ftp site:
> > https://dtcenter.org/met/users/support/met_help.php#ftp
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > Queue: met_help
> > > Subject: MET5.2 question: Dealing with projected inputs
> > > Owner: Nobody
> > > Requestors: steven.chan at metoffice.gov.uk
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > >
> > >
> > >
> > > Dear MET-HELP:
> > >
> > > Sorry to bug you a question, but I am trying to use MODE and MTD
> > > with some model inputs that have unusual map projections (i.e.
the
> "poles"
> > > not being where they are usually; or "rotated pole" in the way
we
> > > called
> > them here).
> > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > after xy dependent latitude and longitude information are added
to
> > > the
> > netcdf file.
> > > It gives errors like:
> > >
> > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
not
> > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > Couldn't figure out projection from information in netCDF file.
> > >
> > > In the past, we got around this problem by regridding the data
to
> > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > to
> > avoid that.
> > > Are there any solutions to directly work with the projected
data?
> > >
> > > Thank you and Best Regards,
> > > Steven
> > >
> > > =============================================================
> > > Steven C Chan, Visiting Scientist from Newcastle University Met
> > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > Kingdom
> > > Tel: +44 (0)1392 886153
> > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: John Halley Gotway
Time: Fri Sep 21 12:30:00 2018
Steven,
Thanks for sending the sample data. As you already know, I confirmed
that
MET can't read these data files as-is. I get the following error from
plot_data_plane:
plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
'name="precipitation_flux"; level="(0,*,*)";'
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
from information in netCDF file.
So we need to add support for reading the rotated lat/lon projection
information from CF-compliant NetCDF files. I was keeping my fingers
crossed that it'd be very simple to do, but it is not. It'll take
some
effort to make sure we do it correctly. And with our release next
week, we
don't have enough time to fit it in. But I did create a development
ticket
for this for the next release.
However, MET will still not be able to read the variable mesh lat/lon
data. We use the grid/projection information from the input data
files to
instantiate a regular grid in the MET libraries. That enables MET to
convert from grid x/y to lat/lon and vice-versa. And for this reason,
MET
doesn't currently support a variable mesh. Its possible that could
change
in the future depending on requirements from our funding agencies.
met-8.0 is coming out next week and includes support for calling a
python
2.7 script... and we can read rotated lat/lon data down that route. I
ran
the following command using the development version of MET:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
schan_eurocordex.nc precipitation_flux";'
In this command, I have...
(1) Replaced the filename (schan_eurocordex.nc) with the constant
"PYTHON_NUMPY".
(2) Set 'name="...";' where "..." is the python script with associated
command line arguments to be run.
I've attached the python script to illustrate how it's reading the
first
time-slice of the gridded data and setting up the metadata. We would
obviously want a more robust python script than this simple one.
And I've attached the resulting plot to see if MET is putting the data
at
the right spot on earth. Looks reasonable to me.
This demonstrates that there is a way forward with met-8.0 until we
get
support for rotated lat/lon projections in CF-compliant files added.
Hope this helps.
Thanks,
John
On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> I have uploaded two files to the FTP:
>
> /incoming/irap/met_help/StevenChan_data/
> schan_rotated_pole_with_different_resolution_near_boundary.nc
> /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
>
> They are both rotated pole, but the resolution of the files are
different
> with one of the files have variable resolution (near the
boundaries).
>
> Please let me know if the files are okay.
>
> Thank you and Best Regards,
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 20 September 2018 18:12
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> Thanks for the update. We're down to crunch time here getting the
met-8.0
> release finalized. Once I get my hands on the sample file, I'll do
my best
> to get it working.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Line manager said it is fine. I am just waiting for Sys Admin to
> > actually make that possible through the Met Office firewall (a
> > technical issue). I will try to get that resolved before the end
of this
> week.
> >
> > Thank you
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 18:27
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > OK, please let us know. The development version for met-8.0
doesn't
> > currently have support for parsing rotated lat/lon grids from
> > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > example we have is from the US Air Force's usage of UK Met Office
> > unified model... which they call GALWEM.
> >
> > If we can get sample data soon enough, hopefully we can get it
added
> > prior to the met-8.0 release.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > and the files are created supposedly according to CF convention
> > > using Met Office python libraries; hence reading the data with
the
> > > help with Python will be extremely helpful function.
> > >
> > > I will need to confirm with line manager if it is possible to
send a
> > > data sample as Met Office has a lot of rules in sharing data. If
> > > that is not possible, I may be able to I can get you fictitious
data
> > > with the right coordinate information. I will get back to you
with
> > > this before the end of this week (line manager is out of town
right
> now).
> > >
> > > Best Regards,
> > > Steven Chan
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 16:35
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > I see that MET is not able to handle the projection on which
your
> > > data lives. We are currently working on the next version of
MET,
> met-8.0.
> > > We've passed the code freeze date and are now in testing. One
of
> > > the features we added was support for rotated lat/lon grids.
Based
> > > on your description, it sounds like that's what you have. met-
8.0
> > > is currently able to read rotated lat/lon data from GRIB2 files,
but
> > > we've been looking for other rotated lat/lon projection
examples.
> > > However, I suspect we may run into other technical issues with
your
> > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > convention,
> > which MET supports?
> > >
> > > Another enhancement in met-8.0 is support for calling python
scripts
> > > to read gridded data. If MET isn't able to read the particular
> > > flavor of NetCDF you have, you could write a python script to
read
> > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > you run MODE, instead of telling it to read a NetCDF input file,
you
> > > can tell it to execute a python script. Then it reads the
gridded
> > > field of data from that python script in memory. We hope that
this
> > > feature will make it possible to run MET on a wider variety of
data
> formats.
> > >
> > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > we can take a look? Here's how you can post data to our
anonymous ftp
> site:
> > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > Owner: Nobody
> > > > Requestors: steven.chan at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > >
> > > >
> > > >
> > > > Dear MET-HELP:
> > > >
> > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > with some model inputs that have unusual map projections (i.e.
the
> > "poles"
> > > > not being where they are usually; or "rotated pole" in the way
we
> > > > called
> > > them here).
> > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > after xy dependent latitude and longitude information are
added to
> > > > the
> > > netcdf file.
> > > > It gives errors like:
> > > >
> > > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
not
> > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > Couldn't figure out projection from information in netCDF
file.
> > > >
> > > > In the past, we got around this problem by regridding the data
to
> > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > to
> > > avoid that.
> > > > Are there any solutions to directly work with the projected
data?
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > =============================================================
> > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > Kingdom
> > > > Tel: +44 (0)1392 886153
> > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Mon Sep 24 10:47:14 2018
Dear John:
It is good to hear MET8.0 coming very soon with Python support. I
guess the near-term challenge will be getting it compiled. Anything
that deals with netCDF libraries can become tricky. The version I
personally use is relatively old (5.2), and it is due for an upgrade.
Thank you very much for all the help again.
If there is something I can help getting rotated pole to work in the
future, please let me know.
Steven
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 21 September 2018 19:30
To: Chan, Steven <steven.chan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
Thanks for sending the sample data. As you already know, I confirmed
that MET can't read these data files as-is. I get the following error
from
plot_data_plane:
plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
'name="precipitation_flux"; level="(0,*,*)";'
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection from information in netCDF file.
So we need to add support for reading the rotated lat/lon projection
information from CF-compliant NetCDF files. I was keeping my fingers
crossed that it'd be very simple to do, but it is not. It'll take
some effort to make sure we do it correctly. And with our release
next week, we don't have enough time to fit it in. But I did create a
development ticket for this for the next release.
However, MET will still not be able to read the variable mesh lat/lon
data. We use the grid/projection information from the input data
files to instantiate a regular grid in the MET libraries. That
enables MET to convert from grid x/y to lat/lon and vice-versa. And
for this reason, MET doesn't currently support a variable mesh. Its
possible that could change in the future depending on requirements
from our funding agencies.
met-8.0 is coming out next week and includes support for calling a
python
2.7 script... and we can read rotated lat/lon data down that route. I
ran the following command using the development version of MET:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py schan_eurocordex.nc precipitation_flux";'
In this command, I have...
(1) Replaced the filename (schan_eurocordex.nc) with the constant
"PYTHON_NUMPY".
(2) Set 'name="...";' where "..." is the python script with associated
command line arguments to be run.
I've attached the python script to illustrate how it's reading the
first time-slice of the gridded data and setting up the metadata. We
would obviously want a more robust python script than this simple one.
And I've attached the resulting plot to see if MET is putting the data
at the right spot on earth. Looks reasonable to me.
This demonstrates that there is a way forward with met-8.0 until we
get support for rotated lat/lon projections in CF-compliant files
added.
Hope this helps.
Thanks,
John
On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> I have uploaded two files to the FTP:
>
> /incoming/irap/met_help/StevenChan_data/
> schan_rotated_pole_with_different_resolution_near_boundary.nc
> /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
>
> They are both rotated pole, but the resolution of the files are
> different with one of the files have variable resolution (near the
boundaries).
>
> Please let me know if the files are okay.
>
> Thank you and Best Regards,
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 20 September 2018 18:12
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> Thanks for the update. We're down to crunch time here getting the
> met-8.0 release finalized. Once I get my hands on the sample file,
> I'll do my best to get it working.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Line manager said it is fine. I am just waiting for Sys Admin to
> > actually make that possible through the Met Office firewall (a
> > technical issue). I will try to get that resolved before the end
of
> > this
> week.
> >
> > Thank you
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 18:27
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > OK, please let us know. The development version for met-8.0
doesn't
> > currently have support for parsing rotated lat/lon grids from
> > CF-compliant NetCDF files since we don't have any examples. The
> > GRIB2 example we have is from the US Air Force's usage of UK Met
> > Office unified model... which they call GALWEM.
> >
> > If we can get sample data soon enough, hopefully we can get it
added
> > prior to the met-8.0 release.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Thank you very much for your reply. Yes, it is rotated lat-lon
> > > grid, and the files are created supposedly according to CF
> > > convention using Met Office python libraries; hence reading the
> > > data with the help with Python will be extremely helpful
function.
> > >
> > > I will need to confirm with line manager if it is possible to
send
> > > a data sample as Met Office has a lot of rules in sharing data.
If
> > > that is not possible, I may be able to I can get you fictitious
> > > data with the right coordinate information. I will get back to
you
> > > with this before the end of this week (line manager is out of
town
> > > right
> now).
> > >
> > > Best Regards,
> > > Steven Chan
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 16:35
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
> > > with projected inputs
> > >
> > > Steven,
> > >
> > > I see that MET is not able to handle the projection on which
your
> > > data lives. We are currently working on the next version of
MET,
> met-8.0.
> > > We've passed the code freeze date and are now in testing. One
of
> > > the features we added was support for rotated lat/lon grids.
> > > Based on your description, it sounds like that's what you have.
> > > met-8.0 is currently able to read rotated lat/lon data from
GRIB2
> > > files, but we've been looking for other rotated lat/lon
projection examples.
> > > However, I suspect we may run into other technical issues with
> > > your data. Do your NetCDF files follow the CF (climate-
forecast)
> > > convention,
> > which MET supports?
> > >
> > > Another enhancement in met-8.0 is support for calling python
> > > scripts to read gridded data. If MET isn't able to read the
> > > particular flavor of NetCDF you have, you could write a python
> > > script to read the data, populate a NumPy array, and dictionary
of
> > > metadata. When you run MODE, instead of telling it to read a
> > > NetCDF input file, you can tell it to execute a python script.
> > > Then it reads the gridded field of data from that python script
in
> > > memory. We hope that this feature will make it possible to run
> > > MET on a wider variety of data
> formats.
> > >
> > > Can you please send us a sample rotated lat/lon NetCDF file so
> > > that we can take a look? Here's how you can post data to our
> > > anonymous ftp
> site:
> > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > Owner: Nobody
> > > > Requestors: steven.chan at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > >
> > > >
> > > >
> > > > Dear MET-HELP:
> > > >
> > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > with some model inputs that have unusual map projections (i.e.
> > > > the
> > "poles"
> > > > not being where they are usually; or "rotated pole" in the way
> > > > we called
> > > them here).
> > > > MODE, MTD and plot_data_plane do not seem wants to read them
> > > > even after xy dependent latitude and longitude information are
> > > > added to the
> > > netcdf file.
> > > > It gives errors like:
> > > >
> > > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
> > > > not a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > Couldn't figure out projection from information in netCDF
file.
> > > >
> > > > In the past, we got around this problem by regridding the data
> > > > to regular lat-lon; due to scientific and technical reasons,
we
> > > > wish to
> > > avoid that.
> > > > Are there any solutions to directly work with the projected
data?
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > =============================================================
> > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
> > > > United Kingdom
> > > > Tel: +44 (0)1392 886153
> > > > Email: steven.chan at metoffice.gov.uk Website:
> > > > www.metoffice.gov.uk
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Thu Apr 11 07:09:11 2019
Dear John:
Please forgive me for the number of emails. This is about the original
email you sent to me last year about rotated pole issue. In fact the
question has nothing to do with rotated pole but with the below:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
schan_eurocordex.nc precipitation_flux";'
For some MET tools, more than one input files are required (for
example "mtd"). How would the Python interface work under that
particular situation? In other words, how would one actually loop
through a list of files?
Particularly for mtd: -fcst/-single/-obs must be followed by a list of
files or a file with list of files. If I understood the manual
correctly, I interpreted that as repeating the string "PYTHON_NUMPY",
so there will be a need of linking a particular "PYTHON_NUMPY" to a
particular file. Please correct me if I am wrong.
Best Regards
Steven Chan
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 21 September 2018 19:30:00
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
Thanks for sending the sample data. As you already know, I confirmed
that
MET can't read these data files as-is. I get the following error from
plot_data_plane:
plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
'name="precipitation_flux"; level="(0,*,*)";'
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
from information in netCDF file.
So we need to add support for reading the rotated lat/lon projection
information from CF-compliant NetCDF files. I was keeping my fingers
crossed that it'd be very simple to do, but it is not. It'll take
some
effort to make sure we do it correctly. And with our release next
week, we
don't have enough time to fit it in. But I did create a development
ticket
for this for the next release.
However, MET will still not be able to read the variable mesh lat/lon
data. We use the grid/projection information from the input data
files to
instantiate a regular grid in the MET libraries. That enables MET to
convert from grid x/y to lat/lon and vice-versa. And for this reason,
MET
doesn't currently support a variable mesh. Its possible that could
change
in the future depending on requirements from our funding agencies.
met-8.0 is coming out next week and includes support for calling a
python
2.7 script... and we can read rotated lat/lon data down that route. I
ran
the following command using the development version of MET:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
schan_eurocordex.nc precipitation_flux";'
In this command, I have...
(1) Replaced the filename (schan_eurocordex.nc) with the constant
"PYTHON_NUMPY".
(2) Set 'name="...";' where "..." is the python script with associated
command line arguments to be run.
I've attached the python script to illustrate how it's reading the
first
time-slice of the gridded data and setting up the metadata. We would
obviously want a more robust python script than this simple one.
And I've attached the resulting plot to see if MET is putting the data
at
the right spot on earth. Looks reasonable to me.
This demonstrates that there is a way forward with met-8.0 until we
get
support for rotated lat/lon projections in CF-compliant files added.
Hope this helps.
Thanks,
John
On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> I have uploaded two files to the FTP:
>
> /incoming/irap/met_help/StevenChan_data/
> schan_rotated_pole_with_different_resolution_near_boundary.nc
> /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
>
> They are both rotated pole, but the resolution of the files are
different
> with one of the files have variable resolution (near the
boundaries).
>
> Please let me know if the files are okay.
>
> Thank you and Best Regards,
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 20 September 2018 18:12
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> Thanks for the update. We're down to crunch time here getting the
met-8.0
> release finalized. Once I get my hands on the sample file, I'll do
my best
> to get it working.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Line manager said it is fine. I am just waiting for Sys Admin to
> > actually make that possible through the Met Office firewall (a
> > technical issue). I will try to get that resolved before the end
of this
> week.
> >
> > Thank you
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 18:27
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > OK, please let us know. The development version for met-8.0
doesn't
> > currently have support for parsing rotated lat/lon grids from
> > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > example we have is from the US Air Force's usage of UK Met Office
> > unified model... which they call GALWEM.
> >
> > If we can get sample data soon enough, hopefully we can get it
added
> > prior to the met-8.0 release.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > and the files are created supposedly according to CF convention
> > > using Met Office python libraries; hence reading the data with
the
> > > help with Python will be extremely helpful function.
> > >
> > > I will need to confirm with line manager if it is possible to
send a
> > > data sample as Met Office has a lot of rules in sharing data. If
> > > that is not possible, I may be able to I can get you fictitious
data
> > > with the right coordinate information. I will get back to you
with
> > > this before the end of this week (line manager is out of town
right
> now).
> > >
> > > Best Regards,
> > > Steven Chan
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 16:35
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > I see that MET is not able to handle the projection on which
your
> > > data lives. We are currently working on the next version of
MET,
> met-8.0.
> > > We've passed the code freeze date and are now in testing. One
of
> > > the features we added was support for rotated lat/lon grids.
Based
> > > on your description, it sounds like that's what you have. met-
8.0
> > > is currently able to read rotated lat/lon data from GRIB2 files,
but
> > > we've been looking for other rotated lat/lon projection
examples.
> > > However, I suspect we may run into other technical issues with
your
> > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > convention,
> > which MET supports?
> > >
> > > Another enhancement in met-8.0 is support for calling python
scripts
> > > to read gridded data. If MET isn't able to read the particular
> > > flavor of NetCDF you have, you could write a python script to
read
> > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > you run MODE, instead of telling it to read a NetCDF input file,
you
> > > can tell it to execute a python script. Then it reads the
gridded
> > > field of data from that python script in memory. We hope that
this
> > > feature will make it possible to run MET on a wider variety of
data
> formats.
> > >
> > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > we can take a look? Here's how you can post data to our
anonymous ftp
> site:
> > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > Owner: Nobody
> > > > Requestors: steven.chan at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > >
> > > >
> > > >
> > > > Dear MET-HELP:
> > > >
> > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > with some model inputs that have unusual map projections (i.e.
the
> > "poles"
> > > > not being where they are usually; or "rotated pole" in the way
we
> > > > called
> > > them here).
> > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > after xy dependent latitude and longitude information are
added to
> > > > the
> > > netcdf file.
> > > > It gives errors like:
> > > >
> > > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
not
> > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > Couldn't figure out projection from information in netCDF
file.
> > > >
> > > > In the past, we got around this problem by regridding the data
to
> > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > to
> > > avoid that.
> > > > Are there any solutions to directly work with the projected
data?
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > =============================================================
> > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > Kingdom
> > > > Tel: +44 (0)1392 886153
> > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk<http://www.metoffice.gov.uk>
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Fri Apr 12 03:33:09 2019
Dear John:
One additional question:
I notice from tests that paths to Python interpreters and modules are
inconsistent. This is quite visible here because we have multiple
versions (and mutually incompatible) of Python interpreters and
modules installed that are managed using shell "module load" command.
For example:
import sys
print('Using: ',sys.executable)
Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
import csv
import numpy as np
import os
print csv
<module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
print np
<module 'numpy' from '/usr/lib64/python2.7/site-
packages/numpy/__init__.pyc'>
print os
<module 'os' from '/usr/lib64/python2.7/os.pyc'>
There is a separate installation of python modules under
/opt/scitools/environments/default_legacy/2019_02_27/; however, for
some reasons, the executable decides to go find modules from a
different place “/usr/lib64/python2.7/”. Is there a way to fix the
above? The above is actually generating errors with actual tests of
using MTD through the Python interface because the correct Python
modules cannot be found, and I tried hand-fixing sys.path but it does
not work as well.
Best Regards,
Steven
________________________________
From: Chan, Steven
Sent: 11 April 2019 14:09:04
To: met_help at ucar.edu
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Dear John:
Please forgive me for the number of emails. This is about the original
email you sent to me last year about rotated pole issue. In fact the
question has nothing to do with rotated pole but with the below:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
schan_eurocordex.nc precipitation_flux";'
For some MET tools, more than one input files are required (for
example "mtd"). How would the Python interface work under that
particular situation? In other words, how would one actually loop
through a list of files?
Particularly for mtd: -fcst/-single/-obs must be followed by a list of
files or a file with list of files. If I understood the manual
correctly, I interpreted that as repeating the string "PYTHON_NUMPY",
so there will be a need of linking a particular "PYTHON_NUMPY" to a
particular file. Please correct me if I am wrong.
Best Regards
Steven Chan
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 21 September 2018 19:30:00
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Steven,
Thanks for sending the sample data. As you already know, I confirmed
that
MET can't read these data files as-is. I get the following error from
plot_data_plane:
plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
'name="precipitation_flux"; level="(0,*,*)";'
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
from information in netCDF file.
So we need to add support for reading the rotated lat/lon projection
information from CF-compliant NetCDF files. I was keeping my fingers
crossed that it'd be very simple to do, but it is not. It'll take
some
effort to make sure we do it correctly. And with our release next
week, we
don't have enough time to fit it in. But I did create a development
ticket
for this for the next release.
However, MET will still not be able to read the variable mesh lat/lon
data. We use the grid/projection information from the input data
files to
instantiate a regular grid in the MET libraries. That enables MET to
convert from grid x/y to lat/lon and vice-versa. And for this reason,
MET
doesn't currently support a variable mesh. Its possible that could
change
in the future depending on requirements from our funding agencies.
met-8.0 is coming out next week and includes support for calling a
python
2.7 script... and we can read rotated lat/lon data down that route. I
ran
the following command using the development version of MET:
plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
schan_eurocordex.nc precipitation_flux";'
In this command, I have...
(1) Replaced the filename (schan_eurocordex.nc) with the constant
"PYTHON_NUMPY".
(2) Set 'name="...";' where "..." is the python script with associated
command line arguments to be run.
I've attached the python script to illustrate how it's reading the
first
time-slice of the gridded data and setting up the metadata. We would
obviously want a more robust python script than this simple one.
And I've attached the resulting plot to see if MET is putting the data
at
the right spot on earth. Looks reasonable to me.
This demonstrates that there is a way forward with met-8.0 until we
get
support for rotated lat/lon projections in CF-compliant files added.
Hope this helps.
Thanks,
John
On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
> I have uploaded two files to the FTP:
>
> /incoming/irap/met_help/StevenChan_data/
> schan_rotated_pole_with_different_resolution_near_boundary.nc
> /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
>
> They are both rotated pole, but the resolution of the files are
different
> with one of the files have variable resolution (near the
boundaries).
>
> Please let me know if the files are okay.
>
> Thank you and Best Regards,
> Steven
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 20 September 2018 18:12
> To: Chan, Steven <steven.chan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Steven,
>
> Thanks for the update. We're down to crunch time here getting the
met-8.0
> release finalized. Once I get my hands on the sample file, I'll do
my best
> to get it working.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > Line manager said it is fine. I am just waiting for Sys Admin to
> > actually make that possible through the Met Office firewall (a
> > technical issue). I will try to get that resolved before the end
of this
> week.
> >
> > Thank you
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 18 September 2018 18:27
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > OK, please let us know. The development version for met-8.0
doesn't
> > currently have support for parsing rotated lat/lon grids from
> > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > example we have is from the US Air Force's usage of UK Met Office
> > unified model... which they call GALWEM.
> >
> > If we can get sample data soon enough, hopefully we can get it
added
> > prior to the met-8.0 release.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > and the files are created supposedly according to CF convention
> > > using Met Office python libraries; hence reading the data with
the
> > > help with Python will be extremely helpful function.
> > >
> > > I will need to confirm with line manager if it is possible to
send a
> > > data sample as Met Office has a lot of rules in sharing data. If
> > > that is not possible, I may be able to I can get you fictitious
data
> > > with the right coordinate information. I will get back to you
with
> > > this before the end of this week (line manager is out of town
right
> now).
> > >
> > > Best Regards,
> > > Steven Chan
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 16:35
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > I see that MET is not able to handle the projection on which
your
> > > data lives. We are currently working on the next version of
MET,
> met-8.0.
> > > We've passed the code freeze date and are now in testing. One
of
> > > the features we added was support for rotated lat/lon grids.
Based
> > > on your description, it sounds like that's what you have. met-
8.0
> > > is currently able to read rotated lat/lon data from GRIB2 files,
but
> > > we've been looking for other rotated lat/lon projection
examples.
> > > However, I suspect we may run into other technical issues with
your
> > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > convention,
> > which MET supports?
> > >
> > > Another enhancement in met-8.0 is support for calling python
scripts
> > > to read gridded data. If MET isn't able to read the particular
> > > flavor of NetCDF you have, you could write a python script to
read
> > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > you run MODE, instead of telling it to read a NetCDF input file,
you
> > > can tell it to execute a python script. Then it reads the
gridded
> > > field of data from that python script in memory. We hope that
this
> > > feature will make it possible to run MET on a wider variety of
data
> formats.
> > >
> > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > we can take a look? Here's how you can post data to our
anonymous ftp
> site:
> > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > Owner: Nobody
> > > > Requestors: steven.chan at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > >
> > > >
> > > >
> > > > Dear MET-HELP:
> > > >
> > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > with some model inputs that have unusual map projections (i.e.
the
> > "poles"
> > > > not being where they are usually; or "rotated pole" in the way
we
> > > > called
> > > them here).
> > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > after xy dependent latitude and longitude information are
added to
> > > > the
> > > netcdf file.
> > > > It gives errors like:
> > > >
> > > > ERROR : plot_data_plane -> file "mi-ag192_h_precip_200001.nc"
not
> > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > Couldn't figure out projection from information in netCDF
file.
> > > >
> > > > In the past, we got around this problem by regridding the data
to
> > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > to
> > > avoid that.
> > > > Are there any solutions to directly work with the projected
data?
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > =============================================================
> > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > Kingdom
> > > > Tel: +44 (0)1392 886153
> > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk<http://www.metoffice.gov.uk>
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: John Halley Gotway
Time: Thu Apr 25 14:48:33 2019
Hi Steven,
It sounds like you're wondering how to configure your environment to
use
one version of python versus another.
While adding support for calling python scripts has solved some
problems in
MET, it has also given rise to new ones. And this is one of them!
Unfortunately, I believe this needs to be handled when compiling MET.
In
particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
MET_PYTHON_CC
Where the python header files are located, e.g.
-I<include
dir>.
Set to the output of 'python-config --cflags'.
MET_PYTHON_LD
Where the python library files are located, e.g. -L<lib
dir>,
followed by the python library names, e.g. -l<library>.
Set to
the output of 'python-config --ldflags'.
It sounds like you have a "good" version of python around and a "bad"
version of python. If the directory where MET was compiled is
available on
your system, you could check "config.log" to see how those were set at
compilation time. For example:
*grep MET_PYTHON_ config.log*
ac_cv_env_MET_PYTHON_CC_set=set
ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
ac_cv_env_MET_PYTHON_LD_set=set
ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
-lutil -lm -lpython2.7 -Xlinker -export-dynamic'
And check to see if you're compiling against your preferred version of
python.
Hope that helps.
John
On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
>
>
> One additional question:
>
> I notice from tests that paths to Python interpreters and modules
are
> inconsistent. This is quite visible here because we have multiple
versions
> (and mutually incompatible) of Python interpreters and modules
installed
> that are managed using shell "module load" command. For example:
>
>
>
> import sys
> print('Using: ',sys.executable)
> Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
>
> import csv
>
> import numpy as np
>
> import os
>
> print csv
>
> <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
>
> print np
>
> <module 'numpy' from
> '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
>
> print os
>
> <module 'os' from '/usr/lib64/python2.7/os.pyc'>
>
>
>
> There is a separate installation of python modules under
> /opt/scitools/environments/default_legacy/2019_02_27/; however, for
some
> reasons, the executable decides to go find modules from a different
place
> “/usr/lib64/python2.7/”. Is there a way to fix the above? The above
is
> actually generating errors with actual tests of using MTD through
the
> Python interface because the correct Python modules cannot be found,
and I
> tried hand-fixing sys.path but it does not work as well.
>
>
> Best Regards,
>
> Steven
>
>
>
> ________________________________
>
> From: Chan, Steven
> Sent: 11 April 2019 14:09:04
> To: met_help at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Dear John:
>
>
>
> Please forgive me for the number of emails. This is about the
original
> email you sent to me last year about rotated pole issue. In fact the
> question has nothing to do with rotated pole but with the below:
>
>
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
>
>
> For some MET tools, more than one input files are required (for
example
> "mtd"). How would the Python interface work under that particular
> situation? In other words, how would one actually loop through a
list of
> files?
>
>
>
> Particularly for mtd: -fcst/-single/-obs must be followed by a list
of
> files or a file with list of files. If I understood the manual
correctly, I
> interpreted that as repeating the string "PYTHON_NUMPY", so there
will be a
> need of linking a particular "PYTHON_NUMPY" to a particular file.
Please
> correct me if I am wrong.
>
>
>
> Best Regards
>
> Steven Chan
>
> ________________________________
>
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 21 September 2018 19:30:00
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Steven,
>
> Thanks for sending the sample data. As you already know, I
confirmed that
> MET can't read these data files as-is. I get the following error
from
> plot_data_plane:
>
> plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> 'name="precipitation_flux"; level="(0,*,*)";'
> ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> from information in netCDF file.
>
> So we need to add support for reading the rotated lat/lon projection
> information from CF-compliant NetCDF files. I was keeping my
fingers
> crossed that it'd be very simple to do, but it is not. It'll take
some
> effort to make sure we do it correctly. And with our release next
week, we
> don't have enough time to fit it in. But I did create a development
ticket
> for this for the next release.
>
> However, MET will still not be able to read the variable mesh
lat/lon
> data. We use the grid/projection information from the input data
files to
> instantiate a regular grid in the MET libraries. That enables MET
to
> convert from grid x/y to lat/lon and vice-versa. And for this
reason, MET
> doesn't currently support a variable mesh. Its possible that could
change
> in the future depending on requirements from our funding agencies.
>
> met-8.0 is coming out next week and includes support for calling a
python
> 2.7 script... and we can read rotated lat/lon data down that route.
I ran
> the following command using the development version of MET:
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
> In this command, I have...
> (1) Replaced the filename (schan_eurocordex.nc) with the constant
> "PYTHON_NUMPY".
> (2) Set 'name="...";' where "..." is the python script with
associated
> command line arguments to be run.
>
> I've attached the python script to illustrate how it's reading the
first
> time-slice of the gridded data and setting up the metadata. We
would
> obviously want a more robust python script than this simple one.
>
> And I've attached the resulting plot to see if MET is putting the
data at
> the right spot on earth. Looks reasonable to me.
>
> This demonstrates that there is a way forward with met-8.0 until we
get
> support for rotated lat/lon projections in CF-compliant files added.
>
> Hope this helps.
>
> Thanks,
> John
>
>
>
>
>
>
> On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > I have uploaded two files to the FTP:
> >
> > /incoming/irap/met_help/StevenChan_data/
> > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> >
> > They are both rotated pole, but the resolution of the files are
different
> > with one of the files have variable resolution (near the
boundaries).
> >
> > Please let me know if the files are okay.
> >
> > Thank you and Best Regards,
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 20 September 2018 18:12
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > Thanks for the update. We're down to crunch time here getting the
> met-8.0
> > release finalized. Once I get my hands on the sample file, I'll
do my
> best
> > to get it working.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Line manager said it is fine. I am just waiting for Sys Admin to
> > > actually make that possible through the Met Office firewall (a
> > > technical issue). I will try to get that resolved before the end
of
> this
> > week.
> > >
> > > Thank you
> > > Steven
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 18:27
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > OK, please let us know. The development version for met-8.0
doesn't
> > > currently have support for parsing rotated lat/lon grids from
> > > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > > example we have is from the US Air Force's usage of UK Met
Office
> > > unified model... which they call GALWEM.
> > >
> > > If we can get sample data soon enough, hopefully we can get it
added
> > > prior to the met-8.0 release.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > > and the files are created supposedly according to CF
convention
> > > > using Met Office python libraries; hence reading the data with
the
> > > > help with Python will be extremely helpful function.
> > > >
> > > > I will need to confirm with line manager if it is possible to
send a
> > > > data sample as Met Office has a lot of rules in sharing data.
If
> > > > that is not possible, I may be able to I can get you
fictitious data
> > > > with the right coordinate information. I will get back to you
with
> > > > this before the end of this week (line manager is out of town
right
> > now).
> > > >
> > > > Best Regards,
> > > > Steven Chan
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 18 September 2018 16:35
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > I see that MET is not able to handle the projection on which
your
> > > > data lives. We are currently working on the next version of
MET,
> > met-8.0.
> > > > We've passed the code freeze date and are now in testing. One
of
> > > > the features we added was support for rotated lat/lon grids.
Based
> > > > on your description, it sounds like that's what you have.
met-8.0
> > > > is currently able to read rotated lat/lon data from GRIB2
files, but
> > > > we've been looking for other rotated lat/lon projection
examples.
> > > > However, I suspect we may run into other technical issues with
your
> > > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > > convention,
> > > which MET supports?
> > > >
> > > > Another enhancement in met-8.0 is support for calling python
scripts
> > > > to read gridded data. If MET isn't able to read the
particular
> > > > flavor of NetCDF you have, you could write a python script to
read
> > > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > > you run MODE, instead of telling it to read a NetCDF input
file, you
> > > > can tell it to execute a python script. Then it reads the
gridded
> > > > field of data from that python script in memory. We hope that
this
> > > > feature will make it possible to run MET on a wider variety of
data
> > formats.
> > > >
> > > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > > we can take a look? Here's how you can post data to our
anonymous
> ftp
> > site:
> > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > >
> > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > > Queue: met_help
> > > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > > Owner: Nobody
> > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET-HELP:
> > > > >
> > > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > > with some model inputs that have unusual map projections
(i.e. the
> > > "poles"
> > > > > not being where they are usually; or "rotated pole" in the
way we
> > > > > called
> > > > them here).
> > > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > > after xy dependent latitude and longitude information are
added to
> > > > > the
> > > > netcdf file.
> > > > > It gives errors like:
> > > > >
> > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc" not
> > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > > Couldn't figure out projection from information in netCDF
file.
> > > > >
> > > > > In the past, we got around this problem by regridding the
data to
> > > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > > to
> > > > avoid that.
> > > > > Are there any solutions to directly work with the projected
data?
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Steven
> > > > >
> > > > >
=============================================================
> > > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > > Kingdom
> > > > > Tel: +44 (0)1392 886153
> > > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk
> <http://www.metoffice.gov.uk>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Fri Apr 26 07:00:31 2019
Dear John:
Yes, indeed there are multiple version of Python installed here -- a
default "bad" version that get installed part of RHEL and another
"good" version that is only accessible via a module.
The actual problem is actually solved now with some caveats. As
according what you have indicated with the MET_PYTHON_?? flags, "make"
compiles with the "good" version of the Python library.
At one time during runtime, somehow computer is still trying to load
the "bad" one because the "bad" one has the libraries in one of the
paths under LD_LIBRARY_PATH. The interesting thing is that MET
actually loads the "good" version of the Python executable (as
indicated by sys.executable), but either Python or MET and then
decides to run off to go use the bad library in LD_LIBRARY_PATH
instead of the library that comes with the Python indicated by
sys.executable. Basically, it can't tell which Python library is the
good or the bad one during runtime.
Attempts to remove that specific path from LD_LIBRARY_PATH leads to
unexpected behaviour of other software which are not related to MET
(like emacs and evince will cause errors and warnings) (the violating
item is something silly like /usr/lib -- i.e. you can't really remove
it). The solution now is basically every MET tool has to run through a
shell script which explicitly remove items from LD_LIBRARY_PATH that
are incompatible with MET. Perhaps there are better way to fix this,
but at least it is working now.
Best Regards,
Steven Chan
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 25 April 2019 21:48:33
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven,
It sounds like you're wondering how to configure your environment to
use
one version of python versus another.
While adding support for calling python scripts has solved some
problems in
MET, it has also given rise to new ones. And this is one of them!
Unfortunately, I believe this needs to be handled when compiling MET.
In
particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
MET_PYTHON_CC
Where the python header files are located, e.g.
-I<include
dir>.
Set to the output of 'python-config --cflags'.
MET_PYTHON_LD
Where the python library files are located, e.g. -L<lib
dir>,
followed by the python library names, e.g. -l<library>.
Set to
the output of 'python-config --ldflags'.
It sounds like you have a "good" version of python around and a "bad"
version of python. If the directory where MET was compiled is
available on
your system, you could check "config.log" to see how those were set at
compilation time. For example:
*grep MET_PYTHON_ config.log*
ac_cv_env_MET_PYTHON_CC_set=set
ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
ac_cv_env_MET_PYTHON_LD_set=set
ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
-lutil -lm -lpython2.7 -Xlinker -export-dynamic'
And check to see if you're compiling against your preferred version of
python.
Hope that helps.
John
On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
>
>
> One additional question:
>
> I notice from tests that paths to Python interpreters and modules
are
> inconsistent. This is quite visible here because we have multiple
versions
> (and mutually incompatible) of Python interpreters and modules
installed
> that are managed using shell "module load" command. For example:
>
>
>
> import sys
> print('Using: ',sys.executable)
> Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
>
> import csv
>
> import numpy as np
>
> import os
>
> print csv
>
> <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
>
> print np
>
> <module 'numpy' from
> '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
>
> print os
>
> <module 'os' from '/usr/lib64/python2.7/os.pyc'>
>
>
>
> There is a separate installation of python modules under
> /opt/scitools/environments/default_legacy/2019_02_27/; however, for
some
> reasons, the executable decides to go find modules from a different
place
> “/usr/lib64/python2.7/”. Is there a way to fix the above? The above
is
> actually generating errors with actual tests of using MTD through
the
> Python interface because the correct Python modules cannot be found,
and I
> tried hand-fixing sys.path but it does not work as well.
>
>
> Best Regards,
>
> Steven
>
>
>
> ________________________________
>
> From: Chan, Steven
> Sent: 11 April 2019 14:09:04
> To: met_help at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Dear John:
>
>
>
> Please forgive me for the number of emails. This is about the
original
> email you sent to me last year about rotated pole issue. In fact the
> question has nothing to do with rotated pole but with the below:
>
>
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
>
>
> For some MET tools, more than one input files are required (for
example
> "mtd"). How would the Python interface work under that particular
> situation? In other words, how would one actually loop through a
list of
> files?
>
>
>
> Particularly for mtd: -fcst/-single/-obs must be followed by a list
of
> files or a file with list of files. If I understood the manual
correctly, I
> interpreted that as repeating the string "PYTHON_NUMPY", so there
will be a
> need of linking a particular "PYTHON_NUMPY" to a particular file.
Please
> correct me if I am wrong.
>
>
>
> Best Regards
>
> Steven Chan
>
> ________________________________
>
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 21 September 2018 19:30:00
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Steven,
>
> Thanks for sending the sample data. As you already know, I
confirmed that
> MET can't read these data files as-is. I get the following error
from
> plot_data_plane:
>
> plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> 'name="precipitation_flux"; level="(0,*,*)";'
> ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> from information in netCDF file.
>
> So we need to add support for reading the rotated lat/lon projection
> information from CF-compliant NetCDF files. I was keeping my
fingers
> crossed that it'd be very simple to do, but it is not. It'll take
some
> effort to make sure we do it correctly. And with our release next
week, we
> don't have enough time to fit it in. But I did create a development
ticket
> for this for the next release.
>
> However, MET will still not be able to read the variable mesh
lat/lon
> data. We use the grid/projection information from the input data
files to
> instantiate a regular grid in the MET libraries. That enables MET
to
> convert from grid x/y to lat/lon and vice-versa. And for this
reason, MET
> doesn't currently support a variable mesh. Its possible that could
change
> in the future depending on requirements from our funding agencies.
>
> met-8.0 is coming out next week and includes support for calling a
python
> 2.7 script... and we can read rotated lat/lon data down that route.
I ran
> the following command using the development version of MET:
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
> In this command, I have...
> (1) Replaced the filename (schan_eurocordex.nc) with the constant
> "PYTHON_NUMPY".
> (2) Set 'name="...";' where "..." is the python script with
associated
> command line arguments to be run.
>
> I've attached the python script to illustrate how it's reading the
first
> time-slice of the gridded data and setting up the metadata. We
would
> obviously want a more robust python script than this simple one.
>
> And I've attached the resulting plot to see if MET is putting the
data at
> the right spot on earth. Looks reasonable to me.
>
> This demonstrates that there is a way forward with met-8.0 until we
get
> support for rotated lat/lon projections in CF-compliant files added.
>
> Hope this helps.
>
> Thanks,
> John
>
>
>
>
>
>
> On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > I have uploaded two files to the FTP:
> >
> > /incoming/irap/met_help/StevenChan_data/
> > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> >
> > They are both rotated pole, but the resolution of the files are
different
> > with one of the files have variable resolution (near the
boundaries).
> >
> > Please let me know if the files are okay.
> >
> > Thank you and Best Regards,
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 20 September 2018 18:12
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > Thanks for the update. We're down to crunch time here getting the
> met-8.0
> > release finalized. Once I get my hands on the sample file, I'll
do my
> best
> > to get it working.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Line manager said it is fine. I am just waiting for Sys Admin to
> > > actually make that possible through the Met Office firewall (a
> > > technical issue). I will try to get that resolved before the end
of
> this
> > week.
> > >
> > > Thank you
> > > Steven
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 18:27
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > OK, please let us know. The development version for met-8.0
doesn't
> > > currently have support for parsing rotated lat/lon grids from
> > > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > > example we have is from the US Air Force's usage of UK Met
Office
> > > unified model... which they call GALWEM.
> > >
> > > If we can get sample data soon enough, hopefully we can get it
added
> > > prior to the met-8.0 release.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > > and the files are created supposedly according to CF
convention
> > > > using Met Office python libraries; hence reading the data with
the
> > > > help with Python will be extremely helpful function.
> > > >
> > > > I will need to confirm with line manager if it is possible to
send a
> > > > data sample as Met Office has a lot of rules in sharing data.
If
> > > > that is not possible, I may be able to I can get you
fictitious data
> > > > with the right coordinate information. I will get back to you
with
> > > > this before the end of this week (line manager is out of town
right
> > now).
> > > >
> > > > Best Regards,
> > > > Steven Chan
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 18 September 2018 16:35
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > I see that MET is not able to handle the projection on which
your
> > > > data lives. We are currently working on the next version of
MET,
> > met-8.0.
> > > > We've passed the code freeze date and are now in testing. One
of
> > > > the features we added was support for rotated lat/lon grids.
Based
> > > > on your description, it sounds like that's what you have.
met-8.0
> > > > is currently able to read rotated lat/lon data from GRIB2
files, but
> > > > we've been looking for other rotated lat/lon projection
examples.
> > > > However, I suspect we may run into other technical issues with
your
> > > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > > convention,
> > > which MET supports?
> > > >
> > > > Another enhancement in met-8.0 is support for calling python
scripts
> > > > to read gridded data. If MET isn't able to read the
particular
> > > > flavor of NetCDF you have, you could write a python script to
read
> > > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > > you run MODE, instead of telling it to read a NetCDF input
file, you
> > > > can tell it to execute a python script. Then it reads the
gridded
> > > > field of data from that python script in memory. We hope that
this
> > > > feature will make it possible to run MET on a wider variety of
data
> > formats.
> > > >
> > > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > > we can take a look? Here's how you can post data to our
anonymous
> ftp
> > site:
> > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > >
> > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > > Queue: met_help
> > > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > > Owner: Nobody
> > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET-HELP:
> > > > >
> > > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > > with some model inputs that have unusual map projections
(i.e. the
> > > "poles"
> > > > > not being where they are usually; or "rotated pole" in the
way we
> > > > > called
> > > > them here).
> > > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > > after xy dependent latitude and longitude information are
added to
> > > > > the
> > > > netcdf file.
> > > > > It gives errors like:
> > > > >
> > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc" not
> > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > > Couldn't figure out projection from information in netCDF
file.
> > > > >
> > > > > In the past, we got around this problem by regridding the
data to
> > > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > > to
> > > > avoid that.
> > > > > Are there any solutions to directly work with the projected
data?
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Steven
> > > > >
> > > > >
=============================================================
> > > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > > Kingdom
> > > > > Tel: +44 (0)1392 886153
> > > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk<http://www.metoffice.gov.uk>
> <http://www.metoffice.gov.uk>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Fri Apr 26 07:02:21 2019
Again thank you very much for all the emails exchange. I hope I wasn't
using up too much of your time.
Steven
________________________________
From: Chan, Steven
Sent: 26 April 2019 14:00:17
To: met_help at ucar.edu
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Dear John:
Yes, indeed there are multiple version of Python installed here -- a
default "bad" version that get installed part of RHEL and another
"good" version that is only accessible via a module.
The actual problem is actually solved now with some caveats. As
according what you have indicated with the MET_PYTHON_?? flags, "make"
compiles with the "good" version of the Python library.
At one time during runtime, somehow computer is still trying to load
the "bad" one because the "bad" one has the libraries in one of the
paths under LD_LIBRARY_PATH. The interesting thing is that MET
actually loads the "good" version of the Python executable (as
indicated by sys.executable), but either Python or MET and then
decides to run off to go use the bad library in LD_LIBRARY_PATH
instead of the library that comes with the Python indicated by
sys.executable. Basically, it can't tell which Python library is the
good or the bad one during runtime.
Attempts to remove that specific path from LD_LIBRARY_PATH leads to
unexpected behaviour of other software which are not related to MET
(like emacs and evince will cause errors and warnings) (the violating
item is something silly like /usr/lib -- i.e. you can't really remove
it). The solution now is basically every MET tool has to run through a
shell script which explicitly remove items from LD_LIBRARY_PATH that
are incompatible with MET. Perhaps there are better way to fix this,
but at least it is working now.
Best Regards,
Steven Chan
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 25 April 2019 21:48:33
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven,
It sounds like you're wondering how to configure your environment to
use
one version of python versus another.
While adding support for calling python scripts has solved some
problems in
MET, it has also given rise to new ones. And this is one of them!
Unfortunately, I believe this needs to be handled when compiling MET.
In
particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
MET_PYTHON_CC
Where the python header files are located, e.g.
-I<include
dir>.
Set to the output of 'python-config --cflags'.
MET_PYTHON_LD
Where the python library files are located, e.g. -L<lib
dir>,
followed by the python library names, e.g. -l<library>.
Set to
the output of 'python-config --ldflags'.
It sounds like you have a "good" version of python around and a "bad"
version of python. If the directory where MET was compiled is
available on
your system, you could check "config.log" to see how those were set at
compilation time. For example:
*grep MET_PYTHON_ config.log*
ac_cv_env_MET_PYTHON_CC_set=set
ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
ac_cv_env_MET_PYTHON_LD_set=set
ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
-lutil -lm -lpython2.7 -Xlinker -export-dynamic'
And check to see if you're compiling against your preferred version of
python.
Hope that helps.
John
On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
>
>
> One additional question:
>
> I notice from tests that paths to Python interpreters and modules
are
> inconsistent. This is quite visible here because we have multiple
versions
> (and mutually incompatible) of Python interpreters and modules
installed
> that are managed using shell "module load" command. For example:
>
>
>
> import sys
> print('Using: ',sys.executable)
> Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
>
> import csv
>
> import numpy as np
>
> import os
>
> print csv
>
> <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
>
> print np
>
> <module 'numpy' from
> '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
>
> print os
>
> <module 'os' from '/usr/lib64/python2.7/os.pyc'>
>
>
>
> There is a separate installation of python modules under
> /opt/scitools/environments/default_legacy/2019_02_27/; however, for
some
> reasons, the executable decides to go find modules from a different
place
> “/usr/lib64/python2.7/”. Is there a way to fix the above? The above
is
> actually generating errors with actual tests of using MTD through
the
> Python interface because the correct Python modules cannot be found,
and I
> tried hand-fixing sys.path but it does not work as well.
>
>
> Best Regards,
>
> Steven
>
>
>
> ________________________________
>
> From: Chan, Steven
> Sent: 11 April 2019 14:09:04
> To: met_help at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Dear John:
>
>
>
> Please forgive me for the number of emails. This is about the
original
> email you sent to me last year about rotated pole issue. In fact the
> question has nothing to do with rotated pole but with the below:
>
>
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
>
>
> For some MET tools, more than one input files are required (for
example
> "mtd"). How would the Python interface work under that particular
> situation? In other words, how would one actually loop through a
list of
> files?
>
>
>
> Particularly for mtd: -fcst/-single/-obs must be followed by a list
of
> files or a file with list of files. If I understood the manual
correctly, I
> interpreted that as repeating the string "PYTHON_NUMPY", so there
will be a
> need of linking a particular "PYTHON_NUMPY" to a particular file.
Please
> correct me if I am wrong.
>
>
>
> Best Regards
>
> Steven Chan
>
> ________________________________
>
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 21 September 2018 19:30:00
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
>
>
> Steven,
>
> Thanks for sending the sample data. As you already know, I
confirmed that
> MET can't read these data files as-is. I get the following error
from
> plot_data_plane:
>
> plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> 'name="precipitation_flux"; level="(0,*,*)";'
> ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> from information in netCDF file.
>
> So we need to add support for reading the rotated lat/lon projection
> information from CF-compliant NetCDF files. I was keeping my
fingers
> crossed that it'd be very simple to do, but it is not. It'll take
some
> effort to make sure we do it correctly. And with our release next
week, we
> don't have enough time to fit it in. But I did create a development
ticket
> for this for the next release.
>
> However, MET will still not be able to read the variable mesh
lat/lon
> data. We use the grid/projection information from the input data
files to
> instantiate a regular grid in the MET libraries. That enables MET
to
> convert from grid x/y to lat/lon and vice-versa. And for this
reason, MET
> doesn't currently support a variable mesh. Its possible that could
change
> in the future depending on requirements from our funding agencies.
>
> met-8.0 is coming out next week and includes support for calling a
python
> 2.7 script... and we can read rotated lat/lon data down that route.
I ran
> the following command using the development version of MET:
>
> plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
'name="read_nccf_rotll.py
> schan_eurocordex.nc precipitation_flux";'
>
> In this command, I have...
> (1) Replaced the filename (schan_eurocordex.nc) with the constant
> "PYTHON_NUMPY".
> (2) Set 'name="...";' where "..." is the python script with
associated
> command line arguments to be run.
>
> I've attached the python script to illustrate how it's reading the
first
> time-slice of the gridded data and setting up the metadata. We
would
> obviously want a more robust python script than this simple one.
>
> And I've attached the resulting plot to see if MET is putting the
data at
> the right spot on earth. Looks reasonable to me.
>
> This demonstrates that there is a way forward with met-8.0 until we
get
> support for rotated lat/lon projections in CF-compliant files added.
>
> Hope this helps.
>
> Thanks,
> John
>
>
>
>
>
>
> On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> > I have uploaded two files to the FTP:
> >
> > /incoming/irap/met_help/StevenChan_data/
> > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> >
> > They are both rotated pole, but the resolution of the files are
different
> > with one of the files have variable resolution (near the
boundaries).
> >
> > Please let me know if the files are okay.
> >
> > Thank you and Best Regards,
> > Steven
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 20 September 2018 18:12
> > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Steven,
> >
> > Thanks for the update. We're down to crunch time here getting the
> met-8.0
> > release finalized. Once I get my hands on the sample file, I'll
do my
> best
> > to get it working.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > Line manager said it is fine. I am just waiting for Sys Admin to
> > > actually make that possible through the Met Office firewall (a
> > > technical issue). I will try to get that resolved before the end
of
> this
> > week.
> > >
> > > Thank you
> > > Steven
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 18 September 2018 18:27
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > OK, please let us know. The development version for met-8.0
doesn't
> > > currently have support for parsing rotated lat/lon grids from
> > > CF-compliant NetCDF files since we don't have any examples. The
GRIB2
> > > example we have is from the US Air Force's usage of UK Met
Office
> > > unified model... which they call GALWEM.
> > >
> > > If we can get sample data soon enough, hopefully we can get it
added
> > > prior to the met-8.0 release.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > Thank you very much for your reply. Yes, it is rotated lat-lon
grid,
> > > > and the files are created supposedly according to CF
convention
> > > > using Met Office python libraries; hence reading the data with
the
> > > > help with Python will be extremely helpful function.
> > > >
> > > > I will need to confirm with line manager if it is possible to
send a
> > > > data sample as Met Office has a lot of rules in sharing data.
If
> > > > that is not possible, I may be able to I can get you
fictitious data
> > > > with the right coordinate information. I will get back to you
with
> > > > this before the end of this week (line manager is out of town
right
> > now).
> > > >
> > > > Best Regards,
> > > > Steven Chan
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 18 September 2018 16:35
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > I see that MET is not able to handle the projection on which
your
> > > > data lives. We are currently working on the next version of
MET,
> > met-8.0.
> > > > We've passed the code freeze date and are now in testing. One
of
> > > > the features we added was support for rotated lat/lon grids.
Based
> > > > on your description, it sounds like that's what you have.
met-8.0
> > > > is currently able to read rotated lat/lon data from GRIB2
files, but
> > > > we've been looking for other rotated lat/lon projection
examples.
> > > > However, I suspect we may run into other technical issues with
your
> > > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > > convention,
> > > which MET supports?
> > > >
> > > > Another enhancement in met-8.0 is support for calling python
scripts
> > > > to read gridded data. If MET isn't able to read the
particular
> > > > flavor of NetCDF you have, you could write a python script to
read
> > > > the data, populate a NumPy array, and dictionary of metadata.
When
> > > > you run MODE, instead of telling it to read a NetCDF input
file, you
> > > > can tell it to execute a python script. Then it reads the
gridded
> > > > field of data from that python script in memory. We hope that
this
> > > > feature will make it possible to run MET on a wider variety of
data
> > formats.
> > > >
> > > > Can you please send us a sample rotated lat/lon NetCDF file so
that
> > > > we can take a look? Here's how you can post data to our
anonymous
> ftp
> > site:
> > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > >
> > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > Transaction: Ticket created by steven.chan at metoffice.gov.uk
> > > > > Queue: met_help
> > > > > Subject: MET5.2 question: Dealing with projected inputs
> > > > > Owner: Nobody
> > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET-HELP:
> > > > >
> > > > > Sorry to bug you a question, but I am trying to use MODE and
MTD
> > > > > with some model inputs that have unusual map projections
(i.e. the
> > > "poles"
> > > > > not being where they are usually; or "rotated pole" in the
way we
> > > > > called
> > > > them here).
> > > > > MODE, MTD and plot_data_plane do not seem wants to read them
even
> > > > > after xy dependent latitude and longitude information are
added to
> > > > > the
> > > > netcdf file.
> > > > > It gives errors like:
> > > > >
> > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc" not
> > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > > Couldn't figure out projection from information in netCDF
file.
> > > > >
> > > > > In the past, we got around this problem by regridding the
data to
> > > > > regular lat-lon; due to scientific and technical reasons, we
wish
> > > > > to
> > > > avoid that.
> > > > > Are there any solutions to directly work with the projected
data?
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Steven
> > > > >
> > > > >
=============================================================
> > > > > Steven C Chan, Visiting Scientist from Newcastle University
Met
> > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
United
> > > > > Kingdom
> > > > > Tel: +44 (0)1392 886153
> > > > > Email: steven.chan at metoffice.gov.uk Website:
www.metoffice.gov.uk<http://www.metoffice.gov.uk>
> <http://www.metoffice.gov.uk>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Julie Prestopnik
Time: Fri Apr 26 08:45:42 2019
Hi Steven. My name is Julie Prestopnik and I work with John. I just
thought I'd jump in here with some further information. Thank you for
describing what is going on.
I think there is potentially a better way to fix this issue. Using
"rpath"
in compiling will embed a specific directory into the runtime library
search path. It takes precedence over a user's LD_LIBRARY_PATH and
might
be worthwhile to try.
Before configuring and compiling MET, you would want to set CPPFLAGS
to the
location of where the python header files are located (likely the the
output of `python-config --cflags`) and set LDFLAGS (using rpath as
shown
below: "-Wl,-rpath,") to the location of the python library directory
(the
-L part of `python-config --ldflags`). For example:
export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
After setting those environment variables, along with the other
environment
variables necessary for the compilation of MET, you would then run
configure and then "make install".
Please let us know how it goes if you decide to give that a try.
Julie
On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
>
> Yes, indeed there are multiple version of Python installed here -- a
> default "bad" version that get installed part of RHEL and another
"good"
> version that is only accessible via a module.
>
>
> The actual problem is actually solved now with some caveats. As
according
> what you have indicated with the MET_PYTHON_?? flags, "make"
compiles with
> the "good" version of the Python library.
>
>
> At one time during runtime, somehow computer is still trying to load
the
> "bad" one because the "bad" one has the libraries in one of the
paths under
> LD_LIBRARY_PATH. The interesting thing is that MET actually loads
the
> "good" version of the Python executable (as indicated by
sys.executable),
> but either Python or MET and then decides to run off to go use the
bad
> library in LD_LIBRARY_PATH instead of the library that comes with
the
> Python indicated by sys.executable. Basically, it can't tell which
Python
> library is the good or the bad one during runtime.
>
>
> Attempts to remove that specific path from LD_LIBRARY_PATH leads to
> unexpected behaviour of other software which are not related to MET
(like
> emacs and evince will cause errors and warnings) (the violating item
is
> something silly like /usr/lib -- i.e. you can't really remove it).
The
> solution now is basically every MET tool has to run through a shell
script
> which explicitly remove items from LD_LIBRARY_PATH that are
incompatible
> with MET. Perhaps there are better way to fix this, but at least it
is
> working now.
>
>
> Best Regards,
>
> Steven Chan
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 25 April 2019 21:48:33
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven,
>
> It sounds like you're wondering how to configure your environment to
use
> one version of python versus another.
>
> While adding support for calling python scripts has solved some
problems in
> MET, it has also given rise to new ones. And this is one of them!
>
> Unfortunately, I believe this needs to be handled when compiling
MET. In
> particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
>
> MET_PYTHON_CC
> Where the python header files are located, e.g.
-I<include
> dir>.
> Set to the output of 'python-config --cflags'.
> MET_PYTHON_LD
> Where the python library files are located, e.g.
-L<lib dir>,
> followed by the python library names, e.g.
-l<library>. Set
> to
> the output of 'python-config --ldflags'.
>
> It sounds like you have a "good" version of python around and a
"bad"
> version of python. If the directory where MET was compiled is
available on
> your system, you could check "config.log" to see how those were set
at
> compilation time. For example:
>
> *grep MET_PYTHON_ config.log*
> ac_cv_env_MET_PYTHON_CC_set=set
> ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> ac_cv_env_MET_PYTHON_LD_set=set
> ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
> -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
>
> And check to see if you're compiling against your preferred version
of
> python.
>
> Hope that helps.
>
> John
>
> On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> >
> >
> > One additional question:
> >
> > I notice from tests that paths to Python interpreters and modules
are
> > inconsistent. This is quite visible here because we have multiple
> versions
> > (and mutually incompatible) of Python interpreters and modules
installed
> > that are managed using shell "module load" command. For example:
> >
> >
> >
> > import sys
> > print('Using: ',sys.executable)
> > Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
> >
> > import csv
> >
> > import numpy as np
> >
> > import os
> >
> > print csv
> >
> > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> >
> > print np
> >
> > <module 'numpy' from
> > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> >
> > print os
> >
> > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> >
> >
> >
> > There is a separate installation of python modules under
> > /opt/scitools/environments/default_legacy/2019_02_27/; however,
for some
> > reasons, the executable decides to go find modules from a
different place
> > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above is
> > actually generating errors with actual tests of using MTD through
the
> > Python interface because the correct Python modules cannot be
found, and
> I
> > tried hand-fixing sys.path but it does not work as well.
> >
> >
> > Best Regards,
> >
> > Steven
> >
> >
> >
> > ________________________________
> >
> > From: Chan, Steven
> > Sent: 11 April 2019 14:09:04
> > To: met_help at ucar.edu
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> >
> >
> > Dear John:
> >
> >
> >
> > Please forgive me for the number of emails. This is about the
original
> > email you sent to me last year about rotated pole issue. In fact
the
> > question has nothing to do with rotated pole but with the below:
> >
> >
> >
> > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> 'name="read_nccf_rotll.py
> > schan_eurocordex.nc precipitation_flux";'
> >
> >
> >
> > For some MET tools, more than one input files are required (for
example
> > "mtd"). How would the Python interface work under that particular
> > situation? In other words, how would one actually loop through a
list of
> > files?
> >
> >
> >
> > Particularly for mtd: -fcst/-single/-obs must be followed by a
list of
> > files or a file with list of files. If I understood the manual
> correctly, I
> > interpreted that as repeating the string "PYTHON_NUMPY", so there
will
> be a
> > need of linking a particular "PYTHON_NUMPY" to a particular file.
Please
> > correct me if I am wrong.
> >
> >
> >
> > Best Regards
> >
> > Steven Chan
> >
> > ________________________________
> >
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 21 September 2018 19:30:00
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> >
> >
> > Steven,
> >
> > Thanks for sending the sample data. As you already know, I
confirmed
> that
> > MET can't read these data files as-is. I get the following error
from
> > plot_data_plane:
> >
> > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > 'name="precipitation_flux"; level="(0,*,*)";'
> > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> > from information in netCDF file.
> >
> > So we need to add support for reading the rotated lat/lon
projection
> > information from CF-compliant NetCDF files. I was keeping my
fingers
> > crossed that it'd be very simple to do, but it is not. It'll take
some
> > effort to make sure we do it correctly. And with our release next
week,
> we
> > don't have enough time to fit it in. But I did create a
development
> ticket
> > for this for the next release.
> >
> > However, MET will still not be able to read the variable mesh
lat/lon
> > data. We use the grid/projection information from the input data
files
> to
> > instantiate a regular grid in the MET libraries. That enables MET
to
> > convert from grid x/y to lat/lon and vice-versa. And for this
reason,
> MET
> > doesn't currently support a variable mesh. Its possible that
could
> change
> > in the future depending on requirements from our funding agencies.
> >
> > met-8.0 is coming out next week and includes support for calling a
python
> > 2.7 script... and we can read rotated lat/lon data down that
route. I
> ran
> > the following command using the development version of MET:
> >
> > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> 'name="read_nccf_rotll.py
> > schan_eurocordex.nc precipitation_flux";'
> >
> > In this command, I have...
> > (1) Replaced the filename (schan_eurocordex.nc) with the constant
> > "PYTHON_NUMPY".
> > (2) Set 'name="...";' where "..." is the python script with
associated
> > command line arguments to be run.
> >
> > I've attached the python script to illustrate how it's reading the
first
> > time-slice of the gridded data and setting up the metadata. We
would
> > obviously want a more robust python script than this simple one.
> >
> > And I've attached the resulting plot to see if MET is putting the
data at
> > the right spot on earth. Looks reasonable to me.
> >
> > This demonstrates that there is a way forward with met-8.0 until
we get
> > support for rotated lat/lon projections in CF-compliant files
added.
> >
> > Hope this helps.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > I have uploaded two files to the FTP:
> > >
> > > /incoming/irap/met_help/StevenChan_data/
> > > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > >
> > > They are both rotated pole, but the resolution of the files are
> different
> > > with one of the files have variable resolution (near the
boundaries).
> > >
> > > Please let me know if the files are okay.
> > >
> > > Thank you and Best Regards,
> > > Steven
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 20 September 2018 18:12
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > Thanks for the update. We're down to crunch time here getting
the
> > met-8.0
> > > release finalized. Once I get my hands on the sample file, I'll
do my
> > best
> > > to get it working.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > Line manager said it is fine. I am just waiting for Sys Admin
to
> > > > actually make that possible through the Met Office firewall (a
> > > > technical issue). I will try to get that resolved before the
end of
> > this
> > > week.
> > > >
> > > > Thank you
> > > > Steven
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 18 September 2018 18:27
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > OK, please let us know. The development version for met-8.0
doesn't
> > > > currently have support for parsing rotated lat/lon grids from
> > > > CF-compliant NetCDF files since we don't have any examples.
The
> GRIB2
> > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > unified model... which they call GALWEM.
> > > >
> > > > If we can get sample data soon enough, hopefully we can get it
added
> > > > prior to the met-8.0 release.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > Thank you very much for your reply. Yes, it is rotated lat-
lon
> grid,
> > > > > and the files are created supposedly according to CF
convention
> > > > > using Met Office python libraries; hence reading the data
with the
> > > > > help with Python will be extremely helpful function.
> > > > >
> > > > > I will need to confirm with line manager if it is possible
to send
> a
> > > > > data sample as Met Office has a lot of rules in sharing
data. If
> > > > > that is not possible, I may be able to I can get you
fictitious
> data
> > > > > with the right coordinate information. I will get back to
you with
> > > > > this before the end of this week (line manager is out of
town right
> > > now).
> > > > >
> > > > > Best Regards,
> > > > > Steven Chan
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 18 September 2018 16:35
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > I see that MET is not able to handle the projection on which
your
> > > > > data lives. We are currently working on the next version of
MET,
> > > met-8.0.
> > > > > We've passed the code freeze date and are now in testing.
One of
> > > > > the features we added was support for rotated lat/lon grids.
Based
> > > > > on your description, it sounds like that's what you have.
met-8.0
> > > > > is currently able to read rotated lat/lon data from GRIB2
files,
> but
> > > > > we've been looking for other rotated lat/lon projection
examples.
> > > > > However, I suspect we may run into other technical issues
with your
> > > > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > > > convention,
> > > > which MET supports?
> > > > >
> > > > > Another enhancement in met-8.0 is support for calling python
> scripts
> > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > flavor of NetCDF you have, you could write a python script
to read
> > > > > the data, populate a NumPy array, and dictionary of
metadata. When
> > > > > you run MODE, instead of telling it to read a NetCDF input
file,
> you
> > > > > can tell it to execute a python script. Then it reads the
gridded
> > > > > field of data from that python script in memory. We hope
that this
> > > > > feature will make it possible to run MET on a wider variety
of data
> > > formats.
> > > > >
> > > > > Can you please send us a sample rotated lat/lon NetCDF file
so that
> > > > > we can take a look? Here's how you can post data to our
anonymous
> > ftp
> > > site:
> > > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > Queue: met_help
> > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > Owner: Nobody
> > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET-HELP:
> > > > > >
> > > > > > Sorry to bug you a question, but I am trying to use MODE
and MTD
> > > > > > with some model inputs that have unusual map projections
(i.e.
> the
> > > > "poles"
> > > > > > not being where they are usually; or "rotated pole" in the
way we
> > > > > > called
> > > > > them here).
> > > > > > MODE, MTD and plot_data_plane do not seem wants to read
them even
> > > > > > after xy dependent latitude and longitude information are
added
> to
> > > > > > the
> > > > > netcdf file.
> > > > > > It gives errors like:
> > > > > >
> > > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc"
> not
> > > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > > > Couldn't figure out projection from information in netCDF
file.
> > > > > >
> > > > > > In the past, we got around this problem by regridding the
data to
> > > > > > regular lat-lon; due to scientific and technical reasons,
we wish
> > > > > > to
> > > > > avoid that.
> > > > > > Are there any solutions to directly work with the
projected data?
> > > > > >
> > > > > > Thank you and Best Regards,
> > > > > > Steven
> > > > > >
> > > > > >
=============================================================
> > > > > > Steven C Chan, Visiting Scientist from Newcastle
University Met
> > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
> United
> > > > > > Kingdom
> > > > > > Tel: +44 (0)1392 886153
> > > > > > Email: steven.chan at metoffice.gov.uk Website:
> www.metoffice.gov.uk<http://www.metoffice.gov.uk>
> > <http://www.metoffice.gov.uk>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Mon Apr 29 04:54:53 2019
Dear Julie:
Thank you very much for your email. It does not seem to work for me,
but perhaps I am doing something wrong?
In particular, should the new LDFLAGS options point to where
libpython2.7.a is? My "python-config --ldflags" has no path
information, but "python-config --cflags" does have path information.
i.e.
[schan at eld277:schan] $ python-config --cflags
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
-fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes
[schan at eld277:schan] $ python-config --ldflags
-lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
Best Regards
Steven
________________________________
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: 26 April 2019 15:45:42
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven. My name is Julie Prestopnik and I work with John. I just
thought I'd jump in here with some further information. Thank you for
describing what is going on.
I think there is potentially a better way to fix this issue. Using
"rpath"
in compiling will embed a specific directory into the runtime library
search path. It takes precedence over a user's LD_LIBRARY_PATH and
might
be worthwhile to try.
Before configuring and compiling MET, you would want to set CPPFLAGS
to the
location of where the python header files are located (likely the the
output of `python-config --cflags`) and set LDFLAGS (using rpath as
shown
below: "-Wl,-rpath,") to the location of the python library directory
(the
-L part of `python-config --ldflags`). For example:
export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
After setting those environment variables, along with the other
environment
variables necessary for the compilation of MET, you would then run
configure and then "make install".
Please let us know how it goes if you decide to give that a try.
Julie
On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear John:
>
>
> Yes, indeed there are multiple version of Python installed here -- a
> default "bad" version that get installed part of RHEL and another
"good"
> version that is only accessible via a module.
>
>
> The actual problem is actually solved now with some caveats. As
according
> what you have indicated with the MET_PYTHON_?? flags, "make"
compiles with
> the "good" version of the Python library.
>
>
> At one time during runtime, somehow computer is still trying to load
the
> "bad" one because the "bad" one has the libraries in one of the
paths under
> LD_LIBRARY_PATH. The interesting thing is that MET actually loads
the
> "good" version of the Python executable (as indicated by
sys.executable),
> but either Python or MET and then decides to run off to go use the
bad
> library in LD_LIBRARY_PATH instead of the library that comes with
the
> Python indicated by sys.executable. Basically, it can't tell which
Python
> library is the good or the bad one during runtime.
>
>
> Attempts to remove that specific path from LD_LIBRARY_PATH leads to
> unexpected behaviour of other software which are not related to MET
(like
> emacs and evince will cause errors and warnings) (the violating item
is
> something silly like /usr/lib -- i.e. you can't really remove it).
The
> solution now is basically every MET tool has to run through a shell
script
> which explicitly remove items from LD_LIBRARY_PATH that are
incompatible
> with MET. Perhaps there are better way to fix this, but at least it
is
> working now.
>
>
> Best Regards,
>
> Steven Chan
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 25 April 2019 21:48:33
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven,
>
> It sounds like you're wondering how to configure your environment to
use
> one version of python versus another.
>
> While adding support for calling python scripts has solved some
problems in
> MET, it has also given rise to new ones. And this is one of them!
>
> Unfortunately, I believe this needs to be handled when compiling
MET. In
> particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
>
> MET_PYTHON_CC
> Where the python header files are located, e.g.
-I<include
> dir>.
> Set to the output of 'python-config --cflags'.
> MET_PYTHON_LD
> Where the python library files are located, e.g.
-L<lib dir>,
> followed by the python library names, e.g.
-l<library>. Set
> to
> the output of 'python-config --ldflags'.
>
> It sounds like you have a "good" version of python around and a
"bad"
> version of python. If the directory where MET was compiled is
available on
> your system, you could check "config.log" to see how those were set
at
> compilation time. For example:
>
> *grep MET_PYTHON_ config.log*
> ac_cv_env_MET_PYTHON_CC_set=set
> ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> ac_cv_env_MET_PYTHON_LD_set=set
> ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
> -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
>
> And check to see if you're compiling against your preferred version
of
> python.
>
> Hope that helps.
>
> John
>
> On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> >
> >
> > One additional question:
> >
> > I notice from tests that paths to Python interpreters and modules
are
> > inconsistent. This is quite visible here because we have multiple
> versions
> > (and mutually incompatible) of Python interpreters and modules
installed
> > that are managed using shell "module load" command. For example:
> >
> >
> >
> > import sys
> > print('Using: ',sys.executable)
> > Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
> >
> > import csv
> >
> > import numpy as np
> >
> > import os
> >
> > print csv
> >
> > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> >
> > print np
> >
> > <module 'numpy' from
> > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> >
> > print os
> >
> > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> >
> >
> >
> > There is a separate installation of python modules under
> > /opt/scitools/environments/default_legacy/2019_02_27/; however,
for some
> > reasons, the executable decides to go find modules from a
different place
> > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above is
> > actually generating errors with actual tests of using MTD through
the
> > Python interface because the correct Python modules cannot be
found, and
> I
> > tried hand-fixing sys.path but it does not work as well.
> >
> >
> > Best Regards,
> >
> > Steven
> >
> >
> >
> > ________________________________
> >
> > From: Chan, Steven
> > Sent: 11 April 2019 14:09:04
> > To: met_help at ucar.edu
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> >
> >
> > Dear John:
> >
> >
> >
> > Please forgive me for the number of emails. This is about the
original
> > email you sent to me last year about rotated pole issue. In fact
the
> > question has nothing to do with rotated pole but with the below:
> >
> >
> >
> > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> 'name="read_nccf_rotll.py
> > schan_eurocordex.nc precipitation_flux";'
> >
> >
> >
> > For some MET tools, more than one input files are required (for
example
> > "mtd"). How would the Python interface work under that particular
> > situation? In other words, how would one actually loop through a
list of
> > files?
> >
> >
> >
> > Particularly for mtd: -fcst/-single/-obs must be followed by a
list of
> > files or a file with list of files. If I understood the manual
> correctly, I
> > interpreted that as repeating the string "PYTHON_NUMPY", so there
will
> be a
> > need of linking a particular "PYTHON_NUMPY" to a particular file.
Please
> > correct me if I am wrong.
> >
> >
> >
> > Best Regards
> >
> > Steven Chan
> >
> > ________________________________
> >
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 21 September 2018 19:30:00
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> >
> >
> > Steven,
> >
> > Thanks for sending the sample data. As you already know, I
confirmed
> that
> > MET can't read these data files as-is. I get the following error
from
> > plot_data_plane:
> >
> > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > 'name="precipitation_flux"; level="(0,*,*)";'
> > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> > from information in netCDF file.
> >
> > So we need to add support for reading the rotated lat/lon
projection
> > information from CF-compliant NetCDF files. I was keeping my
fingers
> > crossed that it'd be very simple to do, but it is not. It'll take
some
> > effort to make sure we do it correctly. And with our release next
week,
> we
> > don't have enough time to fit it in. But I did create a
development
> ticket
> > for this for the next release.
> >
> > However, MET will still not be able to read the variable mesh
lat/lon
> > data. We use the grid/projection information from the input data
files
> to
> > instantiate a regular grid in the MET libraries. That enables MET
to
> > convert from grid x/y to lat/lon and vice-versa. And for this
reason,
> MET
> > doesn't currently support a variable mesh. Its possible that
could
> change
> > in the future depending on requirements from our funding agencies.
> >
> > met-8.0 is coming out next week and includes support for calling a
python
> > 2.7 script... and we can read rotated lat/lon data down that
route. I
> ran
> > the following command using the development version of MET:
> >
> > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> 'name="read_nccf_rotll.py
> > schan_eurocordex.nc precipitation_flux";'
> >
> > In this command, I have...
> > (1) Replaced the filename (schan_eurocordex.nc) with the constant
> > "PYTHON_NUMPY".
> > (2) Set 'name="...";' where "..." is the python script with
associated
> > command line arguments to be run.
> >
> > I've attached the python script to illustrate how it's reading the
first
> > time-slice of the gridded data and setting up the metadata. We
would
> > obviously want a more robust python script than this simple one.
> >
> > And I've attached the resulting plot to see if MET is putting the
data at
> > the right spot on earth. Looks reasonable to me.
> >
> > This demonstrates that there is a way forward with met-8.0 until
we get
> > support for rotated lat/lon projections in CF-compliant files
added.
> >
> > Hope this helps.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > > I have uploaded two files to the FTP:
> > >
> > > /incoming/irap/met_help/StevenChan_data/
> > > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > >
> > > They are both rotated pole, but the resolution of the files are
> different
> > > with one of the files have variable resolution (near the
boundaries).
> > >
> > > Please let me know if the files are okay.
> > >
> > > Thank you and Best Regards,
> > > Steven
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 20 September 2018 18:12
> > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Steven,
> > >
> > > Thanks for the update. We're down to crunch time here getting
the
> > met-8.0
> > > release finalized. Once I get my hands on the sample file, I'll
do my
> > best
> > > to get it working.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > Line manager said it is fine. I am just waiting for Sys Admin
to
> > > > actually make that possible through the Met Office firewall (a
> > > > technical issue). I will try to get that resolved before the
end of
> > this
> > > week.
> > > >
> > > > Thank you
> > > > Steven
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 18 September 2018 18:27
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > OK, please let us know. The development version for met-8.0
doesn't
> > > > currently have support for parsing rotated lat/lon grids from
> > > > CF-compliant NetCDF files since we don't have any examples.
The
> GRIB2
> > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > unified model... which they call GALWEM.
> > > >
> > > > If we can get sample data soon enough, hopefully we can get it
added
> > > > prior to the met-8.0 release.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > Thank you very much for your reply. Yes, it is rotated lat-
lon
> grid,
> > > > > and the files are created supposedly according to CF
convention
> > > > > using Met Office python libraries; hence reading the data
with the
> > > > > help with Python will be extremely helpful function.
> > > > >
> > > > > I will need to confirm with line manager if it is possible
to send
> a
> > > > > data sample as Met Office has a lot of rules in sharing
data. If
> > > > > that is not possible, I may be able to I can get you
fictitious
> data
> > > > > with the right coordinate information. I will get back to
you with
> > > > > this before the end of this week (line manager is out of
town right
> > > now).
> > > > >
> > > > > Best Regards,
> > > > > Steven Chan
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 18 September 2018 16:35
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > I see that MET is not able to handle the projection on which
your
> > > > > data lives. We are currently working on the next version of
MET,
> > > met-8.0.
> > > > > We've passed the code freeze date and are now in testing.
One of
> > > > > the features we added was support for rotated lat/lon grids.
Based
> > > > > on your description, it sounds like that's what you have.
met-8.0
> > > > > is currently able to read rotated lat/lon data from GRIB2
files,
> but
> > > > > we've been looking for other rotated lat/lon projection
examples.
> > > > > However, I suspect we may run into other technical issues
with your
> > > > > data. Do your NetCDF files follow the CF (climate-forecast)
> > > > > convention,
> > > > which MET supports?
> > > > >
> > > > > Another enhancement in met-8.0 is support for calling python
> scripts
> > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > flavor of NetCDF you have, you could write a python script
to read
> > > > > the data, populate a NumPy array, and dictionary of
metadata. When
> > > > > you run MODE, instead of telling it to read a NetCDF input
file,
> you
> > > > > can tell it to execute a python script. Then it reads the
gridded
> > > > > field of data from that python script in memory. We hope
that this
> > > > > feature will make it possible to run MET on a wider variety
of data
> > > formats.
> > > > >
> > > > > Can you please send us a sample rotated lat/lon NetCDF file
so that
> > > > > we can take a look? Here's how you can post data to our
anonymous
> > ftp
> > > site:
> > > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > Queue: met_help
> > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > Owner: Nobody
> > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET-HELP:
> > > > > >
> > > > > > Sorry to bug you a question, but I am trying to use MODE
and MTD
> > > > > > with some model inputs that have unusual map projections
(i.e.
> the
> > > > "poles"
> > > > > > not being where they are usually; or "rotated pole" in the
way we
> > > > > > called
> > > > > them here).
> > > > > > MODE, MTD and plot_data_plane do not seem wants to read
them even
> > > > > > after xy dependent latitude and longitude information are
added
> to
> > > > > > the
> > > > > netcdf file.
> > > > > > It gives errors like:
> > > > > >
> > > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc"
> not
> > > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid() ->
> > > > > > Couldn't figure out projection from information in netCDF
file.
> > > > > >
> > > > > > In the past, we got around this problem by regridding the
data to
> > > > > > regular lat-lon; due to scientific and technical reasons,
we wish
> > > > > > to
> > > > > avoid that.
> > > > > > Are there any solutions to directly work with the
projected data?
> > > > > >
> > > > > > Thank you and Best Regards,
> > > > > > Steven
> > > > > >
> > > > > >
=============================================================
> > > > > > Steven C Chan, Visiting Scientist from Newcastle
University Met
> > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1 3PB
> United
> > > > > > Kingdom
> > > > > > Tel: +44 (0)1392 886153
> > > > > > Email: steven.chan at metoffice.gov.uk Website:
>
www.metoffice.gov.uk<http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > <http://www.metoffice.gov.uk>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Julie Prestopnik
Time: Mon Apr 29 10:25:33 2019
Hi Steven.
Thank you for sending this information. After seeing that "python-
config
--ldflags" has no path information, I think that we likely don't need
to
set CPPFLAGS and LDFLAGS. For some systems, running `python-config
--ldflags` has all of the necessary information, but in this case that
is
not the case. So, we need to dig a little deeper in order to
determine
what MET_PYTHON_LD should be set to.
Since the output of `python-config --cflags` provides the directory
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
it is likely that the directory needed for "-L" is
"/opt/scitools/environments/default_legacy/2019_02_27/lib". Can you
please
take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib and
see if a libpython2.7.a or a libpython2.7.so exists in that directory?
If it does, please set the following:
(bash)
export MET_PYTHON_CC=`python-config --cflags`
export
MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
(csh)
setenv MET_PYTHON_CC `python-config --cflags`
setenv MET_PYTHON_LD
-L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
-lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
If not, please try to find the ".a" or ".so" file location and replace
the
directory after "-L" above with the correct location.
Please do this in a clean shell so that LDFLAGS and CPPFLAGS are no
longer
set. Please set the other necessary MET environment variables and
then
configure and compile MET. If that still does not work, please send
the
error that you are experiencing along with the config.log file and the
make.log file from the build.
Plesae let us know if you have any questions.
Thanks,
Julie
On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
> Thank you very much for your email. It does not seem to work for me,
but
> perhaps I am doing something wrong?
>
>
> In particular, should the new LDFLAGS options point to where
> libpython2.7.a is? My "python-config --ldflags" has no path
information,
> but "python-config --cflags" does have path information.
>
>
> i.e.
>
> [schan at eld277:schan] $ python-config --cflags
>
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> -Wstrict-prototypes
> [schan at eld277:schan] $ python-config --ldflags
> -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
>
> Best Regards
> Steven
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 26 April 2019 15:45:42
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> thought I'd jump in here with some further information. Thank you
for
> describing what is going on.
>
> I think there is potentially a better way to fix this issue. Using
"rpath"
> in compiling will embed a specific directory into the runtime
library
> search path. It takes precedence over a user's LD_LIBRARY_PATH and
might
> be worthwhile to try.
>
> Before configuring and compiling MET, you would want to set CPPFLAGS
to the
> location of where the python header files are located (likely the
the
> output of `python-config --cflags`) and set LDFLAGS (using rpath as
shown
> below: "-Wl,-rpath,") to the location of the python library
directory (the
> -L part of `python-config --ldflags`). For example:
>
> export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
>
> After setting those environment variables, along with the other
environment
> variables necessary for the compilation of MET, you would then run
> configure and then "make install".
>
> Please let us know how it goes if you decide to give that a try.
>
> Julie
>
> On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> >
> > Yes, indeed there are multiple version of Python installed here --
a
> > default "bad" version that get installed part of RHEL and another
"good"
> > version that is only accessible via a module.
> >
> >
> > The actual problem is actually solved now with some caveats. As
according
> > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> with
> > the "good" version of the Python library.
> >
> >
> > At one time during runtime, somehow computer is still trying to
load the
> > "bad" one because the "bad" one has the libraries in one of the
paths
> under
> > LD_LIBRARY_PATH. The interesting thing is that MET actually loads
the
> > "good" version of the Python executable (as indicated by
sys.executable),
> > but either Python or MET and then decides to run off to go use the
bad
> > library in LD_LIBRARY_PATH instead of the library that comes with
the
> > Python indicated by sys.executable. Basically, it can't tell which
Python
> > library is the good or the bad one during runtime.
> >
> >
> > Attempts to remove that specific path from LD_LIBRARY_PATH leads
to
> > unexpected behaviour of other software which are not related to
MET (like
> > emacs and evince will cause errors and warnings) (the violating
item is
> > something silly like /usr/lib -- i.e. you can't really remove it).
The
> > solution now is basically every MET tool has to run through a
shell
> script
> > which explicitly remove items from LD_LIBRARY_PATH that are
incompatible
> > with MET. Perhaps there are better way to fix this, but at least
it is
> > working now.
> >
> >
> > Best Regards,
> >
> > Steven Chan
> >
> > ________________________________
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 25 April 2019 21:48:33
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven,
> >
> > It sounds like you're wondering how to configure your environment
to use
> > one version of python versus another.
> >
> > While adding support for calling python scripts has solved some
problems
> in
> > MET, it has also given rise to new ones. And this is one of them!
> >
> > Unfortunately, I believe this needs to be handled when compiling
MET. In
> > particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
> >
> > MET_PYTHON_CC
> > Where the python header files are located, e.g.
-I<include
> > dir>.
> > Set to the output of 'python-config --cflags'.
> > MET_PYTHON_LD
> > Where the python library files are located, e.g.
-L<lib
> dir>,
> > followed by the python library names, e.g.
-l<library>. Set
> > to
> > the output of 'python-config --ldflags'.
> >
> > It sounds like you have a "good" version of python around and a
"bad"
> > version of python. If the directory where MET was compiled is
available
> on
> > your system, you could check "config.log" to see how those were
set at
> > compilation time. For example:
> >
> > *grep MET_PYTHON_ config.log*
> > ac_cv_env_MET_PYTHON_CC_set=set
> > ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > ac_cv_env_MET_PYTHON_LD_set=set
> > ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
> > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> >
> > And check to see if you're compiling against your preferred
version of
> > python.
> >
> > Hope that helps.
> >
> > John
> >
> > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > >
> > >
> > > One additional question:
> > >
> > > I notice from tests that paths to Python interpreters and
modules are
> > > inconsistent. This is quite visible here because we have
multiple
> > versions
> > > (and mutually incompatible) of Python interpreters and modules
> installed
> > > that are managed using shell "module load" command. For example:
> > >
> > >
> > >
> > > import sys
> > > print('Using: ',sys.executable)
> > > Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > >
> > > import csv
> > >
> > > import numpy as np
> > >
> > > import os
> > >
> > > print csv
> > >
> > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > >
> > > print np
> > >
> > > <module 'numpy' from
> > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > >
> > > print os
> > >
> > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > >
> > >
> > >
> > > There is a separate installation of python modules under
> > > /opt/scitools/environments/default_legacy/2019_02_27/; however,
for
> some
> > > reasons, the executable decides to go find modules from a
different
> place
> > > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above is
> > > actually generating errors with actual tests of using MTD
through the
> > > Python interface because the correct Python modules cannot be
found,
> and
> > I
> > > tried hand-fixing sys.path but it does not work as well.
> > >
> > >
> > > Best Regards,
> > >
> > > Steven
> > >
> > >
> > >
> > > ________________________________
> > >
> > > From: Chan, Steven
> > > Sent: 11 April 2019 14:09:04
> > > To: met_help at ucar.edu
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > >
> > >
> > > Dear John:
> > >
> > >
> > >
> > > Please forgive me for the number of emails. This is about the
original
> > > email you sent to me last year about rotated pole issue. In fact
the
> > > question has nothing to do with rotated pole but with the below:
> > >
> > >
> > >
> > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > 'name="read_nccf_rotll.py
> > > schan_eurocordex.nc precipitation_flux";'
> > >
> > >
> > >
> > > For some MET tools, more than one input files are required (for
example
> > > "mtd"). How would the Python interface work under that
particular
> > > situation? In other words, how would one actually loop through a
list
> of
> > > files?
> > >
> > >
> > >
> > > Particularly for mtd: -fcst/-single/-obs must be followed by a
list of
> > > files or a file with list of files. If I understood the manual
> > correctly, I
> > > interpreted that as repeating the string "PYTHON_NUMPY", so
there will
> > be a
> > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> Please
> > > correct me if I am wrong.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Steven Chan
> > >
> > > ________________________________
> > >
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 21 September 2018 19:30:00
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > >
> > >
> > > Steven,
> > >
> > > Thanks for sending the sample data. As you already know, I
confirmed
> > that
> > > MET can't read these data files as-is. I get the following
error from
> > > plot_data_plane:
> > >
> > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> > > from information in netCDF file.
> > >
> > > So we need to add support for reading the rotated lat/lon
projection
> > > information from CF-compliant NetCDF files. I was keeping my
fingers
> > > crossed that it'd be very simple to do, but it is not. It'll
take some
> > > effort to make sure we do it correctly. And with our release
next
> week,
> > we
> > > don't have enough time to fit it in. But I did create a
development
> > ticket
> > > for this for the next release.
> > >
> > > However, MET will still not be able to read the variable mesh
lat/lon
> > > data. We use the grid/projection information from the input
data files
> > to
> > > instantiate a regular grid in the MET libraries. That enables
MET to
> > > convert from grid x/y to lat/lon and vice-versa. And for this
reason,
> > MET
> > > doesn't currently support a variable mesh. Its possible that
could
> > change
> > > in the future depending on requirements from our funding
agencies.
> > >
> > > met-8.0 is coming out next week and includes support for calling
a
> python
> > > 2.7 script... and we can read rotated lat/lon data down that
route. I
> > ran
> > > the following command using the development version of MET:
> > >
> > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > 'name="read_nccf_rotll.py
> > > schan_eurocordex.nc precipitation_flux";'
> > >
> > > In this command, I have...
> > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > "PYTHON_NUMPY".
> > > (2) Set 'name="...";' where "..." is the python script with
associated
> > > command line arguments to be run.
> > >
> > > I've attached the python script to illustrate how it's reading
the
> first
> > > time-slice of the gridded data and setting up the metadata. We
would
> > > obviously want a more robust python script than this simple one.
> > >
> > > And I've attached the resulting plot to see if MET is putting
the data
> at
> > > the right spot on earth. Looks reasonable to me.
> > >
> > > This demonstrates that there is a way forward with met-8.0 until
we get
> > > support for rotated lat/lon projections in CF-compliant files
added.
> > >
> > > Hope this helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > I have uploaded two files to the FTP:
> > > >
> > > > /incoming/irap/met_help/StevenChan_data/
> > > > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > >
> > > > They are both rotated pole, but the resolution of the files
are
> > different
> > > > with one of the files have variable resolution (near the
boundaries).
> > > >
> > > > Please let me know if the files are okay.
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 20 September 2018 18:12
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > Thanks for the update. We're down to crunch time here getting
the
> > > met-8.0
> > > > release finalized. Once I get my hands on the sample file,
I'll do
> my
> > > best
> > > > to get it working.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > Line manager said it is fine. I am just waiting for Sys
Admin to
> > > > > actually make that possible through the Met Office firewall
(a
> > > > > technical issue). I will try to get that resolved before the
end of
> > > this
> > > > week.
> > > > >
> > > > > Thank you
> > > > > Steven
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 18 September 2018 18:27
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > OK, please let us know. The development version for met-8.0
> doesn't
> > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > CF-compliant NetCDF files since we don't have any examples.
The
> > GRIB2
> > > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > > unified model... which they call GALWEM.
> > > > >
> > > > > If we can get sample data soon enough, hopefully we can get
it
> added
> > > > > prior to the met-8.0 release.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > Thank you very much for your reply. Yes, it is rotated
lat-lon
> > grid,
> > > > > > and the files are created supposedly according to CF
convention
> > > > > > using Met Office python libraries; hence reading the data
with
> the
> > > > > > help with Python will be extremely helpful function.
> > > > > >
> > > > > > I will need to confirm with line manager if it is possible
to
> send
> > a
> > > > > > data sample as Met Office has a lot of rules in sharing
data. If
> > > > > > that is not possible, I may be able to I can get you
fictitious
> > data
> > > > > > with the right coordinate information. I will get back to
you
> with
> > > > > > this before the end of this week (line manager is out of
town
> right
> > > > now).
> > > > > >
> > > > > > Best Regards,
> > > > > > Steven Chan
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 18 September 2018 16:35
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > I see that MET is not able to handle the projection on
which your
> > > > > > data lives. We are currently working on the next version
of MET,
> > > > met-8.0.
> > > > > > We've passed the code freeze date and are now in testing.
One of
> > > > > > the features we added was support for rotated lat/lon
grids.
> Based
> > > > > > on your description, it sounds like that's what you have.
> met-8.0
> > > > > > is currently able to read rotated lat/lon data from GRIB2
files,
> > but
> > > > > > we've been looking for other rotated lat/lon projection
examples.
> > > > > > However, I suspect we may run into other technical issues
with
> your
> > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > convention,
> > > > > which MET supports?
> > > > > >
> > > > > > Another enhancement in met-8.0 is support for calling
python
> > scripts
> > > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > > flavor of NetCDF you have, you could write a python script
to
> read
> > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> When
> > > > > > you run MODE, instead of telling it to read a NetCDF input
file,
> > you
> > > > > > can tell it to execute a python script. Then it reads the
> gridded
> > > > > > field of data from that python script in memory. We hope
that
> this
> > > > > > feature will make it possible to run MET on a wider
variety of
> data
> > > > formats.
> > > > > >
> > > > > > Can you please send us a sample rotated lat/lon NetCDF
file so
> that
> > > > > > we can take a look? Here's how you can post data to our
> anonymous
> > > ftp
> > > > site:
> > > > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > >
> > > > > > Thanks,
> > > > > > John Halley Gotway
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > > Queue: met_help
> > > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > > Owner: Nobody
> > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Dear MET-HELP:
> > > > > > >
> > > > > > > Sorry to bug you a question, but I am trying to use MODE
and
> MTD
> > > > > > > with some model inputs that have unusual map projections
(i.e.
> > the
> > > > > "poles"
> > > > > > > not being where they are usually; or "rotated pole" in
the way
> we
> > > > > > > called
> > > > > > them here).
> > > > > > > MODE, MTD and plot_data_plane do not seem wants to read
them
> even
> > > > > > > after xy dependent latitude and longitude information
are added
> > to
> > > > > > > the
> > > > > > netcdf file.
> > > > > > > It gives errors like:
> > > > > > >
> > > > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc"
> > not
> > > > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid()
->
> > > > > > > Couldn't figure out projection from information in
netCDF file.
> > > > > > >
> > > > > > > In the past, we got around this problem by regridding
the data
> to
> > > > > > > regular lat-lon; due to scientific and technical
reasons, we
> wish
> > > > > > > to
> > > > > > avoid that.
> > > > > > > Are there any solutions to directly work with the
projected
> data?
> > > > > > >
> > > > > > > Thank you and Best Regards,
> > > > > > > Steven
> > > > > > >
> > > > > > >
=============================================================
> > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University Met
> > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1
3PB
> > United
> > > > > > > Kingdom
> > > > > > > Tel: +44 (0)1392 886153
> > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> > www.metoffice.gov.uk<http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > <http://www.metoffice.gov.uk>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Tue Apr 30 10:51:34 2019
Dear Julie:
Thank you for your email.
I can confirm there is an indeed a libpython2.7.so in the directory
you mention. However, making the export command changes do not resolve
the problem; ironically, the fudging the value of LD_LIBRARY_PATH just
before runtime still seems to get around the problem.
About libpython2.7.a: It is located in a different location:
/opt/scitools/environments/default_legacy/2019_02_27/lib/python2.7/config/.
I tried to include that path into -L to MET_PYTHON_LD as well, but it
does not resolve the problem.
Best Regards
Steven
________________________________
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: 29 April 2019 17:25:33
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven.
Thank you for sending this information. After seeing that "python-
config
--ldflags" has no path information, I think that we likely don't need
to
set CPPFLAGS and LDFLAGS. For some systems, running `python-config
--ldflags` has all of the necessary information, but in this case that
is
not the case. So, we need to dig a little deeper in order to
determine
what MET_PYTHON_LD should be set to.
Since the output of `python-config --cflags` provides the directory
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
it is likely that the directory needed for "-L" is
"/opt/scitools/environments/default_legacy/2019_02_27/lib". Can you
please
take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib and
see if a libpython2.7.a or a libpython2.7.so exists in that directory?
If it does, please set the following:
/opt/scitools/environments/${ScitoolsSuperGroup}/${ScitoolsVersion}/
(bash)
export MET_PYTHON_CC=`python-config --cflags`
export
MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
(csh)
setenv MET_PYTHON_CC `python-config --cflags`
setenv MET_PYTHON_LD
-L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
-lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
If not, please try to find the ".a" or ".so" file location and replace
the
directory after "-L" above with the correct location.
Please do this in a clean shell so that LDFLAGS and CPPFLAGS are no
longer
set. Please set the other necessary MET environment variables and
then
configure and compile MET. If that still does not work, please send
the
error that you are experiencing along with the config.log file and the
make.log file from the build.
Plesae let us know if you have any questions.
Thanks,
Julie
On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
> Thank you very much for your email. It does not seem to work for me,
but
> perhaps I am doing something wrong?
>
>
> In particular, should the new LDFLAGS options point to where
> libpython2.7.a is? My "python-config --ldflags" has no path
information,
> but "python-config --cflags" does have path information.
>
>
> i.e.
>
> [schan at eld277:schan] $ python-config --cflags
>
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> -Wstrict-prototypes
> [schan at eld277:schan] $ python-config --ldflags
> -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
>
> Best Regards
> Steven
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 26 April 2019 15:45:42
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> thought I'd jump in here with some further information. Thank you
for
> describing what is going on.
>
> I think there is potentially a better way to fix this issue. Using
"rpath"
> in compiling will embed a specific directory into the runtime
library
> search path. It takes precedence over a user's LD_LIBRARY_PATH and
might
> be worthwhile to try.
>
> Before configuring and compiling MET, you would want to set CPPFLAGS
to the
> location of where the python header files are located (likely the
the
> output of `python-config --cflags`) and set LDFLAGS (using rpath as
shown
> below: "-Wl,-rpath,") to the location of the python library
directory (the
> -L part of `python-config --ldflags`). For example:
>
> export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
>
> After setting those environment variables, along with the other
environment
> variables necessary for the compilation of MET, you would then run
> configure and then "make install".
>
> Please let us know how it goes if you decide to give that a try.
>
> Julie
>
> On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear John:
> >
> >
> > Yes, indeed there are multiple version of Python installed here --
a
> > default "bad" version that get installed part of RHEL and another
"good"
> > version that is only accessible via a module.
> >
> >
> > The actual problem is actually solved now with some caveats. As
according
> > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> with
> > the "good" version of the Python library.
> >
> >
> > At one time during runtime, somehow computer is still trying to
load the
> > "bad" one because the "bad" one has the libraries in one of the
paths
> under
> > LD_LIBRARY_PATH. The interesting thing is that MET actually loads
the
> > "good" version of the Python executable (as indicated by
sys.executable),
> > but either Python or MET and then decides to run off to go use the
bad
> > library in LD_LIBRARY_PATH instead of the library that comes with
the
> > Python indicated by sys.executable. Basically, it can't tell which
Python
> > library is the good or the bad one during runtime.
> >
> >
> > Attempts to remove that specific path from LD_LIBRARY_PATH leads
to
> > unexpected behaviour of other software which are not related to
MET (like
> > emacs and evince will cause errors and warnings) (the violating
item is
> > something silly like /usr/lib -- i.e. you can't really remove it).
The
> > solution now is basically every MET tool has to run through a
shell
> script
> > which explicitly remove items from LD_LIBRARY_PATH that are
incompatible
> > with MET. Perhaps there are better way to fix this, but at least
it is
> > working now.
> >
> >
> > Best Regards,
> >
> > Steven Chan
> >
> > ________________________________
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 25 April 2019 21:48:33
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven,
> >
> > It sounds like you're wondering how to configure your environment
to use
> > one version of python versus another.
> >
> > While adding support for calling python scripts has solved some
problems
> in
> > MET, it has also given rise to new ones. And this is one of them!
> >
> > Unfortunately, I believe this needs to be handled when compiling
MET. In
> > particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
> >
> > MET_PYTHON_CC
> > Where the python header files are located, e.g.
-I<include
> > dir>.
> > Set to the output of 'python-config --cflags'.
> > MET_PYTHON_LD
> > Where the python library files are located, e.g.
-L<lib
> dir>,
> > followed by the python library names, e.g.
-l<library>. Set
> > to
> > the output of 'python-config --ldflags'.
> >
> > It sounds like you have a "good" version of python around and a
"bad"
> > version of python. If the directory where MET was compiled is
available
> on
> > your system, you could check "config.log" to see how those were
set at
> > compilation time. For example:
> >
> > *grep MET_PYTHON_ config.log*
> > ac_cv_env_MET_PYTHON_CC_set=set
> > ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > ac_cv_env_MET_PYTHON_LD_set=set
> > ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7 -lpthread
-ldl
> > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> >
> > And check to see if you're compiling against your preferred
version of
> > python.
> >
> > Hope that helps.
> >
> > John
> >
> > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > >
> > >
> > > One additional question:
> > >
> > > I notice from tests that paths to Python interpreters and
modules are
> > > inconsistent. This is quite visible here because we have
multiple
> > versions
> > > (and mutually incompatible) of Python interpreters and modules
> installed
> > > that are managed using shell "module load" command. For example:
> > >
> > >
> > >
> > > import sys
> > > print('Using: ',sys.executable)
> > > Using:
/opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > >
> > > import csv
> > >
> > > import numpy as np
> > >
> > > import os
> > >
> > > print csv
> > >
> > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > >
> > > print np
> > >
> > > <module 'numpy' from
> > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > >
> > > print os
> > >
> > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > >
> > >
> > >
> > > There is a separate installation of python modules under
> > > /opt/scitools/environments/default_legacy/2019_02_27/; however,
for
> some
> > > reasons, the executable decides to go find modules from a
different
> place
> > > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above is
> > > actually generating errors with actual tests of using MTD
through the
> > > Python interface because the correct Python modules cannot be
found,
> and
> > I
> > > tried hand-fixing sys.path but it does not work as well.
> > >
> > >
> > > Best Regards,
> > >
> > > Steven
> > >
> > >
> > >
> > > ________________________________
> > >
> > > From: Chan, Steven
> > > Sent: 11 April 2019 14:09:04
> > > To: met_help at ucar.edu
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > >
> > >
> > > Dear John:
> > >
> > >
> > >
> > > Please forgive me for the number of emails. This is about the
original
> > > email you sent to me last year about rotated pole issue. In fact
the
> > > question has nothing to do with rotated pole but with the below:
> > >
> > >
> > >
> > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > 'name="read_nccf_rotll.py
> > > schan_eurocordex.nc precipitation_flux";'
> > >
> > >
> > >
> > > For some MET tools, more than one input files are required (for
example
> > > "mtd"). How would the Python interface work under that
particular
> > > situation? In other words, how would one actually loop through a
list
> of
> > > files?
> > >
> > >
> > >
> > > Particularly for mtd: -fcst/-single/-obs must be followed by a
list of
> > > files or a file with list of files. If I understood the manual
> > correctly, I
> > > interpreted that as repeating the string "PYTHON_NUMPY", so
there will
> > be a
> > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> Please
> > > correct me if I am wrong.
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Steven Chan
> > >
> > > ________________________________
> > >
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 21 September 2018 19:30:00
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > >
> > >
> > > Steven,
> > >
> > > Thanks for sending the sample data. As you already know, I
confirmed
> > that
> > > MET can't read these data files as-is. I get the following
error from
> > > plot_data_plane:
> > >
> > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection
> > > from information in netCDF file.
> > >
> > > So we need to add support for reading the rotated lat/lon
projection
> > > information from CF-compliant NetCDF files. I was keeping my
fingers
> > > crossed that it'd be very simple to do, but it is not. It'll
take some
> > > effort to make sure we do it correctly. And with our release
next
> week,
> > we
> > > don't have enough time to fit it in. But I did create a
development
> > ticket
> > > for this for the next release.
> > >
> > > However, MET will still not be able to read the variable mesh
lat/lon
> > > data. We use the grid/projection information from the input
data files
> > to
> > > instantiate a regular grid in the MET libraries. That enables
MET to
> > > convert from grid x/y to lat/lon and vice-versa. And for this
reason,
> > MET
> > > doesn't currently support a variable mesh. Its possible that
could
> > change
> > > in the future depending on requirements from our funding
agencies.
> > >
> > > met-8.0 is coming out next week and includes support for calling
a
> python
> > > 2.7 script... and we can read rotated lat/lon data down that
route. I
> > ran
> > > the following command using the development version of MET:
> > >
> > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > 'name="read_nccf_rotll.py
> > > schan_eurocordex.nc precipitation_flux";'
> > >
> > > In this command, I have...
> > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > "PYTHON_NUMPY".
> > > (2) Set 'name="...";' where "..." is the python script with
associated
> > > command line arguments to be run.
> > >
> > > I've attached the python script to illustrate how it's reading
the
> first
> > > time-slice of the gridded data and setting up the metadata. We
would
> > > obviously want a more robust python script than this simple one.
> > >
> > > And I've attached the resulting plot to see if MET is putting
the data
> at
> > > the right spot on earth. Looks reasonable to me.
> > >
> > > This demonstrates that there is a way forward with met-8.0 until
we get
> > > support for rotated lat/lon projections in CF-compliant files
added.
> > >
> > > Hope this helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > > I have uploaded two files to the FTP:
> > > >
> > > > /incoming/irap/met_help/StevenChan_data/
> > > > schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > >
> > > > They are both rotated pole, but the resolution of the files
are
> > different
> > > > with one of the files have variable resolution (near the
boundaries).
> > > >
> > > > Please let me know if the files are okay.
> > > >
> > > > Thank you and Best Regards,
> > > > Steven
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 20 September 2018 18:12
> > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Steven,
> > > >
> > > > Thanks for the update. We're down to crunch time here getting
the
> > > met-8.0
> > > > release finalized. Once I get my hands on the sample file,
I'll do
> my
> > > best
> > > > to get it working.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > Line manager said it is fine. I am just waiting for Sys
Admin to
> > > > > actually make that possible through the Met Office firewall
(a
> > > > > technical issue). I will try to get that resolved before the
end of
> > > this
> > > > week.
> > > > >
> > > > > Thank you
> > > > > Steven
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 18 September 2018 18:27
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > OK, please let us know. The development version for met-8.0
> doesn't
> > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > CF-compliant NetCDF files since we don't have any examples.
The
> > GRIB2
> > > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > > unified model... which they call GALWEM.
> > > > >
> > > > > If we can get sample data soon enough, hopefully we can get
it
> added
> > > > > prior to the met-8.0 release.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > Thank you very much for your reply. Yes, it is rotated
lat-lon
> > grid,
> > > > > > and the files are created supposedly according to CF
convention
> > > > > > using Met Office python libraries; hence reading the data
with
> the
> > > > > > help with Python will be extremely helpful function.
> > > > > >
> > > > > > I will need to confirm with line manager if it is possible
to
> send
> > a
> > > > > > data sample as Met Office has a lot of rules in sharing
data. If
> > > > > > that is not possible, I may be able to I can get you
fictitious
> > data
> > > > > > with the right coordinate information. I will get back to
you
> with
> > > > > > this before the end of this week (line manager is out of
town
> right
> > > > now).
> > > > > >
> > > > > > Best Regards,
> > > > > > Steven Chan
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 18 September 2018 16:35
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > I see that MET is not able to handle the projection on
which your
> > > > > > data lives. We are currently working on the next version
of MET,
> > > > met-8.0.
> > > > > > We've passed the code freeze date and are now in testing.
One of
> > > > > > the features we added was support for rotated lat/lon
grids.
> Based
> > > > > > on your description, it sounds like that's what you have.
> met-8.0
> > > > > > is currently able to read rotated lat/lon data from GRIB2
files,
> > but
> > > > > > we've been looking for other rotated lat/lon projection
examples.
> > > > > > However, I suspect we may run into other technical issues
with
> your
> > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > convention,
> > > > > which MET supports?
> > > > > >
> > > > > > Another enhancement in met-8.0 is support for calling
python
> > scripts
> > > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > > flavor of NetCDF you have, you could write a python script
to
> read
> > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> When
> > > > > > you run MODE, instead of telling it to read a NetCDF input
file,
> > you
> > > > > > can tell it to execute a python script. Then it reads the
> gridded
> > > > > > field of data from that python script in memory. We hope
that
> this
> > > > > > feature will make it possible to run MET on a wider
variety of
> data
> > > > formats.
> > > > > >
> > > > > > Can you please send us a sample rotated lat/lon NetCDF
file so
> that
> > > > > > we can take a look? Here's how you can post data to our
> anonymous
> > > ftp
> > > > site:
> > > > > > https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > >
> > > > > > Thanks,
> > > > > > John Halley Gotway
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted upon.
> > > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > > Queue: met_help
> > > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > > Owner: Nobody
> > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Dear MET-HELP:
> > > > > > >
> > > > > > > Sorry to bug you a question, but I am trying to use MODE
and
> MTD
> > > > > > > with some model inputs that have unusual map projections
(i.e.
> > the
> > > > > "poles"
> > > > > > > not being where they are usually; or "rotated pole" in
the way
> we
> > > > > > > called
> > > > > > them here).
> > > > > > > MODE, MTD and plot_data_plane do not seem wants to read
them
> even
> > > > > > > after xy dependent latitude and longitude information
are added
> > to
> > > > > > > the
> > > > > > netcdf file.
> > > > > > > It gives errors like:
> > > > > > >
> > > > > > > ERROR : plot_data_plane -> file "mi-
ag192_h_precip_200001.nc"
> > not
> > > > > > > a valid data file ERROR : NcCfFile::read_netcdf_grid()
->
> > > > > > > Couldn't figure out projection from information in
netCDF file.
> > > > > > >
> > > > > > > In the past, we got around this problem by regridding
the data
> to
> > > > > > > regular lat-lon; due to scientific and technical
reasons, we
> wish
> > > > > > > to
> > > > > > avoid that.
> > > > > > > Are there any solutions to directly work with the
projected
> data?
> > > > > > >
> > > > > > > Thank you and Best Regards,
> > > > > > > Steven
> > > > > > >
> > > > > > >
=============================================================
> > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University Met
> > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1
3PB
> > United
> > > > > > > Kingdom
> > > > > > > Tel: +44 (0)1392 886153
> > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> >
www.metoffice.gov.uk<http://www.metoffice.gov.uk<<http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<>
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > <http://www.metoffice.gov.uk>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Julie Prestopnik
Time: Tue Apr 30 11:23:30 2019
Hi Steven.
Could you please send the error that you are experiencing along with
the
config.log file and the make.log file from both builds (where you
point to
libpython2.7.so and also libpython2.7.a)?
Or, because you have a work around with the fudge to the value of
LD_LIBRARY_PATH, would you prefer to consider the problem resolved?
Thanks!
Julie
On Tue, Apr 30, 2019 at 10:52 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
>
> Thank you for your email.
>
>
> I can confirm there is an indeed a libpython2.7.so in the directory
you
> mention. However, making the export command changes do not resolve
the
> problem; ironically, the fudging the value of LD_LIBRARY_PATH just
before
> runtime still seems to get around the problem.
>
>
> About libpython2.7.a: It is located in a different location:
>
>
/opt/scitools/environments/default_legacy/2019_02_27/lib/python2.7/config/.
> I tried to include that path into -L to MET_PYTHON_LD as well, but
it does
> not resolve the problem.
>
>
> Best Regards
>
> Steven
>
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 29 April 2019 17:25:33
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven.
>
> Thank you for sending this information. After seeing that "python-
config
> --ldflags" has no path information, I think that we likely don't
need to
> set CPPFLAGS and LDFLAGS. For some systems, running `python-config
> --ldflags` has all of the necessary information, but in this case
that is
> not the case. So, we need to dig a little deeper in order to
determine
> what MET_PYTHON_LD should be set to.
>
> Since the output of `python-config --cflags` provides the directory
>
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
> it is likely that the directory needed for "-L" is
> "/opt/scitools/environments/default_legacy/2019_02_27/lib". Can you
please
> take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib and
> see if a libpython2.7.a or a libpython2.7.so exists in that
directory?
>
> If it does, please set the following:
> /opt/scitools/environments/${ScitoolsSuperGroup}/${ScitoolsVersion}/
> (bash)
> export MET_PYTHON_CC=`python-config --cflags`
> export
> MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
> -lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
>
> (csh)
> setenv MET_PYTHON_CC `python-config --cflags`
> setenv MET_PYTHON_LD
> -L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
> -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
>
> If not, please try to find the ".a" or ".so" file location and
replace the
> directory after "-L" above with the correct location.
>
> Please do this in a clean shell so that LDFLAGS and CPPFLAGS are no
longer
> set. Please set the other necessary MET environment variables and
then
> configure and compile MET. If that still does not work, please send
the
> error that you are experiencing along with the config.log file and
the
> make.log file from the build.
>
> Plesae let us know if you have any questions.
>
> Thanks,
> Julie
>
> On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear Julie:
> >
> > Thank you very much for your email. It does not seem to work for
me, but
> > perhaps I am doing something wrong?
> >
> >
> > In particular, should the new LDFLAGS options point to where
> > libpython2.7.a is? My "python-config --ldflags" has no path
information,
> > but "python-config --cflags" does have path information.
> >
> >
> > i.e.
> >
> > [schan at eld277:schan] $ python-config --cflags
> >
> >
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> >
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> > -Wstrict-prototypes
> > [schan at eld277:schan] $ python-config --ldflags
> > -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
> >
> > Best Regards
> > Steven
> >
> > ________________________________
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: 26 April 2019 15:45:42
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> > thought I'd jump in here with some further information. Thank you
for
> > describing what is going on.
> >
> > I think there is potentially a better way to fix this issue. Using
> "rpath"
> > in compiling will embed a specific directory into the runtime
library
> > search path. It takes precedence over a user's LD_LIBRARY_PATH
and might
> > be worthwhile to try.
> >
> > Before configuring and compiling MET, you would want to set
CPPFLAGS to
> the
> > location of where the python header files are located (likely the
the
> > output of `python-config --cflags`) and set LDFLAGS (using rpath
as shown
> > below: "-Wl,-rpath,") to the location of the python library
directory
> (the
> > -L part of `python-config --ldflags`). For example:
> >
> > export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> > export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
> >
> > After setting those environment variables, along with the other
> environment
> > variables necessary for the compilation of MET, you would then run
> > configure and then "make install".
> >
> > Please let us know how it goes if you decide to give that a try.
> >
> > Julie
> >
> > On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > >
> > > Yes, indeed there are multiple version of Python installed here
-- a
> > > default "bad" version that get installed part of RHEL and
another
> "good"
> > > version that is only accessible via a module.
> > >
> > >
> > > The actual problem is actually solved now with some caveats. As
> according
> > > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> > with
> > > the "good" version of the Python library.
> > >
> > >
> > > At one time during runtime, somehow computer is still trying to
load
> the
> > > "bad" one because the "bad" one has the libraries in one of the
paths
> > under
> > > LD_LIBRARY_PATH. The interesting thing is that MET actually
loads the
> > > "good" version of the Python executable (as indicated by
> sys.executable),
> > > but either Python or MET and then decides to run off to go use
the bad
> > > library in LD_LIBRARY_PATH instead of the library that comes
with the
> > > Python indicated by sys.executable. Basically, it can't tell
which
> Python
> > > library is the good or the bad one during runtime.
> > >
> > >
> > > Attempts to remove that specific path from LD_LIBRARY_PATH leads
to
> > > unexpected behaviour of other software which are not related to
MET
> (like
> > > emacs and evince will cause errors and warnings) (the violating
item is
> > > something silly like /usr/lib -- i.e. you can't really remove
it). The
> > > solution now is basically every MET tool has to run through a
shell
> > script
> > > which explicitly remove items from LD_LIBRARY_PATH that are
> incompatible
> > > with MET. Perhaps there are better way to fix this, but at least
it is
> > > working now.
> > >
> > >
> > > Best Regards,
> > >
> > > Steven Chan
> > >
> > > ________________________________
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 25 April 2019 21:48:33
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Hi Steven,
> > >
> > > It sounds like you're wondering how to configure your
environment to
> use
> > > one version of python versus another.
> > >
> > > While adding support for calling python scripts has solved some
> problems
> > in
> > > MET, it has also given rise to new ones. And this is one of
them!
> > >
> > > Unfortunately, I believe this needs to be handled when compiling
MET.
> In
> > > particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
> > >
> > > MET_PYTHON_CC
> > > Where the python header files are located, e.g.
> -I<include
> > > dir>.
> > > Set to the output of 'python-config --cflags'.
> > > MET_PYTHON_LD
> > > Where the python library files are located, e.g.
-L<lib
> > dir>,
> > > followed by the python library names, e.g.
-l<library>.
> Set
> > > to
> > > the output of 'python-config --ldflags'.
> > >
> > > It sounds like you have a "good" version of python around and a
"bad"
> > > version of python. If the directory where MET was compiled is
> available
> > on
> > > your system, you could check "config.log" to see how those were
set at
> > > compilation time. For example:
> > >
> > > *grep MET_PYTHON_ config.log*
> > > ac_cv_env_MET_PYTHON_CC_set=set
> > >
> ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > > ac_cv_env_MET_PYTHON_LD_set=set
> > >
> ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7
-lpthread -ldl
> > > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > >
> > > And check to see if you're compiling against your preferred
version of
> > > python.
> > >
> > > Hope that helps.
> > >
> > > John
> > >
> > > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > >
> > > >
> > > > One additional question:
> > > >
> > > > I notice from tests that paths to Python interpreters and
modules are
> > > > inconsistent. This is quite visible here because we have
multiple
> > > versions
> > > > (and mutually incompatible) of Python interpreters and modules
> > installed
> > > > that are managed using shell "module load" command. For
example:
> > > >
> > > >
> > > >
> > > > import sys
> > > > print('Using: ',sys.executable)
> > > > Using:
> /opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > > >
> > > > import csv
> > > >
> > > > import numpy as np
> > > >
> > > > import os
> > > >
> > > > print csv
> > > >
> > > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > > >
> > > > print np
> > > >
> > > > <module 'numpy' from
> > > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > > >
> > > > print os
> > > >
> > > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > > >
> > > >
> > > >
> > > > There is a separate installation of python modules under
> > > > /opt/scitools/environments/default_legacy/2019_02_27/;
however, for
> > some
> > > > reasons, the executable decides to go find modules from a
different
> > place
> > > > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above
> is
> > > > actually generating errors with actual tests of using MTD
through the
> > > > Python interface because the correct Python modules cannot be
found,
> > and
> > > I
> > > > tried hand-fixing sys.path but it does not work as well.
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Steven
> > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > From: Chan, Steven
> > > > Sent: 11 April 2019 14:09:04
> > > > To: met_help at ucar.edu
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > >
> > > >
> > > > Dear John:
> > > >
> > > >
> > > >
> > > > Please forgive me for the number of emails. This is about the
> original
> > > > email you sent to me last year about rotated pole issue. In
fact the
> > > > question has nothing to do with rotated pole but with the
below:
> > > >
> > > >
> > > >
> > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > 'name="read_nccf_rotll.py
> > > > schan_eurocordex.nc precipitation_flux";'
> > > >
> > > >
> > > >
> > > > For some MET tools, more than one input files are required
(for
> example
> > > > "mtd"). How would the Python interface work under that
particular
> > > > situation? In other words, how would one actually loop through
a list
> > of
> > > > files?
> > > >
> > > >
> > > >
> > > > Particularly for mtd: -fcst/-single/-obs must be followed by a
list
> of
> > > > files or a file with list of files. If I understood the manual
> > > correctly, I
> > > > interpreted that as repeating the string "PYTHON_NUMPY", so
there
> will
> > > be a
> > > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> > Please
> > > > correct me if I am wrong.
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Steven Chan
> > > >
> > > > ________________________________
> > > >
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 21 September 2018 19:30:00
> > > > To: Chan, Steven
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > >
> > > >
> > > > Steven,
> > > >
> > > > Thanks for sending the sample data. As you already know, I
confirmed
> > > that
> > > > MET can't read these data files as-is. I get the following
error
> from
> > > > plot_data_plane:
> > > >
> > > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
> projection
> > > > from information in netCDF file.
> > > >
> > > > So we need to add support for reading the rotated lat/lon
projection
> > > > information from CF-compliant NetCDF files. I was keeping my
fingers
> > > > crossed that it'd be very simple to do, but it is not. It'll
take
> some
> > > > effort to make sure we do it correctly. And with our release
next
> > week,
> > > we
> > > > don't have enough time to fit it in. But I did create a
development
> > > ticket
> > > > for this for the next release.
> > > >
> > > > However, MET will still not be able to read the variable mesh
lat/lon
> > > > data. We use the grid/projection information from the input
data
> files
> > > to
> > > > instantiate a regular grid in the MET libraries. That enables
MET to
> > > > convert from grid x/y to lat/lon and vice-versa. And for this
> reason,
> > > MET
> > > > doesn't currently support a variable mesh. Its possible that
could
> > > change
> > > > in the future depending on requirements from our funding
agencies.
> > > >
> > > > met-8.0 is coming out next week and includes support for
calling a
> > python
> > > > 2.7 script... and we can read rotated lat/lon data down that
route.
> I
> > > ran
> > > > the following command using the development version of MET:
> > > >
> > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > 'name="read_nccf_rotll.py
> > > > schan_eurocordex.nc precipitation_flux";'
> > > >
> > > > In this command, I have...
> > > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > > "PYTHON_NUMPY".
> > > > (2) Set 'name="...";' where "..." is the python script with
> associated
> > > > command line arguments to be run.
> > > >
> > > > I've attached the python script to illustrate how it's reading
the
> > first
> > > > time-slice of the gridded data and setting up the metadata.
We would
> > > > obviously want a more robust python script than this simple
one.
> > > >
> > > > And I've attached the resulting plot to see if MET is putting
the
> data
> > at
> > > > the right spot on earth. Looks reasonable to me.
> > > >
> > > > This demonstrates that there is a way forward with met-8.0
until we
> get
> > > > support for rotated lat/lon projections in CF-compliant files
added.
> > > >
> > > > Hope this helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > I have uploaded two files to the FTP:
> > > > >
> > > > > /incoming/irap/met_help/StevenChan_data/
> > > > >
schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > > >
> > > > > They are both rotated pole, but the resolution of the files
are
> > > different
> > > > > with one of the files have variable resolution (near the
> boundaries).
> > > > >
> > > > > Please let me know if the files are okay.
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Steven
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 20 September 2018 18:12
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > Thanks for the update. We're down to crunch time here
getting the
> > > > met-8.0
> > > > > release finalized. Once I get my hands on the sample file,
I'll do
> > my
> > > > best
> > > > > to get it working.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > Line manager said it is fine. I am just waiting for Sys
Admin to
> > > > > > actually make that possible through the Met Office
firewall (a
> > > > > > technical issue). I will try to get that resolved before
the end
> of
> > > > this
> > > > > week.
> > > > > >
> > > > > > Thank you
> > > > > > Steven
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 18 September 2018 18:27
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > OK, please let us know. The development version for met-
8.0
> > doesn't
> > > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > > CF-compliant NetCDF files since we don't have any
examples. The
> > > GRIB2
> > > > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > > > unified model... which they call GALWEM.
> > > > > >
> > > > > > If we can get sample data soon enough, hopefully we can
get it
> > added
> > > > > > prior to the met-8.0 release.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> >
> > > > > > >
> > > > > > > Dear John:
> > > > > > >
> > > > > > > Thank you very much for your reply. Yes, it is rotated
lat-lon
> > > grid,
> > > > > > > and the files are created supposedly according to CF
convention
> > > > > > > using Met Office python libraries; hence reading the
data with
> > the
> > > > > > > help with Python will be extremely helpful function.
> > > > > > >
> > > > > > > I will need to confirm with line manager if it is
possible to
> > send
> > > a
> > > > > > > data sample as Met Office has a lot of rules in sharing
data.
> If
> > > > > > > that is not possible, I may be able to I can get you
fictitious
> > > data
> > > > > > > with the right coordinate information. I will get back
to you
> > with
> > > > > > > this before the end of this week (line manager is out of
town
> > right
> > > > > now).
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Steven Chan
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > Sent: 18 September 2018 16:35
> > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > > with
> > > > > > > projected inputs
> > > > > > >
> > > > > > > Steven,
> > > > > > >
> > > > > > > I see that MET is not able to handle the projection on
which
> your
> > > > > > > data lives. We are currently working on the next
version of
> MET,
> > > > > met-8.0.
> > > > > > > We've passed the code freeze date and are now in
testing. One
> of
> > > > > > > the features we added was support for rotated lat/lon
grids.
> > Based
> > > > > > > on your description, it sounds like that's what you
have.
> > met-8.0
> > > > > > > is currently able to read rotated lat/lon data from
GRIB2
> files,
> > > but
> > > > > > > we've been looking for other rotated lat/lon projection
> examples.
> > > > > > > However, I suspect we may run into other technical
issues with
> > your
> > > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > > convention,
> > > > > > which MET supports?
> > > > > > >
> > > > > > > Another enhancement in met-8.0 is support for calling
python
> > > scripts
> > > > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > > > flavor of NetCDF you have, you could write a python
script to
> > read
> > > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> > When
> > > > > > > you run MODE, instead of telling it to read a NetCDF
input
> file,
> > > you
> > > > > > > can tell it to execute a python script. Then it reads
the
> > gridded
> > > > > > > field of data from that python script in memory. We
hope that
> > this
> > > > > > > feature will make it possible to run MET on a wider
variety of
> > data
> > > > > formats.
> > > > > > >
> > > > > > > Can you please send us a sample rotated lat/lon NetCDF
file so
> > that
> > > > > > > we can take a look? Here's how you can post data to our
> > anonymous
> > > > ftp
> > > > > site:
> > > > > > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John Halley Gotway
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted
upon.
> > > > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > > > Queue: met_help
> > > > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Dear MET-HELP:
> > > > > > > >
> > > > > > > > Sorry to bug you a question, but I am trying to use
MODE and
> > MTD
> > > > > > > > with some model inputs that have unusual map
projections
> (i.e.
> > > the
> > > > > > "poles"
> > > > > > > > not being where they are usually; or "rotated pole" in
the
> way
> > we
> > > > > > > > called
> > > > > > > them here).
> > > > > > > > MODE, MTD and plot_data_plane do not seem wants to
read them
> > even
> > > > > > > > after xy dependent latitude and longitude information
are
> added
> > > to
> > > > > > > > the
> > > > > > > netcdf file.
> > > > > > > > It gives errors like:
> > > > > > > >
> > > > > > > > ERROR : plot_data_plane -> file "
> mi-ag192_h_precip_200001.nc"
> > > not
> > > > > > > > a valid data file ERROR :
NcCfFile::read_netcdf_grid() ->
> > > > > > > > Couldn't figure out projection from information in
netCDF
> file.
> > > > > > > >
> > > > > > > > In the past, we got around this problem by regridding
the
> data
> > to
> > > > > > > > regular lat-lon; due to scientific and technical
reasons, we
> > wish
> > > > > > > > to
> > > > > > > avoid that.
> > > > > > > > Are there any solutions to directly work with the
projected
> > data?
> > > > > > > >
> > > > > > > > Thank you and Best Regards,
> > > > > > > > Steven
> > > > > > > >
> > > > > > > >
=============================================================
> > > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University
> Met
> > > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1
3PB
> > > United
> > > > > > > > Kingdom
> > > > > > > > Tel: +44 (0)1392 886153
> > > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> > > www.metoffice.gov.uk<http://www.metoffice.gov.uk<<
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<>
> > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > > <http://www.metoffice.gov.uk>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Fri May 03 10:15:34 2019
Dear Julie:
Thank you for your email. In the tar gz files:
config.log
output of make
error message in running mtd
I did a grep, neither the string "libpython2.7.so" and
"libpython2.7.a" but you can see the paths that we have tried to add
is there.
About the fudge solution: I am personally happy for that right now.
That does not fix the problem with rotated pole problem that was
communicated in earlier emails, but it does make Python interface to
appear to work.
Best Regards,
Steven
________________________________
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: 30 April 2019 18:23:31
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven.
Could you please send the error that you are experiencing along with
the
config.log file and the make.log file from both builds (where you
point to
libpython2.7.so and also libpython2.7.a)?
Or, because you have a work around with the fudge to the value of
LD_LIBRARY_PATH, would you prefer to consider the problem resolved?
Thanks!
Julie
On Tue, Apr 30, 2019 at 10:52 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
>
> Thank you for your email.
>
>
> I can confirm there is an indeed a libpython2.7.so in the directory
you
> mention. However, making the export command changes do not resolve
the
> problem; ironically, the fudging the value of LD_LIBRARY_PATH just
before
> runtime still seems to get around the problem.
>
>
> About libpython2.7.a: It is located in a different location:
>
>
/opt/scitools/environments/default_legacy/2019_02_27/lib/python2.7/config/.
> I tried to include that path into -L to MET_PYTHON_LD as well, but
it does
> not resolve the problem.
>
>
> Best Regards
>
> Steven
>
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 29 April 2019 17:25:33
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven.
>
> Thank you for sending this information. After seeing that "python-
config
> --ldflags" has no path information, I think that we likely don't
need to
> set CPPFLAGS and LDFLAGS. For some systems, running `python-config
> --ldflags` has all of the necessary information, but in this case
that is
> not the case. So, we need to dig a little deeper in order to
determine
> what MET_PYTHON_LD should be set to.
>
> Since the output of `python-config --cflags` provides the directory
>
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
> it is likely that the directory needed for "-L" is
> "/opt/scitools/environments/default_legacy/2019_02_27/lib". Can you
please
> take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib and
> see if a libpython2.7.a or a libpython2.7.so exists in that
directory?
>
> If it does, please set the following:
> /opt/scitools/environments/${ScitoolsSuperGroup}/${ScitoolsVersion}/
> (bash)
> export MET_PYTHON_CC=`python-config --cflags`
> export
> MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
> -lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
>
> (csh)
> setenv MET_PYTHON_CC `python-config --cflags`
> setenv MET_PYTHON_LD
> -L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
> -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
>
> If not, please try to find the ".a" or ".so" file location and
replace the
> directory after "-L" above with the correct location.
>
> Please do this in a clean shell so that LDFLAGS and CPPFLAGS are no
longer
> set. Please set the other necessary MET environment variables and
then
> configure and compile MET. If that still does not work, please send
the
> error that you are experiencing along with the config.log file and
the
> make.log file from the build.
>
> Plesae let us know if you have any questions.
>
> Thanks,
> Julie
>
> On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear Julie:
> >
> > Thank you very much for your email. It does not seem to work for
me, but
> > perhaps I am doing something wrong?
> >
> >
> > In particular, should the new LDFLAGS options point to where
> > libpython2.7.a is? My "python-config --ldflags" has no path
information,
> > but "python-config --cflags" does have path information.
> >
> >
> > i.e.
> >
> > [schan at eld277:schan] $ python-config --cflags
> >
> >
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> >
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> > -Wstrict-prototypes
> > [schan at eld277:schan] $ python-config --ldflags
> > -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
> >
> > Best Regards
> > Steven
> >
> > ________________________________
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: 26 April 2019 15:45:42
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> > thought I'd jump in here with some further information. Thank you
for
> > describing what is going on.
> >
> > I think there is potentially a better way to fix this issue. Using
> "rpath"
> > in compiling will embed a specific directory into the runtime
library
> > search path. It takes precedence over a user's LD_LIBRARY_PATH
and might
> > be worthwhile to try.
> >
> > Before configuring and compiling MET, you would want to set
CPPFLAGS to
> the
> > location of where the python header files are located (likely the
the
> > output of `python-config --cflags`) and set LDFLAGS (using rpath
as shown
> > below: "-Wl,-rpath,") to the location of the python library
directory
> (the
> > -L part of `python-config --ldflags`). For example:
> >
> > export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> > export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
> >
> > After setting those environment variables, along with the other
> environment
> > variables necessary for the compilation of MET, you would then run
> > configure and then "make install".
> >
> > Please let us know how it goes if you decide to give that a try.
> >
> > Julie
> >
> > On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear John:
> > >
> > >
> > > Yes, indeed there are multiple version of Python installed here
-- a
> > > default "bad" version that get installed part of RHEL and
another
> "good"
> > > version that is only accessible via a module.
> > >
> > >
> > > The actual problem is actually solved now with some caveats. As
> according
> > > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> > with
> > > the "good" version of the Python library.
> > >
> > >
> > > At one time during runtime, somehow computer is still trying to
load
> the
> > > "bad" one because the "bad" one has the libraries in one of the
paths
> > under
> > > LD_LIBRARY_PATH. The interesting thing is that MET actually
loads the
> > > "good" version of the Python executable (as indicated by
> sys.executable),
> > > but either Python or MET and then decides to run off to go use
the bad
> > > library in LD_LIBRARY_PATH instead of the library that comes
with the
> > > Python indicated by sys.executable. Basically, it can't tell
which
> Python
> > > library is the good or the bad one during runtime.
> > >
> > >
> > > Attempts to remove that specific path from LD_LIBRARY_PATH leads
to
> > > unexpected behaviour of other software which are not related to
MET
> (like
> > > emacs and evince will cause errors and warnings) (the violating
item is
> > > something silly like /usr/lib -- i.e. you can't really remove
it). The
> > > solution now is basically every MET tool has to run through a
shell
> > script
> > > which explicitly remove items from LD_LIBRARY_PATH that are
> incompatible
> > > with MET. Perhaps there are better way to fix this, but at least
it is
> > > working now.
> > >
> > >
> > > Best Regards,
> > >
> > > Steven Chan
> > >
> > > ________________________________
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 25 April 2019 21:48:33
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Hi Steven,
> > >
> > > It sounds like you're wondering how to configure your
environment to
> use
> > > one version of python versus another.
> > >
> > > While adding support for calling python scripts has solved some
> problems
> > in
> > > MET, it has also given rise to new ones. And this is one of
them!
> > >
> > > Unfortunately, I believe this needs to be handled when compiling
MET.
> In
> > > particular, using the MET_PYTHON_CC and MET_PYTHON_LD settings:
> > >
> > > MET_PYTHON_CC
> > > Where the python header files are located, e.g.
> -I<include
> > > dir>.
> > > Set to the output of 'python-config --cflags'.
> > > MET_PYTHON_LD
> > > Where the python library files are located, e.g.
-L<lib
> > dir>,
> > > followed by the python library names, e.g.
-l<library>.
> Set
> > > to
> > > the output of 'python-config --ldflags'.
> > >
> > > It sounds like you have a "good" version of python around and a
"bad"
> > > version of python. If the directory where MET was compiled is
> available
> > on
> > > your system, you could check "config.log" to see how those were
set at
> > > compilation time. For example:
> > >
> > > *grep MET_PYTHON_ config.log*
> > > ac_cv_env_MET_PYTHON_CC_set=set
> > >
> ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > > ac_cv_env_MET_PYTHON_LD_set=set
> > >
> ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7
-lpthread -ldl
> > > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > >
> > > And check to see if you're compiling against your preferred
version of
> > > python.
> > >
> > > Hope that helps.
> > >
> > > John
> > >
> > > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > >
> > > >
> > > > One additional question:
> > > >
> > > > I notice from tests that paths to Python interpreters and
modules are
> > > > inconsistent. This is quite visible here because we have
multiple
> > > versions
> > > > (and mutually incompatible) of Python interpreters and modules
> > installed
> > > > that are managed using shell "module load" command. For
example:
> > > >
> > > >
> > > >
> > > > import sys
> > > > print('Using: ',sys.executable)
> > > > Using:
> /opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > > >
> > > > import csv
> > > >
> > > > import numpy as np
> > > >
> > > > import os
> > > >
> > > > print csv
> > > >
> > > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > > >
> > > > print np
> > > >
> > > > <module 'numpy' from
> > > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > > >
> > > > print os
> > > >
> > > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > > >
> > > >
> > > >
> > > > There is a separate installation of python modules under
> > > > /opt/scitools/environments/default_legacy/2019_02_27/;
however, for
> > some
> > > > reasons, the executable decides to go find modules from a
different
> > place
> > > > “/usr/lib64/python2.7/”. Is there a way to fix the above? The
above
> is
> > > > actually generating errors with actual tests of using MTD
through the
> > > > Python interface because the correct Python modules cannot be
found,
> > and
> > > I
> > > > tried hand-fixing sys.path but it does not work as well.
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Steven
> > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > From: Chan, Steven
> > > > Sent: 11 April 2019 14:09:04
> > > > To: met_help at ucar.edu
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > >
> > > >
> > > > Dear John:
> > > >
> > > >
> > > >
> > > > Please forgive me for the number of emails. This is about the
> original
> > > > email you sent to me last year about rotated pole issue. In
fact the
> > > > question has nothing to do with rotated pole but with the
below:
> > > >
> > > >
> > > >
> > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > 'name="read_nccf_rotll.py
> > > > schan_eurocordex.nc precipitation_flux";'
> > > >
> > > >
> > > >
> > > > For some MET tools, more than one input files are required
(for
> example
> > > > "mtd"). How would the Python interface work under that
particular
> > > > situation? In other words, how would one actually loop through
a list
> > of
> > > > files?
> > > >
> > > >
> > > >
> > > > Particularly for mtd: -fcst/-single/-obs must be followed by a
list
> of
> > > > files or a file with list of files. If I understood the manual
> > > correctly, I
> > > > interpreted that as repeating the string "PYTHON_NUMPY", so
there
> will
> > > be a
> > > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> > Please
> > > > correct me if I am wrong.
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Steven Chan
> > > >
> > > > ________________________________
> > > >
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 21 September 2018 19:30:00
> > > > To: Chan, Steven
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > >
> > > >
> > > > Steven,
> > > >
> > > > Thanks for sending the sample data. As you already know, I
confirmed
> > > that
> > > > MET can't read these data files as-is. I get the following
error
> from
> > > > plot_data_plane:
> > > >
> > > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
> projection
> > > > from information in netCDF file.
> > > >
> > > > So we need to add support for reading the rotated lat/lon
projection
> > > > information from CF-compliant NetCDF files. I was keeping my
fingers
> > > > crossed that it'd be very simple to do, but it is not. It'll
take
> some
> > > > effort to make sure we do it correctly. And with our release
next
> > week,
> > > we
> > > > don't have enough time to fit it in. But I did create a
development
> > > ticket
> > > > for this for the next release.
> > > >
> > > > However, MET will still not be able to read the variable mesh
lat/lon
> > > > data. We use the grid/projection information from the input
data
> files
> > > to
> > > > instantiate a regular grid in the MET libraries. That enables
MET to
> > > > convert from grid x/y to lat/lon and vice-versa. And for this
> reason,
> > > MET
> > > > doesn't currently support a variable mesh. Its possible that
could
> > > change
> > > > in the future depending on requirements from our funding
agencies.
> > > >
> > > > met-8.0 is coming out next week and includes support for
calling a
> > python
> > > > 2.7 script... and we can read rotated lat/lon data down that
route.
> I
> > > ran
> > > > the following command using the development version of MET:
> > > >
> > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > 'name="read_nccf_rotll.py
> > > > schan_eurocordex.nc precipitation_flux";'
> > > >
> > > > In this command, I have...
> > > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > > "PYTHON_NUMPY".
> > > > (2) Set 'name="...";' where "..." is the python script with
> associated
> > > > command line arguments to be run.
> > > >
> > > > I've attached the python script to illustrate how it's reading
the
> > first
> > > > time-slice of the gridded data and setting up the metadata.
We would
> > > > obviously want a more robust python script than this simple
one.
> > > >
> > > > And I've attached the resulting plot to see if MET is putting
the
> data
> > at
> > > > the right spot on earth. Looks reasonable to me.
> > > >
> > > > This demonstrates that there is a way forward with met-8.0
until we
> get
> > > > support for rotated lat/lon projections in CF-compliant files
added.
> > > >
> > > > Hope this helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > > I have uploaded two files to the FTP:
> > > > >
> > > > > /incoming/irap/met_help/StevenChan_data/
> > > > >
schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > > /incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > > >
> > > > > They are both rotated pole, but the resolution of the files
are
> > > different
> > > > > with one of the files have variable resolution (near the
> boundaries).
> > > > >
> > > > > Please let me know if the files are okay.
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Steven
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 20 September 2018 18:12
> > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > > Steven,
> > > > >
> > > > > Thanks for the update. We're down to crunch time here
getting the
> > > > met-8.0
> > > > > release finalized. Once I get my hands on the sample file,
I'll do
> > my
> > > > best
> > > > > to get it working.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > Line manager said it is fine. I am just waiting for Sys
Admin to
> > > > > > actually make that possible through the Met Office
firewall (a
> > > > > > technical issue). I will try to get that resolved before
the end
> of
> > > > this
> > > > > week.
> > > > > >
> > > > > > Thank you
> > > > > > Steven
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 18 September 2018 18:27
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > OK, please let us know. The development version for met-
8.0
> > doesn't
> > > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > > CF-compliant NetCDF files since we don't have any
examples. The
> > > GRIB2
> > > > > > example we have is from the US Air Force's usage of UK Met
Office
> > > > > > unified model... which they call GALWEM.
> > > > > >
> > > > > > If we can get sample data soon enough, hopefully we can
get it
> > added
> > > > > > prior to the met-8.0 release.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> >
> > > > > > >
> > > > > > > Dear John:
> > > > > > >
> > > > > > > Thank you very much for your reply. Yes, it is rotated
lat-lon
> > > grid,
> > > > > > > and the files are created supposedly according to CF
convention
> > > > > > > using Met Office python libraries; hence reading the
data with
> > the
> > > > > > > help with Python will be extremely helpful function.
> > > > > > >
> > > > > > > I will need to confirm with line manager if it is
possible to
> > send
> > > a
> > > > > > > data sample as Met Office has a lot of rules in sharing
data.
> If
> > > > > > > that is not possible, I may be able to I can get you
fictitious
> > > data
> > > > > > > with the right coordinate information. I will get back
to you
> > with
> > > > > > > this before the end of this week (line manager is out of
town
> > right
> > > > > now).
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Steven Chan
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > Sent: 18 September 2018 16:35
> > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > > with
> > > > > > > projected inputs
> > > > > > >
> > > > > > > Steven,
> > > > > > >
> > > > > > > I see that MET is not able to handle the projection on
which
> your
> > > > > > > data lives. We are currently working on the next
version of
> MET,
> > > > > met-8.0.
> > > > > > > We've passed the code freeze date and are now in
testing. One
> of
> > > > > > > the features we added was support for rotated lat/lon
grids.
> > Based
> > > > > > > on your description, it sounds like that's what you
have.
> > met-8.0
> > > > > > > is currently able to read rotated lat/lon data from
GRIB2
> files,
> > > but
> > > > > > > we've been looking for other rotated lat/lon projection
> examples.
> > > > > > > However, I suspect we may run into other technical
issues with
> > your
> > > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > > convention,
> > > > > > which MET supports?
> > > > > > >
> > > > > > > Another enhancement in met-8.0 is support for calling
python
> > > scripts
> > > > > > > to read gridded data. If MET isn't able to read the
particular
> > > > > > > flavor of NetCDF you have, you could write a python
script to
> > read
> > > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> > When
> > > > > > > you run MODE, instead of telling it to read a NetCDF
input
> file,
> > > you
> > > > > > > can tell it to execute a python script. Then it reads
the
> > gridded
> > > > > > > field of data from that python script in memory. We
hope that
> > this
> > > > > > > feature will make it possible to run MET on a wider
variety of
> > data
> > > > > formats.
> > > > > > >
> > > > > > > Can you please send us a sample rotated lat/lon NetCDF
file so
> > that
> > > > > > > we can take a look? Here's how you can post data to our
> > anonymous
> > > > ftp
> > > > > site:
> > > > > > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John Halley Gotway
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted
upon.
> > > > > > > > Transaction: Ticket created by
steven.chan at metoffice.gov.uk
> > > > > > > > Queue: met_help
> > > > > > > > Subject: MET5.2 question: Dealing with projected
inputs
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Dear MET-HELP:
> > > > > > > >
> > > > > > > > Sorry to bug you a question, but I am trying to use
MODE and
> > MTD
> > > > > > > > with some model inputs that have unusual map
projections
> (i.e.
> > > the
> > > > > > "poles"
> > > > > > > > not being where they are usually; or "rotated pole" in
the
> way
> > we
> > > > > > > > called
> > > > > > > them here).
> > > > > > > > MODE, MTD and plot_data_plane do not seem wants to
read them
> > even
> > > > > > > > after xy dependent latitude and longitude information
are
> added
> > > to
> > > > > > > > the
> > > > > > > netcdf file.
> > > > > > > > It gives errors like:
> > > > > > > >
> > > > > > > > ERROR : plot_data_plane -> file "
> mi-ag192_h_precip_200001.nc"
> > > not
> > > > > > > > a valid data file ERROR :
NcCfFile::read_netcdf_grid() ->
> > > > > > > > Couldn't figure out projection from information in
netCDF
> file.
> > > > > > > >
> > > > > > > > In the past, we got around this problem by regridding
the
> data
> > to
> > > > > > > > regular lat-lon; due to scientific and technical
reasons, we
> > wish
> > > > > > > > to
> > > > > > > avoid that.
> > > > > > > > Are there any solutions to directly work with the
projected
> > data?
> > > > > > > >
> > > > > > > > Thank you and Best Regards,
> > > > > > > > Steven
> > > > > > > >
> > > > > > > >
=============================================================
> > > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University
> Met
> > > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon EX1
3PB
> > > United
> > > > > > > > Kingdom
> > > > > > > > Tel: +44 (0)1392 886153
> > > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> > >
www.metoffice.gov.uk<http://www.metoffice.gov.uk<<<http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<<>
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<>
> > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > > <http://www.metoffice.gov.uk>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Julie Prestopnik
Time: Tue May 07 09:27:15 2019
Hi Steven.
Thank you for sending along your files.
Looking at the run_MTD.out file, it looks like you are trying to
import a
module "iris" that cannot be found:
ERROR :
> ERROR : python_dataplane() -> an error occurred importing module
>
"/net/home/h04/schan/workspace/AnalysisEurope2.2/MET80/read_cube_numpy.py"
> ERROR :
> Traceback (most recent call last):
> File
>
"/net/home/h04/schan/workspace/AnalysisEurope2.2/MET80/read_cube_numpy.py",
> line 6, in <module>
> import iris
> ImportError: No module named iris
> Execution complete
>
You will need to have iris installed before you can import it.
I hope that helps. Please let us know how it goes after you get iris
installed.
Thanks,
Julie
On Fri, May 3, 2019 at 10:15 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
>
> Thank you for your email. In the tar gz files:
>
>
> config.log
>
> output of make
>
> error message in running mtd
>
>
> I did a grep, neither the string "libpython2.7.so" and
"libpython2.7.a"
> but you can see the paths that we have tried to add is there.
>
>
> About the fudge solution: I am personally happy for that right now.
That
> does not fix the problem with rotated pole problem that was
communicated in
> earlier emails, but it does make Python interface to appear to work.
>
>
> Best Regards,
>
> Steven
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 30 April 2019 18:23:31
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven.
>
> Could you please send the error that you are experiencing along with
the
> config.log file and the make.log file from both builds (where you
point to
> libpython2.7.so and also libpython2.7.a)?
>
> Or, because you have a work around with the fudge to the value of
> LD_LIBRARY_PATH, would you prefer to consider the problem resolved?
>
> Thanks!
>
> Julie
>
> On Tue, Apr 30, 2019 at 10:52 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear Julie:
> >
> >
> > Thank you for your email.
> >
> >
> > I can confirm there is an indeed a libpython2.7.so in the
directory you
> > mention. However, making the export command changes do not resolve
the
> > problem; ironically, the fudging the value of LD_LIBRARY_PATH just
before
> > runtime still seems to get around the problem.
> >
> >
> > About libpython2.7.a: It is located in a different location:
> >
> >
>
/opt/scitools/environments/default_legacy/2019_02_27/lib/python2.7/config/.
> > I tried to include that path into -L to MET_PYTHON_LD as well, but
it
> does
> > not resolve the problem.
> >
> >
> > Best Regards
> >
> > Steven
> >
> >
> > ________________________________
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: 29 April 2019 17:25:33
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven.
> >
> > Thank you for sending this information. After seeing that
"python-config
> > --ldflags" has no path information, I think that we likely don't
need to
> > set CPPFLAGS and LDFLAGS. For some systems, running `python-
config
> > --ldflags` has all of the necessary information, but in this case
that is
> > not the case. So, we need to dig a little deeper in order to
determine
> > what MET_PYTHON_LD should be set to.
> >
> > Since the output of `python-config --cflags` provides the
directory
> >
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
> > it is likely that the directory needed for "-L" is
> > "/opt/scitools/environments/default_legacy/2019_02_27/lib". Can
you
> please
> > take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib
> and
> > see if a libpython2.7.a or a libpython2.7.so exists in that
directory?
> >
> > If it does, please set the following:
> >
/opt/scitools/environments/${ScitoolsSuperGroup}/${ScitoolsVersion}/
> > (bash)
> > export MET_PYTHON_CC=`python-config --cflags`
> > export
> > MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
> > -lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-
dynamic
> >
> > (csh)
> > setenv MET_PYTHON_CC `python-config --cflags`
> > setenv MET_PYTHON_LD
> > -L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
> > -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
> >
> > If not, please try to find the ".a" or ".so" file location and
replace
> the
> > directory after "-L" above with the correct location.
> >
> > Please do this in a clean shell so that LDFLAGS and CPPFLAGS are
no
> longer
> > set. Please set the other necessary MET environment variables and
then
> > configure and compile MET. If that still does not work, please
send the
> > error that you are experiencing along with the config.log file and
the
> > make.log file from the build.
> >
> > Plesae let us know if you have any questions.
> >
> > Thanks,
> > Julie
> >
> > On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear Julie:
> > >
> > > Thank you very much for your email. It does not seem to work for
me,
> but
> > > perhaps I am doing something wrong?
> > >
> > >
> > > In particular, should the new LDFLAGS options point to where
> > > libpython2.7.a is? My "python-config --ldflags" has no path
> information,
> > > but "python-config --cflags" does have path information.
> > >
> > >
> > > i.e.
> > >
> > > [schan at eld277:schan] $ python-config --cflags
> > >
> > >
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > >
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > > -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> > > -Wstrict-prototypes
> > > [schan at eld277:schan] $ python-config --ldflags
> > > -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
> > >
> > > Best Regards
> > > Steven
> > >
> > > ________________________________
> > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > Sent: 26 April 2019 15:45:42
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> > > thought I'd jump in here with some further information. Thank
you for
> > > describing what is going on.
> > >
> > > I think there is potentially a better way to fix this issue.
Using
> > "rpath"
> > > in compiling will embed a specific directory into the runtime
library
> > > search path. It takes precedence over a user's LD_LIBRARY_PATH
and
> might
> > > be worthwhile to try.
> > >
> > > Before configuring and compiling MET, you would want to set
CPPFLAGS to
> > the
> > > location of where the python header files are located (likely
the the
> > > output of `python-config --cflags`) and set LDFLAGS (using rpath
as
> shown
> > > below: "-Wl,-rpath,") to the location of the python library
directory
> > (the
> > > -L part of `python-config --ldflags`). For example:
> > >
> > > export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> > > export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
> > >
> > > After setting those environment variables, along with the other
> > environment
> > > variables necessary for the compilation of MET, you would then
run
> > > configure and then "make install".
> > >
> > > Please let us know how it goes if you decide to give that a try.
> > >
> > > Julie
> > >
> > > On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > >
> > > > Yes, indeed there are multiple version of Python installed
here -- a
> > > > default "bad" version that get installed part of RHEL and
another
> > "good"
> > > > version that is only accessible via a module.
> > > >
> > > >
> > > > The actual problem is actually solved now with some caveats.
As
> > according
> > > > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> > > with
> > > > the "good" version of the Python library.
> > > >
> > > >
> > > > At one time during runtime, somehow computer is still trying
to load
> > the
> > > > "bad" one because the "bad" one has the libraries in one of
the paths
> > > under
> > > > LD_LIBRARY_PATH. The interesting thing is that MET actually
loads the
> > > > "good" version of the Python executable (as indicated by
> > sys.executable),
> > > > but either Python or MET and then decides to run off to go use
the
> bad
> > > > library in LD_LIBRARY_PATH instead of the library that comes
with the
> > > > Python indicated by sys.executable. Basically, it can't tell
which
> > Python
> > > > library is the good or the bad one during runtime.
> > > >
> > > >
> > > > Attempts to remove that specific path from LD_LIBRARY_PATH
leads to
> > > > unexpected behaviour of other software which are not related
to MET
> > (like
> > > > emacs and evince will cause errors and warnings) (the
violating item
> is
> > > > something silly like /usr/lib -- i.e. you can't really remove
it).
> The
> > > > solution now is basically every MET tool has to run through a
shell
> > > script
> > > > which explicitly remove items from LD_LIBRARY_PATH that are
> > incompatible
> > > > with MET. Perhaps there are better way to fix this, but at
least it
> is
> > > > working now.
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Steven Chan
> > > >
> > > > ________________________________
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 25 April 2019 21:48:33
> > > > To: Chan, Steven
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Hi Steven,
> > > >
> > > > It sounds like you're wondering how to configure your
environment to
> > use
> > > > one version of python versus another.
> > > >
> > > > While adding support for calling python scripts has solved
some
> > problems
> > > in
> > > > MET, it has also given rise to new ones. And this is one of
them!
> > > >
> > > > Unfortunately, I believe this needs to be handled when
compiling MET.
> > In
> > > > particular, using the MET_PYTHON_CC and MET_PYTHON_LD
settings:
> > > >
> > > > MET_PYTHON_CC
> > > > Where the python header files are located, e.g.
> > -I<include
> > > > dir>.
> > > > Set to the output of 'python-config --cflags'.
> > > > MET_PYTHON_LD
> > > > Where the python library files are located, e.g.
-L<lib
> > > dir>,
> > > > followed by the python library names, e.g.
-l<library>.
> > Set
> > > > to
> > > > the output of 'python-config --ldflags'.
> > > >
> > > > It sounds like you have a "good" version of python around and
a "bad"
> > > > version of python. If the directory where MET was compiled is
> > available
> > > on
> > > > your system, you could check "config.log" to see how those
were set
> at
> > > > compilation time. For example:
> > > >
> > > > *grep MET_PYTHON_ config.log*
> > > > ac_cv_env_MET_PYTHON_CC_set=set
> > > >
> > ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > > > ac_cv_env_MET_PYTHON_LD_set=set
> > > >
> > ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > > > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-
dynamic'
> > > > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > > > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7
-lpthread
> -ldl
> > > > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > > >
> > > > And check to see if you're compiling against your preferred
version
> of
> > > > python.
> > > >
> > > > Hope that helps.
> > > >
> > > > John
> > > >
> > > > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > >
> > > > >
> > > > > One additional question:
> > > > >
> > > > > I notice from tests that paths to Python interpreters and
modules
> are
> > > > > inconsistent. This is quite visible here because we have
multiple
> > > > versions
> > > > > (and mutually incompatible) of Python interpreters and
modules
> > > installed
> > > > > that are managed using shell "module load" command. For
example:
> > > > >
> > > > >
> > > > >
> > > > > import sys
> > > > > print('Using: ',sys.executable)
> > > > > Using:
> > /opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > > > >
> > > > > import csv
> > > > >
> > > > > import numpy as np
> > > > >
> > > > > import os
> > > > >
> > > > > print csv
> > > > >
> > > > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > > > >
> > > > > print np
> > > > >
> > > > > <module 'numpy' from
> > > > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > > > >
> > > > > print os
> > > > >
> > > > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > > > >
> > > > >
> > > > >
> > > > > There is a separate installation of python modules under
> > > > > /opt/scitools/environments/default_legacy/2019_02_27/;
however, for
> > > some
> > > > > reasons, the executable decides to go find modules from a
different
> > > place
> > > > > “/usr/lib64/python2.7/”. Is there a way to fix the above?
The above
> > is
> > > > > actually generating errors with actual tests of using MTD
through
> the
> > > > > Python interface because the correct Python modules cannot
be
> found,
> > > and
> > > > I
> > > > > tried hand-fixing sys.path but it does not work as well.
> > > > >
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Steven
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > From: Chan, Steven
> > > > > Sent: 11 April 2019 14:09:04
> > > > > To: met_help at ucar.edu
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > >
> > > > >
> > > > > Dear John:
> > > > >
> > > > >
> > > > >
> > > > > Please forgive me for the number of emails. This is about
the
> > original
> > > > > email you sent to me last year about rotated pole issue. In
fact
> the
> > > > > question has nothing to do with rotated pole but with the
below:
> > > > >
> > > > >
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > > 'name="read_nccf_rotll.py
> > > > > schan_eurocordex.nc precipitation_flux";'
> > > > >
> > > > >
> > > > >
> > > > > For some MET tools, more than one input files are required
(for
> > example
> > > > > "mtd"). How would the Python interface work under that
particular
> > > > > situation? In other words, how would one actually loop
through a
> list
> > > of
> > > > > files?
> > > > >
> > > > >
> > > > >
> > > > > Particularly for mtd: -fcst/-single/-obs must be followed by
a list
> > of
> > > > > files or a file with list of files. If I understood the
manual
> > > > correctly, I
> > > > > interpreted that as repeating the string "PYTHON_NUMPY", so
there
> > will
> > > > be a
> > > > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> > > Please
> > > > > correct me if I am wrong.
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Steven Chan
> > > > >
> > > > > ________________________________
> > > > >
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 21 September 2018 19:30:00
> > > > > To: Chan, Steven
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > >
> > > > >
> > > > > Steven,
> > > > >
> > > > > Thanks for sending the sample data. As you already know, I
> confirmed
> > > > that
> > > > > MET can't read these data files as-is. I get the following
error
> > from
> > > > > plot_data_plane:
> > > > >
> > > > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
> > projection
> > > > > from information in netCDF file.
> > > > >
> > > > > So we need to add support for reading the rotated lat/lon
> projection
> > > > > information from CF-compliant NetCDF files. I was keeping
my
> fingers
> > > > > crossed that it'd be very simple to do, but it is not.
It'll take
> > some
> > > > > effort to make sure we do it correctly. And with our
release next
> > > week,
> > > > we
> > > > > don't have enough time to fit it in. But I did create a
> development
> > > > ticket
> > > > > for this for the next release.
> > > > >
> > > > > However, MET will still not be able to read the variable
mesh
> lat/lon
> > > > > data. We use the grid/projection information from the input
data
> > files
> > > > to
> > > > > instantiate a regular grid in the MET libraries. That
enables MET
> to
> > > > > convert from grid x/y to lat/lon and vice-versa. And for
this
> > reason,
> > > > MET
> > > > > doesn't currently support a variable mesh. Its possible
that could
> > > > change
> > > > > in the future depending on requirements from our funding
agencies.
> > > > >
> > > > > met-8.0 is coming out next week and includes support for
calling a
> > > python
> > > > > 2.7 script... and we can read rotated lat/lon data down that
route.
> > I
> > > > ran
> > > > > the following command using the development version of MET:
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > > 'name="read_nccf_rotll.py
> > > > > schan_eurocordex.nc precipitation_flux";'
> > > > >
> > > > > In this command, I have...
> > > > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > > > "PYTHON_NUMPY".
> > > > > (2) Set 'name="...";' where "..." is the python script with
> > associated
> > > > > command line arguments to be run.
> > > > >
> > > > > I've attached the python script to illustrate how it's
reading the
> > > first
> > > > > time-slice of the gridded data and setting up the metadata.
We
> would
> > > > > obviously want a more robust python script than this simple
one.
> > > > >
> > > > > And I've attached the resulting plot to see if MET is
putting the
> > data
> > > at
> > > > > the right spot on earth. Looks reasonable to me.
> > > > >
> > > > > This demonstrates that there is a way forward with met-8.0
until we
> > get
> > > > > support for rotated lat/lon projections in CF-compliant
files
> added.
> > > > >
> > > > > Hope this helps.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > I have uploaded two files to the FTP:
> > > > > >
> > > > > > /incoming/irap/met_help/StevenChan_data/
> > > > > >
schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > > >
/incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > > > >
> > > > > > They are both rotated pole, but the resolution of the
files are
> > > > different
> > > > > > with one of the files have variable resolution (near the
> > boundaries).
> > > > > >
> > > > > > Please let me know if the files are okay.
> > > > > >
> > > > > > Thank you and Best Regards,
> > > > > > Steven
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 20 September 2018 18:12
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > Thanks for the update. We're down to crunch time here
getting
> the
> > > > > met-8.0
> > > > > > release finalized. Once I get my hands on the sample
file, I'll
> do
> > > my
> > > > > best
> > > > > > to get it working.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> >
> > > > > > >
> > > > > > > Dear John:
> > > > > > >
> > > > > > > Line manager said it is fine. I am just waiting for Sys
Admin
> to
> > > > > > > actually make that possible through the Met Office
firewall (a
> > > > > > > technical issue). I will try to get that resolved before
the
> end
> > of
> > > > > this
> > > > > > week.
> > > > > > >
> > > > > > > Thank you
> > > > > > > Steven
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > Sent: 18 September 2018 18:27
> > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > > with
> > > > > > > projected inputs
> > > > > > >
> > > > > > > Steven,
> > > > > > >
> > > > > > > OK, please let us know. The development version for
met-8.0
> > > doesn't
> > > > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > > > CF-compliant NetCDF files since we don't have any
examples.
> The
> > > > GRIB2
> > > > > > > example we have is from the US Air Force's usage of UK
Met
> Office
> > > > > > > unified model... which they call GALWEM.
> > > > > > >
> > > > > > > If we can get sample data soon enough, hopefully we can
get it
> > > added
> > > > > > > prior to the met-8.0 release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > >
> > > > > > > >
> > > > > > > > Dear John:
> > > > > > > >
> > > > > > > > Thank you very much for your reply. Yes, it is rotated
> lat-lon
> > > > grid,
> > > > > > > > and the files are created supposedly according to CF
> convention
> > > > > > > > using Met Office python libraries; hence reading the
data
> with
> > > the
> > > > > > > > help with Python will be extremely helpful function.
> > > > > > > >
> > > > > > > > I will need to confirm with line manager if it is
possible to
> > > send
> > > > a
> > > > > > > > data sample as Met Office has a lot of rules in
sharing data.
> > If
> > > > > > > > that is not possible, I may be able to I can get you
> fictitious
> > > > data
> > > > > > > > with the right coordinate information. I will get back
to you
> > > with
> > > > > > > > this before the end of this week (line manager is out
of town
> > > right
> > > > > > now).
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Steven Chan
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > > Sent: 18 September 2018 16:35
> > > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
> Dealing
> > > > with
> > > > > > > > projected inputs
> > > > > > > >
> > > > > > > > Steven,
> > > > > > > >
> > > > > > > > I see that MET is not able to handle the projection on
which
> > your
> > > > > > > > data lives. We are currently working on the next
version of
> > MET,
> > > > > > met-8.0.
> > > > > > > > We've passed the code freeze date and are now in
testing.
> One
> > of
> > > > > > > > the features we added was support for rotated lat/lon
grids.
> > > Based
> > > > > > > > on your description, it sounds like that's what you
have.
> > > met-8.0
> > > > > > > > is currently able to read rotated lat/lon data from
GRIB2
> > files,
> > > > but
> > > > > > > > we've been looking for other rotated lat/lon
projection
> > examples.
> > > > > > > > However, I suspect we may run into other technical
issues
> with
> > > your
> > > > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > > > convention,
> > > > > > > which MET supports?
> > > > > > > >
> > > > > > > > Another enhancement in met-8.0 is support for calling
python
> > > > scripts
> > > > > > > > to read gridded data. If MET isn't able to read the
> particular
> > > > > > > > flavor of NetCDF you have, you could write a python
script to
> > > read
> > > > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> > > When
> > > > > > > > you run MODE, instead of telling it to read a NetCDF
input
> > file,
> > > > you
> > > > > > > > can tell it to execute a python script. Then it reads
the
> > > gridded
> > > > > > > > field of data from that python script in memory. We
hope
> that
> > > this
> > > > > > > > feature will make it possible to run MET on a wider
variety
> of
> > > data
> > > > > > formats.
> > > > > > > >
> > > > > > > > Can you please send us a sample rotated lat/lon NetCDF
file
> so
> > > that
> > > > > > > > we can take a look? Here's how you can post data to
our
> > > anonymous
> > > > > ftp
> > > > > > site:
> > > > > > > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John Halley Gotway
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > > > <met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> steven.chan at metoffice.gov.uk
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: MET5.2 question: Dealing with
projected
> inputs
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dear MET-HELP:
> > > > > > > > >
> > > > > > > > > Sorry to bug you a question, but I am trying to use
MODE
> and
> > > MTD
> > > > > > > > > with some model inputs that have unusual map
projections
> > (i.e.
> > > > the
> > > > > > > "poles"
> > > > > > > > > not being where they are usually; or "rotated pole"
in the
> > way
> > > we
> > > > > > > > > called
> > > > > > > > them here).
> > > > > > > > > MODE, MTD and plot_data_plane do not seem wants to
read
> them
> > > even
> > > > > > > > > after xy dependent latitude and longitude
information are
> > added
> > > > to
> > > > > > > > > the
> > > > > > > > netcdf file.
> > > > > > > > > It gives errors like:
> > > > > > > > >
> > > > > > > > > ERROR : plot_data_plane -> file "
> > mi-ag192_h_precip_200001.nc"
> > > > not
> > > > > > > > > a valid data file ERROR :
NcCfFile::read_netcdf_grid() ->
> > > > > > > > > Couldn't figure out projection from information in
netCDF
> > file.
> > > > > > > > >
> > > > > > > > > In the past, we got around this problem by
regridding the
> > data
> > > to
> > > > > > > > > regular lat-lon; due to scientific and technical
reasons,
> we
> > > wish
> > > > > > > > > to
> > > > > > > > avoid that.
> > > > > > > > > Are there any solutions to directly work with the
projected
> > > data?
> > > > > > > > >
> > > > > > > > > Thank you and Best Regards,
> > > > > > > > > Steven
> > > > > > > > >
> > > > > > > > >
> =============================================================
> > > > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University
> > Met
> > > > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon
EX1 3PB
> > > > United
> > > > > > > > > Kingdom
> > > > > > > > > Tel: +44 (0)1392 886153
> > > > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> > > > www.metoffice.gov.uk<http://www.metoffice.gov.uk<<<
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<<>
> > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<>
> > > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > > > <http://www.metoffice.gov.uk>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: MET5.2 question: Dealing with projected inputs
From: Chan, Steven
Time: Tue May 07 10:52:02 2019
Dear Julie:
iris is installed, which is the issue. Disabling specific elements
LD_LIBRARY_PATH somehow magically addresses the problem (i.e. iris
imports properly), in which mtd fails later for being unable to
compare rotated pole coordinate correctly. Please see attached.
Steven
________________________________
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: 07 May 2019 16:27:15
To: Chan, Steven
Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
projected inputs
Hi Steven.
Thank you for sending along your files.
Looking at the run_MTD.out file, it looks like you are trying to
import a
module "iris" that cannot be found:
ERROR :
> ERROR : python_dataplane() -> an error occurred importing module
>
"/net/home/h04/schan/workspace/AnalysisEurope2.2/MET80/read_cube_numpy.py"
> ERROR :
> Traceback (most recent call last):
> File
>
"/net/home/h04/schan/workspace/AnalysisEurope2.2/MET80/read_cube_numpy.py",
> line 6, in <module>
> import iris
> ImportError: No module named iris
> Execution complete
>
You will need to have iris installed before you can import it.
I hope that helps. Please let us know how it goes after you get iris
installed.
Thanks,
Julie
On Fri, May 3, 2019 at 10:15 AM Chan, Steven via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
>
> Dear Julie:
>
>
> Thank you for your email. In the tar gz files:
>
>
> config.log
>
> output of make
>
> error message in running mtd
>
>
> I did a grep, neither the string "libpython2.7.so" and
"libpython2.7.a"
> but you can see the paths that we have tried to add is there.
>
>
> About the fudge solution: I am personally happy for that right now.
That
> does not fix the problem with rotated pole problem that was
communicated in
> earlier emails, but it does make Python interface to appear to work.
>
>
> Best Regards,
>
> Steven
>
> ________________________________
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: 30 April 2019 18:23:31
> To: Chan, Steven
> Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing with
> projected inputs
>
> Hi Steven.
>
> Could you please send the error that you are experiencing along with
the
> config.log file and the make.log file from both builds (where you
point to
> libpython2.7.so and also libpython2.7.a)?
>
> Or, because you have a work around with the fudge to the value of
> LD_LIBRARY_PATH, would you prefer to consider the problem resolved?
>
> Thanks!
>
> Julie
>
> On Tue, Apr 30, 2019 at 10:52 AM Chan, Steven via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> >
> > Dear Julie:
> >
> >
> > Thank you for your email.
> >
> >
> > I can confirm there is an indeed a libpython2.7.so in the
directory you
> > mention. However, making the export command changes do not resolve
the
> > problem; ironically, the fudging the value of LD_LIBRARY_PATH just
before
> > runtime still seems to get around the problem.
> >
> >
> > About libpython2.7.a: It is located in a different location:
> >
> >
>
/opt/scitools/environments/default_legacy/2019_02_27/lib/python2.7/config/.
> > I tried to include that path into -L to MET_PYTHON_LD as well, but
it
> does
> > not resolve the problem.
> >
> >
> > Best Regards
> >
> > Steven
> >
> >
> > ________________________________
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: 29 April 2019 17:25:33
> > To: Chan, Steven
> > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > projected inputs
> >
> > Hi Steven.
> >
> > Thank you for sending this information. After seeing that
"python-config
> > --ldflags" has no path information, I think that we likely don't
need to
> > set CPPFLAGS and LDFLAGS. For some systems, running `python-
config
> > --ldflags` has all of the necessary information, but in this case
that is
> > not the case. So, we need to dig a little deeper in order to
determine
> > what MET_PYTHON_LD should be set to.
> >
> > Since the output of `python-config --cflags` provides the
directory
> >
"/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7",
> > it is likely that the directory needed for "-L" is
> > "/opt/scitools/environments/default_legacy/2019_02_27/lib". Can
you
> please
> > take a look in
/opt/scitools/environments/default_legacy/2019_02_27/lib
> and
> > see if a libpython2.7.a or a libpython2.7.so exists in that
directory?
> >
> > If it does, please set the following:
> >
/opt/scitools/environments/${ScitoolsSuperGroup}/${ScitoolsVersion}/
> > (bash)
> > export MET_PYTHON_CC=`python-config --cflags`
> > export
> > MET_PYTHON_LD=-
L/opt/scitools/environments/default_legacy/2019_02_27/lib\
> > -lpython2.7\ -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-
dynamic
> >
> > (csh)
> > setenv MET_PYTHON_CC `python-config --cflags`
> > setenv MET_PYTHON_LD
> > -L/opt/scitools/environments/default_legacy/2019_02_27/lib\
-lpython2.7\
> > -lpthread\ -ldl\ -lutil\ -lm\ -Xlinker\ -export-dynamic
> >
> > If not, please try to find the ".a" or ".so" file location and
replace
> the
> > directory after "-L" above with the correct location.
> >
> > Please do this in a clean shell so that LDFLAGS and CPPFLAGS are
no
> longer
> > set. Please set the other necessary MET environment variables and
then
> > configure and compile MET. If that still does not work, please
send the
> > error that you are experiencing along with the config.log file and
the
> > make.log file from the build.
> >
> > Plesae let us know if you have any questions.
> >
> > Thanks,
> > Julie
> >
> > On Mon, Apr 29, 2019 at 4:55 AM Chan, Steven via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > >
> > > Dear Julie:
> > >
> > > Thank you very much for your email. It does not seem to work for
me,
> but
> > > perhaps I am doing something wrong?
> > >
> > >
> > > In particular, should the new LDFLAGS options point to where
> > > libpython2.7.a is? My "python-config --ldflags" has no path
> information,
> > > but "python-config --cflags" does have path information.
> > >
> > >
> > > i.e.
> > >
> > > [schan at eld277:schan] $ python-config --cflags
> > >
> > >
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > >
>
-I/opt/scitools/environments/default_legacy/2019_02_27/include/python2.7
> > > -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> > > -Wstrict-prototypes
> > > [schan at eld277:schan] $ python-config --ldflags
> > > -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
> > >
> > > Best Regards
> > > Steven
> > >
> > > ________________________________
> > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > Sent: 26 April 2019 15:45:42
> > > To: Chan, Steven
> > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > projected inputs
> > >
> > > Hi Steven. My name is Julie Prestopnik and I work with John. I
just
> > > thought I'd jump in here with some further information. Thank
you for
> > > describing what is going on.
> > >
> > > I think there is potentially a better way to fix this issue.
Using
> > "rpath"
> > > in compiling will embed a specific directory into the runtime
library
> > > search path. It takes precedence over a user's LD_LIBRARY_PATH
and
> might
> > > be worthwhile to try.
> > >
> > > Before configuring and compiling MET, you would want to set
CPPFLAGS to
> > the
> > > location of where the python header files are located (likely
the the
> > > output of `python-config --cflags`) and set LDFLAGS (using rpath
as
> shown
> > > below: "-Wl,-rpath,") to the location of the python library
directory
> > (the
> > > -L part of `python-config --ldflags`). For example:
> > >
> > > export CPPFLAGS=-I/usr/local/python/2.7.13/include/python2.7
> > > export LDFLAGS=-Wl,-rpath,-L/usr/local/python/2.7.13/lib
> > >
> > > After setting those environment variables, along with the other
> > environment
> > > variables necessary for the compilation of MET, you would then
run
> > > configure and then "make install".
> > >
> > > Please let us know how it goes if you decide to give that a try.
> > >
> > > Julie
> > >
> > > On Fri, Apr 26, 2019 at 7:01 AM Chan, Steven via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
>
> > > >
> > > > Dear John:
> > > >
> > > >
> > > > Yes, indeed there are multiple version of Python installed
here -- a
> > > > default "bad" version that get installed part of RHEL and
another
> > "good"
> > > > version that is only accessible via a module.
> > > >
> > > >
> > > > The actual problem is actually solved now with some caveats.
As
> > according
> > > > what you have indicated with the MET_PYTHON_?? flags, "make"
compiles
> > > with
> > > > the "good" version of the Python library.
> > > >
> > > >
> > > > At one time during runtime, somehow computer is still trying
to load
> > the
> > > > "bad" one because the "bad" one has the libraries in one of
the paths
> > > under
> > > > LD_LIBRARY_PATH. The interesting thing is that MET actually
loads the
> > > > "good" version of the Python executable (as indicated by
> > sys.executable),
> > > > but either Python or MET and then decides to run off to go use
the
> bad
> > > > library in LD_LIBRARY_PATH instead of the library that comes
with the
> > > > Python indicated by sys.executable. Basically, it can't tell
which
> > Python
> > > > library is the good or the bad one during runtime.
> > > >
> > > >
> > > > Attempts to remove that specific path from LD_LIBRARY_PATH
leads to
> > > > unexpected behaviour of other software which are not related
to MET
> > (like
> > > > emacs and evince will cause errors and warnings) (the
violating item
> is
> > > > something silly like /usr/lib -- i.e. you can't really remove
it).
> The
> > > > solution now is basically every MET tool has to run through a
shell
> > > script
> > > > which explicitly remove items from LD_LIBRARY_PATH that are
> > incompatible
> > > > with MET. Perhaps there are better way to fix this, but at
least it
> is
> > > > working now.
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Steven Chan
> > > >
> > > > ________________________________
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 25 April 2019 21:48:33
> > > > To: Chan, Steven
> > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question: Dealing
with
> > > > projected inputs
> > > >
> > > > Hi Steven,
> > > >
> > > > It sounds like you're wondering how to configure your
environment to
> > use
> > > > one version of python versus another.
> > > >
> > > > While adding support for calling python scripts has solved
some
> > problems
> > > in
> > > > MET, it has also given rise to new ones. And this is one of
them!
> > > >
> > > > Unfortunately, I believe this needs to be handled when
compiling MET.
> > In
> > > > particular, using the MET_PYTHON_CC and MET_PYTHON_LD
settings:
> > > >
> > > > MET_PYTHON_CC
> > > > Where the python header files are located, e.g.
> > -I<include
> > > > dir>.
> > > > Set to the output of 'python-config --cflags'.
> > > > MET_PYTHON_LD
> > > > Where the python library files are located, e.g.
-L<lib
> > > dir>,
> > > > followed by the python library names, e.g.
-l<library>.
> > Set
> > > > to
> > > > the output of 'python-config --ldflags'.
> > > >
> > > > It sounds like you have a "good" version of python around and
a "bad"
> > > > version of python. If the directory where MET was compiled is
> > available
> > > on
> > > > your system, you could check "config.log" to see how those
were set
> at
> > > > compilation time. For example:
> > > >
> > > > *grep MET_PYTHON_ config.log*
> > > > ac_cv_env_MET_PYTHON_CC_set=set
> > > >
> > ac_cv_env_MET_PYTHON_CC_value=-I/usr/local/python-
2.7.8/include/python2.7
> > > > ac_cv_env_MET_PYTHON_LD_set=set
> > > >
> > ac_cv_env_MET_PYTHON_LD_value='-L-I/usr/local/python-
2.7.8/lib/python2.7
> > > > -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-
dynamic'
> > > > MET_PYTHON_CC='-I/usr/local/python-2.7.8/include/python2.7'
> > > > MET_PYTHON_LD='-L-I/usr/local/python-2.7.8/lib/python2.7
-lpthread
> -ldl
> > > > -lutil -lm -lpython2.7 -Xlinker -export-dynamic'
> > > >
> > > > And check to see if you're compiling against your preferred
version
> of
> > > > python.
> > > >
> > > > Hope that helps.
> > > >
> > > > John
> > > >
> > > > On Fri, Apr 12, 2019 at 3:33 AM Chan, Steven via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > >
> > > > > Dear John:
> > > > >
> > > > >
> > > > >
> > > > > One additional question:
> > > > >
> > > > > I notice from tests that paths to Python interpreters and
modules
> are
> > > > > inconsistent. This is quite visible here because we have
multiple
> > > > versions
> > > > > (and mutually incompatible) of Python interpreters and
modules
> > > installed
> > > > > that are managed using shell "module load" command. For
example:
> > > > >
> > > > >
> > > > >
> > > > > import sys
> > > > > print('Using: ',sys.executable)
> > > > > Using:
> > /opt/scitools/environments/default_legacy/2019_02_27/bin/python
> > > > >
> > > > > import csv
> > > > >
> > > > > import numpy as np
> > > > >
> > > > > import os
> > > > >
> > > > > print csv
> > > > >
> > > > > <module 'csv' from '/usr/lib64/python2.7/csv.pyc'>
> > > > >
> > > > > print np
> > > > >
> > > > > <module 'numpy' from
> > > > > '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
> > > > >
> > > > > print os
> > > > >
> > > > > <module 'os' from '/usr/lib64/python2.7/os.pyc'>
> > > > >
> > > > >
> > > > >
> > > > > There is a separate installation of python modules under
> > > > > /opt/scitools/environments/default_legacy/2019_02_27/;
however, for
> > > some
> > > > > reasons, the executable decides to go find modules from a
different
> > > place
> > > > > “/usr/lib64/python2.7/”. Is there a way to fix the above?
The above
> > is
> > > > > actually generating errors with actual tests of using MTD
through
> the
> > > > > Python interface because the correct Python modules cannot
be
> found,
> > > and
> > > > I
> > > > > tried hand-fixing sys.path but it does not work as well.
> > > > >
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Steven
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > From: Chan, Steven
> > > > > Sent: 11 April 2019 14:09:04
> > > > > To: met_help at ucar.edu
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > >
> > > > >
> > > > > Dear John:
> > > > >
> > > > >
> > > > >
> > > > > Please forgive me for the number of emails. This is about
the
> > original
> > > > > email you sent to me last year about rotated pole issue. In
fact
> the
> > > > > question has nothing to do with rotated pole but with the
below:
> > > > >
> > > > >
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > > 'name="read_nccf_rotll.py
> > > > > schan_eurocordex.nc precipitation_flux";'
> > > > >
> > > > >
> > > > >
> > > > > For some MET tools, more than one input files are required
(for
> > example
> > > > > "mtd"). How would the Python interface work under that
particular
> > > > > situation? In other words, how would one actually loop
through a
> list
> > > of
> > > > > files?
> > > > >
> > > > >
> > > > >
> > > > > Particularly for mtd: -fcst/-single/-obs must be followed by
a list
> > of
> > > > > files or a file with list of files. If I understood the
manual
> > > > correctly, I
> > > > > interpreted that as repeating the string "PYTHON_NUMPY", so
there
> > will
> > > > be a
> > > > > need of linking a particular "PYTHON_NUMPY" to a particular
file.
> > > Please
> > > > > correct me if I am wrong.
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Steven Chan
> > > > >
> > > > > ________________________________
> > > > >
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: 21 September 2018 19:30:00
> > > > > To: Chan, Steven
> > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> with
> > > > > projected inputs
> > > > >
> > > > >
> > > > >
> > > > > Steven,
> > > > >
> > > > > Thanks for sending the sample data. As you already know, I
> confirmed
> > > > that
> > > > > MET can't read these data files as-is. I get the following
error
> > from
> > > > > plot_data_plane:
> > > > >
> > > > > plot_data_plane schan_eurocordex.nc schan_eurocordex.ps
> > > > > 'name="precipitation_flux"; level="(0,*,*)";'
> > > > > ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
> > projection
> > > > > from information in netCDF file.
> > > > >
> > > > > So we need to add support for reading the rotated lat/lon
> projection
> > > > > information from CF-compliant NetCDF files. I was keeping
my
> fingers
> > > > > crossed that it'd be very simple to do, but it is not.
It'll take
> > some
> > > > > effort to make sure we do it correctly. And with our
release next
> > > week,
> > > > we
> > > > > don't have enough time to fit it in. But I did create a
> development
> > > > ticket
> > > > > for this for the next release.
> > > > >
> > > > > However, MET will still not be able to read the variable
mesh
> lat/lon
> > > > > data. We use the grid/projection information from the input
data
> > files
> > > > to
> > > > > instantiate a regular grid in the MET libraries. That
enables MET
> to
> > > > > convert from grid x/y to lat/lon and vice-versa. And for
this
> > reason,
> > > > MET
> > > > > doesn't currently support a variable mesh. Its possible
that could
> > > > change
> > > > > in the future depending on requirements from our funding
agencies.
> > > > >
> > > > > met-8.0 is coming out next week and includes support for
calling a
> > > python
> > > > > 2.7 script... and we can read rotated lat/lon data down that
route.
> > I
> > > > ran
> > > > > the following command using the development version of MET:
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY schan_eurocordex.ps
> > > > 'name="read_nccf_rotll.py
> > > > > schan_eurocordex.nc precipitation_flux";'
> > > > >
> > > > > In this command, I have...
> > > > > (1) Replaced the filename (schan_eurocordex.nc) with the
constant
> > > > > "PYTHON_NUMPY".
> > > > > (2) Set 'name="...";' where "..." is the python script with
> > associated
> > > > > command line arguments to be run.
> > > > >
> > > > > I've attached the python script to illustrate how it's
reading the
> > > first
> > > > > time-slice of the gridded data and setting up the metadata.
We
> would
> > > > > obviously want a more robust python script than this simple
one.
> > > > >
> > > > > And I've attached the resulting plot to see if MET is
putting the
> > data
> > > at
> > > > > the right spot on earth. Looks reasonable to me.
> > > > >
> > > > > This demonstrates that there is a way forward with met-8.0
until we
> > get
> > > > > support for rotated lat/lon projections in CF-compliant
files
> added.
> > > > >
> > > > > Hope this helps.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 21, 2018 at 3:40 AM Chan, Steven via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015 >
> > > > > >
> > > > > > Dear John:
> > > > > >
> > > > > > I have uploaded two files to the FTP:
> > > > > >
> > > > > > /incoming/irap/met_help/StevenChan_data/
> > > > > >
schan_rotated_pole_with_different_resolution_near_boundary.nc
> > > > > >
/incoming/irap/met_help/StevenChan_data/schan_eurocordex.nc
> > > > > >
> > > > > > They are both rotated pole, but the resolution of the
files are
> > > > different
> > > > > > with one of the files have variable resolution (near the
> > boundaries).
> > > > > >
> > > > > > Please let me know if the files are okay.
> > > > > >
> > > > > > Thank you and Best Regards,
> > > > > > Steven
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > Sent: 20 September 2018 18:12
> > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > with
> > > > > > projected inputs
> > > > > >
> > > > > > Steven,
> > > > > >
> > > > > > Thanks for the update. We're down to crunch time here
getting
> the
> > > > > met-8.0
> > > > > > release finalized. Once I get my hands on the sample
file, I'll
> do
> > > my
> > > > > best
> > > > > > to get it working.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Sep 20, 2018 at 3:12 AM Chan, Steven via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> >
> > > > > > >
> > > > > > > Dear John:
> > > > > > >
> > > > > > > Line manager said it is fine. I am just waiting for Sys
Admin
> to
> > > > > > > actually make that possible through the Met Office
firewall (a
> > > > > > > technical issue). I will try to get that resolved before
the
> end
> > of
> > > > > this
> > > > > > week.
> > > > > > >
> > > > > > > Thank you
> > > > > > > Steven
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > Sent: 18 September 2018 18:27
> > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
Dealing
> > > with
> > > > > > > projected inputs
> > > > > > >
> > > > > > > Steven,
> > > > > > >
> > > > > > > OK, please let us know. The development version for
met-8.0
> > > doesn't
> > > > > > > currently have support for parsing rotated lat/lon grids
from
> > > > > > > CF-compliant NetCDF files since we don't have any
examples.
> The
> > > > GRIB2
> > > > > > > example we have is from the US Air Force's usage of UK
Met
> Office
> > > > > > > unified model... which they call GALWEM.
> > > > > > >
> > > > > > > If we can get sample data soon enough, hopefully we can
get it
> > > added
> > > > > > > prior to the met-8.0 release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Sep 18, 2018 at 9:52 AM Chan, Steven via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > >
> > > > > > > >
> > > > > > > > Dear John:
> > > > > > > >
> > > > > > > > Thank you very much for your reply. Yes, it is rotated
> lat-lon
> > > > grid,
> > > > > > > > and the files are created supposedly according to CF
> convention
> > > > > > > > using Met Office python libraries; hence reading the
data
> with
> > > the
> > > > > > > > help with Python will be extremely helpful function.
> > > > > > > >
> > > > > > > > I will need to confirm with line manager if it is
possible to
> > > send
> > > > a
> > > > > > > > data sample as Met Office has a lot of rules in
sharing data.
> > If
> > > > > > > > that is not possible, I may be able to I can get you
> fictitious
> > > > data
> > > > > > > > with the right coordinate information. I will get back
to you
> > > with
> > > > > > > > this before the end of this week (line manager is out
of town
> > > right
> > > > > > now).
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Steven Chan
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > > > > Sent: 18 September 2018 16:35
> > > > > > > > To: Chan, Steven <steven.chan at metoffice.gov.uk>
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #87015] MET5.2 question:
> Dealing
> > > > with
> > > > > > > > projected inputs
> > > > > > > >
> > > > > > > > Steven,
> > > > > > > >
> > > > > > > > I see that MET is not able to handle the projection on
which
> > your
> > > > > > > > data lives. We are currently working on the next
version of
> > MET,
> > > > > > met-8.0.
> > > > > > > > We've passed the code freeze date and are now in
testing.
> One
> > of
> > > > > > > > the features we added was support for rotated lat/lon
grids.
> > > Based
> > > > > > > > on your description, it sounds like that's what you
have.
> > > met-8.0
> > > > > > > > is currently able to read rotated lat/lon data from
GRIB2
> > files,
> > > > but
> > > > > > > > we've been looking for other rotated lat/lon
projection
> > examples.
> > > > > > > > However, I suspect we may run into other technical
issues
> with
> > > your
> > > > > > > > data. Do your NetCDF files follow the CF (climate-
forecast)
> > > > > > > > convention,
> > > > > > > which MET supports?
> > > > > > > >
> > > > > > > > Another enhancement in met-8.0 is support for calling
python
> > > > scripts
> > > > > > > > to read gridded data. If MET isn't able to read the
> particular
> > > > > > > > flavor of NetCDF you have, you could write a python
script to
> > > read
> > > > > > > > the data, populate a NumPy array, and dictionary of
metadata.
> > > When
> > > > > > > > you run MODE, instead of telling it to read a NetCDF
input
> > file,
> > > > you
> > > > > > > > can tell it to execute a python script. Then it reads
the
> > > gridded
> > > > > > > > field of data from that python script in memory. We
hope
> that
> > > this
> > > > > > > > feature will make it possible to run MET on a wider
variety
> of
> > > data
> > > > > > formats.
> > > > > > > >
> > > > > > > > Can you please send us a sample rotated lat/lon NetCDF
file
> so
> > > that
> > > > > > > > we can take a look? Here's how you can post data to
our
> > > anonymous
> > > > > ftp
> > > > > > site:
> > > > > > > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John Halley Gotway
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Sep 18, 2018 at 8:35 AM Chan, Steven via RT
> > > > > > > > <met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Sep 18 08:35:14 2018: Request 87015 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> steven.chan at metoffice.gov.uk
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: MET5.2 question: Dealing with
projected
> inputs
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: steven.chan at metoffice.gov.uk
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87015
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dear MET-HELP:
> > > > > > > > >
> > > > > > > > > Sorry to bug you a question, but I am trying to use
MODE
> and
> > > MTD
> > > > > > > > > with some model inputs that have unusual map
projections
> > (i.e.
> > > > the
> > > > > > > "poles"
> > > > > > > > > not being where they are usually; or "rotated pole"
in the
> > way
> > > we
> > > > > > > > > called
> > > > > > > > them here).
> > > > > > > > > MODE, MTD and plot_data_plane do not seem wants to
read
> them
> > > even
> > > > > > > > > after xy dependent latitude and longitude
information are
> > added
> > > > to
> > > > > > > > > the
> > > > > > > > netcdf file.
> > > > > > > > > It gives errors like:
> > > > > > > > >
> > > > > > > > > ERROR : plot_data_plane -> file "
> > mi-ag192_h_precip_200001.nc"
> > > > not
> > > > > > > > > a valid data file ERROR :
NcCfFile::read_netcdf_grid() ->
> > > > > > > > > Couldn't figure out projection from information in
netCDF
> > file.
> > > > > > > > >
> > > > > > > > > In the past, we got around this problem by
regridding the
> > data
> > > to
> > > > > > > > > regular lat-lon; due to scientific and technical
reasons,
> we
> > > wish
> > > > > > > > > to
> > > > > > > > avoid that.
> > > > > > > > > Are there any solutions to directly work with the
projected
> > > data?
> > > > > > > > >
> > > > > > > > > Thank you and Best Regards,
> > > > > > > > > Steven
> > > > > > > > >
> > > > > > > > >
> =============================================================
> > > > > > > > > Steven C Chan, Visiting Scientist from Newcastle
University
> > Met
> > > > > > > > > Office Hadley Centre FitzRoy Road Exeter Devon
EX1 3PB
> > > > United
> > > > > > > > > Kingdom
> > > > > > > > > Tel: +44 (0)1392 886153
> > > > > > > > > Email: steven.chan at metoffice.gov.uk Website:
> > > >
www.metoffice.gov.uk<http://www.metoffice.gov.uk<<<<http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<<<>
> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<<>
> > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk<>
> > > http://www.metoffice.gov.uk<http://www.metoffice.gov.uk>>
> > > > > <http://www.metoffice.gov.uk>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list