[Met_help] [rt.rap.ucar.edu #87266] History for Question about MET pcp_combine
John Halley Gotway via RT
met_help at ucar.edu
Tue Oct 9 15:10:28 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET help,
I have a question on the MET pcp_combine add tool. I have model data that
is considered my truth from an ECMWF Nature run (long-run simulation used
to represent the true atmosphere). I want to verify how well my forecast
output, in this case from the GFS model, is compared to the Nature run. The
ECMWF nature run data is 6-hourly data, which includes 6-hr precipitation
and other fields in its netcdf files. my GFS data is grib format. I already
can convert the GFS data to 24-hourly accumulations using my prior scripts
and with pcp_combine. I guess my issue is in the ability to read the Nature
run netcdf files. I try the pcp_combine tool and I get this error when
trying to create a 24-hr accumulation.
(0) Current cycle: 2006012800
DEBUG 2: Performing addition command for 4 files.
DEBUG 1: Reading input file: /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_None".
ERROR :
ERROR : get_field() -> can't open data file
"/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc"
ERROR :
I assume MET is having issues reading the variable in the file, which is
denoted at TP_6h for 6-hr total precipitation. Is there a workaround so
that it can successfully process this?
Thank you kindly,
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Wed Oct 03 19:14:16 2018
Hello Andrew,
I see you have a question about reading NetCDF data into MET. Judging
by
the path you sent, I figured that this data lives on theia. So I
logged on
and took a look.
Looks like this ECMWF file doesn't exactly fall into any of the NetCDF
categories that MET currently supports. The closest is CF-compliant
NetCDF. So let's tell MET to interpret it as such using the
"file_type =
NETCDF_NCCF;" option.
When I'm trying to figure out if MET can read gridded data, I usually
use
the plot_data_plane tool.
Here's the commands I ran on theia:
module use /contrib/modulefiles
module load intel met/8.0
plot_data_plane /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
This command ran and produced the attached plot, but it did also
produce a
warning about the valid time:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0.
So MET isn't getting the timing info correct. What can we do about
this?
One option would be reformatting these NetCDF files into a format that
MET
can actually read... either CF-compliant NetCDF or the MET NetCDF
format.
However, that's pretty cumbersome. I would instead recommend that we
take
advantage of a new feature in met-8.0 (released last Friday). We can
write
a Python script to read data from these files, define the metadata,
and
then pass that to MET via memory.
If the timing information doesn't matter for some reason, then you
could
just use the "file_type = NETCDF_NCCF;" option. But if it does
matter, and
presumably it does, then the Python route should work.
Let me know how you'd like to proceed.
Thanks,
John
On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> Transaction: Ticket created by andrew.kren at noaa.gov
> Queue: met_help
> Subject: Question about MET pcp_combine
> Owner: Nobody
> Requestors: andrew.kren at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
>
> Dear MET help,
>
> I have a question on the MET pcp_combine add tool. I have model data
that
> is considered my truth from an ECMWF Nature run (long-run simulation
used
> to represent the true atmosphere). I want to verify how well my
forecast
> output, in this case from the GFS model, is compared to the Nature
run. The
> ECMWF nature run data is 6-hourly data, which includes 6-hr
precipitation
> and other fields in its netcdf files. my GFS data is grib format. I
already
> can convert the GFS data to 24-hourly accumulations using my prior
scripts
> and with pcp_combine. I guess my issue is in the ability to read the
Nature
> run netcdf files. I try the pcp_combine tool and I get this error
when
> trying to create a 24-hr accumulation.
>
> (0) Current cycle: 2006012800
>
> DEBUG 2: Performing addition command for 4 files.
>
> DEBUG 1: Reading input file:
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_None".
>
> ERROR :
>
> ERROR : get_field() -> can't open data file
> "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc"
>
> ERROR :
>
> I assume MET is having issues reading the variable in the file,
which is
> denoted at TP_6h for 6-hr total precipitation. Is there a workaround
so
> that it can successfully process this?
>
> Thank you kindly,
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 08:43:12 2018
John,
That is great. Yes let's go the python route, that seems a lot more
straightforward. Please let me know how I may proceed, and thank you
in
advance for your help on this!!
On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hello Andrew,
>
> I see you have a question about reading NetCDF data into MET.
Judging by
> the path you sent, I figured that this data lives on theia. So I
logged on
> and took a look.
>
> Looks like this ECMWF file doesn't exactly fall into any of the
NetCDF
> categories that MET currently supports. The closest is CF-compliant
> NetCDF. So let's tell MET to interpret it as such using the
"file_type =
> NETCDF_NCCF;" option.
>
> When I'm trying to figure out if MET can read gridded data, I
usually use
> the plot_data_plane tool.
>
> Here's the commands I ran on theia:
>
> module use /contrib/modulefiles
>
> module load intel met/8.0
>
> plot_data_plane /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>
> 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
>
>
> This command ran and produced the attached plot, but it did also
produce a
> warning about the valid time:
>
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
>
>
> So MET isn't getting the timing info correct. What can we do about
this?
> One option would be reformatting these NetCDF files into a format
that MET
> can actually read... either CF-compliant NetCDF or the MET NetCDF
format.
> However, that's pretty cumbersome. I would instead recommend that
we take
> advantage of a new feature in met-8.0 (released last Friday). We can
write
> a Python script to read data from these files, define the metadata,
and
> then pass that to MET via memory.
>
>
> If the timing information doesn't matter for some reason, then you
could
> just use the "file_type = NETCDF_NCCF;" option. But if it does
matter, and
> presumably it does, then the Python route should work.
>
>
> Let me know how you'd like to proceed.
>
>
> Thanks,
> John
>
>
>
>
> On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> > Transaction: Ticket created by andrew.kren at noaa.gov
> > Queue: met_help
> > Subject: Question about MET pcp_combine
> > Owner: Nobody
> > Requestors: andrew.kren at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >
> >
> > Dear MET help,
> >
> > I have a question on the MET pcp_combine add tool. I have model
data that
> > is considered my truth from an ECMWF Nature run (long-run
simulation used
> > to represent the true atmosphere). I want to verify how well my
forecast
> > output, in this case from the GFS model, is compared to the Nature
run.
> The
> > ECMWF nature run data is 6-hourly data, which includes 6-hr
precipitation
> > and other fields in its netcdf files. my GFS data is grib format.
I
> already
> > can convert the GFS data to 24-hourly accumulations using my prior
> scripts
> > and with pcp_combine. I guess my issue is in the ability to read
the
> Nature
> > run netcdf files. I try the pcp_combine tool and I get this error
when
> > trying to create a 24-hr accumulation.
> >
> > (0) Current cycle: 2006012800
> >
> > DEBUG 2: Performing addition command for 4 files.
> >
> > DEBUG 1: Reading input file:
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_None".
> >
> > ERROR :
> >
> > ERROR : get_field() -> can't open data file
> > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc"
> >
> > ERROR :
> >
> > I assume MET is having issues reading the variable in the file,
which is
> > denoted at TP_6h for 6-hr total precipitation. Is there a
workaround so
> > that it can successfully process this?
> >
> > Thank you kindly,
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
> >
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 10:31:47 2018
Andrew,
OK, question for you. Is this ECMWF actually model output with a 6522
forecast hour? Or should this be interpreted as an analysis that's
valid
at 01/28/2006 at 00Z?
That'll determine how best to encode the timing information.
Thanks,
John
On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> John,
> That is great. Yes let's go the python route, that seems a lot more
> straightforward. Please let me know how I may proceed, and thank you
in
> advance for your help on this!!
>
> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hello Andrew,
> >
> > I see you have a question about reading NetCDF data into MET.
Judging by
> > the path you sent, I figured that this data lives on theia. So I
logged
> on
> > and took a look.
> >
> > Looks like this ECMWF file doesn't exactly fall into any of the
NetCDF
> > categories that MET currently supports. The closest is CF-
compliant
> > NetCDF. So let's tell MET to interpret it as such using the
"file_type =
> > NETCDF_NCCF;" option.
> >
> > When I'm trying to figure out if MET can read gridded data, I
usually use
> > the plot_data_plane tool.
> >
> > Here's the commands I ran on theia:
> >
> > module use /contrib/modulefiles
> >
> > module load intel met/8.0
> >
> > plot_data_plane /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >
> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
> >
> >
> > This command ran and produced the attached plot, but it did also
produce
> a
> > warning about the valid time:
> >
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> >
> >
> > So MET isn't getting the timing info correct. What can we do
about this?
> > One option would be reformatting these NetCDF files into a format
that
> MET
> > can actually read... either CF-compliant NetCDF or the MET NetCDF
format.
> > However, that's pretty cumbersome. I would instead recommend that
we
> take
> > advantage of a new feature in met-8.0 (released last Friday). We
can
> write
> > a Python script to read data from these files, define the
metadata, and
> > then pass that to MET via memory.
> >
> >
> > If the timing information doesn't matter for some reason, then you
could
> > just use the "file_type = NETCDF_NCCF;" option. But if it does
matter,
> and
> > presumably it does, then the Python route should work.
> >
> >
> > Let me know how you'd like to proceed.
> >
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> > > Transaction: Ticket created by andrew.kren at noaa.gov
> > > Queue: met_help
> > > Subject: Question about MET pcp_combine
> > > Owner: Nobody
> > > Requestors: andrew.kren at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >
> > >
> > >
> > > Dear MET help,
> > >
> > > I have a question on the MET pcp_combine add tool. I have model
data
> that
> > > is considered my truth from an ECMWF Nature run (long-run
simulation
> used
> > > to represent the true atmosphere). I want to verify how well my
> forecast
> > > output, in this case from the GFS model, is compared to the
Nature run.
> > The
> > > ECMWF nature run data is 6-hourly data, which includes 6-hr
> precipitation
> > > and other fields in its netcdf files. my GFS data is grib
format. I
> > already
> > > can convert the GFS data to 24-hourly accumulations using my
prior
> > scripts
> > > and with pcp_combine. I guess my issue is in the ability to read
the
> > Nature
> > > run netcdf files. I try the pcp_combine tool and I get this
error when
> > > trying to create a 24-hr accumulation.
> > >
> > > (0) Current cycle: 2006012800
> > >
> > > DEBUG 2: Performing addition command for 4 files.
> > >
> > > DEBUG 1: Reading input file:
> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > NR2006.pl.1by1.6522.2006012806.nc
> > >
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_None".
> > >
> > > ERROR :
> > >
> > > ERROR : get_field() -> can't open data file
> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > NR2006.pl.1by1.6522.2006012806.nc"
> > >
> > > ERROR :
> > >
> > > I assume MET is having issues reading the variable in the file,
which
> is
> > > denoted at TP_6h for 6-hr total precipitation. Is there a
workaround so
> > > that it can successfully process this?
> > >
> > > Thank you kindly,
> > >
> > > --
> > > Andrew Kren
> > > Assistant Scientist
> > > University of Miami CIMAS - NOAA/AOML
> > > Duty Station: NCAR Foothills - FL2-3058
> > > Boulder, Colorado, USA
> > > Office: 303-497-2715
> > >
> > >
> >
> >
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 10:46:53 2018
Andrew,
OK, please try running this command on theia. I've attached the
resulting
image:
module use /contrib/modulefiles
module load intel met/8.0
plot_data_plane PYTHON_NUMPY TP_6h.ps \
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
'name=="python
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
This python script (also attached) reads your data and hands it off to
MET.
Here's the steps I followed to do this:
(1) Copy over a sample python script from MET:
cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
read_ECMWF_NC_2D.py
(2) Edit it to read the input netcdf file (argument 1) and the
variable
name (argument 2).
(3) Add code to parse out the timing info and define it in the "attr"
dictionary.
NOTE that I assumed that this is an analysis... so I just set the init
time
to equal the valid time. Feel free to change that however you need.
(4) Add the grid definition info. And I figured this out by running
pcp_combine (see below) and then looking at the NetCDF global
attributes.
pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc 'name="TP_6h";
level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
(5) Along the way, I ran this command to make sure my python script
actually runs on the command line:
python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc TP_6h
Please let me know how this goes for you.
Thanks,
John
On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Andrew,
>
> OK, question for you. Is this ECMWF actually model output with a
6522
> forecast hour? Or should this be interpreted as an analysis that's
valid
> at 01/28/2006 at 00Z?
>
> That'll determine how best to encode the timing information.
>
> Thanks,
> John
>
> On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>>
>> John,
>> That is great. Yes let's go the python route, that seems a lot more
>> straightforward. Please let me know how I may proceed, and thank
you in
>> advance for your help on this!!
>>
>> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Hello Andrew,
>> >
>> > I see you have a question about reading NetCDF data into MET.
Judging
>> by
>> > the path you sent, I figured that this data lives on theia. So I
>> logged on
>> > and took a look.
>> >
>> > Looks like this ECMWF file doesn't exactly fall into any of the
NetCDF
>> > categories that MET currently supports. The closest is CF-
compliant
>> > NetCDF. So let's tell MET to interpret it as such using the
"file_type
>> =
>> > NETCDF_NCCF;" option.
>> >
>> > When I'm trying to figure out if MET can read gridded data, I
usually
>> use
>> > the plot_data_plane tool.
>> >
>> > Here's the commands I ran on theia:
>> >
>> > module use /contrib/modulefiles
>> >
>> > module load intel met/8.0
>> >
>> > plot_data_plane /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>> >
>> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
>> >
>> >
>> > This command ran and produced the attached plot, but it did also
>> produce a
>> > warning about the valid time:
>> >
>> > WARNING: NcCfFile::open() -> could not determine the valid time,
using
>> 0.
>> >
>> >
>> > So MET isn't getting the timing info correct. What can we do
about
>> this?
>> > One option would be reformatting these NetCDF files into a format
that
>> MET
>> > can actually read... either CF-compliant NetCDF or the MET NetCDF
>> format.
>> > However, that's pretty cumbersome. I would instead recommend
that we
>> take
>> > advantage of a new feature in met-8.0 (released last Friday). We
can
>> write
>> > a Python script to read data from these files, define the
metadata, and
>> > then pass that to MET via memory.
>> >
>> >
>> > If the timing information doesn't matter for some reason, then
you could
>> > just use the "file_type = NETCDF_NCCF;" option. But if it does
matter,
>> and
>> > presumably it does, then the Python route should work.
>> >
>> >
>> > Let me know how you'd like to proceed.
>> >
>> >
>> > Thanks,
>> > John
>> >
>> >
>> >
>> >
>> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
>> > > Transaction: Ticket created by andrew.kren at noaa.gov
>> > > Queue: met_help
>> > > Subject: Question about MET pcp_combine
>> > > Owner: Nobody
>> > > Requestors: andrew.kren at noaa.gov
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >
>> > >
>> > >
>> > > Dear MET help,
>> > >
>> > > I have a question on the MET pcp_combine add tool. I have model
data
>> that
>> > > is considered my truth from an ECMWF Nature run (long-run
simulation
>> used
>> > > to represent the true atmosphere). I want to verify how well my
>> forecast
>> > > output, in this case from the GFS model, is compared to the
Nature
>> run.
>> > The
>> > > ECMWF nature run data is 6-hourly data, which includes 6-hr
>> precipitation
>> > > and other fields in its netcdf files. my GFS data is grib
format. I
>> > already
>> > > can convert the GFS data to 24-hourly accumulations using my
prior
>> > scripts
>> > > and with pcp_combine. I guess my issue is in the ability to
read the
>> > Nature
>> > > run netcdf files. I try the pcp_combine tool and I get this
error when
>> > > trying to create a 24-hr accumulation.
>> > >
>> > > (0) Current cycle: 2006012800
>> > >
>> > > DEBUG 2: Performing addition command for 4 files.
>> > >
>> > > DEBUG 1: Reading input file:
>> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > > NR2006.pl.1by1.6522.2006012806.nc
>> > >
>> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
>> > > Met2dDataFile object of type "FileType_None".
>> > >
>> > > ERROR :
>> > >
>> > > ERROR : get_field() -> can't open data file
>> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > > NR2006.pl.1by1.6522.2006012806.nc"
>> > >
>> > > ERROR :
>> > >
>> > > I assume MET is having issues reading the variable in the file,
which
>> is
>> > > denoted at TP_6h for 6-hr total precipitation. Is there a
workaround
>> so
>> > > that it can successfully process this?
>> > >
>> > > Thank you kindly,
>> > >
>> > > --
>> > > Andrew Kren
>> > > Assistant Scientist
>> > > University of Miami CIMAS - NOAA/AOML
>> > > Duty Station: NCAR Foothills - FL2-3058
>> > > Boulder, Colorado, USA
>> > > Office: 303-497-2715
>> > >
>> > >
>> >
>> >
>>
>> --
>> Andrew Kren
>> Assistant Scientist
>> University of Miami CIMAS - NOAA/AOML
>> Duty Station: NCAR Foothills - FL2-3058
>> Boulder, Colorado, USA
>> Office: 303-497-2715
>>
>>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 10:56:33 2018
John,
I believe the 6522 and other numbers (which increase with each date)
were
used as place holders for the filename. The actual dates can be read
as an
analysis that's valid at the time stamp, such as 01/28/2006 at 06Z?
Thus
the filename NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond to the
analysis valid at 2006012806, does that help? I'll try the script.
On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
>
> OK, please try running this command on theia. I've attached the
resulting
> image:
>
>
> module use /contrib/modulefiles
>
> module load intel met/8.0
>
> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>
> 'name=="python
>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
>
>
> This python script (also attached) reads your data and hands it off
to MET.
>
>
> Here's the steps I followed to do this:
>
>
> (1) Copy over a sample python script from MET:
>
> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> read_ECMWF_NC_2D.py
>
>
> (2) Edit it to read the input netcdf file (argument 1) and the
variable
> name (argument 2).
>
>
> (3) Add code to parse out the timing info and define it in the
"attr"
> dictionary.
>
> NOTE that I assumed that this is an analysis... so I just set the
init time
> to equal the valid time. Feel free to change that however you need.
>
>
> (4) Add the grid definition info. And I figured this out by running
> pcp_combine (see below) and then looking at the NetCDF global
attributes.
>
>
> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc 'name="TP_6h";
> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
>
>
> (5) Along the way, I ran this command to make sure my python script
> actually runs on the command line:
>
> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc TP_6h
>
>
> Please let me know how this goes for you.
>
>
> Thanks,
> John
>
> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Andrew,
> >
> > OK, question for you. Is this ECMWF actually model output with a
6522
> > forecast hour? Or should this be interpreted as an analysis
that's valid
> > at 01/28/2006 at 00Z?
> >
> > That'll determine how best to encode the timing information.
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >>
> >> John,
> >> That is great. Yes let's go the python route, that seems a lot
more
> >> straightforward. Please let me know how I may proceed, and thank
you in
> >> advance for your help on this!!
> >>
> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hello Andrew,
> >> >
> >> > I see you have a question about reading NetCDF data into MET.
Judging
> >> by
> >> > the path you sent, I figured that this data lives on theia. So
I
> >> logged on
> >> > and took a look.
> >> >
> >> > Looks like this ECMWF file doesn't exactly fall into any of the
NetCDF
> >> > categories that MET currently supports. The closest is CF-
compliant
> >> > NetCDF. So let's tell MET to interpret it as such using the
> "file_type
> >> =
> >> > NETCDF_NCCF;" option.
> >> >
> >> > When I'm trying to figure out if MET can read gridded data, I
usually
> >> use
> >> > the plot_data_plane tool.
> >> >
> >> > Here's the commands I ran on theia:
> >> >
> >> > module use /contrib/modulefiles
> >> >
> >> > module load intel met/8.0
> >> >
> >> > plot_data_plane /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >> >
> >> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
> >> >
> >> >
> >> > This command ran and produced the attached plot, but it did
also
> >> produce a
> >> > warning about the valid time:
> >> >
> >> > WARNING: NcCfFile::open() -> could not determine the valid
time, using
> >> 0.
> >> >
> >> >
> >> > So MET isn't getting the timing info correct. What can we do
about
> >> this?
> >> > One option would be reformatting these NetCDF files into a
format that
> >> MET
> >> > can actually read... either CF-compliant NetCDF or the MET
NetCDF
> >> format.
> >> > However, that's pretty cumbersome. I would instead recommend
that we
> >> take
> >> > advantage of a new feature in met-8.0 (released last Friday).
We can
> >> write
> >> > a Python script to read data from these files, define the
metadata,
> and
> >> > then pass that to MET via memory.
> >> >
> >> >
> >> > If the timing information doesn't matter for some reason, then
you
> could
> >> > just use the "file_type = NETCDF_NCCF;" option. But if it does
> matter,
> >> and
> >> > presumably it does, then the Python route should work.
> >> >
> >> >
> >> > Let me know how you'd like to proceed.
> >> >
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate via
RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
> >> > > Queue: met_help
> >> > > Subject: Question about MET pcp_combine
> >> > > Owner: Nobody
> >> > > Requestors: andrew.kren at noaa.gov
> >> > > Status: new
> >> > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >
> >> > >
> >> > >
> >> > > Dear MET help,
> >> > >
> >> > > I have a question on the MET pcp_combine add tool. I have
model data
> >> that
> >> > > is considered my truth from an ECMWF Nature run (long-run
simulation
> >> used
> >> > > to represent the true atmosphere). I want to verify how well
my
> >> forecast
> >> > > output, in this case from the GFS model, is compared to the
Nature
> >> run.
> >> > The
> >> > > ECMWF nature run data is 6-hourly data, which includes 6-hr
> >> precipitation
> >> > > and other fields in its netcdf files. my GFS data is grib
format. I
> >> > already
> >> > > can convert the GFS data to 24-hourly accumulations using my
prior
> >> > scripts
> >> > > and with pcp_combine. I guess my issue is in the ability to
read the
> >> > Nature
> >> > > run netcdf files. I try the pcp_combine tool and I get this
error
> when
> >> > > trying to create a 24-hr accumulation.
> >> > >
> >> > > (0) Current cycle: 2006012800
> >> > >
> >> > > DEBUG 2: Performing addition command for 4 files.
> >> > >
> >> > > DEBUG 1: Reading input file:
> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> > >
> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >> > > Met2dDataFile object of type "FileType_None".
> >> > >
> >> > > ERROR :
> >> > >
> >> > > ERROR : get_field() -> can't open data file
> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> >> > >
> >> > > ERROR :
> >> > >
> >> > > I assume MET is having issues reading the variable in the
file,
> which
> >> is
> >> > > denoted at TP_6h for 6-hr total precipitation. Is there a
workaround
> >> so
> >> > > that it can successfully process this?
> >> > >
> >> > > Thank you kindly,
> >> > >
> >> > > --
> >> > > Andrew Kren
> >> > > Assistant Scientist
> >> > > University of Miami CIMAS - NOAA/AOML
> >> > > Duty Station: NCAR Foothills - FL2-3058
> >> > > Boulder, Colorado, USA
> >> > > Office: 303-497-2715
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> Andrew Kren
> >> Assistant Scientist
> >> University of Miami CIMAS - NOAA/AOML
> >> Duty Station: NCAR Foothills - FL2-3058
> >> Boulder, Colorado, USA
> >> Office: 303-497-2715
> >>
> >>
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 11:13:47 2018
John,
I got this output when I ran your python script in a shell script:
The following have been reloaded with a version change:
1) intel/15.0.0 => intel/14.0.2
Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
NR2006.pl.1by1.6522.2006012806.nc'
Variable Name : 'TP_6h'
Data Shape: (181, 360)
Data Type: dtype('float64')
Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000', 'valid':
'20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
'units': 'mm', 'accum': '06'}
Looks like it worked.
On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
andrew.kren at noaa.gov> wrote:
> John,
> I believe the 6522 and other numbers (which increase with each date)
were
> used as place holders for the filename. The actual dates can be read
as an
> analysis that's valid at the time stamp, such as 01/28/2006 at 06Z?
Thus
> the filename NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond to the
> analysis valid at 2006012806, does that help? I'll try the script.
>
> On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Andrew,
>>
>>
>> OK, please try running this command on theia. I've attached the
resulting
>> image:
>>
>>
>> module use /contrib/modulefiles
>>
>> module load intel met/8.0
>>
>> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>>
>> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> NR2006.pl.1by1.6522.2006012806.nc
>> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>>
>> 'name=="python
>>
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
>>
>>
>> This python script (also attached) reads your data and hands it off
to
>> MET.
>>
>>
>> Here's the steps I followed to do this:
>>
>>
>> (1) Copy over a sample python script from MET:
>>
>> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
>> read_ECMWF_NC_2D.py
>>
>>
>> (2) Edit it to read the input netcdf file (argument 1) and the
variable
>> name (argument 2).
>>
>>
>> (3) Add code to parse out the timing info and define it in the
"attr"
>> dictionary.
>>
>> NOTE that I assumed that this is an analysis... so I just set the
init
>> time
>> to equal the valid time. Feel free to change that however you
need.
>>
>>
>> (4) Add the grid definition info. And I figured this out by
running
>> pcp_combine (see below) and then looking at the NetCDF global
attributes.
>>
>>
>> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc 'name="TP_6h";
>> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
>>
>>
>> (5) Along the way, I ran this command to make sure my python script
>> actually runs on the command line:
>>
>> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc TP_6h
>>
>>
>> Please let me know how this goes for you.
>>
>>
>> Thanks,
>> John
>>
>> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>
>> > Andrew,
>> >
>> > OK, question for you. Is this ECMWF actually model output with a
6522
>> > forecast hour? Or should this be interpreted as an analysis
that's
>> valid
>> > at 01/28/2006 at 00Z?
>> >
>> > That'll determine how best to encode the timing information.
>> >
>> > Thanks,
>> > John
>> >
>> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>> >>
>> >> John,
>> >> That is great. Yes let's go the python route, that seems a lot
more
>> >> straightforward. Please let me know how I may proceed, and thank
you in
>> >> advance for your help on this!!
>> >>
>> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
>> >> met_help at ucar.edu>
>> >> wrote:
>> >>
>> >> > Hello Andrew,
>> >> >
>> >> > I see you have a question about reading NetCDF data into MET.
>> Judging
>> >> by
>> >> > the path you sent, I figured that this data lives on theia.
So I
>> >> logged on
>> >> > and took a look.
>> >> >
>> >> > Looks like this ECMWF file doesn't exactly fall into any of
the
>> NetCDF
>> >> > categories that MET currently supports. The closest is CF-
compliant
>> >> > NetCDF. So let's tell MET to interpret it as such using the
>> "file_type
>> >> =
>> >> > NETCDF_NCCF;" option.
>> >> >
>> >> > When I'm trying to figure out if MET can read gridded data, I
usually
>> >> use
>> >> > the plot_data_plane tool.
>> >> >
>> >> > Here's the commands I ran on theia:
>> >> >
>> >> > module use /contrib/modulefiles
>> >> >
>> >> > module load intel met/8.0
>> >> >
>> >> > plot_data_plane
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>> >> >
>> >> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
>> >> >
>> >> >
>> >> > This command ran and produced the attached plot, but it did
also
>> >> produce a
>> >> > warning about the valid time:
>> >> >
>> >> > WARNING: NcCfFile::open() -> could not determine the valid
time,
>> using
>> >> 0.
>> >> >
>> >> >
>> >> > So MET isn't getting the timing info correct. What can we do
about
>> >> this?
>> >> > One option would be reformatting these NetCDF files into a
format
>> that
>> >> MET
>> >> > can actually read... either CF-compliant NetCDF or the MET
NetCDF
>> >> format.
>> >> > However, that's pretty cumbersome. I would instead recommend
that we
>> >> take
>> >> > advantage of a new feature in met-8.0 (released last Friday).
We can
>> >> write
>> >> > a Python script to read data from these files, define the
metadata,
>> and
>> >> > then pass that to MET via memory.
>> >> >
>> >> >
>> >> > If the timing information doesn't matter for some reason, then
you
>> could
>> >> > just use the "file_type = NETCDF_NCCF;" option. But if it
does
>> matter,
>> >> and
>> >> > presumably it does, then the Python route should work.
>> >> >
>> >> >
>> >> > Let me know how you'd like to proceed.
>> >> >
>> >> >
>> >> > Thanks,
>> >> > John
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate
via RT <
>> >> > met_help at ucar.edu> wrote:
>> >> >
>> >> > >
>> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
>> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
>> >> > > Queue: met_help
>> >> > > Subject: Question about MET pcp_combine
>> >> > > Owner: Nobody
>> >> > > Requestors: andrew.kren at noaa.gov
>> >> > > Status: new
>> >> > > Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >> >
>> >> > >
>> >> > >
>> >> > > Dear MET help,
>> >> > >
>> >> > > I have a question on the MET pcp_combine add tool. I have
model
>> data
>> >> that
>> >> > > is considered my truth from an ECMWF Nature run (long-run
>> simulation
>> >> used
>> >> > > to represent the true atmosphere). I want to verify how well
my
>> >> forecast
>> >> > > output, in this case from the GFS model, is compared to the
Nature
>> >> run.
>> >> > The
>> >> > > ECMWF nature run data is 6-hourly data, which includes 6-hr
>> >> precipitation
>> >> > > and other fields in its netcdf files. my GFS data is grib
format. I
>> >> > already
>> >> > > can convert the GFS data to 24-hourly accumulations using my
prior
>> >> > scripts
>> >> > > and with pcp_combine. I guess my issue is in the ability to
read
>> the
>> >> > Nature
>> >> > > run netcdf files. I try the pcp_combine tool and I get this
error
>> when
>> >> > > trying to create a 24-hr accumulation.
>> >> > >
>> >> > > (0) Current cycle: 2006012800
>> >> > >
>> >> > > DEBUG 2: Performing addition command for 4 files.
>> >> > >
>> >> > > DEBUG 1: Reading input file:
>> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> >> > >
>> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> >> > > Met2dDataFile object of type "FileType_None".
>> >> > >
>> >> > > ERROR :
>> >> > >
>> >> > > ERROR : get_field() -> can't open data file
>> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > > NR2006.pl.1by1.6522.2006012806.nc"
>> >> > >
>> >> > > ERROR :
>> >> > >
>> >> > > I assume MET is having issues reading the variable in the
file,
>> which
>> >> is
>> >> > > denoted at TP_6h for 6-hr total precipitation. Is there a
>> workaround
>> >> so
>> >> > > that it can successfully process this?
>> >> > >
>> >> > > Thank you kindly,
>> >> > >
>> >> > > --
>> >> > > Andrew Kren
>> >> > > Assistant Scientist
>> >> > > University of Miami CIMAS - NOAA/AOML
>> >> > > Duty Station: NCAR Foothills - FL2-3058
>> >> > > Boulder, Colorado, USA
>> >> > > Office: 303-497-2715
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >>
>> >> --
>> >> Andrew Kren
>> >> Assistant Scientist
>> >> University of Miami CIMAS - NOAA/AOML
>> >> Duty Station: NCAR Foothills - FL2-3058
>> >> Boulder, Colorado, USA
>> >> Office: 303-497-2715
>> >>
>> >>
>>
>>
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 11:30:37 2018
Great, so running the python script stand-alone read the data well.
The next step is telling MET to call this script. Here's that simple
example of running plot_data_plane:
plot_data_plane PYTHON_NUMPY TP_6h.ps \
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
'name=="python
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
Once this works, look at the output to confirm that MET is putting
your
data on the right spot on the earth.
Presumably it does... so now you're ready to start using this to run
pcp_combine or Grid-Stat or MODE.
Make sense?
John
On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> John,
> I got this output when I ran your python script in a shell script:
>
> The following have been reloaded with a version change:
>
> 1) intel/15.0.0 => intel/14.0.2
>
>
> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> NR2006.pl.1by1.6522.2006012806.nc'
>
> Variable Name : 'TP_6h'
>
> Data Shape: (181, 360)
>
> Data Type: dtype('float64')
>
> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
> 'units': 'mm', 'accum': '06'}
>
>
> Looks like it worked.
>
> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
> andrew.kren at noaa.gov> wrote:
>
> > John,
> > I believe the 6522 and other numbers (which increase with each
date) were
> > used as place holders for the filename. The actual dates can be
read as
> an
> > analysis that's valid at the time stamp, such as 01/28/2006 at
06Z? Thus
> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond to
the
> > analysis valid at 2006012806, does that help? I'll try the script.
> >
> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Andrew,
> >>
> >>
> >> OK, please try running this command on theia. I've attached the
> resulting
> >> image:
> >>
> >>
> >> module use /contrib/modulefiles
> >>
> >> module load intel met/8.0
> >>
> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >>
> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> NR2006.pl.1by1.6522.2006012806.nc
> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >>
> >> 'name=="python
> >>
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> >>
> >>
> >> This python script (also attached) reads your data and hands it
off to
> >> MET.
> >>
> >>
> >> Here's the steps I followed to do this:
> >>
> >>
> >> (1) Copy over a sample python script from MET:
> >>
> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> >> read_ECMWF_NC_2D.py
> >>
> >>
> >> (2) Edit it to read the input netcdf file (argument 1) and the
variable
> >> name (argument 2).
> >>
> >>
> >> (3) Add code to parse out the timing info and define it in the
"attr"
> >> dictionary.
> >>
> >> NOTE that I assumed that this is an analysis... so I just set the
init
> >> time
> >> to equal the valid time. Feel free to change that however you
need.
> >>
> >>
> >> (4) Add the grid definition info. And I figured this out by
running
> >> pcp_combine (see below) and then looking at the NetCDF global
> attributes.
> >>
> >>
> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc 'name="TP_6h";
> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> >>
> >>
> >> (5) Along the way, I ran this command to make sure my python
script
> >> actually runs on the command line:
> >>
> >> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc
TP_6h
> >>
> >>
> >> Please let me know how this goes for you.
> >>
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >> > Andrew,
> >> >
> >> > OK, question for you. Is this ECMWF actually model output with
a 6522
> >> > forecast hour? Or should this be interpreted as an analysis
that's
> >> valid
> >> > at 01/28/2006 at 00Z?
> >> >
> >> > That'll determine how best to encode the timing information.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate via
RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
> >> >>
> >> >> John,
> >> >> That is great. Yes let's go the python route, that seems a lot
more
> >> >> straightforward. Please let me know how I may proceed, and
thank you
> in
> >> >> advance for your help on this!!
> >> >>
> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
> >> >> met_help at ucar.edu>
> >> >> wrote:
> >> >>
> >> >> > Hello Andrew,
> >> >> >
> >> >> > I see you have a question about reading NetCDF data into
MET.
> >> Judging
> >> >> by
> >> >> > the path you sent, I figured that this data lives on theia.
So I
> >> >> logged on
> >> >> > and took a look.
> >> >> >
> >> >> > Looks like this ECMWF file doesn't exactly fall into any of
the
> >> NetCDF
> >> >> > categories that MET currently supports. The closest is
> CF-compliant
> >> >> > NetCDF. So let's tell MET to interpret it as such using the
> >> "file_type
> >> >> =
> >> >> > NETCDF_NCCF;" option.
> >> >> >
> >> >> > When I'm trying to figure out if MET can read gridded data,
I
> usually
> >> >> use
> >> >> > the plot_data_plane tool.
> >> >> >
> >> >> > Here's the commands I ran on theia:
> >> >> >
> >> >> > module use /contrib/modulefiles
> >> >> >
> >> >> > module load intel met/8.0
> >> >> >
> >> >> > plot_data_plane
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >> >> >
> >> >> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
> >> >> >
> >> >> >
> >> >> > This command ran and produced the attached plot, but it did
also
> >> >> produce a
> >> >> > warning about the valid time:
> >> >> >
> >> >> > WARNING: NcCfFile::open() -> could not determine the valid
time,
> >> using
> >> >> 0.
> >> >> >
> >> >> >
> >> >> > So MET isn't getting the timing info correct. What can we
do about
> >> >> this?
> >> >> > One option would be reformatting these NetCDF files into a
format
> >> that
> >> >> MET
> >> >> > can actually read... either CF-compliant NetCDF or the MET
NetCDF
> >> >> format.
> >> >> > However, that's pretty cumbersome. I would instead
recommend that
> we
> >> >> take
> >> >> > advantage of a new feature in met-8.0 (released last
Friday). We
> can
> >> >> write
> >> >> > a Python script to read data from these files, define the
metadata,
> >> and
> >> >> > then pass that to MET via memory.
> >> >> >
> >> >> >
> >> >> > If the timing information doesn't matter for some reason,
then you
> >> could
> >> >> > just use the "file_type = NETCDF_NCCF;" option. But if it
does
> >> matter,
> >> >> and
> >> >> > presumably it does, then the Python route should work.
> >> >> >
> >> >> >
> >> >> > Let me know how you'd like to proceed.
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > John
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate
via RT
> <
> >> >> > met_help at ucar.edu> wrote:
> >> >> >
> >> >> > >
> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> >> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
> >> >> > > Queue: met_help
> >> >> > > Subject: Question about MET pcp_combine
> >> >> > > Owner: Nobody
> >> >> > > Requestors: andrew.kren at noaa.gov
> >> >> > > Status: new
> >> >> > > Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >> >
> >> >> > >
> >> >> > >
> >> >> > > Dear MET help,
> >> >> > >
> >> >> > > I have a question on the MET pcp_combine add tool. I have
model
> >> data
> >> >> that
> >> >> > > is considered my truth from an ECMWF Nature run (long-run
> >> simulation
> >> >> used
> >> >> > > to represent the true atmosphere). I want to verify how
well my
> >> >> forecast
> >> >> > > output, in this case from the GFS model, is compared to
the
> Nature
> >> >> run.
> >> >> > The
> >> >> > > ECMWF nature run data is 6-hourly data, which includes 6-
hr
> >> >> precipitation
> >> >> > > and other fields in its netcdf files. my GFS data is grib
> format. I
> >> >> > already
> >> >> > > can convert the GFS data to 24-hourly accumulations using
my
> prior
> >> >> > scripts
> >> >> > > and with pcp_combine. I guess my issue is in the ability
to read
> >> the
> >> >> > Nature
> >> >> > > run netcdf files. I try the pcp_combine tool and I get
this error
> >> when
> >> >> > > trying to create a 24-hr accumulation.
> >> >> > >
> >> >> > > (0) Current cycle: 2006012800
> >> >> > >
> >> >> > > DEBUG 2: Performing addition command for 4 files.
> >> >> > >
> >> >> > > DEBUG 1: Reading input file:
> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > >
> >> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> >> new
> >> >> > > Met2dDataFile object of type "FileType_None".
> >> >> > >
> >> >> > > ERROR :
> >> >> > >
> >> >> > > ERROR : get_field() -> can't open data file
> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> >> >> > >
> >> >> > > ERROR :
> >> >> > >
> >> >> > > I assume MET is having issues reading the variable in the
file,
> >> which
> >> >> is
> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is there a
> >> workaround
> >> >> so
> >> >> > > that it can successfully process this?
> >> >> > >
> >> >> > > Thank you kindly,
> >> >> > >
> >> >> > > --
> >> >> > > Andrew Kren
> >> >> > > Assistant Scientist
> >> >> > > University of Miami CIMAS - NOAA/AOML
> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> >> >> > > Boulder, Colorado, USA
> >> >> > > Office: 303-497-2715
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Andrew Kren
> >> >> Assistant Scientist
> >> >> University of Miami CIMAS - NOAA/AOML
> >> >> Duty Station: NCAR Foothills - FL2-3058
> >> >> Boulder, Colorado, USA
> >> >> Office: 303-497-2715
> >> >>
> >> >>
> >>
> >>
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
>
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 11:31:41 2018
Andrew,
Sorry, just noticed a typo after I hit send. I had a double-equals
(==)
that should be a single equal (=):
plot_data_plane PYTHON_NUMPY TP_6h.ps \
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
'name="python
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.6522.2006012806.nc
<http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Great, so running the python script stand-alone read the data well.
>
> The next step is telling MET to call this script. Here's that
simple
> example of running plot_data_plane:
>
> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>
> 'name=="python
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>
>
> Once this works, look at the output to confirm that MET is putting
your
> data on the right spot on the earth.
>
>
> Presumably it does... so now you're ready to start using this to run
> pcp_combine or Grid-Stat or MODE.
>
>
> Make sense?
>
>
> John
>
> On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>>
>> John,
>> I got this output when I ran your python script in a shell script:
>>
>> The following have been reloaded with a version change:
>>
>> 1) intel/15.0.0 => intel/14.0.2
>>
>>
>> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> NR2006.pl.1by1.6522.2006012806.nc'
>>
>> Variable Name : 'TP_6h'
>>
>> Data Shape: (181, 360)
>>
>> Data Type: dtype('float64')
>>
>> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
>> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
>> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
>> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
>> 'units': 'mm', 'accum': '06'}
>>
>>
>> Looks like it worked.
>>
>> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
>> andrew.kren at noaa.gov> wrote:
>>
>> > John,
>> > I believe the 6522 and other numbers (which increase with each
date)
>> were
>> > used as place holders for the filename. The actual dates can be
read as
>> an
>> > analysis that's valid at the time stamp, such as 01/28/2006 at
06Z? Thus
>> > the filename NR2006.pl.1by1.6522.2006012806.nc
>> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond to
the
>> > analysis valid at 2006012806, does that help? I'll try the
script.
>> >
>> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> Andrew,
>> >>
>> >>
>> >> OK, please try running this command on theia. I've attached the
>> resulting
>> >> image:
>> >>
>> >>
>> >> module use /contrib/modulefiles
>> >>
>> >> module load intel met/8.0
>> >>
>> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> >>
>> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> NR2006.pl.1by1.6522.2006012806.nc
>> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> >>
>> >> 'name=="python
>> >>
>> >>
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
>> >>
>> >>
>> >> This python script (also attached) reads your data and hands it
off to
>> >> MET.
>> >>
>> >>
>> >> Here's the steps I followed to do this:
>> >>
>> >>
>> >> (1) Copy over a sample python script from MET:
>> >>
>> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
>> >> read_ECMWF_NC_2D.py
>> >>
>> >>
>> >> (2) Edit it to read the input netcdf file (argument 1) and the
variable
>> >> name (argument 2).
>> >>
>> >>
>> >> (3) Add code to parse out the timing info and define it in the
"attr"
>> >> dictionary.
>> >>
>> >> NOTE that I assumed that this is an analysis... so I just set
the init
>> >> time
>> >> to equal the valid time. Feel free to change that however you
need.
>> >>
>> >>
>> >> (4) Add the grid definition info. And I figured this out by
running
>> >> pcp_combine (see below) and then looking at the NetCDF global
>> attributes.
>> >>
>> >>
>> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
'name="TP_6h";
>> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
>> >>
>> >>
>> >> (5) Along the way, I ran this command to make sure my python
script
>> >> actually runs on the command line:
>> >>
>> >> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc
TP_6h
>> >>
>> >>
>> >> Please let me know how this goes for you.
>> >>
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway
<johnhg at ucar.edu>
>> >> wrote:
>> >>
>> >> > Andrew,
>> >> >
>> >> > OK, question for you. Is this ECMWF actually model output
with a
>> 6522
>> >> > forecast hour? Or should this be interpreted as an analysis
that's
>> >> valid
>> >> > at 01/28/2006 at 00Z?
>> >> >
>> >> > That'll determine how best to encode the timing information.
>> >> >
>> >> > Thanks,
>> >> > John
>> >> >
>> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate
via RT <
>> >> > met_help at ucar.edu> wrote:
>> >> >
>> >> >>
>> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
>> >> >>
>> >> >> John,
>> >> >> That is great. Yes let's go the python route, that seems a
lot more
>> >> >> straightforward. Please let me know how I may proceed, and
thank
>> you in
>> >> >> advance for your help on this!!
>> >> >>
>> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
>> >> >> met_help at ucar.edu>
>> >> >> wrote:
>> >> >>
>> >> >> > Hello Andrew,
>> >> >> >
>> >> >> > I see you have a question about reading NetCDF data into
MET.
>> >> Judging
>> >> >> by
>> >> >> > the path you sent, I figured that this data lives on theia.
So I
>> >> >> logged on
>> >> >> > and took a look.
>> >> >> >
>> >> >> > Looks like this ECMWF file doesn't exactly fall into any of
the
>> >> NetCDF
>> >> >> > categories that MET currently supports. The closest is
>> CF-compliant
>> >> >> > NetCDF. So let's tell MET to interpret it as such using
the
>> >> "file_type
>> >> >> =
>> >> >> > NETCDF_NCCF;" option.
>> >> >> >
>> >> >> > When I'm trying to figure out if MET can read gridded data,
I
>> usually
>> >> >> use
>> >> >> > the plot_data_plane tool.
>> >> >> >
>> >> >> > Here's the commands I ran on theia:
>> >> >> >
>> >> >> > module use /contrib/modulefiles
>> >> >> >
>> >> >> > module load intel met/8.0
>> >> >> >
>> >> >> > plot_data_plane
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>> >> >> >
>> >> >> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
>> >> >> >
>> >> >> >
>> >> >> > This command ran and produced the attached plot, but it did
also
>> >> >> produce a
>> >> >> > warning about the valid time:
>> >> >> >
>> >> >> > WARNING: NcCfFile::open() -> could not determine the valid
time,
>> >> using
>> >> >> 0.
>> >> >> >
>> >> >> >
>> >> >> > So MET isn't getting the timing info correct. What can we
do
>> about
>> >> >> this?
>> >> >> > One option would be reformatting these NetCDF files into a
format
>> >> that
>> >> >> MET
>> >> >> > can actually read... either CF-compliant NetCDF or the MET
NetCDF
>> >> >> format.
>> >> >> > However, that's pretty cumbersome. I would instead
recommend
>> that we
>> >> >> take
>> >> >> > advantage of a new feature in met-8.0 (released last
Friday). We
>> can
>> >> >> write
>> >> >> > a Python script to read data from these files, define the
>> metadata,
>> >> and
>> >> >> > then pass that to MET via memory.
>> >> >> >
>> >> >> >
>> >> >> > If the timing information doesn't matter for some reason,
then you
>> >> could
>> >> >> > just use the "file_type = NETCDF_NCCF;" option. But if it
does
>> >> matter,
>> >> >> and
>> >> >> > presumably it does, then the Python route should work.
>> >> >> >
>> >> >> >
>> >> >> > Let me know how you'd like to proceed.
>> >> >> >
>> >> >> >
>> >> >> > Thanks,
>> >> >> > John
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA Affiliate
via
>> RT <
>> >> >> > met_help at ucar.edu> wrote:
>> >> >> >
>> >> >> > >
>> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
>> >> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
>> >> >> > > Queue: met_help
>> >> >> > > Subject: Question about MET pcp_combine
>> >> >> > > Owner: Nobody
>> >> >> > > Requestors: andrew.kren at noaa.gov
>> >> >> > > Status: new
>> >> >> > > Ticket <URL:
>> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >> >> >
>> >> >> > >
>> >> >> > >
>> >> >> > > Dear MET help,
>> >> >> > >
>> >> >> > > I have a question on the MET pcp_combine add tool. I have
model
>> >> data
>> >> >> that
>> >> >> > > is considered my truth from an ECMWF Nature run (long-run
>> >> simulation
>> >> >> used
>> >> >> > > to represent the true atmosphere). I want to verify how
well my
>> >> >> forecast
>> >> >> > > output, in this case from the GFS model, is compared to
the
>> Nature
>> >> >> run.
>> >> >> > The
>> >> >> > > ECMWF nature run data is 6-hourly data, which includes 6-
hr
>> >> >> precipitation
>> >> >> > > and other fields in its netcdf files. my GFS data is grib
>> format. I
>> >> >> > already
>> >> >> > > can convert the GFS data to 24-hourly accumulations using
my
>> prior
>> >> >> > scripts
>> >> >> > > and with pcp_combine. I guess my issue is in the ability
to read
>> >> the
>> >> >> > Nature
>> >> >> > > run netcdf files. I try the pcp_combine tool and I get
this
>> error
>> >> when
>> >> >> > > trying to create a 24-hr accumulation.
>> >> >> > >
>> >> >> > > (0) Current cycle: 2006012800
>> >> >> > >
>> >> >> > > DEBUG 2: Performing addition command for 4 files.
>> >> >> > >
>> >> >> > > DEBUG 1: Reading input file:
>> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> >> >> > >
>> >> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> >> new
>> >> >> > > Met2dDataFile object of type "FileType_None".
>> >> >> > >
>> >> >> > > ERROR :
>> >> >> > >
>> >> >> > > ERROR : get_field() -> can't open data file
>> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
>> >> >> > >
>> >> >> > > ERROR :
>> >> >> > >
>> >> >> > > I assume MET is having issues reading the variable in the
file,
>> >> which
>> >> >> is
>> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is there a
>> >> workaround
>> >> >> so
>> >> >> > > that it can successfully process this?
>> >> >> > >
>> >> >> > > Thank you kindly,
>> >> >> > >
>> >> >> > > --
>> >> >> > > Andrew Kren
>> >> >> > > Assistant Scientist
>> >> >> > > University of Miami CIMAS - NOAA/AOML
>> >> >> > > Duty Station: NCAR Foothills - FL2-3058
>> >> >> > > Boulder, Colorado, USA
>> >> >> > > Office: 303-497-2715
>> >> >> > >
>> >> >> > >
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> Andrew Kren
>> >> >> Assistant Scientist
>> >> >> University of Miami CIMAS - NOAA/AOML
>> >> >> Duty Station: NCAR Foothills - FL2-3058
>> >> >> Boulder, Colorado, USA
>> >> >> Office: 303-497-2715
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>> > --
>> > Andrew Kren
>> > Assistant Scientist
>> > University of Miami CIMAS - NOAA/AOML
>> > Duty Station: NCAR Foothills - FL2-3058
>> > Boulder, Colorado, USA
>> > Office: 303-497-2715
>> >
>>
>>
>> --
>> Andrew Kren
>> Assistant Scientist
>> University of Miami CIMAS - NOAA/AOML
>> Duty Station: NCAR Foothills - FL2-3058
>> Boulder, Colorado, USA
>> Office: 303-497-2715
>>
>>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 11:41:48 2018
John,
I ran this script,
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
and
it gave me this, presumably asking for more information to run the met
command.
The following have been reloaded with a version change:
1) intel/15.0.0 => intel/14.0.2
Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
NR2006.pl.1by1.6522.2006012806.nc'
Variable Name : 'TP_6h'
Data Shape: (181, 360)
Data Type: dtype('float64')
Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000', 'valid':
'20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
'units': 'mm', 'accum': '06'}
*** Model Evaluation Tools (METV8.0) ***
Usage: plot_data_plane
input_filename
output_filename
field_string
[-color_table color_table_name]
[-plot_range min max]
[-title title_string]
[-log file]
[-v level]
where "input_filename" is the name of a gridded data file to be
plotted
(required).
"output_filename" is the name of the output PostScript file to be
written
(required).
"field_string" defines the data to be plotted from the input file
(required).
"-color_table color_table_name" overrides the default color table
("colortables/met_default.ctable") (optional).
"-plot_range min max" defines the range of the data to be plotted
(optional).
"-title title_string" specifies the plot title string (optional).
"-log file" outputs log messages to the specified file (optional).
"-v level" overrides the default level of logging (2) (optional).
On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
> Sorry, just noticed a typo after I hit send. I had a double-equals
(==)
> that should be a single equal (=):
>
> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>
> 'name="python
>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.6522.2006012806.nc
> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>
> On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Great, so running the python script stand-alone read the data
well.
> >
> > The next step is telling MET to call this script. Here's that
simple
> > example of running plot_data_plane:
> >
> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc
> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >
> > 'name=="python
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc
> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >
> >
> > Once this works, look at the output to confirm that MET is putting
your
> > data on the right spot on the earth.
> >
> >
> > Presumably it does... so now you're ready to start using this to
run
> > pcp_combine or Grid-Stat or MODE.
> >
> >
> > Make sense?
> >
> >
> > John
> >
> > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >>
> >> John,
> >> I got this output when I ran your python script in a shell
script:
> >>
> >> The following have been reloaded with a version change:
> >>
> >> 1) intel/15.0.0 => intel/14.0.2
> >>
> >>
> >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> NR2006.pl.1by1.6522.2006012806.nc'
> >>
> >> Variable Name : 'TP_6h'
> >>
> >> Data Shape: (181, 360)
> >>
> >> Data Type: dtype('float64')
> >>
> >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> 'Surface',
> >> 'units': 'mm', 'accum': '06'}
> >>
> >>
> >> Looks like it worked.
> >>
> >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
> >> andrew.kren at noaa.gov> wrote:
> >>
> >> > John,
> >> > I believe the 6522 and other numbers (which increase with each
date)
> >> were
> >> > used as place holders for the filename. The actual dates can be
read
> as
> >> an
> >> > analysis that's valid at the time stamp, such as 01/28/2006 at
06Z?
> Thus
> >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond to
the
> >> > analysis valid at 2006012806, does that help? I'll try the
script.
> >> >
> >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >> Andrew,
> >> >>
> >> >>
> >> >> OK, please try running this command on theia. I've attached
the
> >> resulting
> >> >> image:
> >> >>
> >> >>
> >> >> module use /contrib/modulefiles
> >> >>
> >> >> module load intel met/8.0
> >> >>
> >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> >>
> >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> NR2006.pl.1by1.6522.2006012806.nc
> >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> >>
> >> >> 'name=="python
> >> >>
> >> >>
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> >> >>
> >> >>
> >> >> This python script (also attached) reads your data and hands
it off
> to
> >> >> MET.
> >> >>
> >> >>
> >> >> Here's the steps I followed to do this:
> >> >>
> >> >>
> >> >> (1) Copy over a sample python script from MET:
> >> >>
> >> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> >> >> read_ECMWF_NC_2D.py
> >> >>
> >> >>
> >> >> (2) Edit it to read the input netcdf file (argument 1) and the
> variable
> >> >> name (argument 2).
> >> >>
> >> >>
> >> >> (3) Add code to parse out the timing info and define it in the
"attr"
> >> >> dictionary.
> >> >>
> >> >> NOTE that I assumed that this is an analysis... so I just set
the
> init
> >> >> time
> >> >> to equal the valid time. Feel free to change that however you
need.
> >> >>
> >> >>
> >> >> (4) Add the grid definition info. And I figured this out by
running
> >> >> pcp_combine (see below) and then looking at the NetCDF global
> >> attributes.
> >> >>
> >> >>
> >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
'name="TP_6h";
> >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> >> >>
> >> >>
> >> >> (5) Along the way, I ran this command to make sure my python
script
> >> >> actually runs on the command line:
> >> >>
> >> >> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc
TP_6h
> >> >>
> >> >>
> >> >> Please let me know how this goes for you.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway
<johnhg at ucar.edu>
> >> >> wrote:
> >> >>
> >> >> > Andrew,
> >> >> >
> >> >> > OK, question for you. Is this ECMWF actually model output
with a
> >> 6522
> >> >> > forecast hour? Or should this be interpreted as an analysis
that's
> >> >> valid
> >> >> > at 01/28/2006 at 00Z?
> >> >> >
> >> >> > That'll determine how best to encode the timing information.
> >> >> >
> >> >> > Thanks,
> >> >> > John
> >> >> >
> >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA Affiliate
via RT
> <
> >> >> > met_help at ucar.edu> wrote:
> >> >> >
> >> >> >>
> >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >> >> >>
> >> >> >> John,
> >> >> >> That is great. Yes let's go the python route, that seems a
lot
> more
> >> >> >> straightforward. Please let me know how I may proceed, and
thank
> >> you in
> >> >> >> advance for your help on this!!
> >> >> >>
> >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT <
> >> >> >> met_help at ucar.edu>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Hello Andrew,
> >> >> >> >
> >> >> >> > I see you have a question about reading NetCDF data into
MET.
> >> >> Judging
> >> >> >> by
> >> >> >> > the path you sent, I figured that this data lives on
theia. So
> I
> >> >> >> logged on
> >> >> >> > and took a look.
> >> >> >> >
> >> >> >> > Looks like this ECMWF file doesn't exactly fall into any
of the
> >> >> NetCDF
> >> >> >> > categories that MET currently supports. The closest is
> >> CF-compliant
> >> >> >> > NetCDF. So let's tell MET to interpret it as such using
the
> >> >> "file_type
> >> >> >> =
> >> >> >> > NETCDF_NCCF;" option.
> >> >> >> >
> >> >> >> > When I'm trying to figure out if MET can read gridded
data, I
> >> usually
> >> >> >> use
> >> >> >> > the plot_data_plane tool.
> >> >> >> >
> >> >> >> > Here's the commands I ran on theia:
> >> >> >> >
> >> >> >> > module use /contrib/modulefiles
> >> >> >> >
> >> >> >> > module load intel met/8.0
> >> >> >> >
> >> >> >> > plot_data_plane
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >> >> >> >
> >> >> >> > 'name=="TP_6h"; level="(*,*)"; file_type=NETCDF_NCCF;'
> >> >> >> >
> >> >> >> >
> >> >> >> > This command ran and produced the attached plot, but it
did also
> >> >> >> produce a
> >> >> >> > warning about the valid time:
> >> >> >> >
> >> >> >> > WARNING: NcCfFile::open() -> could not determine the
valid time,
> >> >> using
> >> >> >> 0.
> >> >> >> >
> >> >> >> >
> >> >> >> > So MET isn't getting the timing info correct. What can
we do
> >> about
> >> >> >> this?
> >> >> >> > One option would be reformatting these NetCDF files into
a
> format
> >> >> that
> >> >> >> MET
> >> >> >> > can actually read... either CF-compliant NetCDF or the
MET
> NetCDF
> >> >> >> format.
> >> >> >> > However, that's pretty cumbersome. I would instead
recommend
> >> that we
> >> >> >> take
> >> >> >> > advantage of a new feature in met-8.0 (released last
Friday). We
> >> can
> >> >> >> write
> >> >> >> > a Python script to read data from these files, define the
> >> metadata,
> >> >> and
> >> >> >> > then pass that to MET via memory.
> >> >> >> >
> >> >> >> >
> >> >> >> > If the timing information doesn't matter for some reason,
then
> you
> >> >> could
> >> >> >> > just use the "file_type = NETCDF_NCCF;" option. But if
it does
> >> >> matter,
> >> >> >> and
> >> >> >> > presumably it does, then the Python route should work.
> >> >> >> >
> >> >> >> >
> >> >> >> > Let me know how you'd like to proceed.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > John
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
Affiliate via
> >> RT <
> >> >> >> > met_help at ucar.edu> wrote:
> >> >> >> >
> >> >> >> > >
> >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted upon.
> >> >> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
> >> >> >> > > Queue: met_help
> >> >> >> > > Subject: Question about MET pcp_combine
> >> >> >> > > Owner: Nobody
> >> >> >> > > Requestors: andrew.kren at noaa.gov
> >> >> >> > > Status: new
> >> >> >> > > Ticket <URL:
> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >> >> >
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > Dear MET help,
> >> >> >> > >
> >> >> >> > > I have a question on the MET pcp_combine add tool. I
have
> model
> >> >> data
> >> >> >> that
> >> >> >> > > is considered my truth from an ECMWF Nature run (long-
run
> >> >> simulation
> >> >> >> used
> >> >> >> > > to represent the true atmosphere). I want to verify how
well
> my
> >> >> >> forecast
> >> >> >> > > output, in this case from the GFS model, is compared to
the
> >> Nature
> >> >> >> run.
> >> >> >> > The
> >> >> >> > > ECMWF nature run data is 6-hourly data, which includes
6-hr
> >> >> >> precipitation
> >> >> >> > > and other fields in its netcdf files. my GFS data is
grib
> >> format. I
> >> >> >> > already
> >> >> >> > > can convert the GFS data to 24-hourly accumulations
using my
> >> prior
> >> >> >> > scripts
> >> >> >> > > and with pcp_combine. I guess my issue is in the
ability to
> read
> >> >> the
> >> >> >> > Nature
> >> >> >> > > run netcdf files. I try the pcp_combine tool and I get
this
> >> error
> >> >> when
> >> >> >> > > trying to create a 24-hr accumulation.
> >> >> >> > >
> >> >> >> > > (0) Current cycle: 2006012800
> >> >> >> > >
> >> >> >> > > DEBUG 2: Performing addition command for 4 files.
> >> >> >> > >
> >> >> >> > > DEBUG 1: Reading input file:
> >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> >> >> > >
> >> >> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file()
->
> created
> >> >> new
> >> >> >> > > Met2dDataFile object of type "FileType_None".
> >> >> >> > >
> >> >> >> > > ERROR :
> >> >> >> > >
> >> >> >> > > ERROR : get_field() -> can't open data file
> >> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> >> >> >> > >
> >> >> >> > > ERROR :
> >> >> >> > >
> >> >> >> > > I assume MET is having issues reading the variable in
the
> file,
> >> >> which
> >> >> >> is
> >> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is there
a
> >> >> workaround
> >> >> >> so
> >> >> >> > > that it can successfully process this?
> >> >> >> > >
> >> >> >> > > Thank you kindly,
> >> >> >> > >
> >> >> >> > > --
> >> >> >> > > Andrew Kren
> >> >> >> > > Assistant Scientist
> >> >> >> > > University of Miami CIMAS - NOAA/AOML
> >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> >> >> >> > > Boulder, Colorado, USA
> >> >> >> > > Office: 303-497-2715
> >> >> >> > >
> >> >> >> > >
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >> --
> >> >> >> Andrew Kren
> >> >> >> Assistant Scientist
> >> >> >> University of Miami CIMAS - NOAA/AOML
> >> >> >> Duty Station: NCAR Foothills - FL2-3058
> >> >> >> Boulder, Colorado, USA
> >> >> >> Office: 303-497-2715
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > Andrew Kren
> >> > Assistant Scientist
> >> > University of Miami CIMAS - NOAA/AOML
> >> > Duty Station: NCAR Foothills - FL2-3058
> >> > Boulder, Colorado, USA
> >> > Office: 303-497-2715
> >> >
> >>
> >>
> >> --
> >> Andrew Kren
> >> Assistant Scientist
> >> University of Miami CIMAS - NOAA/AOML
> >> Duty Station: NCAR Foothills - FL2-3058
> >> Boulder, Colorado, USA
> >> Office: 303-497-2715
> >>
> >>
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 12:13:08 2018
Andrew,
Hmmm, we're close. Try running this one:
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
create_24_accum_met_nr.sh
The output is listed below. The punchline is... when I run the python
script on the command line everything is fine. When I run it through
plot_data_plane, I get a runtime error about netCDF4.
I can confirm that this works fine on my project machine at NCAR. So
I
suspect it's an environment issue.
John
[John.H.Gotway at tfe05 kren_data_20181003]$ ./create_24_accum_met_nr.sh
Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
NR2006.pl.1by1.6522.2006012806.nc'
Variable Name : 'TP_6h'
Data Shape: (181, 360)
Data Type: dtype('float64')
Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000', 'valid':
'20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
'units': 'mm', 'accum': '06'}
DEBUG 1: Opening data file: PYTHON_NUMPY
ERROR :
ERROR : python_dataplane() -> an error occurred importing module
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
ERROR :
Traceback (most recent call last):
File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
line 3, in <module>
from netCDF4 import Dataset
ImportError: No module named netCDF4
On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> John,
> I ran this script,
>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
and
> it gave me this, presumably asking for more information to run the
met
> command.
>
> The following have been reloaded with a version change:
>
> 1) intel/15.0.0 => intel/14.0.2
>
>
> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> NR2006.pl.1by1.6522.2006012806.nc'
>
> Variable Name : 'TP_6h'
>
> Data Shape: (181, 360)
>
> Data Type: dtype('float64')
>
> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
> 'units': 'mm', 'accum': '06'}
>
>
> *** Model Evaluation Tools (METV8.0) ***
>
>
> Usage: plot_data_plane
>
> input_filename
>
> output_filename
>
> field_string
>
> [-color_table color_table_name]
>
> [-plot_range min max]
>
> [-title title_string]
>
> [-log file]
>
> [-v level]
>
>
> where "input_filename" is the name of a gridded data file to be
plotted
> (required).
>
> "output_filename" is the name of the output PostScript file to be
written
> (required).
>
> "field_string" defines the data to be plotted from the input file
> (required).
>
> "-color_table color_table_name" overrides the default color table
> ("colortables/met_default.ctable") (optional).
>
> "-plot_range min max" defines the range of the data to be plotted
> (optional).
>
> "-title title_string" specifies the plot title string (optional).
>
> "-log file" outputs log messages to the specified file (optional).
>
> "-v level" overrides the default level of logging (2) (optional).
>
> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Andrew,
> >
> > Sorry, just noticed a typo after I hit send. I had a double-
equals (==)
> > that should be a single equal (=):
> >
> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc
> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >
> > 'name="python
> >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > NR2006.pl.1by1.6522.2006012806.nc
> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >
> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Great, so running the python script stand-alone read the data
well.
> > >
> > > The next step is telling MET to call this script. Here's that
simple
> > > example of running plot_data_plane:
> > >
> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > >
> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > NR2006.pl.1by1.6522.2006012806.nc
> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > >
> > > 'name=="python
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > NR2006.pl.1by1.6522.2006012806.nc
> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > >
> > >
> > > Once this works, look at the output to confirm that MET is
putting your
> > > data on the right spot on the earth.
> > >
> > >
> > > Presumably it does... so now you're ready to start using this to
run
> > > pcp_combine or Grid-Stat or MODE.
> > >
> > >
> > > Make sense?
> > >
> > >
> > > John
> > >
> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > >>
> > >> John,
> > >> I got this output when I ran your python script in a shell
script:
> > >>
> > >> The following have been reloaded with a version change:
> > >>
> > >> 1) intel/15.0.0 => intel/14.0.2
> > >>
> > >>
> > >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > >>
> > >> Variable Name : 'TP_6h'
> > >>
> > >> Data Shape: (181, 360)
> > >>
> > >> Data Type: dtype('float64')
> > >>
> > >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
> 'type':
> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> > 'Surface',
> > >> 'units': 'mm', 'accum': '06'}
> > >>
> > >>
> > >> Looks like it worked.
> > >>
> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
> > >> andrew.kren at noaa.gov> wrote:
> > >>
> > >> > John,
> > >> > I believe the 6522 and other numbers (which increase with
each date)
> > >> were
> > >> > used as place holders for the filename. The actual dates can
be read
> > as
> > >> an
> > >> > analysis that's valid at the time stamp, such as 01/28/2006
at 06Z?
> > Thus
> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond
to the
> > >> > analysis valid at 2006012806, does that help? I'll try the
script.
> > >> >
> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >> Andrew,
> > >> >>
> > >> >>
> > >> >> OK, please try running this command on theia. I've attached
the
> > >> resulting
> > >> >> image:
> > >> >>
> > >> >>
> > >> >> module use /contrib/modulefiles
> > >> >>
> > >> >> module load intel met/8.0
> > >> >>
> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > >> >>
> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > >> >>
> > >> >> 'name=="python
> > >> >>
> > >> >>
> > >>
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > >> >>
> > >> >>
> > >> >> This python script (also attached) reads your data and hands
it off
> > to
> > >> >> MET.
> > >> >>
> > >> >>
> > >> >> Here's the steps I followed to do this:
> > >> >>
> > >> >>
> > >> >> (1) Copy over a sample python script from MET:
> > >> >>
> > >> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > >> >> read_ECMWF_NC_2D.py
> > >> >>
> > >> >>
> > >> >> (2) Edit it to read the input netcdf file (argument 1) and
the
> > variable
> > >> >> name (argument 2).
> > >> >>
> > >> >>
> > >> >> (3) Add code to parse out the timing info and define it in
the
> "attr"
> > >> >> dictionary.
> > >> >>
> > >> >> NOTE that I assumed that this is an analysis... so I just
set the
> > init
> > >> >> time
> > >> >> to equal the valid time. Feel free to change that however
you
> need.
> > >> >>
> > >> >>
> > >> >> (4) Add the grid definition info. And I figured this out by
> running
> > >> >> pcp_combine (see below) and then looking at the NetCDF
global
> > >> attributes.
> > >> >>
> > >> >>
> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
'name="TP_6h";
> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> > >> >>
> > >> >>
> > >> >> (5) Along the way, I ran this command to make sure my python
script
> > >> >> actually runs on the command line:
> > >> >>
> > >> >> python read_ECMWF_NC_2D.py NR2006.pl.1by1.6522.2006012806.nc
TP_6h
> > >> >>
> > >> >>
> > >> >> Please let me know how this goes for you.
> > >> >>
> > >> >>
> > >> >> Thanks,
> > >> >> John
> > >> >>
> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
> johnhg at ucar.edu>
> > >> >> wrote:
> > >> >>
> > >> >> > Andrew,
> > >> >> >
> > >> >> > OK, question for you. Is this ECMWF actually model output
with a
> > >> 6522
> > >> >> > forecast hour? Or should this be interpreted as an
analysis
> that's
> > >> >> valid
> > >> >> > at 01/28/2006 at 00Z?
> > >> >> >
> > >> >> > That'll determine how best to encode the timing
information.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > John
> > >> >> >
> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
Affiliate via
> RT
> > <
> > >> >> > met_help at ucar.edu> wrote:
> > >> >> >
> > >> >> >>
> > >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > >> >> >>
> > >> >> >> John,
> > >> >> >> That is great. Yes let's go the python route, that seems
a lot
> > more
> > >> >> >> straightforward. Please let me know how I may proceed,
and thank
> > >> you in
> > >> >> >> advance for your help on this!!
> > >> >> >>
> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT
<
> > >> >> >> met_help at ucar.edu>
> > >> >> >> wrote:
> > >> >> >>
> > >> >> >> > Hello Andrew,
> > >> >> >> >
> > >> >> >> > I see you have a question about reading NetCDF data
into MET.
> > >> >> Judging
> > >> >> >> by
> > >> >> >> > the path you sent, I figured that this data lives on
theia.
> So
> > I
> > >> >> >> logged on
> > >> >> >> > and took a look.
> > >> >> >> >
> > >> >> >> > Looks like this ECMWF file doesn't exactly fall into
any of
> the
> > >> >> NetCDF
> > >> >> >> > categories that MET currently supports. The closest is
> > >> CF-compliant
> > >> >> >> > NetCDF. So let's tell MET to interpret it as such
using the
> > >> >> "file_type
> > >> >> >> =
> > >> >> >> > NETCDF_NCCF;" option.
> > >> >> >> >
> > >> >> >> > When I'm trying to figure out if MET can read gridded
data, I
> > >> usually
> > >> >> >> use
> > >> >> >> > the plot_data_plane tool.
> > >> >> >> >
> > >> >> >> > Here's the commands I ran on theia:
> > >> >> >> >
> > >> >> >> > module use /contrib/modulefiles
> > >> >> >> >
> > >> >> >> > module load intel met/8.0
> > >> >> >> >
> > >> >> >> > plot_data_plane
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> > >> >> >> >
> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
file_type=NETCDF_NCCF;'
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > This command ran and produced the attached plot, but it
did
> also
> > >> >> >> produce a
> > >> >> >> > warning about the valid time:
> > >> >> >> >
> > >> >> >> > WARNING: NcCfFile::open() -> could not determine the
valid
> time,
> > >> >> using
> > >> >> >> 0.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > So MET isn't getting the timing info correct. What can
we do
> > >> about
> > >> >> >> this?
> > >> >> >> > One option would be reformatting these NetCDF files
into a
> > format
> > >> >> that
> > >> >> >> MET
> > >> >> >> > can actually read... either CF-compliant NetCDF or the
MET
> > NetCDF
> > >> >> >> format.
> > >> >> >> > However, that's pretty cumbersome. I would instead
recommend
> > >> that we
> > >> >> >> take
> > >> >> >> > advantage of a new feature in met-8.0 (released last
Friday).
> We
> > >> can
> > >> >> >> write
> > >> >> >> > a Python script to read data from these files, define
the
> > >> metadata,
> > >> >> and
> > >> >> >> > then pass that to MET via memory.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > If the timing information doesn't matter for some
reason, then
> > you
> > >> >> could
> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option. But if
it
> does
> > >> >> matter,
> > >> >> >> and
> > >> >> >> > presumably it does, then the Python route should work.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > Let me know how you'd like to proceed.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > Thanks,
> > >> >> >> > John
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
Affiliate
> via
> > >> RT <
> > >> >> >> > met_help at ucar.edu> wrote:
> > >> >> >> >
> > >> >> >> > >
> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted
upon.
> > >> >> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
> > >> >> >> > > Queue: met_help
> > >> >> >> > > Subject: Question about MET pcp_combine
> > >> >> >> > > Owner: Nobody
> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > >> >> >> > > Status: new
> > >> >> >> > > Ticket <URL:
> > >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > >> >> >> >
> > >> >> >> > >
> > >> >> >> > >
> > >> >> >> > > Dear MET help,
> > >> >> >> > >
> > >> >> >> > > I have a question on the MET pcp_combine add tool. I
have
> > model
> > >> >> data
> > >> >> >> that
> > >> >> >> > > is considered my truth from an ECMWF Nature run
(long-run
> > >> >> simulation
> > >> >> >> used
> > >> >> >> > > to represent the true atmosphere). I want to verify
how well
> > my
> > >> >> >> forecast
> > >> >> >> > > output, in this case from the GFS model, is compared
to the
> > >> Nature
> > >> >> >> run.
> > >> >> >> > The
> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
includes 6-hr
> > >> >> >> precipitation
> > >> >> >> > > and other fields in its netcdf files. my GFS data is
grib
> > >> format. I
> > >> >> >> > already
> > >> >> >> > > can convert the GFS data to 24-hourly accumulations
using my
> > >> prior
> > >> >> >> > scripts
> > >> >> >> > > and with pcp_combine. I guess my issue is in the
ability to
> > read
> > >> >> the
> > >> >> >> > Nature
> > >> >> >> > > run netcdf files. I try the pcp_combine tool and I
get this
> > >> error
> > >> >> when
> > >> >> >> > > trying to create a 24-hr accumulation.
> > >> >> >> > >
> > >> >> >> > > (0) Current cycle: 2006012800
> > >> >> >> > >
> > >> >> >> > > DEBUG 2: Performing addition command for 4 files.
> > >> >> >> > >
> > >> >> >> > > DEBUG 1: Reading input file:
> > >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> >> > >
> > >> >> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file()
->
> > created
> > >> >> new
> > >> >> >> > > Met2dDataFile object of type "FileType_None".
> > >> >> >> > >
> > >> >> >> > > ERROR :
> > >> >> >> > >
> > >> >> >> > > ERROR : get_field() -> can't open data file
> > >> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > >> >> >> > >
> > >> >> >> > > ERROR :
> > >> >> >> > >
> > >> >> >> > > I assume MET is having issues reading the variable in
the
> > file,
> > >> >> which
> > >> >> >> is
> > >> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is
there a
> > >> >> workaround
> > >> >> >> so
> > >> >> >> > > that it can successfully process this?
> > >> >> >> > >
> > >> >> >> > > Thank you kindly,
> > >> >> >> > >
> > >> >> >> > > --
> > >> >> >> > > Andrew Kren
> > >> >> >> > > Assistant Scientist
> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> > >> >> >> > > Boulder, Colorado, USA
> > >> >> >> > > Office: 303-497-2715
> > >> >> >> > >
> > >> >> >> > >
> > >> >> >> >
> > >> >> >> >
> > >> >> >>
> > >> >> >> --
> > >> >> >> Andrew Kren
> > >> >> >> Assistant Scientist
> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > >> >> >> Boulder, Colorado, USA
> > >> >> >> Office: 303-497-2715
> > >> >> >>
> > >> >> >>
> > >> >>
> > >> >>
> > >> >
> > >> > --
> > >> > Andrew Kren
> > >> > Assistant Scientist
> > >> > University of Miami CIMAS - NOAA/AOML
> > >> > Duty Station: NCAR Foothills - FL2-3058
> > >> > Boulder, Colorado, USA
> > >> > Office: 303-497-2715
> > >> >
> > >>
> > >>
> > >> --
> > >> Andrew Kren
> > >> Assistant Scientist
> > >> University of Miami CIMAS - NOAA/AOML
> > >> Duty Station: NCAR Foothills - FL2-3058
> > >> Boulder, Colorado, USA
> > >> Office: 303-497-2715
> > >>
> > >>
> >
> >
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Thu Oct 04 12:21:57 2018
Andrew,
OK, this script runs successfully on theia with python from my path:
> which python
/contrib/anaconda/anaconda2/4.4.0/bin/python
But when I run the script using /bin/python, I get the same error as
from
MET:
Traceback (most recent call last):
File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
line 3, in <module>
from netCDF4 import Dataset
ImportError: No module named netCDF4
So it would appear that MET is not calling the version of python that
lives
in my path. I need to check with another developer to figure out if
there's a way to tell MET what version of python to run.
Thanks,
John
On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Andrew,
>
> Hmmm, we're close. Try running this one:
>
> /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> create_24_accum_met_nr.sh
>
>
> The output is listed below. The punchline is... when I run the
python
> script on the command line everything is fine. When I run it
through
> plot_data_plane, I get a runtime error about netCDF4.
>
>
> I can confirm that this works fine on my project machine at NCAR.
So I
> suspect it's an environment issue.
>
>
> John
>
>
> [John.H.Gotway at tfe05 kren_data_20181003]$
./create_24_accum_met_nr.sh
>
> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> NR2006.pl.1by1.6522.2006012806.nc'
>
> Variable Name : 'TP_6h'
>
> Data Shape: (181, 360)
>
> Data Type: dtype('float64')
>
> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
> 'units': 'mm', 'accum': '06'}
>
> DEBUG 1: Opening data file: PYTHON_NUMPY
>
> ERROR :
>
> ERROR : python_dataplane() -> an error occurred importing module
> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
>
> ERROR :
>
> Traceback (most recent call last):
>
> File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> line 3, in <module>
>
> from netCDF4 import Dataset
>
> ImportError: No module named netCDF4
>
>
> On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>>
>> John,
>> I ran this script,
>>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
and
>> it gave me this, presumably asking for more information to run the
met
>> command.
>>
>> The following have been reloaded with a version change:
>>
>> 1) intel/15.0.0 => intel/14.0.2
>>
>>
>> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> NR2006.pl.1by1.6522.2006012806.nc'
>>
>> Variable Name : 'TP_6h'
>>
>> Data Shape: (181, 360)
>>
>> Data Type: dtype('float64')
>>
>> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
>> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
>> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
>> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
>> 'units': 'mm', 'accum': '06'}
>>
>>
>> *** Model Evaluation Tools (METV8.0) ***
>>
>>
>> Usage: plot_data_plane
>>
>> input_filename
>>
>> output_filename
>>
>> field_string
>>
>> [-color_table color_table_name]
>>
>> [-plot_range min max]
>>
>> [-title title_string]
>>
>> [-log file]
>>
>> [-v level]
>>
>>
>> where "input_filename" is the name of a gridded data file to be
plotted
>> (required).
>>
>> "output_filename" is the name of the output PostScript file to be
written
>> (required).
>>
>> "field_string" defines the data to be plotted from the input file
>> (required).
>>
>> "-color_table color_table_name" overrides the default color table
>> ("colortables/met_default.ctable") (optional).
>>
>> "-plot_range min max" defines the range of the data to be plotted
>> (optional).
>>
>> "-title title_string" specifies the plot title string (optional).
>>
>> "-log file" outputs log messages to the specified file (optional).
>>
>> "-v level" overrides the default level of logging (2) (optional).
>>
>> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Andrew,
>> >
>> > Sorry, just noticed a typo after I hit send. I had a double-
equals (==)
>> > that should be a single equal (=):
>> >
>> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> >
>> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > NR2006.pl.1by1.6522.2006012806.nc
>> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> >
>> > 'name="python
>> >
>> >
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > NR2006.pl.1by1.6522.2006012806.nc
>> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>> >
>> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway
<johnhg at ucar.edu>
>> > wrote:
>> >
>> > > Great, so running the python script stand-alone read the data
well.
>> > >
>> > > The next step is telling MET to call this script. Here's that
simple
>> > > example of running plot_data_plane:
>> > >
>> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> > >
>> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > > NR2006.pl.1by1.6522.2006012806.nc
>> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> > >
>> > > 'name=="python
>> > >
>> >
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > > NR2006.pl.1by1.6522.2006012806.nc
>> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>> > >
>> > >
>> > > Once this works, look at the output to confirm that MET is
putting
>> your
>> > > data on the right spot on the earth.
>> > >
>> > >
>> > > Presumably it does... so now you're ready to start using this
to run
>> > > pcp_combine or Grid-Stat or MODE.
>> > >
>> > >
>> > > Make sense?
>> > >
>> > >
>> > > John
>> > >
>> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate
via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >>
>> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
>> > >>
>> > >> John,
>> > >> I got this output when I ran your python script in a shell
script:
>> > >>
>> > >> The following have been reloaded with a version change:
>> > >>
>> > >> 1) intel/15.0.0 => intel/14.0.2
>> > >>
>> > >>
>> > >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> > >> NR2006.pl.1by1.6522.2006012806.nc'
>> > >>
>> > >> Variable Name : 'TP_6h'
>> > >>
>> > >> Data Shape: (181, 360)
>> > >>
>> > >> Data Type: dtype('float64')
>> > >>
>> > >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
>> 'valid':
>> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
>> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
>> 'type':
>> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
>> > 'Surface',
>> > >> 'units': 'mm', 'accum': '06'}
>> > >>
>> > >>
>> > >> Looks like it worked.
>> > >>
>> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate <
>> > >> andrew.kren at noaa.gov> wrote:
>> > >>
>> > >> > John,
>> > >> > I believe the 6522 and other numbers (which increase with
each
>> date)
>> > >> were
>> > >> > used as place holders for the filename. The actual dates can
be
>> read
>> > as
>> > >> an
>> > >> > analysis that's valid at the time stamp, such as 01/28/2006
at 06Z?
>> > Thus
>> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
>> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would correspond
to
>> the
>> > >> > analysis valid at 2006012806, does that help? I'll try the
script.
>> > >> >
>> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT <
>> > >> > met_help at ucar.edu> wrote:
>> > >> >
>> > >> >> Andrew,
>> > >> >>
>> > >> >>
>> > >> >> OK, please try running this command on theia. I've
attached the
>> > >> resulting
>> > >> >> image:
>> > >> >>
>> > >> >>
>> > >> >> module use /contrib/modulefiles
>> > >> >>
>> > >> >> module load intel met/8.0
>> > >> >>
>> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> > >> >>
>> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > >> >> NR2006.pl.1by1.6522.2006012806.nc
>> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> > >> >>
>> > >> >> 'name=="python
>> > >> >>
>> > >> >>
>> > >>
>> >
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
>> > >> >>
>> > >> >>
>> > >> >> This python script (also attached) reads your data and
hands it
>> off
>> > to
>> > >> >> MET.
>> > >> >>
>> > >> >>
>> > >> >> Here's the steps I followed to do this:
>> > >> >>
>> > >> >>
>> > >> >> (1) Copy over a sample python script from MET:
>> > >> >>
>> > >> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
>> > >> >> read_ECMWF_NC_2D.py
>> > >> >>
>> > >> >>
>> > >> >> (2) Edit it to read the input netcdf file (argument 1) and
the
>> > variable
>> > >> >> name (argument 2).
>> > >> >>
>> > >> >>
>> > >> >> (3) Add code to parse out the timing info and define it in
the
>> "attr"
>> > >> >> dictionary.
>> > >> >>
>> > >> >> NOTE that I assumed that this is an analysis... so I just
set the
>> > init
>> > >> >> time
>> > >> >> to equal the valid time. Feel free to change that however
you
>> need.
>> > >> >>
>> > >> >>
>> > >> >> (4) Add the grid definition info. And I figured this out
by
>> running
>> > >> >> pcp_combine (see below) and then looking at the NetCDF
global
>> > >> attributes.
>> > >> >>
>> > >> >>
>> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
'name="TP_6h";
>> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
>> > >> >>
>> > >> >>
>> > >> >> (5) Along the way, I ran this command to make sure my
python
>> script
>> > >> >> actually runs on the command line:
>> > >> >>
>> > >> >> python read_ECMWF_NC_2D.py
NR2006.pl.1by1.6522.2006012806.nc
>> TP_6h
>> > >> >>
>> > >> >>
>> > >> >> Please let me know how this goes for you.
>> > >> >>
>> > >> >>
>> > >> >> Thanks,
>> > >> >> John
>> > >> >>
>> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
>> johnhg at ucar.edu>
>> > >> >> wrote:
>> > >> >>
>> > >> >> > Andrew,
>> > >> >> >
>> > >> >> > OK, question for you. Is this ECMWF actually model
output with
>> a
>> > >> 6522
>> > >> >> > forecast hour? Or should this be interpreted as an
analysis
>> that's
>> > >> >> valid
>> > >> >> > at 01/28/2006 at 00Z?
>> > >> >> >
>> > >> >> > That'll determine how best to encode the timing
information.
>> > >> >> >
>> > >> >> > Thanks,
>> > >> >> > John
>> > >> >> >
>> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
Affiliate via
>> RT
>> > <
>> > >> >> > met_help at ucar.edu> wrote:
>> > >> >> >
>> > >> >> >>
>> > >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >
>> > >> >> >>
>> > >> >> >> John,
>> > >> >> >> That is great. Yes let's go the python route, that seems
a lot
>> > more
>> > >> >> >> straightforward. Please let me know how I may proceed,
and
>> thank
>> > >> you in
>> > >> >> >> advance for your help on this!!
>> > >> >> >>
>> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via RT
<
>> > >> >> >> met_help at ucar.edu>
>> > >> >> >> wrote:
>> > >> >> >>
>> > >> >> >> > Hello Andrew,
>> > >> >> >> >
>> > >> >> >> > I see you have a question about reading NetCDF data
into MET.
>> > >> >> Judging
>> > >> >> >> by
>> > >> >> >> > the path you sent, I figured that this data lives on
theia.
>> So
>> > I
>> > >> >> >> logged on
>> > >> >> >> > and took a look.
>> > >> >> >> >
>> > >> >> >> > Looks like this ECMWF file doesn't exactly fall into
any of
>> the
>> > >> >> NetCDF
>> > >> >> >> > categories that MET currently supports. The closest
is
>> > >> CF-compliant
>> > >> >> >> > NetCDF. So let's tell MET to interpret it as such
using the
>> > >> >> "file_type
>> > >> >> >> =
>> > >> >> >> > NETCDF_NCCF;" option.
>> > >> >> >> >
>> > >> >> >> > When I'm trying to figure out if MET can read gridded
data, I
>> > >> usually
>> > >> >> >> use
>> > >> >> >> > the plot_data_plane tool.
>> > >> >> >> >
>> > >> >> >> > Here's the commands I ran on theia:
>> > >> >> >> >
>> > >> >> >> > module use /contrib/modulefiles
>> > >> >> >> >
>> > >> >> >> > module load intel met/8.0
>> > >> >> >> >
>> > >> >> >> > plot_data_plane
>> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>> > >> >> >> >
>> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
file_type=NETCDF_NCCF;'
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > This command ran and produced the attached plot, but
it did
>> also
>> > >> >> >> produce a
>> > >> >> >> > warning about the valid time:
>> > >> >> >> >
>> > >> >> >> > WARNING: NcCfFile::open() -> could not determine the
valid
>> time,
>> > >> >> using
>> > >> >> >> 0.
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > So MET isn't getting the timing info correct. What
can we do
>> > >> about
>> > >> >> >> this?
>> > >> >> >> > One option would be reformatting these NetCDF files
into a
>> > format
>> > >> >> that
>> > >> >> >> MET
>> > >> >> >> > can actually read... either CF-compliant NetCDF or the
MET
>> > NetCDF
>> > >> >> >> format.
>> > >> >> >> > However, that's pretty cumbersome. I would instead
recommend
>> > >> that we
>> > >> >> >> take
>> > >> >> >> > advantage of a new feature in met-8.0 (released last
>> Friday). We
>> > >> can
>> > >> >> >> write
>> > >> >> >> > a Python script to read data from these files, define
the
>> > >> metadata,
>> > >> >> and
>> > >> >> >> > then pass that to MET via memory.
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > If the timing information doesn't matter for some
reason,
>> then
>> > you
>> > >> >> could
>> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option. But
if it
>> does
>> > >> >> matter,
>> > >> >> >> and
>> > >> >> >> > presumably it does, then the Python route should work.
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > Let me know how you'd like to proceed.
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > Thanks,
>> > >> >> >> > John
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> >> >> > met_help at ucar.edu> wrote:
>> > >> >> >> >
>> > >> >> >> > >
>> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted
upon.
>> > >> >> >> > > Transaction: Ticket created by andrew.kren at noaa.gov
>> > >> >> >> > > Queue: met_help
>> > >> >> >> > > Subject: Question about MET pcp_combine
>> > >> >> >> > > Owner: Nobody
>> > >> >> >> > > Requestors: andrew.kren at noaa.gov
>> > >> >> >> > > Status: new
>> > >> >> >> > > Ticket <URL:
>> > >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> > >> >> >> >
>> > >> >> >> > >
>> > >> >> >> > >
>> > >> >> >> > > Dear MET help,
>> > >> >> >> > >
>> > >> >> >> > > I have a question on the MET pcp_combine add tool. I
have
>> > model
>> > >> >> data
>> > >> >> >> that
>> > >> >> >> > > is considered my truth from an ECMWF Nature run
(long-run
>> > >> >> simulation
>> > >> >> >> used
>> > >> >> >> > > to represent the true atmosphere). I want to verify
how
>> well
>> > my
>> > >> >> >> forecast
>> > >> >> >> > > output, in this case from the GFS model, is compared
to the
>> > >> Nature
>> > >> >> >> run.
>> > >> >> >> > The
>> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
includes 6-hr
>> > >> >> >> precipitation
>> > >> >> >> > > and other fields in its netcdf files. my GFS data is
grib
>> > >> format. I
>> > >> >> >> > already
>> > >> >> >> > > can convert the GFS data to 24-hourly accumulations
using
>> my
>> > >> prior
>> > >> >> >> > scripts
>> > >> >> >> > > and with pcp_combine. I guess my issue is in the
ability to
>> > read
>> > >> >> the
>> > >> >> >> > Nature
>> > >> >> >> > > run netcdf files. I try the pcp_combine tool and I
get this
>> > >> error
>> > >> >> when
>> > >> >> >> > > trying to create a 24-hr accumulation.
>> > >> >> >> > >
>> > >> >> >> > > (0) Current cycle: 2006012800
>> > >> >> >> > >
>> > >> >> >> > > DEBUG 2: Performing addition command for 4 files.
>> > >> >> >> > >
>> > >> >> >> > > DEBUG 1: Reading input file:
>> > >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> > >> >> >> > >
>> > >> >> >> > > DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file() ->
>> > created
>> > >> >> new
>> > >> >> >> > > Met2dDataFile object of type "FileType_None".
>> > >> >> >> > >
>> > >> >> >> > > ERROR :
>> > >> >> >> > >
>> > >> >> >> > > ERROR : get_field() -> can't open data file
>> > >> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
>> > >> >> >> > >
>> > >> >> >> > > ERROR :
>> > >> >> >> > >
>> > >> >> >> > > I assume MET is having issues reading the variable
in the
>> > file,
>> > >> >> which
>> > >> >> >> is
>> > >> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is
there a
>> > >> >> workaround
>> > >> >> >> so
>> > >> >> >> > > that it can successfully process this?
>> > >> >> >> > >
>> > >> >> >> > > Thank you kindly,
>> > >> >> >> > >
>> > >> >> >> > > --
>> > >> >> >> > > Andrew Kren
>> > >> >> >> > > Assistant Scientist
>> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
>> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
>> > >> >> >> > > Boulder, Colorado, USA
>> > >> >> >> > > Office: 303-497-2715
>> > >> >> >> > >
>> > >> >> >> > >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >>
>> > >> >> >> --
>> > >> >> >> Andrew Kren
>> > >> >> >> Assistant Scientist
>> > >> >> >> University of Miami CIMAS - NOAA/AOML
>> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
>> > >> >> >> Boulder, Colorado, USA
>> > >> >> >> Office: 303-497-2715
>> > >> >> >>
>> > >> >> >>
>> > >> >>
>> > >> >>
>> > >> >
>> > >> > --
>> > >> > Andrew Kren
>> > >> > Assistant Scientist
>> > >> > University of Miami CIMAS - NOAA/AOML
>> > >> > Duty Station: NCAR Foothills - FL2-3058
>> > >> > Boulder, Colorado, USA
>> > >> > Office: 303-497-2715
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> Andrew Kren
>> > >> Assistant Scientist
>> > >> University of Miami CIMAS - NOAA/AOML
>> > >> Duty Station: NCAR Foothills - FL2-3058
>> > >> Boulder, Colorado, USA
>> > >> Office: 303-497-2715
>> > >>
>> > >>
>> >
>> >
>>
>> --
>> Andrew Kren
>> Assistant Scientist
>> University of Miami CIMAS - NOAA/AOML
>> Duty Station: NCAR Foothills - FL2-3058
>> Boulder, Colorado, USA
>> Office: 303-497-2715
>>
>>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 13:06:04 2018
Thanks John.
On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
> OK, this script runs successfully on theia with python from my path:
>
> > which python
>
> /contrib/anaconda/anaconda2/4.4.0/bin/python
>
>
> But when I run the script using /bin/python, I get the same error as
from
> MET:
>
>
> Traceback (most recent call last):
>
> File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> line 3, in <module>
>
> from netCDF4 import Dataset
>
> ImportError: No module named netCDF4
>
> So it would appear that MET is not calling the version of python
that lives
> in my path. I need to check with another developer to figure out if
> there's a way to tell MET what version of python to run.
>
> Thanks,
> John
>
> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Andrew,
> >
> > Hmmm, we're close. Try running this one:
> >
> > /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > create_24_accum_met_nr.sh
> >
> >
> > The output is listed below. The punchline is... when I run the
python
> > script on the command line everything is fine. When I run it
through
> > plot_data_plane, I get a runtime error about netCDF4.
> >
> >
> > I can confirm that this works fine on my project machine at NCAR.
So I
> > suspect it's an environment issue.
> >
> >
> > John
> >
> >
> > [John.H.Gotway at tfe05 kren_data_20181003]$
./create_24_accum_met_nr.sh
> >
> > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > NR2006.pl.1by1.6522.2006012806.nc'
> >
> > Variable Name : 'TP_6h'
> >
> > Data Shape: (181, 360)
> >
> > Data Type: dtype('float64')
> >
> > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> 'Surface',
> > 'units': 'mm', 'accum': '06'}
> >
> > DEBUG 1: Opening data file: PYTHON_NUMPY
> >
> > ERROR :
> >
> > ERROR : python_dataplane() -> an error occurred importing module
> > "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> >
> > ERROR :
> >
> > Traceback (most recent call last):
> >
> > File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > line 3, in <module>
> >
> > from netCDF4 import Dataset
> >
> > ImportError: No module named netCDF4
> >
> >
> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >>
> >> John,
> >> I ran this script,
> >>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> and
> >> it gave me this, presumably asking for more information to run
the met
> >> command.
> >>
> >> The following have been reloaded with a version change:
> >>
> >> 1) intel/15.0.0 => intel/14.0.2
> >>
> >>
> >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> NR2006.pl.1by1.6522.2006012806.nc'
> >>
> >> Variable Name : 'TP_6h'
> >>
> >> Data Shape: (181, 360)
> >>
> >> Data Type: dtype('float64')
> >>
> >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> 'Surface',
> >> 'units': 'mm', 'accum': '06'}
> >>
> >>
> >> *** Model Evaluation Tools (METV8.0) ***
> >>
> >>
> >> Usage: plot_data_plane
> >>
> >> input_filename
> >>
> >> output_filename
> >>
> >> field_string
> >>
> >> [-color_table color_table_name]
> >>
> >> [-plot_range min max]
> >>
> >> [-title title_string]
> >>
> >> [-log file]
> >>
> >> [-v level]
> >>
> >>
> >> where "input_filename" is the name of a gridded data file to be
plotted
> >> (required).
> >>
> >> "output_filename" is the name of the output PostScript file to be
> written
> >> (required).
> >>
> >> "field_string" defines the data to be plotted from the input file
> >> (required).
> >>
> >> "-color_table color_table_name" overrides the default color table
> >> ("colortables/met_default.ctable") (optional).
> >>
> >> "-plot_range min max" defines the range of the data to be plotted
> >> (optional).
> >>
> >> "-title title_string" specifies the plot title string (optional).
> >>
> >> "-log file" outputs log messages to the specified file
(optional).
> >>
> >> "-v level" overrides the default level of logging (2) (optional).
> >>
> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Andrew,
> >> >
> >> > Sorry, just noticed a typo after I hit send. I had a double-
equals
> (==)
> >> > that should be a single equal (=):
> >> >
> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> >
> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > NR2006.pl.1by1.6522.2006012806.nc
> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> >
> >> > 'name="python
> >> >
> >> >
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > NR2006.pl.1by1.6522.2006012806.nc
> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >> >
> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway
<johnhg at ucar.edu>
> >> > wrote:
> >> >
> >> > > Great, so running the python script stand-alone read the data
well.
> >> > >
> >> > > The next step is telling MET to call this script. Here's
that
> simple
> >> > > example of running plot_data_plane:
> >> > >
> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> > >
> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> > >
> >> > > 'name=="python
> >> > >
> >> >
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >> > >
> >> > >
> >> > > Once this works, look at the output to confirm that MET is
putting
> >> your
> >> > > data on the right spot on the earth.
> >> > >
> >> > >
> >> > > Presumably it does... so now you're ready to start using this
to run
> >> > > pcp_combine or Grid-Stat or MODE.
> >> > >
> >> > >
> >> > > Make sense?
> >> > >
> >> > >
> >> > > John
> >> > >
> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate
via RT
> <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >>
> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >> > >>
> >> > >> John,
> >> > >> I got this output when I ran your python script in a shell
script:
> >> > >>
> >> > >> The following have been reloaded with a version change:
> >> > >>
> >> > >> 1) intel/15.0.0 => intel/14.0.2
> >> > >>
> >> > >>
> >> > >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> >> > >>
> >> > >> Variable Name : 'TP_6h'
> >> > >>
> >> > >> Data Shape: (181, 360)
> >> > >>
> >> > >> Data Type: dtype('float64')
> >> > >>
> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> >> 'valid':
> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
> 'name':
> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> >> 'type':
> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> >> > 'Surface',
> >> > >> 'units': 'mm', 'accum': '06'}
> >> > >>
> >> > >>
> >> > >> Looks like it worked.
> >> > >>
> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA Affiliate
<
> >> > >> andrew.kren at noaa.gov> wrote:
> >> > >>
> >> > >> > John,
> >> > >> > I believe the 6522 and other numbers (which increase with
each
> >> date)
> >> > >> were
> >> > >> > used as place holders for the filename. The actual dates
can be
> >> read
> >> > as
> >> > >> an
> >> > >> > analysis that's valid at the time stamp, such as
01/28/2006 at
> 06Z?
> >> > Thus
> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
correspond to
> >> the
> >> > >> > analysis valid at 2006012806, does that help? I'll try the
> script.
> >> > >> >
> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT
<
> >> > >> > met_help at ucar.edu> wrote:
> >> > >> >
> >> > >> >> Andrew,
> >> > >> >>
> >> > >> >>
> >> > >> >> OK, please try running this command on theia. I've
attached the
> >> > >> resulting
> >> > >> >> image:
> >> > >> >>
> >> > >> >>
> >> > >> >> module use /contrib/modulefiles
> >> > >> >>
> >> > >> >> module load intel met/8.0
> >> > >> >>
> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> > >> >>
> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> > >> >>
> >> > >> >> 'name=="python
> >> > >> >>
> >> > >> >>
> >> > >>
> >> >
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> >> > >> >>
> >> > >> >>
> >> > >> >> This python script (also attached) reads your data and
hands it
> >> off
> >> > to
> >> > >> >> MET.
> >> > >> >>
> >> > >> >>
> >> > >> >> Here's the steps I followed to do this:
> >> > >> >>
> >> > >> >>
> >> > >> >> (1) Copy over a sample python script from MET:
> >> > >> >>
> >> > >> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> >> > >> >> read_ECMWF_NC_2D.py
> >> > >> >>
> >> > >> >>
> >> > >> >> (2) Edit it to read the input netcdf file (argument 1)
and the
> >> > variable
> >> > >> >> name (argument 2).
> >> > >> >>
> >> > >> >>
> >> > >> >> (3) Add code to parse out the timing info and define it
in the
> >> "attr"
> >> > >> >> dictionary.
> >> > >> >>
> >> > >> >> NOTE that I assumed that this is an analysis... so I just
set
> the
> >> > init
> >> > >> >> time
> >> > >> >> to equal the valid time. Feel free to change that
however you
> >> need.
> >> > >> >>
> >> > >> >>
> >> > >> >> (4) Add the grid definition info. And I figured this out
by
> >> running
> >> > >> >> pcp_combine (see below) and then looking at the NetCDF
global
> >> > >> attributes.
> >> > >> >>
> >> > >> >>
> >> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
> 'name="TP_6h";
> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> >> > >> >>
> >> > >> >>
> >> > >> >> (5) Along the way, I ran this command to make sure my
python
> >> script
> >> > >> >> actually runs on the command line:
> >> > >> >>
> >> > >> >> python read_ECMWF_NC_2D.py
NR2006.pl.1by1.6522.2006012806.nc
> >> TP_6h
> >> > >> >>
> >> > >> >>
> >> > >> >> Please let me know how this goes for you.
> >> > >> >>
> >> > >> >>
> >> > >> >> Thanks,
> >> > >> >> John
> >> > >> >>
> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
> >> johnhg at ucar.edu>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Andrew,
> >> > >> >> >
> >> > >> >> > OK, question for you. Is this ECMWF actually model
output
> with
> >> a
> >> > >> 6522
> >> > >> >> > forecast hour? Or should this be interpreted as an
analysis
> >> that's
> >> > >> >> valid
> >> > >> >> > at 01/28/2006 at 00Z?
> >> > >> >> >
> >> > >> >> > That'll determine how best to encode the timing
information.
> >> > >> >> >
> >> > >> >> > Thanks,
> >> > >> >> > John
> >> > >> >> >
> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
Affiliate
> via
> >> RT
> >> > <
> >> > >> >> > met_help at ucar.edu> wrote:
> >> > >> >> >
> >> > >> >> >>
> >> > >> >> >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >
> >> > >> >> >>
> >> > >> >> >> John,
> >> > >> >> >> That is great. Yes let's go the python route, that
seems a
> lot
> >> > more
> >> > >> >> >> straightforward. Please let me know how I may proceed,
and
> >> thank
> >> > >> you in
> >> > >> >> >> advance for your help on this!!
> >> > >> >> >>
> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via
RT <
> >> > >> >> >> met_help at ucar.edu>
> >> > >> >> >> wrote:
> >> > >> >> >>
> >> > >> >> >> > Hello Andrew,
> >> > >> >> >> >
> >> > >> >> >> > I see you have a question about reading NetCDF data
into
> MET.
> >> > >> >> Judging
> >> > >> >> >> by
> >> > >> >> >> > the path you sent, I figured that this data lives on
theia.
> >> So
> >> > I
> >> > >> >> >> logged on
> >> > >> >> >> > and took a look.
> >> > >> >> >> >
> >> > >> >> >> > Looks like this ECMWF file doesn't exactly fall into
any of
> >> the
> >> > >> >> NetCDF
> >> > >> >> >> > categories that MET currently supports. The closest
is
> >> > >> CF-compliant
> >> > >> >> >> > NetCDF. So let's tell MET to interpret it as such
using
> the
> >> > >> >> "file_type
> >> > >> >> >> =
> >> > >> >> >> > NETCDF_NCCF;" option.
> >> > >> >> >> >
> >> > >> >> >> > When I'm trying to figure out if MET can read
gridded
> data, I
> >> > >> usually
> >> > >> >> >> use
> >> > >> >> >> > the plot_data_plane tool.
> >> > >> >> >> >
> >> > >> >> >> > Here's the commands I ran on theia:
> >> > >> >> >> >
> >> > >> >> >> > module use /contrib/modulefiles
> >> > >> >> >> >
> >> > >> >> >> > module load intel met/8.0
> >> > >> >> >> >
> >> > >> >> >> > plot_data_plane
> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >> > >> >> >> >
> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
file_type=NETCDF_NCCF;'
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > This command ran and produced the attached plot, but
it did
> >> also
> >> > >> >> >> produce a
> >> > >> >> >> > warning about the valid time:
> >> > >> >> >> >
> >> > >> >> >> > WARNING: NcCfFile::open() -> could not determine the
valid
> >> time,
> >> > >> >> using
> >> > >> >> >> 0.
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > So MET isn't getting the timing info correct. What
can we
> do
> >> > >> about
> >> > >> >> >> this?
> >> > >> >> >> > One option would be reformatting these NetCDF files
into a
> >> > format
> >> > >> >> that
> >> > >> >> >> MET
> >> > >> >> >> > can actually read... either CF-compliant NetCDF or
the MET
> >> > NetCDF
> >> > >> >> >> format.
> >> > >> >> >> > However, that's pretty cumbersome. I would instead
> recommend
> >> > >> that we
> >> > >> >> >> take
> >> > >> >> >> > advantage of a new feature in met-8.0 (released last
> >> Friday). We
> >> > >> can
> >> > >> >> >> write
> >> > >> >> >> > a Python script to read data from these files,
define the
> >> > >> metadata,
> >> > >> >> and
> >> > >> >> >> > then pass that to MET via memory.
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > If the timing information doesn't matter for some
reason,
> >> then
> >> > you
> >> > >> >> could
> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option. But
if it
> >> does
> >> > >> >> matter,
> >> > >> >> >> and
> >> > >> >> >> > presumably it does, then the Python route should
work.
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > Let me know how you'd like to proceed.
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > Thanks,
> >> > >> >> >> > John
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
Affiliate
> >> via
> >> > >> RT <
> >> > >> >> >> > met_help at ucar.edu> wrote:
> >> > >> >> >> >
> >> > >> >> >> > >
> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted
upon.
> >> > >> >> >> > > Transaction: Ticket created by
andrew.kren at noaa.gov
> >> > >> >> >> > > Queue: met_help
> >> > >> >> >> > > Subject: Question about MET pcp_combine
> >> > >> >> >> > > Owner: Nobody
> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> >> > >> >> >> > > Status: new
> >> > >> >> >> > > Ticket <URL:
> >> > >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> > >> >> >> >
> >> > >> >> >> > >
> >> > >> >> >> > >
> >> > >> >> >> > > Dear MET help,
> >> > >> >> >> > >
> >> > >> >> >> > > I have a question on the MET pcp_combine add tool.
I have
> >> > model
> >> > >> >> data
> >> > >> >> >> that
> >> > >> >> >> > > is considered my truth from an ECMWF Nature run
(long-run
> >> > >> >> simulation
> >> > >> >> >> used
> >> > >> >> >> > > to represent the true atmosphere). I want to
verify how
> >> well
> >> > my
> >> > >> >> >> forecast
> >> > >> >> >> > > output, in this case from the GFS model, is
compared to
> the
> >> > >> Nature
> >> > >> >> >> run.
> >> > >> >> >> > The
> >> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
includes
> 6-hr
> >> > >> >> >> precipitation
> >> > >> >> >> > > and other fields in its netcdf files. my GFS data
is grib
> >> > >> format. I
> >> > >> >> >> > already
> >> > >> >> >> > > can convert the GFS data to 24-hourly
accumulations using
> >> my
> >> > >> prior
> >> > >> >> >> > scripts
> >> > >> >> >> > > and with pcp_combine. I guess my issue is in the
ability
> to
> >> > read
> >> > >> >> the
> >> > >> >> >> > Nature
> >> > >> >> >> > > run netcdf files. I try the pcp_combine tool and I
get
> this
> >> > >> error
> >> > >> >> when
> >> > >> >> >> > > trying to create a 24-hr accumulation.
> >> > >> >> >> > >
> >> > >> >> >> > > (0) Current cycle: 2006012800
> >> > >> >> >> > >
> >> > >> >> >> > > DEBUG 2: Performing addition command for 4 files.
> >> > >> >> >> > >
> >> > >> >> >> > > DEBUG 1: Reading input file:
> >> > >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> > >> >> >> > >
> >> > >> >> >> > > DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file() ->
> >> > created
> >> > >> >> new
> >> > >> >> >> > > Met2dDataFile object of type "FileType_None".
> >> > >> >> >> > >
> >> > >> >> >> > > ERROR :
> >> > >> >> >> > >
> >> > >> >> >> > > ERROR : get_field() -> can't open data file
> >> > >> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> >> > >> >> >> > >
> >> > >> >> >> > > ERROR :
> >> > >> >> >> > >
> >> > >> >> >> > > I assume MET is having issues reading the variable
in the
> >> > file,
> >> > >> >> which
> >> > >> >> >> is
> >> > >> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is
there a
> >> > >> >> workaround
> >> > >> >> >> so
> >> > >> >> >> > > that it can successfully process this?
> >> > >> >> >> > >
> >> > >> >> >> > > Thank you kindly,
> >> > >> >> >> > >
> >> > >> >> >> > > --
> >> > >> >> >> > > Andrew Kren
> >> > >> >> >> > > Assistant Scientist
> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> >> > >> >> >> > > Boulder, Colorado, USA
> >> > >> >> >> > > Office: 303-497-2715
> >> > >> >> >> > >
> >> > >> >> >> > >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >>
> >> > >> >> >> --
> >> > >> >> >> Andrew Kren
> >> > >> >> >> Assistant Scientist
> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> >> > >> >> >> Boulder, Colorado, USA
> >> > >> >> >> Office: 303-497-2715
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >
> >> > >> > --
> >> > >> > Andrew Kren
> >> > >> > Assistant Scientist
> >> > >> > University of Miami CIMAS - NOAA/AOML
> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> >> > >> > Boulder, Colorado, USA
> >> > >> > Office: 303-497-2715
> >> > >> >
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Andrew Kren
> >> > >> Assistant Scientist
> >> > >> University of Miami CIMAS - NOAA/AOML
> >> > >> Duty Station: NCAR Foothills - FL2-3058
> >> > >> Boulder, Colorado, USA
> >> > >> Office: 303-497-2715
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >> --
> >> Andrew Kren
> >> Assistant Scientist
> >> University of Miami CIMAS - NOAA/AOML
> >> Duty Station: NCAR Foothills - FL2-3058
> >> Boulder, Colorado, USA
> >> Office: 303-497-2715
> >>
> >>
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Thu Oct 04 14:17:20 2018
Is there a workaround in the meantime John? Or were you going to check
on
that first? I got your script and confirmed your finding.
On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate <
andrew.kren at noaa.gov> wrote:
> Thanks John.
>
> On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Andrew,
>>
>> OK, this script runs successfully on theia with python from my
path:
>>
>> > which python
>>
>> /contrib/anaconda/anaconda2/4.4.0/bin/python
>>
>>
>> But when I run the script using /bin/python, I get the same error
as from
>> MET:
>>
>>
>> Traceback (most recent call last):
>>
>> File
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
>> line 3, in <module>
>>
>> from netCDF4 import Dataset
>>
>> ImportError: No module named netCDF4
>>
>> So it would appear that MET is not calling the version of python
that
>> lives
>> in my path. I need to check with another developer to figure out
if
>> there's a way to tell MET what version of python to run.
>>
>> Thanks,
>> John
>>
>> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>
>> > Andrew,
>> >
>> > Hmmm, we're close. Try running this one:
>> >
>> > /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
>> > create_24_accum_met_nr.sh
>> >
>> >
>> > The output is listed below. The punchline is... when I run the
python
>> > script on the command line everything is fine. When I run it
through
>> > plot_data_plane, I get a runtime error about netCDF4.
>> >
>> >
>> > I can confirm that this works fine on my project machine at NCAR.
So I
>> > suspect it's an environment issue.
>> >
>> >
>> > John
>> >
>> >
>> > [John.H.Gotway at tfe05 kren_data_20181003]$
./create_24_accum_met_nr.sh
>> >
>> > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> > NR2006.pl.1by1.6522.2006012806.nc'
>> >
>> > Variable Name : 'TP_6h'
>> >
>> > Data Shape: (181, 360)
>> >
>> > Data Type: dtype('float64')
>> >
>> > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
>> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
>> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
>> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
>> 'Surface',
>> > 'units': 'mm', 'accum': '06'}
>> >
>> > DEBUG 1: Opening data file: PYTHON_NUMPY
>> >
>> > ERROR :
>> >
>> > ERROR : python_dataplane() -> an error occurred importing module
>> > "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
>> >
>> > ERROR :
>> >
>> > Traceback (most recent call last):
>> >
>> > File
>> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
>> > line 3, in <module>
>> >
>> > from netCDF4 import Dataset
>> >
>> > ImportError: No module named netCDF4
>> >
>> >
>> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>> >>
>> >> John,
>> >> I ran this script,
>> >>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
>> and
>> >> it gave me this, presumably asking for more information to run
the met
>> >> command.
>> >>
>> >> The following have been reloaded with a version change:
>> >>
>> >> 1) intel/15.0.0 => intel/14.0.2
>> >>
>> >>
>> >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> >> NR2006.pl.1by1.6522.2006012806.nc'
>> >>
>> >> Variable Name : 'TP_6h'
>> >>
>> >> Data Shape: (181, 360)
>> >>
>> >> Data Type: dtype('float64')
>> >>
>> >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
>> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
>> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
>> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
>> 'Surface',
>> >> 'units': 'mm', 'accum': '06'}
>> >>
>> >>
>> >> *** Model Evaluation Tools (METV8.0) ***
>> >>
>> >>
>> >> Usage: plot_data_plane
>> >>
>> >> input_filename
>> >>
>> >> output_filename
>> >>
>> >> field_string
>> >>
>> >> [-color_table color_table_name]
>> >>
>> >> [-plot_range min max]
>> >>
>> >> [-title title_string]
>> >>
>> >> [-log file]
>> >>
>> >> [-v level]
>> >>
>> >>
>> >> where "input_filename" is the name of a gridded data file to be
>> plotted
>> >> (required).
>> >>
>> >> "output_filename" is the name of the output PostScript file to
be
>> written
>> >> (required).
>> >>
>> >> "field_string" defines the data to be plotted from the input
file
>> >> (required).
>> >>
>> >> "-color_table color_table_name" overrides the default color
table
>> >> ("colortables/met_default.ctable") (optional).
>> >>
>> >> "-plot_range min max" defines the range of the data to be
plotted
>> >> (optional).
>> >>
>> >> "-title title_string" specifies the plot title string
(optional).
>> >>
>> >> "-log file" outputs log messages to the specified file
(optional).
>> >>
>> >> "-v level" overrides the default level of logging (2)
(optional).
>> >>
>> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
>> >> met_help at ucar.edu>
>> >> wrote:
>> >>
>> >> > Andrew,
>> >> >
>> >> > Sorry, just noticed a typo after I hit send. I had a double-
equals
>> (==)
>> >> > that should be a single equal (=):
>> >> >
>> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> >> >
>> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > NR2006.pl.1by1.6522.2006012806.nc
>> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> >> >
>> >> > 'name="python
>> >> >
>> >> >
>> >>
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > NR2006.pl.1by1.6522.2006012806.nc
>> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>> >> >
>> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway
<johnhg at ucar.edu>
>> >> > wrote:
>> >> >
>> >> > > Great, so running the python script stand-alone read the
data well.
>> >> > >
>> >> > > The next step is telling MET to call this script. Here's
that
>> simple
>> >> > > example of running plot_data_plane:
>> >> > >
>> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> >> > >
>> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> >> > >
>> >> > > 'name=="python
>> >> > >
>> >> >
>> >>
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
>> >> > >
>> >> > >
>> >> > > Once this works, look at the output to confirm that MET is
putting
>> >> your
>> >> > > data on the right spot on the earth.
>> >> > >
>> >> > >
>> >> > > Presumably it does... so now you're ready to start using
this to
>> run
>> >> > > pcp_combine or Grid-Stat or MODE.
>> >> > >
>> >> > >
>> >> > > Make sense?
>> >> > >
>> >> > >
>> >> > > John
>> >> > >
>> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA Affiliate
via
>> RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > >>
>> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>> >> > >>
>> >> > >> John,
>> >> > >> I got this output when I ran your python script in a shell
script:
>> >> > >>
>> >> > >> The following have been reloaded with a version change:
>> >> > >>
>> >> > >> 1) intel/15.0.0 => intel/14.0.2
>> >> > >>
>> >> > >>
>> >> > >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
>> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
>> >> > >>
>> >> > >> Variable Name : 'TP_6h'
>> >> > >>
>> >> > >> Data Shape: (181, 360)
>> >> > >>
>> >> > >> Data Type: dtype('float64')
>> >> > >>
>> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
>> >> 'valid':
>> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
>> 'name':
>> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
>> >> 'type':
>> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
>> >> > 'Surface',
>> >> > >> 'units': 'mm', 'accum': '06'}
>> >> > >>
>> >> > >>
>> >> > >> Looks like it worked.
>> >> > >>
>> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA
Affiliate <
>> >> > >> andrew.kren at noaa.gov> wrote:
>> >> > >>
>> >> > >> > John,
>> >> > >> > I believe the 6522 and other numbers (which increase with
each
>> >> date)
>> >> > >> were
>> >> > >> > used as place holders for the filename. The actual dates
can be
>> >> read
>> >> > as
>> >> > >> an
>> >> > >> > analysis that's valid at the time stamp, such as
01/28/2006 at
>> 06Z?
>> >> > Thus
>> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
>> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
correspond to
>> >> the
>> >> > >> > analysis valid at 2006012806, does that help? I'll try
the
>> script.
>> >> > >> >
>> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via RT
<
>> >> > >> > met_help at ucar.edu> wrote:
>> >> > >> >
>> >> > >> >> Andrew,
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> OK, please try running this command on theia. I've
attached
>> the
>> >> > >> resulting
>> >> > >> >> image:
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> module use /contrib/modulefiles
>> >> > >> >>
>> >> > >> >> module load intel met/8.0
>> >> > >> >>
>> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
>> >> > >> >>
>> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
>> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
>> >> > >> >>
>> >> > >> >> 'name=="python
>> >> > >> >>
>> >> > >> >>
>> >> > >>
>> >> >
>> >>
>>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
>> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> This python script (also attached) reads your data and
hands it
>> >> off
>> >> > to
>> >> > >> >> MET.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Here's the steps I followed to do this:
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> (1) Copy over a sample python script from MET:
>> >> > >> >>
>> >> > >> >> cp /contrib/met/8.0/share/met/python/read_ascii_numpy.py
>> >> > >> >> read_ECMWF_NC_2D.py
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> (2) Edit it to read the input netcdf file (argument 1)
and the
>> >> > variable
>> >> > >> >> name (argument 2).
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> (3) Add code to parse out the timing info and define it
in the
>> >> "attr"
>> >> > >> >> dictionary.
>> >> > >> >>
>> >> > >> >> NOTE that I assumed that this is an analysis... so I
just set
>> the
>> >> > init
>> >> > >> >> time
>> >> > >> >> to equal the valid time. Feel free to change that
however you
>> >> need.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> (4) Add the grid definition info. And I figured this
out by
>> >> running
>> >> > >> >> pcp_combine (see below) and then looking at the NetCDF
global
>> >> > >> attributes.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
>> 'name="TP_6h";
>> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> (5) Along the way, I ran this command to make sure my
python
>> >> script
>> >> > >> >> actually runs on the command line:
>> >> > >> >>
>> >> > >> >> python read_ECMWF_NC_2D.py
NR2006.pl.1by1.6522.2006012806.nc
>> >> TP_6h
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Please let me know how this goes for you.
>> >> > >> >>
>> >> > >> >>
>> >> > >> >> Thanks,
>> >> > >> >> John
>> >> > >> >>
>> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
>> >> johnhg at ucar.edu>
>> >> > >> >> wrote:
>> >> > >> >>
>> >> > >> >> > Andrew,
>> >> > >> >> >
>> >> > >> >> > OK, question for you. Is this ECMWF actually model
output
>> with
>> >> a
>> >> > >> 6522
>> >> > >> >> > forecast hour? Or should this be interpreted as an
analysis
>> >> that's
>> >> > >> >> valid
>> >> > >> >> > at 01/28/2006 at 00Z?
>> >> > >> >> >
>> >> > >> >> > That'll determine how best to encode the timing
information.
>> >> > >> >> >
>> >> > >> >> > Thanks,
>> >> > >> >> > John
>> >> > >> >> >
>> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
Affiliate
>> via
>> >> RT
>> >> > <
>> >> > >> >> > met_help at ucar.edu> wrote:
>> >> > >> >> >
>> >> > >> >> >>
>> >> > >> >> >> <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >> >
>> >> > >> >> >>
>> >> > >> >> >> John,
>> >> > >> >> >> That is great. Yes let's go the python route, that
seems a
>> lot
>> >> > more
>> >> > >> >> >> straightforward. Please let me know how I may
proceed, and
>> >> thank
>> >> > >> you in
>> >> > >> >> >> advance for your help on this!!
>> >> > >> >> >>
>> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway via
RT <
>> >> > >> >> >> met_help at ucar.edu>
>> >> > >> >> >> wrote:
>> >> > >> >> >>
>> >> > >> >> >> > Hello Andrew,
>> >> > >> >> >> >
>> >> > >> >> >> > I see you have a question about reading NetCDF data
into
>> MET.
>> >> > >> >> Judging
>> >> > >> >> >> by
>> >> > >> >> >> > the path you sent, I figured that this data lives
on
>> theia.
>> >> So
>> >> > I
>> >> > >> >> >> logged on
>> >> > >> >> >> > and took a look.
>> >> > >> >> >> >
>> >> > >> >> >> > Looks like this ECMWF file doesn't exactly fall
into any
>> of
>> >> the
>> >> > >> >> NetCDF
>> >> > >> >> >> > categories that MET currently supports. The
closest is
>> >> > >> CF-compliant
>> >> > >> >> >> > NetCDF. So let's tell MET to interpret it as such
using
>> the
>> >> > >> >> "file_type
>> >> > >> >> >> =
>> >> > >> >> >> > NETCDF_NCCF;" option.
>> >> > >> >> >> >
>> >> > >> >> >> > When I'm trying to figure out if MET can read
gridded
>> data, I
>> >> > >> usually
>> >> > >> >> >> use
>> >> > >> >> >> > the plot_data_plane tool.
>> >> > >> >> >> >
>> >> > >> >> >> > Here's the commands I ran on theia:
>> >> > >> >> >> >
>> >> > >> >> >> > module use /contrib/modulefiles
>> >> > >> >> >> >
>> >> > >> >> >> > module load intel met/8.0
>> >> > >> >> >> >
>> >> > >> >> >> > plot_data_plane
>> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
>> >> > >> >> >> >
>> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
file_type=NETCDF_NCCF;'
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > This command ran and produced the attached plot,
but it
>> did
>> >> also
>> >> > >> >> >> produce a
>> >> > >> >> >> > warning about the valid time:
>> >> > >> >> >> >
>> >> > >> >> >> > WARNING: NcCfFile::open() -> could not determine
the valid
>> >> time,
>> >> > >> >> using
>> >> > >> >> >> 0.
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > So MET isn't getting the timing info correct. What
can
>> we do
>> >> > >> about
>> >> > >> >> >> this?
>> >> > >> >> >> > One option would be reformatting these NetCDF files
into a
>> >> > format
>> >> > >> >> that
>> >> > >> >> >> MET
>> >> > >> >> >> > can actually read... either CF-compliant NetCDF or
the MET
>> >> > NetCDF
>> >> > >> >> >> format.
>> >> > >> >> >> > However, that's pretty cumbersome. I would instead
>> recommend
>> >> > >> that we
>> >> > >> >> >> take
>> >> > >> >> >> > advantage of a new feature in met-8.0 (released
last
>> >> Friday). We
>> >> > >> can
>> >> > >> >> >> write
>> >> > >> >> >> > a Python script to read data from these files,
define the
>> >> > >> metadata,
>> >> > >> >> and
>> >> > >> >> >> > then pass that to MET via memory.
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > If the timing information doesn't matter for some
reason,
>> >> then
>> >> > you
>> >> > >> >> could
>> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option.
But if it
>> >> does
>> >> > >> >> matter,
>> >> > >> >> >> and
>> >> > >> >> >> > presumably it does, then the Python route should
work.
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > Let me know how you'd like to proceed.
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > Thanks,
>> >> > >> >> >> > John
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
>> Affiliate
>> >> via
>> >> > >> RT <
>> >> > >> >> >> > met_help at ucar.edu> wrote:
>> >> > >> >> >> >
>> >> > >> >> >> > >
>> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was acted
upon.
>> >> > >> >> >> > > Transaction: Ticket created by
andrew.kren at noaa.gov
>> >> > >> >> >> > > Queue: met_help
>> >> > >> >> >> > > Subject: Question about MET pcp_combine
>> >> > >> >> >> > > Owner: Nobody
>> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
>> >> > >> >> >> > > Status: new
>> >> > >> >> >> > > Ticket <URL:
>> >> > >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>> >> > >> >> >> >
>> >> > >> >> >> > >
>> >> > >> >> >> > >
>> >> > >> >> >> > > Dear MET help,
>> >> > >> >> >> > >
>> >> > >> >> >> > > I have a question on the MET pcp_combine add
tool. I
>> have
>> >> > model
>> >> > >> >> data
>> >> > >> >> >> that
>> >> > >> >> >> > > is considered my truth from an ECMWF Nature run
>> (long-run
>> >> > >> >> simulation
>> >> > >> >> >> used
>> >> > >> >> >> > > to represent the true atmosphere). I want to
verify how
>> >> well
>> >> > my
>> >> > >> >> >> forecast
>> >> > >> >> >> > > output, in this case from the GFS model, is
compared to
>> the
>> >> > >> Nature
>> >> > >> >> >> run.
>> >> > >> >> >> > The
>> >> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
includes
>> 6-hr
>> >> > >> >> >> precipitation
>> >> > >> >> >> > > and other fields in its netcdf files. my GFS data
is
>> grib
>> >> > >> format. I
>> >> > >> >> >> > already
>> >> > >> >> >> > > can convert the GFS data to 24-hourly
accumulations
>> using
>> >> my
>> >> > >> prior
>> >> > >> >> >> > scripts
>> >> > >> >> >> > > and with pcp_combine. I guess my issue is in the
>> ability to
>> >> > read
>> >> > >> >> the
>> >> > >> >> >> > Nature
>> >> > >> >> >> > > run netcdf files. I try the pcp_combine tool and
I get
>> this
>> >> > >> error
>> >> > >> >> when
>> >> > >> >> >> > > trying to create a 24-hr accumulation.
>> >> > >> >> >> > >
>> >> > >> >> >> > > (0) Current cycle: 2006012800
>> >> > >> >> >> > >
>> >> > >> >> >> > > DEBUG 2: Performing addition command for 4 files.
>> >> > >> >> >> > >
>> >> > >> >> >> > > DEBUG 1: Reading input file:
>> >> > >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
>> >> > >> >> >> > >
>> >> > >> >> >> > > DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file() ->
>> >> > created
>> >> > >> >> new
>> >> > >> >> >> > > Met2dDataFile object of type "FileType_None".
>> >> > >> >> >> > >
>> >> > >> >> >> > > ERROR :
>> >> > >> >> >> > >
>> >> > >> >> >> > > ERROR : get_field() -> can't open data file
>> >> > >> >> >> > > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
>> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
>> >> > >> >> >> > >
>> >> > >> >> >> > > ERROR :
>> >> > >> >> >> > >
>> >> > >> >> >> > > I assume MET is having issues reading the
variable in
>> the
>> >> > file,
>> >> > >> >> which
>> >> > >> >> >> is
>> >> > >> >> >> > > denoted at TP_6h for 6-hr total precipitation. Is
there
>> a
>> >> > >> >> workaround
>> >> > >> >> >> so
>> >> > >> >> >> > > that it can successfully process this?
>> >> > >> >> >> > >
>> >> > >> >> >> > > Thank you kindly,
>> >> > >> >> >> > >
>> >> > >> >> >> > > --
>> >> > >> >> >> > > Andrew Kren
>> >> > >> >> >> > > Assistant Scientist
>> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
>> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
>> >> > >> >> >> > > Boulder, Colorado, USA
>> >> > >> >> >> > > Office: 303-497-2715
>> >> > >> >> >> > >
>> >> > >> >> >> > >
>> >> > >> >> >> >
>> >> > >> >> >> >
>> >> > >> >> >>
>> >> > >> >> >> --
>> >> > >> >> >> Andrew Kren
>> >> > >> >> >> Assistant Scientist
>> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
>> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
>> >> > >> >> >> Boulder, Colorado, USA
>> >> > >> >> >> Office: 303-497-2715
>> >> > >> >> >>
>> >> > >> >> >>
>> >> > >> >>
>> >> > >> >>
>> >> > >> >
>> >> > >> > --
>> >> > >> > Andrew Kren
>> >> > >> > Assistant Scientist
>> >> > >> > University of Miami CIMAS - NOAA/AOML
>> >> > >> > Duty Station: NCAR Foothills - FL2-3058
>> >> > >> > Boulder, Colorado, USA
>> >> > >> > Office: 303-497-2715
>> >> > >> >
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Andrew Kren
>> >> > >> Assistant Scientist
>> >> > >> University of Miami CIMAS - NOAA/AOML
>> >> > >> Duty Station: NCAR Foothills - FL2-3058
>> >> > >> Boulder, Colorado, USA
>> >> > >> Office: 303-497-2715
>> >> > >>
>> >> > >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> Andrew Kren
>> >> Assistant Scientist
>> >> University of Miami CIMAS - NOAA/AOML
>> >> Duty Station: NCAR Foothills - FL2-3058
>> >> Boulder, Colorado, USA
>> >> Office: 303-497-2715
>> >>
>> >>
>>
>>
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Mon Oct 08 13:22:52 2018
Andrew,
I did some more hunting around on this issue. Please try adding the
following lines to the top of the python script you're running:
import sys
sys.path.insert(0,
'/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages')
sys.path.insert(0, '/contrib/anaconda/anaconda2/4.4.0/lib/site-
python')
The problem is that when Python is called by MET, the "site-packages"
settings differ from when invoking the python interpreter on the
command
line.
On the command line, we're pointing to the right site-packages:
python -c "import site; print(site.getsitepackages())"
['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages',
'/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
When called via MET, it points here:
['/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-
packages',
'/usr/lib/site-python']
We need to do some more digging to see if we can recompile MET to
point to
the *correct* site packages.
But please confirm that this change to the script enable MET to work
for
you.
Thanks,
John
On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> Is there a workaround in the meantime John? Or were you going to
check on
> that first? I got your script and confirmed your finding.
>
> On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate <
> andrew.kren at noaa.gov> wrote:
>
> > Thanks John.
> >
> > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Andrew,
> >>
> >> OK, this script runs successfully on theia with python from my
path:
> >>
> >> > which python
> >>
> >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> >>
> >>
> >> But when I run the script using /bin/python, I get the same error
as
> from
> >> MET:
> >>
> >>
> >> Traceback (most recent call last):
> >>
> >> File
> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> >> line 3, in <module>
> >>
> >> from netCDF4 import Dataset
> >>
> >> ImportError: No module named netCDF4
> >>
> >> So it would appear that MET is not calling the version of python
that
> >> lives
> >> in my path. I need to check with another developer to figure out
if
> >> there's a way to tell MET what version of python to run.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >> > Andrew,
> >> >
> >> > Hmmm, we're close. Try running this one:
> >> >
> >> >
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> >> > create_24_accum_met_nr.sh
> >> >
> >> >
> >> > The output is listed below. The punchline is... when I run the
python
> >> > script on the command line everything is fine. When I run it
through
> >> > plot_data_plane, I get a runtime error about netCDF4.
> >> >
> >> >
> >> > I can confirm that this works fine on my project machine at
NCAR. So
> I
> >> > suspect it's an environment issue.
> >> >
> >> >
> >> > John
> >> >
> >> >
> >> > [John.H.Gotway at tfe05 kren_data_20181003]$
./create_24_accum_met_nr.sh
> >> >
> >> > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> > NR2006.pl.1by1.6522.2006012806.nc'
> >> >
> >> > Variable Name : 'TP_6h'
> >> >
> >> > Data Shape: (181, 360)
> >> >
> >> > Data Type: dtype('float64')
> >> >
> >> > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> >> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
> 'type':
> >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> >> 'Surface',
> >> > 'units': 'mm', 'accum': '06'}
> >> >
> >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> >> >
> >> > ERROR :
> >> >
> >> > ERROR : python_dataplane() -> an error occurred importing
module
> >> >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> >> >
> >> > ERROR :
> >> >
> >> > Traceback (most recent call last):
> >> >
> >> > File
> >>
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> >> > line 3, in <module>
> >> >
> >> > from netCDF4 import Dataset
> >> >
> >> > ImportError: No module named netCDF4
> >> >
> >> >
> >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate
via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
> >> >>
> >> >> John,
> >> >> I ran this script,
> >> >>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> >> and
> >> >> it gave me this, presumably asking for more information to run
the
> met
> >> >> command.
> >> >>
> >> >> The following have been reloaded with a version change:
> >> >>
> >> >> 1) intel/15.0.0 => intel/14.0.2
> >> >>
> >> >>
> >> >> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> >> NR2006.pl.1by1.6522.2006012806.nc'
> >> >>
> >> >> Variable Name : 'TP_6h'
> >> >>
> >> >> Data Shape: (181, 360)
> >> >>
> >> >> Data Type: dtype('float64')
> >> >>
> >> >> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
> 'valid':
> >> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> 'type':
> >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> >> 'Surface',
> >> >> 'units': 'mm', 'accum': '06'}
> >> >>
> >> >>
> >> >> *** Model Evaluation Tools (METV8.0) ***
> >> >>
> >> >>
> >> >> Usage: plot_data_plane
> >> >>
> >> >> input_filename
> >> >>
> >> >> output_filename
> >> >>
> >> >> field_string
> >> >>
> >> >> [-color_table color_table_name]
> >> >>
> >> >> [-plot_range min max]
> >> >>
> >> >> [-title title_string]
> >> >>
> >> >> [-log file]
> >> >>
> >> >> [-v level]
> >> >>
> >> >>
> >> >> where "input_filename" is the name of a gridded data file to
be
> >> plotted
> >> >> (required).
> >> >>
> >> >> "output_filename" is the name of the output PostScript file to
be
> >> written
> >> >> (required).
> >> >>
> >> >> "field_string" defines the data to be plotted from the input
file
> >> >> (required).
> >> >>
> >> >> "-color_table color_table_name" overrides the default color
table
> >> >> ("colortables/met_default.ctable") (optional).
> >> >>
> >> >> "-plot_range min max" defines the range of the data to be
plotted
> >> >> (optional).
> >> >>
> >> >> "-title title_string" specifies the plot title string
(optional).
> >> >>
> >> >> "-log file" outputs log messages to the specified file
(optional).
> >> >>
> >> >> "-v level" overrides the default level of logging (2)
(optional).
> >> >>
> >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
> >> >> met_help at ucar.edu>
> >> >> wrote:
> >> >>
> >> >> > Andrew,
> >> >> >
> >> >> > Sorry, just noticed a typo after I hit send. I had a
double-equals
> >> (==)
> >> >> > that should be a single equal (=):
> >> >> >
> >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> >> >
> >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> >> >
> >> >> > 'name="python
> >> >> >
> >> >> >
> >> >>
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >> >> >
> >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <
> johnhg at ucar.edu>
> >> >> > wrote:
> >> >> >
> >> >> > > Great, so running the python script stand-alone read the
data
> well.
> >> >> > >
> >> >> > > The next step is telling MET to call this script. Here's
that
> >> simple
> >> >> > > example of running plot_data_plane:
> >> >> > >
> >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> >> > >
> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> >> > >
> >> >> > > 'name=="python
> >> >> > >
> >> >> >
> >> >>
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> >> >> > >
> >> >> > >
> >> >> > > Once this works, look at the output to confirm that MET is
> putting
> >> >> your
> >> >> > > data on the right spot on the earth.
> >> >> > >
> >> >> > >
> >> >> > > Presumably it does... so now you're ready to start using
this to
> >> run
> >> >> > > pcp_combine or Grid-Stat or MODE.
> >> >> > >
> >> >> > >
> >> >> > > Make sense?
> >> >> > >
> >> >> > >
> >> >> > > John
> >> >> > >
> >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA
Affiliate via
> >> RT <
> >> >> > > met_help at ucar.edu> wrote:
> >> >> > >
> >> >> > >>
> >> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >> >> > >>
> >> >> > >> John,
> >> >> > >> I got this output when I ran your python script in a
shell
> script:
> >> >> > >>
> >> >> > >> The following have been reloaded with a version change:
> >> >> > >>
> >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> >> >> > >>
> >> >> > >>
> >> >> > >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> >> >> > >>
> >> >> > >> Variable Name : 'TP_6h'
> >> >> > >>
> >> >> > >> Data Shape: (181, 360)
> >> >> > >>
> >> >> > >> Data Type: dtype('float64')
> >> >> > >>
> >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> >> >> 'valid':
> >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> >> 'name':
> >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon': 1.0,
> >> >> 'type':
> >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> >> >> > 'Surface',
> >> >> > >> 'units': 'mm', 'accum': '06'}
> >> >> > >>
> >> >> > >>
> >> >> > >> Looks like it worked.
> >> >> > >>
> >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA
Affiliate <
> >> >> > >> andrew.kren at noaa.gov> wrote:
> >> >> > >>
> >> >> > >> > John,
> >> >> > >> > I believe the 6522 and other numbers (which increase
with each
> >> >> date)
> >> >> > >> were
> >> >> > >> > used as place holders for the filename. The actual
dates can
> be
> >> >> read
> >> >> > as
> >> >> > >> an
> >> >> > >> > analysis that's valid at the time stamp, such as
01/28/2006 at
> >> 06Z?
> >> >> > Thus
> >> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
correspond
> to
> >> >> the
> >> >> > >> > analysis valid at 2006012806, does that help? I'll try
the
> >> script.
> >> >> > >> >
> >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway via
RT <
> >> >> > >> > met_help at ucar.edu> wrote:
> >> >> > >> >
> >> >> > >> >> Andrew,
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> OK, please try running this command on theia. I've
attached
> >> the
> >> >> > >> resulting
> >> >> > >> >> image:
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> module use /contrib/modulefiles
> >> >> > >> >>
> >> >> > >> >> module load intel met/8.0
> >> >> > >> >>
> >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> >> >> > >> >>
> >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> >> >> > >> >>
> >> >> > >> >> 'name=="python
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> This python script (also attached) reads your data and
hands
> it
> >> >> off
> >> >> > to
> >> >> > >> >> MET.
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> Here's the steps I followed to do this:
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> (1) Copy over a sample python script from MET:
> >> >> > >> >>
> >> >> > >> >> cp
/contrib/met/8.0/share/met/python/read_ascii_numpy.py
> >> >> > >> >> read_ECMWF_NC_2D.py
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> (2) Edit it to read the input netcdf file (argument 1)
and
> the
> >> >> > variable
> >> >> > >> >> name (argument 2).
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> (3) Add code to parse out the timing info and define
it in
> the
> >> >> "attr"
> >> >> > >> >> dictionary.
> >> >> > >> >>
> >> >> > >> >> NOTE that I assumed that this is an analysis... so I
just set
> >> the
> >> >> > init
> >> >> > >> >> time
> >> >> > >> >> to equal the valid time. Feel free to change that
however
> you
> >> >> need.
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> (4) Add the grid definition info. And I figured this
out by
> >> >> running
> >> >> > >> >> pcp_combine (see below) and then looking at the NetCDF
global
> >> >> > >> attributes.
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
> >> 'name="TP_6h";
> >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> (5) Along the way, I ran this command to make sure my
python
> >> >> script
> >> >> > >> >> actually runs on the command line:
> >> >> > >> >>
> >> >> > >> >> python read_ECMWF_NC_2D.py
NR2006.pl.1by1.6522.2006012806.nc
> >> >> TP_6h
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> Please let me know how this goes for you.
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >> Thanks,
> >> >> > >> >> John
> >> >> > >> >>
> >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
> >> >> johnhg at ucar.edu>
> >> >> > >> >> wrote:
> >> >> > >> >>
> >> >> > >> >> > Andrew,
> >> >> > >> >> >
> >> >> > >> >> > OK, question for you. Is this ECMWF actually model
output
> >> with
> >> >> a
> >> >> > >> 6522
> >> >> > >> >> > forecast hour? Or should this be interpreted as an
> analysis
> >> >> that's
> >> >> > >> >> valid
> >> >> > >> >> > at 01/28/2006 at 00Z?
> >> >> > >> >> >
> >> >> > >> >> > That'll determine how best to encode the timing
> information.
> >> >> > >> >> >
> >> >> > >> >> > Thanks,
> >> >> > >> >> > John
> >> >> > >> >> >
> >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
Affiliate
> >> via
> >> >> RT
> >> >> > <
> >> >> > >> >> > met_help at ucar.edu> wrote:
> >> >> > >> >> >
> >> >> > >> >> >>
> >> >> > >> >> >> <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >> >
> >> >> > >> >> >>
> >> >> > >> >> >> John,
> >> >> > >> >> >> That is great. Yes let's go the python route, that
seems a
> >> lot
> >> >> > more
> >> >> > >> >> >> straightforward. Please let me know how I may
proceed, and
> >> >> thank
> >> >> > >> you in
> >> >> > >> >> >> advance for your help on this!!
> >> >> > >> >> >>
> >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway
via RT <
> >> >> > >> >> >> met_help at ucar.edu>
> >> >> > >> >> >> wrote:
> >> >> > >> >> >>
> >> >> > >> >> >> > Hello Andrew,
> >> >> > >> >> >> >
> >> >> > >> >> >> > I see you have a question about reading NetCDF
data into
> >> MET.
> >> >> > >> >> Judging
> >> >> > >> >> >> by
> >> >> > >> >> >> > the path you sent, I figured that this data lives
on
> >> theia.
> >> >> So
> >> >> > I
> >> >> > >> >> >> logged on
> >> >> > >> >> >> > and took a look.
> >> >> > >> >> >> >
> >> >> > >> >> >> > Looks like this ECMWF file doesn't exactly fall
into any
> >> of
> >> >> the
> >> >> > >> >> NetCDF
> >> >> > >> >> >> > categories that MET currently supports. The
closest is
> >> >> > >> CF-compliant
> >> >> > >> >> >> > NetCDF. So let's tell MET to interpret it as
such using
> >> the
> >> >> > >> >> "file_type
> >> >> > >> >> >> =
> >> >> > >> >> >> > NETCDF_NCCF;" option.
> >> >> > >> >> >> >
> >> >> > >> >> >> > When I'm trying to figure out if MET can read
gridded
> >> data, I
> >> >> > >> usually
> >> >> > >> >> >> use
> >> >> > >> >> >> > the plot_data_plane tool.
> >> >> > >> >> >> >
> >> >> > >> >> >> > Here's the commands I ran on theia:
> >> >> > >> >> >> >
> >> >> > >> >> >> > module use /contrib/modulefiles
> >> >> > >> >> >> >
> >> >> > >> >> >> > module load intel met/8.0
> >> >> > >> >> >> >
> >> >> > >> >> >> > plot_data_plane
> >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> >> >> > >> >> >> >
> >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> file_type=NETCDF_NCCF;'
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > This command ran and produced the attached plot,
but it
> >> did
> >> >> also
> >> >> > >> >> >> produce a
> >> >> > >> >> >> > warning about the valid time:
> >> >> > >> >> >> >
> >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not determine
the
> valid
> >> >> time,
> >> >> > >> >> using
> >> >> > >> >> >> 0.
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > So MET isn't getting the timing info correct.
What can
> >> we do
> >> >> > >> about
> >> >> > >> >> >> this?
> >> >> > >> >> >> > One option would be reformatting these NetCDF
files
> into a
> >> >> > format
> >> >> > >> >> that
> >> >> > >> >> >> MET
> >> >> > >> >> >> > can actually read... either CF-compliant NetCDF
or the
> MET
> >> >> > NetCDF
> >> >> > >> >> >> format.
> >> >> > >> >> >> > However, that's pretty cumbersome. I would
instead
> >> recommend
> >> >> > >> that we
> >> >> > >> >> >> take
> >> >> > >> >> >> > advantage of a new feature in met-8.0 (released
last
> >> >> Friday). We
> >> >> > >> can
> >> >> > >> >> >> write
> >> >> > >> >> >> > a Python script to read data from these files,
define
> the
> >> >> > >> metadata,
> >> >> > >> >> and
> >> >> > >> >> >> > then pass that to MET via memory.
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > If the timing information doesn't matter for some
> reason,
> >> >> then
> >> >> > you
> >> >> > >> >> could
> >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option.
But if
> it
> >> >> does
> >> >> > >> >> matter,
> >> >> > >> >> >> and
> >> >> > >> >> >> > presumably it does, then the Python route should
work.
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > Let me know how you'd like to proceed.
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > Thanks,
> >> >> > >> >> >> > John
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren - NOAA
> >> Affiliate
> >> >> via
> >> >> > >> RT <
> >> >> > >> >> >> > met_help at ucar.edu> wrote:
> >> >> > >> >> >> >
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was
acted
> upon.
> >> >> > >> >> >> > > Transaction: Ticket created by
andrew.kren at noaa.gov
> >> >> > >> >> >> > > Queue: met_help
> >> >> > >> >> >> > > Subject: Question about MET pcp_combine
> >> >> > >> >> >> > > Owner: Nobody
> >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> >> >> > >> >> >> > > Status: new
> >> >> > >> >> >> > > Ticket <URL:
> >> >> > >> >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >> >> > >> >> >> >
> >> >> > >> >> >> > >
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > Dear MET help,
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > I have a question on the MET pcp_combine add
tool. I
> >> have
> >> >> > model
> >> >> > >> >> data
> >> >> > >> >> >> that
> >> >> > >> >> >> > > is considered my truth from an ECMWF Nature run
> >> (long-run
> >> >> > >> >> simulation
> >> >> > >> >> >> used
> >> >> > >> >> >> > > to represent the true atmosphere). I want to
verify
> how
> >> >> well
> >> >> > my
> >> >> > >> >> >> forecast
> >> >> > >> >> >> > > output, in this case from the GFS model, is
compared
> to
> >> the
> >> >> > >> Nature
> >> >> > >> >> >> run.
> >> >> > >> >> >> > The
> >> >> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
includes
> >> 6-hr
> >> >> > >> >> >> precipitation
> >> >> > >> >> >> > > and other fields in its netcdf files. my GFS
data is
> >> grib
> >> >> > >> format. I
> >> >> > >> >> >> > already
> >> >> > >> >> >> > > can convert the GFS data to 24-hourly
accumulations
> >> using
> >> >> my
> >> >> > >> prior
> >> >> > >> >> >> > scripts
> >> >> > >> >> >> > > and with pcp_combine. I guess my issue is in
the
> >> ability to
> >> >> > read
> >> >> > >> >> the
> >> >> > >> >> >> > Nature
> >> >> > >> >> >> > > run netcdf files. I try the pcp_combine tool
and I get
> >> this
> >> >> > >> error
> >> >> > >> >> when
> >> >> > >> >> >> > > trying to create a 24-hr accumulation.
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > (0) Current cycle: 2006012800
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > DEBUG 2: Performing addition command for 4
files.
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > DEBUG 1: Reading input file:
> >> >> > >> >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file()
> ->
> >> >> > created
> >> >> > >> >> new
> >> >> > >> >> >> > > Met2dDataFile object of type "FileType_None".
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > ERROR :
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > ERROR : get_field() -> can't open data file
> >> >> > >> >> >> > >
"/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > ERROR :
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > I assume MET is having issues reading the
variable in
> >> the
> >> >> > file,
> >> >> > >> >> which
> >> >> > >> >> >> is
> >> >> > >> >> >> > > denoted at TP_6h for 6-hr total precipitation.
Is
> there
> >> a
> >> >> > >> >> workaround
> >> >> > >> >> >> so
> >> >> > >> >> >> > > that it can successfully process this?
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > Thank you kindly,
> >> >> > >> >> >> > >
> >> >> > >> >> >> > > --
> >> >> > >> >> >> > > Andrew Kren
> >> >> > >> >> >> > > Assistant Scientist
> >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> >> >> > >> >> >> > > Boulder, Colorado, USA
> >> >> > >> >> >> > > Office: 303-497-2715
> >> >> > >> >> >> > >
> >> >> > >> >> >> > >
> >> >> > >> >> >> >
> >> >> > >> >> >> >
> >> >> > >> >> >>
> >> >> > >> >> >> --
> >> >> > >> >> >> Andrew Kren
> >> >> > >> >> >> Assistant Scientist
> >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> >> >> > >> >> >> Boulder, Colorado, USA
> >> >> > >> >> >> Office: 303-497-2715
> >> >> > >> >> >>
> >> >> > >> >> >>
> >> >> > >> >>
> >> >> > >> >>
> >> >> > >> >
> >> >> > >> > --
> >> >> > >> > Andrew Kren
> >> >> > >> > Assistant Scientist
> >> >> > >> > University of Miami CIMAS - NOAA/AOML
> >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> >> >> > >> > Boulder, Colorado, USA
> >> >> > >> > Office: 303-497-2715
> >> >> > >> >
> >> >> > >>
> >> >> > >>
> >> >> > >> --
> >> >> > >> Andrew Kren
> >> >> > >> Assistant Scientist
> >> >> > >> University of Miami CIMAS - NOAA/AOML
> >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> >> >> > >> Boulder, Colorado, USA
> >> >> > >> Office: 303-497-2715
> >> >> > >>
> >> >> > >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Andrew Kren
> >> >> Assistant Scientist
> >> >> University of Miami CIMAS - NOAA/AOML
> >> >> Duty Station: NCAR Foothills - FL2-3058
> >> >> Boulder, Colorado, USA
> >> >> Office: 303-497-2715
> >> >>
> >> >>
> >>
> >>
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
>
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Tue Oct 09 08:35:21 2018
John,
Ok I tried this morning putting that at the top of my python script,
read_ECMWF_NC_2D.py. I then ran my shell script as usual, but it still
gave
me this error:
The following have been reloaded with a version change:
1) intel/15.0.0 => intel/14.0.2
/contrib/anaconda/anaconda2/4.4.0/bin/python
Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
NR2006.pl.1by1.6522.2006012806.nc'
Variable Name : 'TP_6h'
Data Shape: (181, 360)
Data Type: dtype('float64')
Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000', 'valid':
'20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
'units': 'mm', 'accum': '06'}
DEBUG 1: Opening data file: PYTHON_NUMPY
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of
type "FileType_Python_Numpy".
ERROR :
ERROR : python_dataplane() -> an error occurred importing module
"./python.py"
ERROR :
ImportError: No module named python
/contrib/anaconda/anaconda2/4.4.0/bin/python
Do I need to add it to the shell script as well?
On Mon, Oct 8, 2018 at 1:22 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
> I did some more hunting around on this issue. Please try adding the
> following lines to the top of the python script you're running:
>
> import sys
> sys.path.insert(0,
> '/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages')
> sys.path.insert(0, '/contrib/anaconda/anaconda2/4.4.0/lib/site-
python')
>
> The problem is that when Python is called by MET, the "site-
packages"
> settings differ from when invoking the python interpreter on the
command
> line.
>
> On the command line, we're pointing to the right site-packages:
> python -c "import site; print(site.getsitepackages())"
> ['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages',
> '/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
>
> When called via MET, it points here:
> ['/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-
packages',
> '/usr/lib/site-python']
>
> We need to do some more digging to see if we can recompile MET to
point to
> the *correct* site packages.
>
> But please confirm that this change to the script enable MET to work
for
> you.
>
> Thanks,
> John
>
>
> On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >
> > Is there a workaround in the meantime John? Or were you going to
check on
> > that first? I got your script and confirmed your finding.
> >
> > On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate <
> > andrew.kren at noaa.gov> wrote:
> >
> > > Thanks John.
> > >
> > > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Andrew,
> > >>
> > >> OK, this script runs successfully on theia with python from my
path:
> > >>
> > >> > which python
> > >>
> > >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> > >>
> > >>
> > >> But when I run the script using /bin/python, I get the same
error as
> > from
> > >> MET:
> > >>
> > >>
> > >> Traceback (most recent call last):
> > >>
> > >> File
> > "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > >> line 3, in <module>
> > >>
> > >> from netCDF4 import Dataset
> > >>
> > >> ImportError: No module named netCDF4
> > >>
> > >> So it would appear that MET is not calling the version of
python that
> > >> lives
> > >> in my path. I need to check with another developer to figure
out if
> > >> there's a way to tell MET what version of python to run.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway
<johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >> > Andrew,
> > >> >
> > >> > Hmmm, we're close. Try running this one:
> > >> >
> > >> >
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > >> > create_24_accum_met_nr.sh
> > >> >
> > >> >
> > >> > The output is listed below. The punchline is... when I run
the
> python
> > >> > script on the command line everything is fine. When I run it
> through
> > >> > plot_data_plane, I get a runtime error about netCDF4.
> > >> >
> > >> >
> > >> > I can confirm that this works fine on my project machine at
NCAR.
> So
> > I
> > >> > suspect it's an environment issue.
> > >> >
> > >> >
> > >> > John
> > >> >
> > >> >
> > >> > [John.H.Gotway at tfe05 kren_data_20181003]$
> ./create_24_accum_met_nr.sh
> > >> >
> > >> > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > >> > NR2006.pl.1by1.6522.2006012806.nc'
> > >> >
> > >> > Variable Name : 'TP_6h'
> > >> >
> > >> > Data Shape: (181, 360)
> > >> >
> > >> > Data Type: dtype('float64')
> > >> >
> > >> > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
> 'valid':
> > >> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> > 'type':
> > >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > >> 'Surface',
> > >> > 'units': 'mm', 'accum': '06'}
> > >> >
> > >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> > >> >
> > >> > ERROR :
> > >> >
> > >> > ERROR : python_dataplane() -> an error occurred importing
module
> > >> >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> > >> >
> > >> > ERROR :
> > >> >
> > >> > Traceback (most recent call last):
> > >> >
> > >> > File
> > >>
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > >> > line 3, in <module>
> > >> >
> > >> > from netCDF4 import Dataset
> > >> >
> > >> > ImportError: No module named netCDF4
> > >> >
> > >> >
> > >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA Affiliate
via RT
> <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > >> >>
> > >> >> John,
> > >> >> I ran this script,
> > >> >>
>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> > >> and
> > >> >> it gave me this, presumably asking for more information to
run the
> > met
> > >> >> command.
> > >> >>
> > >> >> The following have been reloaded with a version change:
> > >> >>
> > >> >> 1) intel/15.0.0 => intel/14.0.2
> > >> >>
> > >> >>
> > >> >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > >> >> NR2006.pl.1by1.6522.2006012806.nc'
> > >> >>
> > >> >> Variable Name : 'TP_6h'
> > >> >>
> > >> >> Data Shape: (181, 360)
> > >> >>
> > >> >> Data Type: dtype('float64')
> > >> >>
> > >> >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > 'valid':
> > >> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
> 'name':
> > >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> > 'type':
> > >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > >> 'Surface',
> > >> >> 'units': 'mm', 'accum': '06'}
> > >> >>
> > >> >>
> > >> >> *** Model Evaluation Tools (METV8.0) ***
> > >> >>
> > >> >>
> > >> >> Usage: plot_data_plane
> > >> >>
> > >> >> input_filename
> > >> >>
> > >> >> output_filename
> > >> >>
> > >> >> field_string
> > >> >>
> > >> >> [-color_table color_table_name]
> > >> >>
> > >> >> [-plot_range min max]
> > >> >>
> > >> >> [-title title_string]
> > >> >>
> > >> >> [-log file]
> > >> >>
> > >> >> [-v level]
> > >> >>
> > >> >>
> > >> >> where "input_filename" is the name of a gridded data file
to be
> > >> plotted
> > >> >> (required).
> > >> >>
> > >> >> "output_filename" is the name of the output PostScript file
to be
> > >> written
> > >> >> (required).
> > >> >>
> > >> >> "field_string" defines the data to be plotted from the input
file
> > >> >> (required).
> > >> >>
> > >> >> "-color_table color_table_name" overrides the default color
table
> > >> >> ("colortables/met_default.ctable") (optional).
> > >> >>
> > >> >> "-plot_range min max" defines the range of the data to be
plotted
> > >> >> (optional).
> > >> >>
> > >> >> "-title title_string" specifies the plot title string
(optional).
> > >> >>
> > >> >> "-log file" outputs log messages to the specified file
(optional).
> > >> >>
> > >> >> "-v level" overrides the default level of logging (2)
(optional).
> > >> >>
> > >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT <
> > >> >> met_help at ucar.edu>
> > >> >> wrote:
> > >> >>
> > >> >> > Andrew,
> > >> >> >
> > >> >> > Sorry, just noticed a typo after I hit send. I had a
> double-equals
> > >> (==)
> > >> >> > that should be a single equal (=):
> > >> >> >
> > >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > >> >> >
> > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > >> >> >
> > >> >> > 'name="python
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > >> >> >
> > >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <
> > johnhg at ucar.edu>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Great, so running the python script stand-alone read the
data
> > well.
> > >> >> > >
> > >> >> > > The next step is telling MET to call this script.
Here's that
> > >> simple
> > >> >> > > example of running plot_data_plane:
> > >> >> > >
> > >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > >> >> > >
> > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > >> >> > >
> > >> >> > > 'name=="python
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > >> >> > >
> > >> >> > >
> > >> >> > > Once this works, look at the output to confirm that MET
is
> > putting
> > >> >> your
> > >> >> > > data on the right spot on the earth.
> > >> >> > >
> > >> >> > >
> > >> >> > > Presumably it does... so now you're ready to start using
this
> to
> > >> run
> > >> >> > > pcp_combine or Grid-Stat or MODE.
> > >> >> > >
> > >> >> > >
> > >> >> > > Make sense?
> > >> >> > >
> > >> >> > >
> > >> >> > > John
> > >> >> > >
> > >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA
Affiliate
> via
> > >> RT <
> > >> >> > > met_help at ucar.edu> wrote:
> > >> >> > >
> > >> >> > >>
> > >> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >
> > >> >> > >>
> > >> >> > >> John,
> > >> >> > >> I got this output when I ran your python script in a
shell
> > script:
> > >> >> > >>
> > >> >> > >> The following have been reloaded with a version change:
> > >> >> > >>
> > >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> > >> >> > >>
> > >> >> > >>
> > >> >> > >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > >> >> > >>
> > >> >> > >> Variable Name : 'TP_6h'
> > >> >> > >>
> > >> >> > >> Data Shape: (181, 360)
> > >> >> > >>
> > >> >> > >> Data Type: dtype('float64')
> > >> >> > >>
> > >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > >> >> 'valid':
> > >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > >> 'name':
> > >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> 1.0,
> > >> >> 'type':
> > >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
> 'level':
> > >> >> > 'Surface',
> > >> >> > >> 'units': 'mm', 'accum': '06'}
> > >> >> > >>
> > >> >> > >>
> > >> >> > >> Looks like it worked.
> > >> >> > >>
> > >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA
Affiliate <
> > >> >> > >> andrew.kren at noaa.gov> wrote:
> > >> >> > >>
> > >> >> > >> > John,
> > >> >> > >> > I believe the 6522 and other numbers (which increase
with
> each
> > >> >> date)
> > >> >> > >> were
> > >> >> > >> > used as place holders for the filename. The actual
dates can
> > be
> > >> >> read
> > >> >> > as
> > >> >> > >> an
> > >> >> > >> > analysis that's valid at the time stamp, such as
01/28/2006
> at
> > >> 06Z?
> > >> >> > Thus
> > >> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
> correspond
> > to
> > >> >> the
> > >> >> > >> > analysis valid at 2006012806, does that help? I'll
try the
> > >> script.
> > >> >> > >> >
> > >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway
via RT <
> > >> >> > >> > met_help at ucar.edu> wrote:
> > >> >> > >> >
> > >> >> > >> >> Andrew,
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> OK, please try running this command on theia. I've
> attached
> > >> the
> > >> >> > >> resulting
> > >> >> > >> >> image:
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> module use /contrib/modulefiles
> > >> >> > >> >>
> > >> >> > >> >> module load intel met/8.0
> > >> >> > >> >>
> > >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > >> >> > >> >>
> > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps
\
> > >> >> > >> >>
> > >> >> > >> >> 'name=="python
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >>
> > >> >> >
> > >> >>
> > >>
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> This python script (also attached) reads your data
and
> hands
> > it
> > >> >> off
> > >> >> > to
> > >> >> > >> >> MET.
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> Here's the steps I followed to do this:
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> (1) Copy over a sample python script from MET:
> > >> >> > >> >>
> > >> >> > >> >> cp
/contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > >> >> > >> >> read_ECMWF_NC_2D.py
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> (2) Edit it to read the input netcdf file (argument
1) and
> > the
> > >> >> > variable
> > >> >> > >> >> name (argument 2).
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> (3) Add code to parse out the timing info and define
it in
> > the
> > >> >> "attr"
> > >> >> > >> >> dictionary.
> > >> >> > >> >>
> > >> >> > >> >> NOTE that I assumed that this is an analysis... so I
just
> set
> > >> the
> > >> >> > init
> > >> >> > >> >> time
> > >> >> > >> >> to equal the valid time. Feel free to change that
however
> > you
> > >> >> need.
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> (4) Add the grid definition info. And I figured
this out
> by
> > >> >> running
> > >> >> > >> >> pcp_combine (see below) and then looking at the
NetCDF
> global
> > >> >> > >> attributes.
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
> > >> 'name="TP_6h";
> > >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> (5) Along the way, I ran this command to make sure
my
> python
> > >> >> script
> > >> >> > >> >> actually runs on the command line:
> > >> >> > >> >>
> > >> >> > >> >> python read_ECMWF_NC_2D.py
> NR2006.pl.1by1.6522.2006012806.nc
> > >> >> TP_6h
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> Please let me know how this goes for you.
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >> Thanks,
> > >> >> > >> >> John
> > >> >> > >> >>
> > >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway <
> > >> >> johnhg at ucar.edu>
> > >> >> > >> >> wrote:
> > >> >> > >> >>
> > >> >> > >> >> > Andrew,
> > >> >> > >> >> >
> > >> >> > >> >> > OK, question for you. Is this ECMWF actually
model
> output
> > >> with
> > >> >> a
> > >> >> > >> 6522
> > >> >> > >> >> > forecast hour? Or should this be interpreted as
an
> > analysis
> > >> >> that's
> > >> >> > >> >> valid
> > >> >> > >> >> > at 01/28/2006 at 00Z?
> > >> >> > >> >> >
> > >> >> > >> >> > That'll determine how best to encode the timing
> > information.
> > >> >> > >> >> >
> > >> >> > >> >> > Thanks,
> > >> >> > >> >> > John
> > >> >> > >> >> >
> > >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren - NOAA
> Affiliate
> > >> via
> > >> >> RT
> > >> >> > <
> > >> >> > >> >> > met_help at ucar.edu> wrote:
> > >> >> > >> >> >
> > >> >> > >> >> >>
> > >> >> > >> >> >> <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > >> >> >
> > >> >> > >> >> >>
> > >> >> > >> >> >> John,
> > >> >> > >> >> >> That is great. Yes let's go the python route,
that
> seems a
> > >> lot
> > >> >> > more
> > >> >> > >> >> >> straightforward. Please let me know how I may
proceed,
> and
> > >> >> thank
> > >> >> > >> you in
> > >> >> > >> >> >> advance for your help on this!!
> > >> >> > >> >> >>
> > >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley Gotway
via
> RT <
> > >> >> > >> >> >> met_help at ucar.edu>
> > >> >> > >> >> >> wrote:
> > >> >> > >> >> >>
> > >> >> > >> >> >> > Hello Andrew,
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > I see you have a question about reading NetCDF
data
> into
> > >> MET.
> > >> >> > >> >> Judging
> > >> >> > >> >> >> by
> > >> >> > >> >> >> > the path you sent, I figured that this data
lives on
> > >> theia.
> > >> >> So
> > >> >> > I
> > >> >> > >> >> >> logged on
> > >> >> > >> >> >> > and took a look.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > Looks like this ECMWF file doesn't exactly fall
into
> any
> > >> of
> > >> >> the
> > >> >> > >> >> NetCDF
> > >> >> > >> >> >> > categories that MET currently supports. The
closest
> is
> > >> >> > >> CF-compliant
> > >> >> > >> >> >> > NetCDF. So let's tell MET to interpret it as
such
> using
> > >> the
> > >> >> > >> >> "file_type
> > >> >> > >> >> >> =
> > >> >> > >> >> >> > NETCDF_NCCF;" option.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > When I'm trying to figure out if MET can read
gridded
> > >> data, I
> > >> >> > >> usually
> > >> >> > >> >> >> use
> > >> >> > >> >> >> > the plot_data_plane tool.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > Here's the commands I ran on theia:
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > module use /contrib/modulefiles
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > module load intel met/8.0
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > plot_data_plane
> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> > file_type=NETCDF_NCCF;'
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > This command ran and produced the attached
plot, but
> it
> > >> did
> > >> >> also
> > >> >> > >> >> >> produce a
> > >> >> > >> >> >> > warning about the valid time:
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not
determine the
> > valid
> > >> >> time,
> > >> >> > >> >> using
> > >> >> > >> >> >> 0.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > So MET isn't getting the timing info correct.
What
> can
> > >> we do
> > >> >> > >> about
> > >> >> > >> >> >> this?
> > >> >> > >> >> >> > One option would be reformatting these NetCDF
files
> > into a
> > >> >> > format
> > >> >> > >> >> that
> > >> >> > >> >> >> MET
> > >> >> > >> >> >> > can actually read... either CF-compliant NetCDF
or the
> > MET
> > >> >> > NetCDF
> > >> >> > >> >> >> format.
> > >> >> > >> >> >> > However, that's pretty cumbersome. I would
instead
> > >> recommend
> > >> >> > >> that we
> > >> >> > >> >> >> take
> > >> >> > >> >> >> > advantage of a new feature in met-8.0 (released
last
> > >> >> Friday). We
> > >> >> > >> can
> > >> >> > >> >> >> write
> > >> >> > >> >> >> > a Python script to read data from these files,
define
> > the
> > >> >> > >> metadata,
> > >> >> > >> >> and
> > >> >> > >> >> >> > then pass that to MET via memory.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > If the timing information doesn't matter for
some
> > reason,
> > >> >> then
> > >> >> > you
> > >> >> > >> >> could
> > >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;" option.
But
> if
> > it
> > >> >> does
> > >> >> > >> >> matter,
> > >> >> > >> >> >> and
> > >> >> > >> >> >> > presumably it does, then the Python route
should work.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > Let me know how you'd like to proceed.
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > Thanks,
> > >> >> > >> >> >> > John
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren -
NOAA
> > >> Affiliate
> > >> >> via
> > >> >> > >> RT <
> > >> >> > >> >> >> > met_help at ucar.edu> wrote:
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was
acted
> > upon.
> > >> >> > >> >> >> > > Transaction: Ticket created by
andrew.kren at noaa.gov
> > >> >> > >> >> >> > > Queue: met_help
> > >> >> > >> >> >> > > Subject: Question about MET pcp_combine
> > >> >> > >> >> >> > > Owner: Nobody
> > >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > >> >> > >> >> >> > > Status: new
> > >> >> > >> >> >> > > Ticket <URL:
> > >> >> > >> >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > >> >> > >> >> >> >
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > Dear MET help,
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > I have a question on the MET pcp_combine add
tool. I
> > >> have
> > >> >> > model
> > >> >> > >> >> data
> > >> >> > >> >> >> that
> > >> >> > >> >> >> > > is considered my truth from an ECMWF Nature
run
> > >> (long-run
> > >> >> > >> >> simulation
> > >> >> > >> >> >> used
> > >> >> > >> >> >> > > to represent the true atmosphere). I want to
verify
> > how
> > >> >> well
> > >> >> > my
> > >> >> > >> >> >> forecast
> > >> >> > >> >> >> > > output, in this case from the GFS model, is
compared
> > to
> > >> the
> > >> >> > >> Nature
> > >> >> > >> >> >> run.
> > >> >> > >> >> >> > The
> > >> >> > >> >> >> > > ECMWF nature run data is 6-hourly data, which
> includes
> > >> 6-hr
> > >> >> > >> >> >> precipitation
> > >> >> > >> >> >> > > and other fields in its netcdf files. my GFS
data is
> > >> grib
> > >> >> > >> format. I
> > >> >> > >> >> >> > already
> > >> >> > >> >> >> > > can convert the GFS data to 24-hourly
accumulations
> > >> using
> > >> >> my
> > >> >> > >> prior
> > >> >> > >> >> >> > scripts
> > >> >> > >> >> >> > > and with pcp_combine. I guess my issue is in
the
> > >> ability to
> > >> >> > read
> > >> >> > >> >> the
> > >> >> > >> >> >> > Nature
> > >> >> > >> >> >> > > run netcdf files. I try the pcp_combine tool
and I
> get
> > >> this
> > >> >> > >> error
> > >> >> > >> >> when
> > >> >> > >> >> >> > > trying to create a 24-hr accumulation.
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > (0) Current cycle: 2006012800
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > DEBUG 2: Performing addition command for 4
files.
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > DEBUG 1: Reading input file:
> > >> >> > >> >> >> > >
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > DEBUG 4:
> Met2dDataFileFactory::new_met_2d_data_file()
> > ->
> > >> >> > created
> > >> >> > >> >> new
> > >> >> > >> >> >> > > Met2dDataFile object of type "FileType_None".
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > ERROR :
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > ERROR : get_field() -> can't open data file
> > >> >> > >> >> >> > >
"/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > ERROR :
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > I assume MET is having issues reading the
variable
> in
> > >> the
> > >> >> > file,
> > >> >> > >> >> which
> > >> >> > >> >> >> is
> > >> >> > >> >> >> > > denoted at TP_6h for 6-hr total
precipitation. Is
> > there
> > >> a
> > >> >> > >> >> workaround
> > >> >> > >> >> >> so
> > >> >> > >> >> >> > > that it can successfully process this?
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > Thank you kindly,
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > > --
> > >> >> > >> >> >> > > Andrew Kren
> > >> >> > >> >> >> > > Assistant Scientist
> > >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> > >> >> > >> >> >> > > Boulder, Colorado, USA
> > >> >> > >> >> >> > > Office: 303-497-2715
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> > >
> > >> >> > >> >> >> >
> > >> >> > >> >> >> >
> > >> >> > >> >> >>
> > >> >> > >> >> >> --
> > >> >> > >> >> >> Andrew Kren
> > >> >> > >> >> >> Assistant Scientist
> > >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > >> >> > >> >> >> Boulder, Colorado, USA
> > >> >> > >> >> >> Office: 303-497-2715
> > >> >> > >> >> >>
> > >> >> > >> >> >>
> > >> >> > >> >>
> > >> >> > >> >>
> > >> >> > >> >
> > >> >> > >> > --
> > >> >> > >> > Andrew Kren
> > >> >> > >> > Assistant Scientist
> > >> >> > >> > University of Miami CIMAS - NOAA/AOML
> > >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> > >> >> > >> > Boulder, Colorado, USA
> > >> >> > >> > Office: 303-497-2715
> > >> >> > >> >
> > >> >> > >>
> > >> >> > >>
> > >> >> > >> --
> > >> >> > >> Andrew Kren
> > >> >> > >> Assistant Scientist
> > >> >> > >> University of Miami CIMAS - NOAA/AOML
> > >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> > >> >> > >> Boulder, Colorado, USA
> > >> >> > >> Office: 303-497-2715
> > >> >> > >>
> > >> >> > >>
> > >> >> >
> > >> >> >
> > >> >>
> > >> >> --
> > >> >> Andrew Kren
> > >> >> Assistant Scientist
> > >> >> University of Miami CIMAS - NOAA/AOML
> > >> >> Duty Station: NCAR Foothills - FL2-3058
> > >> >> Boulder, Colorado, USA
> > >> >> Office: 303-497-2715
> > >> >>
> > >> >>
> > >>
> > >>
> > >
> > > --
> > > Andrew Kren
> > > Assistant Scientist
> > > University of Miami CIMAS - NOAA/AOML
> > > Duty Station: NCAR Foothills - FL2-3058
> > > Boulder, Colorado, USA
> > > Office: 303-497-2715
> > >
> >
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
> >
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Tue Oct 09 09:31:07 2018
Andrew,
I have a good guess as to what's happening here and this is easy to
fix.
Let's say you have a python script named "myscript.py". To run it on
the
command line, you run:
python myscript.py
But when running it via a MET tool, remove the leading "python"
command,
like this:
plot_data_plane PYTHON_NUMPY plot.ps 'name="myscript.py";'
Notice that I used 'name="myscript.py";' and NOT 'name="python
myscript.py";'.
Please make sure that's how you're specifying it and try again.
Thanks,
John
On Tue, Oct 9, 2018 at 8:35 AM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> John,
> Ok I tried this morning putting that at the top of my python script,
> read_ECMWF_NC_2D.py. I then ran my shell script as usual, but it
still gave
> me this error:
>
> The following have been reloaded with a version change:
>
> 1) intel/15.0.0 => intel/14.0.2
>
>
> /contrib/anaconda/anaconda2/4.4.0/bin/python
>
> Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> NR2006.pl.1by1.6522.2006012806.nc'
>
> Variable Name : 'TP_6h'
>
> Data Shape: (181, 360)
>
> Data Type: dtype('float64')
>
> Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0, 'name':
> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
'Surface',
> 'units': 'mm', 'accum': '06'}
>
> DEBUG 1: Opening data file: PYTHON_NUMPY
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Python_Numpy".
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Python_Numpy".
>
> ERROR :
>
> ERROR : python_dataplane() -> an error occurred importing module
> "./python.py"
>
> ERROR :
>
> ImportError: No module named python
>
> /contrib/anaconda/anaconda2/4.4.0/bin/python
>
>
> Do I need to add it to the shell script as well?
>
> On Mon, Oct 8, 2018 at 1:22 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Andrew,
> >
> > I did some more hunting around on this issue. Please try adding
the
> > following lines to the top of the python script you're running:
> >
> > import sys
> > sys.path.insert(0,
> > '/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages')
> > sys.path.insert(0, '/contrib/anaconda/anaconda2/4.4.0/lib/site-
python')
> >
> > The problem is that when Python is called by MET, the "site-
packages"
> > settings differ from when invoking the python interpreter on the
command
> > line.
> >
> > On the command line, we're pointing to the right site-packages:
> > python -c "import site; print(site.getsitepackages())"
> > ['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages',
> > '/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
> >
> > When called via MET, it points here:
> > ['/usr/lib64/python2.7/site-packages',
> '/usr/lib/python2.7/site-packages',
> > '/usr/lib/site-python']
> >
> > We need to do some more digging to see if we can recompile MET to
point
> to
> > the *correct* site packages.
> >
> > But please confirm that this change to the script enable MET to
work for
> > you.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > >
> > > Is there a workaround in the meantime John? Or were you going to
check
> on
> > > that first? I got your script and confirmed your finding.
> > >
> > > On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate <
> > > andrew.kren at noaa.gov> wrote:
> > >
> > > > Thanks John.
> > > >
> > > > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Andrew,
> > > >>
> > > >> OK, this script runs successfully on theia with python from
my path:
> > > >>
> > > >> > which python
> > > >>
> > > >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > >>
> > > >>
> > > >> But when I run the script using /bin/python, I get the same
error as
> > > from
> > > >> MET:
> > > >>
> > > >>
> > > >> Traceback (most recent call last):
> > > >>
> > > >> File
> > >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > >> line 3, in <module>
> > > >>
> > > >> from netCDF4 import Dataset
> > > >>
> > > >> ImportError: No module named netCDF4
> > > >>
> > > >> So it would appear that MET is not calling the version of
python
> that
> > > >> lives
> > > >> in my path. I need to check with another developer to figure
out if
> > > >> there's a way to tell MET what version of python to run.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway
<johnhg at ucar.edu
> >
> > > >> wrote:
> > > >>
> > > >> > Andrew,
> > > >> >
> > > >> > Hmmm, we're close. Try running this one:
> > > >> >
> > > >> >
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > > >> > create_24_accum_met_nr.sh
> > > >> >
> > > >> >
> > > >> > The output is listed below. The punchline is... when I run
the
> > python
> > > >> > script on the command line everything is fine. When I run
it
> > through
> > > >> > plot_data_plane, I get a runtime error about netCDF4.
> > > >> >
> > > >> >
> > > >> > I can confirm that this works fine on my project machine at
NCAR.
> > So
> > > I
> > > >> > suspect it's an environment issue.
> > > >> >
> > > >> >
> > > >> > John
> > > >> >
> > > >> >
> > > >> > [John.H.Gotway at tfe05 kren_data_20181003]$
> > ./create_24_accum_met_nr.sh
> > > >> >
> > > >> > Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > >> > NR2006.pl.1by1.6522.2006012806.nc'
> > > >> >
> > > >> > Variable Name : 'TP_6h'
> > > >> >
> > > >> > Data Shape: (181, 360)
> > > >> >
> > > >> > Data Type: dtype('float64')
> > > >> >
> > > >> > Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > 'valid':
> > > >> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> 'name':
> > > >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> > > 'type':
> > > >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > > >> 'Surface',
> > > >> > 'units': 'mm', 'accum': '06'}
> > > >> >
> > > >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> > > >> >
> > > >> > ERROR :
> > > >> >
> > > >> > ERROR : python_dataplane() -> an error occurred importing
module
> > > >> >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> > > >> >
> > > >> > ERROR :
> > > >> >
> > > >> > Traceback (most recent call last):
> > > >> >
> > > >> > File
> > > >>
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > >> > line 3, in <module>
> > > >> >
> > > >> > from netCDF4 import Dataset
> > > >> >
> > > >> > ImportError: No module named netCDF4
> > > >> >
> > > >> >
> > > >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA
Affiliate via
> RT
> > <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> >>
> > > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > > >> >>
> > > >> >> John,
> > > >> >> I ran this script,
> > > >> >>
> >
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> > > >> and
> > > >> >> it gave me this, presumably asking for more information to
run
> the
> > > met
> > > >> >> command.
> > > >> >>
> > > >> >> The following have been reloaded with a version change:
> > > >> >>
> > > >> >> 1) intel/15.0.0 => intel/14.0.2
> > > >> >>
> > > >> >>
> > > >> >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > >> >> NR2006.pl.1by1.6522.2006012806.nc'
> > > >> >>
> > > >> >> Variable Name : 'TP_6h'
> > > >> >>
> > > >> >> Data Shape: (181, 360)
> > > >> >>
> > > >> >> Data Type: dtype('float64')
> > > >> >>
> > > >> >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > > 'valid':
> > > >> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > 'name':
> > > >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> > > 'type':
> > > >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > > >> 'Surface',
> > > >> >> 'units': 'mm', 'accum': '06'}
> > > >> >>
> > > >> >>
> > > >> >> *** Model Evaluation Tools (METV8.0) ***
> > > >> >>
> > > >> >>
> > > >> >> Usage: plot_data_plane
> > > >> >>
> > > >> >> input_filename
> > > >> >>
> > > >> >> output_filename
> > > >> >>
> > > >> >> field_string
> > > >> >>
> > > >> >> [-color_table color_table_name]
> > > >> >>
> > > >> >> [-plot_range min max]
> > > >> >>
> > > >> >> [-title title_string]
> > > >> >>
> > > >> >> [-log file]
> > > >> >>
> > > >> >> [-v level]
> > > >> >>
> > > >> >>
> > > >> >> where "input_filename" is the name of a gridded data file
to be
> > > >> plotted
> > > >> >> (required).
> > > >> >>
> > > >> >> "output_filename" is the name of the output PostScript
file to be
> > > >> written
> > > >> >> (required).
> > > >> >>
> > > >> >> "field_string" defines the data to be plotted from the
input file
> > > >> >> (required).
> > > >> >>
> > > >> >> "-color_table color_table_name" overrides the default
color table
> > > >> >> ("colortables/met_default.ctable") (optional).
> > > >> >>
> > > >> >> "-plot_range min max" defines the range of the data to be
plotted
> > > >> >> (optional).
> > > >> >>
> > > >> >> "-title title_string" specifies the plot title string
(optional).
> > > >> >>
> > > >> >> "-log file" outputs log messages to the specified file
> (optional).
> > > >> >>
> > > >> >> "-v level" overrides the default level of logging (2)
(optional).
> > > >> >>
> > > >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via RT
<
> > > >> >> met_help at ucar.edu>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Andrew,
> > > >> >> >
> > > >> >> > Sorry, just noticed a typo after I hit send. I had a
> > double-equals
> > > >> (==)
> > > >> >> > that should be a single equal (=):
> > > >> >> >
> > > >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > >> >> >
> > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > > >> >> >
> > > >> >> > 'name="python
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > > >> >> >
> > > >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > Great, so running the python script stand-alone read
the data
> > > well.
> > > >> >> > >
> > > >> >> > > The next step is telling MET to call this script.
Here's
> that
> > > >> simple
> > > >> >> > > example of running plot_data_plane:
> > > >> >> > >
> > > >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > >> >> > >
> > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > > >> >> > >
> > > >> >> > > 'name=="python
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > Once this works, look at the output to confirm that
MET is
> > > putting
> > > >> >> your
> > > >> >> > > data on the right spot on the earth.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > Presumably it does... so now you're ready to start
using this
> > to
> > > >> run
> > > >> >> > > pcp_combine or Grid-Stat or MODE.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > Make sense?
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > John
> > > >> >> > >
> > > >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA
Affiliate
> > via
> > > >> RT <
> > > >> >> > > met_help at ucar.edu> wrote:
> > > >> >> > >
> > > >> >> > >>
> > > >> >> > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > >
> > > >> >> > >>
> > > >> >> > >> John,
> > > >> >> > >> I got this output when I ran your python script in a
shell
> > > script:
> > > >> >> > >>
> > > >> >> > >> The following have been reloaded with a version
change:
> > > >> >> > >>
> > > >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >> Input File:
> '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > > >> >> > >>
> > > >> >> > >> Variable Name : 'TP_6h'
> > > >> >> > >>
> > > >> >> > >> Data Shape: (181, 360)
> > > >> >> > >>
> > > >> >> > >> Data Type: dtype('float64')
> > > >> >> > >>
> > > >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
> '20060128_060000',
> > > >> >> 'valid':
> > > >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll': 0.0,
> > > >> 'name':
> > > >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> > 1.0,
> > > >> >> 'type':
> > > >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead':
'00',
> > 'level':
> > > >> >> > 'Surface',
> > > >> >> > >> 'units': 'mm', 'accum': '06'}
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >> Looks like it worked.
> > > >> >> > >>
> > > >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA
> Affiliate <
> > > >> >> > >> andrew.kren at noaa.gov> wrote:
> > > >> >> > >>
> > > >> >> > >> > John,
> > > >> >> > >> > I believe the 6522 and other numbers (which
increase with
> > each
> > > >> >> date)
> > > >> >> > >> were
> > > >> >> > >> > used as place holders for the filename. The actual
dates
> can
> > > be
> > > >> >> read
> > > >> >> > as
> > > >> >> > >> an
> > > >> >> > >> > analysis that's valid at the time stamp, such as
> 01/28/2006
> > at
> > > >> 06Z?
> > > >> >> > Thus
> > > >> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
> > correspond
> > > to
> > > >> >> the
> > > >> >> > >> > analysis valid at 2006012806, does that help? I'll
try the
> > > >> script.
> > > >> >> > >> >
> > > >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley Gotway
via RT
> <
> > > >> >> > >> > met_help at ucar.edu> wrote:
> > > >> >> > >> >
> > > >> >> > >> >> Andrew,
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> OK, please try running this command on theia.
I've
> > attached
> > > >> the
> > > >> >> > >> resulting
> > > >> >> > >> >> image:
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> module use /contrib/modulefiles
> > > >> >> > >> >>
> > > >> >> > >> >> module load intel met/8.0
> > > >> >> > >> >>
> > > >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > >> >> > >> >>
> > > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h.ps \
> > > >> >> > >> >>
> > > >> >> > >> >> 'name=="python
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >>
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> This python script (also attached) reads your data
and
> > hands
> > > it
> > > >> >> off
> > > >> >> > to
> > > >> >> > >> >> MET.
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> Here's the steps I followed to do this:
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> (1) Copy over a sample python script from MET:
> > > >> >> > >> >>
> > > >> >> > >> >> cp
/contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > > >> >> > >> >> read_ECMWF_NC_2D.py
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> (2) Edit it to read the input netcdf file
(argument 1)
> and
> > > the
> > > >> >> > variable
> > > >> >> > >> >> name (argument 2).
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> (3) Add code to parse out the timing info and
define it
> in
> > > the
> > > >> >> "attr"
> > > >> >> > >> >> dictionary.
> > > >> >> > >> >>
> > > >> >> > >> >> NOTE that I assumed that this is an analysis... so
I just
> > set
> > > >> the
> > > >> >> > init
> > > >> >> > >> >> time
> > > >> >> > >> >> to equal the valid time. Feel free to change that
> however
> > > you
> > > >> >> need.
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> (4) Add the grid definition info. And I figured
this out
> > by
> > > >> >> running
> > > >> >> > >> >> pcp_combine (see below) and then looking at the
NetCDF
> > global
> > > >> >> > >> attributes.
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> pcp_combine -add NR2006.pl.1by1.6522.2006012806.nc
> > > >> 'name="TP_6h";
> > > >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> (5) Along the way, I ran this command to make sure
my
> > python
> > > >> >> script
> > > >> >> > >> >> actually runs on the command line:
> > > >> >> > >> >>
> > > >> >> > >> >> python read_ECMWF_NC_2D.py
> > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> TP_6h
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> Please let me know how this goes for you.
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >> Thanks,
> > > >> >> > >> >> John
> > > >> >> > >> >>
> > > >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley Gotway
<
> > > >> >> johnhg at ucar.edu>
> > > >> >> > >> >> wrote:
> > > >> >> > >> >>
> > > >> >> > >> >> > Andrew,
> > > >> >> > >> >> >
> > > >> >> > >> >> > OK, question for you. Is this ECMWF actually
model
> > output
> > > >> with
> > > >> >> a
> > > >> >> > >> 6522
> > > >> >> > >> >> > forecast hour? Or should this be interpreted as
an
> > > analysis
> > > >> >> that's
> > > >> >> > >> >> valid
> > > >> >> > >> >> > at 01/28/2006 at 00Z?
> > > >> >> > >> >> >
> > > >> >> > >> >> > That'll determine how best to encode the timing
> > > information.
> > > >> >> > >> >> >
> > > >> >> > >> >> > Thanks,
> > > >> >> > >> >> > John
> > > >> >> > >> >> >
> > > >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren -
NOAA
> > Affiliate
> > > >> via
> > > >> >> RT
> > > >> >> > <
> > > >> >> > >> >> > met_help at ucar.edu> wrote:
> > > >> >> > >> >> >
> > > >> >> > >> >> >>
> > > >> >> > >> >> >> <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > >> >> >
> > > >> >> > >> >> >>
> > > >> >> > >> >> >> John,
> > > >> >> > >> >> >> That is great. Yes let's go the python route,
that
> > seems a
> > > >> lot
> > > >> >> > more
> > > >> >> > >> >> >> straightforward. Please let me know how I may
proceed,
> > and
> > > >> >> thank
> > > >> >> > >> you in
> > > >> >> > >> >> >> advance for your help on this!!
> > > >> >> > >> >> >>
> > > >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley
Gotway via
> > RT <
> > > >> >> > >> >> >> met_help at ucar.edu>
> > > >> >> > >> >> >> wrote:
> > > >> >> > >> >> >>
> > > >> >> > >> >> >> > Hello Andrew,
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > I see you have a question about reading
NetCDF data
> > into
> > > >> MET.
> > > >> >> > >> >> Judging
> > > >> >> > >> >> >> by
> > > >> >> > >> >> >> > the path you sent, I figured that this data
lives on
> > > >> theia.
> > > >> >> So
> > > >> >> > I
> > > >> >> > >> >> >> logged on
> > > >> >> > >> >> >> > and took a look.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > Looks like this ECMWF file doesn't exactly
fall into
> > any
> > > >> of
> > > >> >> the
> > > >> >> > >> >> NetCDF
> > > >> >> > >> >> >> > categories that MET currently supports. The
closest
> > is
> > > >> >> > >> CF-compliant
> > > >> >> > >> >> >> > NetCDF. So let's tell MET to interpret it as
such
> > using
> > > >> the
> > > >> >> > >> >> "file_type
> > > >> >> > >> >> >> =
> > > >> >> > >> >> >> > NETCDF_NCCF;" option.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > When I'm trying to figure out if MET can read
> gridded
> > > >> data, I
> > > >> >> > >> usually
> > > >> >> > >> >> >> use
> > > >> >> > >> >> >> > the plot_data_plane tool.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > Here's the commands I ran on theia:
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > module use /contrib/modulefiles
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > module load intel met/8.0
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > plot_data_plane
> > > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps \
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> > > file_type=NETCDF_NCCF;'
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > This command ran and produced the attached
plot, but
> > it
> > > >> did
> > > >> >> also
> > > >> >> > >> >> >> produce a
> > > >> >> > >> >> >> > warning about the valid time:
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not
determine the
> > > valid
> > > >> >> time,
> > > >> >> > >> >> using
> > > >> >> > >> >> >> 0.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > So MET isn't getting the timing info correct.
What
> > can
> > > >> we do
> > > >> >> > >> about
> > > >> >> > >> >> >> this?
> > > >> >> > >> >> >> > One option would be reformatting these NetCDF
files
> > > into a
> > > >> >> > format
> > > >> >> > >> >> that
> > > >> >> > >> >> >> MET
> > > >> >> > >> >> >> > can actually read... either CF-compliant
NetCDF or
> the
> > > MET
> > > >> >> > NetCDF
> > > >> >> > >> >> >> format.
> > > >> >> > >> >> >> > However, that's pretty cumbersome. I would
instead
> > > >> recommend
> > > >> >> > >> that we
> > > >> >> > >> >> >> take
> > > >> >> > >> >> >> > advantage of a new feature in met-8.0
(released last
> > > >> >> Friday). We
> > > >> >> > >> can
> > > >> >> > >> >> >> write
> > > >> >> > >> >> >> > a Python script to read data from these
files,
> define
> > > the
> > > >> >> > >> metadata,
> > > >> >> > >> >> and
> > > >> >> > >> >> >> > then pass that to MET via memory.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > If the timing information doesn't matter for
some
> > > reason,
> > > >> >> then
> > > >> >> > you
> > > >> >> > >> >> could
> > > >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;"
option. But
> > if
> > > it
> > > >> >> does
> > > >> >> > >> >> matter,
> > > >> >> > >> >> >> and
> > > >> >> > >> >> >> > presumably it does, then the Python route
should
> work.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > Let me know how you'd like to proceed.
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > Thanks,
> > > >> >> > >> >> >> > John
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren -
NOAA
> > > >> Affiliate
> > > >> >> via
> > > >> >> > >> RT <
> > > >> >> > >> >> >> > met_help at ucar.edu> wrote:
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266 was
acted
> > > upon.
> > > >> >> > >> >> >> > > Transaction: Ticket created by
> andrew.kren at noaa.gov
> > > >> >> > >> >> >> > > Queue: met_help
> > > >> >> > >> >> >> > > Subject: Question about MET
pcp_combine
> > > >> >> > >> >> >> > > Owner: Nobody
> > > >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > > >> >> > >> >> >> > > Status: new
> > > >> >> > >> >> >> > > Ticket <URL:
> > > >> >> > >> >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > Dear MET help,
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > I have a question on the MET pcp_combine
add
> tool. I
> > > >> have
> > > >> >> > model
> > > >> >> > >> >> data
> > > >> >> > >> >> >> that
> > > >> >> > >> >> >> > > is considered my truth from an ECMWF Nature
run
> > > >> (long-run
> > > >> >> > >> >> simulation
> > > >> >> > >> >> >> used
> > > >> >> > >> >> >> > > to represent the true atmosphere). I want
to
> verify
> > > how
> > > >> >> well
> > > >> >> > my
> > > >> >> > >> >> >> forecast
> > > >> >> > >> >> >> > > output, in this case from the GFS model, is
> compared
> > > to
> > > >> the
> > > >> >> > >> Nature
> > > >> >> > >> >> >> run.
> > > >> >> > >> >> >> > The
> > > >> >> > >> >> >> > > ECMWF nature run data is 6-hourly data,
which
> > includes
> > > >> 6-hr
> > > >> >> > >> >> >> precipitation
> > > >> >> > >> >> >> > > and other fields in its netcdf files. my
GFS data
> is
> > > >> grib
> > > >> >> > >> format. I
> > > >> >> > >> >> >> > already
> > > >> >> > >> >> >> > > can convert the GFS data to 24-hourly
> accumulations
> > > >> using
> > > >> >> my
> > > >> >> > >> prior
> > > >> >> > >> >> >> > scripts
> > > >> >> > >> >> >> > > and with pcp_combine. I guess my issue is
in the
> > > >> ability to
> > > >> >> > read
> > > >> >> > >> >> the
> > > >> >> > >> >> >> > Nature
> > > >> >> > >> >> >> > > run netcdf files. I try the pcp_combine
tool and I
> > get
> > > >> this
> > > >> >> > >> error
> > > >> >> > >> >> when
> > > >> >> > >> >> >> > > trying to create a 24-hr accumulation.
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > (0) Current cycle: 2006012800
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > DEBUG 2: Performing addition command for 4
files.
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > DEBUG 1: Reading input file:
> > > >> >> > >> >> >> > >
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > DEBUG 4:
> > Met2dDataFileFactory::new_met_2d_data_file()
> > > ->
> > > >> >> > created
> > > >> >> > >> >> new
> > > >> >> > >> >> >> > > Met2dDataFile object of type
"FileType_None".
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > ERROR :
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > ERROR : get_field() -> can't open data
file
> > > >> >> > >> >> >> > >
"/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > ERROR :
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > I assume MET is having issues reading the
variable
> > in
> > > >> the
> > > >> >> > file,
> > > >> >> > >> >> which
> > > >> >> > >> >> >> is
> > > >> >> > >> >> >> > > denoted at TP_6h for 6-hr total
precipitation. Is
> > > there
> > > >> a
> > > >> >> > >> >> workaround
> > > >> >> > >> >> >> so
> > > >> >> > >> >> >> > > that it can successfully process this?
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > Thank you kindly,
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > > --
> > > >> >> > >> >> >> > > Andrew Kren
> > > >> >> > >> >> >> > > Assistant Scientist
> > > >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > > >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> > > >> >> > >> >> >> > > Boulder, Colorado, USA
> > > >> >> > >> >> >> > > Office: 303-497-2715
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> > >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >> >
> > > >> >> > >> >> >>
> > > >> >> > >> >> >> --
> > > >> >> > >> >> >> Andrew Kren
> > > >> >> > >> >> >> Assistant Scientist
> > > >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > > >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > > >> >> > >> >> >> Boulder, Colorado, USA
> > > >> >> > >> >> >> Office: 303-497-2715
> > > >> >> > >> >> >>
> > > >> >> > >> >> >>
> > > >> >> > >> >>
> > > >> >> > >> >>
> > > >> >> > >> >
> > > >> >> > >> > --
> > > >> >> > >> > Andrew Kren
> > > >> >> > >> > Assistant Scientist
> > > >> >> > >> > University of Miami CIMAS - NOAA/AOML
> > > >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> > > >> >> > >> > Boulder, Colorado, USA
> > > >> >> > >> > Office: 303-497-2715
> > > >> >> > >> >
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >> --
> > > >> >> > >> Andrew Kren
> > > >> >> > >> Assistant Scientist
> > > >> >> > >> University of Miami CIMAS - NOAA/AOML
> > > >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> > > >> >> > >> Boulder, Colorado, USA
> > > >> >> > >> Office: 303-497-2715
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >> --
> > > >> >> Andrew Kren
> > > >> >> Assistant Scientist
> > > >> >> University of Miami CIMAS - NOAA/AOML
> > > >> >> Duty Station: NCAR Foothills - FL2-3058
> > > >> >> Boulder, Colorado, USA
> > > >> >> Office: 303-497-2715
> > > >> >>
> > > >> >>
> > > >>
> > > >>
> > > >
> > > > --
> > > > Andrew Kren
> > > > Assistant Scientist
> > > > University of Miami CIMAS - NOAA/AOML
> > > > Duty Station: NCAR Foothills - FL2-3058
> > > > Boulder, Colorado, USA
> > > > Office: 303-497-2715
> > > >
> > >
> > >
> > > --
> > > Andrew Kren
> > > Assistant Scientist
> > > University of Miami CIMAS - NOAA/AOML
> > > Duty Station: NCAR Foothills - FL2-3058
> > > Boulder, Colorado, USA
> > > Office: 303-497-2715
> > >
> > >
> >
> >
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Tue Oct 09 09:58:11 2018
John,
That worked! Thanks.
So regarding what I want to do know with the files. I want to
accumulate
the precipitation over selected periods using the pcp_combine. If I
want to
use MET in this instance do I follow the same procedure, i.e. say
pcp_combine then name="/python script...?
Thanks,
On Tue, Oct 9, 2018 at 9:31 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
> I have a good guess as to what's happening here and this is easy to
fix.
> Let's say you have a python script named "myscript.py". To run it
on the
> command line, you run:
> python myscript.py
>
> But when running it via a MET tool, remove the leading "python"
command,
> like this:
> plot_data_plane PYTHON_NUMPY plot.ps 'name="myscript.py";'
>
> Notice that I used 'name="myscript.py";' and NOT 'name="python
> myscript.py";'.
>
> Please make sure that's how you're specifying it and try again.
>
> Thanks,
> John
>
> On Tue, Oct 9, 2018 at 8:35 AM Andrew Kren - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >
> > John,
> > Ok I tried this morning putting that at the top of my python
script,
> > read_ECMWF_NC_2D.py. I then ran my shell script as usual, but it
still
> gave
> > me this error:
> >
> > The following have been reloaded with a version change:
> >
> > 1) intel/15.0.0 => intel/14.0.2
> >
> >
> > /contrib/anaconda/anaconda2/4.4.0/bin/python
> >
> > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > NR2006.pl.1by1.6522.2006012806.nc'
> >
> > Variable Name : 'TP_6h'
> >
> > Data Shape: (181, 360)
> >
> > Data Type: dtype('float64')
> >
> > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> 'Surface',
> > 'units': 'mm', 'accum': '06'}
> >
> > DEBUG 1: Opening data file: PYTHON_NUMPY
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Python_Numpy".
> >
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> > type "FileType_Python_Numpy".
> >
> > ERROR :
> >
> > ERROR : python_dataplane() -> an error occurred importing module
> > "./python.py"
> >
> > ERROR :
> >
> > ImportError: No module named python
> >
> > /contrib/anaconda/anaconda2/4.4.0/bin/python
> >
> >
> > Do I need to add it to the shell script as well?
> >
> > On Mon, Oct 8, 2018 at 1:22 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Andrew,
> > >
> > > I did some more hunting around on this issue. Please try adding
the
> > > following lines to the top of the python script you're running:
> > >
> > > import sys
> > > sys.path.insert(0,
> > > '/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-packages')
> > > sys.path.insert(0, '/contrib/anaconda/anaconda2/4.4.0/lib/site-
python')
> > >
> > > The problem is that when Python is called by MET, the "site-
packages"
> > > settings differ from when invoking the python interpreter on the
> command
> > > line.
> > >
> > > On the command line, we're pointing to the right site-packages:
> > > python -c "import site; print(site.getsitepackages())"
> > > ['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-
packages',
> > > '/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
> > >
> > > When called via MET, it points here:
> > > ['/usr/lib64/python2.7/site-packages',
> > '/usr/lib/python2.7/site-packages',
> > > '/usr/lib/site-python']
> > >
> > > We need to do some more digging to see if we can recompile MET
to point
> > to
> > > the *correct* site packages.
> > >
> > > But please confirm that this change to the script enable MET to
work
> for
> > > you.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
> > > >
> > > > Is there a workaround in the meantime John? Or were you going
to
> check
> > on
> > > > that first? I got your script and confirmed your finding.
> > > >
> > > > On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate <
> > > > andrew.kren at noaa.gov> wrote:
> > > >
> > > > > Thanks John.
> > > > >
> > > > > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Andrew,
> > > > >>
> > > > >> OK, this script runs successfully on theia with python from
my
> path:
> > > > >>
> > > > >> > which python
> > > > >>
> > > > >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > > >>
> > > > >>
> > > > >> But when I run the script using /bin/python, I get the same
error
> as
> > > > from
> > > > >> MET:
> > > > >>
> > > > >>
> > > > >> Traceback (most recent call last):
> > > > >>
> > > > >> File
> > > >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > >> line 3, in <module>
> > > > >>
> > > > >> from netCDF4 import Dataset
> > > > >>
> > > > >> ImportError: No module named netCDF4
> > > > >>
> > > > >> So it would appear that MET is not calling the version of
python
> > that
> > > > >> lives
> > > > >> in my path. I need to check with another developer to
figure out
> if
> > > > >> there's a way to tell MET what version of python to run.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > Andrew,
> > > > >> >
> > > > >> > Hmmm, we're close. Try running this one:
> > > > >> >
> > > > >> >
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > > > >> > create_24_accum_met_nr.sh
> > > > >> >
> > > > >> >
> > > > >> > The output is listed below. The punchline is... when I
run the
> > > python
> > > > >> > script on the command line everything is fine. When I
run it
> > > through
> > > > >> > plot_data_plane, I get a runtime error about netCDF4.
> > > > >> >
> > > > >> >
> > > > >> > I can confirm that this works fine on my project machine
at
> NCAR.
> > > So
> > > > I
> > > > >> > suspect it's an environment issue.
> > > > >> >
> > > > >> >
> > > > >> > John
> > > > >> >
> > > > >> >
> > > > >> > [John.H.Gotway at tfe05 kren_data_20181003]$
> > > ./create_24_accum_met_nr.sh
> > > > >> >
> > > > >> > Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > >> > NR2006.pl.1by1.6522.2006012806.nc'
> > > > >> >
> > > > >> > Variable Name : 'TP_6h'
> > > > >> >
> > > > >> > Data Shape: (181, 360)
> > > > >> >
> > > > >> > Data Type: dtype('float64')
> > > > >> >
> > > > >> > Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > > 'valid':
> > > > >> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > 'name':
> > > > >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon': 1.0,
> > > > 'type':
> > > > >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > > > >> 'Surface',
> > > > >> > 'units': 'mm', 'accum': '06'}
> > > > >> >
> > > > >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> > > > >> >
> > > > >> > ERROR :
> > > > >> >
> > > > >> > ERROR : python_dataplane() -> an error occurred
importing
> module
> > > > >> >
> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> > > > >> >
> > > > >> > ERROR :
> > > > >> >
> > > > >> > Traceback (most recent call last):
> > > > >> >
> > > > >> > File
> > > > >>
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > >> > line 3, in <module>
> > > > >> >
> > > > >> > from netCDF4 import Dataset
> > > > >> >
> > > > >> > ImportError: No module named netCDF4
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA
Affiliate via
> > RT
> > > <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> >>
> > > > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> >
> > > > >> >>
> > > > >> >> John,
> > > > >> >> I ran this script,
> > > > >> >>
> > >
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> > > > >> and
> > > > >> >> it gave me this, presumably asking for more information
to run
> > the
> > > > met
> > > > >> >> command.
> > > > >> >>
> > > > >> >> The following have been reloaded with a version change:
> > > > >> >>
> > > > >> >> 1) intel/15.0.0 => intel/14.0.2
> > > > >> >>
> > > > >> >>
> > > > >> >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > >> >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > >> >>
> > > > >> >> Variable Name : 'TP_6h'
> > > > >> >>
> > > > >> >> Data Shape: (181, 360)
> > > > >> >>
> > > > >> >> Data Type: dtype('float64')
> > > > >> >>
> > > > >> >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > > > 'valid':
> > > > >> >> '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > > 'name':
> > > > >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon': 1.0,
> > > > 'type':
> > > > >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > > > >> 'Surface',
> > > > >> >> 'units': 'mm', 'accum': '06'}
> > > > >> >>
> > > > >> >>
> > > > >> >> *** Model Evaluation Tools (METV8.0) ***
> > > > >> >>
> > > > >> >>
> > > > >> >> Usage: plot_data_plane
> > > > >> >>
> > > > >> >> input_filename
> > > > >> >>
> > > > >> >> output_filename
> > > > >> >>
> > > > >> >> field_string
> > > > >> >>
> > > > >> >> [-color_table color_table_name]
> > > > >> >>
> > > > >> >> [-plot_range min max]
> > > > >> >>
> > > > >> >> [-title title_string]
> > > > >> >>
> > > > >> >> [-log file]
> > > > >> >>
> > > > >> >> [-v level]
> > > > >> >>
> > > > >> >>
> > > > >> >> where "input_filename" is the name of a gridded data
file to
> be
> > > > >> plotted
> > > > >> >> (required).
> > > > >> >>
> > > > >> >> "output_filename" is the name of the output PostScript
file to
> be
> > > > >> written
> > > > >> >> (required).
> > > > >> >>
> > > > >> >> "field_string" defines the data to be plotted from the
input
> file
> > > > >> >> (required).
> > > > >> >>
> > > > >> >> "-color_table color_table_name" overrides the default
color
> table
> > > > >> >> ("colortables/met_default.ctable") (optional).
> > > > >> >>
> > > > >> >> "-plot_range min max" defines the range of the data to
be
> plotted
> > > > >> >> (optional).
> > > > >> >>
> > > > >> >> "-title title_string" specifies the plot title string
> (optional).
> > > > >> >>
> > > > >> >> "-log file" outputs log messages to the specified file
> > (optional).
> > > > >> >>
> > > > >> >> "-v level" overrides the default level of logging (2)
> (optional).
> > > > >> >>
> > > > >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via
RT <
> > > > >> >> met_help at ucar.edu>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Andrew,
> > > > >> >> >
> > > > >> >> > Sorry, just noticed a typo after I hit send. I had a
> > > double-equals
> > > > >> (==)
> > > > >> >> > that should be a single equal (=):
> > > > >> >> >
> > > > >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > >> >> >
> > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps \
> > > > >> >> >
> > > > >> >> > 'name="python
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > > > >> >> >
> > > > >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <
> > > > johnhg at ucar.edu>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> > > Great, so running the python script stand-alone read
the
> data
> > > > well.
> > > > >> >> > >
> > > > >> >> > > The next step is telling MET to call this script.
Here's
> > that
> > > > >> simple
> > > > >> >> > > example of running plot_data_plane:
> > > > >> >> > >
> > > > >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > >> >> > >
> > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps
\
> > > > >> >> > >
> > > > >> >> > > 'name=="python
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > Once this works, look at the output to confirm that
MET is
> > > > putting
> > > > >> >> your
> > > > >> >> > > data on the right spot on the earth.
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > Presumably it does... so now you're ready to start
using
> this
> > > to
> > > > >> run
> > > > >> >> > > pcp_combine or Grid-Stat or MODE.
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > Make sense?
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > John
> > > > >> >> > >
> > > > >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA
> Affiliate
> > > via
> > > > >> RT <
> > > > >> >> > > met_help at ucar.edu> wrote:
> > > > >> >> > >
> > > > >> >> > >>
> > > > >> >> > >> <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > >
> > > > >> >> > >>
> > > > >> >> > >> John,
> > > > >> >> > >> I got this output when I ran your python script in
a shell
> > > > script:
> > > > >> >> > >>
> > > > >> >> > >> The following have been reloaded with a version
change:
> > > > >> >> > >>
> > > > >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >> Input File:
> > '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > >> >> > >>
> > > > >> >> > >> Variable Name : 'TP_6h'
> > > > >> >> > >>
> > > > >> >> > >> Data Shape: (181, 360)
> > > > >> >> > >>
> > > > >> >> > >> Data Type: dtype('float64')
> > > > >> >> > >>
> > > > >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
> > '20060128_060000',
> > > > >> >> 'valid':
> > > > >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll':
> 0.0,
> > > > >> 'name':
> > > > >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> > > 1.0,
> > > > >> >> 'type':
> > > > >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead':
'00',
> > > 'level':
> > > > >> >> > 'Surface',
> > > > >> >> > >> 'units': 'mm', 'accum': '06'}
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >> Looks like it worked.
> > > > >> >> > >>
> > > > >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren - NOAA
> > Affiliate <
> > > > >> >> > >> andrew.kren at noaa.gov> wrote:
> > > > >> >> > >>
> > > > >> >> > >> > John,
> > > > >> >> > >> > I believe the 6522 and other numbers (which
increase
> with
> > > each
> > > > >> >> date)
> > > > >> >> > >> were
> > > > >> >> > >> > used as place holders for the filename. The
actual dates
> > can
> > > > be
> > > > >> >> read
> > > > >> >> > as
> > > > >> >> > >> an
> > > > >> >> > >> > analysis that's valid at the time stamp, such as
> > 01/28/2006
> > > at
> > > > >> 06Z?
> > > > >> >> > Thus
> > > > >> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> would
> > > correspond
> > > > to
> > > > >> >> the
> > > > >> >> > >> > analysis valid at 2006012806, does that help?
I'll try
> the
> > > > >> script.
> > > > >> >> > >> >
> > > > >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley
Gotway via
> RT
> > <
> > > > >> >> > >> > met_help at ucar.edu> wrote:
> > > > >> >> > >> >
> > > > >> >> > >> >> Andrew,
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> OK, please try running this command on theia.
I've
> > > attached
> > > > >> the
> > > > >> >> > >> resulting
> > > > >> >> > >> >> image:
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> module use /contrib/modulefiles
> > > > >> >> > >> >>
> > > > >> >> > >> >> module load intel met/8.0
> > > > >> >> > >> >>
> > > > >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > >> >> > >> >>
> > > > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h.ps \
> > > > >> >> > >> >>
> > > > >> >> > >> >> 'name=="python
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >>
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > >> >> > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> This python script (also attached) reads your
data and
> > > hands
> > > > it
> > > > >> >> off
> > > > >> >> > to
> > > > >> >> > >> >> MET.
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> Here's the steps I followed to do this:
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> (1) Copy over a sample python script from MET:
> > > > >> >> > >> >>
> > > > >> >> > >> >> cp
> /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > > > >> >> > >> >> read_ECMWF_NC_2D.py
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> (2) Edit it to read the input netcdf file
(argument 1)
> > and
> > > > the
> > > > >> >> > variable
> > > > >> >> > >> >> name (argument 2).
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> (3) Add code to parse out the timing info and
define it
> > in
> > > > the
> > > > >> >> "attr"
> > > > >> >> > >> >> dictionary.
> > > > >> >> > >> >>
> > > > >> >> > >> >> NOTE that I assumed that this is an analysis...
so I
> just
> > > set
> > > > >> the
> > > > >> >> > init
> > > > >> >> > >> >> time
> > > > >> >> > >> >> to equal the valid time. Feel free to change
that
> > however
> > > > you
> > > > >> >> need.
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> (4) Add the grid definition info. And I figured
this
> out
> > > by
> > > > >> >> running
> > > > >> >> > >> >> pcp_combine (see below) and then looking at the
NetCDF
> > > global
> > > > >> >> > >> attributes.
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> pcp_combine -add
NR2006.pl.1by1.6522.2006012806.nc
> > > > >> 'name="TP_6h";
> > > > >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;' TP_6h.nc
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> (5) Along the way, I ran this command to make
sure my
> > > python
> > > > >> >> script
> > > > >> >> > >> >> actually runs on the command line:
> > > > >> >> > >> >>
> > > > >> >> > >> >> python read_ECMWF_NC_2D.py
> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> TP_6h
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> Please let me know how this goes for you.
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >> Thanks,
> > > > >> >> > >> >> John
> > > > >> >> > >> >>
> > > > >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley
Gotway <
> > > > >> >> johnhg at ucar.edu>
> > > > >> >> > >> >> wrote:
> > > > >> >> > >> >>
> > > > >> >> > >> >> > Andrew,
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > OK, question for you. Is this ECMWF actually
model
> > > output
> > > > >> with
> > > > >> >> a
> > > > >> >> > >> 6522
> > > > >> >> > >> >> > forecast hour? Or should this be interpreted
as an
> > > > analysis
> > > > >> >> that's
> > > > >> >> > >> >> valid
> > > > >> >> > >> >> > at 01/28/2006 at 00Z?
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > That'll determine how best to encode the
timing
> > > > information.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > Thanks,
> > > > >> >> > >> >> > John
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren -
NOAA
> > > Affiliate
> > > > >> via
> > > > >> >> RT
> > > > >> >> > <
> > > > >> >> > >> >> > met_help at ucar.edu> wrote:
> > > > >> >> > >> >> >
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >> <URL:
> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > >> >> >
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >> John,
> > > > >> >> > >> >> >> That is great. Yes let's go the python route,
that
> > > seems a
> > > > >> lot
> > > > >> >> > more
> > > > >> >> > >> >> >> straightforward. Please let me know how I may
> proceed,
> > > and
> > > > >> >> thank
> > > > >> >> > >> you in
> > > > >> >> > >> >> >> advance for your help on this!!
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley
Gotway
> via
> > > RT <
> > > > >> >> > >> >> >> met_help at ucar.edu>
> > > > >> >> > >> >> >> wrote:
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >> > Hello Andrew,
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > I see you have a question about reading
NetCDF
> data
> > > into
> > > > >> MET.
> > > > >> >> > >> >> Judging
> > > > >> >> > >> >> >> by
> > > > >> >> > >> >> >> > the path you sent, I figured that this data
lives
> on
> > > > >> theia.
> > > > >> >> So
> > > > >> >> > I
> > > > >> >> > >> >> >> logged on
> > > > >> >> > >> >> >> > and took a look.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > Looks like this ECMWF file doesn't exactly
fall
> into
> > > any
> > > > >> of
> > > > >> >> the
> > > > >> >> > >> >> NetCDF
> > > > >> >> > >> >> >> > categories that MET currently supports.
The
> closest
> > > is
> > > > >> >> > >> CF-compliant
> > > > >> >> > >> >> >> > NetCDF. So let's tell MET to interpret it
as such
> > > using
> > > > >> the
> > > > >> >> > >> >> "file_type
> > > > >> >> > >> >> >> =
> > > > >> >> > >> >> >> > NETCDF_NCCF;" option.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > When I'm trying to figure out if MET can
read
> > gridded
> > > > >> data, I
> > > > >> >> > >> usually
> > > > >> >> > >> >> >> use
> > > > >> >> > >> >> >> > the plot_data_plane tool.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > Here's the commands I ran on theia:
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > module use /contrib/modulefiles
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > module load intel met/8.0
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > plot_data_plane
> > > > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc TP_6h.ps
\
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> > > > file_type=NETCDF_NCCF;'
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > This command ran and produced the attached
plot,
> but
> > > it
> > > > >> did
> > > > >> >> also
> > > > >> >> > >> >> >> produce a
> > > > >> >> > >> >> >> > warning about the valid time:
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not
determine
> the
> > > > valid
> > > > >> >> time,
> > > > >> >> > >> >> using
> > > > >> >> > >> >> >> 0.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > So MET isn't getting the timing info
correct.
> What
> > > can
> > > > >> we do
> > > > >> >> > >> about
> > > > >> >> > >> >> >> this?
> > > > >> >> > >> >> >> > One option would be reformatting these
NetCDF
> files
> > > > into a
> > > > >> >> > format
> > > > >> >> > >> >> that
> > > > >> >> > >> >> >> MET
> > > > >> >> > >> >> >> > can actually read... either CF-compliant
NetCDF or
> > the
> > > > MET
> > > > >> >> > NetCDF
> > > > >> >> > >> >> >> format.
> > > > >> >> > >> >> >> > However, that's pretty cumbersome. I would
> instead
> > > > >> recommend
> > > > >> >> > >> that we
> > > > >> >> > >> >> >> take
> > > > >> >> > >> >> >> > advantage of a new feature in met-8.0
(released
> last
> > > > >> >> Friday). We
> > > > >> >> > >> can
> > > > >> >> > >> >> >> write
> > > > >> >> > >> >> >> > a Python script to read data from these
files,
> > define
> > > > the
> > > > >> >> > >> metadata,
> > > > >> >> > >> >> and
> > > > >> >> > >> >> >> > then pass that to MET via memory.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > If the timing information doesn't matter
for some
> > > > reason,
> > > > >> >> then
> > > > >> >> > you
> > > > >> >> > >> >> could
> > > > >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;"
option.
> But
> > > if
> > > > it
> > > > >> >> does
> > > > >> >> > >> >> matter,
> > > > >> >> > >> >> >> and
> > > > >> >> > >> >> >> > presumably it does, then the Python route
should
> > work.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > Let me know how you'd like to proceed.
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > Thanks,
> > > > >> >> > >> >> >> > John
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew Kren
- NOAA
> > > > >> Affiliate
> > > > >> >> via
> > > > >> >> > >> RT <
> > > > >> >> > >> >> >> > met_help at ucar.edu> wrote:
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266
was
> acted
> > > > upon.
> > > > >> >> > >> >> >> > > Transaction: Ticket created by
> > andrew.kren at noaa.gov
> > > > >> >> > >> >> >> > > Queue: met_help
> > > > >> >> > >> >> >> > > Subject: Question about MET
pcp_combine
> > > > >> >> > >> >> >> > > Owner: Nobody
> > > > >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > > > >> >> > >> >> >> > > Status: new
> > > > >> >> > >> >> >> > > Ticket <URL:
> > > > >> >> > >> >>
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > Dear MET help,
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > I have a question on the MET pcp_combine
add
> > tool. I
> > > > >> have
> > > > >> >> > model
> > > > >> >> > >> >> data
> > > > >> >> > >> >> >> that
> > > > >> >> > >> >> >> > > is considered my truth from an ECMWF
Nature run
> > > > >> (long-run
> > > > >> >> > >> >> simulation
> > > > >> >> > >> >> >> used
> > > > >> >> > >> >> >> > > to represent the true atmosphere). I want
to
> > verify
> > > > how
> > > > >> >> well
> > > > >> >> > my
> > > > >> >> > >> >> >> forecast
> > > > >> >> > >> >> >> > > output, in this case from the GFS model,
is
> > compared
> > > > to
> > > > >> the
> > > > >> >> > >> Nature
> > > > >> >> > >> >> >> run.
> > > > >> >> > >> >> >> > The
> > > > >> >> > >> >> >> > > ECMWF nature run data is 6-hourly data,
which
> > > includes
> > > > >> 6-hr
> > > > >> >> > >> >> >> precipitation
> > > > >> >> > >> >> >> > > and other fields in its netcdf files. my
GFS
> data
> > is
> > > > >> grib
> > > > >> >> > >> format. I
> > > > >> >> > >> >> >> > already
> > > > >> >> > >> >> >> > > can convert the GFS data to 24-hourly
> > accumulations
> > > > >> using
> > > > >> >> my
> > > > >> >> > >> prior
> > > > >> >> > >> >> >> > scripts
> > > > >> >> > >> >> >> > > and with pcp_combine. I guess my issue is
in the
> > > > >> ability to
> > > > >> >> > read
> > > > >> >> > >> >> the
> > > > >> >> > >> >> >> > Nature
> > > > >> >> > >> >> >> > > run netcdf files. I try the pcp_combine
tool
> and I
> > > get
> > > > >> this
> > > > >> >> > >> error
> > > > >> >> > >> >> when
> > > > >> >> > >> >> >> > > trying to create a 24-hr accumulation.
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > (0) Current cycle: 2006012800
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > DEBUG 2: Performing addition command for
4
> files.
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > DEBUG 1: Reading input file:
> > > > >> >> > >> >> >> > >
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > DEBUG 4:
> > > Met2dDataFileFactory::new_met_2d_data_file()
> > > > ->
> > > > >> >> > created
> > > > >> >> > >> >> new
> > > > >> >> > >> >> >> > > Met2dDataFile object of type
"FileType_None".
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > ERROR :
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > ERROR : get_field() -> can't open data
file
> > > > >> >> > >> >> >> > >
"/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > ERROR :
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > I assume MET is having issues reading the
> variable
> > > in
> > > > >> the
> > > > >> >> > file,
> > > > >> >> > >> >> which
> > > > >> >> > >> >> >> is
> > > > >> >> > >> >> >> > > denoted at TP_6h for 6-hr total
precipitation.
> Is
> > > > there
> > > > >> a
> > > > >> >> > >> >> workaround
> > > > >> >> > >> >> >> so
> > > > >> >> > >> >> >> > > that it can successfully process this?
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > Thank you kindly,
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > > --
> > > > >> >> > >> >> >> > > Andrew Kren
> > > > >> >> > >> >> >> > > Assistant Scientist
> > > > >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > > > >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> > > > >> >> > >> >> >> > > Boulder, Colorado, USA
> > > > >> >> > >> >> >> > > Office: 303-497-2715
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> > >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >> >
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >> --
> > > > >> >> > >> >> >> Andrew Kren
> > > > >> >> > >> >> >> Assistant Scientist
> > > > >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > > > >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > >> >> > >> >> >> Boulder, Colorado, USA
> > > > >> >> > >> >> >> Office: 303-497-2715
> > > > >> >> > >> >> >>
> > > > >> >> > >> >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >>
> > > > >> >> > >> >
> > > > >> >> > >> > --
> > > > >> >> > >> > Andrew Kren
> > > > >> >> > >> > Assistant Scientist
> > > > >> >> > >> > University of Miami CIMAS - NOAA/AOML
> > > > >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> > > > >> >> > >> > Boulder, Colorado, USA
> > > > >> >> > >> > Office: 303-497-2715
> > > > >> >> > >> >
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >> --
> > > > >> >> > >> Andrew Kren
> > > > >> >> > >> Assistant Scientist
> > > > >> >> > >> University of Miami CIMAS - NOAA/AOML
> > > > >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> > > > >> >> > >> Boulder, Colorado, USA
> > > > >> >> > >> Office: 303-497-2715
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >> --
> > > > >> >> Andrew Kren
> > > > >> >> Assistant Scientist
> > > > >> >> University of Miami CIMAS - NOAA/AOML
> > > > >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > >> >> Boulder, Colorado, USA
> > > > >> >> Office: 303-497-2715
> > > > >> >>
> > > > >> >>
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Andrew Kren
> > > > > Assistant Scientist
> > > > > University of Miami CIMAS - NOAA/AOML
> > > > > Duty Station: NCAR Foothills - FL2-3058
> > > > > Boulder, Colorado, USA
> > > > > Office: 303-497-2715
> > > > >
> > > >
> > > >
> > > > --
> > > > Andrew Kren
> > > > Assistant Scientist
> > > > University of Miami CIMAS - NOAA/AOML
> > > > Duty Station: NCAR Foothills - FL2-3058
> > > > Boulder, Colorado, USA
> > > > Office: 303-497-2715
> > > >
> > > >
> > >
> > >
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
> >
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
Subject: Question about MET pcp_combine
From: John Halley Gotway
Time: Tue Oct 09 11:32:07 2018
Andrew,
Here's an example of running pcp_combine with the python options to
add up
the TP_6h variable across 4 input files.
There are probably more succinct ways of summing up variables across
NetCDF
files using the NCO tools, but this output a NetCDF that is easily
read by
the other MET tools.
Be aware that in my testing, I ran across a problem when you do not
use the
"-name" option to define the output variable name. I'll debug that
and
include a fix in the next set of patches.
But as long as you use the "-name" option, you won't run into that.
Thanks,
John
##########################################
module use /contrib/modulefiles
module load intel met/8.0
pcp_combine -add \
PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.7986.2006033006.nc TP_6h";' \
PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.7992.2006033012.nc TP_6h";' \
PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.7998.2006033018.nc TP_6h";' \
PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
NR2006.pl.1by1.8004.2006033100.nc TP_6h";' \
TP_24h_2006033100.nc -name APCP_24
On Tue, Oct 9, 2018 at 9:58 AM Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
>
> John,
> That worked! Thanks.
> So regarding what I want to do know with the files. I want to
accumulate
> the precipitation over selected periods using the pcp_combine. If I
want to
> use MET in this instance do I follow the same procedure, i.e. say
> pcp_combine then name="/python script...?
>
> Thanks,
>
> On Tue, Oct 9, 2018 at 9:31 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Andrew,
> >
> > I have a good guess as to what's happening here and this is easy
to fix.
> > Let's say you have a python script named "myscript.py". To run it
on the
> > command line, you run:
> > python myscript.py
> >
> > But when running it via a MET tool, remove the leading "python"
command,
> > like this:
> > plot_data_plane PYTHON_NUMPY plot.ps 'name="myscript.py";'
> >
> > Notice that I used 'name="myscript.py";' and NOT 'name="python
> > myscript.py";'.
> >
> > Please make sure that's how you're specifying it and try again.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 9, 2018 at 8:35 AM Andrew Kren - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > >
> > > John,
> > > Ok I tried this morning putting that at the top of my python
script,
> > > read_ECMWF_NC_2D.py. I then ran my shell script as usual, but it
still
> > gave
> > > me this error:
> > >
> > > The following have been reloaded with a version change:
> > >
> > > 1) intel/15.0.0 => intel/14.0.2
> > >
> > >
> > > /contrib/anaconda/anaconda2/4.4.0/bin/python
> > >
> > > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > NR2006.pl.1by1.6522.2006012806.nc'
> > >
> > > Variable Name : 'TP_6h'
> > >
> > > Data Shape: (181, 360)
> > >
> > > Data Type: dtype('float64')
> > >
> > > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
'valid':
> > > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon': 1.0,
'type':
> > > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00', 'level':
> > 'Surface',
> > > 'units': 'mm', 'accum': '06'}
> > >
> > > DEBUG 1: Opening data file: PYTHON_NUMPY
> > >
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_Python_Numpy".
> > >
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of
> > > type "FileType_Python_Numpy".
> > >
> > > ERROR :
> > >
> > > ERROR : python_dataplane() -> an error occurred importing
module
> > > "./python.py"
> > >
> > > ERROR :
> > >
> > > ImportError: No module named python
> > >
> > > /contrib/anaconda/anaconda2/4.4.0/bin/python
> > >
> > >
> > > Do I need to add it to the shell script as well?
> > >
> > > On Mon, Oct 8, 2018 at 1:22 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Andrew,
> > > >
> > > > I did some more hunting around on this issue. Please try
adding the
> > > > following lines to the top of the python script you're
running:
> > > >
> > > > import sys
> > > > sys.path.insert(0,
> > > > '/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-
packages')
> > > > sys.path.insert(0,
> '/contrib/anaconda/anaconda2/4.4.0/lib/site-python')
> > > >
> > > > The problem is that when Python is called by MET, the "site-
packages"
> > > > settings differ from when invoking the python interpreter on
the
> > command
> > > > line.
> > > >
> > > > On the command line, we're pointing to the right site-
packages:
> > > > python -c "import site; print(site.getsitepackages())"
> > > > ['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-
packages',
> > > > '/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
> > > >
> > > > When called via MET, it points here:
> > > > ['/usr/lib64/python2.7/site-packages',
> > > '/usr/lib/python2.7/site-packages',
> > > > '/usr/lib/site-python']
> > > >
> > > > We need to do some more digging to see if we can recompile MET
to
> point
> > > to
> > > > the *correct* site packages.
> > > >
> > > > But please confirm that this change to the script enable MET
to work
> > for
> > > > you.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > > > >
> > > > > Is there a workaround in the meantime John? Or were you
going to
> > check
> > > on
> > > > > that first? I got your script and confirmed your finding.
> > > > >
> > > > > On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA Affiliate
<
> > > > > andrew.kren at noaa.gov> wrote:
> > > > >
> > > > > > Thanks John.
> > > > > >
> > > > > > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >> Andrew,
> > > > > >>
> > > > > >> OK, this script runs successfully on theia with python
from my
> > path:
> > > > > >>
> > > > > >> > which python
> > > > > >>
> > > > > >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > > > >>
> > > > > >>
> > > > > >> But when I run the script using /bin/python, I get the
same
> error
> > as
> > > > > from
> > > > > >> MET:
> > > > > >>
> > > > > >>
> > > > > >> Traceback (most recent call last):
> > > > > >>
> > > > > >> File
> > > > >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > > >> line 3, in <module>
> > > > > >>
> > > > > >> from netCDF4 import Dataset
> > > > > >>
> > > > > >> ImportError: No module named netCDF4
> > > > > >>
> > > > > >> So it would appear that MET is not calling the version of
python
> > > that
> > > > > >> lives
> > > > > >> in my path. I need to check with another developer to
figure
> out
> > if
> > > > > >> there's a way to tell MET what version of python to run.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Andrew,
> > > > > >> >
> > > > > >> > Hmmm, we're close. Try running this one:
> > > > > >> >
> > > > > >> >
> /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > > > > >> > create_24_accum_met_nr.sh
> > > > > >> >
> > > > > >> >
> > > > > >> > The output is listed below. The punchline is... when I
run
> the
> > > > python
> > > > > >> > script on the command line everything is fine. When I
run it
> > > > through
> > > > > >> > plot_data_plane, I get a runtime error about netCDF4.
> > > > > >> >
> > > > > >> >
> > > > > >> > I can confirm that this works fine on my project
machine at
> > NCAR.
> > > > So
> > > > > I
> > > > > >> > suspect it's an environment issue.
> > > > > >> >
> > > > > >> >
> > > > > >> > John
> > > > > >> >
> > > > > >> >
> > > > > >> > [John.H.Gotway at tfe05 kren_data_20181003]$
> > > > ./create_24_accum_met_nr.sh
> > > > > >> >
> > > > > >> > Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > >> > NR2006.pl.1by1.6522.2006012806.nc'
> > > > > >> >
> > > > > >> > Variable Name : 'TP_6h'
> > > > > >> >
> > > > > >> > Data Shape: (181, 360)
> > > > > >> >
> > > > > >> > Data Type: dtype('float64')
> > > > > >> >
> > > > > >> > Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > > > 'valid':
> > > > > >> > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll':
0.0,
> > > 'name':
> > > > > >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> 1.0,
> > > > > 'type':
> > > > > >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
> 'level':
> > > > > >> 'Surface',
> > > > > >> > 'units': 'mm', 'accum': '06'}
> > > > > >> >
> > > > > >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> > > > > >> >
> > > > > >> > ERROR :
> > > > > >> >
> > > > > >> > ERROR : python_dataplane() -> an error occurred
importing
> > module
> > > > > >> >
> > "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> > > > > >> >
> > > > > >> > ERROR :
> > > > > >> >
> > > > > >> > Traceback (most recent call last):
> > > > > >> >
> > > > > >> > File
> > > > > >>
> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > > >> > line 3, in <module>
> > > > > >> >
> > > > > >> > from netCDF4 import Dataset
> > > > > >> >
> > > > > >> > ImportError: No module named netCDF4
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> >>
> > > > > >> >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > >
> > > > > >> >>
> > > > > >> >> John,
> > > > > >> >> I ran this script,
> > > > > >> >>
> > > >
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> > > > > >> and
> > > > > >> >> it gave me this, presumably asking for more
information to
> run
> > > the
> > > > > met
> > > > > >> >> command.
> > > > > >> >>
> > > > > >> >> The following have been reloaded with a version
change:
> > > > > >> >>
> > > > > >> >> 1) intel/15.0.0 => intel/14.0.2
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> Input File:
'/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > >> >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > > >> >>
> > > > > >> >> Variable Name : 'TP_6h'
> > > > > >> >>
> > > > > >> >> Data Shape: (181, 360)
> > > > > >> >>
> > > > > >> >> Data Type: dtype('float64')
> > > > > >> >>
> > > > > >> >> Attributes: {'long_name': 'TP_6h', 'init':
'20060128_060000',
> > > > > 'valid':
> > > > > >> >> '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll': 0.0,
> > > > 'name':
> > > > > >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> 1.0,
> > > > > 'type':
> > > > > >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
> 'level':
> > > > > >> 'Surface',
> > > > > >> >> 'units': 'mm', 'accum': '06'}
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> *** Model Evaluation Tools (METV8.0) ***
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> Usage: plot_data_plane
> > > > > >> >>
> > > > > >> >> input_filename
> > > > > >> >>
> > > > > >> >> output_filename
> > > > > >> >>
> > > > > >> >> field_string
> > > > > >> >>
> > > > > >> >> [-color_table color_table_name]
> > > > > >> >>
> > > > > >> >> [-plot_range min max]
> > > > > >> >>
> > > > > >> >> [-title title_string]
> > > > > >> >>
> > > > > >> >> [-log file]
> > > > > >> >>
> > > > > >> >> [-v level]
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> where "input_filename" is the name of a gridded data
file to
> > be
> > > > > >> plotted
> > > > > >> >> (required).
> > > > > >> >>
> > > > > >> >> "output_filename" is the name of the output PostScript
file
> to
> > be
> > > > > >> written
> > > > > >> >> (required).
> > > > > >> >>
> > > > > >> >> "field_string" defines the data to be plotted from the
input
> > file
> > > > > >> >> (required).
> > > > > >> >>
> > > > > >> >> "-color_table color_table_name" overrides the default
color
> > table
> > > > > >> >> ("colortables/met_default.ctable") (optional).
> > > > > >> >>
> > > > > >> >> "-plot_range min max" defines the range of the data to
be
> > plotted
> > > > > >> >> (optional).
> > > > > >> >>
> > > > > >> >> "-title title_string" specifies the plot title string
> > (optional).
> > > > > >> >>
> > > > > >> >> "-log file" outputs log messages to the specified file
> > > (optional).
> > > > > >> >>
> > > > > >> >> "-v level" overrides the default level of logging (2)
> > (optional).
> > > > > >> >>
> > > > > >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway via
RT <
> > > > > >> >> met_help at ucar.edu>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Andrew,
> > > > > >> >> >
> > > > > >> >> > Sorry, just noticed a typo after I hit send. I had
a
> > > > double-equals
> > > > > >> (==)
> > > > > >> >> > that should be a single equal (=):
> > > > > >> >> >
> > > > > >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > >> >> >
> > > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h.ps
\
> > > > > >> >> >
> > > > > >> >> > 'name="python
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/> TP_6h";'
> > > > > >> >> >
> > > > > >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway <
> > > > > johnhg at ucar.edu>
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> > > Great, so running the python script stand-alone
read the
> > data
> > > > > well.
> > > > > >> >> > >
> > > > > >> >> > > The next step is telling MET to call this script.
Here's
> > > that
> > > > > >> simple
> > > > > >> >> > > example of running plot_data_plane:
> > > > > >> >> > >
> > > > > >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > >> >> > >
> > > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h.ps \
> > > > > >> >> > >
> > > > > >> >> > > 'name=="python
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h";'
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > Once this works, look at the output to confirm
that MET
> is
> > > > > putting
> > > > > >> >> your
> > > > > >> >> > > data on the right spot on the earth.
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > Presumably it does... so now you're ready to start
using
> > this
> > > > to
> > > > > >> run
> > > > > >> >> > > pcp_combine or Grid-Stat or MODE.
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > Make sense?
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > John
> > > > > >> >> > >
> > > > > >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren - NOAA
> > Affiliate
> > > > via
> > > > > >> RT <
> > > > > >> >> > > met_help at ucar.edu> wrote:
> > > > > >> >> > >
> > > > > >> >> > >>
> > > > > >> >> > >> <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > >
> > > > > >> >> > >>
> > > > > >> >> > >> John,
> > > > > >> >> > >> I got this output when I ran your python script
in a
> shell
> > > > > script:
> > > > > >> >> > >>
> > > > > >> >> > >> The following have been reloaded with a version
change:
> > > > > >> >> > >>
> > > > > >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >> Input File:
> > > '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > > >> >> > >>
> > > > > >> >> > >> Variable Name : 'TP_6h'
> > > > > >> >> > >>
> > > > > >> >> > >> Data Shape: (181, 360)
> > > > > >> >> > >>
> > > > > >> >> > >> Data Type: dtype('float64')
> > > > > >> >> > >>
> > > > > >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
> > > '20060128_060000',
> > > > > >> >> 'valid':
> > > > > >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll':
> > 0.0,
> > > > > >> 'name':
> > > > > >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
> 'delta_lon':
> > > > 1.0,
> > > > > >> >> 'type':
> > > > > >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead':
'00',
> > > > 'level':
> > > > > >> >> > 'Surface',
> > > > > >> >> > >> 'units': 'mm', 'accum': '06'}
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >> Looks like it worked.
> > > > > >> >> > >>
> > > > > >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren -
NOAA
> > > Affiliate <
> > > > > >> >> > >> andrew.kren at noaa.gov> wrote:
> > > > > >> >> > >>
> > > > > >> >> > >> > John,
> > > > > >> >> > >> > I believe the 6522 and other numbers (which
increase
> > with
> > > > each
> > > > > >> >> date)
> > > > > >> >> > >> were
> > > > > >> >> > >> > used as place holders for the filename. The
actual
> dates
> > > can
> > > > > be
> > > > > >> >> read
> > > > > >> >> > as
> > > > > >> >> > >> an
> > > > > >> >> > >> > analysis that's valid at the time stamp, such
as
> > > 01/28/2006
> > > > at
> > > > > >> 06Z?
> > > > > >> >> > Thus
> > > > > >> >> > >> > the filename NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/>
would
> > > > correspond
> > > > > to
> > > > > >> >> the
> > > > > >> >> > >> > analysis valid at 2006012806, does that help?
I'll try
> > the
> > > > > >> script.
> > > > > >> >> > >> >
> > > > > >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley
Gotway via
> > RT
> > > <
> > > > > >> >> > >> > met_help at ucar.edu> wrote:
> > > > > >> >> > >> >
> > > > > >> >> > >> >> Andrew,
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> OK, please try running this command on theia.
I've
> > > > attached
> > > > > >> the
> > > > > >> >> > >> resulting
> > > > > >> >> > >> >> image:
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> module use /contrib/modulefiles
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> module load intel met/8.0
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/>
> TP_6h.ps \
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> 'name=="python
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >>
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > >> >> > >> >>
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> This python script (also attached) reads your
data
> and
> > > > hands
> > > > > it
> > > > > >> >> off
> > > > > >> >> > to
> > > > > >> >> > >> >> MET.
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> Here's the steps I followed to do this:
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> (1) Copy over a sample python script from MET:
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> cp
> > /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > > > > >> >> > >> >> read_ECMWF_NC_2D.py
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> (2) Edit it to read the input netcdf file
(argument
> 1)
> > > and
> > > > > the
> > > > > >> >> > variable
> > > > > >> >> > >> >> name (argument 2).
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> (3) Add code to parse out the timing info and
define
> it
> > > in
> > > > > the
> > > > > >> >> "attr"
> > > > > >> >> > >> >> dictionary.
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> NOTE that I assumed that this is an
analysis... so I
> > just
> > > > set
> > > > > >> the
> > > > > >> >> > init
> > > > > >> >> > >> >> time
> > > > > >> >> > >> >> to equal the valid time. Feel free to change
that
> > > however
> > > > > you
> > > > > >> >> need.
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> (4) Add the grid definition info. And I
figured this
> > out
> > > > by
> > > > > >> >> running
> > > > > >> >> > >> >> pcp_combine (see below) and then looking at
the
> NetCDF
> > > > global
> > > > > >> >> > >> attributes.
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> pcp_combine -add
NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> 'name="TP_6h";
> > > > > >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;'
TP_6h.nc
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> (5) Along the way, I ran this command to make
sure my
> > > > python
> > > > > >> >> script
> > > > > >> >> > >> >> actually runs on the command line:
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> python read_ECMWF_NC_2D.py
> > > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> TP_6h
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> Please let me know how this goes for you.
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> Thanks,
> > > > > >> >> > >> >> John
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley
Gotway <
> > > > > >> >> johnhg at ucar.edu>
> > > > > >> >> > >> >> wrote:
> > > > > >> >> > >> >>
> > > > > >> >> > >> >> > Andrew,
> > > > > >> >> > >> >> >
> > > > > >> >> > >> >> > OK, question for you. Is this ECMWF
actually model
> > > > output
> > > > > >> with
> > > > > >> >> a
> > > > > >> >> > >> 6522
> > > > > >> >> > >> >> > forecast hour? Or should this be
interpreted as an
> > > > > analysis
> > > > > >> >> that's
> > > > > >> >> > >> >> valid
> > > > > >> >> > >> >> > at 01/28/2006 at 00Z?
> > > > > >> >> > >> >> >
> > > > > >> >> > >> >> > That'll determine how best to encode the
timing
> > > > > information.
> > > > > >> >> > >> >> >
> > > > > >> >> > >> >> > Thanks,
> > > > > >> >> > >> >> > John
> > > > > >> >> > >> >> >
> > > > > >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren -
NOAA
> > > > Affiliate
> > > > > >> via
> > > > > >> >> RT
> > > > > >> >> > <
> > > > > >> >> > >> >> > met_help at ucar.edu> wrote:
> > > > > >> >> > >> >> >
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >> <URL:
> > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > > >> >> >
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >> John,
> > > > > >> >> > >> >> >> That is great. Yes let's go the python
route, that
> > > > seems a
> > > > > >> lot
> > > > > >> >> > more
> > > > > >> >> > >> >> >> straightforward. Please let me know how I
may
> > proceed,
> > > > and
> > > > > >> >> thank
> > > > > >> >> > >> you in
> > > > > >> >> > >> >> >> advance for your help on this!!
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John Halley
Gotway
> > via
> > > > RT <
> > > > > >> >> > >> >> >> met_help at ucar.edu>
> > > > > >> >> > >> >> >> wrote:
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >> > Hello Andrew,
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > I see you have a question about reading
NetCDF
> > data
> > > > into
> > > > > >> MET.
> > > > > >> >> > >> >> Judging
> > > > > >> >> > >> >> >> by
> > > > > >> >> > >> >> >> > the path you sent, I figured that this
data
> lives
> > on
> > > > > >> theia.
> > > > > >> >> So
> > > > > >> >> > I
> > > > > >> >> > >> >> >> logged on
> > > > > >> >> > >> >> >> > and took a look.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > Looks like this ECMWF file doesn't
exactly fall
> > into
> > > > any
> > > > > >> of
> > > > > >> >> the
> > > > > >> >> > >> >> NetCDF
> > > > > >> >> > >> >> >> > categories that MET currently supports.
The
> > closest
> > > > is
> > > > > >> >> > >> CF-compliant
> > > > > >> >> > >> >> >> > NetCDF. So let's tell MET to interpret
it as
> such
> > > > using
> > > > > >> the
> > > > > >> >> > >> >> "file_type
> > > > > >> >> > >> >> >> =
> > > > > >> >> > >> >> >> > NETCDF_NCCF;" option.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > When I'm trying to figure out if MET can
read
> > > gridded
> > > > > >> data, I
> > > > > >> >> > >> usually
> > > > > >> >> > >> >> >> use
> > > > > >> >> > >> >> >> > the plot_data_plane tool.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > Here's the commands I ran on theia:
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > module use /contrib/modulefiles
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > module load intel met/8.0
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > plot_data_plane
> > > > > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc
TP_6h.ps \
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> > > > > file_type=NETCDF_NCCF;'
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > This command ran and produced the
attached plot,
> > but
> > > > it
> > > > > >> did
> > > > > >> >> also
> > > > > >> >> > >> >> >> produce a
> > > > > >> >> > >> >> >> > warning about the valid time:
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not
determine
> > the
> > > > > valid
> > > > > >> >> time,
> > > > > >> >> > >> >> using
> > > > > >> >> > >> >> >> 0.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > So MET isn't getting the timing info
correct.
> > What
> > > > can
> > > > > >> we do
> > > > > >> >> > >> about
> > > > > >> >> > >> >> >> this?
> > > > > >> >> > >> >> >> > One option would be reformatting these
NetCDF
> > files
> > > > > into a
> > > > > >> >> > format
> > > > > >> >> > >> >> that
> > > > > >> >> > >> >> >> MET
> > > > > >> >> > >> >> >> > can actually read... either CF-compliant
NetCDF
> or
> > > the
> > > > > MET
> > > > > >> >> > NetCDF
> > > > > >> >> > >> >> >> format.
> > > > > >> >> > >> >> >> > However, that's pretty cumbersome. I
would
> > instead
> > > > > >> recommend
> > > > > >> >> > >> that we
> > > > > >> >> > >> >> >> take
> > > > > >> >> > >> >> >> > advantage of a new feature in met-8.0
(released
> > last
> > > > > >> >> Friday). We
> > > > > >> >> > >> can
> > > > > >> >> > >> >> >> write
> > > > > >> >> > >> >> >> > a Python script to read data from these
files,
> > > define
> > > > > the
> > > > > >> >> > >> metadata,
> > > > > >> >> > >> >> and
> > > > > >> >> > >> >> >> > then pass that to MET via memory.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > If the timing information doesn't matter
for
> some
> > > > > reason,
> > > > > >> >> then
> > > > > >> >> > you
> > > > > >> >> > >> >> could
> > > > > >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;"
option.
> > But
> > > > if
> > > > > it
> > > > > >> >> does
> > > > > >> >> > >> >> matter,
> > > > > >> >> > >> >> >> and
> > > > > >> >> > >> >> >> > presumably it does, then the Python route
should
> > > work.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > Let me know how you'd like to proceed.
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > Thanks,
> > > > > >> >> > >> >> >> > John
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew
Kren -
> NOAA
> > > > > >> Affiliate
> > > > > >> >> via
> > > > > >> >> > >> RT <
> > > > > >> >> > >> >> >> > met_help at ucar.edu> wrote:
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request 87266
was
> > acted
> > > > > upon.
> > > > > >> >> > >> >> >> > > Transaction: Ticket created by
> > > andrew.kren at noaa.gov
> > > > > >> >> > >> >> >> > > Queue: met_help
> > > > > >> >> > >> >> >> > > Subject: Question about MET
pcp_combine
> > > > > >> >> > >> >> >> > > Owner: Nobody
> > > > > >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > > > > >> >> > >> >> >> > > Status: new
> > > > > >> >> > >> >> >> > > Ticket <URL:
> > > > > >> >> > >> >>
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > Dear MET help,
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > I have a question on the MET
pcp_combine add
> > > tool. I
> > > > > >> have
> > > > > >> >> > model
> > > > > >> >> > >> >> data
> > > > > >> >> > >> >> >> that
> > > > > >> >> > >> >> >> > > is considered my truth from an ECMWF
Nature
> run
> > > > > >> (long-run
> > > > > >> >> > >> >> simulation
> > > > > >> >> > >> >> >> used
> > > > > >> >> > >> >> >> > > to represent the true atmosphere). I
want to
> > > verify
> > > > > how
> > > > > >> >> well
> > > > > >> >> > my
> > > > > >> >> > >> >> >> forecast
> > > > > >> >> > >> >> >> > > output, in this case from the GFS
model, is
> > > compared
> > > > > to
> > > > > >> the
> > > > > >> >> > >> Nature
> > > > > >> >> > >> >> >> run.
> > > > > >> >> > >> >> >> > The
> > > > > >> >> > >> >> >> > > ECMWF nature run data is 6-hourly data,
which
> > > > includes
> > > > > >> 6-hr
> > > > > >> >> > >> >> >> precipitation
> > > > > >> >> > >> >> >> > > and other fields in its netcdf files.
my GFS
> > data
> > > is
> > > > > >> grib
> > > > > >> >> > >> format. I
> > > > > >> >> > >> >> >> > already
> > > > > >> >> > >> >> >> > > can convert the GFS data to 24-hourly
> > > accumulations
> > > > > >> using
> > > > > >> >> my
> > > > > >> >> > >> prior
> > > > > >> >> > >> >> >> > scripts
> > > > > >> >> > >> >> >> > > and with pcp_combine. I guess my issue
is in
> the
> > > > > >> ability to
> > > > > >> >> > read
> > > > > >> >> > >> >> the
> > > > > >> >> > >> >> >> > Nature
> > > > > >> >> > >> >> >> > > run netcdf files. I try the pcp_combine
tool
> > and I
> > > > get
> > > > > >> this
> > > > > >> >> > >> error
> > > > > >> >> > >> >> when
> > > > > >> >> > >> >> >> > > trying to create a 24-hr accumulation.
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > (0) Current cycle: 2006012800
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > DEBUG 2: Performing addition command
for 4
> > files.
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > DEBUG 1: Reading input file:
> > > > > >> >> > >> >> >> > >
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > DEBUG 4:
> > > > Met2dDataFileFactory::new_met_2d_data_file()
> > > > > ->
> > > > > >> >> > created
> > > > > >> >> > >> >> new
> > > > > >> >> > >> >> >> > > Met2dDataFile object of type
"FileType_None".
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > ERROR :
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > ERROR : get_field() -> can't open data
file
> > > > > >> >> > >> >> >> > >
> "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > ERROR :
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > I assume MET is having issues reading
the
> > variable
> > > > in
> > > > > >> the
> > > > > >> >> > file,
> > > > > >> >> > >> >> which
> > > > > >> >> > >> >> >> is
> > > > > >> >> > >> >> >> > > denoted at TP_6h for 6-hr total
precipitation.
> > Is
> > > > > there
> > > > > >> a
> > > > > >> >> > >> >> workaround
> > > > > >> >> > >> >> >> so
> > > > > >> >> > >> >> >> > > that it can successfully process this?
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > Thank you kindly,
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > > --
> > > > > >> >> > >> >> >> > > Andrew Kren
> > > > > >> >> > >> >> >> > > Assistant Scientist
> > > > > >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > > > > >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-3058
> > > > > >> >> > >> >> >> > > Boulder, Colorado, USA
> > > > > >> >> > >> >> >> > > Office: 303-497-2715
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> > >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >> >
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >> --
> > > > > >> >> > >> >> >> Andrew Kren
> > > > > >> >> > >> >> >> Assistant Scientist
> > > > > >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > > > > >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > > >> >> > >> >> >> Boulder, Colorado, USA
> > > > > >> >> > >> >> >> Office: 303-497-2715
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >>
> > > > > >> >> > >> >
> > > > > >> >> > >> > --
> > > > > >> >> > >> > Andrew Kren
> > > > > >> >> > >> > Assistant Scientist
> > > > > >> >> > >> > University of Miami CIMAS - NOAA/AOML
> > > > > >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> > > > > >> >> > >> > Boulder, Colorado, USA
> > > > > >> >> > >> > Office: 303-497-2715
> > > > > >> >> > >> >
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >> --
> > > > > >> >> > >> Andrew Kren
> > > > > >> >> > >> Assistant Scientist
> > > > > >> >> > >> University of Miami CIMAS - NOAA/AOML
> > > > > >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> > > > > >> >> > >> Boulder, Colorado, USA
> > > > > >> >> > >> Office: 303-497-2715
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Andrew Kren
> > > > > >> >> Assistant Scientist
> > > > > >> >> University of Miami CIMAS - NOAA/AOML
> > > > > >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > > >> >> Boulder, Colorado, USA
> > > > > >> >> Office: 303-497-2715
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > > > --
> > > > > > Andrew Kren
> > > > > > Assistant Scientist
> > > > > > University of Miami CIMAS - NOAA/AOML
> > > > > > Duty Station: NCAR Foothills - FL2-3058
> > > > > > Boulder, Colorado, USA
> > > > > > Office: 303-497-2715
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Andrew Kren
> > > > > Assistant Scientist
> > > > > University of Miami CIMAS - NOAA/AOML
> > > > > Duty Station: NCAR Foothills - FL2-3058
> > > > > Boulder, Colorado, USA
> > > > > Office: 303-497-2715
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Andrew Kren
> > > Assistant Scientist
> > > University of Miami CIMAS - NOAA/AOML
> > > Duty Station: NCAR Foothills - FL2-3058
> > > Boulder, Colorado, USA
> > > Office: 303-497-2715
> > >
> > >
> >
> >
>
> --
> Andrew Kren
> Assistant Scientist
> University of Miami CIMAS - NOAA/AOML
> Duty Station: NCAR Foothills - FL2-3058
> Boulder, Colorado, USA
> Office: 303-497-2715
>
>
------------------------------------------------
Subject: Question about MET pcp_combine
From: Andrew Kren - NOAA Affiliate
Time: Tue Oct 09 14:54:48 2018
Thank you John
On Tue, Oct 9, 2018 at 11:32 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Andrew,
>
>
> Here's an example of running pcp_combine with the python options to
add up
> the TP_6h variable across 4 input files.
>
>
> There are probably more succinct ways of summing up variables across
NetCDF
> files using the NCO tools, but this output a NetCDF that is easily
read by
> the other MET tools.
>
>
> Be aware that in my testing, I ran across a problem when you do not
use the
> "-name" option to define the output variable name. I'll debug that
and
> include a fix in the next set of patches.
>
>
> But as long as you use the "-name" option, you won't run into that.
>
>
> Thanks,
>
> John
>
>
> ##########################################
>
>
> module use /contrib/modulefiles
>
> module load intel met/8.0
>
>
> pcp_combine -add \
>
> PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.7986.2006033006.nc TP_6h";' \
>
> PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.7992.2006033012.nc TP_6h";' \
>
> PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.7998.2006033018.nc TP_6h";' \
>
> PYTHON_NUMPY 'name="read_ECMWF_NC_2D.py
> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> NR2006.pl.1by1.8004.2006033100.nc TP_6h";' \
>
> TP_24h_2006033100.nc -name APCP_24
>
>
>
> On Tue, Oct 9, 2018 at 9:58 AM Andrew Kren - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> >
> > John,
> > That worked! Thanks.
> > So regarding what I want to do know with the files. I want to
accumulate
> > the precipitation over selected periods using the pcp_combine. If
I want
> to
> > use MET in this instance do I follow the same procedure, i.e. say
> > pcp_combine then name="/python script...?
> >
> > Thanks,
> >
> > On Tue, Oct 9, 2018 at 9:31 AM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Andrew,
> > >
> > > I have a good guess as to what's happening here and this is easy
to
> fix.
> > > Let's say you have a python script named "myscript.py". To run
it on
> the
> > > command line, you run:
> > > python myscript.py
> > >
> > > But when running it via a MET tool, remove the leading "python"
> command,
> > > like this:
> > > plot_data_plane PYTHON_NUMPY plot.ps 'name="myscript.py";'
> > >
> > > Notice that I used 'name="myscript.py";' and NOT 'name="python
> > > myscript.py";'.
> > >
> > > Please make sure that's how you're specifying it and try again.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Oct 9, 2018 at 8:35 AM Andrew Kren - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
>
> > > >
> > > > John,
> > > > Ok I tried this morning putting that at the top of my python
script,
> > > > read_ECMWF_NC_2D.py. I then ran my shell script as usual, but
it
> still
> > > gave
> > > > me this error:
> > > >
> > > > The following have been reloaded with a version change:
> > > >
> > > > 1) intel/15.0.0 => intel/14.0.2
> > > >
> > > >
> > > > /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > >
> > > > Input File: '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > NR2006.pl.1by1.6522.2006012806.nc'
> > > >
> > > > Variable Name : 'TP_6h'
> > > >
> > > > Data Shape: (181, 360)
> > > >
> > > > Data Type: dtype('float64')
> > > >
> > > > Attributes: {'long_name': 'TP_6h', 'init': '20060128_060000',
> 'valid':
> > > > '20060128_060000', 'grid': {'delta_lat': 1.0, 'lon_ll': 0.0,
'name':
> > > > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0, 'delta_lon':
1.0,
> 'type':
> > > > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead': '00',
'level':
> > > 'Surface',
> > > > 'units': 'mm', 'accum': '06'}
> > > >
> > > > DEBUG 1: Opening data file: PYTHON_NUMPY
> > > >
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> > > > Met2dDataFile object of type "FileType_Python_Numpy".
> > > >
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of
> > > > type "FileType_Python_Numpy".
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : python_dataplane() -> an error occurred importing
module
> > > > "./python.py"
> > > >
> > > > ERROR :
> > > >
> > > > ImportError: No module named python
> > > >
> > > > /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > >
> > > >
> > > > Do I need to add it to the shell script as well?
> > > >
> > > > On Mon, Oct 8, 2018 at 1:22 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Andrew,
> > > > >
> > > > > I did some more hunting around on this issue. Please try
adding
> the
> > > > > following lines to the top of the python script you're
running:
> > > > >
> > > > > import sys
> > > > > sys.path.insert(0,
> > > > > '/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-
packages')
> > > > > sys.path.insert(0,
> > '/contrib/anaconda/anaconda2/4.4.0/lib/site-python')
> > > > >
> > > > > The problem is that when Python is called by MET, the
> "site-packages"
> > > > > settings differ from when invoking the python interpreter on
the
> > > command
> > > > > line.
> > > > >
> > > > > On the command line, we're pointing to the right site-
packages:
> > > > > python -c "import site; print(site.getsitepackages())"
> > > > > ['/contrib/anaconda/anaconda2/4.4.0/lib/python2.7/site-
packages',
> > > > > '/contrib/anaconda/anaconda2/4.4.0/lib/site-python']
> > > > >
> > > > > When called via MET, it points here:
> > > > > ['/usr/lib64/python2.7/site-packages',
> > > > '/usr/lib/python2.7/site-packages',
> > > > > '/usr/lib/site-python']
> > > > >
> > > > > We need to do some more digging to see if we can recompile
MET to
> > point
> > > > to
> > > > > the *correct* site packages.
> > > > >
> > > > > But please confirm that this change to the script enable MET
to
> work
> > > for
> > > > > you.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Oct 4, 2018 at 2:17 PM Andrew Kren - NOAA Affiliate
via RT
> <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266 >
> > > > > >
> > > > > > Is there a workaround in the meantime John? Or were you
going to
> > > check
> > > > on
> > > > > > that first? I got your script and confirmed your finding.
> > > > > >
> > > > > > On Thu, Oct 4, 2018 at 1:05 PM Andrew Kren - NOAA
Affiliate <
> > > > > > andrew.kren at noaa.gov> wrote:
> > > > > >
> > > > > > > Thanks John.
> > > > > > >
> > > > > > > On Thu, Oct 4, 2018 at 12:21 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >> Andrew,
> > > > > > >>
> > > > > > >> OK, this script runs successfully on theia with python
from my
> > > path:
> > > > > > >>
> > > > > > >> > which python
> > > > > > >>
> > > > > > >> /contrib/anaconda/anaconda2/4.4.0/bin/python
> > > > > > >>
> > > > > > >>
> > > > > > >> But when I run the script using /bin/python, I get the
same
> > error
> > > as
> > > > > > from
> > > > > > >> MET:
> > > > > > >>
> > > > > > >>
> > > > > > >> Traceback (most recent call last):
> > > > > > >>
> > > > > > >> File
> > > > > >
> "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > > > >> line 3, in <module>
> > > > > > >>
> > > > > > >> from netCDF4 import Dataset
> > > > > > >>
> > > > > > >> ImportError: No module named netCDF4
> > > > > > >>
> > > > > > >> So it would appear that MET is not calling the version
of
> python
> > > > that
> > > > > > >> lives
> > > > > > >> in my path. I need to check with another developer to
figure
> > out
> > > if
> > > > > > >> there's a way to tell MET what version of python to
run.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Thu, Oct 4, 2018 at 12:16 PM John Halley Gotway <
> > > johnhg at ucar.edu
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Andrew,
> > > > > > >> >
> > > > > > >> > Hmmm, we're close. Try running this one:
> > > > > > >> >
> > > > > > >> >
> > /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/
> > > > > > >> > create_24_accum_met_nr.sh
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > The output is listed below. The punchline is... when
I run
> > the
> > > > > python
> > > > > > >> > script on the command line everything is fine. When
I run
> it
> > > > > through
> > > > > > >> > plot_data_plane, I get a runtime error about netCDF4.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > I can confirm that this works fine on my project
machine at
> > > NCAR.
> > > > > So
> > > > > > I
> > > > > > >> > suspect it's an environment issue.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > John
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > [John.H.Gotway at tfe05 kren_data_20181003]$
> > > > > ./create_24_accum_met_nr.sh
> > > > > > >> >
> > > > > > >> > Input File:
> '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > > >> > NR2006.pl.1by1.6522.2006012806.nc'
> > > > > > >> >
> > > > > > >> > Variable Name : 'TP_6h'
> > > > > > >> >
> > > > > > >> > Data Shape: (181, 360)
> > > > > > >> >
> > > > > > >> > Data Type: dtype('float64')
> > > > > > >> >
> > > > > > >> > Attributes: {'long_name': 'TP_6h', 'init':
> '20060128_060000',
> > > > > 'valid':
> > > > > > >> > '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll': 0.0,
> > > > 'name':
> > > > > > >> > 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> > 1.0,
> > > > > > 'type':
> > > > > > >> > 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead':
'00',
> > 'level':
> > > > > > >> 'Surface',
> > > > > > >> > 'units': 'mm', 'accum': '06'}
> > > > > > >> >
> > > > > > >> > DEBUG 1: Opening data file: PYTHON_NUMPY
> > > > > > >> >
> > > > > > >> > ERROR :
> > > > > > >> >
> > > > > > >> > ERROR : python_dataplane() -> an error occurred
importing
> > > module
> > > > > > >> >
> > >
"/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py"
> > > > > > >> >
> > > > > > >> > ERROR :
> > > > > > >> >
> > > > > > >> > Traceback (most recent call last):
> > > > > > >> >
> > > > > > >> > File
> > > > > > >>
> > "/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/read_ECMWF_NC_2D.py",
> > > > > > >> > line 3, in <module>
> > > > > > >> >
> > > > > > >> > from netCDF4 import Dataset
> > > > > > >> >
> > > > > > >> > ImportError: No module named netCDF4
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, Oct 4, 2018 at 11:42 AM Andrew Kren - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > >> > met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> >>
> > > > > > >> >> <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > >
> > > > > > >> >>
> > > > > > >> >> John,
> > > > > > >> >> I ran this script,
> > > > > > >> >>
> > > > >
>
/scratch4/BMC/shout/Andrew.Kren/ncl_scripts/create_24_accum_met_nr.sh
> > > > > > >> and
> > > > > > >> >> it gave me this, presumably asking for more
information to
> > run
> > > > the
> > > > > > met
> > > > > > >> >> command.
> > > > > > >> >>
> > > > > > >> >> The following have been reloaded with a version
change:
> > > > > > >> >>
> > > > > > >> >> 1) intel/15.0.0 => intel/14.0.2
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> Input File:
> '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > > >> >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > > > >> >>
> > > > > > >> >> Variable Name : 'TP_6h'
> > > > > > >> >>
> > > > > > >> >> Data Shape: (181, 360)
> > > > > > >> >>
> > > > > > >> >> Data Type: dtype('float64')
> > > > > > >> >>
> > > > > > >> >> Attributes: {'long_name': 'TP_6h', 'init':
> '20060128_060000',
> > > > > > 'valid':
> > > > > > >> >> '20060128_060000', 'grid': {'delta_lat': 1.0,
'lon_ll':
> 0.0,
> > > > > 'name':
> > > > > > >> >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
'delta_lon':
> > 1.0,
> > > > > > 'type':
> > > > > > >> >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h', 'lead':
'00',
> > 'level':
> > > > > > >> 'Surface',
> > > > > > >> >> 'units': 'mm', 'accum': '06'}
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> *** Model Evaluation Tools (METV8.0) ***
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> Usage: plot_data_plane
> > > > > > >> >>
> > > > > > >> >> input_filename
> > > > > > >> >>
> > > > > > >> >> output_filename
> > > > > > >> >>
> > > > > > >> >> field_string
> > > > > > >> >>
> > > > > > >> >> [-color_table color_table_name]
> > > > > > >> >>
> > > > > > >> >> [-plot_range min max]
> > > > > > >> >>
> > > > > > >> >> [-title title_string]
> > > > > > >> >>
> > > > > > >> >> [-log file]
> > > > > > >> >>
> > > > > > >> >> [-v level]
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> where "input_filename" is the name of a gridded
data file
> to
> > > be
> > > > > > >> plotted
> > > > > > >> >> (required).
> > > > > > >> >>
> > > > > > >> >> "output_filename" is the name of the output
PostScript file
> > to
> > > be
> > > > > > >> written
> > > > > > >> >> (required).
> > > > > > >> >>
> > > > > > >> >> "field_string" defines the data to be plotted from
the
> input
> > > file
> > > > > > >> >> (required).
> > > > > > >> >>
> > > > > > >> >> "-color_table color_table_name" overrides the
default color
> > > table
> > > > > > >> >> ("colortables/met_default.ctable") (optional).
> > > > > > >> >>
> > > > > > >> >> "-plot_range min max" defines the range of the data
to be
> > > plotted
> > > > > > >> >> (optional).
> > > > > > >> >>
> > > > > > >> >> "-title title_string" specifies the plot title
string
> > > (optional).
> > > > > > >> >>
> > > > > > >> >> "-log file" outputs log messages to the specified
file
> > > > (optional).
> > > > > > >> >>
> > > > > > >> >> "-v level" overrides the default level of logging
(2)
> > > (optional).
> > > > > > >> >>
> > > > > > >> >> On Thu, Oct 4, 2018 at 11:37 AM John Halley Gotway
via RT <
> > > > > > >> >> met_help at ucar.edu>
> > > > > > >> >> wrote:
> > > > > > >> >>
> > > > > > >> >> > Andrew,
> > > > > > >> >> >
> > > > > > >> >> > Sorry, just noticed a typo after I hit send. I
had a
> > > > > double-equals
> > > > > > >> (==)
> > > > > > >> >> > that should be a single equal (=):
> > > > > > >> >> >
> > > > > > >> >> > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > > >> >> >
> > > > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h.ps \
> > > > > > >> >> >
> > > > > > >> >> > 'name="python
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > > >> >> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h";'
> > > > > > >> >> >
> > > > > > >> >> > On Thu, Oct 4, 2018 at 11:34 AM John Halley Gotway
<
> > > > > > johnhg at ucar.edu>
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> > > Great, so running the python script stand-alone
read
> the
> > > data
> > > > > > well.
> > > > > > >> >> > >
> > > > > > >> >> > > The next step is telling MET to call this
script.
> Here's
> > > > that
> > > > > > >> simple
> > > > > > >> >> > > example of running plot_data_plane:
> > > > > > >> >> > >
> > > > > > >> >> > > plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > > >> >> > >
> > > > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h.ps \
> > > > > > >> >> > >
> > > > > > >> >> > > 'name=="python
> > > > > > >> >> > >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > > >> >> > > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > > <http://nr2006.pl.1by1.6522.2006012806.nc/>
TP_6h";'
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > Once this works, look at the output to confirm
that MET
> > is
> > > > > > putting
> > > > > > >> >> your
> > > > > > >> >> > > data on the right spot on the earth.
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > Presumably it does... so now you're ready to
start
> using
> > > this
> > > > > to
> > > > > > >> run
> > > > > > >> >> > > pcp_combine or Grid-Stat or MODE.
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > Make sense?
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > John
> > > > > > >> >> > >
> > > > > > >> >> > > On Thu, Oct 4, 2018 at 11:14 AM Andrew Kren -
NOAA
> > > Affiliate
> > > > > via
> > > > > > >> RT <
> > > > > > >> >> > > met_help at ucar.edu> wrote:
> > > > > > >> >> > >
> > > > > > >> >> > >>
> > > > > > >> >> > >> <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > > >
> > > > > > >> >> > >>
> > > > > > >> >> > >> John,
> > > > > > >> >> > >> I got this output when I ran your python script
in a
> > shell
> > > > > > script:
> > > > > > >> >> > >>
> > > > > > >> >> > >> The following have been reloaded with a version
> change:
> > > > > > >> >> > >>
> > > > > > >> >> > >> 1) intel/15.0.0 => intel/14.0.2
> > > > > > >> >> > >>
> > > > > > >> >> > >>
> > > > > > >> >> > >> Input File:
> > > > '/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc//
> > > > > > >> >> > >> NR2006.pl.1by1.6522.2006012806.nc'
> > > > > > >> >> > >>
> > > > > > >> >> > >> Variable Name : 'TP_6h'
> > > > > > >> >> > >>
> > > > > > >> >> > >> Data Shape: (181, 360)
> > > > > > >> >> > >>
> > > > > > >> >> > >> Data Type: dtype('float64')
> > > > > > >> >> > >>
> > > > > > >> >> > >> Attributes: {'long_name': 'TP_6h', 'init':
> > > > '20060128_060000',
> > > > > > >> >> 'valid':
> > > > > > >> >> > >> '20060128_060000', 'grid': {'delta_lat': 1.0,
> 'lon_ll':
> > > 0.0,
> > > > > > >> 'name':
> > > > > > >> >> > >> 'ECMWF-Global', 'Nlat': 181, 'lat_ll': -90.0,
> > 'delta_lon':
> > > > > 1.0,
> > > > > > >> >> 'type':
> > > > > > >> >> > >> 'LatLon', 'Nlon': 360}, 'name': 'TP_6h',
'lead': '00',
> > > > > 'level':
> > > > > > >> >> > 'Surface',
> > > > > > >> >> > >> 'units': 'mm', 'accum': '06'}
> > > > > > >> >> > >>
> > > > > > >> >> > >>
> > > > > > >> >> > >> Looks like it worked.
> > > > > > >> >> > >>
> > > > > > >> >> > >> On Thu, Oct 4, 2018 at 10:55 AM Andrew Kren -
NOAA
> > > > Affiliate <
> > > > > > >> >> > >> andrew.kren at noaa.gov> wrote:
> > > > > > >> >> > >>
> > > > > > >> >> > >> > John,
> > > > > > >> >> > >> > I believe the 6522 and other numbers (which
increase
> > > with
> > > > > each
> > > > > > >> >> date)
> > > > > > >> >> > >> were
> > > > > > >> >> > >> > used as place holders for the filename. The
actual
> > dates
> > > > can
> > > > > > be
> > > > > > >> >> read
> > > > > > >> >> > as
> > > > > > >> >> > >> an
> > > > > > >> >> > >> > analysis that's valid at the time stamp, such
as
> > > > 01/28/2006
> > > > > at
> > > > > > >> 06Z?
> > > > > > >> >> > Thus
> > > > > > >> >> > >> > the filename
NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > >> > <http://nr2006.pl.1by1.6522.2006012806.nc/>
would
> > > > > correspond
> > > > > > to
> > > > > > >> >> the
> > > > > > >> >> > >> > analysis valid at 2006012806, does that help?
I'll
> try
> > > the
> > > > > > >> script.
> > > > > > >> >> > >> >
> > > > > > >> >> > >> > On Thu, Oct 4, 2018 at 10:46 AM John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > >> >> > >> > met_help at ucar.edu> wrote:
> > > > > > >> >> > >> >
> > > > > > >> >> > >> >> Andrew,
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> OK, please try running this command on
theia. I've
> > > > > attached
> > > > > > >> the
> > > > > > >> >> > >> resulting
> > > > > > >> >> > >> >> image:
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> module use /contrib/modulefiles
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> module load intel met/8.0
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> plot_data_plane PYTHON_NUMPY TP_6h.ps \
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > >> >> <http://nr2006.pl.1by1.6522.2006012806.nc/>
> > TP_6h.ps \
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> 'name=="python
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >>
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/kren_data_20181003/read_ECMWF_NC_2D.py
> > > > > > >> >> > >> >>
/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > >> >> NR2006.pl.1by1.6522.2006012806.nc TP_6h";'
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> This python script (also attached) reads
your data
> > and
> > > > > hands
> > > > > > it
> > > > > > >> >> off
> > > > > > >> >> > to
> > > > > > >> >> > >> >> MET.
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> Here's the steps I followed to do this:
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> (1) Copy over a sample python script from
MET:
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> cp
> > > /contrib/met/8.0/share/met/python/read_ascii_numpy.py
> > > > > > >> >> > >> >> read_ECMWF_NC_2D.py
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> (2) Edit it to read the input netcdf file
(argument
> > 1)
> > > > and
> > > > > > the
> > > > > > >> >> > variable
> > > > > > >> >> > >> >> name (argument 2).
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> (3) Add code to parse out the timing info
and
> define
> > it
> > > > in
> > > > > > the
> > > > > > >> >> "attr"
> > > > > > >> >> > >> >> dictionary.
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> NOTE that I assumed that this is an
analysis... so
> I
> > > just
> > > > > set
> > > > > > >> the
> > > > > > >> >> > init
> > > > > > >> >> > >> >> time
> > > > > > >> >> > >> >> to equal the valid time. Feel free to
change that
> > > > however
> > > > > > you
> > > > > > >> >> need.
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> (4) Add the grid definition info. And I
figured
> this
> > > out
> > > > > by
> > > > > > >> >> running
> > > > > > >> >> > >> >> pcp_combine (see below) and then looking at
the
> > NetCDF
> > > > > global
> > > > > > >> >> > >> attributes.
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> pcp_combine -add
NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> 'name="TP_6h";
> > > > > > >> >> > >> >> level="(*,*)"; file_type=NETCDF_NCCF;'
TP_6h.nc
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> (5) Along the way, I ran this command to
make sure
> my
> > > > > python
> > > > > > >> >> script
> > > > > > >> >> > >> >> actually runs on the command line:
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> python read_ECMWF_NC_2D.py
> > > > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> TP_6h
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> Please let me know how this goes for you.
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> Thanks,
> > > > > > >> >> > >> >> John
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> On Thu, Oct 4, 2018 at 10:35 AM John Halley
Gotway
> <
> > > > > > >> >> johnhg at ucar.edu>
> > > > > > >> >> > >> >> wrote:
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >> > Andrew,
> > > > > > >> >> > >> >> >
> > > > > > >> >> > >> >> > OK, question for you. Is this ECMWF
actually
> model
> > > > > output
> > > > > > >> with
> > > > > > >> >> a
> > > > > > >> >> > >> 6522
> > > > > > >> >> > >> >> > forecast hour? Or should this be
interpreted as
> an
> > > > > > analysis
> > > > > > >> >> that's
> > > > > > >> >> > >> >> valid
> > > > > > >> >> > >> >> > at 01/28/2006 at 00Z?
> > > > > > >> >> > >> >> >
> > > > > > >> >> > >> >> > That'll determine how best to encode the
timing
> > > > > > information.
> > > > > > >> >> > >> >> >
> > > > > > >> >> > >> >> > Thanks,
> > > > > > >> >> > >> >> > John
> > > > > > >> >> > >> >> >
> > > > > > >> >> > >> >> > On Thu, Oct 4, 2018 at 8:43 AM Andrew Kren
- NOAA
> > > > > Affiliate
> > > > > > >> via
> > > > > > >> >> RT
> > > > > > >> >> > <
> > > > > > >> >> > >> >> > met_help at ucar.edu> wrote:
> > > > > > >> >> > >> >> >
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >> <URL:
> > > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > > > >> >> >
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >> John,
> > > > > > >> >> > >> >> >> That is great. Yes let's go the python
route,
> that
> > > > > seems a
> > > > > > >> lot
> > > > > > >> >> > more
> > > > > > >> >> > >> >> >> straightforward. Please let me know how I
may
> > > proceed,
> > > > > and
> > > > > > >> >> thank
> > > > > > >> >> > >> you in
> > > > > > >> >> > >> >> >> advance for your help on this!!
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >> On Wed, Oct 3, 2018 at 7:14 PM John
Halley
> Gotway
> > > via
> > > > > RT <
> > > > > > >> >> > >> >> >> met_help at ucar.edu>
> > > > > > >> >> > >> >> >> wrote:
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >> > Hello Andrew,
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > I see you have a question about reading
NetCDF
> > > data
> > > > > into
> > > > > > >> MET.
> > > > > > >> >> > >> >> Judging
> > > > > > >> >> > >> >> >> by
> > > > > > >> >> > >> >> >> > the path you sent, I figured that this
data
> > lives
> > > on
> > > > > > >> theia.
> > > > > > >> >> So
> > > > > > >> >> > I
> > > > > > >> >> > >> >> >> logged on
> > > > > > >> >> > >> >> >> > and took a look.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > Looks like this ECMWF file doesn't
exactly
> fall
> > > into
> > > > > any
> > > > > > >> of
> > > > > > >> >> the
> > > > > > >> >> > >> >> NetCDF
> > > > > > >> >> > >> >> >> > categories that MET currently supports.
The
> > > closest
> > > > > is
> > > > > > >> >> > >> CF-compliant
> > > > > > >> >> > >> >> >> > NetCDF. So let's tell MET to interpret
it as
> > such
> > > > > using
> > > > > > >> the
> > > > > > >> >> > >> >> "file_type
> > > > > > >> >> > >> >> >> =
> > > > > > >> >> > >> >> >> > NETCDF_NCCF;" option.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > When I'm trying to figure out if MET
can read
> > > > gridded
> > > > > > >> data, I
> > > > > > >> >> > >> usually
> > > > > > >> >> > >> >> >> use
> > > > > > >> >> > >> >> >> > the plot_data_plane tool.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > Here's the commands I ran on theia:
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > module use /contrib/modulefiles
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > module load intel met/8.0
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > plot_data_plane
> > > > > > >> >> /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > >> >> >> > NR2006.pl.1by1.6522.2006012806.nc
TP_6h.ps \
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > 'name=="TP_6h"; level="(*,*)";
> > > > > > file_type=NETCDF_NCCF;'
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > This command ran and produced the
attached
> plot,
> > > but
> > > > > it
> > > > > > >> did
> > > > > > >> >> also
> > > > > > >> >> > >> >> >> produce a
> > > > > > >> >> > >> >> >> > warning about the valid time:
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > WARNING: NcCfFile::open() -> could not
> determine
> > > the
> > > > > > valid
> > > > > > >> >> time,
> > > > > > >> >> > >> >> using
> > > > > > >> >> > >> >> >> 0.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > So MET isn't getting the timing info
correct.
> > > What
> > > > > can
> > > > > > >> we do
> > > > > > >> >> > >> about
> > > > > > >> >> > >> >> >> this?
> > > > > > >> >> > >> >> >> > One option would be reformatting these
NetCDF
> > > files
> > > > > > into a
> > > > > > >> >> > format
> > > > > > >> >> > >> >> that
> > > > > > >> >> > >> >> >> MET
> > > > > > >> >> > >> >> >> > can actually read... either CF-
compliant
> NetCDF
> > or
> > > > the
> > > > > > MET
> > > > > > >> >> > NetCDF
> > > > > > >> >> > >> >> >> format.
> > > > > > >> >> > >> >> >> > However, that's pretty cumbersome. I
would
> > > instead
> > > > > > >> recommend
> > > > > > >> >> > >> that we
> > > > > > >> >> > >> >> >> take
> > > > > > >> >> > >> >> >> > advantage of a new feature in met-8.0
> (released
> > > last
> > > > > > >> >> Friday). We
> > > > > > >> >> > >> can
> > > > > > >> >> > >> >> >> write
> > > > > > >> >> > >> >> >> > a Python script to read data from these
files,
> > > > define
> > > > > > the
> > > > > > >> >> > >> metadata,
> > > > > > >> >> > >> >> and
> > > > > > >> >> > >> >> >> > then pass that to MET via memory.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > If the timing information doesn't
matter for
> > some
> > > > > > reason,
> > > > > > >> >> then
> > > > > > >> >> > you
> > > > > > >> >> > >> >> could
> > > > > > >> >> > >> >> >> > just use the "file_type = NETCDF_NCCF;"
> option.
> > > But
> > > > > if
> > > > > > it
> > > > > > >> >> does
> > > > > > >> >> > >> >> matter,
> > > > > > >> >> > >> >> >> and
> > > > > > >> >> > >> >> >> > presumably it does, then the Python
route
> should
> > > > work.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > Let me know how you'd like to proceed.
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > Thanks,
> > > > > > >> >> > >> >> >> > John
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > On Wed, Oct 3, 2018 at 3:35 PM Andrew
Kren -
> > NOAA
> > > > > > >> Affiliate
> > > > > > >> >> via
> > > > > > >> >> > >> RT <
> > > > > > >> >> > >> >> >> > met_help at ucar.edu> wrote:
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > Wed Oct 03 15:35:15 2018: Request
87266 was
> > > acted
> > > > > > upon.
> > > > > > >> >> > >> >> >> > > Transaction: Ticket created by
> > > > andrew.kren at noaa.gov
> > > > > > >> >> > >> >> >> > > Queue: met_help
> > > > > > >> >> > >> >> >> > > Subject: Question about MET
pcp_combine
> > > > > > >> >> > >> >> >> > > Owner: Nobody
> > > > > > >> >> > >> >> >> > > Requestors: andrew.kren at noaa.gov
> > > > > > >> >> > >> >> >> > > Status: new
> > > > > > >> >> > >> >> >> > > Ticket <URL:
> > > > > > >> >> > >> >>
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87266
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > Dear MET help,
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > I have a question on the MET
pcp_combine add
> > > > tool. I
> > > > > > >> have
> > > > > > >> >> > model
> > > > > > >> >> > >> >> data
> > > > > > >> >> > >> >> >> that
> > > > > > >> >> > >> >> >> > > is considered my truth from an ECMWF
Nature
> > run
> > > > > > >> (long-run
> > > > > > >> >> > >> >> simulation
> > > > > > >> >> > >> >> >> used
> > > > > > >> >> > >> >> >> > > to represent the true atmosphere). I
want to
> > > > verify
> > > > > > how
> > > > > > >> >> well
> > > > > > >> >> > my
> > > > > > >> >> > >> >> >> forecast
> > > > > > >> >> > >> >> >> > > output, in this case from the GFS
model, is
> > > > compared
> > > > > > to
> > > > > > >> the
> > > > > > >> >> > >> Nature
> > > > > > >> >> > >> >> >> run.
> > > > > > >> >> > >> >> >> > The
> > > > > > >> >> > >> >> >> > > ECMWF nature run data is 6-hourly
data,
> which
> > > > > includes
> > > > > > >> 6-hr
> > > > > > >> >> > >> >> >> precipitation
> > > > > > >> >> > >> >> >> > > and other fields in its netcdf files.
my GFS
> > > data
> > > > is
> > > > > > >> grib
> > > > > > >> >> > >> format. I
> > > > > > >> >> > >> >> >> > already
> > > > > > >> >> > >> >> >> > > can convert the GFS data to 24-hourly
> > > > accumulations
> > > > > > >> using
> > > > > > >> >> my
> > > > > > >> >> > >> prior
> > > > > > >> >> > >> >> >> > scripts
> > > > > > >> >> > >> >> >> > > and with pcp_combine. I guess my
issue is in
> > the
> > > > > > >> ability to
> > > > > > >> >> > read
> > > > > > >> >> > >> >> the
> > > > > > >> >> > >> >> >> > Nature
> > > > > > >> >> > >> >> >> > > run netcdf files. I try the
pcp_combine tool
> > > and I
> > > > > get
> > > > > > >> this
> > > > > > >> >> > >> error
> > > > > > >> >> > >> >> when
> > > > > > >> >> > >> >> >> > > trying to create a 24-hr
accumulation.
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > (0) Current cycle: 2006012800
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > DEBUG 2: Performing addition command
for 4
> > > files.
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > DEBUG 1: Reading input file:
> > > > > > >> >> > >> >> >> > >
> > /scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > DEBUG 4:
> > > > > Met2dDataFileFactory::new_met_2d_data_file()
> > > > > > ->
> > > > > > >> >> > created
> > > > > > >> >> > >> >> new
> > > > > > >> >> > >> >> >> > > Met2dDataFile object of type
> "FileType_None".
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > ERROR :
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > ERROR : get_field() -> can't open
data file
> > > > > > >> >> > >> >> >> > >
> > "/scratch4/BMC/shout/Andrew.Kren/T511/nr_dev/nc/
> > > > > > >> >> > >> >> >> > > NR2006.pl.1by1.6522.2006012806.nc"
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > ERROR :
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > I assume MET is having issues reading
the
> > > variable
> > > > > in
> > > > > > >> the
> > > > > > >> >> > file,
> > > > > > >> >> > >> >> which
> > > > > > >> >> > >> >> >> is
> > > > > > >> >> > >> >> >> > > denoted at TP_6h for 6-hr total
> precipitation.
> > > Is
> > > > > > there
> > > > > > >> a
> > > > > > >> >> > >> >> workaround
> > > > > > >> >> > >> >> >> so
> > > > > > >> >> > >> >> >> > > that it can successfully process
this?
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > Thank you kindly,
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > > --
> > > > > > >> >> > >> >> >> > > Andrew Kren
> > > > > > >> >> > >> >> >> > > Assistant Scientist
> > > > > > >> >> > >> >> >> > > University of Miami CIMAS - NOAA/AOML
> > > > > > >> >> > >> >> >> > > Duty Station: NCAR Foothills - FL2-
3058
> > > > > > >> >> > >> >> >> > > Boulder, Colorado, USA
> > > > > > >> >> > >> >> >> > > Office: 303-497-2715
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> > >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >> >
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >> --
> > > > > > >> >> > >> >> >> Andrew Kren
> > > > > > >> >> > >> >> >> Assistant Scientist
> > > > > > >> >> > >> >> >> University of Miami CIMAS - NOAA/AOML
> > > > > > >> >> > >> >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > > > >> >> > >> >> >> Boulder, Colorado, USA
> > > > > > >> >> > >> >> >> Office: 303-497-2715
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >>
> > > > > > >> >> > >> >
> > > > > > >> >> > >> > --
> > > > > > >> >> > >> > Andrew Kren
> > > > > > >> >> > >> > Assistant Scientist
> > > > > > >> >> > >> > University of Miami CIMAS - NOAA/AOML
> > > > > > >> >> > >> > Duty Station: NCAR Foothills - FL2-3058
> > > > > > >> >> > >> > Boulder, Colorado, USA
> > > > > > >> >> > >> > Office: 303-497-2715
> > > > > > >> >> > >> >
> > > > > > >> >> > >>
> > > > > > >> >> > >>
> > > > > > >> >> > >> --
> > > > > > >> >> > >> Andrew Kren
> > > > > > >> >> > >> Assistant Scientist
> > > > > > >> >> > >> University of Miami CIMAS - NOAA/AOML
> > > > > > >> >> > >> Duty Station: NCAR Foothills - FL2-3058
> > > > > > >> >> > >> Boulder, Colorado, USA
> > > > > > >> >> > >> Office: 303-497-2715
> > > > > > >> >> > >>
> > > > > > >> >> > >>
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Andrew Kren
> > > > > > >> >> Assistant Scientist
> > > > > > >> >> University of Miami CIMAS - NOAA/AOML
> > > > > > >> >> Duty Station: NCAR Foothills - FL2-3058
> > > > > > >> >> Boulder, Colorado, USA
> > > > > > >> >> Office: 303-497-2715
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > Andrew Kren
> > > > > > > Assistant Scientist
> > > > > > > University of Miami CIMAS - NOAA/AOML
> > > > > > > Duty Station: NCAR Foothills - FL2-3058
> > > > > > > Boulder, Colorado, USA
> > > > > > > Office: 303-497-2715
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Andrew Kren
> > > > > > Assistant Scientist
> > > > > > University of Miami CIMAS - NOAA/AOML
> > > > > > Duty Station: NCAR Foothills - FL2-3058
> > > > > > Boulder, Colorado, USA
> > > > > > Office: 303-497-2715
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Andrew Kren
> > > > Assistant Scientist
> > > > University of Miami CIMAS - NOAA/AOML
> > > > Duty Station: NCAR Foothills - FL2-3058
> > > > Boulder, Colorado, USA
> > > > Office: 303-497-2715
> > > >
> > > >
> > >
> > >
> >
> > --
> > Andrew Kren
> > Assistant Scientist
> > University of Miami CIMAS - NOAA/AOML
> > Duty Station: NCAR Foothills - FL2-3058
> > Boulder, Colorado, USA
> > Office: 303-497-2715
> >
> >
>
>
--
Andrew Kren
Assistant Scientist
University of Miami CIMAS - NOAA/AOML
Duty Station: NCAR Foothills - FL2-3058
Boulder, Colorado, USA
Office: 303-497-2715
------------------------------------------------
More information about the Met_help
mailing list