[Met_help] [rt.rap.ucar.edu #80322] History for MET Series analysis
John Halley Gotway via RT
met_help at ucar.edu
Mon May 1 17:03:07 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
MET-Help,
My files are located in Theia at : /scratch4/NCEPDEV/global/noscrub/Partha.Bhattacharjee/MET6
I was trying to run "series_stat.sh" in that directory using "SeriesAnlConfig_AOTK_vs_AOD" configuration file. I am getting wrong time information from input netcdf file. However, in that same directory, when I run "grid_stat.sh" using the same nedcdf files and that creates output.
I appreciate your help on this.
Thanks.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET Series analysis
From: John Halley Gotway
Time: Mon May 01 15:19:36 2017
Partha,
I took a look at your Series-Analysis run on theia. I'm working in
this directory:
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/partha_data_20170501
When I ran your script as-is, I got the following error:
ERROR : get_series_entry() -> Could not find data for
total_aod_mean(4,*,*) in file list:
ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
By default, Series-Analysis tries to find the exact same timing
information in the list of forecast and observed files. If it can't
find an exact match, it errors out with this type of message.
Adding the "-paired" command line option tells the tools that you've
already paired up the data. So the 1st forecast file should go with
the 1st obs file. The 2nd with the 2nd, and so on. Now, instead of
getting an error message, you just get warnings about the timing info
not matching:
WARNING: get_series_data() -> Forecast and observation valid times do
not match 20170415_060000 != 20170416_000000 for AOTK/A0 versus
total_aod_mean(4,*,*).
But it now produces an output file. I still need to figure out the
multiple GRIB matches for "AOTK/A0", but hopefully using the "-paired"
option will help.
Thanks,
John
------------------------------------------------
Subject: MET Series analysis
From: Partha Bhattacharjee - NOAA Affiliate
Time: Mon May 01 15:54:22 2017
Thanks John. I have one question. I used "paired" option for 6 hourly
files
and I am getting the following messages :
DEBUG 2: Processing series entry 1 of 4: AOTK/A0 versus
total_aod_mean(4,*,*)
DEBUG 2: Processing series entry 2 of 4: AOTK/A0 versus
total_aod_mean(4,*,*)
DEBUG 2: Processing series entry 3 of 4: AOTK/A0 versus
total_aod_mean(4,*,*)
DEBUG 2: Processing series entry 4 of 4: AOTK/A0 versus
total_aod_mean(4,*,*)
Does it imply that it is matching/pairing all 4 grib files with the
same
time step of the ncetcdf observation file ? Ideally we should expect
first
grib file (6 hourly) will match with "total_aod_mean(1,*,*):, second
(12
hourly) with "total_aod_mean(2,*,*), 3rd with "total_aod_mean(3 ,*,*)
and
the fourth (24 hourly) will match "total_aod_mean(4,*,*). And is that
the
reason, we are getting those WARNING messages that time step between
the
files are not matching ?
Is it something to do with the configuration file I have used
"SeriesAnlConfig_AOTK_vs_AOD" ? I repeat "fcst" and "obs" for 4
different
grib files and 4 different time steps for a single verifying netcdf
observation in the config file. Let me know what you think.
Thanks.
Partha
On Mon, May 1, 2017 at 5:19 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Partha,
>
> I took a look at your Series-Analysis run on theia. I'm working in
this
> directory:
> /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/partha_data_20170501
>
> When I ran your script as-is, I got the following error:
> ERROR : get_series_entry() -> Could not find data for
> total_aod_mean(4,*,*) in file list:
> ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
>
> By default, Series-Analysis tries to find the exact same timing
> information in the list of forecast and observed files. If it can't
find
> an exact match, it errors out with this type of message.
>
> Adding the "-paired" command line option tells the tools that you've
> already paired up the data. So the 1st forecast file should go with
the
> 1st obs file. The 2nd with the 2nd, and so on. Now, instead of
getting an
> error message, you just get warnings about the timing info not
matching:
>
> WARNING: get_series_data() -> Forecast and observation valid times
do not
> match 20170415_060000 != 20170416_000000 for AOTK/A0 versus
> total_aod_mean(4,*,*).
>
> But it now produces an output file. I still need to figure out the
> multiple GRIB matches for "AOTK/A0", but hopefully using the "-
paired"
> option will help.
>
> Thanks,
> John
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
Subject: MET Series analysis
From: John Halley Gotway
Time: Mon May 01 16:10:13 2017
Partha,
Good news. I figured out how to distinguish between the different
AOTK records in the NGAC GRIB2 files. I ran MET in the debugger to
compare the contents of the different AOTK records. And I found a
difference in how the "Process" is set. Plesae refer to the values
listed in this GRIB2 table which is specific to Product Definition
Template #48:
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
When you run MET, specify the GRIB2_process you'd like to select. For
example 62008 corresponds to "Sea Salt Dry":
plot_data_plane ngac.t00z.a2df12.grib2 aotk.ps 'name="AOTK";
level="A0"; GRIB2_process=62008;' -v 3
Look at the wgrib2 output at the description of the aerosol type, and
refer to that table to figure out how to set the GRIB2_process. For
example, "Particulate Organic Matter Dry" from the wgrib2 output
corresponds to "-GRIB2_process=62010;".
Thanks,
John
------------------------------------------------
Subject: MET Series analysis
From: John Halley Gotway
Time: Mon May 01 16:22:29 2017
Partha,
OK, I took a closer look at your config file and see what's going on.
You've listed the "fcst" and "obs" dictionary 4 times. What's
happening is
that each of those settings is over-writing the previous one. I
should
have been more clear. Instead, you should have only 1 "fcst" and 1
"obs"
dictionary setting, but you should make the field array have length 4.
And
actually, since the variable name remains fixed, we can just list
multiple
level values instead. See below, and note that I'm just guessing on
the
"GRIB2_process=62008;" setting. Please refer to this table to figure
out
exactly how you want to set that:
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
fcst = {
field = [ { name = "AOTK";
level = [ "A0", "A0", "A0", "A0" ];
GRIB2_process = 62008; }
];
};
obs = {
field = [ { name = "total_aod_mean";
level = [ "(1,*,*)", "(2,*,*)", "(3,*,*)", "(4,*,*)"
]; }
];
};
When I run this configuration on your test data, all the times match
up
nicely. And note the following DEBUG messages that get printed:
DEBUG 1: Length of configuration "fcst.field" = 4
DEBUG 1: Length of configuration "obs.field" = 4
DEBUG 1: Length of forecast file list = 4
DEBUG 1: Length of observation file list = 4
This confirms that you've passed it 4 forecast files, 4 observation
files,
4 forecast fields, and 4 observation fields.
Thanks,
John
On Mon, May 1, 2017 at 3:54 PM, Partha Bhattacharjee - NOAA Affiliate
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80322 >
>
> Thanks John. I have one question. I used "paired" option for 6
hourly files
> and I am getting the following messages :
>
> DEBUG 2: Processing series entry 1 of 4: AOTK/A0 versus
> total_aod_mean(4,*,*)
>
> DEBUG 2: Processing series entry 2 of 4: AOTK/A0 versus
> total_aod_mean(4,*,*)
>
> DEBUG 2: Processing series entry 3 of 4: AOTK/A0 versus
> total_aod_mean(4,*,*)
>
> DEBUG 2: Processing series entry 4 of 4: AOTK/A0 versus
> total_aod_mean(4,*,*)
>
> Does it imply that it is matching/pairing all 4 grib files with
the same
> time step of the ncetcdf observation file ? Ideally we should expect
first
> grib file (6 hourly) will match with "total_aod_mean(1,*,*):, second
(12
> hourly) with "total_aod_mean(2,*,*), 3rd with "total_aod_mean(3
,*,*) and
> the fourth (24 hourly) will match "total_aod_mean(4,*,*). And is
that the
> reason, we are getting those WARNING messages that time step between
the
> files are not matching ?
>
> Is it something to do with the configuration file I have used
> "SeriesAnlConfig_AOTK_vs_AOD" ? I repeat "fcst" and "obs" for 4
different
> grib files and 4 different time steps for a single verifying netcdf
> observation in the config file. Let me know what you think.
>
> Thanks.
> Partha
>
>
> On Mon, May 1, 2017 at 5:19 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Partha,
> >
> > I took a look at your Series-Analysis run on theia. I'm working
in this
> > directory:
> > /scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/partha_data_20170501
> >
> > When I ran your script as-is, I got the following error:
> > ERROR : get_series_entry() -> Could not find data for
> > total_aod_mean(4,*,*) in file list:
> > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> >
> > By default, Series-Analysis tries to find the exact same timing
> > information in the list of forecast and observed files. If it
can't find
> > an exact match, it errors out with this type of message.
> >
> > Adding the "-paired" command line option tells the tools that
you've
> > already paired up the data. So the 1st forecast file should go
with the
> > 1st obs file. The 2nd with the 2nd, and so on. Now, instead of
getting
> an
> > error message, you just get warnings about the timing info not
matching:
> >
> > WARNING: get_series_data() -> Forecast and observation valid times
do not
> > match 20170415_060000 != 20170416_000000 for AOTK/A0 versus
> > total_aod_mean(4,*,*).
> >
> > But it now produces an output file. I still need to figure out
the
> > multiple GRIB matches for "AOTK/A0", but hopefully using the "-
paired"
> > option will help.
> >
> > Thanks,
> > John
> >
>
>
>
> --
> Partha S. Bhattacharjee, Ph.D.
> I.M. Systems Group
> Environmental Modeling Centre
> NCWCP
> 5830 University Research Court.
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MET Series analysis
From: Partha Bhattacharjee - NOAA Affiliate
Time: Mon May 01 16:50:07 2017
Many thanks John, it is working fine. Thanks for looking into it.
Partha
On Mon, May 1, 2017 at 6:22 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Partha,
>
> OK, I took a closer look at your config file and see what's going
on.
> You've listed the "fcst" and "obs" dictionary 4 times. What's
happening is
> that each of those settings is over-writing the previous one. I
should
> have been more clear. Instead, you should have only 1 "fcst" and 1
"obs"
> dictionary setting, but you should make the field array have length
4. And
> actually, since the variable name remains fixed, we can just list
multiple
> level values instead. See below, and note that I'm just guessing on
the
> "GRIB2_process=62008;" setting. Please refer to this table to
figure out
> exactly how you want to set that:
> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-
233.shtml
>
> fcst = {
> field = [ { name = "AOTK";
> level = [ "A0", "A0", "A0", "A0" ];
> GRIB2_process = 62008; }
> ];
> };
>
> obs = {
> field = [ { name = "total_aod_mean";
> level = [ "(1,*,*)", "(2,*,*)", "(3,*,*)",
"(4,*,*)" ]; }
> ];
> };
>
> When I run this configuration on your test data, all the times match
up
> nicely. And note the following DEBUG messages that get printed:
> DEBUG 1: Length of configuration "fcst.field" = 4
> DEBUG 1: Length of configuration "obs.field" = 4
> DEBUG 1: Length of forecast file list = 4
> DEBUG 1: Length of observation file list = 4
>
> This confirms that you've passed it 4 forecast files, 4 observation
files,
> 4 forecast fields, and 4 observation fields.
>
> Thanks,
> John
>
> On Mon, May 1, 2017 at 3:54 PM, Partha Bhattacharjee - NOAA
Affiliate via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80322 >
> >
> > Thanks John. I have one question. I used "paired" option for 6
hourly
> files
> > and I am getting the following messages :
> >
> > DEBUG 2: Processing series entry 1 of 4: AOTK/A0 versus
> > total_aod_mean(4,*,*)
> >
> > DEBUG 2: Processing series entry 2 of 4: AOTK/A0 versus
> > total_aod_mean(4,*,*)
> >
> > DEBUG 2: Processing series entry 3 of 4: AOTK/A0 versus
> > total_aod_mean(4,*,*)
> >
> > DEBUG 2: Processing series entry 4 of 4: AOTK/A0 versus
> > total_aod_mean(4,*,*)
> >
> > Does it imply that it is matching/pairing all 4 grib files with
the
> same
> > time step of the ncetcdf observation file ? Ideally we should
expect
> first
> > grib file (6 hourly) will match with "total_aod_mean(1,*,*):,
second (12
> > hourly) with "total_aod_mean(2,*,*), 3rd with "total_aod_mean(3
,*,*) and
> > the fourth (24 hourly) will match "total_aod_mean(4,*,*). And is
that the
> > reason, we are getting those WARNING messages that time step
between the
> > files are not matching ?
> >
> > Is it something to do with the configuration file I have used
> > "SeriesAnlConfig_AOTK_vs_AOD" ? I repeat "fcst" and "obs" for 4
different
> > grib files and 4 different time steps for a single verifying
netcdf
> > observation in the config file. Let me know what you think.
> >
> > Thanks.
> > Partha
> >
> >
> > On Mon, May 1, 2017 at 5:19 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Partha,
> > >
> > > I took a look at your Series-Analysis run on theia. I'm working
in
> this
> > > directory:
> > >
/scratch4/BMC/dtc/John.H.Gotway/MET/MET_Help/partha_data_20170501
> > >
> > > When I ran your script as-is, I got the following error:
> > > ERROR : get_series_entry() -> Could not find data for
> > > total_aod_mean(4,*,*) in file list:
> > > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > > ERROR : input_ICAP/icap_2017041500_MME_modeaod550.nc
> > >
> > > By default, Series-Analysis tries to find the exact same timing
> > > information in the list of forecast and observed files. If it
can't
> find
> > > an exact match, it errors out with this type of message.
> > >
> > > Adding the "-paired" command line option tells the tools that
you've
> > > already paired up the data. So the 1st forecast file should go
with
> the
> > > 1st obs file. The 2nd with the 2nd, and so on. Now, instead of
> getting
> > an
> > > error message, you just get warnings about the timing info not
> matching:
> > >
> > > WARNING: get_series_data() -> Forecast and observation valid
times do
> not
> > > match 20170415_060000 != 20170416_000000 for AOTK/A0 versus
> > > total_aod_mean(4,*,*).
> > >
> > > But it now produces an output file. I still need to figure out
the
> > > multiple GRIB matches for "AOTK/A0", but hopefully using the "-
paired"
> > > option will help.
> > >
> > > Thanks,
> > > John
> > >
> >
> >
> >
> > --
> > Partha S. Bhattacharjee, Ph.D.
> > I.M. Systems Group
> > Environmental Modeling Centre
> > NCWCP
> > 5830 University Research Court.
> > College Park, MD 20740
> >
> >
>
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
More information about the Met_help
mailing list