[Met_help] [rt.rap.ucar.edu #97672] History for grid-diag functionality

Tue Dec 22 12:36:47 MST 2020

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi George,
I am running grid-diag using the script found here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh

It all appears to be running to completion but the functionality is not what I was expecting. To test the tool, I first set the script to run one day (INIT_BEG/END=2020061500) and it created two file lists (idx0/1) that I would have expected that included that one day. For example: 
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
I did the same for all my single days. It created the files I would expect. An example netcdf can be seen here:
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-21.nc

Then, I wanted to run something similar for all my available data (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400) but I would like to have all days aggregated together and one netcdf file created in the end. It did create one netcdf file at the end, but it only included the data for the last date (the 15th).
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-2020061500_f00-21.nc
If you diff the two nc files 
grid_diag_out_GFS_20200615_f00-21.nc and grid_diag_out_GFS_2020060800-2020061500_f00-21.nc
they are identical.

I could be confused on the intent of the tool. Perhaps this is the behavior you want it to have. If so, I still think there is a bug because it should be outputting a file per day, and it appears to only output a file at the end for the last day.

What I thought would happen is I would get one file, but it would include all days/times requested. So, what I need it to do is create a list that includes all model (idx0) and obs (idx1) files that I want to use in one file. I can do this by hand and I am quite certain it would run and do what I want, but I can't get it to work in an automated fashion. (Essentially I want it to do something similar to series/stat-analysis, for example - aggregate over time)

It is very possible I am doing something wrong on my end. I hope so as that would be an easy fix. If this doesn't make sense or if you would like to discuss this on the phone at some point, I am happy to do so. Just let me know what you think. Thanks in advance for your help. If my dir structure is confusing or you need me to send you anything else please don't hesitate to ask.

Thanks!
Jamie

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: grid-diag functionality
From: George McCabe
Time: Mon Nov 30 15:42:28 2020

Hi Jamie,

I moved this email over to MET Help.

The configuration you have set up loops over each run time in the
range you specified (2020060800 to 2020061500 with 24 hour increments
and processing the list of forecast leads for each time). However, the
output template will be the same for each run:

GRID_DIAG_OUTPUT_TEMPLATE = grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc

This is why you are seeing only the results of the last run in the
output. Each run overwrites the data from the previous run.

Wrappers like SeriesAnalysis and StatAnalysis are designed to process
a range of times at once, but GridDiag was written to process once for
each time. However, this may not be the intended way to use the tool.
This is how I implemented it based on the example I was provided.

John HG (copied here) may have some insight on the best way to achieve
what you are trying to do. Perhaps you would want to run GridDiag for
each run time, then run another tool to aggregate the stats across all
of the output.

If that is not the case, then you should be able to get the wrapper to
run the way you are intending by configuring it to process a single
run time (INIT_BEG==INIT_END) and expand the list of values for
GRID_DIAG_INPUT_TEMPLATE to include the times you need using the
'shift' keyword in the filename templates, i.e.

{init?fmt=...}, {init?fmt=...?shift=24H}, {init?fmt=...?shift=48H},
and so on.

This approach is not very flexible and would require you to make a lot
of changes to the METplus config file for each range of times you
desire. If this is the only way to achieve what you need, then we
should make changes to the wrapper to allow you to configure this more
easily.

Thanks,
George

On Mon Nov 30 15:27:50 2020, mccabe wrote:
> Hi George,
> I am running grid-diag using the script found
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
>
> It all appears to be running to completion but the functionality is
> not what I was expecting. To test the tool, I first set the script
to
> run one day (INIT_BEG/END=2020061500) and it created two file lists
> (idx0/1) that I would have expected that included that one day. For
> example:
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> I did the same for all my single days. It created the files I would
> expect. An example netcdf can be seen here:
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> 21.nc
>
> Then, I wanted to run something similar for all my available data
> (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400) but I
would
> like to have all days aggregated together and one netcdf file
created
> in the end. It did create one netcdf file at the end, but it only
> included the data for the last date (the 15th).
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> 2020061500_f00-21.nc
>  If you diff the two nc files
> grid_diag_out_GFS_20200615_f00-21.nc and
grid_diag_out_GFS_2020060800-
> 2020061500_f00-21.nc
> they are identical.
>
> I could be confused on the intent of the tool. Perhaps this is the
> behavior you want it to have. If so, I still think there is a bug
> because it should be outputting a file per day, and it appears to
only
> output a file at the end for the last day.
>
> What I thought would happen is I would get one file, but it would
> include all days/times requested. So, what I need it to do is create
a
> list that includes all model (idx0) and obs (idx1) files that I want
> to use in one file. I can do this by hand and I am quite certain it
> would run and do what I want, but I can't get it to work in an
> automated fashion. (Essentially I want it to do something similar to
> series/stat-analysis, for example - aggregate over time)
>
> It is very possible I am doing something wrong on my end. I hope so
as
> that would be an easy fix. If this doesn't make sense or if you
would
> like to discuss this on the phone at some point, I am happy to do
so.
> Just let me know what you think. Thanks in advance for your help. If
> my dir structure is confusing or you need me to send you anything
else
> please don't hesitate to ask.
>
> Thanks!
> Jamie

------------------------------------------------
Subject: grid-diag functionality
From: John Halley Gotway
Time: Mon Nov 30 16:26:40 2020

George,

The logic for running the Grid-Diag is more like Series-Analysis and
Stat-Analysis than it is like Grid-Stat. While a user could run it for
a
single time, it is more likely they'll want to run it over a set of
times.
It really is pretty analogous to Series-Analysis... you may want to
hold
the init time fixed and run over all lead times... or hold the lead
time
fixed and run over all init times... or just run over all init and
lead
times in one big batch.

George, is there any way to generalize the support for how times are
processed?

Thanks,
John

On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> Hi Jamie,
>
> I moved this email over to MET Help.
>
> The configuration you have set up loops over each run time in the
range
> you specified (2020060800 to 2020061500 with 24 hour increments and
> processing the list of forecast leads for each time). However, the
output
> template will be the same for each run:
>
> GRID_DIAG_OUTPUT_TEMPLATE =
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
>
> This is why you are seeing only the results of the last run in the
output.
> Each run overwrites the data from the previous run.
>
> Wrappers like SeriesAnalysis and StatAnalysis are designed to
process a
> range of times at once, but GridDiag was written to process once for
each
> time. However, this may not be the intended way to use the tool.
This is
> how I implemented it based on the example I was provided.
>
> John HG (copied here) may have some insight on the best way to
achieve
> what you are trying to do. Perhaps you would want to run GridDiag
for each
> run time, then run another tool to aggregate the stats across all of
the
> output.
>
> If that is not the case, then you should be able to get the wrapper
to run
> the way you are intending by configuring it to process a single run
time
> (INIT_BEG==INIT_END) and expand the list of values for
> GRID_DIAG_INPUT_TEMPLATE to include the times you need using the
'shift'
> keyword in the filename templates, i.e.
>
> {init?fmt=...}, {init?fmt=...?shift=24H}, {init?fmt=...?shift=48H},
and so
> on.
>
> This approach is not very flexible and would require you to make a
lot of
> changes to the METplus config file for each range of times you
desire. If
> this is the only way to achieve what you need, then we should make
changes
> to the wrapper to allow you to configure this more easily.
>
> Thanks,
> George
>
> On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > Hi George,
> > I am running grid-diag using the script found
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> >
> > It all appears to be running to completion but the functionality
is
> > not what I was expecting. To test the tool, I first set the script
to
> > run one day (INIT_BEG/END=2020061500) and it created two file
lists
> > (idx0/1) that I would have expected that included that one day.
For
> > example:
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > I did the same for all my single days. It created the files I
would
> > expect. An example netcdf can be seen here:
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > 21.nc
> >
> > Then, I wanted to run something similar for all my available data
> > (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400) but I
would
> > like to have all days aggregated together and one netcdf file
created
> > in the end. It did create one netcdf file at the end, but it only
> > included the data for the last date (the 15th).
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > 2020061500_f00-21.nc
> >  If you diff the two nc files
> > grid_diag_out_GFS_20200615_f00-21.nc and
grid_diag_out_GFS_2020060800-
> > 2020061500_f00-21.nc
> > they are identical.
> >
> > I could be confused on the intent of the tool. Perhaps this is the
> > behavior you want it to have. If so, I still think there is a bug
> > because it should be outputting a file per day, and it appears to
only
> > output a file at the end for the last day.
> >
> > What I thought would happen is I would get one file, but it would
> > include all days/times requested. So, what I need it to do is
create a
> > list that includes all model (idx0) and obs (idx1) files that I
want
> > to use in one file. I can do this by hand and I am quite certain
it
> > would run and do what I want, but I can't get it to work in an
> > automated fashion. (Essentially I want it to do something similar
to
> > series/stat-analysis, for example - aggregate over time)
> >
> > It is very possible I am doing something wrong on my end. I hope
so as
> > that would be an easy fix. If this doesn't make sense or if you
would
> > like to discuss this on the phone at some point, I am happy to do
so.
> > Just let me know what you think. Thanks in advance for your help.
If
> > my dir structure is confusing or you need me to send you anything
else
> > please don't hesitate to ask.
> >
> > Thanks!
> > Jamie
>
>
>
>

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Nov 30 16:43:20 2020

OK, it sounds like I misunderstood the intended use for the tool when
I
implemented the wrapper. Thanks for pointing this out, Jamie!

John, I think we should have a discussion about how we would want to
have
tools like SeriesAnalysis and GridDiag to be configured to run in
these
cases. I'll follow up with you to set something up. SeriesAnalysis is
currently separated into SeriesByInit and SeriesByLead which perform
the
grouping. However, I am currently refactoring these tools to hopefully
combine the logic and allow options to specify different ways to
filter/group the data. This would be a good time to come up with a
good way
to define how you want to group your data in a general way that could
apply
to other similar wrappers, like GridDiag.

Jamie, to confirm what you are saying, you expected the list of init
times
and forecast leads to determine which files to use in a single
analysis. Is
that correct?

Thanks,
George

On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> George,
>
> The logic for running the Grid-Diag is more like Series-Analysis and
> Stat-Analysis than it is like Grid-Stat. While a user could run it
for a
> single time, it is more likely they'll want to run it over a set of
times.
> It really is pretty analogous to Series-Analysis... you may want to
hold
> the init time fixed and run over all lead times... or hold the lead
time
> fixed and run over all init times... or just run over all init and
lead
> times in one big batch.
>
> George, is there any way to generalize the support for how times are
> processed?
>
> Thanks,
> John
>
> On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > Hi Jamie,
> >
> > I moved this email over to MET Help.
> >
> > The configuration you have set up loops over each run time in the
range
> > you specified (2020060800 to 2020061500 with 24 hour increments
and
> > processing the list of forecast leads for each time). However, the
output
> > template will be the same for each run:
> >
> > GRID_DIAG_OUTPUT_TEMPLATE =
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> >
> > This is why you are seeing only the results of the last run in the
> output.
> > Each run overwrites the data from the previous run.
> >
> > Wrappers like SeriesAnalysis and StatAnalysis are designed to
process a
> > range of times at once, but GridDiag was written to process once
for each
> > time. However, this may not be the intended way to use the tool.
This is
> > how I implemented it based on the example I was provided.
> >
> > John HG (copied here) may have some insight on the best way to
achieve
> > what you are trying to do. Perhaps you would want to run GridDiag
for
> each
> > run time, then run another tool to aggregate the stats across all
of the
> > output.
> >
> > If that is not the case, then you should be able to get the
wrapper to
> run
> > the way you are intending by configuring it to process a single
run time
> > (INIT_BEG==INIT_END) and expand the list of values for
> > GRID_DIAG_INPUT_TEMPLATE to include the times you need using the
'shift'
> > keyword in the filename templates, i.e.
> >
> > {init?fmt=...}, {init?fmt=...?shift=24H},
{init?fmt=...?shift=48H}, and
> so
> > on.
> >
> > This approach is not very flexible and would require you to make a
lot of
> > changes to the METplus config file for each range of times you
desire. If
> > this is the only way to achieve what you need, then we should make
> changes
> > to the wrapper to allow you to configure this more easily.
> >
> > Thanks,
> > George
> >
> > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > Hi George,
> > > I am running grid-diag using the script found
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > >
> > > It all appears to be running to completion but the functionality
is
> > > not what I was expecting. To test the tool, I first set the
script to
> > > run one day (INIT_BEG/END=2020061500) and it created two file
lists
> > > (idx0/1) that I would have expected that included that one day.
For
> > > example:
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > I did the same for all my single days. It created the files I
would
> > > expect. An example netcdf can be seen here:
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > 21.nc
> > >
> > > Then, I wanted to run something similar for all my available
data
> > > (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400) but I
would
> > > like to have all days aggregated together and one netcdf file
created
> > > in the end. It did create one netcdf file at the end, but it
only
> > > included the data for the last date (the 15th).
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > 2020061500_f00-21.nc
> > >  If you diff the two nc files
> > > grid_diag_out_GFS_20200615_f00-21.nc and
grid_diag_out_GFS_2020060800-
> > > 2020061500_f00-21.nc
> > > they are identical.
> > >
> > > I could be confused on the intent of the tool. Perhaps this is
the
> > > behavior you want it to have. If so, I still think there is a
bug
> > > because it should be outputting a file per day, and it appears
to only
> > > output a file at the end for the last day.
> > >
> > > What I thought would happen is I would get one file, but it
would
> > > include all days/times requested. So, what I need it to do is
create a
> > > list that includes all model (idx0) and obs (idx1) files that I
want
> > > to use in one file. I can do this by hand and I am quite certain
it
> > > would run and do what I want, but I can't get it to work in an
> > > automated fashion. (Essentially I want it to do something
similar to
> > > series/stat-analysis, for example - aggregate over time)
> > >
> > > It is very possible I am doing something wrong on my end. I hope
so as
> > > that would be an easy fix. If this doesn't make sense or if you
would
> > > like to discuss this on the phone at some point, I am happy to
do so.
> > > Just let me know what you think. Thanks in advance for your
help. If
> > > my dir structure is confusing or you need me to send you
anything else
> > > please don't hesitate to ask.
> > >
> > > Thanks!
> > > Jamie
> >
> >
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Nov 30 17:31:30 2020

Hi George and John,
Sounds like the modifications you are discussing will address my need
(correct - the list of init times
and forecast leads to determine which files to use in a single
analysis).
Thanks!
Jamie

On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> OK, it sounds like I misunderstood the intended use for the tool
when I
> implemented the wrapper. Thanks for pointing this out, Jamie!
>
> John, I think we should have a discussion about how we would want to
have
> tools like SeriesAnalysis and GridDiag to be configured to run in
these
> cases. I'll follow up with you to set something up. SeriesAnalysis
is
> currently separated into SeriesByInit and SeriesByLead which perform
the
> grouping. However, I am currently refactoring these tools to
hopefully
> combine the logic and allow options to specify different ways to
> filter/group the data. This would be a good time to come up with a
good way
> to define how you want to group your data in a general way that
could apply
> to other similar wrappers, like GridDiag.
>
> Jamie, to confirm what you are saying, you expected the list of init
times
> and forecast leads to determine which files to use in a single
analysis. Is
> that correct?
>
> Thanks,
> George
>
> On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > George,
> >
> > The logic for running the Grid-Diag is more like Series-Analysis
and
> > Stat-Analysis than it is like Grid-Stat. While a user could run it
for a
> > single time, it is more likely they'll want to run it over a set
of
> times.
> > It really is pretty analogous to Series-Analysis... you may want
to hold
> > the init time fixed and run over all lead times... or hold the
lead time
> > fixed and run over all init times... or just run over all init and
lead
> > times in one big batch.
> >
> > George, is there any way to generalize the support for how times
are
> > processed?
> >
> > Thanks,
> > John
> >
> > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > Hi Jamie,
> > >
> > > I moved this email over to MET Help.
> > >
> > > The configuration you have set up loops over each run time in
the range
> > > you specified (2020060800 to 2020061500 with 24 hour increments
and
> > > processing the list of forecast leads for each time). However,
the
> output
> > > template will be the same for each run:
> > >
> > > GRID_DIAG_OUTPUT_TEMPLATE =
> > >
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > >
> > > This is why you are seeing only the results of the last run in
the
> > output.
> > > Each run overwrites the data from the previous run.
> > >
> > > Wrappers like SeriesAnalysis and StatAnalysis are designed to
process a
> > > range of times at once, but GridDiag was written to process once
for
> each
> > > time. However, this may not be the intended way to use the tool.
This
> is
> > > how I implemented it based on the example I was provided.
> > >
> > > John HG (copied here) may have some insight on the best way to
achieve
> > > what you are trying to do. Perhaps you would want to run
GridDiag for
> > each
> > > run time, then run another tool to aggregate the stats across
all of
> the
> > > output.
> > >
> > > If that is not the case, then you should be able to get the
wrapper to
> > run
> > > the way you are intending by configuring it to process a single
run
> time
> > > (INIT_BEG==INIT_END) and expand the list of values for
> > > GRID_DIAG_INPUT_TEMPLATE to include the times you need using the
> 'shift'
> > > keyword in the filename templates, i.e.
> > >
> > > {init?fmt=...}, {init?fmt=...?shift=24H},
{init?fmt=...?shift=48H}, and
> > so
> > > on.
> > >
> > > This approach is not very flexible and would require you to make
a lot
> of
> > > changes to the METplus config file for each range of times you
desire.
> If
> > > this is the only way to achieve what you need, then we should
make
> > changes
> > > to the wrapper to allow you to configure this more easily.
> > >
> > > Thanks,
> > > George
> > >
> > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > Hi George,
> > > > I am running grid-diag using the script found
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > >
> > > > It all appears to be running to completion but the
functionality is
> > > > not what I was expecting. To test the tool, I first set the
script to
> > > > run one day (INIT_BEG/END=2020061500) and it created two file
lists
> > > > (idx0/1) that I would have expected that included that one
day. For
> > > > example:
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > I did the same for all my single days. It created the files I
would
> > > > expect. An example netcdf can be seen here:
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > 21.nc
> > > >
> > > > Then, I wanted to run something similar for all my available
data
> > > > (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400) but
I
> would
> > > > like to have all days aggregated together and one netcdf file
created
> > > > in the end. It did create one netcdf file at the end, but it
only
> > > > included the data for the last date (the 15th).
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > 2020061500_f00-21.nc
> > > >  If you diff the two nc files
> > > > grid_diag_out_GFS_20200615_f00-21.nc and
> grid_diag_out_GFS_2020060800-
> > > > 2020061500_f00-21.nc
> > > > they are identical.
> > > >
> > > > I could be confused on the intent of the tool. Perhaps this is
the
> > > > behavior you want it to have. If so, I still think there is a
bug
> > > > because it should be outputting a file per day, and it appears
to
> only
> > > > output a file at the end for the last day.
> > > >
> > > > What I thought would happen is I would get one file, but it
would
> > > > include all days/times requested. So, what I need it to do is
create
> a
> > > > list that includes all model (idx0) and obs (idx1) files that
I want
> > > > to use in one file. I can do this by hand and I am quite
certain it
> > > > would run and do what I want, but I can't get it to work in an
> > > > automated fashion. (Essentially I want it to do something
similar to
> > > > series/stat-analysis, for example - aggregate over time)
> > > >
> > > > It is very possible I am doing something wrong on my end. I
hope so
> as
> > > > that would be an easy fix. If this doesn't make sense or if
you would
> > > > like to discuss this on the phone at some point, I am happy to
do so.
> > > > Just let me know what you think. Thanks in advance for your
help. If
> > > > my dir structure is confusing or you need me to send you
anything
> else
> > > > please don't hesitate to ask.
> > > >
> > > > Thanks!
> > > > Jamie
> > >
> > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 14 10:36:58 2020

Hi Jamie,

I created a GitHub issue for this work:

https://github.com/dtcenter/METplus/issues/733

I have implemented functionality to call certain wrappers in different
intervals to process groups of files instead of just once per file. I
am now working on incorporating this functionality into the GridDiag
wrapper. If you don't mind, I was planning on assigning the pull
request review to you so you can test these changes on your use case
to ensure it runs the way you expect.

Thanks,
George

On Mon Nov 30 17:31:30 2020, jwolff wrote:
> Hi George and John,
> Sounds like the modifications you are discussing will address my
need
> (correct - the list of init times
> and forecast leads to determine which files to use in a single
> analysis).
> Thanks!
> Jamie
>
> On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> <met_help at ucar.edu>
> wrote:
>
> > OK, it sounds like I misunderstood the intended use for the tool
when
> > I
> > implemented the wrapper. Thanks for pointing this out, Jamie!
> >
> > John, I think we should have a discussion about how we would want
to
> > have
> > tools like SeriesAnalysis and GridDiag to be configured to run in
> > these
> > cases. I'll follow up with you to set something up. SeriesAnalysis
is
> > currently separated into SeriesByInit and SeriesByLead which
perform
> > the
> > grouping. However, I am currently refactoring these tools to
> > hopefully
> > combine the logic and allow options to specify different ways to
> > filter/group the data. This would be a good time to come up with a
> > good way
> > to define how you want to group your data in a general way that
could
> > apply
> > to other similar wrappers, like GridDiag.
> >
> > Jamie, to confirm what you are saying, you expected the list of
init
> > times
> > and forecast leads to determine which files to use in a single
> > analysis. Is
> > that correct?
> >
> > Thanks,
> > George
> >
> > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > George,
> > >
> > > The logic for running the Grid-Diag is more like Series-Analysis
> > > and
> > > Stat-Analysis than it is like Grid-Stat. While a user could run
it
> > > for a
> > > single time, it is more likely they'll want to run it over a set
of
> > times.
> > > It really is pretty analogous to Series-Analysis... you may want
to
> > > hold
> > > the init time fixed and run over all lead times... or hold the
lead
> > > time
> > > fixed and run over all init times... or just run over all init
and
> > > lead
> > > times in one big batch.
> > >
> > > George, is there any way to generalize the support for how times
> > > are
> > > processed?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > Hi Jamie,
> > > >
> > > > I moved this email over to MET Help.
> > > >
> > > > The configuration you have set up loops over each run time in
the
> > > > range
> > > > you specified (2020060800 to 2020061500 with 24 hour
increments
> > > > and
> > > > processing the list of forecast leads for each time). However,
> > > > the
> > output
> > > > template will be the same for each run:
> > > >
> > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > >
> > >
> > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > >
> > > > This is why you are seeing only the results of the last run in
> > > > the
> > > output.
> > > > Each run overwrites the data from the previous run.
> > > >
> > > > Wrappers like SeriesAnalysis and StatAnalysis are designed to
> > > > process a
> > > > range of times at once, but GridDiag was written to process
once
> > > > for
> > each
> > > > time. However, this may not be the intended way to use the
tool.
> > > > This
> > is
> > > > how I implemented it based on the example I was provided.
> > > >
> > > > John HG (copied here) may have some insight on the best way to
> > > > achieve
> > > > what you are trying to do. Perhaps you would want to run
GridDiag
> > > > for
> > > each
> > > > run time, then run another tool to aggregate the stats across
all
> > > > of
> > the
> > > > output.
> > > >
> > > > If that is not the case, then you should be able to get the
> > > > wrapper to
> > > run
> > > > the way you are intending by configuring it to process a
single
> > > > run
> > time
> > > > (INIT_BEG==INIT_END) and expand the list of values for
> > > > GRID_DIAG_INPUT_TEMPLATE to include the times you need using
the
> > 'shift'
> > > > keyword in the filename templates, i.e.
> > > >
> > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > {init?fmt=...?shift=48H}, and
> > > so
> > > > on.
> > > >
> > > > This approach is not very flexible and would require you to
make
> > > > a lot
> > of
> > > > changes to the METplus config file for each range of times you
> > > > desire.
> > If
> > > > this is the only way to achieve what you need, then we should
> > > > make
> > > changes
> > > > to the wrapper to allow you to configure this more easily.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > Hi George,
> > > > > I am running grid-diag using the script found
> > > > >
> > > >
> > >
> >
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > >
> > > > > It all appears to be running to completion but the
> > > > > functionality is
> > > > > not what I was expecting. To test the tool, I first set the
> > > > > script to
> > > > > run one day (INIT_BEG/END=2020061500) and it created two
file
> > > > > lists
> > > > > (idx0/1) that I would have expected that included that one
day.
> > > > > For
> > > > > example:
> > > > >
> > > >
> > >
> >
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > I did the same for all my single days. It created the files
I
> > > > > would
> > > > > expect. An example netcdf can be seen here:
> > > > >
> > > >
> > >
> >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > 21.nc
> > > > >
> > > > > Then, I wanted to run something similar for all my available
> > > > > data
> > > > > (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400)
but
> > > > > I
> > would
> > > > > like to have all days aggregated together and one netcdf
file
> > > > > created
> > > > > in the end. It did create one netcdf file at the end, but it
> > > > > only
> > > > > included the data for the last date (the 15th).
> > > > >
> > > >
> > >
> >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > 2020061500_f00-21.nc
> > > > >  If you diff the two nc files
> > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > grid_diag_out_GFS_2020060800-
> > > > > 2020061500_f00-21.nc
> > > > > they are identical.
> > > > >
> > > > > I could be confused on the intent of the tool. Perhaps this
is
> > > > > the
> > > > > behavior you want it to have. If so, I still think there is
a
> > > > > bug
> > > > > because it should be outputting a file per day, and it
appears
> > > > > to
> > only
> > > > > output a file at the end for the last day.
> > > > >
> > > > > What I thought would happen is I would get one file, but it
> > > > > would
> > > > > include all days/times requested. So, what I need it to do
is
> > > > > create
> > a
> > > > > list that includes all model (idx0) and obs (idx1) files
that I
> > > > > want
> > > > > to use in one file. I can do this by hand and I am quite
> > > > > certain it
> > > > > would run and do what I want, but I can't get it to work in
an
> > > > > automated fashion. (Essentially I want it to do something
> > > > > similar to
> > > > > series/stat-analysis, for example - aggregate over time)
> > > > >
> > > > > It is very possible I am doing something wrong on my end. I
> > > > > hope so
> > as
> > > > > that would be an easy fix. If this doesn't make sense or if
you
> > > > > would
> > > > > like to discuss this on the phone at some point, I am happy
to
> > > > > do so.
> > > > > Just let me know what you think. Thanks in advance for your
> > > > > help. If
> > > > > my dir structure is confusing or you need me to send you
> > > > > anything
> > else
> > > > > please don't hesitate to ask.
> > > > >
> > > > > Thanks!
> > > > > Jamie
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
> > obliged to
> > reply to this email outside of your normal working hours.
> >
> >

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 14 10:51:07 2020

Sure. No problem. Thank you!
Jamie

On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Jamie,
>
> I created a GitHub issue for this work:
>
> https://github.com/dtcenter/METplus/issues/733
>
> I have implemented functionality to call certain wrappers in
different
> intervals to process groups of files instead of just once per file.
I am
> now working on incorporating this functionality into the GridDiag
wrapper.
> If you don't mind, I was planning on assigning the pull request
review to
> you so you can test these changes on your use case to ensure it runs
the
> way you expect.
>
> Thanks,
> George
>
> On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > Hi George and John,
> > Sounds like the modifications you are discussing will address my
need
> > (correct - the list of init times
> > and forecast leads to determine which files to use in a single
> > analysis).
> > Thanks!
> > Jamie
> >
> > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > OK, it sounds like I misunderstood the intended use for the tool
when
> > > I
> > > implemented the wrapper. Thanks for pointing this out, Jamie!
> > >
> > > John, I think we should have a discussion about how we would
want to
> > > have
> > > tools like SeriesAnalysis and GridDiag to be configured to run
in
> > > these
> > > cases. I'll follow up with you to set something up.
SeriesAnalysis is
> > > currently separated into SeriesByInit and SeriesByLead which
perform
> > > the
> > > grouping. However, I am currently refactoring these tools to
> > > hopefully
> > > combine the logic and allow options to specify different ways to
> > > filter/group the data. This would be a good time to come up with
a
> > > good way
> > > to define how you want to group your data in a general way that
could
> > > apply
> > > to other similar wrappers, like GridDiag.
> > >
> > > Jamie, to confirm what you are saying, you expected the list of
init
> > > times
> > > and forecast leads to determine which files to use in a single
> > > analysis. Is
> > > that correct?
> > >
> > > Thanks,
> > > George
> > >
> > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > George,
> > > >
> > > > The logic for running the Grid-Diag is more like Series-
Analysis
> > > > and
> > > > Stat-Analysis than it is like Grid-Stat. While a user could
run it
> > > > for a
> > > > single time, it is more likely they'll want to run it over a
set of
> > > times.
> > > > It really is pretty analogous to Series-Analysis... you may
want to
> > > > hold
> > > > the init time fixed and run over all lead times... or hold the
lead
> > > > time
> > > > fixed and run over all init times... or just run over all init
and
> > > > lead
> > > > times in one big batch.
> > > >
> > > > George, is there any way to generalize the support for how
times
> > > > are
> > > > processed?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > Hi Jamie,
> > > > >
> > > > > I moved this email over to MET Help.
> > > > >
> > > > > The configuration you have set up loops over each run time
in the
> > > > > range
> > > > > you specified (2020060800 to 2020061500 with 24 hour
increments
> > > > > and
> > > > > processing the list of forecast leads for each time).
However,
> > > > > the
> > > output
> > > > > template will be the same for each run:
> > > > >
> > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > >
> > > >
> > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > >
> > > > > This is why you are seeing only the results of the last run
in
> > > > > the
> > > > output.
> > > > > Each run overwrites the data from the previous run.
> > > > >
> > > > > Wrappers like SeriesAnalysis and StatAnalysis are designed
to
> > > > > process a
> > > > > range of times at once, but GridDiag was written to process
once
> > > > > for
> > > each
> > > > > time. However, this may not be the intended way to use the
tool.
> > > > > This
> > > is
> > > > > how I implemented it based on the example I was provided.
> > > > >
> > > > > John HG (copied here) may have some insight on the best way
to
> > > > > achieve
> > > > > what you are trying to do. Perhaps you would want to run
GridDiag
> > > > > for
> > > > each
> > > > > run time, then run another tool to aggregate the stats
across all
> > > > > of
> > > the
> > > > > output.
> > > > >
> > > > > If that is not the case, then you should be able to get the
> > > > > wrapper to
> > > > run
> > > > > the way you are intending by configuring it to process a
single
> > > > > run
> > > time
> > > > > (INIT_BEG==INIT_END) and expand the list of values for
> > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you need using
the
> > > 'shift'
> > > > > keyword in the filename templates, i.e.
> > > > >
> > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > {init?fmt=...?shift=48H}, and
> > > > so
> > > > > on.
> > > > >
> > > > > This approach is not very flexible and would require you to
make
> > > > > a lot
> > > of
> > > > > changes to the METplus config file for each range of times
you
> > > > > desire.
> > > If
> > > > > this is the only way to achieve what you need, then we
should
> > > > > make
> > > > changes
> > > > > to the wrapper to allow you to configure this more easily.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > Hi George,
> > > > > > I am running grid-diag using the script found
> > > > > >
> > > > >
> > > >
> > >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > >
> > > > > > It all appears to be running to completion but the
> > > > > > functionality is
> > > > > > not what I was expecting. To test the tool, I first set
the
> > > > > > script to
> > > > > > run one day (INIT_BEG/END=2020061500) and it created two
file
> > > > > > lists
> > > > > > (idx0/1) that I would have expected that included that one
day.
> > > > > > For
> > > > > > example:
> > > > > >
> > > > >
> > > >
> > >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > I did the same for all my single days. It created the
files I
> > > > > > would
> > > > > > expect. An example netcdf can be seen here:
> > > > > >
> > > > >
> > > >
> > >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > 21.nc
> > > > > >
> > > > > > Then, I wanted to run something similar for all my
available
> > > > > > data
> > > > > > (INIT_BEG=2020060800, INIT_END=2020061500, INIT_INC=86400)
but
> > > > > > I
> > > would
> > > > > > like to have all days aggregated together and one netcdf
file
> > > > > > created
> > > > > > in the end. It did create one netcdf file at the end, but
it
> > > > > > only
> > > > > > included the data for the last date (the 15th).
> > > > > >
> > > > >
> > > >
> > >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > 2020061500_f00-21.nc
> > > > > >  If you diff the two nc files
> > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > grid_diag_out_GFS_2020060800-
> > > > > > 2020061500_f00-21.nc
> > > > > > they are identical.
> > > > > >
> > > > > > I could be confused on the intent of the tool. Perhaps
this is
> > > > > > the
> > > > > > behavior you want it to have. If so, I still think there
is a
> > > > > > bug
> > > > > > because it should be outputting a file per day, and it
appears
> > > > > > to
> > > only
> > > > > > output a file at the end for the last day.
> > > > > >
> > > > > > What I thought would happen is I would get one file, but
it
> > > > > > would
> > > > > > include all days/times requested. So, what I need it to do
is
> > > > > > create
> > > a
> > > > > > list that includes all model (idx0) and obs (idx1) files
that I
> > > > > > want
> > > > > > to use in one file. I can do this by hand and I am quite
> > > > > > certain it
> > > > > > would run and do what I want, but I can't get it to work
in an
> > > > > > automated fashion. (Essentially I want it to do something
> > > > > > similar to
> > > > > > series/stat-analysis, for example - aggregate over time)
> > > > > >
> > > > > > It is very possible I am doing something wrong on my end.
I
> > > > > > hope so
> > > as
> > > > > > that would be an easy fix. If this doesn't make sense or
if you
> > > > > > would
> > > > > > like to discuss this on the phone at some point, I am
happy to
> > > > > > do so.
> > > > > > Just let me know what you think. Thanks in advance for
your
> > > > > > help. If
> > > > > > my dir structure is confusing or you need me to send you
> > > > > > anything
> > > else
> > > > > > please don't hesitate to ask.
> > > > > >
> > > > > > Thanks!
> > > > > > Jamie
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
> > >
>
>
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 14 16:26:08 2020

Hi Jamie,

I implemented the changes to GridDiag that were necessary to run the
tool as you expected. I assigned you to the pull request:

https://github.com/dtcenter/METplus/pull/735

The pull request has notes regarding which branch to checkout on
cheyenne and which setting to add to get it to work. Let me know if
you have any questions!

Thanks,
George

On Mon Dec 14 10:51:07 2020, jwolff wrote:
> Sure. No problem. Thank you!
> Jamie
>
> On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Jamie,
> >
> > I created a GitHub issue for this work:
> >
> > https://github.com/dtcenter/METplus/issues/733
> >
> > I have implemented functionality to call certain wrappers in
> > different
> > intervals to process groups of files instead of just once per
file. I
> > am
> > now working on incorporating this functionality into the GridDiag
> > wrapper.
> > If you don't mind, I was planning on assigning the pull request
> > review to
> > you so you can test these changes on your use case to ensure it
runs
> > the
> > way you expect.
> >
> > Thanks,
> > George
> >
> > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > Hi George and John,
> > > Sounds like the modifications you are discussing will address my
> > > need
> > > (correct - the list of init times
> > > and forecast leads to determine which files to use in a single
> > > analysis).
> > > Thanks!
> > > Jamie
> > >
> > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > OK, it sounds like I misunderstood the intended use for the
tool
> > > > when
> > > > I
> > > > implemented the wrapper. Thanks for pointing this out, Jamie!
> > > >
> > > > John, I think we should have a discussion about how we would
want
> > > > to
> > > > have
> > > > tools like SeriesAnalysis and GridDiag to be configured to run
in
> > > > these
> > > > cases. I'll follow up with you to set something up.
> > > > SeriesAnalysis is
> > > > currently separated into SeriesByInit and SeriesByLead which
> > > > perform
> > > > the
> > > > grouping. However, I am currently refactoring these tools to
> > > > hopefully
> > > > combine the logic and allow options to specify different ways
to
> > > > filter/group the data. This would be a good time to come up
with
> > > > a
> > > > good way
> > > > to define how you want to group your data in a general way
that
> > > > could
> > > > apply
> > > > to other similar wrappers, like GridDiag.
> > > >
> > > > Jamie, to confirm what you are saying, you expected the list
of
> > > > init
> > > > times
> > > > and forecast leads to determine which files to use in a single
> > > > analysis. Is
> > > > that correct?
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > George,
> > > > >
> > > > > The logic for running the Grid-Diag is more like Series-
> > > > > Analysis
> > > > > and
> > > > > Stat-Analysis than it is like Grid-Stat. While a user could
run
> > > > > it
> > > > > for a
> > > > > single time, it is more likely they'll want to run it over a
> > > > > set of
> > > > times.
> > > > > It really is pretty analogous to Series-Analysis... you may
> > > > > want to
> > > > > hold
> > > > > the init time fixed and run over all lead times... or hold
the
> > > > > lead
> > > > > time
> > > > > fixed and run over all init times... or just run over all
init
> > > > > and
> > > > > lead
> > > > > times in one big batch.
> > > > >
> > > > > George, is there any way to generalize the support for how
> > > > > times
> > > > > are
> > > > > processed?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > >
> > > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > I moved this email over to MET Help.
> > > > > >
> > > > > > The configuration you have set up loops over each run time
in
> > > > > > the
> > > > > > range
> > > > > > you specified (2020060800 to 2020061500 with 24 hour
> > > > > > increments
> > > > > > and
> > > > > > processing the list of forecast leads for each time).
> > > > > > However,
> > > > > > the
> > > > output
> > > > > > template will be the same for each run:
> > > > > >
> > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > >
> > > > >
> > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > >
> > > > > > This is why you are seeing only the results of the last
run
> > > > > > in
> > > > > > the
> > > > > output.
> > > > > > Each run overwrites the data from the previous run.
> > > > > >
> > > > > > Wrappers like SeriesAnalysis and StatAnalysis are designed
to
> > > > > > process a
> > > > > > range of times at once, but GridDiag was written to
process
> > > > > > once
> > > > > > for
> > > > each
> > > > > > time. However, this may not be the intended way to use the
> > > > > > tool.
> > > > > > This
> > > > is
> > > > > > how I implemented it based on the example I was provided.
> > > > > >
> > > > > > John HG (copied here) may have some insight on the best
way
> > > > > > to
> > > > > > achieve
> > > > > > what you are trying to do. Perhaps you would want to run
> > > > > > GridDiag
> > > > > > for
> > > > > each
> > > > > > run time, then run another tool to aggregate the stats
across
> > > > > > all
> > > > > > of
> > > > the
> > > > > > output.
> > > > > >
> > > > > > If that is not the case, then you should be able to get
the
> > > > > > wrapper to
> > > > > run
> > > > > > the way you are intending by configuring it to process a
> > > > > > single
> > > > > > run
> > > > time
> > > > > > (INIT_BEG==INIT_END) and expand the list of values for
> > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you need
using
> > > > > > the
> > > > 'shift'
> > > > > > keyword in the filename templates, i.e.
> > > > > >
> > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > {init?fmt=...?shift=48H}, and
> > > > > so
> > > > > > on.
> > > > > >
> > > > > > This approach is not very flexible and would require you
to
> > > > > > make
> > > > > > a lot
> > > > of
> > > > > > changes to the METplus config file for each range of times
> > > > > > you
> > > > > > desire.
> > > > If
> > > > > > this is the only way to achieve what you need, then we
should
> > > > > > make
> > > > > changes
> > > > > > to the wrapper to allow you to configure this more easily.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > Hi George,
> > > > > > > I am running grid-diag using the script found
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > >
> > > > > > > It all appears to be running to completion but the
> > > > > > > functionality is
> > > > > > > not what I was expecting. To test the tool, I first set
the
> > > > > > > script to
> > > > > > > run one day (INIT_BEG/END=2020061500) and it created two
> > > > > > > file
> > > > > > > lists
> > > > > > > (idx0/1) that I would have expected that included that
one
> > > > > > > day.
> > > > > > > For
> > > > > > > example:
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > I did the same for all my single days. It created the
files
> > > > > > > I
> > > > > > > would
> > > > > > > expect. An example netcdf can be seen here:
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > 21.nc
> > > > > > >
> > > > > > > Then, I wanted to run something similar for all my
> > > > > > > available
> > > > > > > data
> > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
INIT_INC=86400)
> > > > > > > but
> > > > > > > I
> > > > would
> > > > > > > like to have all days aggregated together and one netcdf
> > > > > > > file
> > > > > > > created
> > > > > > > in the end. It did create one netcdf file at the end,
but
> > > > > > > it
> > > > > > > only
> > > > > > > included the data for the last date (the 15th).
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > 2020061500_f00-21.nc
> > > > > > >  If you diff the two nc files
> > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > grid_diag_out_GFS_2020060800-
> > > > > > > 2020061500_f00-21.nc
> > > > > > > they are identical.
> > > > > > >
> > > > > > > I could be confused on the intent of the tool. Perhaps
this
> > > > > > > is
> > > > > > > the
> > > > > > > behavior you want it to have. If so, I still think there
is
> > > > > > > a
> > > > > > > bug
> > > > > > > because it should be outputting a file per day, and it
> > > > > > > appears
> > > > > > > to
> > > > only
> > > > > > > output a file at the end for the last day.
> > > > > > >
> > > > > > > What I thought would happen is I would get one file, but
it
> > > > > > > would
> > > > > > > include all days/times requested. So, what I need it to
do
> > > > > > > is
> > > > > > > create
> > > > a
> > > > > > > list that includes all model (idx0) and obs (idx1) files
> > > > > > > that I
> > > > > > > want
> > > > > > > to use in one file. I can do this by hand and I am quite
> > > > > > > certain it
> > > > > > > would run and do what I want, but I can't get it to work
in
> > > > > > > an
> > > > > > > automated fashion. (Essentially I want it to do
something
> > > > > > > similar to
> > > > > > > series/stat-analysis, for example - aggregate over time)
> > > > > > >
> > > > > > > It is very possible I am doing something wrong on my
end. I
> > > > > > > hope so
> > > > as
> > > > > > > that would be an easy fix. If this doesn't make sense or
if
> > > > > > > you
> > > > > > > would
> > > > > > > like to discuss this on the phone at some point, I am
happy
> > > > > > > to
> > > > > > > do so.
> > > > > > > Just let me know what you think. Thanks in advance for
your
> > > > > > > help. If
> > > > > > > my dir structure is confusing or you need me to send you
> > > > > > > anything
> > > > else
> > > > > > > please don't hesitate to ask.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Jamie
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> >
> >
> >
> >

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Thu Dec 17 12:54:46 2020

Hi Jamie,

Just following up on this in case this email was buried. Let me know
if you need any help getting this to work!

- George

On Mon Dec 14 16:26:08 2020, mccabe wrote:
> Hi Jamie,
>
> I implemented the changes to GridDiag that were necessary to run the
> tool as you expected. I assigned you to the pull request:
>
> https://github.com/dtcenter/METplus/pull/735
>
> The pull request has notes regarding which branch to checkout on
> cheyenne and which setting to add to get it to work. Let me know if
> you have any questions!
>
> Thanks,
> George
>
> On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > Sure. No problem. Thank you!
> > Jamie
> >
> > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Jamie,
> > >
> > > I created a GitHub issue for this work:
> > >
> > > https://github.com/dtcenter/METplus/issues/733
> > >
> > > I have implemented functionality to call certain wrappers in
> > > different
> > > intervals to process groups of files instead of just once per
file.
> > > I
> > > am
> > > now working on incorporating this functionality into the
GridDiag
> > > wrapper.
> > > If you don't mind, I was planning on assigning the pull request
> > > review to
> > > you so you can test these changes on your use case to ensure it
> > > runs
> > > the
> > > way you expect.
> > >
> > > Thanks,
> > > George
> > >
> > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > Hi George and John,
> > > > Sounds like the modifications you are discussing will address
my
> > > > need
> > > > (correct - the list of init times
> > > > and forecast leads to determine which files to use in a single
> > > > analysis).
> > > > Thanks!
> > > > Jamie
> > > >
> > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > OK, it sounds like I misunderstood the intended use for the
> > > > > tool
> > > > > when
> > > > > I
> > > > > implemented the wrapper. Thanks for pointing this out,
Jamie!
> > > > >
> > > > > John, I think we should have a discussion about how we would
> > > > > want
> > > > > to
> > > > > have
> > > > > tools like SeriesAnalysis and GridDiag to be configured to
run
> > > > > in
> > > > > these
> > > > > cases. I'll follow up with you to set something up.
> > > > > SeriesAnalysis is
> > > > > currently separated into SeriesByInit and SeriesByLead which
> > > > > perform
> > > > > the
> > > > > grouping. However, I am currently refactoring these tools to
> > > > > hopefully
> > > > > combine the logic and allow options to specify different
ways
> > > > > to
> > > > > filter/group the data. This would be a good time to come up
> > > > > with
> > > > > a
> > > > > good way
> > > > > to define how you want to group your data in a general way
that
> > > > > could
> > > > > apply
> > > > > to other similar wrappers, like GridDiag.
> > > > >
> > > > > Jamie, to confirm what you are saying, you expected the list
of
> > > > > init
> > > > > times
> > > > > and forecast leads to determine which files to use in a
single
> > > > > analysis. Is
> > > > > that correct?
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > >
> > > > > >
> > > > > > George,
> > > > > >
> > > > > > The logic for running the Grid-Diag is more like Series-
> > > > > > Analysis
> > > > > > and
> > > > > > Stat-Analysis than it is like Grid-Stat. While a user
could
> > > > > > run
> > > > > > it
> > > > > > for a
> > > > > > single time, it is more likely they'll want to run it over
a
> > > > > > set of
> > > > > times.
> > > > > > It really is pretty analogous to Series-Analysis... you
may
> > > > > > want to
> > > > > > hold
> > > > > > the init time fixed and run over all lead times... or hold
> > > > > > the
> > > > > > lead
> > > > > > time
> > > > > > fixed and run over all init times... or just run over all
> > > > > > init
> > > > > > and
> > > > > > lead
> > > > > > times in one big batch.
> > > > > >
> > > > > > George, is there any way to generalize the support for how
> > > > > > times
> > > > > > are
> > > > > > processed?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > I moved this email over to MET Help.
> > > > > > >
> > > > > > > The configuration you have set up loops over each run
time
> > > > > > > in
> > > > > > > the
> > > > > > > range
> > > > > > > you specified (2020060800 to 2020061500 with 24 hour
> > > > > > > increments
> > > > > > > and
> > > > > > > processing the list of forecast leads for each time).
> > > > > > > However,
> > > > > > > the
> > > > > output
> > > > > > > template will be the same for each run:
> > > > > > >
> > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > >
> > > > > >
> > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > >
> > > > > > > This is why you are seeing only the results of the last
run
> > > > > > > in
> > > > > > > the
> > > > > > output.
> > > > > > > Each run overwrites the data from the previous run.
> > > > > > >
> > > > > > > Wrappers like SeriesAnalysis and StatAnalysis are
designed
> > > > > > > to
> > > > > > > process a
> > > > > > > range of times at once, but GridDiag was written to
process
> > > > > > > once
> > > > > > > for
> > > > > each
> > > > > > > time. However, this may not be the intended way to use
the
> > > > > > > tool.
> > > > > > > This
> > > > > is
> > > > > > > how I implemented it based on the example I was
provided.
> > > > > > >
> > > > > > > John HG (copied here) may have some insight on the best
way
> > > > > > > to
> > > > > > > achieve
> > > > > > > what you are trying to do. Perhaps you would want to run
> > > > > > > GridDiag
> > > > > > > for
> > > > > > each
> > > > > > > run time, then run another tool to aggregate the stats
> > > > > > > across
> > > > > > > all
> > > > > > > of
> > > > > the
> > > > > > > output.
> > > > > > >
> > > > > > > If that is not the case, then you should be able to get
the
> > > > > > > wrapper to
> > > > > > run
> > > > > > > the way you are intending by configuring it to process a
> > > > > > > single
> > > > > > > run
> > > > > time
> > > > > > > (INIT_BEG==INIT_END) and expand the list of values for
> > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you need
> > > > > > > using
> > > > > > > the
> > > > > 'shift'
> > > > > > > keyword in the filename templates, i.e.
> > > > > > >
> > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > so
> > > > > > > on.
> > > > > > >
> > > > > > > This approach is not very flexible and would require you
to
> > > > > > > make
> > > > > > > a lot
> > > > > of
> > > > > > > changes to the METplus config file for each range of
times
> > > > > > > you
> > > > > > > desire.
> > > > > If
> > > > > > > this is the only way to achieve what you need, then we
> > > > > > > should
> > > > > > > make
> > > > > > changes
> > > > > > > to the wrapper to allow you to configure this more
easily.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > Hi George,
> > > > > > > > I am running grid-diag using the script found
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > >
> > > > > > > > It all appears to be running to completion but the
> > > > > > > > functionality is
> > > > > > > > not what I was expecting. To test the tool, I first
set
> > > > > > > > the
> > > > > > > > script to
> > > > > > > > run one day (INIT_BEG/END=2020061500) and it created
two
> > > > > > > > file
> > > > > > > > lists
> > > > > > > > (idx0/1) that I would have expected that included that
> > > > > > > > one
> > > > > > > > day.
> > > > > > > > For
> > > > > > > > example:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > I did the same for all my single days. It created the
> > > > > > > > files
> > > > > > > > I
> > > > > > > > would
> > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > 21.nc
> > > > > > > >
> > > > > > > > Then, I wanted to run something similar for all my
> > > > > > > > available
> > > > > > > > data
> > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > INIT_INC=86400)
> > > > > > > > but
> > > > > > > > I
> > > > > would
> > > > > > > > like to have all days aggregated together and one
netcdf
> > > > > > > > file
> > > > > > > > created
> > > > > > > > in the end. It did create one netcdf file at the end,
but
> > > > > > > > it
> > > > > > > > only
> > > > > > > > included the data for the last date (the 15th).
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > 2020061500_f00-21.nc
> > > > > > > >  If you diff the two nc files
> > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > 2020061500_f00-21.nc
> > > > > > > > they are identical.
> > > > > > > >
> > > > > > > > I could be confused on the intent of the tool. Perhaps
> > > > > > > > this
> > > > > > > > is
> > > > > > > > the
> > > > > > > > behavior you want it to have. If so, I still think
there
> > > > > > > > is
> > > > > > > > a
> > > > > > > > bug
> > > > > > > > because it should be outputting a file per day, and it
> > > > > > > > appears
> > > > > > > > to
> > > > > only
> > > > > > > > output a file at the end for the last day.
> > > > > > > >
> > > > > > > > What I thought would happen is I would get one file,
but
> > > > > > > > it
> > > > > > > > would
> > > > > > > > include all days/times requested. So, what I need it
to
> > > > > > > > do
> > > > > > > > is
> > > > > > > > create
> > > > > a
> > > > > > > > list that includes all model (idx0) and obs (idx1)
files
> > > > > > > > that I
> > > > > > > > want
> > > > > > > > to use in one file. I can do this by hand and I am
quite
> > > > > > > > certain it
> > > > > > > > would run and do what I want, but I can't get it to
work
> > > > > > > > in
> > > > > > > > an
> > > > > > > > automated fashion. (Essentially I want it to do
something
> > > > > > > > similar to
> > > > > > > > series/stat-analysis, for example - aggregate over
time)
> > > > > > > >
> > > > > > > > It is very possible I am doing something wrong on my
end.
> > > > > > > > I
> > > > > > > > hope so
> > > > > as
> > > > > > > > that would be an easy fix. If this doesn't make sense
or
> > > > > > > > if
> > > > > > > > you
> > > > > > > > would
> > > > > > > > like to discuss this on the phone at some point, I am
> > > > > > > > happy
> > > > > > > > to
> > > > > > > > do so.
> > > > > > > > Just let me know what you think. Thanks in advance for
> > > > > > > > your
> > > > > > > > help. If
> > > > > > > > my dir structure is confusing or you need me to send
you
> > > > > > > > anything
> > > > > else
> > > > > > > > please don't hesitate to ask.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Jamie
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > >
> > >
> > >
> > >

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Thu Dec 17 13:10:06 2020

Hi George,
I  am sorry, I am swamped this week. I hope to get to this next week
or the
week after.  In the meantime, are there any build instructions you can
share with me to use when  I pull this branch on cheyenne?
I will be in touch as soon as I can.
Thanks,
Jamie

On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Jamie,
>
> Just following up on this in case this email was buried. Let me know
if
> you need any help getting this to work!
>
> - George
>
> On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > Hi Jamie,
> >
> > I implemented the changes to GridDiag that were necessary to run
the
> > tool as you expected. I assigned you to the pull request:
> >
> > https://github.com/dtcenter/METplus/pull/735
> >
> > The pull request has notes regarding which branch to checkout on
> > cheyenne and which setting to add to get it to work. Let me know
if
> > you have any questions!
> >
> > Thanks,
> > George
> >
> > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > Sure. No problem. Thank you!
> > > Jamie
> > >
> > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Jamie,
> > > >
> > > > I created a GitHub issue for this work:
> > > >
> > > > https://github.com/dtcenter/METplus/issues/733
> > > >
> > > > I have implemented functionality to call certain wrappers in
> > > > different
> > > > intervals to process groups of files instead of just once per
file.
> > > > I
> > > > am
> > > > now working on incorporating this functionality into the
GridDiag
> > > > wrapper.
> > > > If you don't mind, I was planning on assigning the pull
request
> > > > review to
> > > > you so you can test these changes on your use case to ensure
it
> > > > runs
> > > > the
> > > > way you expect.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > Hi George and John,
> > > > > Sounds like the modifications you are discussing will
address my
> > > > > need
> > > > > (correct - the list of init times
> > > > > and forecast leads to determine which files to use in a
single
> > > > > analysis).
> > > > > Thanks!
> > > > > Jamie
> > > > >
> > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > OK, it sounds like I misunderstood the intended use for
the
> > > > > > tool
> > > > > > when
> > > > > > I
> > > > > > implemented the wrapper. Thanks for pointing this out,
Jamie!
> > > > > >
> > > > > > John, I think we should have a discussion about how we
would
> > > > > > want
> > > > > > to
> > > > > > have
> > > > > > tools like SeriesAnalysis and GridDiag to be configured to
run
> > > > > > in
> > > > > > these
> > > > > > cases. I'll follow up with you to set something up.
> > > > > > SeriesAnalysis is
> > > > > > currently separated into SeriesByInit and SeriesByLead
which
> > > > > > perform
> > > > > > the
> > > > > > grouping. However, I am currently refactoring these tools
to
> > > > > > hopefully
> > > > > > combine the logic and allow options to specify different
ways
> > > > > > to
> > > > > > filter/group the data. This would be a good time to come
up
> > > > > > with
> > > > > > a
> > > > > > good way
> > > > > > to define how you want to group your data in a general way
that
> > > > > > could
> > > > > > apply
> > > > > > to other similar wrappers, like GridDiag.
> > > > > >
> > > > > > Jamie, to confirm what you are saying, you expected the
list of
> > > > > > init
> > > > > > times
> > > > > > and forecast leads to determine which files to use in a
single
> > > > > > analysis. Is
> > > > > > that correct?
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > >
> > > > > > >
> > > > > > > George,
> > > > > > >
> > > > > > > The logic for running the Grid-Diag is more like Series-
> > > > > > > Analysis
> > > > > > > and
> > > > > > > Stat-Analysis than it is like Grid-Stat. While a user
could
> > > > > > > run
> > > > > > > it
> > > > > > > for a
> > > > > > > single time, it is more likely they'll want to run it
over a
> > > > > > > set of
> > > > > > times.
> > > > > > > It really is pretty analogous to Series-Analysis... you
may
> > > > > > > want to
> > > > > > > hold
> > > > > > > the init time fixed and run over all lead times... or
hold
> > > > > > > the
> > > > > > > lead
> > > > > > > time
> > > > > > > fixed and run over all init times... or just run over
all
> > > > > > > init
> > > > > > > and
> > > > > > > lead
> > > > > > > times in one big batch.
> > > > > > >
> > > > > > > George, is there any way to generalize the support for
how
> > > > > > > times
> > > > > > > are
> > > > > > > processed?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > I moved this email over to MET Help.
> > > > > > > >
> > > > > > > > The configuration you have set up loops over each run
time
> > > > > > > > in
> > > > > > > > the
> > > > > > > > range
> > > > > > > > you specified (2020060800 to 2020061500 with 24 hour
> > > > > > > > increments
> > > > > > > > and
> > > > > > > > processing the list of forecast leads for each time).
> > > > > > > > However,
> > > > > > > > the
> > > > > > output
> > > > > > > > template will be the same for each run:
> > > > > > > >
> > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > >
> > > > > > >
> > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > >
> > > > > > > > This is why you are seeing only the results of the
last run
> > > > > > > > in
> > > > > > > > the
> > > > > > > output.
> > > > > > > > Each run overwrites the data from the previous run.
> > > > > > > >
> > > > > > > > Wrappers like SeriesAnalysis and StatAnalysis are
designed
> > > > > > > > to
> > > > > > > > process a
> > > > > > > > range of times at once, but GridDiag was written to
process
> > > > > > > > once
> > > > > > > > for
> > > > > > each
> > > > > > > > time. However, this may not be the intended way to use
the
> > > > > > > > tool.
> > > > > > > > This
> > > > > > is
> > > > > > > > how I implemented it based on the example I was
provided.
> > > > > > > >
> > > > > > > > John HG (copied here) may have some insight on the
best way
> > > > > > > > to
> > > > > > > > achieve
> > > > > > > > what you are trying to do. Perhaps you would want to
run
> > > > > > > > GridDiag
> > > > > > > > for
> > > > > > > each
> > > > > > > > run time, then run another tool to aggregate the stats
> > > > > > > > across
> > > > > > > > all
> > > > > > > > of
> > > > > > the
> > > > > > > > output.
> > > > > > > >
> > > > > > > > If that is not the case, then you should be able to
get the
> > > > > > > > wrapper to
> > > > > > > run
> > > > > > > > the way you are intending by configuring it to process
a
> > > > > > > > single
> > > > > > > > run
> > > > > > time
> > > > > > > > (INIT_BEG==INIT_END) and expand the list of values for
> > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you need
> > > > > > > > using
> > > > > > > > the
> > > > > > 'shift'
> > > > > > > > keyword in the filename templates, i.e.
> > > > > > > >
> > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > so
> > > > > > > > on.
> > > > > > > >
> > > > > > > > This approach is not very flexible and would require
you to
> > > > > > > > make
> > > > > > > > a lot
> > > > > > of
> > > > > > > > changes to the METplus config file for each range of
times
> > > > > > > > you
> > > > > > > > desire.
> > > > > > If
> > > > > > > > this is the only way to achieve what you need, then we
> > > > > > > > should
> > > > > > > > make
> > > > > > > changes
> > > > > > > > to the wrapper to allow you to configure this more
easily.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > Hi George,
> > > > > > > > > I am running grid-diag using the script found
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > >
> > > > > > > > > It all appears to be running to completion but the
> > > > > > > > > functionality is
> > > > > > > > > not what I was expecting. To test the tool, I first
set
> > > > > > > > > the
> > > > > > > > > script to
> > > > > > > > > run one day (INIT_BEG/END=2020061500) and it created
two
> > > > > > > > > file
> > > > > > > > > lists
> > > > > > > > > (idx0/1) that I would have expected that included
that
> > > > > > > > > one
> > > > > > > > > day.
> > > > > > > > > For
> > > > > > > > > example:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > I did the same for all my single days. It created
the
> > > > > > > > > files
> > > > > > > > > I
> > > > > > > > > would
> > > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > 21.nc
> > > > > > > > >
> > > > > > > > > Then, I wanted to run something similar for all my
> > > > > > > > > available
> > > > > > > > > data
> > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > INIT_INC=86400)
> > > > > > > > > but
> > > > > > > > > I
> > > > > > would
> > > > > > > > > like to have all days aggregated together and one
netcdf
> > > > > > > > > file
> > > > > > > > > created
> > > > > > > > > in the end. It did create one netcdf file at the
end, but
> > > > > > > > > it
> > > > > > > > > only
> > > > > > > > > included the data for the last date (the 15th).
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > >  If you diff the two nc files
> > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > they are identical.
> > > > > > > > >
> > > > > > > > > I could be confused on the intent of the tool.
Perhaps
> > > > > > > > > this
> > > > > > > > > is
> > > > > > > > > the
> > > > > > > > > behavior you want it to have. If so, I still think
there
> > > > > > > > > is
> > > > > > > > > a
> > > > > > > > > bug
> > > > > > > > > because it should be outputting a file per day, and
it
> > > > > > > > > appears
> > > > > > > > > to
> > > > > > only
> > > > > > > > > output a file at the end for the last day.
> > > > > > > > >
> > > > > > > > > What I thought would happen is I would get one file,
but
> > > > > > > > > it
> > > > > > > > > would
> > > > > > > > > include all days/times requested. So, what I need it
to
> > > > > > > > > do
> > > > > > > > > is
> > > > > > > > > create
> > > > > > a
> > > > > > > > > list that includes all model (idx0) and obs (idx1)
files
> > > > > > > > > that I
> > > > > > > > > want
> > > > > > > > > to use in one file. I can do this by hand and I am
quite
> > > > > > > > > certain it
> > > > > > > > > would run and do what I want, but I can't get it to
work
> > > > > > > > > in
> > > > > > > > > an
> > > > > > > > > automated fashion. (Essentially I want it to do
something
> > > > > > > > > similar to
> > > > > > > > > series/stat-analysis, for example - aggregate over
time)
> > > > > > > > >
> > > > > > > > > It is very possible I am doing something wrong on my
end.
> > > > > > > > > I
> > > > > > > > > hope so
> > > > > > as
> > > > > > > > > that would be an easy fix. If this doesn't make
sense or
> > > > > > > > > if
> > > > > > > > > you
> > > > > > > > > would
> > > > > > > > > like to discuss this on the phone at some point, I
am
> > > > > > > > > happy
> > > > > > > > > to
> > > > > > > > > do so.
> > > > > > > > > Just let me know what you think. Thanks in advance
for
> > > > > > > > > your
> > > > > > > > > help. If
> > > > > > > > > my dir structure is confusing or you need me to send
you
> > > > > > > > > anything
> > > > > > else
> > > > > > > > > please don't hesitate to ask.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Jamie
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > > > > obliged to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
>
>
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Thu Dec 17 13:30:48 2020

Hi Jamie,

Not a problem at all. You should just be able to obtain the branch and
call
that version of master_metplus.py instead of the one you are currently
using. Let me know if you run into any issues and I can help.

Thanks,
George

On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> Hi George,
> I  am sorry, I am swamped this week. I hope to get to this next week
or the
> week after.  In the meantime, are there any build instructions you
can
> share with me to use when  I pull this branch on cheyenne?
> I will be in touch as soon as I can.
> Thanks,
> Jamie
>
> On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Jamie,
> >
> > Just following up on this in case this email was buried. Let me
know if
> > you need any help getting this to work!
> >
> > - George
> >
> > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > Hi Jamie,
> > >
> > > I implemented the changes to GridDiag that were necessary to run
the
> > > tool as you expected. I assigned you to the pull request:
> > >
> > > https://github.com/dtcenter/METplus/pull/735
> > >
> > > The pull request has notes regarding which branch to checkout on
> > > cheyenne and which setting to add to get it to work. Let me know
if
> > > you have any questions!
> > >
> > > Thanks,
> > > George
> > >
> > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > Sure. No problem. Thank you!
> > > > Jamie
> > > >
> > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Jamie,
> > > > >
> > > > > I created a GitHub issue for this work:
> > > > >
> > > > > https://github.com/dtcenter/METplus/issues/733
> > > > >
> > > > > I have implemented functionality to call certain wrappers in
> > > > > different
> > > > > intervals to process groups of files instead of just once
per file.
> > > > > I
> > > > > am
> > > > > now working on incorporating this functionality into the
GridDiag
> > > > > wrapper.
> > > > > If you don't mind, I was planning on assigning the pull
request
> > > > > review to
> > > > > you so you can test these changes on your use case to ensure
it
> > > > > runs
> > > > > the
> > > > > way you expect.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > Hi George and John,
> > > > > > Sounds like the modifications you are discussing will
address my
> > > > > > need
> > > > > > (correct - the list of init times
> > > > > > and forecast leads to determine which files to use in a
single
> > > > > > analysis).
> > > > > > Thanks!
> > > > > > Jamie
> > > > > >
> > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > OK, it sounds like I misunderstood the intended use for
the
> > > > > > > tool
> > > > > > > when
> > > > > > > I
> > > > > > > implemented the wrapper. Thanks for pointing this out,
Jamie!
> > > > > > >
> > > > > > > John, I think we should have a discussion about how we
would
> > > > > > > want
> > > > > > > to
> > > > > > > have
> > > > > > > tools like SeriesAnalysis and GridDiag to be configured
to run
> > > > > > > in
> > > > > > > these
> > > > > > > cases. I'll follow up with you to set something up.
> > > > > > > SeriesAnalysis is
> > > > > > > currently separated into SeriesByInit and SeriesByLead
which
> > > > > > > perform
> > > > > > > the
> > > > > > > grouping. However, I am currently refactoring these
tools to
> > > > > > > hopefully
> > > > > > > combine the logic and allow options to specify different
ways
> > > > > > > to
> > > > > > > filter/group the data. This would be a good time to come
up
> > > > > > > with
> > > > > > > a
> > > > > > > good way
> > > > > > > to define how you want to group your data in a general
way that
> > > > > > > could
> > > > > > > apply
> > > > > > > to other similar wrappers, like GridDiag.
> > > > > > >
> > > > > > > Jamie, to confirm what you are saying, you expected the
list of
> > > > > > > init
> > > > > > > times
> > > > > > > and forecast leads to determine which files to use in a
single
> > > > > > > analysis. Is
> > > > > > > that correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > >
> > > > > > > >
> > > > > > > > George,
> > > > > > > >
> > > > > > > > The logic for running the Grid-Diag is more like
Series-
> > > > > > > > Analysis
> > > > > > > > and
> > > > > > > > Stat-Analysis than it is like Grid-Stat. While a user
could
> > > > > > > > run
> > > > > > > > it
> > > > > > > > for a
> > > > > > > > single time, it is more likely they'll want to run it
over a
> > > > > > > > set of
> > > > > > > times.
> > > > > > > > It really is pretty analogous to Series-Analysis...
you may
> > > > > > > > want to
> > > > > > > > hold
> > > > > > > > the init time fixed and run over all lead times... or
hold
> > > > > > > > the
> > > > > > > > lead
> > > > > > > > time
> > > > > > > > fixed and run over all init times... or just run over
all
> > > > > > > > init
> > > > > > > > and
> > > > > > > > lead
> > > > > > > > times in one big batch.
> > > > > > > >
> > > > > > > > George, is there any way to generalize the support for
how
> > > > > > > > times
> > > > > > > > are
> > > > > > > > processed?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > > > > > <met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jamie,
> > > > > > > > >
> > > > > > > > > I moved this email over to MET Help.
> > > > > > > > >
> > > > > > > > > The configuration you have set up loops over each
run time
> > > > > > > > > in
> > > > > > > > > the
> > > > > > > > > range
> > > > > > > > > you specified (2020060800 to 2020061500 with 24 hour
> > > > > > > > > increments
> > > > > > > > > and
> > > > > > > > > processing the list of forecast leads for each
time).
> > > > > > > > > However,
> > > > > > > > > the
> > > > > > > output
> > > > > > > > > template will be the same for each run:
> > > > > > > > >
> > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > >
> > > > > > > >
> > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > >
> > > > > > > > > This is why you are seeing only the results of the
last run
> > > > > > > > > in
> > > > > > > > > the
> > > > > > > > output.
> > > > > > > > > Each run overwrites the data from the previous run.
> > > > > > > > >
> > > > > > > > > Wrappers like SeriesAnalysis and StatAnalysis are
designed
> > > > > > > > > to
> > > > > > > > > process a
> > > > > > > > > range of times at once, but GridDiag was written to
process
> > > > > > > > > once
> > > > > > > > > for
> > > > > > > each
> > > > > > > > > time. However, this may not be the intended way to
use the
> > > > > > > > > tool.
> > > > > > > > > This
> > > > > > > is
> > > > > > > > > how I implemented it based on the example I was
provided.
> > > > > > > > >
> > > > > > > > > John HG (copied here) may have some insight on the
best way
> > > > > > > > > to
> > > > > > > > > achieve
> > > > > > > > > what you are trying to do. Perhaps you would want to
run
> > > > > > > > > GridDiag
> > > > > > > > > for
> > > > > > > > each
> > > > > > > > > run time, then run another tool to aggregate the
stats
> > > > > > > > > across
> > > > > > > > > all
> > > > > > > > > of
> > > > > > > the
> > > > > > > > > output.
> > > > > > > > >
> > > > > > > > > If that is not the case, then you should be able to
get the
> > > > > > > > > wrapper to
> > > > > > > > run
> > > > > > > > > the way you are intending by configuring it to
process a
> > > > > > > > > single
> > > > > > > > > run
> > > > > > > time
> > > > > > > > > (INIT_BEG==INIT_END) and expand the list of values
for
> > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you
need
> > > > > > > > > using
> > > > > > > > > the
> > > > > > > 'shift'
> > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > >
> > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > so
> > > > > > > > > on.
> > > > > > > > >
> > > > > > > > > This approach is not very flexible and would require
you to
> > > > > > > > > make
> > > > > > > > > a lot
> > > > > > > of
> > > > > > > > > changes to the METplus config file for each range of
times
> > > > > > > > > you
> > > > > > > > > desire.
> > > > > > > If
> > > > > > > > > this is the only way to achieve what you need, then
we
> > > > > > > > > should
> > > > > > > > > make
> > > > > > > > changes
> > > > > > > > > to the wrapper to allow you to configure this more
easily.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > Hi George,
> > > > > > > > > > I am running grid-diag using the script found
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > >
> > > > > > > > > > It all appears to be running to completion but the
> > > > > > > > > > functionality is
> > > > > > > > > > not what I was expecting. To test the tool, I
first set
> > > > > > > > > > the
> > > > > > > > > > script to
> > > > > > > > > > run one day (INIT_BEG/END=2020061500) and it
created two
> > > > > > > > > > file
> > > > > > > > > > lists
> > > > > > > > > > (idx0/1) that I would have expected that included
that
> > > > > > > > > > one
> > > > > > > > > > day.
> > > > > > > > > > For
> > > > > > > > > > example:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > I did the same for all my single days. It created
the
> > > > > > > > > > files
> > > > > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > 21.nc
> > > > > > > > > >
> > > > > > > > > > Then, I wanted to run something similar for all my
> > > > > > > > > > available
> > > > > > > > > > data
> > > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > but
> > > > > > > > > > I
> > > > > > > would
> > > > > > > > > > like to have all days aggregated together and one
netcdf
> > > > > > > > > > file
> > > > > > > > > > created
> > > > > > > > > > in the end. It did create one netcdf file at the
end, but
> > > > > > > > > > it
> > > > > > > > > > only
> > > > > > > > > > included the data for the last date (the 15th).
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > they are identical.
> > > > > > > > > >
> > > > > > > > > > I could be confused on the intent of the tool.
Perhaps
> > > > > > > > > > this
> > > > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > behavior you want it to have. If so, I still think
there
> > > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > bug
> > > > > > > > > > because it should be outputting a file per day,
and it
> > > > > > > > > > appears
> > > > > > > > > > to
> > > > > > > only
> > > > > > > > > > output a file at the end for the last day.
> > > > > > > > > >
> > > > > > > > > > What I thought would happen is I would get one
file, but
> > > > > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > include all days/times requested. So, what I need
it to
> > > > > > > > > > do
> > > > > > > > > > is
> > > > > > > > > > create
> > > > > > > a
> > > > > > > > > > list that includes all model (idx0) and obs (idx1)
files
> > > > > > > > > > that I
> > > > > > > > > > want
> > > > > > > > > > to use in one file. I can do this by hand and I am
quite
> > > > > > > > > > certain it
> > > > > > > > > > would run and do what I want, but I can't get it
to work
> > > > > > > > > > in
> > > > > > > > > > an
> > > > > > > > > > automated fashion. (Essentially I want it to do
something
> > > > > > > > > > similar to
> > > > > > > > > > series/stat-analysis, for example - aggregate over
time)
> > > > > > > > > >
> > > > > > > > > > It is very possible I am doing something wrong on
my end.
> > > > > > > > > > I
> > > > > > > > > > hope so
> > > > > > > as
> > > > > > > > > > that would be an easy fix. If this doesn't make
sense or
> > > > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > would
> > > > > > > > > > like to discuss this on the phone at some point, I
am
> > > > > > > > > > happy
> > > > > > > > > > to
> > > > > > > > > > do so.
> > > > > > > > > > Just let me know what you think. Thanks in advance
for
> > > > > > > > > > your
> > > > > > > > > > help. If
> > > > > > > > > > my dir structure is confusing or you need me to
send you
> > > > > > > > > > anything
> > > > > > > else
> > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Jamie
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > > > > obliged to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> >
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 21 13:42:50 2020

Hi George,
I am trying to test your new mods in my processing.
I have a script to run grid_diag here:
/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh

I made the change you mentioned needed to be added
(GRID_DIAG_RUNTIME_FREQ
= RUN_ONCE) to my METplus parm and you can see the parm file
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf

However, when I run it I get the following error:
12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
(runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode unles
s LOOP_BY = processes

But that is confusing because I do not think LOOP_BY has an option of
processes. Should that be LOOP_ORDER? That is what I tried and that
seemed
to get me further.

Before, there would be two file_lists created one for the model and
one for
the obs
(For example:
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
But now it looks like they are both in one file:
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt

I think this is causing a problem because I am getting an error:
DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
/glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
DEBUG 1: Default Config File:
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File:
/glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
ERROR  :
ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
abbreviation
'tb_g16'
ERROR  :

I am guessing this is a problem with my config file, not necessarily
your
logic. Could you take a look at my config file to see if I have
additional
logic that needs to change for this to work? Let me know if you need
more
information from me.
Thanks!
Jamie

On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Jamie,
>
> Not a problem at all. You should just be able to obtain the branch
and call
> that version of master_metplus.py instead of the one you are
currently
> using. Let me know if you run into any issues and I can help.
>
> Thanks,
> George
>
> On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > Hi George,
> > I  am sorry, I am swamped this week. I hope to get to this next
week or
> the
> > week after.  In the meantime, are there any build instructions you
can
> > share with me to use when  I pull this branch on cheyenne?
> > I will be in touch as soon as I can.
> > Thanks,
> > Jamie
> >
> > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Jamie,
> > >
> > > Just following up on this in case this email was buried. Let me
know if
> > > you need any help getting this to work!
> > >
> > > - George
> > >
> > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > Hi Jamie,
> > > >
> > > > I implemented the changes to GridDiag that were necessary to
run the
> > > > tool as you expected. I assigned you to the pull request:
> > > >
> > > > https://github.com/dtcenter/METplus/pull/735
> > > >
> > > > The pull request has notes regarding which branch to checkout
on
> > > > cheyenne and which setting to add to get it to work. Let me
know if
> > > > you have any questions!
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > Sure. No problem. Thank you!
> > > > > Jamie
> > > > >
> > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > I created a GitHub issue for this work:
> > > > > >
> > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > >
> > > > > > I have implemented functionality to call certain wrappers
in
> > > > > > different
> > > > > > intervals to process groups of files instead of just once
per
> file.
> > > > > > I
> > > > > > am
> > > > > > now working on incorporating this functionality into the
GridDiag
> > > > > > wrapper.
> > > > > > If you don't mind, I was planning on assigning the pull
request
> > > > > > review to
> > > > > > you so you can test these changes on your use case to
ensure it
> > > > > > runs
> > > > > > the
> > > > > > way you expect.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > Hi George and John,
> > > > > > > Sounds like the modifications you are discussing will
address
> my
> > > > > > > need
> > > > > > > (correct - the list of init times
> > > > > > > and forecast leads to determine which files to use in a
single
> > > > > > > analysis).
> > > > > > > Thanks!
> > > > > > > Jamie
> > > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > OK, it sounds like I misunderstood the intended use
for the
> > > > > > > > tool
> > > > > > > > when
> > > > > > > > I
> > > > > > > > implemented the wrapper. Thanks for pointing this out,
Jamie!
> > > > > > > >
> > > > > > > > John, I think we should have a discussion about how we
would
> > > > > > > > want
> > > > > > > > to
> > > > > > > > have
> > > > > > > > tools like SeriesAnalysis and GridDiag to be
configured to
> run
> > > > > > > > in
> > > > > > > > these
> > > > > > > > cases. I'll follow up with you to set something up.
> > > > > > > > SeriesAnalysis is
> > > > > > > > currently separated into SeriesByInit and SeriesByLead
which
> > > > > > > > perform
> > > > > > > > the
> > > > > > > > grouping. However, I am currently refactoring these
tools to
> > > > > > > > hopefully
> > > > > > > > combine the logic and allow options to specify
different ways
> > > > > > > > to
> > > > > > > > filter/group the data. This would be a good time to
come up
> > > > > > > > with
> > > > > > > > a
> > > > > > > > good way
> > > > > > > > to define how you want to group your data in a general
way
> that
> > > > > > > > could
> > > > > > > > apply
> > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > >
> > > > > > > > Jamie, to confirm what you are saying, you expected
the list
> of
> > > > > > > > init
> > > > > > > > times
> > > > > > > > and forecast leads to determine which files to use in
a
> single
> > > > > > > > analysis. Is
> > > > > > > > that correct?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > George,
> > > > > > > > >
> > > > > > > > > The logic for running the Grid-Diag is more like
Series-
> > > > > > > > > Analysis
> > > > > > > > > and
> > > > > > > > > Stat-Analysis than it is like Grid-Stat. While a
user could
> > > > > > > > > run
> > > > > > > > > it
> > > > > > > > > for a
> > > > > > > > > single time, it is more likely they'll want to run
it over
> a
> > > > > > > > > set of
> > > > > > > > times.
> > > > > > > > > It really is pretty analogous to Series-Analysis...
you may
> > > > > > > > > want to
> > > > > > > > > hold
> > > > > > > > > the init time fixed and run over all lead times...
or hold
> > > > > > > > > the
> > > > > > > > > lead
> > > > > > > > > time
> > > > > > > > > fixed and run over all init times... or just run
over all
> > > > > > > > > init
> > > > > > > > > and
> > > > > > > > > lead
> > > > > > > > > times in one big batch.
> > > > > > > > >
> > > > > > > > > George, is there any way to generalize the support
for how
> > > > > > > > > times
> > > > > > > > > are
> > > > > > > > > processed?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via RT
> > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jamie,
> > > > > > > > > >
> > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > >
> > > > > > > > > > The configuration you have set up loops over each
run
> time
> > > > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > range
> > > > > > > > > > you specified (2020060800 to 2020061500 with 24
hour
> > > > > > > > > > increments
> > > > > > > > > > and
> > > > > > > > > > processing the list of forecast leads for each
time).
> > > > > > > > > > However,
> > > > > > > > > > the
> > > > > > > > output
> > > > > > > > > > template will be the same for each run:
> > > > > > > > > >
> > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > >
> > > > > > > > > > This is why you are seeing only the results of the
last
> run
> > > > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > output.
> > > > > > > > > > Each run overwrites the data from the previous
run.
> > > > > > > > > >
> > > > > > > > > > Wrappers like SeriesAnalysis and StatAnalysis are
> designed
> > > > > > > > > > to
> > > > > > > > > > process a
> > > > > > > > > > range of times at once, but GridDiag was written
to
> process
> > > > > > > > > > once
> > > > > > > > > > for
> > > > > > > > each
> > > > > > > > > > time. However, this may not be the intended way to
use
> the
> > > > > > > > > > tool.
> > > > > > > > > > This
> > > > > > > > is
> > > > > > > > > > how I implemented it based on the example I was
provided.
> > > > > > > > > >
> > > > > > > > > > John HG (copied here) may have some insight on the
best
> way
> > > > > > > > > > to
> > > > > > > > > > achieve
> > > > > > > > > > what you are trying to do. Perhaps you would want
to run
> > > > > > > > > > GridDiag
> > > > > > > > > > for
> > > > > > > > > each
> > > > > > > > > > run time, then run another tool to aggregate the
stats
> > > > > > > > > > across
> > > > > > > > > > all
> > > > > > > > > > of
> > > > > > > > the
> > > > > > > > > > output.
> > > > > > > > > >
> > > > > > > > > > If that is not the case, then you should be able
to get
> the
> > > > > > > > > > wrapper to
> > > > > > > > > run
> > > > > > > > > > the way you are intending by configuring it to
process a
> > > > > > > > > > single
> > > > > > > > > > run
> > > > > > > > time
> > > > > > > > > > (INIT_BEG==INIT_END) and expand the list of values
for
> > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times you
need
> > > > > > > > > > using
> > > > > > > > > > the
> > > > > > > > 'shift'
> > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > >
> > > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > so
> > > > > > > > > > on.
> > > > > > > > > >
> > > > > > > > > > This approach is not very flexible and would
require you
> to
> > > > > > > > > > make
> > > > > > > > > > a lot
> > > > > > > > of
> > > > > > > > > > changes to the METplus config file for each range
of
> times
> > > > > > > > > > you
> > > > > > > > > > desire.
> > > > > > > > If
> > > > > > > > > > this is the only way to achieve what you need,
then we
> > > > > > > > > > should
> > > > > > > > > > make
> > > > > > > > > changes
> > > > > > > > > > to the wrapper to allow you to configure this more
> easily.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > > Hi George,
> > > > > > > > > > > I am running grid-diag using the script found
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > >
> > > > > > > > > > > It all appears to be running to completion but
the
> > > > > > > > > > > functionality is
> > > > > > > > > > > not what I was expecting. To test the tool, I
first set
> > > > > > > > > > > the
> > > > > > > > > > > script to
> > > > > > > > > > > run one day (INIT_BEG/END=2020061500) and it
created
> two
> > > > > > > > > > > file
> > > > > > > > > > > lists
> > > > > > > > > > > (idx0/1) that I would have expected that
included that
> > > > > > > > > > > one
> > > > > > > > > > > day.
> > > > > > > > > > > For
> > > > > > > > > > > example:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > I did the same for all my single days. It
created the
> > > > > > > > > > > files
> > > > > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > 21.nc
> > > > > > > > > > >
> > > > > > > > > > > Then, I wanted to run something similar for all
my
> > > > > > > > > > > available
> > > > > > > > > > > data
> > > > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > but
> > > > > > > > > > > I
> > > > > > > > would
> > > > > > > > > > > like to have all days aggregated together and
one
> netcdf
> > > > > > > > > > > file
> > > > > > > > > > > created
> > > > > > > > > > > in the end. It did create one netcdf file at the
end,
> but
> > > > > > > > > > > it
> > > > > > > > > > > only
> > > > > > > > > > > included the data for the last date (the 15th).
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > they are identical.
> > > > > > > > > > >
> > > > > > > > > > > I could be confused on the intent of the tool.
Perhaps
> > > > > > > > > > > this
> > > > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > behavior you want it to have. If so, I still
think
> there
> > > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > bug
> > > > > > > > > > > because it should be outputting a file per day,
and it
> > > > > > > > > > > appears
> > > > > > > > > > > to
> > > > > > > > only
> > > > > > > > > > > output a file at the end for the last day.
> > > > > > > > > > >
> > > > > > > > > > > What I thought would happen is I would get one
file,
> but
> > > > > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > include all days/times requested. So, what I
need it to
> > > > > > > > > > > do
> > > > > > > > > > > is
> > > > > > > > > > > create
> > > > > > > > a
> > > > > > > > > > > list that includes all model (idx0) and obs
(idx1)
> files
> > > > > > > > > > > that I
> > > > > > > > > > > want
> > > > > > > > > > > to use in one file. I can do this by hand and I
am
> quite
> > > > > > > > > > > certain it
> > > > > > > > > > > would run and do what I want, but I can't get it
to
> work
> > > > > > > > > > > in
> > > > > > > > > > > an
> > > > > > > > > > > automated fashion. (Essentially I want it to do
> something
> > > > > > > > > > > similar to
> > > > > > > > > > > series/stat-analysis, for example - aggregate
over
> time)
> > > > > > > > > > >
> > > > > > > > > > > It is very possible I am doing something wrong
on my
> end.
> > > > > > > > > > > I
> > > > > > > > > > > hope so
> > > > > > > > as
> > > > > > > > > > > that would be an easy fix. If this doesn't make
sense
> or
> > > > > > > > > > > if
> > > > > > > > > > > you
> > > > > > > > > > > would
> > > > > > > > > > > like to discuss this on the phone at some point,
I am
> > > > > > > > > > > happy
> > > > > > > > > > > to
> > > > > > > > > > > do so.
> > > > > > > > > > > Just let me know what you think. Thanks in
advance for
> > > > > > > > > > > your
> > > > > > > > > > > help. If
> > > > > > > > > > > my dir structure is confusing or you need me to
send
> you
> > > > > > > > > > > anything
> > > > > > > > else
> > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Jamie
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > > > > obliged to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > >
> > >
> > >
> >
> > --
> >
> > Jamie K. Wolff
> > National Center for Atmospheric Research (NCAR)
> > Research Applications Laboratory (RAL)
> > Phone: 303.497.2812  Email: jwolff at ucar.edu
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 21 14:10:06 2020

Hi Jamie,

Yes, that error message should say LOOP_ORDER. Thanks for catching
that!

Are you saying the error is occurring because all of the files are
lumped
together instead of specified in separate -data arguments? If so, I
can
modify the code to separate out each item in the template list.

On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> Hi George,
> I am trying to test your new mods in my processing.
> I have a script to run grid_diag here:
> /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
>
> I made the change you mentioned needed to be added
(GRID_DIAG_RUNTIME_FREQ
> = RUN_ONCE) to my METplus parm and you can see the parm file
> here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
>
> However, when I run it I get the following error:
> 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode unles
> s LOOP_BY = processes
>
> But that is confusing because I do not think LOOP_BY has an option
of
> processes. Should that be LOOP_ORDER? That is what I tried and that
seemed
> to get me further.
>
> Before, there would be two file_lists created one for the model and
one for
> the obs
> (For example:
>
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
>
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> But now it looks like they are both in one file:
>
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
>
> I think this is causing a problem because I am getting an error:
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> DEBUG 1: Default Config File:
>
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> DEBUG 1: User Config File:
> /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> ERROR  :
> ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
abbreviation
> 'tb_g16'
> ERROR  :
>
> I am guessing this is a problem with my config file, not necessarily
your
> logic. Could you take a look at my config file to see if I have
additional
> logic that needs to change for this to work? Let me know if you need
more
> information from me.
> Thanks!
> Jamie
>
> On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Jamie,
> >
> > Not a problem at all. You should just be able to obtain the branch
and
> call
> > that version of master_metplus.py instead of the one you are
currently
> > using. Let me know if you run into any issues and I can help.
> >
> > Thanks,
> > George
> >
> > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > Hi George,
> > > I  am sorry, I am swamped this week. I hope to get to this next
week or
> > the
> > > week after.  In the meantime, are there any build instructions
you can
> > > share with me to use when  I pull this branch on cheyenne?
> > > I will be in touch as soon as I can.
> > > Thanks,
> > > Jamie
> > >
> > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Jamie,
> > > >
> > > > Just following up on this in case this email was buried. Let
me know
> if
> > > > you need any help getting this to work!
> > > >
> > > > - George
> > > >
> > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > Hi Jamie,
> > > > >
> > > > > I implemented the changes to GridDiag that were necessary to
run
> the
> > > > > tool as you expected. I assigned you to the pull request:
> > > > >
> > > > > https://github.com/dtcenter/METplus/pull/735
> > > > >
> > > > > The pull request has notes regarding which branch to
checkout on
> > > > > cheyenne and which setting to add to get it to work. Let me
know if
> > > > > you have any questions!
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > Sure. No problem. Thank you!
> > > > > > Jamie
> > > > > >
> > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > I created a GitHub issue for this work:
> > > > > > >
> > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > >
> > > > > > > I have implemented functionality to call certain
wrappers in
> > > > > > > different
> > > > > > > intervals to process groups of files instead of just
once per
> > file.
> > > > > > > I
> > > > > > > am
> > > > > > > now working on incorporating this functionality into the
> GridDiag
> > > > > > > wrapper.
> > > > > > > If you don't mind, I was planning on assigning the pull
request
> > > > > > > review to
> > > > > > > you so you can test these changes on your use case to
ensure it
> > > > > > > runs
> > > > > > > the
> > > > > > > way you expect.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > Hi George and John,
> > > > > > > > Sounds like the modifications you are discussing will
address
> > my
> > > > > > > > need
> > > > > > > > (correct - the list of init times
> > > > > > > > and forecast leads to determine which files to use in
a
> single
> > > > > > > > analysis).
> > > > > > > > Thanks!
> > > > > > > > Jamie
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > > > > > <met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > OK, it sounds like I misunderstood the intended use
for the
> > > > > > > > > tool
> > > > > > > > > when
> > > > > > > > > I
> > > > > > > > > implemented the wrapper. Thanks for pointing this
out,
> Jamie!
> > > > > > > > >
> > > > > > > > > John, I think we should have a discussion about how
we
> would
> > > > > > > > > want
> > > > > > > > > to
> > > > > > > > > have
> > > > > > > > > tools like SeriesAnalysis and GridDiag to be
configured to
> > run
> > > > > > > > > in
> > > > > > > > > these
> > > > > > > > > cases. I'll follow up with you to set something up.
> > > > > > > > > SeriesAnalysis is
> > > > > > > > > currently separated into SeriesByInit and
SeriesByLead
> which
> > > > > > > > > perform
> > > > > > > > > the
> > > > > > > > > grouping. However, I am currently refactoring these
tools
> to
> > > > > > > > > hopefully
> > > > > > > > > combine the logic and allow options to specify
different
> ways
> > > > > > > > > to
> > > > > > > > > filter/group the data. This would be a good time to
come up
> > > > > > > > > with
> > > > > > > > > a
> > > > > > > > > good way
> > > > > > > > > to define how you want to group your data in a
general way
> > that
> > > > > > > > > could
> > > > > > > > > apply
> > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > >
> > > > > > > > > Jamie, to confirm what you are saying, you expected
the
> list
> > of
> > > > > > > > > init
> > > > > > > > > times
> > > > > > > > > and forecast leads to determine which files to use
in a
> > single
> > > > > > > > > analysis. Is
> > > > > > > > > that correct?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > George,
> > > > > > > > > >
> > > > > > > > > > The logic for running the Grid-Diag is more like
Series-
> > > > > > > > > > Analysis
> > > > > > > > > > and
> > > > > > > > > > Stat-Analysis than it is like Grid-Stat. While a
user
> could
> > > > > > > > > > run
> > > > > > > > > > it
> > > > > > > > > > for a
> > > > > > > > > > single time, it is more likely they'll want to run
it
> over
> > a
> > > > > > > > > > set of
> > > > > > > > > times.
> > > > > > > > > > It really is pretty analogous to Series-
Analysis... you
> may
> > > > > > > > > > want to
> > > > > > > > > > hold
> > > > > > > > > > the init time fixed and run over all lead times...
or
> hold
> > > > > > > > > > the
> > > > > > > > > > lead
> > > > > > > > > > time
> > > > > > > > > > fixed and run over all init times... or just run
over all
> > > > > > > > > > init
> > > > > > > > > > and
> > > > > > > > > > lead
> > > > > > > > > > times in one big batch.
> > > > > > > > > >
> > > > > > > > > > George, is there any way to generalize the support
for
> how
> > > > > > > > > > times
> > > > > > > > > > are
> > > > > > > > > > processed?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe via
RT
> > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jamie,
> > > > > > > > > > >
> > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > >
> > > > > > > > > > > The configuration you have set up loops over
each run
> > time
> > > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > range
> > > > > > > > > > > you specified (2020060800 to 2020061500 with 24
hour
> > > > > > > > > > > increments
> > > > > > > > > > > and
> > > > > > > > > > > processing the list of forecast leads for each
time).
> > > > > > > > > > > However,
> > > > > > > > > > > the
> > > > > > > > > output
> > > > > > > > > > > template will be the same for each run:
> > > > > > > > > > >
> > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > >
> > > > > > > > > > > This is why you are seeing only the results of
the last
> > run
> > > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > output.
> > > > > > > > > > > Each run overwrites the data from the previous
run.
> > > > > > > > > > >
> > > > > > > > > > > Wrappers like SeriesAnalysis and StatAnalysis
are
> > designed
> > > > > > > > > > > to
> > > > > > > > > > > process a
> > > > > > > > > > > range of times at once, but GridDiag was written
to
> > process
> > > > > > > > > > > once
> > > > > > > > > > > for
> > > > > > > > > each
> > > > > > > > > > > time. However, this may not be the intended way
to use
> > the
> > > > > > > > > > > tool.
> > > > > > > > > > > This
> > > > > > > > > is
> > > > > > > > > > > how I implemented it based on the example I was
> provided.
> > > > > > > > > > >
> > > > > > > > > > > John HG (copied here) may have some insight on
the best
> > way
> > > > > > > > > > > to
> > > > > > > > > > > achieve
> > > > > > > > > > > what you are trying to do. Perhaps you would
want to
> run
> > > > > > > > > > > GridDiag
> > > > > > > > > > > for
> > > > > > > > > > each
> > > > > > > > > > > run time, then run another tool to aggregate the
stats
> > > > > > > > > > > across
> > > > > > > > > > > all
> > > > > > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > output.
> > > > > > > > > > >
> > > > > > > > > > > If that is not the case, then you should be able
to get
> > the
> > > > > > > > > > > wrapper to
> > > > > > > > > > run
> > > > > > > > > > > the way you are intending by configuring it to
process
> a
> > > > > > > > > > > single
> > > > > > > > > > > run
> > > > > > > > > time
> > > > > > > > > > > (INIT_BEG==INIT_END) and expand the list of
values for
> > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times
you need
> > > > > > > > > > > using
> > > > > > > > > > > the
> > > > > > > > > 'shift'
> > > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > > >
> > > > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > so
> > > > > > > > > > > on.
> > > > > > > > > > >
> > > > > > > > > > > This approach is not very flexible and would
require
> you
> > to
> > > > > > > > > > > make
> > > > > > > > > > > a lot
> > > > > > > > > of
> > > > > > > > > > > changes to the METplus config file for each
range of
> > times
> > > > > > > > > > > you
> > > > > > > > > > > desire.
> > > > > > > > > If
> > > > > > > > > > > this is the only way to achieve what you need,
then we
> > > > > > > > > > > should
> > > > > > > > > > > make
> > > > > > > > > > changes
> > > > > > > > > > > to the wrapper to allow you to configure this
more
> > easily.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > > > Hi George,
> > > > > > > > > > > > I am running grid-diag using the script found
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > >
> > > > > > > > > > > > It all appears to be running to completion but
the
> > > > > > > > > > > > functionality is
> > > > > > > > > > > > not what I was expecting. To test the tool, I
first
> set
> > > > > > > > > > > > the
> > > > > > > > > > > > script to
> > > > > > > > > > > > run one day (INIT_BEG/END=2020061500) and it
created
> > two
> > > > > > > > > > > > file
> > > > > > > > > > > > lists
> > > > > > > > > > > > (idx0/1) that I would have expected that
included
> that
> > > > > > > > > > > > one
> > > > > > > > > > > > day.
> > > > > > > > > > > > For
> > > > > > > > > > > > example:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > I did the same for all my single days. It
created the
> > > > > > > > > > > > files
> > > > > > > > > > > > I
> > > > > > > > > > > > would
> > > > > > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > 21.nc
> > > > > > > > > > > >
> > > > > > > > > > > > Then, I wanted to run something similar for
all my
> > > > > > > > > > > > available
> > > > > > > > > > > > data
> > > > > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > but
> > > > > > > > > > > > I
> > > > > > > > > would
> > > > > > > > > > > > like to have all days aggregated together and
one
> > netcdf
> > > > > > > > > > > > file
> > > > > > > > > > > > created
> > > > > > > > > > > > in the end. It did create one netcdf file at
the end,
> > but
> > > > > > > > > > > > it
> > > > > > > > > > > > only
> > > > > > > > > > > > included the data for the last date (the
15th).
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > they are identical.
> > > > > > > > > > > >
> > > > > > > > > > > > I could be confused on the intent of the tool.
> Perhaps
> > > > > > > > > > > > this
> > > > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > behavior you want it to have. If so, I still
think
> > there
> > > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > bug
> > > > > > > > > > > > because it should be outputting a file per
day, and
> it
> > > > > > > > > > > > appears
> > > > > > > > > > > > to
> > > > > > > > > only
> > > > > > > > > > > > output a file at the end for the last day.
> > > > > > > > > > > >
> > > > > > > > > > > > What I thought would happen is I would get one
file,
> > but
> > > > > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > include all days/times requested. So, what I
need it
> to
> > > > > > > > > > > > do
> > > > > > > > > > > > is
> > > > > > > > > > > > create
> > > > > > > > > a
> > > > > > > > > > > > list that includes all model (idx0) and obs
(idx1)
> > files
> > > > > > > > > > > > that I
> > > > > > > > > > > > want
> > > > > > > > > > > > to use in one file. I can do this by hand and
I am
> > quite
> > > > > > > > > > > > certain it
> > > > > > > > > > > > would run and do what I want, but I can't get
it to
> > work
> > > > > > > > > > > > in
> > > > > > > > > > > > an
> > > > > > > > > > > > automated fashion. (Essentially I want it to
do
> > something
> > > > > > > > > > > > similar to
> > > > > > > > > > > > series/stat-analysis, for example - aggregate
over
> > time)
> > > > > > > > > > > >
> > > > > > > > > > > > It is very possible I am doing something wrong
on my
> > end.
> > > > > > > > > > > > I
> > > > > > > > > > > > hope so
> > > > > > > > > as
> > > > > > > > > > > > that would be an easy fix. If this doesn't
make sense
> > or
> > > > > > > > > > > > if
> > > > > > > > > > > > you
> > > > > > > > > > > > would
> > > > > > > > > > > > like to discuss this on the phone at some
point, I am
> > > > > > > > > > > > happy
> > > > > > > > > > > > to
> > > > > > > > > > > > do so.
> > > > > > > > > > > > Just let me know what you think. Thanks in
advance
> for
> > > > > > > > > > > > your
> > > > > > > > > > > > help. If
> > > > > > > > > > > > my dir structure is confusing or you need me
to send
> > you
> > > > > > > > > > > > anything
> > > > > > > > > else
> > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Jamie
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > > > > obliged to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > >
> > > Jamie K. Wolff
> > > National Center for Atmospheric Research (NCAR)
> > > Research Applications Laboratory (RAL)
> > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 21 14:58:50 2020

I'm not entirely sure that is the cause of the error, but it is new
behavior that may be having an impact.

Here is how I used to run grid_diag (with the lists I created by
hand):
grid_diag -data
stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-21.txt
-data
stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-21.txt
-out
test_new.nc -config GridDiagConfig -v 4

With the list created now I get the error I sent:
grid_diag -data
stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt  -out
test_new_ALL.nc -config GridDiagConfig -v 4

...
MetGrib2DataFile::read_grib2_record_list() - processed 4 records from
GRIB2
file:
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of
type "FileType_Gb2".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of
type "FileType_Gb2".
ERROR  :
ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
abbreviation
'tb_g16'

Does that help explain?

You should be able to run both of those commands on the command line.
I am
using version 9.1.1 of MET with this environment:
module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
module load met/9.1.1
ncar_pylib

Thanks!
Jamie

On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Jamie,
>
> Yes, that error message should say LOOP_ORDER. Thanks for catching
that!
>
> Are you saying the error is occurring because all of the files are
lumped
> together instead of specified in separate -data arguments? If so, I
can
> modify the code to separate out each item in the template list.
>
> On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > Hi George,
> > I am trying to test your new mods in my processing.
> > I have a script to run grid_diag here:
> > /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> >
> > I made the change you mentioned needed to be added
> (GRID_DIAG_RUNTIME_FREQ
> > = RUN_ONCE) to my METplus parm and you can see the parm file
> >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> >
> > However, when I run it I get the following error:
> > 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode unles
> > s LOOP_BY = processes
> >
> > But that is confusing because I do not think LOOP_BY has an option
of
> > processes. Should that be LOOP_ORDER? That is what I tried and
that
> seemed
> > to get me further.
> >
> > Before, there would be two file_lists created one for the model
and one
> for
> > the obs
> > (For example:
> >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > But now it looks like they are both in one file:
> >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> >
> > I think this is causing a problem because I am getting an error:
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > DEBUG 1: Default Config File:
> >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > DEBUG 1: User Config File:
> > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > ERROR  :
> > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> abbreviation
> > 'tb_g16'
> > ERROR  :
> >
> > I am guessing this is a problem with my config file, not
necessarily your
> > logic. Could you take a look at my config file to see if I have
> additional
> > logic that needs to change for this to work? Let me know if you
need more
> > information from me.
> > Thanks!
> > Jamie
> >
> > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Jamie,
> > >
> > > Not a problem at all. You should just be able to obtain the
branch and
> > call
> > > that version of master_metplus.py instead of the one you are
currently
> > > using. Let me know if you run into any issues and I can help.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > Hi George,
> > > > I  am sorry, I am swamped this week. I hope to get to this
next week
> or
> > > the
> > > > week after.  In the meantime, are there any build instructions
you
> can
> > > > share with me to use when  I pull this branch on cheyenne?
> > > > I will be in touch as soon as I can.
> > > > Thanks,
> > > > Jamie
> > > >
> > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Jamie,
> > > > >
> > > > > Just following up on this in case this email was buried. Let
me
> know
> > if
> > > > > you need any help getting this to work!
> > > > >
> > > > > - George
> > > > >
> > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > Hi Jamie,
> > > > > >
> > > > > > I implemented the changes to GridDiag that were necessary
to run
> > the
> > > > > > tool as you expected. I assigned you to the pull request:
> > > > > >
> > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > >
> > > > > > The pull request has notes regarding which branch to
checkout on
> > > > > > cheyenne and which setting to add to get it to work. Let
me know
> if
> > > > > > you have any questions!
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > Sure. No problem. Thank you!
> > > > > > > Jamie
> > > > > > >
> > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > > > > > <met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > I created a GitHub issue for this work:
> > > > > > > >
> > > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > > >
> > > > > > > > I have implemented functionality to call certain
wrappers in
> > > > > > > > different
> > > > > > > > intervals to process groups of files instead of just
once per
> > > file.
> > > > > > > > I
> > > > > > > > am
> > > > > > > > now working on incorporating this functionality into
the
> > GridDiag
> > > > > > > > wrapper.
> > > > > > > > If you don't mind, I was planning on assigning the
pull
> request
> > > > > > > > review to
> > > > > > > > you so you can test these changes on your use case to
ensure
> it
> > > > > > > > runs
> > > > > > > > the
> > > > > > > > way you expect.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > Hi George and John,
> > > > > > > > > Sounds like the modifications you are discussing
will
> address
> > > my
> > > > > > > > > need
> > > > > > > > > (correct - the list of init times
> > > > > > > > > and forecast leads to determine which files to use
in a
> > single
> > > > > > > > > analysis).
> > > > > > > > > Thanks!
> > > > > > > > > Jamie
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via RT
> > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > OK, it sounds like I misunderstood the intended
use for
> the
> > > > > > > > > > tool
> > > > > > > > > > when
> > > > > > > > > > I
> > > > > > > > > > implemented the wrapper. Thanks for pointing this
out,
> > Jamie!
> > > > > > > > > >
> > > > > > > > > > John, I think we should have a discussion about
how we
> > would
> > > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > tools like SeriesAnalysis and GridDiag to be
configured
> to
> > > run
> > > > > > > > > > in
> > > > > > > > > > these
> > > > > > > > > > cases. I'll follow up with you to set something
up.
> > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > currently separated into SeriesByInit and
SeriesByLead
> > which
> > > > > > > > > > perform
> > > > > > > > > > the
> > > > > > > > > > grouping. However, I am currently refactoring
these tools
> > to
> > > > > > > > > > hopefully
> > > > > > > > > > combine the logic and allow options to specify
different
> > ways
> > > > > > > > > > to
> > > > > > > > > > filter/group the data. This would be a good time
to come
> up
> > > > > > > > > > with
> > > > > > > > > > a
> > > > > > > > > > good way
> > > > > > > > > > to define how you want to group your data in a
general
> way
> > > that
> > > > > > > > > > could
> > > > > > > > > > apply
> > > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > > >
> > > > > > > > > > Jamie, to confirm what you are saying, you
expected the
> > list
> > > of
> > > > > > > > > > init
> > > > > > > > > > times
> > > > > > > > > > and forecast leads to determine which files to use
in a
> > > single
> > > > > > > > > > analysis. Is
> > > > > > > > > > that correct?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley Gotway
via
> RT <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > George,
> > > > > > > > > > >
> > > > > > > > > > > The logic for running the Grid-Diag is more like
> Series-
> > > > > > > > > > > Analysis
> > > > > > > > > > > and
> > > > > > > > > > > Stat-Analysis than it is like Grid-Stat. While a
user
> > could
> > > > > > > > > > > run
> > > > > > > > > > > it
> > > > > > > > > > > for a
> > > > > > > > > > > single time, it is more likely they'll want to
run it
> > over
> > > a
> > > > > > > > > > > set of
> > > > > > > > > > times.
> > > > > > > > > > > It really is pretty analogous to Series-
Analysis... you
> > may
> > > > > > > > > > > want to
> > > > > > > > > > > hold
> > > > > > > > > > > the init time fixed and run over all lead
times... or
> > hold
> > > > > > > > > > > the
> > > > > > > > > > > lead
> > > > > > > > > > > time
> > > > > > > > > > > fixed and run over all init times... or just run
over
> all
> > > > > > > > > > > init
> > > > > > > > > > > and
> > > > > > > > > > > lead
> > > > > > > > > > > times in one big batch.
> > > > > > > > > > >
> > > > > > > > > > > George, is there any way to generalize the
support for
> > how
> > > > > > > > > > > times
> > > > > > > > > > > are
> > > > > > > > > > > processed?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe
via RT
> > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > > > > > > > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > >
> > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > >
> > > > > > > > > > > > The configuration you have set up loops over
each run
> > > time
> > > > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > range
> > > > > > > > > > > > you specified (2020060800 to 2020061500 with
24 hour
> > > > > > > > > > > > increments
> > > > > > > > > > > > and
> > > > > > > > > > > > processing the list of forecast leads for each
time).
> > > > > > > > > > > > However,
> > > > > > > > > > > > the
> > > > > > > > > > output
> > > > > > > > > > > > template will be the same for each run:
> > > > > > > > > > > >
> > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-
{ENV[FHR_LAST]}.nc
> > > > > > > > > > > >
> > > > > > > > > > > > This is why you are seeing only the results of
the
> last
> > > run
> > > > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > output.
> > > > > > > > > > > > Each run overwrites the data from the previous
run.
> > > > > > > > > > > >
> > > > > > > > > > > > Wrappers like SeriesAnalysis and StatAnalysis
are
> > > designed
> > > > > > > > > > > > to
> > > > > > > > > > > > process a
> > > > > > > > > > > > range of times at once, but GridDiag was
written to
> > > process
> > > > > > > > > > > > once
> > > > > > > > > > > > for
> > > > > > > > > > each
> > > > > > > > > > > > time. However, this may not be the intended
way to
> use
> > > the
> > > > > > > > > > > > tool.
> > > > > > > > > > > > This
> > > > > > > > > > is
> > > > > > > > > > > > how I implemented it based on the example I
was
> > provided.
> > > > > > > > > > > >
> > > > > > > > > > > > John HG (copied here) may have some insight on
the
> best
> > > way
> > > > > > > > > > > > to
> > > > > > > > > > > > achieve
> > > > > > > > > > > > what you are trying to do. Perhaps you would
want to
> > run
> > > > > > > > > > > > GridDiag
> > > > > > > > > > > > for
> > > > > > > > > > > each
> > > > > > > > > > > > run time, then run another tool to aggregate
the
> stats
> > > > > > > > > > > > across
> > > > > > > > > > > > all
> > > > > > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > output.
> > > > > > > > > > > >
> > > > > > > > > > > > If that is not the case, then you should be
able to
> get
> > > the
> > > > > > > > > > > > wrapper to
> > > > > > > > > > > run
> > > > > > > > > > > > the way you are intending by configuring it to
> process
> > a
> > > > > > > > > > > > single
> > > > > > > > > > > > run
> > > > > > > > > > time
> > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the list of
values
> for
> > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the times
you
> need
> > > > > > > > > > > > using
> > > > > > > > > > > > the
> > > > > > > > > > 'shift'
> > > > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > > > >
> > > > > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > so
> > > > > > > > > > > > on.
> > > > > > > > > > > >
> > > > > > > > > > > > This approach is not very flexible and would
require
> > you
> > > to
> > > > > > > > > > > > make
> > > > > > > > > > > > a lot
> > > > > > > > > > of
> > > > > > > > > > > > changes to the METplus config file for each
range of
> > > times
> > > > > > > > > > > > you
> > > > > > > > > > > > desire.
> > > > > > > > > > If
> > > > > > > > > > > > this is the only way to achieve what you need,
then
> we
> > > > > > > > > > > > should
> > > > > > > > > > > > make
> > > > > > > > > > > changes
> > > > > > > > > > > > to the wrapper to allow you to configure this
more
> > > easily.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > I am running grid-diag using the script
found
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > >
> > > > > > > > > > > > > It all appears to be running to completion
but the
> > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > not what I was expecting. To test the tool,
I first
> > set
> > > > > > > > > > > > > the
> > > > > > > > > > > > > script to
> > > > > > > > > > > > > run one day (INIT_BEG/END=2020061500) and it
> created
> > > two
> > > > > > > > > > > > > file
> > > > > > > > > > > > > lists
> > > > > > > > > > > > > (idx0/1) that I would have expected that
included
> > that
> > > > > > > > > > > > > one
> > > > > > > > > > > > > day.
> > > > > > > > > > > > > For
> > > > > > > > > > > > > example:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > I did the same for all my single days. It
created
> the
> > > > > > > > > > > > > files
> > > > > > > > > > > > > I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > expect. An example netcdf can be seen here:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > >
> > > > > > > > > > > > > Then, I wanted to run something similar for
all my
> > > > > > > > > > > > > available
> > > > > > > > > > > > > data
> > > > > > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > but
> > > > > > > > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > > > > like to have all days aggregated together
and one
> > > netcdf
> > > > > > > > > > > > > file
> > > > > > > > > > > > > created
> > > > > > > > > > > > > in the end. It did create one netcdf file at
the
> end,
> > > but
> > > > > > > > > > > > > it
> > > > > > > > > > > > > only
> > > > > > > > > > > > > included the data for the last date (the
15th).
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I could be confused on the intent of the
tool.
> > Perhaps
> > > > > > > > > > > > > this
> > > > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > behavior you want it to have. If so, I still
think
> > > there
> > > > > > > > > > > > > is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > bug
> > > > > > > > > > > > > because it should be outputting a file per
day, and
> > it
> > > > > > > > > > > > > appears
> > > > > > > > > > > > > to
> > > > > > > > > > only
> > > > > > > > > > > > > output a file at the end for the last day.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What I thought would happen is I would get
one
> file,
> > > but
> > > > > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > include all days/times requested. So, what I
need
> it
> > to
> > > > > > > > > > > > > do
> > > > > > > > > > > > > is
> > > > > > > > > > > > > create
> > > > > > > > > > a
> > > > > > > > > > > > > list that includes all model (idx0) and obs
(idx1)
> > > files
> > > > > > > > > > > > > that I
> > > > > > > > > > > > > want
> > > > > > > > > > > > > to use in one file. I can do this by hand
and I am
> > > quite
> > > > > > > > > > > > > certain it
> > > > > > > > > > > > > would run and do what I want, but I can't
get it to
> > > work
> > > > > > > > > > > > > in
> > > > > > > > > > > > > an
> > > > > > > > > > > > > automated fashion. (Essentially I want it to
do
> > > something
> > > > > > > > > > > > > similar to
> > > > > > > > > > > > > series/stat-analysis, for example -
aggregate over
> > > time)
> > > > > > > > > > > > >
> > > > > > > > > > > > > It is very possible I am doing something
wrong on
> my
> > > end.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > hope so
> > > > > > > > > > as
> > > > > > > > > > > > > that would be an easy fix. If this doesn't
make
> sense
> > > or
> > > > > > > > > > > > > if
> > > > > > > > > > > > > you
> > > > > > > > > > > > > would
> > > > > > > > > > > > > like to discuss this on the phone at some
point, I
> am
> > > > > > > > > > > > > happy
> > > > > > > > > > > > > to
> > > > > > > > > > > > > do so.
> > > > > > > > > > > > > Just let me know what you think. Thanks in
advance
> > for
> > > > > > > > > > > > > your
> > > > > > > > > > > > > help. If
> > > > > > > > > > > > > my dir structure is confusing or you need me
to
> send
> > > you
> > > > > > > > > > > > > anything
> > > > > > > > > > else
> > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > Jamie
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > > > > obliged to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Jamie K. Wolff
> > > > National Center for Atmospheric Research (NCAR)
> > > > Research Applications Laboratory (RAL)
> > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > Jamie K. Wolff
> > National Center for Atmospheric Research (NCAR)
> > Research Applications Laboratory (RAL)
> > Phone: 303.497.2812  Email: jwolff at ucar.edu
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 21 15:03:25 2020

Hi Jamie,

>From my understanding that is likely the cause of the error. I made
some
changes that should fix it for your case. Could you try running your
case
using this master_metplus.py on cheyenne?

/glade/u/home/mccabe/METplus/ush/master_metplus.py

You could also run "git pull" on the feature_733_grid_diag_freq branch
if
that is easier. Let me know how it goes.

Thanks,
George

On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> I'm not entirely sure that is the cause of the error, but it is new
> behavior that may be having an impact.
>
> Here is how I used to run grid_diag (with the lists I created by
hand):
> grid_diag -data
> stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt -data
> stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt -out
> test_new.nc -config GridDiagConfig -v 4
>
> With the list created now I get the error I sent:
> grid_diag -data
> stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
-out
> test_new_ALL.nc -config GridDiagConfig -v 4
>
> ...
> MetGrib2DataFile::read_grib2_record_list() - processed 4 records
from GRIB2
> file:
>
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Gb2".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Gb2".
> ERROR  :
> ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
abbreviation
> 'tb_g16'
>
> Does that help explain?
>
> You should be able to run both of those commands on the command
line. I am
> using version 9.1.1 of MET with this environment:
> module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> module load met/9.1.1
> ncar_pylib
>
> Thanks!
> Jamie
>
>
> On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Jamie,
> >
> > Yes, that error message should say LOOP_ORDER. Thanks for catching
that!
> >
> > Are you saying the error is occurring because all of the files are
lumped
> > together instead of specified in separate -data arguments? If so,
I can
> > modify the code to separate out each item in the template list.
> >
> > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > Hi George,
> > > I am trying to test your new mods in my processing.
> > > I have a script to run grid_diag here:
> > > /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > >
> > > I made the change you mentioned needed to be added
> > (GRID_DIAG_RUNTIME_FREQ
> > > = RUN_ONCE) to my METplus parm and you can see the parm file
> > >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > >
> > > However, when I run it I get the following error:
> > > 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode
unles
> > > s LOOP_BY = processes
> > >
> > > But that is confusing because I do not think LOOP_BY has an
option of
> > > processes. Should that be LOOP_ORDER? That is what I tried and
that
> > seemed
> > > to get me further.
> > >
> > > Before, there would be two file_lists created one for the model
and one
> > for
> > > the obs
> > > (For example:
> > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > But now it looks like they are both in one file:
> > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > >
> > > I think this is causing a problem because I am getting an error:
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > DEBUG 1: Default Config File:
> > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > DEBUG 1: User Config File:
> > > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > ERROR  :
> > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> > abbreviation
> > > 'tb_g16'
> > > ERROR  :
> > >
> > > I am guessing this is a problem with my config file, not
necessarily
> your
> > > logic. Could you take a look at my config file to see if I have
> > additional
> > > logic that needs to change for this to work? Let me know if you
need
> more
> > > information from me.
> > > Thanks!
> > > Jamie
> > >
> > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Jamie,
> > > >
> > > > Not a problem at all. You should just be able to obtain the
branch
> and
> > > call
> > > > that version of master_metplus.py instead of the one you are
> currently
> > > > using. Let me know if you run into any issues and I can help.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > Hi George,
> > > > > I  am sorry, I am swamped this week. I hope to get to this
next
> week
> > or
> > > > the
> > > > > week after.  In the meantime, are there any build
instructions you
> > can
> > > > > share with me to use when  I pull this branch on cheyenne?
> > > > > I will be in touch as soon as I can.
> > > > > Thanks,
> > > > > Jamie
> > > > >
> > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > Just following up on this in case this email was buried.
Let me
> > know
> > > if
> > > > > > you need any help getting this to work!
> > > > > >
> > > > > > - George
> > > > > >
> > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > I implemented the changes to GridDiag that were
necessary to
> run
> > > the
> > > > > > > tool as you expected. I assigned you to the pull
request:
> > > > > > >
> > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > >
> > > > > > > The pull request has notes regarding which branch to
checkout
> on
> > > > > > > cheyenne and which setting to add to get it to work. Let
me
> know
> > if
> > > > > > > you have any questions!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > Sure. No problem. Thank you!
> > > > > > > > Jamie
> > > > > > > >
> > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via RT
> > > > > > > > <met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jamie,
> > > > > > > > >
> > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > >
> > > > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > > > >
> > > > > > > > > I have implemented functionality to call certain
wrappers
> in
> > > > > > > > > different
> > > > > > > > > intervals to process groups of files instead of just
once
> per
> > > > file.
> > > > > > > > > I
> > > > > > > > > am
> > > > > > > > > now working on incorporating this functionality into
the
> > > GridDiag
> > > > > > > > > wrapper.
> > > > > > > > > If you don't mind, I was planning on assigning the
pull
> > request
> > > > > > > > > review to
> > > > > > > > > you so you can test these changes on your use case
to
> ensure
> > it
> > > > > > > > > runs
> > > > > > > > > the
> > > > > > > > > way you expect.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > Hi George and John,
> > > > > > > > > > Sounds like the modifications you are discussing
will
> > address
> > > > my
> > > > > > > > > > need
> > > > > > > > > > (correct - the list of init times
> > > > > > > > > > and forecast leads to determine which files to use
in a
> > > single
> > > > > > > > > > analysis).
> > > > > > > > > > Thanks!
> > > > > > > > > > Jamie
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe via
RT
> > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > OK, it sounds like I misunderstood the intended
use for
> > the
> > > > > > > > > > > tool
> > > > > > > > > > > when
> > > > > > > > > > > I
> > > > > > > > > > > implemented the wrapper. Thanks for pointing
this out,
> > > Jamie!
> > > > > > > > > > >
> > > > > > > > > > > John, I think we should have a discussion about
how we
> > > would
> > > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > tools like SeriesAnalysis and GridDiag to be
configured
> > to
> > > > run
> > > > > > > > > > > in
> > > > > > > > > > > these
> > > > > > > > > > > cases. I'll follow up with you to set something
up.
> > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > currently separated into SeriesByInit and
SeriesByLead
> > > which
> > > > > > > > > > > perform
> > > > > > > > > > > the
> > > > > > > > > > > grouping. However, I am currently refactoring
these
> tools
> > > to
> > > > > > > > > > > hopefully
> > > > > > > > > > > combine the logic and allow options to specify
> different
> > > ways
> > > > > > > > > > > to
> > > > > > > > > > > filter/group the data. This would be a good time
to
> come
> > up
> > > > > > > > > > > with
> > > > > > > > > > > a
> > > > > > > > > > > good way
> > > > > > > > > > > to define how you want to group your data in a
general
> > way
> > > > that
> > > > > > > > > > > could
> > > > > > > > > > > apply
> > > > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > > > >
> > > > > > > > > > > Jamie, to confirm what you are saying, you
expected the
> > > list
> > > > of
> > > > > > > > > > > init
> > > > > > > > > > > times
> > > > > > > > > > > and forecast leads to determine which files to
use in a
> > > > single
> > > > > > > > > > > analysis. Is
> > > > > > > > > > > that correct?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley
Gotway via
> > RT <
> > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > George,
> > > > > > > > > > > >
> > > > > > > > > > > > The logic for running the Grid-Diag is more
like
> > Series-
> > > > > > > > > > > > Analysis
> > > > > > > > > > > > and
> > > > > > > > > > > > Stat-Analysis than it is like Grid-Stat. While
a user
> > > could
> > > > > > > > > > > > run
> > > > > > > > > > > > it
> > > > > > > > > > > > for a
> > > > > > > > > > > > single time, it is more likely they'll want to
run it
> > > over
> > > > a
> > > > > > > > > > > > set of
> > > > > > > > > > > times.
> > > > > > > > > > > > It really is pretty analogous to Series-
Analysis...
> you
> > > may
> > > > > > > > > > > > want to
> > > > > > > > > > > > hold
> > > > > > > > > > > > the init time fixed and run over all lead
times... or
> > > hold
> > > > > > > > > > > > the
> > > > > > > > > > > > lead
> > > > > > > > > > > > time
> > > > > > > > > > > > fixed and run over all init times... or just
run over
> > all
> > > > > > > > > > > > init
> > > > > > > > > > > > and
> > > > > > > > > > > > lead
> > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > >
> > > > > > > > > > > > George, is there any way to generalize the
support
> for
> > > how
> > > > > > > > > > > > times
> > > > > > > > > > > > are
> > > > > > > > > > > > processed?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George McCabe
via RT
> > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL:
> > > > > > > > > > > > >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The configuration you have set up loops over
each
> run
> > > > time
> > > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > range
> > > > > > > > > > > > > you specified (2020060800 to 2020061500 with
24
> hour
> > > > > > > > > > > > > increments
> > > > > > > > > > > > > and
> > > > > > > > > > > > > processing the list of forecast leads for
each
> time).
> > > > > > > > > > > > > However,
> > > > > > > > > > > > > the
> > > > > > > > > > > output
> > > > > > > > > > > > > template will be the same for each run:
> > > > > > > > > > > > >
> > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-
{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is why you are seeing only the results
of the
> > last
> > > > run
> > > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > output.
> > > > > > > > > > > > > Each run overwrites the data from the
previous run.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Wrappers like SeriesAnalysis and
StatAnalysis are
> > > > designed
> > > > > > > > > > > > > to
> > > > > > > > > > > > > process a
> > > > > > > > > > > > > range of times at once, but GridDiag was
written to
> > > > process
> > > > > > > > > > > > > once
> > > > > > > > > > > > > for
> > > > > > > > > > > each
> > > > > > > > > > > > > time. However, this may not be the intended
way to
> > use
> > > > the
> > > > > > > > > > > > > tool.
> > > > > > > > > > > > > This
> > > > > > > > > > > is
> > > > > > > > > > > > > how I implemented it based on the example I
was
> > > provided.
> > > > > > > > > > > > >
> > > > > > > > > > > > > John HG (copied here) may have some insight
on the
> > best
> > > > way
> > > > > > > > > > > > > to
> > > > > > > > > > > > > achieve
> > > > > > > > > > > > > what you are trying to do. Perhaps you would
want
> to
> > > run
> > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > for
> > > > > > > > > > > > each
> > > > > > > > > > > > > run time, then run another tool to aggregate
the
> > stats
> > > > > > > > > > > > > across
> > > > > > > > > > > > > all
> > > > > > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > output.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If that is not the case, then you should be
able to
> > get
> > > > the
> > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > run
> > > > > > > > > > > > > the way you are intending by configuring it
to
> > process
> > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > run
> > > > > > > > > > > time
> > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the list of
values
> > for
> > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the
times you
> > need
> > > > > > > > > > > > > using
> > > > > > > > > > > > > the
> > > > > > > > > > > 'shift'
> > > > > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > > > > >
> > > > > > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > so
> > > > > > > > > > > > > on.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This approach is not very flexible and would
> require
> > > you
> > > > to
> > > > > > > > > > > > > make
> > > > > > > > > > > > > a lot
> > > > > > > > > > > of
> > > > > > > > > > > > > changes to the METplus config file for each
range
> of
> > > > times
> > > > > > > > > > > > > you
> > > > > > > > > > > > > desire.
> > > > > > > > > > > If
> > > > > > > > > > > > > this is the only way to achieve what you
need, then
> > we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > make
> > > > > > > > > > > > changes
> > > > > > > > > > > > > to the wrapper to allow you to configure
this more
> > > > easily.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > I am running grid-diag using the script
found
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It all appears to be running to completion
but
> the
> > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > not what I was expecting. To test the
tool, I
> first
> > > set
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > run one day (INIT_BEG/END=2020061500) and
it
> > created
> > > > two
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > (idx0/1) that I would have expected that
included
> > > that
> > > > > > > > > > > > > > one
> > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > For
> > > > > > > > > > > > > > example:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > I did the same for all my single days. It
created
> > the
> > > > > > > > > > > > > > files
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > expect. An example netcdf can be seen
here:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Then, I wanted to run something similar
for all
> my
> > > > > > > > > > > > > > available
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > (INIT_BEG=2020060800, INIT_END=2020061500,
> > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > > > like to have all days aggregated together
and one
> > > > netcdf
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > created
> > > > > > > > > > > > > > in the end. It did create one netcdf file
at the
> > end,
> > > > but
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > included the data for the last date (the
15th).
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I could be confused on the intent of the
tool.
> > > Perhaps
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > behavior you want it to have. If so, I
still
> think
> > > > there
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > because it should be outputting a file per
day,
> and
> > > it
> > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > to
> > > > > > > > > > > only
> > > > > > > > > > > > > > output a file at the end for the last day.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What I thought would happen is I would get
one
> > file,
> > > > but
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > include all days/times requested. So, what
I need
> > it
> > > to
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > create
> > > > > > > > > > > a
> > > > > > > > > > > > > > list that includes all model (idx0) and
obs
> (idx1)
> > > > files
> > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > to use in one file. I can do this by hand
and I
> am
> > > > quite
> > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > would run and do what I want, but I can't
get it
> to
> > > > work
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > automated fashion. (Essentially I want it
to do
> > > > something
> > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > series/stat-analysis, for example -
aggregate
> over
> > > > time)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It is very possible I am doing something
wrong on
> > my
> > > > end.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > hope so
> > > > > > > > > > > as
> > > > > > > > > > > > > > that would be an easy fix. If this doesn't
make
> > sense
> > > > or
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > like to discuss this on the phone at some
point,
> I
> > am
> > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > Just let me know what you think. Thanks in
> advance
> > > for
> > > > > > > > > > > > > > your
> > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > my dir structure is confusing or you need
me to
> > send
> > > > you
> > > > > > > > > > > > > > anything
> > > > > > > > > > > else
> > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > feel
> > > > > > > > > > > obliged to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jamie K. Wolff
> > > > > National Center for Atmospheric Research (NCAR)
> > > > > Research Applications Laboratory (RAL)
> > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > Jamie K. Wolff
> > > National Center for Atmospheric Research (NCAR)
> > > Research Applications Laboratory (RAL)
> > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 21 15:32:19 2020

That did the trick! I can now reproduce what I was doing by hand.

I tested all four options and they all seemed to work great. This is a
great enhancement - thank you!

One remaining question is figuring out the best way to set the
template
file name so it is appropriately filled and not overwriting files. Can
you
suggest what I might change the below line to so it would work for the
new
options?

GRID_DIAG_OUTPUT_TEMPLATE =
grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc

Thank you!
Jamie

On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Jamie,
>
> From my understanding that is likely the cause of the error. I made
some
> changes that should fix it for your case. Could you try running your
case
> using this master_metplus.py on cheyenne?
>
> /glade/u/home/mccabe/METplus/ush/master_metplus.py
>
> You could also run "git pull" on the feature_733_grid_diag_freq
branch if
> that is easier. Let me know how it goes.
>
> Thanks,
> George
>
> On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > I'm not entirely sure that is the cause of the error, but it is
new
> > behavior that may be having an impact.
> >
> > Here is how I used to run grid_diag (with the lists I created by
hand):
> > grid_diag -data
> > stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt
> -data
> > stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt
> -out
> > test_new.nc -config GridDiagConfig -v 4
> >
> > With the list created now I get the error I sent:
> > grid_diag -data
> > stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
-out
> > test_new_ALL.nc -config GridDiagConfig -v 4
> >
> > ...
> > MetGrib2DataFile::read_grib2_record_list() - processed 4 records
from
> GRIB2
> > file:
> >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> > type "FileType_Gb2".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> > type "FileType_Gb2".
> > ERROR  :
> > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> abbreviation
> > 'tb_g16'
> >
> > Does that help explain?
> >
> > You should be able to run both of those commands on the command
line. I
> am
> > using version 9.1.1 of MET with this environment:
> > module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> > module load met/9.1.1
> > ncar_pylib
> >
> > Thanks!
> > Jamie
> >
> >
> > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Jamie,
> > >
> > > Yes, that error message should say LOOP_ORDER. Thanks for
catching
> that!
> > >
> > > Are you saying the error is occurring because all of the files
are
> lumped
> > > together instead of specified in separate -data arguments? If
so, I can
> > > modify the code to separate out each item in the template list.
> > >
> > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > Hi George,
> > > > I am trying to test your new mods in my processing.
> > > > I have a script to run grid_diag here:
> > > > /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > >
> > > > I made the change you mentioned needed to be added
> > > (GRID_DIAG_RUNTIME_FREQ
> > > > = RUN_ONCE) to my METplus parm and you can see the parm file
> > > >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > >
> > > > However, when I run it I get the following error:
> > > > 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> > > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode
unles
> > > > s LOOP_BY = processes
> > > >
> > > > But that is confusing because I do not think LOOP_BY has an
option of
> > > > processes. Should that be LOOP_ORDER? That is what I tried and
that
> > > seemed
> > > > to get me further.
> > > >
> > > > Before, there would be two file_lists created one for the
model and
> one
> > > for
> > > > the obs
> > > > (For example:
> > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > But now it looks like they are both in one file:
> > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > >
> > > > I think this is causing a problem because I am getting an
error:
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > DEBUG 1: Default Config File:
> > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > DEBUG 1: User Config File:
> > > > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > ERROR  :
> > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> > > abbreviation
> > > > 'tb_g16'
> > > > ERROR  :
> > > >
> > > > I am guessing this is a problem with my config file, not
necessarily
> > your
> > > > logic. Could you take a look at my config file to see if I
have
> > > additional
> > > > logic that needs to change for this to work? Let me know if
you need
> > more
> > > > information from me.
> > > > Thanks!
> > > > Jamie
> > > >
> > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Jamie,
> > > > >
> > > > > Not a problem at all. You should just be able to obtain the
branch
> > and
> > > > call
> > > > > that version of master_metplus.py instead of the one you are
> > currently
> > > > > using. Let me know if you run into any issues and I can
help.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > > >
> > > > > > Hi George,
> > > > > > I  am sorry, I am swamped this week. I hope to get to this
next
> > week
> > > or
> > > > > the
> > > > > > week after.  In the meantime, are there any build
instructions
> you
> > > can
> > > > > > share with me to use when  I pull this branch on cheyenne?
> > > > > > I will be in touch as soon as I can.
> > > > > > Thanks,
> > > > > > Jamie
> > > > > >
> > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > Just following up on this in case this email was buried.
Let me
> > > know
> > > > if
> > > > > > > you need any help getting this to work!
> > > > > > >
> > > > > > > - George
> > > > > > >
> > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > I implemented the changes to GridDiag that were
necessary to
> > run
> > > > the
> > > > > > > > tool as you expected. I assigned you to the pull
request:
> > > > > > > >
> > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > >
> > > > > > > > The pull request has notes regarding which branch to
checkout
> > on
> > > > > > > > cheyenne and which setting to add to get it to work.
Let me
> > know
> > > if
> > > > > > > > you have any questions!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > Jamie
> > > > > > > > >
> > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via
RT
> > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jamie,
> > > > > > > > > >
> > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > >
> > > > > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > >
> > > > > > > > > > I have implemented functionality to call certain
wrappers
> > in
> > > > > > > > > > different
> > > > > > > > > > intervals to process groups of files instead of
just once
> > per
> > > > > file.
> > > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > now working on incorporating this functionality
into the
> > > > GridDiag
> > > > > > > > > > wrapper.
> > > > > > > > > > If you don't mind, I was planning on assigning the
pull
> > > request
> > > > > > > > > > review to
> > > > > > > > > > you so you can test these changes on your use case
to
> > ensure
> > > it
> > > > > > > > > > runs
> > > > > > > > > > the
> > > > > > > > > > way you expect.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > > Hi George and John,
> > > > > > > > > > > Sounds like the modifications you are discussing
will
> > > address
> > > > > my
> > > > > > > > > > > need
> > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > and forecast leads to determine which files to
use in a
> > > > single
> > > > > > > > > > > analysis).
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Jamie
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe
via RT
> > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > OK, it sounds like I misunderstood the
intended use
> for
> > > the
> > > > > > > > > > > > tool
> > > > > > > > > > > > when
> > > > > > > > > > > > I
> > > > > > > > > > > > implemented the wrapper. Thanks for pointing
this
> out,
> > > > Jamie!
> > > > > > > > > > > >
> > > > > > > > > > > > John, I think we should have a discussion
about how
> we
> > > > would
> > > > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > have
> > > > > > > > > > > > tools like SeriesAnalysis and GridDiag to be
> configured
> > > to
> > > > > run
> > > > > > > > > > > > in
> > > > > > > > > > > > these
> > > > > > > > > > > > cases. I'll follow up with you to set
something up.
> > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > currently separated into SeriesByInit and
> SeriesByLead
> > > > which
> > > > > > > > > > > > perform
> > > > > > > > > > > > the
> > > > > > > > > > > > grouping. However, I am currently refactoring
these
> > tools
> > > > to
> > > > > > > > > > > > hopefully
> > > > > > > > > > > > combine the logic and allow options to specify
> > different
> > > > ways
> > > > > > > > > > > > to
> > > > > > > > > > > > filter/group the data. This would be a good
time to
> > come
> > > up
> > > > > > > > > > > > with
> > > > > > > > > > > > a
> > > > > > > > > > > > good way
> > > > > > > > > > > > to define how you want to group your data in a
> general
> > > way
> > > > > that
> > > > > > > > > > > > could
> > > > > > > > > > > > apply
> > > > > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > > > > >
> > > > > > > > > > > > Jamie, to confirm what you are saying, you
expected
> the
> > > > list
> > > > > of
> > > > > > > > > > > > init
> > > > > > > > > > > > times
> > > > > > > > > > > > and forecast leads to determine which files to
use
> in a
> > > > > single
> > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > that correct?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley
Gotway
> via
> > > RT <
> > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > George,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The logic for running the Grid-Diag is more
like
> > > Series-
> > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > and
> > > > > > > > > > > > > Stat-Analysis than it is like Grid-Stat.
While a
> user
> > > > could
> > > > > > > > > > > > > run
> > > > > > > > > > > > > it
> > > > > > > > > > > > > for a
> > > > > > > > > > > > > single time, it is more likely they'll want
to run
> it
> > > > over
> > > > > a
> > > > > > > > > > > > > set of
> > > > > > > > > > > > times.
> > > > > > > > > > > > > It really is pretty analogous to Series-
Analysis...
> > you
> > > > may
> > > > > > > > > > > > > want to
> > > > > > > > > > > > > hold
> > > > > > > > > > > > > the init time fixed and run over all lead
times...
> or
> > > > hold
> > > > > > > > > > > > > the
> > > > > > > > > > > > > lead
> > > > > > > > > > > > > time
> > > > > > > > > > > > > fixed and run over all init times... or just
run
> over
> > > all
> > > > > > > > > > > > > init
> > > > > > > > > > > > > and
> > > > > > > > > > > > > lead
> > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > >
> > > > > > > > > > > > > George, is there any way to generalize the
support
> > for
> > > > how
> > > > > > > > > > > > > times
> > > > > > > > > > > > > are
> > > > > > > > > > > > > processed?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George
McCabe via
> RT
> > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > >
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The configuration you have set up loops
over each
> > run
> > > > > time
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > range
> > > > > > > > > > > > > > you specified (2020060800 to 2020061500
with 24
> > hour
> > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > processing the list of forecast leads for
each
> > time).
> > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > output
> > > > > > > > > > > > > > template will be the same for each run:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-
{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is why you are seeing only the
results of
> the
> > > last
> > > > > run
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > output.
> > > > > > > > > > > > > > Each run overwrites the data from the
previous
> run.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
StatAnalysis are
> > > > > designed
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > range of times at once, but GridDiag was
written
> to
> > > > > process
> > > > > > > > > > > > > > once
> > > > > > > > > > > > > > for
> > > > > > > > > > > > each
> > > > > > > > > > > > > > time. However, this may not be the
intended way
> to
> > > use
> > > > > the
> > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > is
> > > > > > > > > > > > > > how I implemented it based on the example
I was
> > > > provided.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > John HG (copied here) may have some
insight on
> the
> > > best
> > > > > way
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > what you are trying to do. Perhaps you
would want
> > to
> > > > run
> > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > run time, then run another tool to
aggregate the
> > > stats
> > > > > > > > > > > > > > across
> > > > > > > > > > > > > > all
> > > > > > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > output.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If that is not the case, then you should
be able
> to
> > > get
> > > > > the
> > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > the way you are intending by configuring
it to
> > > process
> > > > a
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > run
> > > > > > > > > > > > time
> > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the list
of
> values
> > > for
> > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the
times you
> > > need
> > > > > > > > > > > > > > using
> > > > > > > > > > > > > > the
> > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {init?fmt=...}, {init?fmt=...?shift=24H},
> > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > on.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This approach is not very flexible and
would
> > require
> > > > you
> > > > > to
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > a lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > > changes to the METplus config file for
each range
> > of
> > > > > times
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > desire.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > this is the only way to achieve what you
need,
> then
> > > we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > to the wrapper to allow you to configure
this
> more
> > > > > easily.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > George
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe wrote:
> > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > I am running grid-diag using the script
found
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It all appears to be running to
completion but
> > the
> > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > not what I was expecting. To test the
tool, I
> > first
> > > > set
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > run one day (INIT_BEG/END=2020061500)
and it
> > > created
> > > > > two
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > (idx0/1) that I would have expected that
> included
> > > > that
> > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > I did the same for all my single days.
It
> created
> > > the
> > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > expect. An example netcdf can be seen
here:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Then, I wanted to run something similar
for all
> > my
> > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > (INIT_BEG=2020060800,
INIT_END=2020061500,
> > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > like to have all days aggregated
together and
> one
> > > > > netcdf
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > in the end. It did create one netcdf
file at
> the
> > > end,
> > > > > but
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > included the data for the last date (the
15th).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc and
> > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I could be confused on the intent of the
tool.
> > > > Perhaps
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > behavior you want it to have. If so, I
still
> > think
> > > > > there
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > because it should be outputting a file
per day,
> > and
> > > > it
> > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > only
> > > > > > > > > > > > > > > output a file at the end for the last
day.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What I thought would happen is I would
get one
> > > file,
> > > > > but
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > include all days/times requested. So,
what I
> need
> > > it
> > > > to
> > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > create
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > list that includes all model (idx0) and
obs
> > (idx1)
> > > > > files
> > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to use in one file. I can do this by
hand and I
> > am
> > > > > quite
> > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > would run and do what I want, but I
can't get
> it
> > to
> > > > > work
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > automated fashion. (Essentially I want
it to do
> > > > > something
> > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > series/stat-analysis, for example -
aggregate
> > over
> > > > > time)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It is very possible I am doing something
wrong
> on
> > > my
> > > > > end.
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > that would be an easy fix. If this
doesn't make
> > > sense
> > > > > or
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > like to discuss this on the phone at
some
> point,
> > I
> > > am
> > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > Just let me know what you think. Thanks
in
> > advance
> > > > for
> > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > my dir structure is confusing or you
need me to
> > > send
> > > > > you
> > > > > > > > > > > > > > > anything
> > > > > > > > > > > > else
> > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > ---
> > > > > > > > > > > > My working day may not be your working day.
Please do
> > not
> > > > > feel
> > > > > > > > > > > > obliged to
> > > > > > > > > > > > reply to this email outside of your normal
working
> > hours.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jamie K. Wolff
> > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > Research Applications Laboratory (RAL)
> > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > >
> > > > > > *My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Jamie K. Wolff
> > > > National Center for Atmospheric Research (NCAR)
> > > > Research Applications Laboratory (RAL)
> > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > Jamie K. Wolff
> > National Center for Atmospheric Research (NCAR)
> > Research Applications Laboratory (RAL)
> > Phone: 303.497.2812  Email: jwolff at ucar.edu
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 21 15:37:00 2020

Glad it worked! I'm not sure I understand the issue with the output
template. If Grid-Diag is run once for all of the files, isn't the
template
you have there unique enough to that case (model, time ranges) that it
wouldn't be overwritten?

On Mon, Dec 21, 2020 at 3:32 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> That did the trick! I can now reproduce what I was doing by hand.
>
> I tested all four options and they all seemed to work great. This is
a
> great enhancement - thank you!
>
> One remaining question is figuring out the best way to set the
template
> file name so it is appropriately filled and not overwriting files.
Can you
> suggest what I might change the below line to so it would work for
the new
> options?
>
> GRID_DIAG_OUTPUT_TEMPLATE =
>
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
>
> Thank you!
> Jamie
>
> On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Jamie,
> >
> > From my understanding that is likely the cause of the error. I
made some
> > changes that should fix it for your case. Could you try running
your case
> > using this master_metplus.py on cheyenne?
> >
> > /glade/u/home/mccabe/METplus/ush/master_metplus.py
> >
> > You could also run "git pull" on the feature_733_grid_diag_freq
branch if
> > that is easier. Let me know how it goes.
> >
> > Thanks,
> > George
> >
> > On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > I'm not entirely sure that is the cause of the error, but it is
new
> > > behavior that may be having an impact.
> > >
> > > Here is how I used to run grid_diag (with the lists I created by
hand):
> > > grid_diag -data
> > > stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt
> > -data
> > > stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt
> > -out
> > > test_new.nc -config GridDiagConfig -v 4
> > >
> > > With the list created now I get the error I sent:
> > > grid_diag -data
> > > stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
-out
> > > test_new_ALL.nc -config GridDiagConfig -v 4
> > >
> > > ...
> > > MetGrib2DataFile::read_grib2_record_list() - processed 4 records
from
> > GRIB2
> > > file:
> > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of
> > > type "FileType_Gb2".
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of
> > > type "FileType_Gb2".
> > > ERROR  :
> > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> > abbreviation
> > > 'tb_g16'
> > >
> > > Does that help explain?
> > >
> > > You should be able to run both of those commands on the command
line. I
> > am
> > > using version 9.1.1 of MET with this environment:
> > > module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> > > module load met/9.1.1
> > > ncar_pylib
> > >
> > > Thanks!
> > > Jamie
> > >
> > >
> > > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Jamie,
> > > >
> > > > Yes, that error message should say LOOP_ORDER. Thanks for
catching
> > that!
> > > >
> > > > Are you saying the error is occurring because all of the files
are
> > lumped
> > > > together instead of specified in separate -data arguments? If
so, I
> can
> > > > modify the code to separate out each item in the template
list.
> > > >
> > > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > Hi George,
> > > > > I am trying to test your new mods in my processing.
> > > > > I have a script to run grid_diag here:
> > > > >
/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > >
> > > > > I made the change you mentioned needed to be added
> > > > (GRID_DIAG_RUNTIME_FREQ
> > > > > = RUN_ONCE) to my METplus parm and you can see the parm file
> > > > >
> here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > > >
> > > > > However, when I run it I get the following error:
> > > > > 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> > > > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE mode
unles
> > > > > s LOOP_BY = processes
> > > > >
> > > > > But that is confusing because I do not think LOOP_BY has an
option
> of
> > > > > processes. Should that be LOOP_ORDER? That is what I tried
and that
> > > > seemed
> > > > > to get me further.
> > > > >
> > > > > Before, there would be two file_lists created one for the
model and
> > one
> > > > for
> > > > > the obs
> > > > > (For example:
> > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > > But now it looks like they are both in one file:
> > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > >
> > > > > I think this is causing a problem because I am getting an
error:
> > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > > DEBUG 1: Default Config File:
> > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > > DEBUG 1: User Config File:
> > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > > ERROR  :
> > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > abbreviation
> > > > > 'tb_g16'
> > > > > ERROR  :
> > > > >
> > > > > I am guessing this is a problem with my config file, not
> necessarily
> > > your
> > > > > logic. Could you take a look at my config file to see if I
have
> > > > additional
> > > > > logic that needs to change for this to work? Let me know if
you
> need
> > > more
> > > > > information from me.
> > > > > Thanks!
> > > > > Jamie
> > > > >
> > > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > Not a problem at all. You should just be able to obtain
the
> branch
> > > and
> > > > > call
> > > > > > that version of master_metplus.py instead of the one you
are
> > > currently
> > > > > > using. Let me know if you run into any issues and I can
help.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> >
> > > > > > >
> > > > > > > Hi George,
> > > > > > > I  am sorry, I am swamped this week. I hope to get to
this next
> > > week
> > > > or
> > > > > > the
> > > > > > > week after.  In the meantime, are there any build
instructions
> > you
> > > > can
> > > > > > > share with me to use when  I pull this branch on
cheyenne?
> > > > > > > I will be in touch as soon as I can.
> > > > > > > Thanks,
> > > > > > > Jamie
> > > > > > >
> > > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > Just following up on this in case this email was
buried. Let
> me
> > > > know
> > > > > if
> > > > > > > > you need any help getting this to work!
> > > > > > > >
> > > > > > > > - George
> > > > > > > >
> > > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > > Hi Jamie,
> > > > > > > > >
> > > > > > > > > I implemented the changes to GridDiag that were
necessary
> to
> > > run
> > > > > the
> > > > > > > > > tool as you expected. I assigned you to the pull
request:
> > > > > > > > >
> > > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > > >
> > > > > > > > > The pull request has notes regarding which branch to
> checkout
> > > on
> > > > > > > > > cheyenne and which setting to add to get it to work.
Let me
> > > know
> > > > if
> > > > > > > > > you have any questions!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > > Jamie
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe via
RT
> > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jamie,
> > > > > > > > > > >
> > > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > > >
> > > > > > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > > >
> > > > > > > > > > > I have implemented functionality to call certain
> wrappers
> > > in
> > > > > > > > > > > different
> > > > > > > > > > > intervals to process groups of files instead of
just
> once
> > > per
> > > > > > file.
> > > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > now working on incorporating this functionality
into
> the
> > > > > GridDiag
> > > > > > > > > > > wrapper.
> > > > > > > > > > > If you don't mind, I was planning on assigning
the pull
> > > > request
> > > > > > > > > > > review to
> > > > > > > > > > > you so you can test these changes on your use
case to
> > > ensure
> > > > it
> > > > > > > > > > > runs
> > > > > > > > > > > the
> > > > > > > > > > > way you expect.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > > > Hi George and John,
> > > > > > > > > > > > Sounds like the modifications you are
discussing will
> > > > address
> > > > > > my
> > > > > > > > > > > > need
> > > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > > and forecast leads to determine which files to
use
> in a
> > > > > single
> > > > > > > > > > > > analysis).
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Jamie
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George McCabe
via RT
> > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > OK, it sounds like I misunderstood the
intended use
> > for
> > > > the
> > > > > > > > > > > > > tool
> > > > > > > > > > > > > when
> > > > > > > > > > > > > I
> > > > > > > > > > > > > implemented the wrapper. Thanks for pointing
this
> > out,
> > > > > Jamie!
> > > > > > > > > > > > >
> > > > > > > > > > > > > John, I think we should have a discussion
about how
> > we
> > > > > would
> > > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > have
> > > > > > > > > > > > > tools like SeriesAnalysis and GridDiag to be
> > configured
> > > > to
> > > > > > run
> > > > > > > > > > > > > in
> > > > > > > > > > > > > these
> > > > > > > > > > > > > cases. I'll follow up with you to set
something up.
> > > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > > currently separated into SeriesByInit and
> > SeriesByLead
> > > > > which
> > > > > > > > > > > > > perform
> > > > > > > > > > > > > the
> > > > > > > > > > > > > grouping. However, I am currently
refactoring these
> > > tools
> > > > > to
> > > > > > > > > > > > > hopefully
> > > > > > > > > > > > > combine the logic and allow options to
specify
> > > different
> > > > > ways
> > > > > > > > > > > > > to
> > > > > > > > > > > > > filter/group the data. This would be a good
time to
> > > come
> > > > up
> > > > > > > > > > > > > with
> > > > > > > > > > > > > a
> > > > > > > > > > > > > good way
> > > > > > > > > > > > > to define how you want to group your data in
a
> > general
> > > > way
> > > > > > that
> > > > > > > > > > > > > could
> > > > > > > > > > > > > apply
> > > > > > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jamie, to confirm what you are saying, you
expected
> > the
> > > > > list
> > > > > > of
> > > > > > > > > > > > > init
> > > > > > > > > > > > > times
> > > > > > > > > > > > > and forecast leads to determine which files
to use
> > in a
> > > > > > single
> > > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > > that correct?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John Halley
Gotway
> > via
> > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > George,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The logic for running the Grid-Diag is
more like
> > > > Series-
> > > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > Stat-Analysis than it is like Grid-Stat.
While a
> > user
> > > > > could
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > single time, it is more likely they'll
want to
> run
> > it
> > > > > over
> > > > > > a
> > > > > > > > > > > > > > set of
> > > > > > > > > > > > > times.
> > > > > > > > > > > > > > It really is pretty analogous to
> Series-Analysis...
> > > you
> > > > > may
> > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > the init time fixed and run over all lead
> times...
> > or
> > > > > hold
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > time
> > > > > > > > > > > > > > fixed and run over all init times... or
just run
> > over
> > > > all
> > > > > > > > > > > > > > init
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > George, is there any way to generalize the
> support
> > > for
> > > > > how
> > > > > > > > > > > > > > times
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > processed?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > John
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George
McCabe via
> > RT
> > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > > >
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The configuration you have set up loops
over
> each
> > > run
> > > > > > time
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > range
> > > > > > > > > > > > > > > you specified (2020060800 to 2020061500
with 24
> > > hour
> > > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > processing the list of forecast leads
for each
> > > time).
> > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > output
> > > > > > > > > > > > > > > template will be the same for each run:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > >
> {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is why you are seeing only the
results of
> > the
> > > > last
> > > > > > run
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > Each run overwrites the data from the
previous
> > run.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
StatAnalysis
> are
> > > > > > designed
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > > range of times at once, but GridDiag was
> written
> > to
> > > > > > process
> > > > > > > > > > > > > > > once
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > > time. However, this may not be the
intended way
> > to
> > > > use
> > > > > > the
> > > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > how I implemented it based on the
example I was
> > > > > provided.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > John HG (copied here) may have some
insight on
> > the
> > > > best
> > > > > > way
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > what you are trying to do. Perhaps you
would
> want
> > > to
> > > > > run
> > > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > run time, then run another tool to
aggregate
> the
> > > > stats
> > > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If that is not the case, then you should
be
> able
> > to
> > > > get
> > > > > > the
> > > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > the way you are intending by configuring
it to
> > > > process
> > > > > a
> > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the list
of
> > values
> > > > for
> > > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include the
times
> you
> > > > need
> > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > > keyword in the filename templates, i.e.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {init?fmt=...},
{init?fmt=...?shift=24H},
> > > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > on.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This approach is not very flexible and
would
> > > require
> > > > > you
> > > > > > to
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > a lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > changes to the METplus config file for
each
> range
> > > of
> > > > > > times
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > desire.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > this is the only way to achieve what you
need,
> > then
> > > > we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > to the wrapper to allow you to configure
this
> > more
> > > > > > easily.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe
wrote:
> > > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > > I am running grid-diag using the
script found
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It all appears to be running to
completion
> but
> > > the
> > > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > > not what I was expecting. To test the
tool, I
> > > first
> > > > > set
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > > run one day (INIT_BEG/END=2020061500)
and it
> > > > created
> > > > > > two
> > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > > (idx0/1) that I would have expected
that
> > included
> > > > > that
> > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > > I did the same for all my single days.
It
> > created
> > > > the
> > > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > expect. An example netcdf can be seen
here:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Then, I wanted to run something
similar for
> all
> > > my
> > > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > (INIT_BEG=2020060800,
INIT_END=2020061500,
> > > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > like to have all days aggregated
together and
> > one
> > > > > > netcdf
> > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > > in the end. It did create one netcdf
file at
> > the
> > > > end,
> > > > > > but
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > included the data for the last date
(the
> 15th).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc
and
> > > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I could be confused on the intent of
the
> tool.
> > > > > Perhaps
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > behavior you want it to have. If so, I
still
> > > think
> > > > > > there
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > because it should be outputting a file
per
> day,
> > > and
> > > > > it
> > > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > output a file at the end for the last
day.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What I thought would happen is I would
get
> one
> > > > file,
> > > > > > but
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > include all days/times requested. So,
what I
> > need
> > > > it
> > > > > to
> > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > create
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > list that includes all model (idx0)
and obs
> > > (idx1)
> > > > > > files
> > > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to use in one file. I can do this by
hand
> and I
> > > am
> > > > > > quite
> > > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > > would run and do what I want, but I
can't get
> > it
> > > to
> > > > > > work
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > automated fashion. (Essentially I want
it to
> do
> > > > > > something
> > > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > > series/stat-analysis, for example -
aggregate
> > > over
> > > > > > time)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It is very possible I am doing
something
> wrong
> > on
> > > > my
> > > > > > end.
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > that would be an easy fix. If this
doesn't
> make
> > > > sense
> > > > > > or
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > like to discuss this on the phone at
some
> > point,
> > > I
> > > > am
> > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > > Just let me know what you think.
Thanks in
> > > advance
> > > > > for
> > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > > my dir structure is confusing or you
need me
> to
> > > > send
> > > > > > you
> > > > > > > > > > > > > > > > anything
> > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > My working day may not be your working day.
Please
> do
> > > not
> > > > > > feel
> > > > > > > > > > > > > obliged to
> > > > > > > > > > > > > reply to this email outside of your normal
working
> > > hours.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jamie K. Wolff
> > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > Research Applications Laboratory (RAL)
> > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > >
> > > > > > > *My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jamie K. Wolff
> > > > > National Center for Atmospheric Research (NCAR)
> > > > > Research Applications Laboratory (RAL)
> > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > Jamie K. Wolff
> > > National Center for Atmospheric Research (NCAR)
> > > Research Applications Laboratory (RAL)
> > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 21 15:41:20 2020

Sorry, yes. In the case when I set RUN_ONCE it works perfectly fine.
However, when I change to run the other options that logic breaks
down.
Maybe it isn't a big deal right now (since I plan to use RUN_ONCE) but
in
the future I would want to know how to change the template so it could
work
for any of the four options.
Make sense?
Jamie

On Mon, Dec 21, 2020 at 3:37 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Glad it worked! I'm not sure I understand the issue with the output
> template. If Grid-Diag is run once for all of the files, isn't the
template
> you have there unique enough to that case (model, time ranges) that
it
> wouldn't be overwritten?
>
> On Mon, Dec 21, 2020 at 3:32 PM Jamie Wolff via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > That did the trick! I can now reproduce what I was doing by hand.
> >
> > I tested all four options and they all seemed to work great. This
is a
> > great enhancement - thank you!
> >
> > One remaining question is figuring out the best way to set the
template
> > file name so it is appropriately filled and not overwriting files.
Can
> you
> > suggest what I might change the below line to so it would work for
the
> new
> > options?
> >
> > GRID_DIAG_OUTPUT_TEMPLATE =
> >
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> >
> > Thank you!
> > Jamie
> >
> > On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Jamie,
> > >
> > > From my understanding that is likely the cause of the error. I
made
> some
> > > changes that should fix it for your case. Could you try running
your
> case
> > > using this master_metplus.py on cheyenne?
> > >
> > > /glade/u/home/mccabe/METplus/ush/master_metplus.py
> > >
> > > You could also run "git pull" on the feature_733_grid_diag_freq
branch
> if
> > > that is easier. Let me know how it goes.
> > >
> > > Thanks,
> > > George
> > >
> > > On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > I'm not entirely sure that is the cause of the error, but it
is new
> > > > behavior that may be having an impact.
> > > >
> > > > Here is how I used to run grid_diag (with the lists I created
by
> hand):
> > > > grid_diag -data
> > > > stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > -data
> > > > stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt
> > > -out
> > > > test_new.nc -config GridDiagConfig -v 4
> > > >
> > > > With the list created now I get the error I sent:
> > > > grid_diag -data
> > > >
stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> -out
> > > > test_new_ALL.nc -config GridDiagConfig -v 4
> > > >
> > > > ...
> > > > MetGrib2DataFile::read_grib2_record_list() - processed 4
records from
> > > GRIB2
> > > > file:
> > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of
> > > > type "FileType_Gb2".
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of
> > > > type "FileType_Gb2".
> > > > ERROR  :
> > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2 field
> > > abbreviation
> > > > 'tb_g16'
> > > >
> > > > Does that help explain?
> > > >
> > > > You should be able to run both of those commands on the
command
> line. I
> > > am
> > > > using version 9.1.1 of MET with this environment:
> > > > module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> > > > module load met/9.1.1
> > > > ncar_pylib
> > > >
> > > > Thanks!
> > > > Jamie
> > > >
> > > >
> > > > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Jamie,
> > > > >
> > > > > Yes, that error message should say LOOP_ORDER. Thanks for
catching
> > > that!
> > > > >
> > > > > Are you saying the error is occurring because all of the
files are
> > > lumped
> > > > > together instead of specified in separate -data arguments?
If so, I
> > can
> > > > > modify the code to separate out each item in the template
list.
> > > > >
> > > > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <
> > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > > >
> > > > > > Hi George,
> > > > > > I am trying to test your new mods in my processing.
> > > > > > I have a script to run grid_diag here:
> > > > > >
/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > >
> > > > > > I made the change you mentioned needed to be added
> > > > > (GRID_DIAG_RUNTIME_FREQ
> > > > > > = RUN_ONCE) to my METplus parm and you can see the parm
file
> > > > > >
> >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > > > >
> > > > > > However, when I run it I get the following error:
> > > > > > 12/21 13:20:13.449 metplus (command_builder.py:163) ERROR:
> > > > > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE
mode unles
> > > > > > s LOOP_BY = processes
> > > > > >
> > > > > > But that is confusing because I do not think LOOP_BY has
an
> option
> > of
> > > > > > processes. Should that be LOOP_ORDER? That is what I tried
and
> that
> > > > > seemed
> > > > > > to get me further.
> > > > > >
> > > > > > Before, there would be two file_lists created one for the
model
> and
> > > one
> > > > > for
> > > > > > the obs
> > > > > > (For example:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > > > But now it looks like they are both in one file:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > > >
> > > > > > I think this is causing a problem because I am getting an
error:
> > > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > > > DEBUG 1: Default Config File:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > > > DEBUG 1: User Config File:
> > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > > > ERROR  :
> > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > > abbreviation
> > > > > > 'tb_g16'
> > > > > > ERROR  :
> > > > > >
> > > > > > I am guessing this is a problem with my config file, not
> > necessarily
> > > > your
> > > > > > logic. Could you take a look at my config file to see if I
have
> > > > > additional
> > > > > > logic that needs to change for this to work? Let me know
if you
> > need
> > > > more
> > > > > > information from me.
> > > > > > Thanks!
> > > > > > Jamie
> > > > > >
> > > > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > Not a problem at all. You should just be able to obtain
the
> > branch
> > > > and
> > > > > > call
> > > > > > > that version of master_metplus.py instead of the one you
are
> > > > currently
> > > > > > > using. Let me know if you run into any issues and I can
help.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > > I  am sorry, I am swamped this week. I hope to get to
this
> next
> > > > week
> > > > > or
> > > > > > > the
> > > > > > > > week after.  In the meantime, are there any build
> instructions
> > > you
> > > > > can
> > > > > > > > share with me to use when  I pull this branch on
cheyenne?
> > > > > > > > I will be in touch as soon as I can.
> > > > > > > > Thanks,
> > > > > > > > Jamie
> > > > > > > >
> > > > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jamie,
> > > > > > > > >
> > > > > > > > > Just following up on this in case this email was
buried.
> Let
> > me
> > > > > know
> > > > > > if
> > > > > > > > > you need any help getting this to work!
> > > > > > > > >
> > > > > > > > > - George
> > > > > > > > >
> > > > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > > > Hi Jamie,
> > > > > > > > > >
> > > > > > > > > > I implemented the changes to GridDiag that were
necessary
> > to
> > > > run
> > > > > > the
> > > > > > > > > > tool as you expected. I assigned you to the pull
request:
> > > > > > > > > >
> > > > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > > > >
> > > > > > > > > > The pull request has notes regarding which branch
to
> > checkout
> > > > on
> > > > > > > > > > cheyenne and which setting to add to get it to
work. Let
> me
> > > > know
> > > > > if
> > > > > > > > > > you have any questions!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > > > Jamie
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe
via RT
> > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > >
> > > > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > > > >
> > > > > > > > > > > > https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > > > >
> > > > > > > > > > > > I have implemented functionality to call
certain
> > wrappers
> > > > in
> > > > > > > > > > > > different
> > > > > > > > > > > > intervals to process groups of files instead
of just
> > once
> > > > per
> > > > > > > file.
> > > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > now working on incorporating this
functionality into
> > the
> > > > > > GridDiag
> > > > > > > > > > > > wrapper.
> > > > > > > > > > > > If you don't mind, I was planning on assigning
the
> pull
> > > > > request
> > > > > > > > > > > > review to
> > > > > > > > > > > > you so you can test these changes on your use
case to
> > > > ensure
> > > > > it
> > > > > > > > > > > > runs
> > > > > > > > > > > > the
> > > > > > > > > > > > way you expect.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > > > > Hi George and John,
> > > > > > > > > > > > > Sounds like the modifications you are
discussing
> will
> > > > > address
> > > > > > > my
> > > > > > > > > > > > > need
> > > > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > > > and forecast leads to determine which files
to use
> > in a
> > > > > > single
> > > > > > > > > > > > > analysis).
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > Jamie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George
McCabe via
> RT
> > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > OK, it sounds like I misunderstood the
intended
> use
> > > for
> > > > > the
> > > > > > > > > > > > > > tool
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > implemented the wrapper. Thanks for
pointing this
> > > out,
> > > > > > Jamie!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > John, I think we should have a discussion
about
> how
> > > we
> > > > > > would
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > tools like SeriesAnalysis and GridDiag to
be
> > > configured
> > > > > to
> > > > > > > run
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > cases. I'll follow up with you to set
something
> up.
> > > > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > > > currently separated into SeriesByInit and
> > > SeriesByLead
> > > > > > which
> > > > > > > > > > > > > > perform
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > grouping. However, I am currently
refactoring
> these
> > > > tools
> > > > > > to
> > > > > > > > > > > > > > hopefully
> > > > > > > > > > > > > > combine the logic and allow options to
specify
> > > > different
> > > > > > ways
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > filter/group the data. This would be a
good time
> to
> > > > come
> > > > > up
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > good way
> > > > > > > > > > > > > > to define how you want to group your data
in a
> > > general
> > > > > way
> > > > > > > that
> > > > > > > > > > > > > > could
> > > > > > > > > > > > > > apply
> > > > > > > > > > > > > > to other similar wrappers, like GridDiag.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jamie, to confirm what you are saying, you
> expected
> > > the
> > > > > > list
> > > > > > > of
> > > > > > > > > > > > > > init
> > > > > > > > > > > > > > times
> > > > > > > > > > > > > > and forecast leads to determine which
files to
> use
> > > in a
> > > > > > > single
> > > > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > > > that correct?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > George
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John
Halley
> Gotway
> > > via
> > > > > RT <
> > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > George,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The logic for running the Grid-Diag is
more
> like
> > > > > Series-
> > > > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > Stat-Analysis than it is like Grid-Stat.
While
> a
> > > user
> > > > > > could
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > single time, it is more likely they'll
want to
> > run
> > > it
> > > > > > over
> > > > > > > a
> > > > > > > > > > > > > > > set of
> > > > > > > > > > > > > > times.
> > > > > > > > > > > > > > > It really is pretty analogous to
> > Series-Analysis...
> > > > you
> > > > > > may
> > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > the init time fixed and run over all
lead
> > times...
> > > or
> > > > > > hold
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > fixed and run over all init times... or
just
> run
> > > over
> > > > > all
> > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > George, is there any way to generalize
the
> > support
> > > > for
> > > > > > how
> > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > processed?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George
McCabe
> via
> > > RT
> > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > > > >
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The configuration you have set up
loops over
> > each
> > > > run
> > > > > > > time
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > range
> > > > > > > > > > > > > > > > you specified (2020060800 to
2020061500 with
> 24
> > > > hour
> > > > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > processing the list of forecast leads
for
> each
> > > > time).
> > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > > template will be the same for each
run:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > > >
> > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is why you are seeing only the
results
> of
> > > the
> > > > > last
> > > > > > > run
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > Each run overwrites the data from the
> previous
> > > run.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
StatAnalysis
> > are
> > > > > > > designed
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > > > range of times at once, but GridDiag
was
> > written
> > > to
> > > > > > > process
> > > > > > > > > > > > > > > > once
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > time. However, this may not be the
intended
> way
> > > to
> > > > > use
> > > > > > > the
> > > > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > how I implemented it based on the
example I
> was
> > > > > > provided.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > John HG (copied here) may have some
insight
> on
> > > the
> > > > > best
> > > > > > > way
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > > what you are trying to do. Perhaps you
would
> > want
> > > > to
> > > > > > run
> > > > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > run time, then run another tool to
aggregate
> > the
> > > > > stats
> > > > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If that is not the case, then you
should be
> > able
> > > to
> > > > > get
> > > > > > > the
> > > > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > the way you are intending by
configuring it
> to
> > > > > process
> > > > > > a
> > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the
list of
> > > values
> > > > > for
> > > > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include
the times
> > you
> > > > > need
> > > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > > > keyword in the filename templates,
i.e.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {init?fmt=...},
{init?fmt=...?shift=24H},
> > > > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > on.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This approach is not very flexible and
would
> > > > require
> > > > > > you
> > > > > > > to
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > a lot
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > changes to the METplus config file for
each
> > range
> > > > of
> > > > > > > times
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > desire.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > this is the only way to achieve what
you
> need,
> > > then
> > > > > we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > to the wrapper to allow you to
configure this
> > > more
> > > > > > > easily.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe
wrote:
> > > > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > > > I am running grid-diag using the
script
> found
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It all appears to be running to
completion
> > but
> > > > the
> > > > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > > > not what I was expecting. To test
the
> tool, I
> > > > first
> > > > > > set
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > > > run one day
(INIT_BEG/END=2020061500) and
> it
> > > > > created
> > > > > > > two
> > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > > > (idx0/1) that I would have expected
that
> > > included
> > > > > > that
> > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > > > I did the same for all my single
days. It
> > > created
> > > > > the
> > > > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > expect. An example netcdf can be
seen here:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Then, I wanted to run something
similar for
> > all
> > > > my
> > > > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > (INIT_BEG=2020060800,
INIT_END=2020061500,
> > > > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > like to have all days aggregated
together
> and
> > > one
> > > > > > > netcdf
> > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > > > in the end. It did create one netcdf
file
> at
> > > the
> > > > > end,
> > > > > > > but
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > included the data for the last date
(the
> > 15th).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-21.nc
and
> > > > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I could be confused on the intent of
the
> > tool.
> > > > > > Perhaps
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > behavior you want it to have. If so,
I
> still
> > > > think
> > > > > > > there
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > because it should be outputting a
file per
> > day,
> > > > and
> > > > > > it
> > > > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > output a file at the end for the
last day.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What I thought would happen is I
would get
> > one
> > > > > file,
> > > > > > > but
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > include all days/times requested.
So, what
> I
> > > need
> > > > > it
> > > > > > to
> > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > list that includes all model (idx0)
and obs
> > > > (idx1)
> > > > > > > files
> > > > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > to use in one file. I can do this by
hand
> > and I
> > > > am
> > > > > > > quite
> > > > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > > > would run and do what I want, but I
can't
> get
> > > it
> > > > to
> > > > > > > work
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > automated fashion. (Essentially I
want it
> to
> > do
> > > > > > > something
> > > > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > > > series/stat-analysis, for example -
> aggregate
> > > > over
> > > > > > > time)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It is very possible I am doing
something
> > wrong
> > > on
> > > > > my
> > > > > > > end.
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > that would be an easy fix. If this
doesn't
> > make
> > > > > sense
> > > > > > > or
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > like to discuss this on the phone at
some
> > > point,
> > > > I
> > > > > am
> > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > > > Just let me know what you think.
Thanks in
> > > > advance
> > > > > > for
> > > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > > > my dir structure is confusing or you
need
> me
> > to
> > > > > send
> > > > > > > you
> > > > > > > > > > > > > > > > > anything
> > > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > My working day may not be your working
day.
> Please
> > do
> > > > not
> > > > > > > feel
> > > > > > > > > > > > > > obliged to
> > > > > > > > > > > > > > reply to this email outside of your normal
> working
> > > > hours.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jamie K. Wolff
> > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > >
> > > > > > > > *My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jamie K. Wolff
> > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > Research Applications Laboratory (RAL)
> > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > >
> > > > > > *My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Jamie K. Wolff
> > > > National Center for Atmospheric Research (NCAR)
> > > > Research Applications Laboratory (RAL)
> > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > Jamie K. Wolff
> > National Center for Atmospheric Research (NCAR)
> > Research Applications Laboratory (RAL)
> > Phone: 303.497.2812  Email: jwolff at ucar.edu
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 21 15:45:34 2020

Ah, yes, I understand now. If you were to run using
RUN_ONCE_PER_INIT_OR_VALID, you would want to use a template that
references the init or valid time. Something like this would work:

GRID_DIAG_OUTPUT_TEMPLATE =
grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}.nc

If you were to run using RUN_ONCE_PER_LEAD, you would want to
reference the
lead tag instead.

GRID_DIAG_OUTPUT_TEMPLATE =
grid_diag_out_{MODEL}_{lead?fmt=%3H}.nc

If you were running with RUN_ONCE_FOR_EACH, you would want to
reference
lead and either init or valid, depending on how you are running. Since
you
are looping over init times, I would recommend something like:

GRID_DIAG_OUTPUT_TEMPLATE =
grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc

Does that make sense?

- George

On Mon, Dec 21, 2020 at 3:41 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> Sorry, yes. In the case when I set RUN_ONCE it works perfectly fine.
> However, when I change to run the other options that logic breaks
down.
> Maybe it isn't a big deal right now (since I plan to use RUN_ONCE)
but in
> the future I would want to know how to change the template so it
could work
> for any of the four options.
> Make sense?
> Jamie
>
> On Mon, Dec 21, 2020 at 3:37 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Glad it worked! I'm not sure I understand the issue with the
output
> > template. If Grid-Diag is run once for all of the files, isn't the
> template
> > you have there unique enough to that case (model, time ranges)
that it
> > wouldn't be overwritten?
> >
> > On Mon, Dec 21, 2020 at 3:32 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > That did the trick! I can now reproduce what I was doing by
hand.
> > >
> > > I tested all four options and they all seemed to work great.
This is a
> > > great enhancement - thank you!
> > >
> > > One remaining question is figuring out the best way to set the
template
> > > file name so it is appropriately filled and not overwriting
files. Can
> > you
> > > suggest what I might change the below line to so it would work
for the
> > new
> > > options?
> > >
> > > GRID_DIAG_OUTPUT_TEMPLATE =
> > >
> > >
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > >
> > > Thank you!
> > > Jamie
> > >
> > > On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Jamie,
> > > >
> > > > From my understanding that is likely the cause of the error. I
made
> > some
> > > > changes that should fix it for your case. Could you try
running your
> > case
> > > > using this master_metplus.py on cheyenne?
> > > >
> > > > /glade/u/home/mccabe/METplus/ush/master_metplus.py
> > > >
> > > > You could also run "git pull" on the
feature_733_grid_diag_freq
> branch
> > if
> > > > that is easier. Let me know how it goes.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > I'm not entirely sure that is the cause of the error, but it
is new
> > > > > behavior that may be having an impact.
> > > > >
> > > > > Here is how I used to run grid_diag (with the lists I
created by
> > hand):
> > > > > grid_diag -data
> > > > >
> stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt
> > > > -data
> > > > >
> stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt
> > > > -out
> > > > > test_new.nc -config GridDiagConfig -v 4
> > > > >
> > > > > With the list created now I get the error I sent:
> > > > > grid_diag -data
> > > > >
stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > -out
> > > > > test_new_ALL.nc -config GridDiagConfig -v 4
> > > > >
> > > > > ...
> > > > > MetGrib2DataFile::read_grib2_record_list() - processed 4
records
> from
> > > > GRIB2
> > > > > file:
> > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> object
> > > of
> > > > > type "FileType_Gb2".
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> object
> > > of
> > > > > type "FileType_Gb2".
> > > > > ERROR  :
> > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > abbreviation
> > > > > 'tb_g16'
> > > > >
> > > > > Does that help explain?
> > > > >
> > > > > You should be able to run both of those commands on the
command
> > line. I
> > > > am
> > > > > using version 9.1.1 of MET with this environment:
> > > > > module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> > > > > module load met/9.1.1
> > > > > ncar_pylib
> > > > >
> > > > > Thanks!
> > > > > Jamie
> > > > >
> > > > >
> > > > > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > Yes, that error message should say LOOP_ORDER. Thanks for
> catching
> > > > that!
> > > > > >
> > > > > > Are you saying the error is occurring because all of the
files
> are
> > > > lumped
> > > > > > together instead of specified in separate -data arguments?
If
> so, I
> > > can
> > > > > > modify the code to separate out each item in the template
list.
> > > > > >
> > > > > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <
> > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> >
> > > > > > >
> > > > > > > Hi George,
> > > > > > > I am trying to test your new mods in my processing.
> > > > > > > I have a script to run grid_diag here:
> > > > > > >
/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > >
> > > > > > > I made the change you mentioned needed to be added
> > > > > > (GRID_DIAG_RUNTIME_FREQ
> > > > > > > = RUN_ONCE) to my METplus parm and you can see the parm
file
> > > > > > >
> > >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > > > > >
> > > > > > > However, when I run it I get the following error:
> > > > > > > 12/21 13:20:13.449 metplus (command_builder.py:163)
ERROR:
> > > > > > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE
mode
> unles
> > > > > > > s LOOP_BY = processes
> > > > > > >
> > > > > > > But that is confusing because I do not think LOOP_BY has
an
> > option
> > > of
> > > > > > > processes. Should that be LOOP_ORDER? That is what I
tried and
> > that
> > > > > > seemed
> > > > > > > to get me further.
> > > > > > >
> > > > > > > Before, there would be two file_lists created one for
the model
> > and
> > > > one
> > > > > > for
> > > > > > > the obs
> > > > > > > (For example:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > > > > But now it looks like they are both in one file:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > > > >
> > > > > > > I think this is causing a problem because I am getting
an
> error:
> > > > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES
file:
> > > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > > > > DEBUG 1: Default Config File:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > > > > DEBUG 1: User Config File:
> > > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > > > > ERROR  :
> > > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > > > abbreviation
> > > > > > > 'tb_g16'
> > > > > > > ERROR  :
> > > > > > >
> > > > > > > I am guessing this is a problem with my config file, not
> > > necessarily
> > > > > your
> > > > > > > logic. Could you take a look at my config file to see if
I have
> > > > > > additional
> > > > > > > logic that needs to change for this to work? Let me know
if you
> > > need
> > > > > more
> > > > > > > information from me.
> > > > > > > Thanks!
> > > > > > > Jamie
> > > > > > >
> > > > > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > Not a problem at all. You should just be able to
obtain the
> > > branch
> > > > > and
> > > > > > > call
> > > > > > > > that version of master_metplus.py instead of the one
you are
> > > > > currently
> > > > > > > > using. Let me know if you run into any issues and I
can help.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > >
> > > > > > > > >
> > > > > > > > > Hi George,
> > > > > > > > > I  am sorry, I am swamped this week. I hope to get
to this
> > next
> > > > > week
> > > > > > or
> > > > > > > > the
> > > > > > > > > week after.  In the meantime, are there any build
> > instructions
> > > > you
> > > > > > can
> > > > > > > > > share with me to use when  I pull this branch on
cheyenne?
> > > > > > > > > I will be in touch as soon as I can.
> > > > > > > > > Thanks,
> > > > > > > > > Jamie
> > > > > > > > >
> > > > > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jamie,
> > > > > > > > > >
> > > > > > > > > > Just following up on this in case this email was
buried.
> > Let
> > > me
> > > > > > know
> > > > > > > if
> > > > > > > > > > you need any help getting this to work!
> > > > > > > > > >
> > > > > > > > > > - George
> > > > > > > > > >
> > > > > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > > > > Hi Jamie,
> > > > > > > > > > >
> > > > > > > > > > > I implemented the changes to GridDiag that were
> necessary
> > > to
> > > > > run
> > > > > > > the
> > > > > > > > > > > tool as you expected. I assigned you to the pull
> request:
> > > > > > > > > > >
> > > > > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > > > > >
> > > > > > > > > > > The pull request has notes regarding which
branch to
> > > checkout
> > > > > on
> > > > > > > > > > > cheyenne and which setting to add to get it to
work.
> Let
> > me
> > > > > know
> > > > > > if
> > > > > > > > > > > you have any questions!
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > > > > Jamie
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George McCabe
via RT
> > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have implemented functionality to call
certain
> > > wrappers
> > > > > in
> > > > > > > > > > > > > different
> > > > > > > > > > > > > intervals to process groups of files instead
of
> just
> > > once
> > > > > per
> > > > > > > > file.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > now working on incorporating this
functionality
> into
> > > the
> > > > > > > GridDiag
> > > > > > > > > > > > > wrapper.
> > > > > > > > > > > > > If you don't mind, I was planning on
assigning the
> > pull
> > > > > > request
> > > > > > > > > > > > > review to
> > > > > > > > > > > > > you so you can test these changes on your
use case
> to
> > > > > ensure
> > > > > > it
> > > > > > > > > > > > > runs
> > > > > > > > > > > > > the
> > > > > > > > > > > > > way you expect.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > > > > > Hi George and John,
> > > > > > > > > > > > > > Sounds like the modifications you are
discussing
> > will
> > > > > > address
> > > > > > > > my
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > > > > and forecast leads to determine which
files to
> use
> > > in a
> > > > > > > single
> > > > > > > > > > > > > > analysis).
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George
McCabe via
> > RT
> > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > OK, it sounds like I misunderstood the
intended
> > use
> > > > for
> > > > > > the
> > > > > > > > > > > > > > > tool
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > implemented the wrapper. Thanks for
pointing
> this
> > > > out,
> > > > > > > Jamie!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > John, I think we should have a
discussion about
> > how
> > > > we
> > > > > > > would
> > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > tools like SeriesAnalysis and GridDiag
to be
> > > > configured
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > cases. I'll follow up with you to set
something
> > up.
> > > > > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > > > > currently separated into SeriesByInit
and
> > > > SeriesByLead
> > > > > > > which
> > > > > > > > > > > > > > > perform
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > grouping. However, I am currently
refactoring
> > these
> > > > > tools
> > > > > > > to
> > > > > > > > > > > > > > > hopefully
> > > > > > > > > > > > > > > combine the logic and allow options to
specify
> > > > > different
> > > > > > > ways
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > filter/group the data. This would be a
good
> time
> > to
> > > > > come
> > > > > > up
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > good way
> > > > > > > > > > > > > > > to define how you want to group your
data in a
> > > > general
> > > > > > way
> > > > > > > > that
> > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > apply
> > > > > > > > > > > > > > > to other similar wrappers, like
GridDiag.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jamie, to confirm what you are saying,
you
> > expected
> > > > the
> > > > > > > list
> > > > > > > > of
> > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > and forecast leads to determine which
files to
> > use
> > > > in a
> > > > > > > > single
> > > > > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > > > > that correct?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John
Halley
> > Gotway
> > > > via
> > > > > > RT <
> > > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > George,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The logic for running the Grid-Diag is
more
> > like
> > > > > > Series-
> > > > > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > Stat-Analysis than it is like Grid-
Stat.
> While
> > a
> > > > user
> > > > > > > could
> > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > single time, it is more likely they'll
want
> to
> > > run
> > > > it
> > > > > > > over
> > > > > > > > a
> > > > > > > > > > > > > > > > set of
> > > > > > > > > > > > > > > times.
> > > > > > > > > > > > > > > > It really is pretty analogous to
> > > Series-Analysis...
> > > > > you
> > > > > > > may
> > > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > the init time fixed and run over all
lead
> > > times...
> > > > or
> > > > > > > hold
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > fixed and run over all init times...
or just
> > run
> > > > over
> > > > > > all
> > > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > George, is there any way to generalize
the
> > > support
> > > > > for
> > > > > > > how
> > > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > processed?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM George
McCabe
> > via
> > > > RT
> > > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > > > > >
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I moved this email over to MET Help.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The configuration you have set up
loops
> over
> > > each
> > > > > run
> > > > > > > > time
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > range
> > > > > > > > > > > > > > > > > you specified (2020060800 to
2020061500
> with
> > 24
> > > > > hour
> > > > > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > processing the list of forecast
leads for
> > each
> > > > > time).
> > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > > > template will be the same for each
run:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > > > >
> > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is why you are seeing only the
results
> > of
> > > > the
> > > > > > last
> > > > > > > > run
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > > Each run overwrites the data from
the
> > previous
> > > > run.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
> StatAnalysis
> > > are
> > > > > > > > designed
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > > > > range of times at once, but GridDiag
was
> > > written
> > > > to
> > > > > > > > process
> > > > > > > > > > > > > > > > > once
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > time. However, this may not be the
intended
> > way
> > > > to
> > > > > > use
> > > > > > > > the
> > > > > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > how I implemented it based on the
example I
> > was
> > > > > > > provided.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > John HG (copied here) may have some
insight
> > on
> > > > the
> > > > > > best
> > > > > > > > way
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > > > what you are trying to do. Perhaps
you
> would
> > > want
> > > > > to
> > > > > > > run
> > > > > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > run time, then run another tool to
> aggregate
> > > the
> > > > > > stats
> > > > > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If that is not the case, then you
should be
> > > able
> > > > to
> > > > > > get
> > > > > > > > the
> > > > > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > the way you are intending by
configuring it
> > to
> > > > > > process
> > > > > > > a
> > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand the
list of
> > > > values
> > > > > > for
> > > > > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to include
the
> times
> > > you
> > > > > > need
> > > > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > > > > keyword in the filename templates,
i.e.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > {init?fmt=...},
{init?fmt=...?shift=24H},
> > > > > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > on.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This approach is not very flexible
and
> would
> > > > > require
> > > > > > > you
> > > > > > > > to
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > a lot
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > changes to the METplus config file
for each
> > > range
> > > > > of
> > > > > > > > times
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > desire.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > this is the only way to achieve what
you
> > need,
> > > > then
> > > > > > we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > to the wrapper to allow you to
configure
> this
> > > > more
> > > > > > > > easily.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020, mccabe
wrote:
> > > > > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > > > > I am running grid-diag using the
script
> > found
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It all appears to be running to
> completion
> > > but
> > > > > the
> > > > > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > > > > not what I was expecting. To test
the
> > tool, I
> > > > > first
> > > > > > > set
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > > > > run one day
(INIT_BEG/END=2020061500) and
> > it
> > > > > > created
> > > > > > > > two
> > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > > > > (idx0/1) that I would have
expected that
> > > > included
> > > > > > > that
> > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > > > > I did the same for all my single
days. It
> > > > created
> > > > > > the
> > > > > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > expect. An example netcdf can be
seen
> here:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Then, I wanted to run something
similar
> for
> > > all
> > > > > my
> > > > > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > (INIT_BEG=2020060800,
> INIT_END=2020061500,
> > > > > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > like to have all days aggregated
together
> > and
> > > > one
> > > > > > > > netcdf
> > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > > > > in the end. It did create one
netcdf file
> > at
> > > > the
> > > > > > end,
> > > > > > > > but
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > included the data for the last
date (the
> > > 15th).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-
21.nc and
> > > > > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I could be confused on the intent
of the
> > > tool.
> > > > > > > Perhaps
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > behavior you want it to have. If
so, I
> > still
> > > > > think
> > > > > > > > there
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > > because it should be outputting a
file
> per
> > > day,
> > > > > and
> > > > > > > it
> > > > > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > output a file at the end for the
last
> day.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > What I thought would happen is I
would
> get
> > > one
> > > > > > file,
> > > > > > > > but
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > include all days/times requested.
So,
> what
> > I
> > > > need
> > > > > > it
> > > > > > > to
> > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > list that includes all model
(idx0) and
> obs
> > > > > (idx1)
> > > > > > > > files
> > > > > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > to use in one file. I can do this
by hand
> > > and I
> > > > > am
> > > > > > > > quite
> > > > > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > > > > would run and do what I want, but
I can't
> > get
> > > > it
> > > > > to
> > > > > > > > work
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > automated fashion. (Essentially I
want it
> > to
> > > do
> > > > > > > > something
> > > > > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > > > > series/stat-analysis, for example
-
> > aggregate
> > > > > over
> > > > > > > > time)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It is very possible I am doing
something
> > > wrong
> > > > on
> > > > > > my
> > > > > > > > end.
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > that would be an easy fix. If this
> doesn't
> > > make
> > > > > > sense
> > > > > > > > or
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > like to discuss this on the phone
at some
> > > > point,
> > > > > I
> > > > > > am
> > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > > > > Just let me know what you think.
Thanks
> in
> > > > > advance
> > > > > > > for
> > > > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > > > > my dir structure is confusing or
you need
> > me
> > > to
> > > > > > send
> > > > > > > > you
> > > > > > > > > > > > > > > > > > anything
> > > > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > My working day may not be your working
day.
> > Please
> > > do
> > > > > not
> > > > > > > > feel
> > > > > > > > > > > > > > > obliged to
> > > > > > > > > > > > > > > reply to this email outside of your
normal
> > working
> > > > > hours.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jamie K. Wolff
> > > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > > >
> > > > > > > > > *My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jamie K. Wolff
> > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > Research Applications Laboratory (RAL)
> > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > >
> > > > > > > *My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jamie K. Wolff
> > > > > National Center for Atmospheric Research (NCAR)
> > > > > Research Applications Laboratory (RAL)
> > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > Jamie K. Wolff
> > > National Center for Atmospheric Research (NCAR)
> > > Research Applications Laboratory (RAL)
> > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: grid-diag functionality
From: Jamie Wolff
Time: Mon Dec 21 15:55:04 2020

Perfect sense.

I would recommend this PR be approved.

Thanks!
Jamie

On Mon, Dec 21, 2020 at 3:45 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Ah, yes, I understand now. If you were to run using
> RUN_ONCE_PER_INIT_OR_VALID, you would want to use a template that
> references the init or valid time. Something like this would work:
>
> GRID_DIAG_OUTPUT_TEMPLATE =
> grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}.nc
>
> If you were to run using RUN_ONCE_PER_LEAD, you would want to
reference the
> lead tag instead.
>
>
> GRID_DIAG_OUTPUT_TEMPLATE =
> grid_diag_out_{MODEL}_{lead?fmt=%3H}.nc
>
> If you were running with RUN_ONCE_FOR_EACH, you would want to
reference
> lead and either init or valid, depending on how you are running.
Since you
> are looping over init times, I would recommend something like:
>
>
> GRID_DIAG_OUTPUT_TEMPLATE =
> grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc
>
> Does that make sense?
>
> - George
>
> On Mon, Dec 21, 2020 at 3:41 PM Jamie Wolff via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> >
> > Sorry, yes. In the case when I set RUN_ONCE it works perfectly
fine.
> > However, when I change to run the other options that logic breaks
down.
> > Maybe it isn't a big deal right now (since I plan to use RUN_ONCE)
but in
> > the future I would want to know how to change the template so it
could
> work
> > for any of the four options.
> > Make sense?
> > Jamie
> >
> > On Mon, Dec 21, 2020 at 3:37 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Glad it worked! I'm not sure I understand the issue with the
output
> > > template. If Grid-Diag is run once for all of the files, isn't
the
> > template
> > > you have there unique enough to that case (model, time ranges)
that it
> > > wouldn't be overwritten?
> > >
> > > On Mon, Dec 21, 2020 at 3:32 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
>
> > > >
> > > > That did the trick! I can now reproduce what I was doing by
hand.
> > > >
> > > > I tested all four options and they all seemed to work great.
This is
> a
> > > > great enhancement - thank you!
> > > >
> > > > One remaining question is figuring out the best way to set the
> template
> > > > file name so it is appropriately filled and not overwriting
files.
> Can
> > > you
> > > > suggest what I might change the below line to so it would work
for
> the
> > > new
> > > > options?
> > > >
> > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > >
> > > >
> > >
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > >
> > > > Thank you!
> > > > Jamie
> > > >
> > > > On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Jamie,
> > > > >
> > > > > From my understanding that is likely the cause of the error.
I made
> > > some
> > > > > changes that should fix it for your case. Could you try
running
> your
> > > case
> > > > > using this master_metplus.py on cheyenne?
> > > > >
> > > > > /glade/u/home/mccabe/METplus/ush/master_metplus.py
> > > > >
> > > > > You could also run "git pull" on the
feature_733_grid_diag_freq
> > branch
> > > if
> > > > > that is easier. Let me know how it goes.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT <
> > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > > >
> > > > > > I'm not entirely sure that is the cause of the error, but
it is
> new
> > > > > > behavior that may be having an impact.
> > > > > >
> > > > > > Here is how I used to run grid_diag (with the lists I
created by
> > > hand):
> > > > > > grid_diag -data
> > > > > >
> > stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt
> > > > > -data
> > > > > >
> > stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt
> > > > > -out
> > > > > > test_new.nc -config GridDiagConfig -v 4
> > > > > >
> > > > > > With the list created now I get the error I sent:
> > > > > > grid_diag -data
> > > > > >
stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > -out
> > > > > > test_new_ALL.nc -config GridDiagConfig -v 4
> > > > > >
> > > > > > ...
> > > > > > MetGrib2DataFile::read_grib2_record_list() - processed 4
records
> > from
> > > > > GRIB2
> > > > > > file:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > object
> > > > of
> > > > > > type "FileType_Gb2".
> > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > object
> > > > of
> > > > > > type "FileType_Gb2".
> > > > > > ERROR  :
> > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > > abbreviation
> > > > > > 'tb_g16'
> > > > > >
> > > > > > Does that help explain?
> > > > > >
> > > > > > You should be able to run both of those commands on the
command
> > > line. I
> > > > > am
> > > > > > using version 9.1.1 of MET with this environment:
> > > > > > module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
> > > > > > module load met/9.1.1
> > > > > > ncar_pylib
> > > > > >
> > > > > > Thanks!
> > > > > > Jamie
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jamie,
> > > > > > >
> > > > > > > Yes, that error message should say LOOP_ORDER. Thanks
for
> > catching
> > > > > that!
> > > > > > >
> > > > > > > Are you saying the error is occurring because all of the
files
> > are
> > > > > lumped
> > > > > > > together instead of specified in separate -data
arguments? If
> > so, I
> > > > can
> > > > > > > modify the code to separate out each item in the
template list.
> > > > > > >
> > > > > > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <
> > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > > I am trying to test your new mods in my processing.
> > > > > > > > I have a script to run grid_diag here:
> > > > > > > >
> /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > >
> > > > > > > > I made the change you mentioned needed to be added
> > > > > > > (GRID_DIAG_RUNTIME_FREQ
> > > > > > > > = RUN_ONCE) to my METplus parm and you can see the
parm file
> > > > > > > >
> > > >
here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > > > > > >
> > > > > > > > However, when I run it I get the following error:
> > > > > > > > 12/21 13:20:13.449 metplus (command_builder.py:163)
ERROR:
> > > > > > > > (runtime_freq_wrapper.py:85) Cannot run using RUN_ONCE
mode
> > unles
> > > > > > > > s LOOP_BY = processes
> > > > > > > >
> > > > > > > > But that is confusing because I do not think LOOP_BY
has an
> > > option
> > > > of
> > > > > > > > processes. Should that be LOOP_ORDER? That is what I
tried
> and
> > > that
> > > > > > > seemed
> > > > > > > > to get me further.
> > > > > > > >
> > > > > > > > Before, there would be two file_lists created one for
the
> model
> > > and
> > > > > one
> > > > > > > for
> > > > > > > > the obs
> > > > > > > > (For example:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > > > > > But now it looks like they are both in one file:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > > > > >
> > > > > > > > I think this is causing a problem because I am getting
an
> > error:
> > > > > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES
file:
> > > > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > > > > > DEBUG 1: Default Config File:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > > > > > DEBUG 1: User Config File:
> > > > > > > >
/glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > > > > > ERROR  :
> > > > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized
GRIB2 field
> > > > > > > abbreviation
> > > > > > > > 'tb_g16'
> > > > > > > > ERROR  :
> > > > > > > >
> > > > > > > > I am guessing this is a problem with my config file,
not
> > > > necessarily
> > > > > > your
> > > > > > > > logic. Could you take a look at my config file to see
if I
> have
> > > > > > > additional
> > > > > > > > logic that needs to change for this to work? Let me
know if
> you
> > > > need
> > > > > > more
> > > > > > > > information from me.
> > > > > > > > Thanks!
> > > > > > > > Jamie
> > > > > > > >
> > > > > > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jamie,
> > > > > > > > >
> > > > > > > > > Not a problem at all. You should just be able to
obtain the
> > > > branch
> > > > > > and
> > > > > > > > call
> > > > > > > > > that version of master_metplus.py instead of the one
you
> are
> > > > > > currently
> > > > > > > > > using. Let me know if you run into any issues and I
can
> help.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi George,
> > > > > > > > > > I  am sorry, I am swamped this week. I hope to get
to
> this
> > > next
> > > > > > week
> > > > > > > or
> > > > > > > > > the
> > > > > > > > > > week after.  In the meantime, are there any build
> > > instructions
> > > > > you
> > > > > > > can
> > > > > > > > > > share with me to use when  I pull this branch on
> cheyenne?
> > > > > > > > > > I will be in touch as soon as I can.
> > > > > > > > > > Thanks,
> > > > > > > > > > Jamie
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jamie,
> > > > > > > > > > >
> > > > > > > > > > > Just following up on this in case this email was
> buried.
> > > Let
> > > > me
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > > you need any help getting this to work!
> > > > > > > > > > >
> > > > > > > > > > > - George
> > > > > > > > > > >
> > > > > > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > >
> > > > > > > > > > > > I implemented the changes to GridDiag that
were
> > necessary
> > > > to
> > > > > > run
> > > > > > > > the
> > > > > > > > > > > > tool as you expected. I assigned you to the
pull
> > request:
> > > > > > > > > > > >
> > > > > > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > > > > > >
> > > > > > > > > > > > The pull request has notes regarding which
branch to
> > > > checkout
> > > > > > on
> > > > > > > > > > > > cheyenne and which setting to add to get it to
work.
> > Let
> > > me
> > > > > > know
> > > > > > > if
> > > > > > > > > > > > you have any questions!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > > > > > Jamie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George
McCabe via
> RT
> > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have implemented functionality to call
certain
> > > > wrappers
> > > > > > in
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > intervals to process groups of files
instead of
> > just
> > > > once
> > > > > > per
> > > > > > > > > file.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > now working on incorporating this
functionality
> > into
> > > > the
> > > > > > > > GridDiag
> > > > > > > > > > > > > > wrapper.
> > > > > > > > > > > > > > If you don't mind, I was planning on
assigning
> the
> > > pull
> > > > > > > request
> > > > > > > > > > > > > > review to
> > > > > > > > > > > > > > you so you can test these changes on your
use
> case
> > to
> > > > > > ensure
> > > > > > > it
> > > > > > > > > > > > > > runs
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > way you expect.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > George
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff wrote:
> > > > > > > > > > > > > > > Hi George and John,
> > > > > > > > > > > > > > > Sounds like the modifications you are
> discussing
> > > will
> > > > > > > address
> > > > > > > > > my
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > > > > > and forecast leads to determine which
files to
> > use
> > > > in a
> > > > > > > > single
> > > > > > > > > > > > > > > analysis).
> > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George
McCabe
> via
> > > RT
> > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > OK, it sounds like I misunderstood the
> intended
> > > use
> > > > > for
> > > > > > > the
> > > > > > > > > > > > > > > > tool
> > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > implemented the wrapper. Thanks for
pointing
> > this
> > > > > out,
> > > > > > > > Jamie!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > John, I think we should have a
discussion
> about
> > > how
> > > > > we
> > > > > > > > would
> > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > tools like SeriesAnalysis and GridDiag
to be
> > > > > configured
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > cases. I'll follow up with you to set
> something
> > > up.
> > > > > > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > > > > > currently separated into SeriesByInit
and
> > > > > SeriesByLead
> > > > > > > > which
> > > > > > > > > > > > > > > > perform
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > grouping. However, I am currently
refactoring
> > > these
> > > > > > tools
> > > > > > > > to
> > > > > > > > > > > > > > > > hopefully
> > > > > > > > > > > > > > > > combine the logic and allow options to
> specify
> > > > > > different
> > > > > > > > ways
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > filter/group the data. This would be a
good
> > time
> > > to
> > > > > > come
> > > > > > > up
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > good way
> > > > > > > > > > > > > > > > to define how you want to group your
data in
> a
> > > > > general
> > > > > > > way
> > > > > > > > > that
> > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > apply
> > > > > > > > > > > > > > > > to other similar wrappers, like
GridDiag.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jamie, to confirm what you are saying,
you
> > > expected
> > > > > the
> > > > > > > > list
> > > > > > > > > of
> > > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > > and forecast leads to determine which
files
> to
> > > use
> > > > > in a
> > > > > > > > > single
> > > > > > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > > > > > that correct?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John
Halley
> > > Gotway
> > > > > via
> > > > > > > RT <
> > > > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > George,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The logic for running the Grid-Diag
is more
> > > like
> > > > > > > Series-
> > > > > > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > Stat-Analysis than it is like Grid-
Stat.
> > While
> > > a
> > > > > user
> > > > > > > > could
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > single time, it is more likely
they'll want
> > to
> > > > run
> > > > > it
> > > > > > > > over
> > > > > > > > > a
> > > > > > > > > > > > > > > > > set of
> > > > > > > > > > > > > > > > times.
> > > > > > > > > > > > > > > > > It really is pretty analogous to
> > > > Series-Analysis...
> > > > > > you
> > > > > > > > may
> > > > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > > the init time fixed and run over all
lead
> > > > times...
> > > > > or
> > > > > > > > hold
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > fixed and run over all init times...
or
> just
> > > run
> > > > > over
> > > > > > > all
> > > > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > George, is there any way to
generalize the
> > > > support
> > > > > > for
> > > > > > > > how
> > > > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > processed?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM
George
> McCabe
> > > via
> > > > > RT
> > > > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > > > > > >
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I moved this email over to MET
Help.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The configuration you have set up
loops
> > over
> > > > each
> > > > > > run
> > > > > > > > > time
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > range
> > > > > > > > > > > > > > > > > > you specified (2020060800 to
2020061500
> > with
> > > 24
> > > > > > hour
> > > > > > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > processing the list of forecast
leads for
> > > each
> > > > > > time).
> > > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > > > > template will be the same for each
run:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > > > > >
> > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is why you are seeing only
the
> results
> > > of
> > > > > the
> > > > > > > last
> > > > > > > > > run
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > > > Each run overwrites the data from
the
> > > previous
> > > > > run.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
> > StatAnalysis
> > > > are
> > > > > > > > > designed
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > > > > > range of times at once, but
GridDiag was
> > > > written
> > > > > to
> > > > > > > > > process
> > > > > > > > > > > > > > > > > > once
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > time. However, this may not be the
> intended
> > > way
> > > > > to
> > > > > > > use
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > how I implemented it based on the
> example I
> > > was
> > > > > > > > provided.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > John HG (copied here) may have
some
> insight
> > > on
> > > > > the
> > > > > > > best
> > > > > > > > > way
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > > > > what you are trying to do. Perhaps
you
> > would
> > > > want
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > run time, then run another tool to
> > aggregate
> > > > the
> > > > > > > stats
> > > > > > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If that is not the case, then you
should
> be
> > > > able
> > > > > to
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > the way you are intending by
configuring
> it
> > > to
> > > > > > > process
> > > > > > > > a
> > > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand
the list
> of
> > > > > values
> > > > > > > for
> > > > > > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to
include the
> > times
> > > > you
> > > > > > > need
> > > > > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > > > > > keyword in the filename templates,
i.e.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > {init?fmt=...},
{init?fmt=...?shift=24H},
> > > > > > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > on.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This approach is not very flexible
and
> > would
> > > > > > require
> > > > > > > > you
> > > > > > > > > to
> > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > a lot
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > changes to the METplus config file
for
> each
> > > > range
> > > > > > of
> > > > > > > > > times
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > desire.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > this is the only way to achieve
what you
> > > need,
> > > > > then
> > > > > > > we
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > to the wrapper to allow you to
configure
> > this
> > > > > more
> > > > > > > > > easily.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020,
mccabe
> wrote:
> > > > > > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > > > > > I am running grid-diag using the
script
> > > found
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It all appears to be running to
> > completion
> > > > but
> > > > > > the
> > > > > > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > > > > > not what I was expecting. To
test the
> > > tool, I
> > > > > > first
> > > > > > > > set
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > > > > > run one day
(INIT_BEG/END=2020061500)
> and
> > > it
> > > > > > > created
> > > > > > > > > two
> > > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > > > > > (idx0/1) that I would have
expected
> that
> > > > > included
> > > > > > > > that
> > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > > > > > I did the same for all my single
days.
> It
> > > > > created
> > > > > > > the
> > > > > > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > expect. An example netcdf can be
seen
> > here:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Then, I wanted to run something
similar
> > for
> > > > all
> > > > > > my
> > > > > > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > (INIT_BEG=2020060800,
> > INIT_END=2020061500,
> > > > > > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > like to have all days aggregated
> together
> > > and
> > > > > one
> > > > > > > > > netcdf
> > > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > > > > > in the end. It did create one
netcdf
> file
> > > at
> > > > > the
> > > > > > > end,
> > > > > > > > > but
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > included the data for the last
date
> (the
> > > > 15th).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > > > > > grid_diag_out_GFS_20200615_f00-
21.nc
> and
> > > > > > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I could be confused on the
intent of
> the
> > > > tool.
> > > > > > > > Perhaps
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > behavior you want it to have. If
so, I
> > > still
> > > > > > think
> > > > > > > > > there
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > > > because it should be outputting
a file
> > per
> > > > day,
> > > > > > and
> > > > > > > > it
> > > > > > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > output a file at the end for the
last
> > day.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > What I thought would happen is I
would
> > get
> > > > one
> > > > > > > file,
> > > > > > > > > but
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > include all days/times
requested. So,
> > what
> > > I
> > > > > need
> > > > > > > it
> > > > > > > > to
> > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > list that includes all model
(idx0) and
> > obs
> > > > > > (idx1)
> > > > > > > > > files
> > > > > > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > > to use in one file. I can do
this by
> hand
> > > > and I
> > > > > > am
> > > > > > > > > quite
> > > > > > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > > > > > would run and do what I want,
but I
> can't
> > > get
> > > > > it
> > > > > > to
> > > > > > > > > work
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > automated fashion. (Essentially
I want
> it
> > > to
> > > > do
> > > > > > > > > something
> > > > > > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > > > > > series/stat-analysis, for
example -
> > > aggregate
> > > > > > over
> > > > > > > > > time)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It is very possible I am doing
> something
> > > > wrong
> > > > > on
> > > > > > > my
> > > > > > > > > end.
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > that would be an easy fix. If
this
> > doesn't
> > > > make
> > > > > > > sense
> > > > > > > > > or
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > like to discuss this on the
phone at
> some
> > > > > point,
> > > > > > I
> > > > > > > am
> > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > > > > > Just let me know what you think.
Thanks
> > in
> > > > > > advance
> > > > > > > > for
> > > > > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > > > > > my dir structure is confusing or
you
> need
> > > me
> > > > to
> > > > > > > send
> > > > > > > > > you
> > > > > > > > > > > > > > > > > > > anything
> > > > > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > > > > > National Center for Atmospheric
Research
> > > > > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > My working day may not be your working
day.
> > > Please
> > > > do
> > > > > > not
> > > > > > > > > feel
> > > > > > > > > > > > > > > > obliged to
> > > > > > > > > > > > > > > > reply to this email outside of your
normal
> > > working
> > > > > > hours.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Jamie K. Wolff
> > > > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > > > >
> > > > > > > > > > *My working day may not be your working day.
Please do
> not
> > > feel
> > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
> hours.*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jamie K. Wolff
> > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > >
> > > > > > > > *My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jamie K. Wolff
> > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > Research Applications Laboratory (RAL)
> > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > >
> > > > > > *My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Jamie K. Wolff
> > > > National Center for Atmospheric Research (NCAR)
> > > > Research Applications Laboratory (RAL)
> > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > Jamie K. Wolff
> > National Center for Atmospheric Research (NCAR)
> > Research Applications Laboratory (RAL)
> > Phone: 303.497.2812  Email: jwolff at ucar.edu
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812  Email: jwolff at ucar.edu

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: grid-diag functionality
From: George McCabe
Time: Mon Dec 21 15:57:59 2020

Awesome, thank you so much for reviewing it and making sure it worked
well
for your use case!

- George

On Mon, Dec 21, 2020 at 3:55 PM Jamie Wolff via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
>
> Perfect sense.
>
> I would recommend this PR be approved.
>
> Thanks!
> Jamie
>
> On Mon, Dec 21, 2020 at 3:45 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Ah, yes, I understand now. If you were to run using
> > RUN_ONCE_PER_INIT_OR_VALID, you would want to use a template that
> > references the init or valid time. Something like this would work:
> >
> > GRID_DIAG_OUTPUT_TEMPLATE =
> > grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}.nc
> >
> > If you were to run using RUN_ONCE_PER_LEAD, you would want to
reference
> the
> > lead tag instead.
> >
> >
> > GRID_DIAG_OUTPUT_TEMPLATE =
> > grid_diag_out_{MODEL}_{lead?fmt=%3H}.nc
> >
> > If you were running with RUN_ONCE_FOR_EACH, you would want to
reference
> > lead and either init or valid, depending on how you are running.
Since
> you
> > are looping over init times, I would recommend something like:
> >
> >
> > GRID_DIAG_OUTPUT_TEMPLATE =
> > grid_diag_out_{MODEL}_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc
> >
> > Does that make sense?
> >
> > - George
> >
> > On Mon, Dec 21, 2020 at 3:41 PM Jamie Wolff via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > >
> > > Sorry, yes. In the case when I set RUN_ONCE it works perfectly
fine.
> > > However, when I change to run the other options that logic
breaks down.
> > > Maybe it isn't a big deal right now (since I plan to use
RUN_ONCE) but
> in
> > > the future I would want to know how to change the template so it
could
> > work
> > > for any of the four options.
> > > Make sense?
> > > Jamie
> > >
> > > On Mon, Dec 21, 2020 at 3:37 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Glad it worked! I'm not sure I understand the issue with the
output
> > > > template. If Grid-Diag is run once for all of the files, isn't
the
> > > template
> > > > you have there unique enough to that case (model, time ranges)
that
> it
> > > > wouldn't be overwritten?
> > > >
> > > > On Mon, Dec 21, 2020 at 3:32 PM Jamie Wolff via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672 >
> > > > >
> > > > > That did the trick! I can now reproduce what I was doing by
hand.
> > > > >
> > > > > I tested all four options and they all seemed to work great.
This
> is
> > a
> > > > > great enhancement - thank you!
> > > > >
> > > > > One remaining question is figuring out the best way to set
the
> > template
> > > > > file name so it is appropriately filled and not overwriting
files.
> > Can
> > > > you
> > > > > suggest what I might change the below line to so it would
work for
> > the
> > > > new
> > > > > options?
> > > > >
> > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > >
> > > > >
> > > >
> > >
> >
> grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
{ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > >
> > > > > Thank you!
> > > > > Jamie
> > > > >
> > > > > On Mon, Dec 21, 2020 at 3:03 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Jamie,
> > > > > >
> > > > > > From my understanding that is likely the cause of the
error. I
> made
> > > > some
> > > > > > changes that should fix it for your case. Could you try
running
> > your
> > > > case
> > > > > > using this master_metplus.py on cheyenne?
> > > > > >
> > > > > > /glade/u/home/mccabe/METplus/ush/master_metplus.py
> > > > > >
> > > > > > You could also run "git pull" on the
feature_733_grid_diag_freq
> > > branch
> > > > if
> > > > > > that is easier. Let me know how it goes.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Mon, Dec 21, 2020 at 2:59 PM Jamie Wolff via RT <
> > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> >
> > > > > > >
> > > > > > > I'm not entirely sure that is the cause of the error,
but it is
> > new
> > > > > > > behavior that may be having an impact.
> > > > > > >
> > > > > > > Here is how I used to run grid_diag (with the lists I
created
> by
> > > > hand):
> > > > > > > grid_diag -data
> > > > > > >
> > > stage/file_lists.sav/grid_diag_data_files_idx0_20200608-15_f00-
21.txt
> > > > > > -data
> > > > > > >
> > > stage/file_lists.sav/grid_diag_data_files_idx1_20200608-15_f00-
21.txt
> > > > > > -out
> > > > > > > test_new.nc -config GridDiagConfig -v 4
> > > > > > >
> > > > > > > With the list created now I get the error I sent:
> > > > > > > grid_diag -data
> > > > > > >
> stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > -out
> > > > > > > test_new_ALL.nc -config GridDiagConfig -v 4
> > > > > > >
> > > > > > > ...
> > > > > > > MetGrib2DataFile::read_grib2_record_list() - processed 4
> records
> > > from
> > > > > > GRIB2
> > > > > > > file:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/model_data/GFS/gfs.20200608/00/gfs.t00z.special.grb2f000
> > > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > object
> > > > > of
> > > > > > > type "FileType_Gb2".
> > > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > object
> > > > > of
> > > > > > > type "FileType_Gb2".
> > > > > > > ERROR  :
> > > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized GRIB2
field
> > > > > > abbreviation
> > > > > > > 'tb_g16'
> > > > > > >
> > > > > > > Does that help explain?
> > > > > > >
> > > > > > > You should be able to run both of those commands on the
command
> > > > line. I
> > > > > > am
> > > > > > > using version 9.1.1 of MET with this environment:
> > > > > > > module use
/glade/p/ral/jntp/MET/MET_releases/modulefiles
> > > > > > > module load met/9.1.1
> > > > > > > ncar_pylib
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Jamie
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 21, 2020 at 2:10 PM George McCabe via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jamie,
> > > > > > > >
> > > > > > > > Yes, that error message should say LOOP_ORDER. Thanks
for
> > > catching
> > > > > > that!
> > > > > > > >
> > > > > > > > Are you saying the error is occurring because all of
the
> files
> > > are
> > > > > > lumped
> > > > > > > > together instead of specified in separate -data
arguments? If
> > > so, I
> > > > > can
> > > > > > > > modify the code to separate out each item in the
template
> list.
> > > > > > > >
> > > > > > > > On Mon, Dec 21, 2020 at 1:42 PM Jamie Wolff via RT <
> > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > >
> > > > > > > > >
> > > > > > > > > Hi George,
> > > > > > > > > I am trying to test your new mods in my processing.
> > > > > > > > > I have a script to run grid_diag here:
> > > > > > > > >
> > /glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > >
> > > > > > > > > I made the change you mentioned needed to be added
> > > > > > > > (GRID_DIAG_RUNTIME_FREQ
> > > > > > > > > = RUN_ONCE) to my METplus parm and you can see the
parm
> file
> > > > > > > > >
> > > > >
> here:/glade/p/ral/jntp/AF/cloud_vx/parm/metplus/gfs_griddiag_tb.conf
> > > > > > > > >
> > > > > > > > > However, when I run it I get the following error:
> > > > > > > > > 12/21 13:20:13.449 metplus (command_builder.py:163)
ERROR:
> > > > > > > > > (runtime_freq_wrapper.py:85) Cannot run using
RUN_ONCE mode
> > > unles
> > > > > > > > > s LOOP_BY = processes
> > > > > > > > >
> > > > > > > > > But that is confusing because I do not think LOOP_BY
has an
> > > > option
> > > > > of
> > > > > > > > > processes. Should that be LOOP_ORDER? That is what I
tried
> > and
> > > > that
> > > > > > > > seemed
> > > > > > > > > to get me further.
> > > > > > > > >
> > > > > > > > > Before, there would be two file_lists created one
for the
> > model
> > > > and
> > > > > > one
> > > > > > > > for
> > > > > > > > > the obs
> > > > > > > > > (For example:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx0_20200608-
15_f00-21.txt
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists.sav/grid_diag_data_files_idx1_20200608-
15_f00-21.txt)
> > > > > > > > > But now it looks like they are both in one file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_files_init_ALL_valid_ALL_lead_ALL.txt
> > > > > > > > >
> > > > > > > > > I think this is causing a problem because I am
getting an
> > > error:
> > > > > > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES
file:
> > > > > > > > > /glade/p/ral/jntp/AF/cloud_vx/parm/grib2_tb.txt
> > > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/MET/MET_releases/9.1.1/share/met/config/GridDiagConfig_default
> > > > > > > > > DEBUG 1: User Config File:
> > > > > > > > >
/glade/p/ral/jntp/AF/cloud_vx/parm/met/GridDiagConfig_Tb
> > > > > > > > > ERROR  :
> > > > > > > > > ERROR  : VarInfoGrib2::set_dict() -> unrecognized
GRIB2
> field
> > > > > > > > abbreviation
> > > > > > > > > 'tb_g16'
> > > > > > > > > ERROR  :
> > > > > > > > >
> > > > > > > > > I am guessing this is a problem with my config file,
not
> > > > > necessarily
> > > > > > > your
> > > > > > > > > logic. Could you take a look at my config file to
see if I
> > have
> > > > > > > > additional
> > > > > > > > > logic that needs to change for this to work? Let me
know if
> > you
> > > > > need
> > > > > > > more
> > > > > > > > > information from me.
> > > > > > > > > Thanks!
> > > > > > > > > Jamie
> > > > > > > > >
> > > > > > > > > On Thu, Dec 17, 2020 at 1:31 PM George McCabe via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jamie,
> > > > > > > > > >
> > > > > > > > > > Not a problem at all. You should just be able to
obtain
> the
> > > > > branch
> > > > > > > and
> > > > > > > > > call
> > > > > > > > > > that version of master_metplus.py instead of the
one you
> > are
> > > > > > > currently
> > > > > > > > > > using. Let me know if you run into any issues and
I can
> > help.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 17, 2020 at 1:10 PM Jamie Wolff via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi George,
> > > > > > > > > > > I  am sorry, I am swamped this week. I hope to
get to
> > this
> > > > next
> > > > > > > week
> > > > > > > > or
> > > > > > > > > > the
> > > > > > > > > > > week after.  In the meantime, are there any
build
> > > > instructions
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > share with me to use when  I pull this branch on
> > cheyenne?
> > > > > > > > > > > I will be in touch as soon as I can.
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Jamie
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 17, 2020 at 1:05 PM George McCabe
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > >
> > > > > > > > > > > > Just following up on this in case this email
was
> > buried.
> > > > Let
> > > > > me
> > > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > > you need any help getting this to work!
> > > > > > > > > > > >
> > > > > > > > > > > > - George
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon Dec 14 16:26:08 2020, mccabe wrote:
> > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I implemented the changes to GridDiag that
were
> > > necessary
> > > > > to
> > > > > > > run
> > > > > > > > > the
> > > > > > > > > > > > > tool as you expected. I assigned you to the
pull
> > > request:
> > > > > > > > > > > > >
> > > > > > > > > > > > > https://github.com/dtcenter/METplus/pull/735
> > > > > > > > > > > > >
> > > > > > > > > > > > > The pull request has notes regarding which
branch
> to
> > > > > checkout
> > > > > > > on
> > > > > > > > > > > > > cheyenne and which setting to add to get it
to
> work.
> > > Let
> > > > me
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > > > > you have any questions!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon Dec 14 10:51:07 2020, jwolff wrote:
> > > > > > > > > > > > > > Sure. No problem. Thank you!
> > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Dec 14, 2020 at 10:37 AM George
McCabe
> via
> > RT
> > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I created a GitHub issue for this work:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
https://github.com/dtcenter/METplus/issues/733
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have implemented functionality to call
> certain
> > > > > wrappers
> > > > > > > in
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > intervals to process groups of files
instead of
> > > just
> > > > > once
> > > > > > > per
> > > > > > > > > > file.
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > now working on incorporating this
functionality
> > > into
> > > > > the
> > > > > > > > > GridDiag
> > > > > > > > > > > > > > > wrapper.
> > > > > > > > > > > > > > > If you don't mind, I was planning on
assigning
> > the
> > > > pull
> > > > > > > > request
> > > > > > > > > > > > > > > review to
> > > > > > > > > > > > > > > you so you can test these changes on
your use
> > case
> > > to
> > > > > > > ensure
> > > > > > > > it
> > > > > > > > > > > > > > > runs
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > way you expect.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon Nov 30 17:31:30 2020, jwolff
wrote:
> > > > > > > > > > > > > > > > Hi George and John,
> > > > > > > > > > > > > > > > Sounds like the modifications you are
> > discussing
> > > > will
> > > > > > > > address
> > > > > > > > > > my
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > (correct - the list of init times
> > > > > > > > > > > > > > > > and forecast leads to determine which
files
> to
> > > use
> > > > > in a
> > > > > > > > > single
> > > > > > > > > > > > > > > > analysis).
> > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:43 PM George
McCabe
> > via
> > > > RT
> > > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > OK, it sounds like I misunderstood
the
> > intended
> > > > use
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > > > > > > tool
> > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > implemented the wrapper. Thanks for
> pointing
> > > this
> > > > > > out,
> > > > > > > > > Jamie!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > John, I think we should have a
discussion
> > about
> > > > how
> > > > > > we
> > > > > > > > > would
> > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > tools like SeriesAnalysis and
GridDiag to
> be
> > > > > > configured
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > cases. I'll follow up with you to
set
> > something
> > > > up.
> > > > > > > > > > > > > > > > > SeriesAnalysis is
> > > > > > > > > > > > > > > > > currently separated into
SeriesByInit and
> > > > > > SeriesByLead
> > > > > > > > > which
> > > > > > > > > > > > > > > > > perform
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > grouping. However, I am currently
> refactoring
> > > > these
> > > > > > > tools
> > > > > > > > > to
> > > > > > > > > > > > > > > > > hopefully
> > > > > > > > > > > > > > > > > combine the logic and allow options
to
> > specify
> > > > > > > different
> > > > > > > > > ways
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > filter/group the data. This would be
a good
> > > time
> > > > to
> > > > > > > come
> > > > > > > > up
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > good way
> > > > > > > > > > > > > > > > > to define how you want to group your
data
> in
> > a
> > > > > > general
> > > > > > > > way
> > > > > > > > > > that
> > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > apply
> > > > > > > > > > > > > > > > > to other similar wrappers, like
GridDiag.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jamie, to confirm what you are
saying, you
> > > > expected
> > > > > > the
> > > > > > > > > list
> > > > > > > > > > of
> > > > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > > > and forecast leads to determine
which files
> > to
> > > > use
> > > > > > in a
> > > > > > > > > > single
> > > > > > > > > > > > > > > > > analysis. Is
> > > > > > > > > > > > > > > > > that correct?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 4:26 PM John
Halley
> > > > Gotway
> > > > > > via
> > > > > > > > RT <
> > > > > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > <URL:
> > > > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > George,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The logic for running the Grid-
Diag is
> more
> > > > like
> > > > > > > > Series-
> > > > > > > > > > > > > > > > > > Analysis
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > Stat-Analysis than it is like
Grid-Stat.
> > > While
> > > > a
> > > > > > user
> > > > > > > > > could
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > single time, it is more likely
they'll
> want
> > > to
> > > > > run
> > > > > > it
> > > > > > > > > over
> > > > > > > > > > a
> > > > > > > > > > > > > > > > > > set of
> > > > > > > > > > > > > > > > > times.
> > > > > > > > > > > > > > > > > > It really is pretty analogous to
> > > > > Series-Analysis...
> > > > > > > you
> > > > > > > > > may
> > > > > > > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > > > the init time fixed and run over
all lead
> > > > > times...
> > > > > > or
> > > > > > > > > hold
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > fixed and run over all init
times... or
> > just
> > > > run
> > > > > > over
> > > > > > > > all
> > > > > > > > > > > > > > > > > > init
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > lead
> > > > > > > > > > > > > > > > > > times in one big batch.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > George, is there any way to
generalize
> the
> > > > > support
> > > > > > > for
> > > > > > > > > how
> > > > > > > > > > > > > > > > > > times
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > processed?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 3:42 PM
George
> > McCabe
> > > > via
> > > > > > RT
> > > > > > > > > > > > > > > > > > <met_help at ucar.edu>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > <URL:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97672
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jamie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I moved this email over to MET
Help.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The configuration you have set
up loops
> > > over
> > > > > each
> > > > > > > run
> > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > range
> > > > > > > > > > > > > > > > > > > you specified (2020060800 to
2020061500
> > > with
> > > > 24
> > > > > > > hour
> > > > > > > > > > > > > > > > > > > increments
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > processing the list of forecast
leads
> for
> > > > each
> > > > > > > time).
> > > > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > > > > > template will be the same for
each run:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > GRID_DIAG_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
grid_diag_out_{MODEL}_{ENV[INIT_BEG]}-
> > > > > > > > > > > > > > > > >
> > > > > {ENV[INIT_END]}_f{ENV[FHR_FIRST]}-{ENV[FHR_LAST]}.nc
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is why you are seeing only
the
> > results
> > > > of
> > > > > > the
> > > > > > > > last
> > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > > > > Each run overwrites the data
from the
> > > > previous
> > > > > > run.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Wrappers like SeriesAnalysis and
> > > StatAnalysis
> > > > > are
> > > > > > > > > > designed
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > process a
> > > > > > > > > > > > > > > > > > > range of times at once, but
GridDiag
> was
> > > > > written
> > > > > > to
> > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > once
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > > time. However, this may not be
the
> > intended
> > > > way
> > > > > > to
> > > > > > > > use
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > tool.
> > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > how I implemented it based on
the
> > example I
> > > > was
> > > > > > > > > provided.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > John HG (copied here) may have
some
> > insight
> > > > on
> > > > > > the
> > > > > > > > best
> > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > > > > > what you are trying to do.
Perhaps you
> > > would
> > > > > want
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > > > > > > > GridDiag
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > > run time, then run another tool
to
> > > aggregate
> > > > > the
> > > > > > > > stats
> > > > > > > > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > output.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > If that is not the case, then
you
> should
> > be
> > > > > able
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > wrapper to
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > the way you are intending by
> configuring
> > it
> > > > to
> > > > > > > > process
> > > > > > > > > a
> > > > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > (INIT_BEG==INIT_END) and expand
the
> list
> > of
> > > > > > values
> > > > > > > > for
> > > > > > > > > > > > > > > > > > > GRID_DIAG_INPUT_TEMPLATE to
include the
> > > times
> > > > > you
> > > > > > > > need
> > > > > > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > 'shift'
> > > > > > > > > > > > > > > > > > > keyword in the filename
templates, i.e.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > {init?fmt=...},
> {init?fmt=...?shift=24H},
> > > > > > > > > > > > > > > > > > > {init?fmt=...?shift=48H}, and
> > > > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > > on.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This approach is not very
flexible and
> > > would
> > > > > > > require
> > > > > > > > > you
> > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > a lot
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > changes to the METplus config
file for
> > each
> > > > > range
> > > > > > > of
> > > > > > > > > > times
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > desire.
> > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > this is the only way to achieve
what
> you
> > > > need,
> > > > > > then
> > > > > > > > we
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > to the wrapper to allow you to
> configure
> > > this
> > > > > > more
> > > > > > > > > > easily.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > George
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon Nov 30 15:27:50 2020,
mccabe
> > wrote:
> > > > > > > > > > > > > > > > > > > > Hi George,
> > > > > > > > > > > > > > > > > > > > I am running grid-diag using
the
> script
> > > > found
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
here:cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/scripts/run_metplus_griddiag.ksh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It all appears to be running
to
> > > completion
> > > > > but
> > > > > > > the
> > > > > > > > > > > > > > > > > > > > functionality is
> > > > > > > > > > > > > > > > > > > > not what I was expecting. To
test the
> > > > tool, I
> > > > > > > first
> > > > > > > > > set
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > script to
> > > > > > > > > > > > > > > > > > > > run one day
(INIT_BEG/END=2020061500)
> > and
> > > > it
> > > > > > > > created
> > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > > lists
> > > > > > > > > > > > > > > > > > > > (idx0/1) that I would have
expected
> > that
> > > > > > included
> > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > day.
> > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > example:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/p/ral/jntp/AF/cloud_vx/grid_diag/stage/file_lists/grid_diag_data_files_idx0_20200615210000.txt
> > > > > > > > > > > > > > > > > > > > I did the same for all my
single
> days.
> > It
> > > > > > created
> > > > > > > > the
> > > > > > > > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > expect. An example netcdf can
be seen
> > > here:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_20200615_f00-
> > > > > > > > > > > > > > > > > > > > 21.nc
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Then, I wanted to run
something
> similar
> > > for
> > > > > all
> > > > > > > my
> > > > > > > > > > > > > > > > > > > > available
> > > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > (INIT_BEG=2020060800,
> > > INIT_END=2020061500,
> > > > > > > > > > > > > > > > > > > > INIT_INC=86400)
> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > like to have all days
aggregated
> > together
> > > > and
> > > > > > one
> > > > > > > > > > netcdf
> > > > > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > > created
> > > > > > > > > > > > > > > > > > > > in the end. It did create one
netcdf
> > file
> > > > at
> > > > > > the
> > > > > > > > end,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > included the data for the last
date
> > (the
> > > > > 15th).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
cheyenne3:/glade/p/ral/jntp/AF/cloud_vx/grid_diag/GridDiag/grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > > > >  If you diff the two nc files
> > > > > > > > > > > > > > > > > > > >
grid_diag_out_GFS_20200615_f00-21.nc
> > and
> > > > > > > > > > > > > > > > > grid_diag_out_GFS_2020060800-
> > > > > > > > > > > > > > > > > > > > 2020061500_f00-21.nc
> > > > > > > > > > > > > > > > > > > > they are identical.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I could be confused on the
intent of
> > the
> > > > > tool.
> > > > > > > > > Perhaps
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > behavior you want it to have.
If so,
> I
> > > > still
> > > > > > > think
> > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > > > > because it should be
outputting a
> file
> > > per
> > > > > day,
> > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > appears
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > output a file at the end for
the last
> > > day.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > What I thought would happen is
I
> would
> > > get
> > > > > one
> > > > > > > > file,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > include all days/times
requested. So,
> > > what
> > > > I
> > > > > > need
> > > > > > > > it
> > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > list that includes all model
(idx0)
> and
> > > obs
> > > > > > > (idx1)
> > > > > > > > > > files
> > > > > > > > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > > > to use in one file. I can do
this by
> > hand
> > > > > and I
> > > > > > > am
> > > > > > > > > > quite
> > > > > > > > > > > > > > > > > > > > certain it
> > > > > > > > > > > > > > > > > > > > would run and do what I want,
but I
> > can't
> > > > get
> > > > > > it
> > > > > > > to
> > > > > > > > > > work
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > automated fashion.
(Essentially I
> want
> > it
> > > > to
> > > > > do
> > > > > > > > > > something
> > > > > > > > > > > > > > > > > > > > similar to
> > > > > > > > > > > > > > > > > > > > series/stat-analysis, for
example -
> > > > aggregate
> > > > > > > over
> > > > > > > > > > time)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It is very possible I am doing
> > something
> > > > > wrong
> > > > > > on
> > > > > > > > my
> > > > > > > > > > end.
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > hope so
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > that would be an easy fix. If
this
> > > doesn't
> > > > > make
> > > > > > > > sense
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > like to discuss this on the
phone at
> > some
> > > > > > point,
> > > > > > > I
> > > > > > > > am
> > > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > do so.
> > > > > > > > > > > > > > > > > > > > Just let me know what you
think.
> Thanks
> > > in
> > > > > > > advance
> > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > > > help. If
> > > > > > > > > > > > > > > > > > > > my dir structure is confusing
or you
> > need
> > > > me
> > > > > to
> > > > > > > > send
> > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > anything
> > > > > > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > > > > > please don't hesitate to ask.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > Jamie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > George McCabe - Software Engineer
III
> > > > > > > > > > > > > > > > > National Center for Atmospheric
Research
> > > > > > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > My working day may not be your
working day.
> > > > Please
> > > > > do
> > > > > > > not
> > > > > > > > > > feel
> > > > > > > > > > > > > > > > > obliged to
> > > > > > > > > > > > > > > > > reply to this email outside of your
normal
> > > > working
> > > > > > > hours.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Jamie K. Wolff
> > > > > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > > > > >
> > > > > > > > > > > *My working day may not be your working day.
Please do
> > not
> > > > feel
> > > > > > > > obliged
> > > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> > hours.*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jamie K. Wolff
> > > > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > > > Research Applications Laboratory (RAL)
> > > > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > > > >
> > > > > > > > > *My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jamie K. Wolff
> > > > > > > National Center for Atmospheric Research (NCAR)
> > > > > > > Research Applications Laboratory (RAL)
> > > > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > > > >
> > > > > > > *My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jamie K. Wolff
> > > > > National Center for Atmospheric Research (NCAR)
> > > > > Research Applications Laboratory (RAL)
> > > > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > Jamie K. Wolff
> > > National Center for Atmospheric Research (NCAR)
> > > Research Applications Laboratory (RAL)
> > > Phone: 303.497.2812  Email: jwolff at ucar.edu
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> Jamie K. Wolff
> National Center for Atmospheric Research (NCAR)
> Research Applications Laboratory (RAL)
> Phone: 303.497.2812  Email: jwolff at ucar.edu
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------