[Met_help] [rt.rap.ucar.edu #92739] History for PCP Combine Wrapper
George McCabe via RT
met_help at ucar.edu
Mon Mar 2 09:38:52 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I have an interesting PCP combine case that I wanted to bring to your attention. I'm attempting to use the METplus wrappers instead of my own, but haven't been able to crack the configuration code that would create the proper pcp_combine command. Here is my issue:
I am processing individual members of the HREF, but the APCP variable accumulation period of members are inconsistently stored. Typically when doing ensemble precip verification, I use the total accumulation from cycle init to forecast hour, which is simple enough for those grib2 output files that include a "APCP:0-{fcst_len}" variable. The problem is that EMC sends us files where precip variables are encoded like this (for some models):
1-hr forecast - APCP:surface:0-1 hour acc fcst:
2-hr forecast - APCP:surface:0-2 hour acc fcst:
3-hr forecast - APCP:surface:0-3 hour acc fcst:
4-hr forecast - APCP:surface:3-4 hour acc fcst:
5-hr forecast - APCP:surface:3-5 hour acc fcst:
6-hr forecast - APCP:surface:3-6 hour acc fcst:
7-hr forecast - APCP:surface:6-7 hour acc fcst:
...
After experimenting with various combinations of config variables and looking through the python code, I'm not sure the METplus PCP combine wrapper can currently handle a case like this as it would require two different types of accumulation levels to be set dynamically based on what the forecast hour is (among other things). Can you correct me if I'm (hopefully) wrong?
For reference, these are the commands that I use to convert/add all 7 of the examples listed above:
1. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_01.grib2 01 nam_nest_2019102112_01.nc
2. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_02.grib2 02 nam_nest_2019102112_02.nc
3. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.grib2 03 nam_nest_2019102112_03.nc
4. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3 nam_nest_2019102112_04.grib2 1 nam_nest_2019102112_04.nc
5. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3 nam_nest_2019102112_05.grib2 2 nam_nest_2019102112_05.nc
6. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3 nam_nest_2019102112_06.grib2 3 nam_nest_2019102112_06.nc
7. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_06.nc 6 nam_nest_2019102112_07.grib2 1 nam_nest_2019102112_07.nc
Thanks,
Jeff
--
-------------------------------------------------------------------------------
Jeff Hamilton Jeffrey.A.Hamilton at noaa.gov
CIRES Research Associate phone: 303-497-7312
NOAA/OAR/ESRL/GSD/ADB fax: 303-497-4176
325 Broadway, R/GSD1 DSRC Room 2B604
Boulder, CO 80305
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: PCP Combine Wrapper
From: George McCabe
Time: Mon Oct 21 16:34:27 2019
Hi Jeff,
In the upcoming METplus 3.0 release (and available in the METplus
v3.0beta-1 release that just came out), you will be able to specify a
list
of accumulation levels that are available in the input data in order
of
preference. You are correct that you can't specify what is in the
files
based on the file time. It assumes that all of the files available
have all
of the accumulation amounts that you specified. For example, if you
are
trying to build a 7-hr accumulation at 20191021_19Z and you specify
you
have 3, 2, 1 as the accumulations available, the METplus wrapper will
build
a command adding these 3 files:
FCST_PCP_COMBINE_INPUT_ACCUMS = 3, 2, 1
nam_nest_2019102112_07.grib2 3
nam_nest_2019102112_04.grib2 3
nam_nest_2019102112_01.grib2 1
The command would fail if the files specified didn't actually have the
accumulation amounts that are listed.
If you just specify you want to use 1 hr accumulations, it would add:
FCST_PCP_COMBINE_INPUT_ACCUMS = 1
nam_nest_2019102112_07.grib2 1
nam_nest_2019102112_06.grib2 1
nam_nest_2019102112_05.grib2 1
nam_nest_2019102112_04.grib2 1
nam_nest_2019102112_03.grib2 1
nam_nest_2019102112_02.grib2 1
nam_nest_2019102112_01.grib2 1
This would work if each forecast always had a 1 hr accumulation in it,
but
it would not utilize the 3 or 2 hour accumulation when it is
available.
In METplus 3.0 you can also build a custom call to pcp_combine
specifying
the args as you wish, but since it changes from run to run, you
wouldn't be
able to write a custom command that fits all situations.
If we would add an configurations to the wrappers to be able to
specify the
available contents of each file based on the forecast lead, or add the
ability to query the actual file and to see what is available to use,
then
the tools could build the commands you need. Unfortunately, I don't
think
what you need is possible via METplus at the moment. I am copying John
(who
is out today) in case he has an idea of how to handle this case.
Thanks,
George
On Mon, Oct 21, 2019 at 3:35 PM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> Mon Oct 21 15:35:53 2019: Request 92739 was acted upon.
> Transaction: Ticket created by minnawin
> Queue: met_help
> Subject: PCP Combine Wrapper
> Owner: mccabe
> Requestors: jeffrey.a.hamilton at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92739 >
>
>
> I have an interesting PCP combine case that I wanted to bring to
your
> attention. I'm attempting to use the METplus wrappers instead of my
own,
> but haven't been able to crack the configuration code that would
create the
> proper pcp_combine command. Here is my issue:
>
> I am processing individual members of the HREF, but the APCP
variable
> accumulation period of members are inconsistently stored. Typically
when
> doing ensemble precip verification, I use the total accumulation
from cycle
> init to forecast hour, which is simple enough for those grib2 output
files
> that include a "APCP:0-{fcst_len}" variable. The problem is that EMC
sends
> us files where precip variables are encoded like this (for some
models):
>
> 1-hr forecast - APCP:surface:0-1 hour acc fcst:
> 2-hr forecast - APCP:surface:0-2 hour acc fcst:
> 3-hr forecast - APCP:surface:0-3 hour acc fcst:
> 4-hr forecast - APCP:surface:3-4 hour acc fcst:
> 5-hr forecast - APCP:surface:3-5 hour acc fcst:
> 6-hr forecast - APCP:surface:3-6 hour acc fcst:
> 7-hr forecast - APCP:surface:6-7 hour acc fcst:
> ...
>
> After experimenting with various combinations of config variables
and
> looking through the python code, I'm not sure the METplus PCP
combine
> wrapper can currently handle a case like this as it would require
two
> different types of accumulation levels to be set dynamically based
on what
> the forecast hour is (among other things). Can you correct me if I'm
> (hopefully) wrong?
>
> For reference, these are the commands that I use to convert/add all
7 of
> the examples listed above:
>
> 1. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_01.grib2 01
> nam_nest_2019102112_01.nc
> 2. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_02.grib2 02
> nam_nest_2019102112_02.nc
> 3. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.grib2 03
> nam_nest_2019102112_03.nc
> 4. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> nam_nest_2019102112_04.grib2 1 nam_nest_2019102112_04.nc
> 5. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> nam_nest_2019102112_05.grib2 2 nam_nest_2019102112_05.nc
> 6. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> nam_nest_2019102112_06.grib2 3 nam_nest_2019102112_06.nc
> 7. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_06.nc 6
> nam_nest_2019102112_07.grib2 1 nam_nest_2019102112_07.nc
>
> Thanks,
>
> Jeff
> --
>
>
-------------------------------------------------------------------------------
> Jeff Hamilton
> Jeffrey.A.Hamilton at noaa.gov
> CIRES Research Associate phone: 303-497-7312
> NOAA/OAR/ESRL/GSD/ADB fax: 303-497-4176
> 325 Broadway, R/GSD1 DSRC Room 2B604
> Boulder, CO 80305
>
--
------------------------------------------------------------
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
------------------------------------------------------------
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PCP Combine Wrapper
From: George McCabe
Time: Mon Mar 02 09:33:16 2020
Hi Jeff,
I apologize that this ticket was buried for so long and if this was
already addressed outside of MET Help. PCPCombine now supports bucket
intervals, so you can specify the accum amounts based on forecast
lead. There is an example use case that demonstrates this behavior:
parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
The new configuration variables of interest are:
FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
This says that you should use the forecast lead as the input
accumulation and reset the bucket every 6 hours. If you wanted to tell
METplus to use the forecast lead or a 1 hour accum (if the available
accum from the forecast lead is larger than what is remaining to build
the desired accum), then you can set:
FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
The order of the input accums are the order that they are attempted to
use.
Let me know if you have any questions.
Thanks,
George
On Mon Oct 21 16:34:27 2019, mccabe wrote:
> Hi Jeff,
>
> In the upcoming METplus 3.0 release (and available in the METplus
> v3.0beta-1 release that just came out), you will be able to specify
a
> list
> of accumulation levels that are available in the input data in order
> of
> preference. You are correct that you can't specify what is in the
> files
> based on the file time. It assumes that all of the files available
> have all
> of the accumulation amounts that you specified. For example, if you
> are
> trying to build a 7-hr accumulation at 20191021_19Z and you specify
> you
> have 3, 2, 1 as the accumulations available, the METplus wrapper
will
> build
> a command adding these 3 files:
>
> FCST_PCP_COMBINE_INPUT_ACCUMS = 3, 2, 1
>
> nam_nest_2019102112_07.grib2 3
> nam_nest_2019102112_04.grib2 3
> nam_nest_2019102112_01.grib2 1
>
> The command would fail if the files specified didn't actually have
the
> accumulation amounts that are listed.
>
> If you just specify you want to use 1 hr accumulations, it would
add:
>
> FCST_PCP_COMBINE_INPUT_ACCUMS = 1
>
> nam_nest_2019102112_07.grib2 1
> nam_nest_2019102112_06.grib2 1
> nam_nest_2019102112_05.grib2 1
> nam_nest_2019102112_04.grib2 1
> nam_nest_2019102112_03.grib2 1
> nam_nest_2019102112_02.grib2 1
> nam_nest_2019102112_01.grib2 1
>
> This would work if each forecast always had a 1 hr accumulation in
it,
> but
> it would not utilize the 3 or 2 hour accumulation when it is
> available.
>
> In METplus 3.0 you can also build a custom call to pcp_combine
> specifying
> the args as you wish, but since it changes from run to run, you
> wouldn't be
> able to write a custom command that fits all situations.
>
> If we would add an configurations to the wrappers to be able to
> specify the
> available contents of each file based on the forecast lead, or add
the
> ability to query the actual file and to see what is available to
use,
> then
> the tools could build the commands you need. Unfortunately, I don't
> think
> what you need is possible via METplus at the moment. I am copying
John
> (who
> is out today) in case he has an idea of how to handle this case.
>
> Thanks,
> George
>
> On Mon, Oct 21, 2019 at 3:35 PM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > Mon Oct 21 15:35:53 2019: Request 92739 was acted upon.
> > Transaction: Ticket created by minnawin
> > Queue: met_help
> > Subject: PCP Combine Wrapper
> > Owner: mccabe
> > Requestors: jeffrey.a.hamilton at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92739
> > >
> >
> >
> > I have an interesting PCP combine case that I wanted to bring to
your
> > attention. I'm attempting to use the METplus wrappers instead of
my
> > own,
> > but haven't been able to crack the configuration code that would
> > create the
> > proper pcp_combine command. Here is my issue:
> >
> > I am processing individual members of the HREF, but the APCP
variable
> > accumulation period of members are inconsistently stored.
Typically
> > when
> > doing ensemble precip verification, I use the total accumulation
from
> > cycle
> > init to forecast hour, which is simple enough for those grib2
output
> > files
> > that include a "APCP:0-{fcst_len}" variable. The problem is that
EMC
> > sends
> > us files where precip variables are encoded like this (for some
> > models):
> >
> > 1-hr forecast - APCP:surface:0-1 hour acc fcst:
> > 2-hr forecast - APCP:surface:0-2 hour acc fcst:
> > 3-hr forecast - APCP:surface:0-3 hour acc fcst:
> > 4-hr forecast - APCP:surface:3-4 hour acc fcst:
> > 5-hr forecast - APCP:surface:3-5 hour acc fcst:
> > 6-hr forecast - APCP:surface:3-6 hour acc fcst:
> > 7-hr forecast - APCP:surface:6-7 hour acc fcst:
> > ...
> >
> > After experimenting with various combinations of config variables
and
> > looking through the python code, I'm not sure the METplus PCP
combine
> > wrapper can currently handle a case like this as it would require
two
> > different types of accumulation levels to be set dynamically based
on
> > what
> > the forecast hour is (among other things). Can you correct me if
I'm
> > (hopefully) wrong?
> >
> > For reference, these are the commands that I use to convert/add
all 7
> > of
> > the examples listed above:
> >
> > 1. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_01.grib2
01
> > nam_nest_2019102112_01.nc
> > 2. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_02.grib2
02
> > nam_nest_2019102112_02.nc
> > 3. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.grib2
03
> > nam_nest_2019102112_03.nc
> > 4. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> > nam_nest_2019102112_04.grib2 1 nam_nest_2019102112_04.nc
> > 5. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> > nam_nest_2019102112_05.grib2 2 nam_nest_2019102112_05.nc
> > 6. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_03.nc 3
> > nam_nest_2019102112_06.grib2 3 nam_nest_2019102112_06.nc
> > 7. pcp_combine -v 2 -name APCP -add nam_nest_2019102112_06.nc 6
> > nam_nest_2019102112_07.grib2 1 nam_nest_2019102112_07.nc
> >
> > Thanks,
> >
> > Jeff
> > --
> >
> >
-------------------------------------------------------------------------------
> > Jeff Hamilton
> > Jeffrey.A.Hamilton at noaa.gov
> > CIRES Research Associate phone: 303-497-7312
> > NOAA/OAR/ESRL/GSD/ADB fax: 303-497-4176
> > 325 Broadway, R/GSD1 DSRC Room 2B604
> > Boulder, CO 80305
> >
------------------------------------------------
More information about the Met_help
mailing list