[Met_help] [rt.rap.ucar.edu #94302] History for METplus pcp_combine questions
George McCabe via RT
met_help at ucar.edu
Mon Mar 2 15:59:00 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi there MET help desk,
I have been running METplus v. 2.2 and had a couple of questions regarding the pcp_combine tool. I’m currently working on processing an ensemble of FV3 members to generate statistics for that ensemble. The ensemble is quite large though (40 members) so I was wondering if there was any shortcut I could specify in the config file to read in each member of the ensemble or if each member’s file path and file has to be specified individually in the INPUT_TEMPLATE section of the config file?
The other question I have is regarding the lead time and accumulations created. I’m looking to build precipitation accumulations based on the lead time (e.g., 2-hour accumulation for a 2-hour lead time, 3-hour accumulation at 3-hour lead time, and so on) and have each desired accumulation specified in the config file along with the incremented lead times (“begin_end_incr(1,6,1)”) using the ADD function. However, I’ll notice that for some lead times, the accumulation does not line up correctly (i.e., I’ll have a 4-hour accumulation at 3-hour lead time). This behavior doesn’t exist when my accumulation-leadtime pairs are specified in odd or even increments, so 1,3,5 or 2,4,6 as an example. Is there support in METplus 2.2 to be able to bucket the accumulations by the lead time, or if not, is there a workaround that could be useful?
Any help would be greatly appreciated.
Thanks!
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: METplus pcp_combine questions
From: George McCabe
Time: Tue Feb 25 10:50:38 2020
Hi Brian,
Regarding a shortcut to process multiple ensembles through PCPCombine
easily, METplus does not currently support a good way to do this. It
makes
sense to enhance the logic to make this easier. I will have to think
of a
good way to handle it. If you have any suggestions of how you would
like to
be able to configure METplus to do this, please let me know. Other
wrappers
do support wildcard characters to find multiple files, so we could
potentially use this logic slightly modified here.
Regarding accumulation buckets, functionality was added to METplus
that
will be found in version 3.0 that can handle this case. It is
available in
3.0 beta3 if you would like to obtain that version to test. Please
note
that there are a few major changes moving from METplus 2 to 3. METplus
3
uses Python 3, so you will need to switch to that version of Python to
run.
There are also a good number of configuration variable name changes.
There
is a utility included with the release to make this transition easier.
You
can substitute master_metplus.py with validate_config.py, passing in
the
same list of configuration files you will be running. This script will
prompt you if you would like the script to automatically replace each
deprecated configuration variables in your files one-by-one.
There is an example of using bucket intervals with PCPCombine here:
https://github.com/NCAR/METplus/blob/master_v3.0-
beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
The new variables of interest are:
FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
This says that the accumulation buckets reset every 6 hours. The input
accums variable is a list of accumulations that are found in the data.
This
configuration says to use the forecast lead as the accum, resetting
every 6
hours. This value can be a comma-separated list, so if the files have
a 1
hour accumulation as well as the accum that corresponds to the
forecast
lead, you could set it to:
FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
Please let me know if you have any questions.
Thanks,
George
On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
>
> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
> Transaction: Given to mccabe (George McCabe) by johnhg
> Queue: met_help
> Subject: METplus pcp_combine questions
> Owner: mccabe
> Requestors: brian.matilla at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>
>
> This transaction appears to have no content
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94302] METplus pcp_combine questions
From: Brian Matilla - NOAA Affiliate
Time: Tue Feb 25 16:18:57 2020
Hi George,
Thanks for the detailed information! I think it would be useful to
incorporate some member-based logic into the pcp_combine wrappers.
I’ll see what I can come up with and follow up with you.
It’s great to hear that bucketing is included with the newest version.
I’ll give the METplus 3 beta a try since I’m currently playing around
with MET 9.0 and its NEP/NMEP capabilities so it would make sense to
trial both at the same time. I think that having the bucket feature
will be of great help for what we’re trying to accomplish.
So more out of curiosity, is there a timetable for when the formal
release of MET 9 and METplus 3 will take place?
-Brian
> On Feb 25, 2020, at 11:50 AM, George McCabe via RT
<met_help at ucar.edu> wrote:
>
> Hi Brian,
>
> Regarding a shortcut to process multiple ensembles through
PCPCombine
> easily, METplus does not currently support a good way to do this. It
makes
> sense to enhance the logic to make this easier. I will have to think
of a
> good way to handle it. If you have any suggestions of how you would
like to
> be able to configure METplus to do this, please let me know. Other
wrappers
> do support wildcard characters to find multiple files, so we could
> potentially use this logic slightly modified here.
>
> Regarding accumulation buckets, functionality was added to METplus
that
> will be found in version 3.0 that can handle this case. It is
available in
> 3.0 beta3 if you would like to obtain that version to test. Please
note
> that there are a few major changes moving from METplus 2 to 3.
METplus 3
> uses Python 3, so you will need to switch to that version of Python
to run.
> There are also a good number of configuration variable name changes.
There
> is a utility included with the release to make this transition
easier. You
> can substitute master_metplus.py with validate_config.py, passing in
the
> same list of configuration files you will be running. This script
will
> prompt you if you would like the script to automatically replace
each
> deprecated configuration variables in your files one-by-one.
>
> There is an example of using bucket intervals with PCPCombine here:
>
> https://github.com/NCAR/METplus/blob/master_v3.0-
beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
>
> The new variables of interest are:
>
> FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
>
> This says that the accumulation buckets reset every 6 hours. The
input
> accums variable is a list of accumulations that are found in the
data. This
> configuration says to use the forecast lead as the accum, resetting
every 6
> hours. This value can be a comma-separated list, so if the files
have a 1
> hour accumulation as well as the accum that corresponds to the
forecast
> lead, you could set it to:
>
> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
>
> Please let me know if you have any questions.
>
> Thanks,
> George
>
> On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
>> Transaction: Given to mccabe (George McCabe) by johnhg
>> Queue: met_help
>> Subject: METplus pcp_combine questions
>> Owner: mccabe
>> Requestors: brian.matilla at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>
>>
>> This transaction appears to have no content
>>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
------------------------------------------------
Subject: METplus pcp_combine questions
From: George McCabe
Time: Tue Feb 25 17:15:34 2020
Hi Brian,
I thought about it more and it seems like it would be more useful to
have a
list of strings that can be looped over to provide additional
information
to use for each run time. A wildcard would require you to use all
ensemble
members, when maybe you only want to process say the first 3.
Something
like:
CUSTOM_RUN_STRINGS = ens1, ens2, ens3
Then for each init or valid and forecast lead run time combination,
METplus
would loop over each value in the list and make it available in the
filename template functionality so it could be used in many places.
Your
template would look something like:
..._INPUT_TEMPLATE = (custom}_{init?fmt=%Y%m%d%H}_f{lead?fmt=%3H}.nc
For init = 20190224_3Z and 3 hr forecast lead, the tool would run 3
times,
using
ens1_2019022403_f003.nc
ens2_2019022403_f003.nc
ens3_2019022403_f003.nc
Ideally I would enhance the begin_end_incr logic to allow strings as
part
of it, so you could more easily define that list.
The MET engineers are working an issue that has delayed the release
for
some time. They are intending to get the release out as soon as
possible,
but the broken logic is imperative to the release. The METplus 3.0
release
will occur when the MET release is ready. I am not sure that I will be
able
to get the above functionality implemented and testing in time for the
release. If I cannot, then I can let you know when the functionality
is
available in the develop branch of our GitHub repository and you could
easily obtain the changes for your case.
On Tue, Feb 25, 2020 at 11:18 PM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>
> Hi George,
>
> Thanks for the detailed information! I think it would be useful to
> incorporate some member-based logic into the pcp_combine wrappers.
I’ll see
> what I can come up with and follow up with you.
>
> It’s great to hear that bucketing is included with the newest
version.
> I’ll give the METplus 3 beta a try since I’m currently playing
around with
> MET 9.0 and its NEP/NMEP capabilities so it would make sense to
trial both
> at the same time. I think that having the bucket feature will be of
great
> help for what we’re trying to accomplish.
>
> So more out of curiosity, is there a timetable for when the formal
release
> of MET 9 and METplus 3 will take place?
>
> -Brian
>
> > On Feb 25, 2020, at 11:50 AM, George McCabe via RT
<met_help at ucar.edu>
> wrote:
> >
> > Hi Brian,
> >
> > Regarding a shortcut to process multiple ensembles through
PCPCombine
> > easily, METplus does not currently support a good way to do this.
It
> makes
> > sense to enhance the logic to make this easier. I will have to
think of a
> > good way to handle it. If you have any suggestions of how you
would like
> to
> > be able to configure METplus to do this, please let me know. Other
> wrappers
> > do support wildcard characters to find multiple files, so we could
> > potentially use this logic slightly modified here.
> >
> > Regarding accumulation buckets, functionality was added to METplus
that
> > will be found in version 3.0 that can handle this case. It is
available
> in
> > 3.0 beta3 if you would like to obtain that version to test. Please
note
> > that there are a few major changes moving from METplus 2 to 3.
METplus 3
> > uses Python 3, so you will need to switch to that version of
Python to
> run.
> > There are also a good number of configuration variable name
changes.
> There
> > is a utility included with the release to make this transition
easier.
> You
> > can substitute master_metplus.py with validate_config.py, passing
in the
> > same list of configuration files you will be running. This script
will
> > prompt you if you would like the script to automatically replace
each
> > deprecated configuration variables in your files one-by-one.
> >
> > There is an example of using bucket intervals with PCPCombine
here:
> >
> >
> https://github.com/NCAR/METplus/blob/master_v3.0-
beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
> >
> > The new variables of interest are:
> >
> > FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
> > FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
> >
> > This says that the accumulation buckets reset every 6 hours. The
input
> > accums variable is a list of accumulations that are found in the
data.
> This
> > configuration says to use the forecast lead as the accum,
resetting
> every 6
> > hours. This value can be a comma-separated list, so if the files
have a 1
> > hour accumulation as well as the accum that corresponds to the
forecast
> > lead, you could set it to:
> >
> > FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
> >
> > Please let me know if you have any questions.
> >
> > Thanks,
> > George
> >
> > On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >>
> >> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
> >> Transaction: Given to mccabe (George McCabe) by johnhg
> >> Queue: met_help
> >> Subject: METplus pcp_combine questions
> >> Owner: mccabe
> >> Requestors: brian.matilla at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus pcp_combine questions
From: George McCabe
Time: Tue Feb 25 17:21:58 2020
I created a GitHub issue to address this new functionality:
https://github.com/NCAR/METplus/issues/383
On Wed, Feb 26, 2020 at 12:15 AM George McCabe <mccabe at ucar.edu>
wrote:
> Hi Brian,
>
> I thought about it more and it seems like it would be more useful to
have
> a list of strings that can be looped over to provide additional
information
> to use for each run time. A wildcard would require you to use all
ensemble
> members, when maybe you only want to process say the first 3.
Something
> like:
>
> CUSTOM_RUN_STRINGS = ens1, ens2, ens3
>
> Then for each init or valid and forecast lead run time combination,
> METplus would loop over each value in the list and make it available
in the
> filename template functionality so it could be used in many places.
Your
> template would look something like:
>
> ..._INPUT_TEMPLATE = (custom}_{init?fmt=%Y%m%d%H}_f{lead?fmt=%3H}.nc
>
> For init = 20190224_3Z and 3 hr forecast lead, the tool would run 3
times,
> using
>
> ens1_2019022403_f003.nc
> ens2_2019022403_f003.nc
> ens3_2019022403_f003.nc
>
> Ideally I would enhance the begin_end_incr logic to allow strings as
part
> of it, so you could more easily define that list.
>
> The MET engineers are working an issue that has delayed the release
for
> some time. They are intending to get the release out as soon as
possible,
> but the broken logic is imperative to the release. The METplus 3.0
release
> will occur when the MET release is ready. I am not sure that I will
be able
> to get the above functionality implemented and testing in time for
the
> release. If I cannot, then I can let you know when the functionality
is
> available in the develop branch of our GitHub repository and you
could
> easily obtain the changes for your case.
>
> On Tue, Feb 25, 2020 at 11:18 PM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>
>> Hi George,
>>
>> Thanks for the detailed information! I think it would be useful to
>> incorporate some member-based logic into the pcp_combine wrappers.
I’ll see
>> what I can come up with and follow up with you.
>>
>> It’s great to hear that bucketing is included with the newest
version.
>> I’ll give the METplus 3 beta a try since I’m currently playing
around with
>> MET 9.0 and its NEP/NMEP capabilities so it would make sense to
trial both
>> at the same time. I think that having the bucket feature will be of
great
>> help for what we’re trying to accomplish.
>>
>> So more out of curiosity, is there a timetable for when the formal
>> release of MET 9 and METplus 3 will take place?
>>
>> -Brian
>>
>> > On Feb 25, 2020, at 11:50 AM, George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>> >
>> > Hi Brian,
>> >
>> > Regarding a shortcut to process multiple ensembles through
PCPCombine
>> > easily, METplus does not currently support a good way to do this.
It
>> makes
>> > sense to enhance the logic to make this easier. I will have to
think of
>> a
>> > good way to handle it. If you have any suggestions of how you
would
>> like to
>> > be able to configure METplus to do this, please let me know.
Other
>> wrappers
>> > do support wildcard characters to find multiple files, so we
could
>> > potentially use this logic slightly modified here.
>> >
>> > Regarding accumulation buckets, functionality was added to
METplus that
>> > will be found in version 3.0 that can handle this case. It is
available
>> in
>> > 3.0 beta3 if you would like to obtain that version to test.
Please note
>> > that there are a few major changes moving from METplus 2 to 3.
METplus 3
>> > uses Python 3, so you will need to switch to that version of
Python to
>> run.
>> > There are also a good number of configuration variable name
changes.
>> There
>> > is a utility included with the release to make this transition
easier.
>> You
>> > can substitute master_metplus.py with validate_config.py, passing
in the
>> > same list of configuration files you will be running. This script
will
>> > prompt you if you would like the script to automatically replace
each
>> > deprecated configuration variables in your files one-by-one.
>> >
>> > There is an example of using bucket intervals with PCPCombine
here:
>> >
>> >
>> https://github.com/NCAR/METplus/blob/master_v3.0-
beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
>> >
>> > The new variables of interest are:
>> >
>> > FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
>> > FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
>> >
>> > This says that the accumulation buckets reset every 6 hours. The
input
>> > accums variable is a list of accumulations that are found in the
data.
>> This
>> > configuration says to use the forecast lead as the accum,
resetting
>> every 6
>> > hours. This value can be a comma-separated list, so if the files
have a
>> 1
>> > hour accumulation as well as the accum that corresponds to the
forecast
>> > lead, you could set it to:
>> >
>> > FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
>> >
>> > Please let me know if you have any questions.
>> >
>> > Thanks,
>> > George
>> >
>> > On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> > wrote:
>> >
>> >>
>> >> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
>> >> Transaction: Given to mccabe (George McCabe) by johnhg
>> >> Queue: met_help
>> >> Subject: METplus pcp_combine questions
>> >> Owner: mccabe
>> >> Requestors: brian.matilla at noaa.gov
>> >> Status: new
>> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>> >>
>> >>
>> >> This transaction appears to have no content
>> >>
>> >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>>
>>
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94302] METplus pcp_combine questions
From: Brian Matilla - NOAA Affiliate
Time: Fri Feb 28 08:50:33 2020
Hi George,
That idea makes a lot of sense. I thought about this a bit over the
last couple of days and just to piggy back on your thoughts, perhaps
one thing that could be implemented would be a format flag for the run
strings such that if member strings happen to be formatted in such a
way (e.g., 001) then it can be ingested that way (something along the
lines of {member?fmt=%03d} assuming Python syntax for number strings,
or a similar flavor that METplus could support). For instance, the
data I’m working with is located in subdirectories labeled as:
“mem_001” so enhancing the begin_end_incr logic for a situation like
that would be useful. Either way, support for even custom strings
would help a bunch to accommodate any need for utilizing multiple
ensemble members.
Regarding the MET release, I understand the delay. I’m actually
looking to open a separate ticket to address an issue with
ensemble_stat and NMEP in MET 9.0 beta 3, unless I can bring it up
here.
-Brian
> On Feb 25, 2020, at 6:21 PM, George McCabe via RT
<met_help at ucar.edu> wrote:
>
> I created a GitHub issue to address this new functionality:
> https://github.com/NCAR/METplus/issues/383
>
> On Wed, Feb 26, 2020 at 12:15 AM George McCabe <mccabe at ucar.edu>
wrote:
>
>> Hi Brian,
>>
>> I thought about it more and it seems like it would be more useful
to have
>> a list of strings that can be looped over to provide additional
information
>> to use for each run time. A wildcard would require you to use all
ensemble
>> members, when maybe you only want to process say the first 3.
Something
>> like:
>>
>> CUSTOM_RUN_STRINGS = ens1, ens2, ens3
>>
>> Then for each init or valid and forecast lead run time combination,
>> METplus would loop over each value in the list and make it
available in the
>> filename template functionality so it could be used in many places.
Your
>> template would look something like:
>>
>> ..._INPUT_TEMPLATE =
(custom}_{init?fmt=%Y%m%d%H}_f{lead?fmt=%3H}.nc
>>
>> For init = 20190224_3Z and 3 hr forecast lead, the tool would run 3
times,
>> using
>>
>> ens1_2019022403_f003.nc
>> ens2_2019022403_f003.nc
>> ens3_2019022403_f003.nc
>>
>> Ideally I would enhance the begin_end_incr logic to allow strings
as part
>> of it, so you could more easily define that list.
>>
>> The MET engineers are working an issue that has delayed the release
for
>> some time. They are intending to get the release out as soon as
possible,
>> but the broken logic is imperative to the release. The METplus 3.0
release
>> will occur when the MET release is ready. I am not sure that I will
be able
>> to get the above functionality implemented and testing in time for
the
>> release. If I cannot, then I can let you know when the
functionality is
>> available in the develop branch of our GitHub repository and you
could
>> easily obtain the changes for your case.
>>
>> On Tue, Feb 25, 2020 at 11:18 PM Brian Matilla - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>>
>>> Hi George,
>>>
>>> Thanks for the detailed information! I think it would be useful to
>>> incorporate some member-based logic into the pcp_combine wrappers.
I’ll see
>>> what I can come up with and follow up with you.
>>>
>>> It’s great to hear that bucketing is included with the newest
version.
>>> I’ll give the METplus 3 beta a try since I’m currently playing
around with
>>> MET 9.0 and its NEP/NMEP capabilities so it would make sense to
trial both
>>> at the same time. I think that having the bucket feature will be
of great
>>> help for what we’re trying to accomplish.
>>>
>>> So more out of curiosity, is there a timetable for when the formal
>>> release of MET 9 and METplus 3 will take place?
>>>
>>> -Brian
>>>
>>>> On Feb 25, 2020, at 11:50 AM, George McCabe via RT
<met_help at ucar.edu>
>>> wrote:
>>>>
>>>> Hi Brian,
>>>>
>>>> Regarding a shortcut to process multiple ensembles through
PCPCombine
>>>> easily, METplus does not currently support a good way to do this.
It
>>> makes
>>>> sense to enhance the logic to make this easier. I will have to
think of
>>> a
>>>> good way to handle it. If you have any suggestions of how you
would
>>> like to
>>>> be able to configure METplus to do this, please let me know.
Other
>>> wrappers
>>>> do support wildcard characters to find multiple files, so we
could
>>>> potentially use this logic slightly modified here.
>>>>
>>>> Regarding accumulation buckets, functionality was added to
METplus that
>>>> will be found in version 3.0 that can handle this case. It is
available
>>> in
>>>> 3.0 beta3 if you would like to obtain that version to test.
Please note
>>>> that there are a few major changes moving from METplus 2 to 3.
METplus 3
>>>> uses Python 3, so you will need to switch to that version of
Python to
>>> run.
>>>> There are also a good number of configuration variable name
changes.
>>> There
>>>> is a utility included with the release to make this transition
easier.
>>> You
>>>> can substitute master_metplus.py with validate_config.py, passing
in the
>>>> same list of configuration files you will be running. This script
will
>>>> prompt you if you would like the script to automatically replace
each
>>>> deprecated configuration variables in your files one-by-one.
>>>>
>>>> There is an example of using bucket intervals with PCPCombine
here:
>>>>
>>>>
>>> https://github.com/NCAR/METplus/blob/master_v3.0-
beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
>>>>
>>>> The new variables of interest are:
>>>>
>>>> FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
>>>>
>>>> This says that the accumulation buckets reset every 6 hours. The
input
>>>> accums variable is a list of accumulations that are found in the
data.
>>> This
>>>> configuration says to use the forecast lead as the accum,
resetting
>>> every 6
>>>> hours. This value can be a comma-separated list, so if the files
have a
>>> 1
>>>> hour accumulation as well as the accum that corresponds to the
forecast
>>>> lead, you could set it to:
>>>>
>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
>>>>
>>>> Please let me know if you have any questions.
>>>>
>>>> Thanks,
>>>> George
>>>>
>>>> On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT <
>>> met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>>
>>>>> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>> Queue: met_help
>>>>> Subject: METplus pcp_combine questions
>>>>> Owner: mccabe
>>>>> Requestors: brian.matilla at noaa.gov
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>>>>
>>>>>
>>>>> This transaction appears to have no content
>>>>>
>>>>
>>>>
>>>> --
>>>> George McCabe - Software Engineer III
>>>> National Center for Atmospheric Research
>>>> Research Applications Laboratory
>>>> 303-497-2768
>>>> ---
>>>> My working day may not be your working day. Please do not feel
obliged
>>> to
>>>> reply to this email outside of your normal working hours.
>>>>
>>>
>>>
>>>
>>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
------------------------------------------------
Subject: METplus pcp_combine questions
From: George McCabe
Time: Mon Mar 02 09:19:13 2020
Hi Brian,
Supporting the number precision formatting seems to be a good idea to
include in this logic. I will make a note to make sure that is
included.
I would recommend creating a separate ticket for your EnsembleStat and
NMEP issue to keep things organized.
Thanks,
George
On Fri Feb 28 08:50:33 2020, brian.matilla at noaa.gov wrote:
> Hi George,
>
> That idea makes a lot of sense. I thought about this a bit over the
> last couple of days and just to piggy back on your thoughts, perhaps
> one thing that could be implemented would be a format flag for the
run
> strings such that if member strings happen to be formatted in such a
> way (e.g., 001) then it can be ingested that way (something along
the
> lines of {member?fmt=%03d} assuming Python syntax for number
strings,
> or a similar flavor that METplus could support). For instance, the
> data I’m working with is located in subdirectories labeled as:
> “mem_001” so enhancing the begin_end_incr logic for a situation like
> that would be useful. Either way, support for even custom strings
> would help a bunch to accommodate any need for utilizing multiple
> ensemble members.
>
> Regarding the MET release, I understand the delay. I’m actually
> looking to open a separate ticket to address an issue with
> ensemble_stat and NMEP in MET 9.0 beta 3, unless I can bring it up
> here.
>
> -Brian
>
> > On Feb 25, 2020, at 6:21 PM, George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > I created a GitHub issue to address this new functionality:
> > https://github.com/NCAR/METplus/issues/383
> >
> > On Wed, Feb 26, 2020 at 12:15 AM George McCabe <mccabe at ucar.edu>
> > wrote:
> >
> >> Hi Brian,
> >>
> >> I thought about it more and it seems like it would be more useful
to
> >> have
> >> a list of strings that can be looped over to provide additional
> >> information
> >> to use for each run time. A wildcard would require you to use all
> >> ensemble
> >> members, when maybe you only want to process say the first 3.
> >> Something
> >> like:
> >>
> >> CUSTOM_RUN_STRINGS = ens1, ens2, ens3
> >>
> >> Then for each init or valid and forecast lead run time
combination,
> >> METplus would loop over each value in the list and make it
available
> >> in the
> >> filename template functionality so it could be used in many
places.
> >> Your
> >> template would look something like:
> >>
> >> ..._INPUT_TEMPLATE =
(custom}_{init?fmt=%Y%m%d%H}_f{lead?fmt=%3H}.nc
> >>
> >> For init = 20190224_3Z and 3 hr forecast lead, the tool would run
3
> >> times,
> >> using
> >>
> >> ens1_2019022403_f003.nc
> >> ens2_2019022403_f003.nc
> >> ens3_2019022403_f003.nc
> >>
> >> Ideally I would enhance the begin_end_incr logic to allow strings
as
> >> part
> >> of it, so you could more easily define that list.
> >>
> >> The MET engineers are working an issue that has delayed the
release
> >> for
> >> some time. They are intending to get the release out as soon as
> >> possible,
> >> but the broken logic is imperative to the release. The METplus
3.0
> >> release
> >> will occur when the MET release is ready. I am not sure that I
will
> >> be able
> >> to get the above functionality implemented and testing in time
for
> >> the
> >> release. If I cannot, then I can let you know when the
functionality
> >> is
> >> available in the develop branch of our GitHub repository and you
> >> could
> >> easily obtain the changes for your case.
> >>
> >> On Tue, Feb 25, 2020 at 11:18 PM Brian Matilla - NOAA Affiliate
via
> >> RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
> >>>
> >>> Hi George,
> >>>
> >>> Thanks for the detailed information! I think it would be useful
to
> >>> incorporate some member-based logic into the pcp_combine
wrappers.
> >>> I’ll see
> >>> what I can come up with and follow up with you.
> >>>
> >>> It’s great to hear that bucketing is included with the newest
> >>> version.
> >>> I’ll give the METplus 3 beta a try since I’m currently playing
> >>> around with
> >>> MET 9.0 and its NEP/NMEP capabilities so it would make sense to
> >>> trial both
> >>> at the same time. I think that having the bucket feature will be
of
> >>> great
> >>> help for what we’re trying to accomplish.
> >>>
> >>> So more out of curiosity, is there a timetable for when the
formal
> >>> release of MET 9 and METplus 3 will take place?
> >>>
> >>> -Brian
> >>>
> >>>> On Feb 25, 2020, at 11:50 AM, George McCabe via RT
> >>>> <met_help at ucar.edu>
> >>> wrote:
> >>>>
> >>>> Hi Brian,
> >>>>
> >>>> Regarding a shortcut to process multiple ensembles through
> >>>> PCPCombine
> >>>> easily, METplus does not currently support a good way to do
this.
> >>>> It
> >>> makes
> >>>> sense to enhance the logic to make this easier. I will have to
> >>>> think of
> >>> a
> >>>> good way to handle it. If you have any suggestions of how you
> >>>> would
> >>> like to
> >>>> be able to configure METplus to do this, please let me know.
Other
> >>> wrappers
> >>>> do support wildcard characters to find multiple files, so we
could
> >>>> potentially use this logic slightly modified here.
> >>>>
> >>>> Regarding accumulation buckets, functionality was added to
METplus
> >>>> that
> >>>> will be found in version 3.0 that can handle this case. It is
> >>>> available
> >>> in
> >>>> 3.0 beta3 if you would like to obtain that version to test.
Please
> >>>> note
> >>>> that there are a few major changes moving from METplus 2 to 3.
> >>>> METplus 3
> >>>> uses Python 3, so you will need to switch to that version of
> >>>> Python to
> >>> run.
> >>>> There are also a good number of configuration variable name
> >>>> changes.
> >>> There
> >>>> is a utility included with the release to make this transition
> >>>> easier.
> >>> You
> >>>> can substitute master_metplus.py with validate_config.py,
passing
> >>>> in the
> >>>> same list of configuration files you will be running. This
script
> >>>> will
> >>>> prompt you if you would like the script to automatically
replace
> >>>> each
> >>>> deprecated configuration variables in your files one-by-one.
> >>>>
> >>>> There is an example of using bucket intervals with PCPCombine
> >>>> here:
> >>>>
> >>>>
> >>> https://github.com/NCAR/METplus/blob/master_v3.0-
> >>> beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
> >>>>
> >>>> The new variables of interest are:
> >>>>
> >>>> FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
> >>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
> >>>>
> >>>> This says that the accumulation buckets reset every 6 hours.
The
> >>>> input
> >>>> accums variable is a list of accumulations that are found in
the
> >>>> data.
> >>> This
> >>>> configuration says to use the forecast lead as the accum,
> >>>> resetting
> >>> every 6
> >>>> hours. This value can be a comma-separated list, so if the
files
> >>>> have a
> >>> 1
> >>>> hour accumulation as well as the accum that corresponds to the
> >>>> forecast
> >>>> lead, you could set it to:
> >>>>
> >>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
> >>>>
> >>>> Please let me know if you have any questions.
> >>>>
> >>>> Thanks,
> >>>> George
> >>>>
> >>>> On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT <
> >>> met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>>
> >>>>> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
> >>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>> Queue: met_help
> >>>>> Subject: METplus pcp_combine questions
> >>>>> Owner: mccabe
> >>>>> Requestors: brian.matilla at noaa.gov
> >>>>> Status: new
> >>>>> Ticket <URL:
> >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
> >>>>>
> >>>>>
> >>>>> This transaction appears to have no content
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> George McCabe - Software Engineer III
> >>>> National Center for Atmospheric Research
> >>>> Research Applications Laboratory
> >>>> 303-497-2768
> >>>> ---
> >>>> My working day may not be your working day. Please do not feel
> >>>> obliged
> >>> to
> >>>> reply to this email outside of your normal working hours.
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
> >> obliged to
> >> reply to this email outside of your normal working hours.
> >>
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
> > obliged to
> > reply to this email outside of your normal working hours.
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94302] METplus pcp_combine questions
From: Brian Matilla - NOAA Affiliate
Time: Mon Mar 02 15:57:14 2020
Hi George,
Thanks for that! And yes, I’ll go ahead and create a new ticket to
address the NMEP issue. For now, I think we can close this ticket.
-Brian
> On Mar 2, 2020, at 10:19 AM, George McCabe via RT
<met_help at ucar.edu> wrote:
>
> Hi Brian,
>
> Supporting the number precision formatting seems to be a good idea
to include in this logic. I will make a note to make sure that is
included.
>
> I would recommend creating a separate ticket for your EnsembleStat
and NMEP issue to keep things organized.
>
> Thanks,
> George
>
> On Fri Feb 28 08:50:33 2020, brian.matilla at noaa.gov wrote:
>> Hi George,
>>
>> That idea makes a lot of sense. I thought about this a bit over the
>> last couple of days and just to piggy back on your thoughts,
perhaps
>> one thing that could be implemented would be a format flag for the
run
>> strings such that if member strings happen to be formatted in such
a
>> way (e.g., 001) then it can be ingested that way (something along
the
>> lines of {member?fmt=%03d} assuming Python syntax for number
strings,
>> or a similar flavor that METplus could support). For instance, the
>> data I’m working with is located in subdirectories labeled as:
>> “mem_001” so enhancing the begin_end_incr logic for a situation
like
>> that would be useful. Either way, support for even custom strings
>> would help a bunch to accommodate any need for utilizing multiple
>> ensemble members.
>>
>> Regarding the MET release, I understand the delay. I’m actually
>> looking to open a separate ticket to address an issue with
>> ensemble_stat and NMEP in MET 9.0 beta 3, unless I can bring it up
>> here.
>>
>> -Brian
>>
>>> On Feb 25, 2020, at 6:21 PM, George McCabe via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>> I created a GitHub issue to address this new functionality:
>>> https://github.com/NCAR/METplus/issues/383
>>>
>>> On Wed, Feb 26, 2020 at 12:15 AM George McCabe <mccabe at ucar.edu>
>>> wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> I thought about it more and it seems like it would be more useful
to
>>>> have
>>>> a list of strings that can be looped over to provide additional
>>>> information
>>>> to use for each run time. A wildcard would require you to use all
>>>> ensemble
>>>> members, when maybe you only want to process say the first 3.
>>>> Something
>>>> like:
>>>>
>>>> CUSTOM_RUN_STRINGS = ens1, ens2, ens3
>>>>
>>>> Then for each init or valid and forecast lead run time
combination,
>>>> METplus would loop over each value in the list and make it
available
>>>> in the
>>>> filename template functionality so it could be used in many
places.
>>>> Your
>>>> template would look something like:
>>>>
>>>> ..._INPUT_TEMPLATE =
(custom}_{init?fmt=%Y%m%d%H}_f{lead?fmt=%3H}.nc
>>>>
>>>> For init = 20190224_3Z and 3 hr forecast lead, the tool would run
3
>>>> times,
>>>> using
>>>>
>>>> ens1_2019022403_f003.nc
>>>> ens2_2019022403_f003.nc
>>>> ens3_2019022403_f003.nc
>>>>
>>>> Ideally I would enhance the begin_end_incr logic to allow strings
as
>>>> part
>>>> of it, so you could more easily define that list.
>>>>
>>>> The MET engineers are working an issue that has delayed the
release
>>>> for
>>>> some time. They are intending to get the release out as soon as
>>>> possible,
>>>> but the broken logic is imperative to the release. The METplus
3.0
>>>> release
>>>> will occur when the MET release is ready. I am not sure that I
will
>>>> be able
>>>> to get the above functionality implemented and testing in time
for
>>>> the
>>>> release. If I cannot, then I can let you know when the
functionality
>>>> is
>>>> available in the develop branch of our GitHub repository and you
>>>> could
>>>> easily obtain the changes for your case.
>>>>
>>>> On Tue, Feb 25, 2020 at 11:18 PM Brian Matilla - NOAA Affiliate
via
>>>> RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>>>>
>>>>> Hi George,
>>>>>
>>>>> Thanks for the detailed information! I think it would be useful
to
>>>>> incorporate some member-based logic into the pcp_combine
wrappers.
>>>>> I’ll see
>>>>> what I can come up with and follow up with you.
>>>>>
>>>>> It’s great to hear that bucketing is included with the newest
>>>>> version.
>>>>> I’ll give the METplus 3 beta a try since I’m currently playing
>>>>> around with
>>>>> MET 9.0 and its NEP/NMEP capabilities so it would make sense to
>>>>> trial both
>>>>> at the same time. I think that having the bucket feature will be
of
>>>>> great
>>>>> help for what we’re trying to accomplish.
>>>>>
>>>>> So more out of curiosity, is there a timetable for when the
formal
>>>>> release of MET 9 and METplus 3 will take place?
>>>>>
>>>>> -Brian
>>>>>
>>>>>> On Feb 25, 2020, at 11:50 AM, George McCabe via RT
>>>>>> <met_help at ucar.edu>
>>>>> wrote:
>>>>>>
>>>>>> Hi Brian,
>>>>>>
>>>>>> Regarding a shortcut to process multiple ensembles through
>>>>>> PCPCombine
>>>>>> easily, METplus does not currently support a good way to do
this.
>>>>>> It
>>>>> makes
>>>>>> sense to enhance the logic to make this easier. I will have to
>>>>>> think of
>>>>> a
>>>>>> good way to handle it. If you have any suggestions of how you
>>>>>> would
>>>>> like to
>>>>>> be able to configure METplus to do this, please let me know.
Other
>>>>> wrappers
>>>>>> do support wildcard characters to find multiple files, so we
could
>>>>>> potentially use this logic slightly modified here.
>>>>>>
>>>>>> Regarding accumulation buckets, functionality was added to
METplus
>>>>>> that
>>>>>> will be found in version 3.0 that can handle this case. It is
>>>>>> available
>>>>> in
>>>>>> 3.0 beta3 if you would like to obtain that version to test.
Please
>>>>>> note
>>>>>> that there are a few major changes moving from METplus 2 to 3.
>>>>>> METplus 3
>>>>>> uses Python 3, so you will need to switch to that version of
>>>>>> Python to
>>>>> run.
>>>>>> There are also a good number of configuration variable name
>>>>>> changes.
>>>>> There
>>>>>> is a utility included with the release to make this transition
>>>>>> easier.
>>>>> You
>>>>>> can substitute master_metplus.py with validate_config.py,
passing
>>>>>> in the
>>>>>> same list of configuration files you will be running. This
script
>>>>>> will
>>>>>> prompt you if you would like the script to automatically
replace
>>>>>> each
>>>>>> deprecated configuration variables in your files one-by-one.
>>>>>>
>>>>>> There is an example of using bucket intervals with PCPCombine
>>>>>> here:
>>>>>>
>>>>>>
>>>>> https://github.com/NCAR/METplus/blob/master_v3.0-
>>>>> beta3/parm/use_cases/met_tool_wrapper/PCPCombine/pcp_bucket.conf
>>>>>>
>>>>>> The new variables of interest are:
>>>>>>
>>>>>> FCST_PCP_COMBINE_BUCKET_INTERVAL = 6H
>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}
>>>>>>
>>>>>> This says that the accumulation buckets reset every 6 hours.
The
>>>>>> input
>>>>>> accums variable is a list of accumulations that are found in
the
>>>>>> data.
>>>>> This
>>>>>> configuration says to use the forecast lead as the accum,
>>>>>> resetting
>>>>> every 6
>>>>>> hours. This value can be a comma-separated list, so if the
files
>>>>>> have a
>>>>> 1
>>>>>> hour accumulation as well as the accum that corresponds to the
>>>>>> forecast
>>>>>> lead, you could set it to:
>>>>>>
>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = {lead}, 1
>>>>>>
>>>>>> Please let me know if you have any questions.
>>>>>>
>>>>>> Thanks,
>>>>>> George
>>>>>>
>>>>>> On Tue, Feb 25, 2020 at 9:42 AM John Halley Gotway via RT <
>>>>> met_help at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Tue Feb 25 09:42:24 2020: Request 94302 was acted upon.
>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>> Queue: met_help
>>>>>>> Subject: METplus pcp_combine questions
>>>>>>> Owner: mccabe
>>>>>>> Requestors: brian.matilla at noaa.gov
>>>>>>> Status: new
>>>>>>> Ticket <URL:
>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94302 >
>>>>>>>
>>>>>>>
>>>>>>> This transaction appears to have no content
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> George McCabe - Software Engineer III
>>>>>> National Center for Atmospheric Research
>>>>>> Research Applications Laboratory
>>>>>> 303-497-2768
>>>>>> ---
>>>>>> My working day may not be your working day. Please do not feel
>>>>>> obliged
>>>>> to
>>>>>> reply to this email outside of your normal working hours.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> George McCabe - Software Engineer III
>>>> National Center for Atmospheric Research
>>>> Research Applications Laboratory
>>>> 303-497-2768
>>>> ---
>>>> My working day may not be your working day. Please do not feel
>>>> obliged to
>>>> reply to this email outside of your normal working hours.
>>>>
>>>
>>>
>>> --
>>> George McCabe - Software Engineer III
>>> National Center for Atmospheric Research
>>> Research Applications Laboratory
>>> 303-497-2768
>>> ---
>>> My working day may not be your working day. Please do not feel
>>> obliged to
>>> reply to this email outside of your normal working hours.
>>>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list