[Met_help] [rt.rap.ucar.edu #91715] History for METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
George McCabe via RT
met_help at ucar.edu
Wed Mar 11 09:52:18 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. I'm trying to run METplus' PointStat - want to sum up 24h
1-hourly precip analysis (pcprtma) and verify it against daily gauges.
There are some issues I couldn't figure out myself and hope you can help
me with. Attached are
pcprtma_24h.ksh: run script
metplus_final.conf
master_metplus.log.20190827
pcprtma_24h.conf: my config for pcprtma
PointStatConfig_gauges: my general PointStatConfig
test_add.sh, test_sum.sh: "manual" run of MET8.1's pcp_combine on
the pcprtma data to rule out data problem
Issues:
1) Right off the bat I could not sum up the 1-hourly 'pcprtma' into 24h
totals: in the attached log file, there is "metplus.PointStat
(point_stat_wrapper.py:174) ERROR: Could not find file in
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
matching template pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h". For
a while I wondered whether the problem was with my pcprtma data, so I
tried out MET8.1's pcp_combine's 'add' and 'sum' on the data (attached
test_add.sh and test_sum.sh) and the data seem to be OK, so most likely
it's error in my config files
2) Mask files: my pcprtma_24h.conf was originally copied over from
parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see that in
this config file, for POINT_STAT_POLY there is
# List of full path to poly masking files. NOTE: Only short lists of poly
# files work (those that fit on one line), a long list will result in an
# environment variable that is too long, resulting in an error. For long
# lists of poly masking files (i.e. all the mask files in the NCEP_mask
# directory), define these in the MET point_stat configuration file.
I wanted to just test it out first so I pointed to only one mask file,
the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS RFCs).
Eventually I'd like to have 13 masked regions for this. How do I
"define these in the MET point_stat configuration file"? Currently my
PointStatConfig_gauges (cribbed from
parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has this:
poly = ${POINT_STAT_POLY};
Do I change this to
poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
..."${MASKDIR}/WGRFC.nc"; ?
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
3) This is more MET than METplus: in the attached test_sum.sh, I used
'pcp_combine -sum' to sum up 12 1-h precip into a 12h total, and all the
hourly files are in /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
What I wanted to do initially was to sum up 24 1-h precip into a 24h
(12Z-12Z) total and the hourly files are in /gpfs/.../pcprtma.20190824
and /gpfs/.../pcprtma.20190825. Could you tell me how to make the
-pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
I've been running point_stat in MET but have not run it in METplus
before, other than the use case
parm/use_cases/grid_to_obs/examples/conus_surface.conf. Haven't been
able to figure things out myself in the past few weeks. If you happen
to see anything else wrong in my set-up I would appreciate your pointing
them out too.
Thank you -
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: George McCabe
Time: Tue Aug 27 16:26:11 2019
Hi Ying,
(John, see #3)
1) It looks like your input dir for PcpCombine is slightly wrong. You
have:
FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
and I think it should be
FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
That is the directory you are using for the -add command in your
test_add.sh script. The current version of METplus does not list the
directory and template used in the logs when PcpCombine fails, but
this
will be in the next version.
2) You can change this value to a list of mask files, but you will
need to
add []s around the values, i.e.
poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
..."${MASKDIR}/WGRFC.nc" ];
However, you should also be able to keep the MET config file as is and
set
these values in your METplus config file like this:
POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
..."${MASKDIR}/WGRFC.nc" ]
I don't think passing in []s in the variable has been tested, but in
theory
it should work. Let me know it if does not.
3) This question may be better suited for John. I believe the -pcpdir
needs
to specify the directory containing the files, but I am not sure if
you are
able to specify 2 directories to look in. You could try using the *
character, i.e. gpfs/.../pcprtma.*
If you have many other directories besides those 2 dates, PcpCombine
would
look in all of those directories, so it would take a very long time to
run.
I'm CC'ing John in case he has more input related to #3. Let me know
if my
suggestions don't work or if you have any other questions.
Thanks,
George
On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: METplus PointStat; MET pcp_combine -sum (wildcard in
-pcpdir
> argument?)
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>
>
> Hello. I'm trying to run METplus' PointStat - want to sum up 24h
> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
> There are some issues I couldn't figure out myself and hope you can
help
> me with. Attached are
>
> pcprtma_24h.ksh: run script
>
> metplus_final.conf
>
> master_metplus.log.20190827
>
> pcprtma_24h.conf: my config for pcprtma
>
> PointStatConfig_gauges: my general PointStatConfig
> test_add.sh, test_sum.sh: "manual" run of MET8.1's pcp_combine
on
> the pcprtma data to rule out data problem
>
>
> Issues:
>
> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma' into
24h
> totals: in the attached log file, there is "metplus.PointStat
> (point_stat_wrapper.py:174) ERROR: Could not find file in
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
> matching template pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
For
> a while I wondered whether the problem was with my pcprtma data, so
I
> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
> test_add.sh and test_sum.sh) and the data seem to be OK, so most
likely
> it's error in my config files
>
> 2) Mask files: my pcprtma_24h.conf was originally copied over from
> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see that
in
> this config file, for POINT_STAT_POLY there is
>
> # List of full path to poly masking files. NOTE: Only short lists
of poly
> # files work (those that fit on one line), a long list will result
in an
> # environment variable that is too long, resulting in an error. For
long
> # lists of poly masking files (i.e. all the mask files in the
NCEP_mask
> # directory), define these in the MET point_stat configuration file.
>
> I wanted to just test it out first so I pointed to only one mask
file,
> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS RFCs).
> Eventually I'd like to have 13 masked regions for this. How do I
> "define these in the MET point_stat configuration file"? Currently
my
> PointStatConfig_gauges (cribbed from
> parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has
this:
>
> poly = ${POINT_STAT_POLY};
>
> Do I change this to
>
> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> ..."${MASKDIR}/WGRFC.nc"; ?
>
>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
>
> 3) This is more MET than METplus: in the attached test_sum.sh, I
used
> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total, and all
the
> hourly files are in
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
> What I wanted to do initially was to sum up 24 1-h precip into a 24h
> (12Z-12Z) total and the hourly files are in
/gpfs/.../pcprtma.20190824
> and /gpfs/.../pcprtma.20190825. Could you tell me how to make the
> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
>
> I've been running point_stat in MET but have not run it in METplus
> before, other than the use case
> parm/use_cases/grid_to_obs/examples/conus_surface.conf. Haven't
been
> able to figure things out myself in the past few weeks. If you
happen
> to see anything else wrong in my set-up I would appreciate your
pointing
> them out too.
>
> Thank you -
>
> Ying
>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91715] METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: Ying Lin
Time: Wed Aug 28 09:11:03 2019
Thanks George! I've put the masks into the pcprtma_24h.config. If it
doesn't work then I'll try the "higher level" PointStatConfig. If
either
works that'll be really useful.
Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
/gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
transitioned to the old phase2 (tide/gyre) machine to the phase3/dell
a
week ago. When I ran the test_add.sh it was three weeks ago and the
RTMA on the Dell was still the 'para'. It's now officially 'prod'.
Anyway I ran the script again with the new multiple-mask set up in the
config and there is still a problem with the pcp_combine_wrapper.py:
08/28 14:41:42.152 metplus.PcpCombine (pcp_combine_wrapper.py:547)
ERROR: pcp_combine could not generate command for init 201908241200
and
forecast lead 24
Not sure how to go from here. Is there a way to find out why
"pcp_combine could not generate command"?
Thanks again -
Ying
On 8/27/19 6:26 PM, George McCabe via RT wrote:
> Hi Ying,
>
> (John, see #3)
>
> 1) It looks like your input dir for PcpCombine is slightly wrong.
You have:
>
> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
> and I think it should be
> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
>
> That is the directory you are using for the -add command in your
> test_add.sh script. The current version of METplus does not list the
> directory and template used in the logs when PcpCombine fails, but
this
> will be in the next version.
>
> 2) You can change this value to a list of mask files, but you will
need to
> add []s around the values, i.e.
>
> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> ..."${MASKDIR}/WGRFC.nc" ];
>
> However, you should also be able to keep the MET config file as is
and set
> these values in your METplus config file like this:
>
> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> ..."${MASKDIR}/WGRFC.nc" ]
>
> I don't think passing in []s in the variable has been tested, but in
theory
> it should work. Let me know it if does not.
>
> 3) This question may be better suited for John. I believe the
-pcpdir needs
> to specify the directory containing the files, but I am not sure if
you are
> able to specify 2 directories to look in. You could try using the *
> character, i.e. gpfs/.../pcprtma.*
> If you have many other directories besides those 2 dates, PcpCombine
would
> look in all of those directories, so it would take a very long time
to run.
>
> I'm CC'ing John in case he has more input related to #3. Let me know
if my
> suggestions don't work or if you have any other questions.
>
> Thanks,
> George
>
>
> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: METplus PointStat; MET pcp_combine -sum (wildcard in
-pcpdir
>> argument?)
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>
>>
>> Hello. I'm trying to run METplus' PointStat - want to sum up 24h
>> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
>> There are some issues I couldn't figure out myself and hope you can
help
>> me with. Attached are
>>
>> pcprtma_24h.ksh: run script
>>
>> metplus_final.conf
>>
>> master_metplus.log.20190827
>>
>> pcprtma_24h.conf: my config for pcprtma
>>
>> PointStatConfig_gauges: my general PointStatConfig
>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
pcp_combine on
>> the pcprtma data to rule out data problem
>>
>>
>> Issues:
>>
>> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma' into
24h
>> totals: in the attached log file, there is "metplus.PointStat
>> (point_stat_wrapper.py:174) ERROR: Could not find file in
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
>> matching template pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
For
>> a while I wondered whether the problem was with my pcprtma data, so
I
>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
>> test_add.sh and test_sum.sh) and the data seem to be OK, so most
likely
>> it's error in my config files
>>
>> 2) Mask files: my pcprtma_24h.conf was originally copied over from
>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see
that in
>> this config file, for POINT_STAT_POLY there is
>>
>> # List of full path to poly masking files. NOTE: Only short lists
of poly
>> # files work (those that fit on one line), a long list will result
in an
>> # environment variable that is too long, resulting in an error. For
long
>> # lists of poly masking files (i.e. all the mask files in the
NCEP_mask
>> # directory), define these in the MET point_stat configuration
file.
>>
>> I wanted to just test it out first so I pointed to only one mask
file,
>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
>> Eventually I'd like to have 13 masked regions for this. How do I
>> "define these in the MET point_stat configuration file"? Currently
my
>> PointStatConfig_gauges (cribbed from
>> parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc)
has this:
>>
>> poly = ${POINT_STAT_POLY};
>>
>> Do I change this to
>>
>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>> ..."${MASKDIR}/WGRFC.nc"; ?
>>
>>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
>>
>> 3) This is more MET than METplus: in the attached test_sum.sh, I
used
>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total, and
all the
>> hourly files are in
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
>> What I wanted to do initially was to sum up 24 1-h precip into a
24h
>> (12Z-12Z) total and the hourly files are in
/gpfs/.../pcprtma.20190824
>> and /gpfs/.../pcprtma.20190825. Could you tell me how to make the
>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
>>
>> I've been running point_stat in MET but have not run it in
METplus
>> before, other than the use case
>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. Haven't
been
>> able to figure things out myself in the past few weeks. If you
happen
>> to see anything else wrong in my set-up I would appreciate your
pointing
>> them out too.
>>
>> Thank you -
>>
>> Ying
>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: George McCabe
Time: Wed Aug 28 09:28:35 2019
Hi Ying,
I noticed that one of your variables is not all upper case:
# Accumulation interval available in forecast data
fcst_PCP_COMBINE_INPUT_LEVEL = 01
I don't think that would cause PcpCombine wrapper to not find files,
but it
may help. Could you send me the directory listing for
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
and
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
When PcpCombine wrapper cannot build an accumulation, the first place
I
look is the input directory and template to make sure all of the files
it
will need are available. Often times there is a missing file so it
can't
build a 24 hour accumulation. Besides that, the logging doesn't give
you
much insight as to where it failed to find the required data. I can
add
more debugging information to wrapper for this processing, but it will
create a lot of extra logging information.
On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>
> Thanks George! I've put the masks into the pcprtma_24h.config. If
it
> doesn't work then I'll try the "higher level" PointStatConfig. If
either
> works that'll be really useful.
>
> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell a
> week ago. When I ran the test_add.sh it was three weeks ago and the
> RTMA on the Dell was still the 'para'. It's now officially 'prod'.
> Anyway I ran the script again with the new multiple-mask set up in
the
> config and there is still a problem with the pcp_combine_wrapper.py:
>
> 08/28 14:41:42.152 metplus.PcpCombine (pcp_combine_wrapper.py:547)
> ERROR: pcp_combine could not generate command for init 201908241200
and
> forecast lead 24
>
> Not sure how to go from here. Is there a way to find out why
> "pcp_combine could not generate command"?
>
> Thanks again -
>
> Ying
>
> On 8/27/19 6:26 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > (John, see #3)
> >
> > 1) It looks like your input dir for PcpCombine is slightly wrong.
You
> have:
> >
> > FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
> > and I think it should be
> > FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
> >
> > That is the directory you are using for the -add command in your
> > test_add.sh script. The current version of METplus does not list
the
> > directory and template used in the logs when PcpCombine fails, but
this
> > will be in the next version.
> >
> > 2) You can change this value to a list of mask files, but you will
need
> to
> > add []s around the values, i.e.
> >
> > poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> > ..."${MASKDIR}/WGRFC.nc" ];
> >
> > However, you should also be able to keep the MET config file as is
and
> set
> > these values in your METplus config file like this:
> >
> > POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> > ..."${MASKDIR}/WGRFC.nc" ]
> >
> > I don't think passing in []s in the variable has been tested, but
in
> theory
> > it should work. Let me know it if does not.
> >
> > 3) This question may be better suited for John. I believe the
-pcpdir
> needs
> > to specify the directory containing the files, but I am not sure
if you
> are
> > able to specify 2 directories to look in. You could try using the
*
> > character, i.e. gpfs/.../pcprtma.*
> > If you have many other directories besides those 2 dates,
PcpCombine
> would
> > look in all of those directories, so it would take a very long
time to
> run.
> >
> > I'm CC'ing John in case he has more input related to #3. Let me
know if
> my
> > suggestions don't work or if you have any other questions.
> >
> > Thanks,
> > George
> >
> >
> > On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
> >> Transaction: Ticket created by ying.lin at noaa.gov
> >> Queue: met_help
> >> Subject: METplus PointStat; MET pcp_combine -sum (wildcard
in
> -pcpdir
> >> argument?)
> >> Owner: Nobody
> >> Requestors: ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
> >
> >>
> >>
> >> Hello. I'm trying to run METplus' PointStat - want to sum up 24h
> >> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
> >> There are some issues I couldn't figure out myself and hope you
can help
> >> me with. Attached are
> >>
> >> pcprtma_24h.ksh: run script
> >>
> >> metplus_final.conf
> >>
> >> master_metplus.log.20190827
> >>
> >> pcprtma_24h.conf: my config for pcprtma
> >>
> >> PointStatConfig_gauges: my general PointStatConfig
> >> test_add.sh, test_sum.sh: "manual" run of MET8.1's
pcp_combine on
> >> the pcprtma data to rule out data problem
> >>
> >>
> >> Issues:
> >>
> >> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma'
into 24h
> >> totals: in the attached log file, there is "metplus.PointStat
> >> (point_stat_wrapper.py:174) ERROR: Could not find file in
> >>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
> >> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h". For
> >> a while I wondered whether the problem was with my pcprtma data,
so I
> >> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
> >> test_add.sh and test_sum.sh) and the data seem to be OK, so most
likely
> >> it's error in my config files
> >>
> >> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
> >> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see
that in
> >> this config file, for POINT_STAT_POLY there is
> >>
> >> # List of full path to poly masking files. NOTE: Only short
lists of
> poly
> >> # files work (those that fit on one line), a long list will
result in an
> >> # environment variable that is too long, resulting in an error.
For long
> >> # lists of poly masking files (i.e. all the mask files in the
NCEP_mask
> >> # directory), define these in the MET point_stat configuration
file.
> >>
> >> I wanted to just test it out first so I pointed to only one mask
file,
> >> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
> >> Eventually I'd like to have 13 masked regions for this. How do I
> >> "define these in the MET point_stat configuration file"?
Currently my
> >> PointStatConfig_gauges (cribbed from
> >> parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc)
has
> this:
> >>
> >> poly = ${POINT_STAT_POLY};
> >>
> >> Do I change this to
> >>
> >> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> >> ..."${MASKDIR}/WGRFC.nc"; ?
> >>
> >>
>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
> >>
> >> 3) This is more MET than METplus: in the attached test_sum.sh, I
used
> >> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total, and
all the
> >> hourly files are in
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
> >> What I wanted to do initially was to sum up 24 1-h precip into a
24h
> >> (12Z-12Z) total and the hourly files are in
/gpfs/.../pcprtma.20190824
> >> and /gpfs/.../pcprtma.20190825. Could you tell me how to make
the
> >> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
> >>
> >> I've been running point_stat in MET but have not run it in
METplus
> >> before, other than the use case
> >> parm/use_cases/grid_to_obs/examples/conus_surface.conf. Haven't
been
> >> able to figure things out myself in the past few weeks. If you
happen
> >> to see anything else wrong in my set-up I would appreciate your
pointing
> >> them out too.
> >>
> >> Thank you -
> >>
> >> Ying
> >>
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: Ying Lin
Time: Wed Aug 28 10:38:36 2019
Hi George,
Thanks for catching the *fcst*_PCP_COMBINE_INPUT_LEVEL! I fixed
that, it didn't make a difference though.
Here's a listing of the two pcprtma directories:
m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
pcprtma.2019082400.gif pcprtma.2019082417.gif
pcprtma2.2019082410.grb2
pcprtma.2019082401.gif pcprtma.2019082418.gif
pcprtma2.2019082411.grb2
pcprtma.2019082402.gif pcprtma.2019082419.gif
pcprtma2.2019082412.grb2
pcprtma.2019082403.gif pcprtma.2019082420.gif
pcprtma2.2019082413.grb2
pcprtma.2019082404.gif pcprtma.2019082421.gif
pcprtma2.2019082414.grb2
pcprtma.2019082405.gif pcprtma.2019082422.gif
pcprtma2.2019082415.grb2
pcprtma.2019082406.gif pcprtma.2019082423.gif
pcprtma2.2019082416.grb2
pcprtma.2019082407.gif pcprtma2.2019082400.grb2
pcprtma2.2019082417.grb2
pcprtma.2019082408.gif pcprtma2.2019082401.grb2
pcprtma2.2019082418.grb2
pcprtma.2019082409.gif pcprtma2.2019082402.grb2
pcprtma2.2019082419.grb2
pcprtma.2019082410.gif pcprtma2.2019082403.grb2
pcprtma2.2019082420.grb2
pcprtma.2019082411.gif pcprtma2.2019082404.grb2
pcprtma2.2019082421.grb2
pcprtma.2019082412.gif pcprtma2.2019082405.grb2
pcprtma2.2019082422.grb2
pcprtma.2019082413.gif pcprtma2.2019082406.grb2
pcprtma2.2019082423.grb2
pcprtma.2019082414.gif pcprtma2.2019082407.grb2 wmo
pcprtma.2019082415.gif pcprtma2.2019082408.grb2
pcprtma.2019082416.gif pcprtma2.2019082409.grb2
m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
pcprtma.2019082500.gif pcprtma.2019082517.gif
pcprtma2.2019082510.grb2
pcprtma.2019082501.gif pcprtma.2019082518.gif
pcprtma2.2019082511.grb2
pcprtma.2019082502.gif pcprtma.2019082519.gif
pcprtma2.2019082512.grb2
pcprtma.2019082503.gif pcprtma.2019082520.gif
pcprtma2.2019082513.grb2
pcprtma.2019082504.gif pcprtma.2019082521.gif
pcprtma2.2019082514.grb2
pcprtma.2019082505.gif pcprtma.2019082522.gif
pcprtma2.2019082515.grb2
pcprtma.2019082506.gif pcprtma.2019082523.gif
pcprtma2.2019082516.grb2
pcprtma.2019082507.gif pcprtma2.2019082500.grb2
pcprtma2.2019082517.grb2
pcprtma.2019082508.gif pcprtma2.2019082501.grb2
pcprtma2.2019082518.grb2
pcprtma.2019082509.gif pcprtma2.2019082502.grb2
pcprtma2.2019082519.grb2
pcprtma.2019082510.gif pcprtma2.2019082503.grb2
pcprtma2.2019082520.grb2
pcprtma.2019082511.gif pcprtma2.2019082504.grb2
pcprtma2.2019082521.grb2
pcprtma.2019082512.gif pcprtma2.2019082505.grb2
pcprtma2.2019082522.grb2
pcprtma.2019082513.gif pcprtma2.2019082506.grb2
pcprtma2.2019082523.grb2
pcprtma.2019082514.gif pcprtma2.2019082507.grb2 wmo
pcprtma.2019082515.gif pcprtma2.2019082508.grb2
pcprtma.2019082516.gif pcprtma2.2019082509.grb2
I also placed a copy of all the *.grb2 files from above in
ftp://ftp.emc.ncep.noaa.gov/mmb/precip/pcprtma.0824-25.tar.gz, in case
there might be something odd in the grib header. There doesn't seem
to
have been a missing file though.
Thanks again -
Ying
On 8/28/19 11:28 AM, George McCabe via RT wrote:
> Hi Ying,
>
> I noticed that one of your variables is not all upper case:
>
> # Accumulation interval available in forecast data
> fcst_PCP_COMBINE_INPUT_LEVEL = 01
>
> I don't think that would cause PcpCombine wrapper to not find files,
but it
> may help. Could you send me the directory listing for
>
> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
> and
> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
>
> When PcpCombine wrapper cannot build an accumulation, the first
place I
> look is the input directory and template to make sure all of the
files it
> will need are available. Often times there is a missing file so it
can't
> build a 24 hour accumulation. Besides that, the logging doesn't give
you
> much insight as to where it failed to find the required data. I can
add
> more debugging information to wrapper for this processing, but it
will
> create a lot of extra logging information.
>
> On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>
>> Thanks George! I've put the masks into the pcprtma_24h.config. If
it
>> doesn't work then I'll try the "higher level" PointStatConfig. If
either
>> works that'll be really useful.
>>
>> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
>> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
>> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell a
>> week ago. When I ran the test_add.sh it was three weeks ago and
the
>> RTMA on the Dell was still the 'para'. It's now officially 'prod'.
>> Anyway I ran the script again with the new multiple-mask set up in
the
>> config and there is still a problem with the
pcp_combine_wrapper.py:
>>
>> 08/28 14:41:42.152 metplus.PcpCombine (pcp_combine_wrapper.py:547)
>> ERROR: pcp_combine could not generate command for init 201908241200
and
>> forecast lead 24
>>
>> Not sure how to go from here. Is there a way to find out why
>> "pcp_combine could not generate command"?
>>
>> Thanks again -
>>
>> Ying
>>
>> On 8/27/19 6:26 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> (John, see #3)
>>>
>>> 1) It looks like your input dir for PcpCombine is slightly wrong.
You
>> have:
>>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
>>> and I think it should be
>>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
>>>
>>> That is the directory you are using for the -add command in your
>>> test_add.sh script. The current version of METplus does not list
the
>>> directory and template used in the logs when PcpCombine fails, but
this
>>> will be in the next version.
>>>
>>> 2) You can change this value to a list of mask files, but you will
need
>> to
>>> add []s around the values, i.e.
>>>
>>> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>> ..."${MASKDIR}/WGRFC.nc" ];
>>>
>>> However, you should also be able to keep the MET config file as is
and
>> set
>>> these values in your METplus config file like this:
>>>
>>> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>> ..."${MASKDIR}/WGRFC.nc" ]
>>>
>>> I don't think passing in []s in the variable has been tested, but
in
>> theory
>>> it should work. Let me know it if does not.
>>>
>>> 3) This question may be better suited for John. I believe the
-pcpdir
>> needs
>>> to specify the directory containing the files, but I am not sure
if you
>> are
>>> able to specify 2 directories to look in. You could try using the
*
>>> character, i.e. gpfs/.../pcprtma.*
>>> If you have many other directories besides those 2 dates,
PcpCombine
>> would
>>> look in all of those directories, so it would take a very long
time to
>> run.
>>> I'm CC'ing John in case he has more input related to #3. Let me
know if
>> my
>>> suggestions don't work or if you have any other questions.
>>>
>>> Thanks,
>>> George
>>>
>>>
>>> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>> Queue: met_help
>>>> Subject: METplus PointStat; MET pcp_combine -sum (wildcard
in
>> -pcpdir
>>>> argument?)
>>>> Owner: Nobody
>>>> Requestors: ying.lin at noaa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
>>>>
>>>> Hello. I'm trying to run METplus' PointStat - want to sum up 24h
>>>> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
>>>> There are some issues I couldn't figure out myself and hope you
can help
>>>> me with. Attached are
>>>>
>>>> pcprtma_24h.ksh: run script
>>>>
>>>> metplus_final.conf
>>>>
>>>> master_metplus.log.20190827
>>>>
>>>> pcprtma_24h.conf: my config for pcprtma
>>>>
>>>> PointStatConfig_gauges: my general PointStatConfig
>>>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
pcp_combine on
>>>> the pcprtma data to rule out data problem
>>>>
>>>>
>>>> Issues:
>>>>
>>>> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma'
into 24h
>>>> totals: in the attached log file, there is "metplus.PointStat
>>>> (point_stat_wrapper.py:174) ERROR: Could not find file in
>>>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
>>>> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h". For
>>>> a while I wondered whether the problem was with my pcprtma data,
so I
>>>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
>>>> test_add.sh and test_sum.sh) and the data seem to be OK, so most
likely
>>>> it's error in my config files
>>>>
>>>> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see
that in
>>>> this config file, for POINT_STAT_POLY there is
>>>>
>>>> # List of full path to poly masking files. NOTE: Only short
lists of
>> poly
>>>> # files work (those that fit on one line), a long list will
result in an
>>>> # environment variable that is too long, resulting in an error.
For long
>>>> # lists of poly masking files (i.e. all the mask files in the
NCEP_mask
>>>> # directory), define these in the MET point_stat configuration
file.
>>>>
>>>> I wanted to just test it out first so I pointed to only one mask
file,
>>>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
>>>> Eventually I'd like to have 13 masked regions for this. How do I
>>>> "define these in the MET point_stat configuration file"?
Currently my
>>>> PointStatConfig_gauges (cribbed from
>>>> parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc)
has
>> this:
>>>> poly = ${POINT_STAT_POLY};
>>>>
>>>> Do I change this to
>>>>
>>>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>>> ..."${MASKDIR}/WGRFC.nc"; ?
>>>>
>>>>
>>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
>>>> 3) This is more MET than METplus: in the attached test_sum.sh, I
used
>>>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total, and
all the
>>>> hourly files are in
/gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
>>>> What I wanted to do initially was to sum up 24 1-h precip into a
24h
>>>> (12Z-12Z) total and the hourly files are in
/gpfs/.../pcprtma.20190824
>>>> and /gpfs/.../pcprtma.20190825. Could you tell me how to make
the
>>>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
>>>>
>>>> I've been running point_stat in MET but have not run it in
METplus
>>>> before, other than the use case
>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. Haven't
been
>>>> able to figure things out myself in the past few weeks. If you
happen
>>>> to see anything else wrong in my set-up I would appreciate your
pointing
>>>> them out too.
>>>>
>>>> Thank you -
>>>>
>>>> Ying
>>>>
>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: John Halley Gotway
Time: Wed Aug 28 16:19:37 2019
Hi all,
Sorry for being so late in responding. I took a look at the "-pcpdir"
command line option for pcp_combine and see that it expects a single
argument. Using a wildcard in the directory name passes multiple
options
to "-pcpdir" when it only expects one. But you can use "-pcpdir"
multiple
times if needed.
To clarify, the following won't work:
-pcpdir directory*
But the following will work:
-pcpdir directory1 -pcpdir directory2 -pcpdir directory3
This behavior is actual very, very simple to change in pcp_combine.
It's
really just a couple of characters to change in the source code file.
This
issue just hasn't come up before, so we've never been prompted to
support
it.
We could probably even include the change in the met-8.1.2 bugfix
release
if that'd be helpful. Please just let me know.
Thanks,
John
On Wed, Aug 28, 2019 at 10:38 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>
> Hi George,
>
> Thanks for catching the *fcst*_PCP_COMBINE_INPUT_LEVEL! I fixed
> that, it didn't make a difference though.
>
> Here's a listing of the two pcprtma directories:
> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
> pcprtma.2019082400.gif pcprtma.2019082417.gif
pcprtma2.2019082410.grb2
> pcprtma.2019082401.gif pcprtma.2019082418.gif
pcprtma2.2019082411.grb2
> pcprtma.2019082402.gif pcprtma.2019082419.gif
pcprtma2.2019082412.grb2
> pcprtma.2019082403.gif pcprtma.2019082420.gif
pcprtma2.2019082413.grb2
> pcprtma.2019082404.gif pcprtma.2019082421.gif
pcprtma2.2019082414.grb2
> pcprtma.2019082405.gif pcprtma.2019082422.gif
pcprtma2.2019082415.grb2
> pcprtma.2019082406.gif pcprtma.2019082423.gif
pcprtma2.2019082416.grb2
> pcprtma.2019082407.gif pcprtma2.2019082400.grb2
pcprtma2.2019082417.grb2
> pcprtma.2019082408.gif pcprtma2.2019082401.grb2
pcprtma2.2019082418.grb2
> pcprtma.2019082409.gif pcprtma2.2019082402.grb2
pcprtma2.2019082419.grb2
> pcprtma.2019082410.gif pcprtma2.2019082403.grb2
pcprtma2.2019082420.grb2
> pcprtma.2019082411.gif pcprtma2.2019082404.grb2
pcprtma2.2019082421.grb2
> pcprtma.2019082412.gif pcprtma2.2019082405.grb2
pcprtma2.2019082422.grb2
> pcprtma.2019082413.gif pcprtma2.2019082406.grb2
pcprtma2.2019082423.grb2
> pcprtma.2019082414.gif pcprtma2.2019082407.grb2 wmo
> pcprtma.2019082415.gif pcprtma2.2019082408.grb2
> pcprtma.2019082416.gif pcprtma2.2019082409.grb2
> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
> pcprtma.2019082500.gif pcprtma.2019082517.gif
pcprtma2.2019082510.grb2
> pcprtma.2019082501.gif pcprtma.2019082518.gif
pcprtma2.2019082511.grb2
> pcprtma.2019082502.gif pcprtma.2019082519.gif
pcprtma2.2019082512.grb2
> pcprtma.2019082503.gif pcprtma.2019082520.gif
pcprtma2.2019082513.grb2
> pcprtma.2019082504.gif pcprtma.2019082521.gif
pcprtma2.2019082514.grb2
> pcprtma.2019082505.gif pcprtma.2019082522.gif
pcprtma2.2019082515.grb2
> pcprtma.2019082506.gif pcprtma.2019082523.gif
pcprtma2.2019082516.grb2
> pcprtma.2019082507.gif pcprtma2.2019082500.grb2
pcprtma2.2019082517.grb2
> pcprtma.2019082508.gif pcprtma2.2019082501.grb2
pcprtma2.2019082518.grb2
> pcprtma.2019082509.gif pcprtma2.2019082502.grb2
pcprtma2.2019082519.grb2
> pcprtma.2019082510.gif pcprtma2.2019082503.grb2
pcprtma2.2019082520.grb2
> pcprtma.2019082511.gif pcprtma2.2019082504.grb2
pcprtma2.2019082521.grb2
> pcprtma.2019082512.gif pcprtma2.2019082505.grb2
pcprtma2.2019082522.grb2
> pcprtma.2019082513.gif pcprtma2.2019082506.grb2
pcprtma2.2019082523.grb2
> pcprtma.2019082514.gif pcprtma2.2019082507.grb2 wmo
> pcprtma.2019082515.gif pcprtma2.2019082508.grb2
> pcprtma.2019082516.gif pcprtma2.2019082509.grb2
>
> I also placed a copy of all the *.grb2 files from above in
> ftp://ftp.emc.ncep.noaa.gov/mmb/precip/pcprtma.0824-25.tar.gz, in
case
> there might be something odd in the grib header. There doesn't seem
to
> have been a missing file though.
>
> Thanks again -
> Ying
>
>
> On 8/28/19 11:28 AM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > I noticed that one of your variables is not all upper case:
> >
> > # Accumulation interval available in forecast data
> > fcst_PCP_COMBINE_INPUT_LEVEL = 01
> >
> > I don't think that would cause PcpCombine wrapper to not find
files, but
> it
> > may help. Could you send me the directory listing for
> >
> > /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
> > and
> > /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
> >
> > When PcpCombine wrapper cannot build an accumulation, the first
place I
> > look is the input directory and template to make sure all of the
files it
> > will need are available. Often times there is a missing file so it
can't
> > build a 24 hour accumulation. Besides that, the logging doesn't
give you
> > much insight as to where it failed to find the required data. I
can add
> > more debugging information to wrapper for this processing, but it
will
> > create a lot of extra logging information.
> >
> > On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
> >>
> >> Thanks George! I've put the masks into the pcprtma_24h.config.
If it
> >> doesn't work then I'll try the "higher level" PointStatConfig. If
either
> >> works that'll be really useful.
> >>
> >> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
> >> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
> >> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell a
> >> week ago. When I ran the test_add.sh it was three weeks ago and
the
> >> RTMA on the Dell was still the 'para'. It's now officially
'prod'.
> >> Anyway I ran the script again with the new multiple-mask set up
in the
> >> config and there is still a problem with the
pcp_combine_wrapper.py:
> >>
> >> 08/28 14:41:42.152 metplus.PcpCombine
(pcp_combine_wrapper.py:547)
> >> ERROR: pcp_combine could not generate command for init
201908241200 and
> >> forecast lead 24
> >>
> >> Not sure how to go from here. Is there a way to find out why
> >> "pcp_combine could not generate command"?
> >>
> >> Thanks again -
> >>
> >> Ying
> >>
> >> On 8/27/19 6:26 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> (John, see #3)
> >>>
> >>> 1) It looks like your input dir for PcpCombine is slightly
wrong. You
> >> have:
> >>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
> >>> and I think it should be
> >>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
> >>>
> >>> That is the directory you are using for the -add command in your
> >>> test_add.sh script. The current version of METplus does not list
the
> >>> directory and template used in the logs when PcpCombine fails,
but this
> >>> will be in the next version.
> >>>
> >>> 2) You can change this value to a list of mask files, but you
will need
> >> to
> >>> add []s around the values, i.e.
> >>>
> >>> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> >>> ..."${MASKDIR}/WGRFC.nc" ];
> >>>
> >>> However, you should also be able to keep the MET config file as
is and
> >> set
> >>> these values in your METplus config file like this:
> >>>
> >>> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc",
"${MASKDIR}/CBRFC.nc",
> >>> ..."${MASKDIR}/WGRFC.nc" ]
> >>>
> >>> I don't think passing in []s in the variable has been tested,
but in
> >> theory
> >>> it should work. Let me know it if does not.
> >>>
> >>> 3) This question may be better suited for John. I believe the
-pcpdir
> >> needs
> >>> to specify the directory containing the files, but I am not sure
if you
> >> are
> >>> able to specify 2 directories to look in. You could try using
the *
> >>> character, i.e. gpfs/.../pcprtma.*
> >>> If you have many other directories besides those 2 dates,
PcpCombine
> >> would
> >>> look in all of those directories, so it would take a very long
time to
> >> run.
> >>> I'm CC'ing John in case he has more input related to #3. Let me
know if
> >> my
> >>> suggestions don't work or if you have any other questions.
> >>>
> >>> Thanks,
> >>> George
> >>>
> >>>
> >>> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
> >>>> Transaction: Ticket created by ying.lin at noaa.gov
> >>>> Queue: met_help
> >>>> Subject: METplus PointStat; MET pcp_combine -sum
(wildcard in
> >> -pcpdir
> >>>> argument?)
> >>>> Owner: Nobody
> >>>> Requestors: ying.lin at noaa.gov
> >>>> Status: new
> >>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
> >>>>
> >>>> Hello. I'm trying to run METplus' PointStat - want to sum up
24h
> >>>> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
> >>>> There are some issues I couldn't figure out myself and hope you
can
> help
> >>>> me with. Attached are
> >>>>
> >>>> pcprtma_24h.ksh: run script
> >>>>
> >>>> metplus_final.conf
> >>>>
> >>>> master_metplus.log.20190827
> >>>>
> >>>> pcprtma_24h.conf: my config for pcprtma
> >>>>
> >>>> PointStatConfig_gauges: my general PointStatConfig
> >>>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
pcp_combine
> on
> >>>> the pcprtma data to rule out data problem
> >>>>
> >>>>
> >>>> Issues:
> >>>>
> >>>> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma'
into
> 24h
> >>>> totals: in the attached log file, there is "metplus.PointStat
> >>>> (point_stat_wrapper.py:174) ERROR: Could not find file in
> >>>>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
> >>>> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
> For
> >>>> a while I wondered whether the problem was with my pcprtma
data, so I
> >>>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
> >>>> test_add.sh and test_sum.sh) and the data seem to be OK, so
most
> likely
> >>>> it's error in my config files
> >>>>
> >>>> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
> >>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see
that
> in
> >>>> this config file, for POINT_STAT_POLY there is
> >>>>
> >>>> # List of full path to poly masking files. NOTE: Only short
lists of
> >> poly
> >>>> # files work (those that fit on one line), a long list will
result in
> an
> >>>> # environment variable that is too long, resulting in an error.
For
> long
> >>>> # lists of poly masking files (i.e. all the mask files in the
> NCEP_mask
> >>>> # directory), define these in the MET point_stat configuration
file.
> >>>>
> >>>> I wanted to just test it out first so I pointed to only one
mask file,
> >>>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
> >>>> Eventually I'd like to have 13 masked regions for this. How do
I
> >>>> "define these in the MET point_stat configuration file"?
Currently my
> >>>> PointStatConfig_gauges (cribbed from
> >>>>
parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has
> >> this:
> >>>> poly = ${POINT_STAT_POLY};
> >>>>
> >>>> Do I change this to
> >>>>
> >>>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> >>>> ..."${MASKDIR}/WGRFC.nc"; ?
> >>>>
> >>>>
> >>
>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
> >>>> 3) This is more MET than METplus: in the attached test_sum.sh,
I used
> >>>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total,
and all
> the
> >>>> hourly files are in
> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
> >>>> What I wanted to do initially was to sum up 24 1-h precip into
a 24h
> >>>> (12Z-12Z) total and the hourly files are in
> /gpfs/.../pcprtma.20190824
> >>>> and /gpfs/.../pcprtma.20190825. Could you tell me how to make
the
> >>>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
> >>>>
> >>>> I've been running point_stat in MET but have not run it in
METplus
> >>>> before, other than the use case
> >>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf.
Haven't been
> >>>> able to figure things out myself in the past few weeks. If you
happen
> >>>> to see anything else wrong in my set-up I would appreciate your
> pointing
> >>>> them out too.
> >>>>
> >>>> Thank you -
> >>>>
> >>>> Ying
> >>>>
> >>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: Ying Lin
Time: Wed Aug 28 16:37:53 2019
Hi John,
I tried it out like this:
$PCPCOMBINE -sum 00000000_000000 1 \
${day}_120000 24 \
pcprtma.${day}12_A24h.nc \
-pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$daym1 \
-pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$day \
-pcprx pcprtma2.*.grb2
It works great, I can see it first finding the files in the two
directories matching the requested 24 valid times, then reading each
of
the file to create the 24h sum. I didn't know -pcpdir can be applied
twice, this is even better than -pcpdir allowing wildcard IMO since as
George said that would send the script looking for all directories
with
the name pcprtma.%Y%M%D. *Thank you* for your reply - I'm very happy
with the above set up.
Ying
On 8/28/19 6:19 PM, John Halley Gotway via RT wrote:
> Hi all,
>
> Sorry for being so late in responding. I took a look at the "-
pcpdir"
> command line option for pcp_combine and see that it expects a single
> argument. Using a wildcard in the directory name passes multiple
options
> to "-pcpdir" when it only expects one. But you can use "-pcpdir"
multiple
> times if needed.
>
> To clarify, the following won't work:
> -pcpdir directory*
>
> But the following will work:
> -pcpdir directory1 -pcpdir directory2 -pcpdir directory3
>
> This behavior is actual very, very simple to change in pcp_combine.
It's
> really just a couple of characters to change in the source code
file. This
> issue just hasn't come up before, so we've never been prompted to
support
> it.
>
> We could probably even include the change in the met-8.1.2 bugfix
release
> if that'd be helpful. Please just let me know.
>
> Thanks,
> John
>
> On Wed, Aug 28, 2019 at 10:38 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>
>> Hi George,
>>
>> Thanks for catching the *fcst*_PCP_COMBINE_INPUT_LEVEL! I
fixed
>> that, it didn't make a difference though.
>>
>> Here's a listing of the two pcprtma directories:
>> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
>> pcprtma.2019082400.gif pcprtma.2019082417.gif
pcprtma2.2019082410.grb2
>> pcprtma.2019082401.gif pcprtma.2019082418.gif
pcprtma2.2019082411.grb2
>> pcprtma.2019082402.gif pcprtma.2019082419.gif
pcprtma2.2019082412.grb2
>> pcprtma.2019082403.gif pcprtma.2019082420.gif
pcprtma2.2019082413.grb2
>> pcprtma.2019082404.gif pcprtma.2019082421.gif
pcprtma2.2019082414.grb2
>> pcprtma.2019082405.gif pcprtma.2019082422.gif
pcprtma2.2019082415.grb2
>> pcprtma.2019082406.gif pcprtma.2019082423.gif
pcprtma2.2019082416.grb2
>> pcprtma.2019082407.gif pcprtma2.2019082400.grb2
pcprtma2.2019082417.grb2
>> pcprtma.2019082408.gif pcprtma2.2019082401.grb2
pcprtma2.2019082418.grb2
>> pcprtma.2019082409.gif pcprtma2.2019082402.grb2
pcprtma2.2019082419.grb2
>> pcprtma.2019082410.gif pcprtma2.2019082403.grb2
pcprtma2.2019082420.grb2
>> pcprtma.2019082411.gif pcprtma2.2019082404.grb2
pcprtma2.2019082421.grb2
>> pcprtma.2019082412.gif pcprtma2.2019082405.grb2
pcprtma2.2019082422.grb2
>> pcprtma.2019082413.gif pcprtma2.2019082406.grb2
pcprtma2.2019082423.grb2
>> pcprtma.2019082414.gif pcprtma2.2019082407.grb2 wmo
>> pcprtma.2019082415.gif pcprtma2.2019082408.grb2
>> pcprtma.2019082416.gif pcprtma2.2019082409.grb2
>> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
>> pcprtma.2019082500.gif pcprtma.2019082517.gif
pcprtma2.2019082510.grb2
>> pcprtma.2019082501.gif pcprtma.2019082518.gif
pcprtma2.2019082511.grb2
>> pcprtma.2019082502.gif pcprtma.2019082519.gif
pcprtma2.2019082512.grb2
>> pcprtma.2019082503.gif pcprtma.2019082520.gif
pcprtma2.2019082513.grb2
>> pcprtma.2019082504.gif pcprtma.2019082521.gif
pcprtma2.2019082514.grb2
>> pcprtma.2019082505.gif pcprtma.2019082522.gif
pcprtma2.2019082515.grb2
>> pcprtma.2019082506.gif pcprtma.2019082523.gif
pcprtma2.2019082516.grb2
>> pcprtma.2019082507.gif pcprtma2.2019082500.grb2
pcprtma2.2019082517.grb2
>> pcprtma.2019082508.gif pcprtma2.2019082501.grb2
pcprtma2.2019082518.grb2
>> pcprtma.2019082509.gif pcprtma2.2019082502.grb2
pcprtma2.2019082519.grb2
>> pcprtma.2019082510.gif pcprtma2.2019082503.grb2
pcprtma2.2019082520.grb2
>> pcprtma.2019082511.gif pcprtma2.2019082504.grb2
pcprtma2.2019082521.grb2
>> pcprtma.2019082512.gif pcprtma2.2019082505.grb2
pcprtma2.2019082522.grb2
>> pcprtma.2019082513.gif pcprtma2.2019082506.grb2
pcprtma2.2019082523.grb2
>> pcprtma.2019082514.gif pcprtma2.2019082507.grb2 wmo
>> pcprtma.2019082515.gif pcprtma2.2019082508.grb2
>> pcprtma.2019082516.gif pcprtma2.2019082509.grb2
>>
>> I also placed a copy of all the *.grb2 files from above in
>> ftp://ftp.emc.ncep.noaa.gov/mmb/precip/pcprtma.0824-25.tar.gz, in
case
>> there might be something odd in the grib header. There doesn't
seem to
>> have been a missing file though.
>>
>> Thanks again -
>> Ying
>>
>>
>> On 8/28/19 11:28 AM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> I noticed that one of your variables is not all upper case:
>>>
>>> # Accumulation interval available in forecast data
>>> fcst_PCP_COMBINE_INPUT_LEVEL = 01
>>>
>>> I don't think that would cause PcpCombine wrapper to not find
files, but
>> it
>>> may help. Could you send me the directory listing for
>>>
>>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
>>> and
>>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
>>>
>>> When PcpCombine wrapper cannot build an accumulation, the first
place I
>>> look is the input directory and template to make sure all of the
files it
>>> will need are available. Often times there is a missing file so it
can't
>>> build a 24 hour accumulation. Besides that, the logging doesn't
give you
>>> much insight as to where it failed to find the required data. I
can add
>>> more debugging information to wrapper for this processing, but it
will
>>> create a lot of extra logging information.
>>>
>>> On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>>>
>>>> Thanks George! I've put the masks into the pcprtma_24h.config.
If it
>>>> doesn't work then I'll try the "higher level" PointStatConfig. If
either
>>>> works that'll be really useful.
>>>>
>>>> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
>>>> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
>>>> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell a
>>>> week ago. When I ran the test_add.sh it was three weeks ago and
the
>>>> RTMA on the Dell was still the 'para'. It's now officially
'prod'.
>>>> Anyway I ran the script again with the new multiple-mask set up
in the
>>>> config and there is still a problem with the
pcp_combine_wrapper.py:
>>>>
>>>> 08/28 14:41:42.152 metplus.PcpCombine
(pcp_combine_wrapper.py:547)
>>>> ERROR: pcp_combine could not generate command for init
201908241200 and
>>>> forecast lead 24
>>>>
>>>> Not sure how to go from here. Is there a way to find out why
>>>> "pcp_combine could not generate command"?
>>>>
>>>> Thanks again -
>>>>
>>>> Ying
>>>>
>>>> On 8/27/19 6:26 PM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> (John, see #3)
>>>>>
>>>>> 1) It looks like your input dir for PcpCombine is slightly
wrong. You
>>>> have:
>>>>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*prod*
>>>>> and I think it should be
>>>>> FCST_PCP_COMBINE_INPUT_DIR = /gpfs/dell2/nco/ops/com/rtma/*para*
>>>>>
>>>>> That is the directory you are using for the -add command in your
>>>>> test_add.sh script. The current version of METplus does not list
the
>>>>> directory and template used in the logs when PcpCombine fails,
but this
>>>>> will be in the next version.
>>>>>
>>>>> 2) You can change this value to a list of mask files, but you
will need
>>>> to
>>>>> add []s around the values, i.e.
>>>>>
>>>>> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>>>> ..."${MASKDIR}/WGRFC.nc" ];
>>>>>
>>>>> However, you should also be able to keep the MET config file as
is and
>>>> set
>>>>> these values in your METplus config file like this:
>>>>>
>>>>> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc",
"${MASKDIR}/CBRFC.nc",
>>>>> ..."${MASKDIR}/WGRFC.nc" ]
>>>>>
>>>>> I don't think passing in []s in the variable has been tested,
but in
>>>> theory
>>>>> it should work. Let me know it if does not.
>>>>>
>>>>> 3) This question may be better suited for John. I believe the
-pcpdir
>>>> needs
>>>>> to specify the directory containing the files, but I am not sure
if you
>>>> are
>>>>> able to specify 2 directories to look in. You could try using
the *
>>>>> character, i.e. gpfs/.../pcprtma.*
>>>>> If you have many other directories besides those 2 dates,
PcpCombine
>>>> would
>>>>> look in all of those directories, so it would take a very long
time to
>>>> run.
>>>>> I'm CC'ing John in case he has more input related to #3. Let me
know if
>>>> my
>>>>> suggestions don't work or if you have any other questions.
>>>>>
>>>>> Thanks,
>>>>> George
>>>>>
>>>>>
>>>>> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
>>>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>>>> Queue: met_help
>>>>>> Subject: METplus PointStat; MET pcp_combine -sum
(wildcard in
>>>> -pcpdir
>>>>>> argument?)
>>>>>> Owner: Nobody
>>>>>> Requestors: ying.lin at noaa.gov
>>>>>> Status: new
>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
>>>>>> Hello. I'm trying to run METplus' PointStat - want to sum up
24h
>>>>>> 1-hourly precip analysis (pcprtma) and verify it against daily
gauges.
>>>>>> There are some issues I couldn't figure out myself and hope you
can
>> help
>>>>>> me with. Attached are
>>>>>>
>>>>>> pcprtma_24h.ksh: run script
>>>>>>
>>>>>> metplus_final.conf
>>>>>>
>>>>>> master_metplus.log.20190827
>>>>>>
>>>>>> pcprtma_24h.conf: my config for pcprtma
>>>>>>
>>>>>> PointStatConfig_gauges: my general PointStatConfig
>>>>>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
pcp_combine
>> on
>>>>>> the pcprtma data to rule out data problem
>>>>>>
>>>>>>
>>>>>> Issues:
>>>>>>
>>>>>> 1) Right off the bat I could not sum up the 1-hourly 'pcprtma'
into
>> 24h
>>>>>> totals: in the attached log file, there is "metplus.PointStat
>>>>>> (point_stat_wrapper.py:174) ERROR: Could not find file in
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
>>>>>> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
>> For
>>>>>> a while I wondered whether the problem was with my pcprtma
data, so I
>>>>>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
(attached
>>>>>> test_add.sh and test_sum.sh) and the data seem to be OK, so
most
>> likely
>>>>>> it's error in my config files
>>>>>>
>>>>>> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
>>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I see
that
>> in
>>>>>> this config file, for POINT_STAT_POLY there is
>>>>>>
>>>>>> # List of full path to poly masking files. NOTE: Only short
lists of
>>>> poly
>>>>>> # files work (those that fit on one line), a long list will
result in
>> an
>>>>>> # environment variable that is too long, resulting in an error.
For
>> long
>>>>>> # lists of poly masking files (i.e. all the mask files in the
>> NCEP_mask
>>>>>> # directory), define these in the MET point_stat configuration
file.
>>>>>>
>>>>>> I wanted to just test it out first so I pointed to only one
mask file,
>>>>>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
>>>>>> Eventually I'd like to have 13 masked regions for this. How do
I
>>>>>> "define these in the MET point_stat configuration file"?
Currently my
>>>>>> PointStatConfig_gauges (cribbed from
>>>>>>
parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has
>>>> this:
>>>>>> poly = ${POINT_STAT_POLY};
>>>>>>
>>>>>> Do I change this to
>>>>>>
>>>>>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>>>>> ..."${MASKDIR}/WGRFC.nc"; ?
>>>>>>
>>>>>>
>>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
>>>>>> 3) This is more MET than METplus: in the attached test_sum.sh,
I used
>>>>>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total,
and all
>> the
>>>>>> hourly files are in
>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
>>>>>> What I wanted to do initially was to sum up 24 1-h precip into
a 24h
>>>>>> (12Z-12Z) total and the hourly files are in
>> /gpfs/.../pcprtma.20190824
>>>>>> and /gpfs/.../pcprtma.20190825. Could you tell me how to make
the
>>>>>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
>>>>>>
>>>>>> I've been running point_stat in MET but have not run it in
METplus
>>>>>> before, other than the use case
>>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf.
Haven't been
>>>>>> able to figure things out myself in the past few weeks. If you
happen
>>>>>> to see anything else wrong in my set-up I would appreciate your
>> pointing
>>>>>> them out too.
>>>>>>
>>>>>> Thank you -
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: George McCabe
Time: Wed Aug 28 17:28:21 2019
OK, I downloaded your data and conf file and tried to see what was
wrong.
It turns out your config was correct, but there is a bug in the
PcpCombine
wrapper. We have never processed grib2 forecast data using this tool
and
the logic to do so was flawed. I will work on a solution and issue a
bugfix.
On Wed, Aug 28, 2019 at 4:37 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>
> Hi John,
>
> I tried it out like this:
>
> $PCPCOMBINE -sum 00000000_000000 1 \
> ${day}_120000 24 \
> pcprtma.${day}12_A24h.nc \
> -pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$daym1 \
> -pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$day \
> -pcprx pcprtma2.*.grb2
>
> It works great, I can see it first finding the files in the two
> directories matching the requested 24 valid times, then reading each
of
> the file to create the 24h sum. I didn't know -pcpdir can be applied
> twice, this is even better than -pcpdir allowing wildcard IMO since
as
> George said that would send the script looking for all directories
with
> the name pcprtma.%Y%M%D. *Thank you* for your reply - I'm very happy
> with the above set up.
>
> Ying
>
> On 8/28/19 6:19 PM, John Halley Gotway via RT wrote:
> > Hi all,
> >
> > Sorry for being so late in responding. I took a look at the "-
pcpdir"
> > command line option for pcp_combine and see that it expects a
single
> > argument. Using a wildcard in the directory name passes multiple
options
> > to "-pcpdir" when it only expects one. But you can use "-pcpdir"
> multiple
> > times if needed.
> >
> > To clarify, the following won't work:
> > -pcpdir directory*
> >
> > But the following will work:
> > -pcpdir directory1 -pcpdir directory2 -pcpdir directory3
> >
> > This behavior is actual very, very simple to change in
pcp_combine. It's
> > really just a couple of characters to change in the source code
file.
> This
> > issue just hasn't come up before, so we've never been prompted to
support
> > it.
> >
> > We could probably even include the change in the met-8.1.2 bugfix
release
> > if that'd be helpful. Please just let me know.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 28, 2019 at 10:38 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
> >>
> >> Hi George,
> >>
> >> Thanks for catching the *fcst*_PCP_COMBINE_INPUT_LEVEL! I
fixed
> >> that, it didn't make a difference though.
> >>
> >> Here's a listing of the two pcprtma directories:
> >> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
> >> pcprtma.2019082400.gif pcprtma.2019082417.gif
pcprtma2.2019082410.grb2
> >> pcprtma.2019082401.gif pcprtma.2019082418.gif
pcprtma2.2019082411.grb2
> >> pcprtma.2019082402.gif pcprtma.2019082419.gif
pcprtma2.2019082412.grb2
> >> pcprtma.2019082403.gif pcprtma.2019082420.gif
pcprtma2.2019082413.grb2
> >> pcprtma.2019082404.gif pcprtma.2019082421.gif
pcprtma2.2019082414.grb2
> >> pcprtma.2019082405.gif pcprtma.2019082422.gif
pcprtma2.2019082415.grb2
> >> pcprtma.2019082406.gif pcprtma.2019082423.gif
pcprtma2.2019082416.grb2
> >> pcprtma.2019082407.gif pcprtma2.2019082400.grb2
> pcprtma2.2019082417.grb2
> >> pcprtma.2019082408.gif pcprtma2.2019082401.grb2
> pcprtma2.2019082418.grb2
> >> pcprtma.2019082409.gif pcprtma2.2019082402.grb2
> pcprtma2.2019082419.grb2
> >> pcprtma.2019082410.gif pcprtma2.2019082403.grb2
> pcprtma2.2019082420.grb2
> >> pcprtma.2019082411.gif pcprtma2.2019082404.grb2
> pcprtma2.2019082421.grb2
> >> pcprtma.2019082412.gif pcprtma2.2019082405.grb2
> pcprtma2.2019082422.grb2
> >> pcprtma.2019082413.gif pcprtma2.2019082406.grb2
> pcprtma2.2019082423.grb2
> >> pcprtma.2019082414.gif pcprtma2.2019082407.grb2 wmo
> >> pcprtma.2019082415.gif pcprtma2.2019082408.grb2
> >> pcprtma.2019082416.gif pcprtma2.2019082409.grb2
> >> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
> >> pcprtma.2019082500.gif pcprtma.2019082517.gif
pcprtma2.2019082510.grb2
> >> pcprtma.2019082501.gif pcprtma.2019082518.gif
pcprtma2.2019082511.grb2
> >> pcprtma.2019082502.gif pcprtma.2019082519.gif
pcprtma2.2019082512.grb2
> >> pcprtma.2019082503.gif pcprtma.2019082520.gif
pcprtma2.2019082513.grb2
> >> pcprtma.2019082504.gif pcprtma.2019082521.gif
pcprtma2.2019082514.grb2
> >> pcprtma.2019082505.gif pcprtma.2019082522.gif
pcprtma2.2019082515.grb2
> >> pcprtma.2019082506.gif pcprtma.2019082523.gif
pcprtma2.2019082516.grb2
> >> pcprtma.2019082507.gif pcprtma2.2019082500.grb2
> pcprtma2.2019082517.grb2
> >> pcprtma.2019082508.gif pcprtma2.2019082501.grb2
> pcprtma2.2019082518.grb2
> >> pcprtma.2019082509.gif pcprtma2.2019082502.grb2
> pcprtma2.2019082519.grb2
> >> pcprtma.2019082510.gif pcprtma2.2019082503.grb2
> pcprtma2.2019082520.grb2
> >> pcprtma.2019082511.gif pcprtma2.2019082504.grb2
> pcprtma2.2019082521.grb2
> >> pcprtma.2019082512.gif pcprtma2.2019082505.grb2
> pcprtma2.2019082522.grb2
> >> pcprtma.2019082513.gif pcprtma2.2019082506.grb2
> pcprtma2.2019082523.grb2
> >> pcprtma.2019082514.gif pcprtma2.2019082507.grb2 wmo
> >> pcprtma.2019082515.gif pcprtma2.2019082508.grb2
> >> pcprtma.2019082516.gif pcprtma2.2019082509.grb2
> >>
> >> I also placed a copy of all the *.grb2 files from above in
> >> ftp://ftp.emc.ncep.noaa.gov/mmb/precip/pcprtma.0824-25.tar.gz, in
case
> >> there might be something odd in the grib header. There doesn't
seem to
> >> have been a missing file though.
> >>
> >> Thanks again -
> >> Ying
> >>
> >>
> >> On 8/28/19 11:28 AM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> I noticed that one of your variables is not all upper case:
> >>>
> >>> # Accumulation interval available in forecast data
> >>> fcst_PCP_COMBINE_INPUT_LEVEL = 01
> >>>
> >>> I don't think that would cause PcpCombine wrapper to not find
files,
> but
> >> it
> >>> may help. Could you send me the directory listing for
> >>>
> >>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
> >>> and
> >>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
> >>>
> >>> When PcpCombine wrapper cannot build an accumulation, the first
place I
> >>> look is the input directory and template to make sure all of the
files
> it
> >>> will need are available. Often times there is a missing file so
it
> can't
> >>> build a 24 hour accumulation. Besides that, the logging doesn't
give
> you
> >>> much insight as to where it failed to find the required data. I
can add
> >>> more debugging information to wrapper for this processing, but
it will
> >>> create a lot of extra logging information.
> >>>
> >>> On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
> >>>>
> >>>> Thanks George! I've put the masks into the pcprtma_24h.config.
If it
> >>>> doesn't work then I'll try the "higher level" PointStatConfig.
If
> either
> >>>> works that'll be really useful.
> >>>>
> >>>> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
> >>>> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
> >>>> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell
> a
> >>>> week ago. When I ran the test_add.sh it was three weeks ago
and the
> >>>> RTMA on the Dell was still the 'para'. It's now officially
'prod'.
> >>>> Anyway I ran the script again with the new multiple-mask set up
in the
> >>>> config and there is still a problem with the
pcp_combine_wrapper.py:
> >>>>
> >>>> 08/28 14:41:42.152 metplus.PcpCombine
(pcp_combine_wrapper.py:547)
> >>>> ERROR: pcp_combine could not generate command for init
201908241200
> and
> >>>> forecast lead 24
> >>>>
> >>>> Not sure how to go from here. Is there a way to find out why
> >>>> "pcp_combine could not generate command"?
> >>>>
> >>>> Thanks again -
> >>>>
> >>>> Ying
> >>>>
> >>>> On 8/27/19 6:26 PM, George McCabe via RT wrote:
> >>>>> Hi Ying,
> >>>>>
> >>>>> (John, see #3)
> >>>>>
> >>>>> 1) It looks like your input dir for PcpCombine is slightly
wrong. You
> >>>> have:
> >>>>> FCST_PCP_COMBINE_INPUT_DIR =
/gpfs/dell2/nco/ops/com/rtma/*prod*
> >>>>> and I think it should be
> >>>>> FCST_PCP_COMBINE_INPUT_DIR =
/gpfs/dell2/nco/ops/com/rtma/*para*
> >>>>>
> >>>>> That is the directory you are using for the -add command in
your
> >>>>> test_add.sh script. The current version of METplus does not
list the
> >>>>> directory and template used in the logs when PcpCombine fails,
but
> this
> >>>>> will be in the next version.
> >>>>>
> >>>>> 2) You can change this value to a list of mask files, but you
will
> need
> >>>> to
> >>>>> add []s around the values, i.e.
> >>>>>
> >>>>> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> >>>>> ..."${MASKDIR}/WGRFC.nc" ];
> >>>>>
> >>>>> However, you should also be able to keep the MET config file
as is
> and
> >>>> set
> >>>>> these values in your METplus config file like this:
> >>>>>
> >>>>> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc",
"${MASKDIR}/CBRFC.nc",
> >>>>> ..."${MASKDIR}/WGRFC.nc" ]
> >>>>>
> >>>>> I don't think passing in []s in the variable has been tested,
but in
> >>>> theory
> >>>>> it should work. Let me know it if does not.
> >>>>>
> >>>>> 3) This question may be better suited for John. I believe the
-pcpdir
> >>>> needs
> >>>>> to specify the directory containing the files, but I am not
sure if
> you
> >>>> are
> >>>>> able to specify 2 directories to look in. You could try using
the *
> >>>>> character, i.e. gpfs/.../pcprtma.*
> >>>>> If you have many other directories besides those 2 dates,
PcpCombine
> >>>> would
> >>>>> look in all of those directories, so it would take a very long
time
> to
> >>>> run.
> >>>>> I'm CC'ing John in case he has more input related to #3. Let
me know
> if
> >>>> my
> >>>>> suggestions don't work or if you have any other questions.
> >>>>>
> >>>>> Thanks,
> >>>>> George
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
> >>>>>> Transaction: Ticket created by ying.lin at noaa.gov
> >>>>>> Queue: met_help
> >>>>>> Subject: METplus PointStat; MET pcp_combine -sum
(wildcard
> in
> >>>> -pcpdir
> >>>>>> argument?)
> >>>>>> Owner: Nobody
> >>>>>> Requestors: ying.lin at noaa.gov
> >>>>>> Status: new
> >>>>>> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
> >>>>>> Hello. I'm trying to run METplus' PointStat - want to sum up
24h
> >>>>>> 1-hourly precip analysis (pcprtma) and verify it against
daily
> gauges.
> >>>>>> There are some issues I couldn't figure out myself and hope
you can
> >> help
> >>>>>> me with. Attached are
> >>>>>>
> >>>>>> pcprtma_24h.ksh: run script
> >>>>>>
> >>>>>> metplus_final.conf
> >>>>>>
> >>>>>> master_metplus.log.20190827
> >>>>>>
> >>>>>> pcprtma_24h.conf: my config for pcprtma
> >>>>>>
> >>>>>> PointStatConfig_gauges: my general PointStatConfig
> >>>>>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
> pcp_combine
> >> on
> >>>>>> the pcprtma data to rule out data problem
> >>>>>>
> >>>>>>
> >>>>>> Issues:
> >>>>>>
> >>>>>> 1) Right off the bat I could not sum up the 1-hourly
'pcprtma' into
> >> 24h
> >>>>>> totals: in the attached log file, there is "metplus.PointStat
> >>>>>> (point_stat_wrapper.py:174) ERROR: Could not find file in
> >>>>>>
> >>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
> >>>>>> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
> >> For
> >>>>>> a while I wondered whether the problem was with my pcprtma
data, so
> I
> >>>>>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
> (attached
> >>>>>> test_add.sh and test_sum.sh) and the data seem to be OK, so
most
> >> likely
> >>>>>> it's error in my config files
> >>>>>>
> >>>>>> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
> >>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I
see that
> >> in
> >>>>>> this config file, for POINT_STAT_POLY there is
> >>>>>>
> >>>>>> # List of full path to poly masking files. NOTE: Only short
lists
> of
> >>>> poly
> >>>>>> # files work (those that fit on one line), a long list will
result
> in
> >> an
> >>>>>> # environment variable that is too long, resulting in an
error. For
> >> long
> >>>>>> # lists of poly masking files (i.e. all the mask files in the
> >> NCEP_mask
> >>>>>> # directory), define these in the MET point_stat
configuration file.
> >>>>>>
> >>>>>> I wanted to just test it out first so I pointed to only one
mask
> file,
> >>>>>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
> >>>>>> Eventually I'd like to have 13 masked regions for this. How
do I
> >>>>>> "define these in the MET point_stat configuration file"?
Currently
> my
> >>>>>> PointStatConfig_gauges (cribbed from
> >>>>>>
parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has
> >>>> this:
> >>>>>> poly = ${POINT_STAT_POLY};
> >>>>>>
> >>>>>> Do I change this to
> >>>>>>
> >>>>>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
> >>>>>> ..."${MASKDIR}/WGRFC.nc"; ?
> >>>>>>
> >>>>>>
> >>
>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
> >>>>>> 3) This is more MET than METplus: in the attached
test_sum.sh, I
> used
> >>>>>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total,
and all
> >> the
> >>>>>> hourly files are in
> >> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
> >>>>>> What I wanted to do initially was to sum up 24 1-h precip
into a 24h
> >>>>>> (12Z-12Z) total and the hourly files are in
> >> /gpfs/.../pcprtma.20190824
> >>>>>> and /gpfs/.../pcprtma.20190825. Could you tell me how to
make the
> >>>>>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
> >>>>>>
> >>>>>> I've been running point_stat in MET but have not run it
in
> METplus
> >>>>>> before, other than the use case
> >>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf.
Haven't
> been
> >>>>>> able to figure things out myself in the past few weeks. If
you
> happen
> >>>>>> to see anything else wrong in my set-up I would appreciate
your
> >> pointing
> >>>>>> them out too.
> >>>>>>
> >>>>>> Thank you -
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Ying Lin
> >>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>>> NCWCP Cubicle No. 2015
> >>>>>> Ying.Lin at noaa.gov
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91715] METplus PointStat; MET pcp_combine -sum (wildcard in -pcpdir argument?)
From: Ying Lin
Time: Wed Aug 28 22:11:43 2019
Hi George, thank you for taking the time to look into this. I’ve had
this “analysis to gauges” validation working in MET before and it’s
relatively simple, so METplus for this is not a high priority item.
Ying
> On Aug 28, 2019, at 19:28, George McCabe via RT <met_help at ucar.edu>
wrote:
>
> OK, I downloaded your data and conf file and tried to see what was
wrong.
> It turns out your config was correct, but there is a bug in the
PcpCombine
> wrapper. We have never processed grib2 forecast data using this tool
and
> the logic to do so was flawed. I will work on a solution and issue a
bugfix.
>
>> On Wed, Aug 28, 2019 at 4:37 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>
>> Hi John,
>>
>> I tried it out like this:
>>
>> $PCPCOMBINE -sum 00000000_000000 1 \
>> ${day}_120000 24 \
>> pcprtma.${day}12_A24h.nc \
>> -pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$daym1 \
>> -pcpdir /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.$day \
>> -pcprx pcprtma2.*.grb2
>>
>> It works great, I can see it first finding the files in the two
>> directories matching the requested 24 valid times, then reading
each of
>> the file to create the 24h sum. I didn't know -pcpdir can be
applied
>> twice, this is even better than -pcpdir allowing wildcard IMO since
as
>> George said that would send the script looking for all directories
with
>> the name pcprtma.%Y%M%D. *Thank you* for your reply - I'm very
happy
>> with the above set up.
>>
>> Ying
>>
>>> On 8/28/19 6:19 PM, John Halley Gotway via RT wrote:
>>> Hi all,
>>>
>>> Sorry for being so late in responding. I took a look at the "-
pcpdir"
>>> command line option for pcp_combine and see that it expects a
single
>>> argument. Using a wildcard in the directory name passes multiple
options
>>> to "-pcpdir" when it only expects one. But you can use "-pcpdir"
>> multiple
>>> times if needed.
>>>
>>> To clarify, the following won't work:
>>> -pcpdir directory*
>>>
>>> But the following will work:
>>> -pcpdir directory1 -pcpdir directory2 -pcpdir directory3
>>>
>>> This behavior is actual very, very simple to change in
pcp_combine. It's
>>> really just a couple of characters to change in the source code
file.
>> This
>>> issue just hasn't come up before, so we've never been prompted to
support
>>> it.
>>>
>>> We could probably even include the change in the met-8.1.2 bugfix
release
>>> if that'd be helpful. Please just let me know.
>>>
>>> Thanks,
>>> John
>>>
>>> On Wed, Aug 28, 2019 at 10:38 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>>>
>>>> Hi George,
>>>>
>>>> Thanks for catching the *fcst*_PCP_COMBINE_INPUT_LEVEL! I
fixed
>>>> that, it didn't make a difference though.
>>>>
>>>> Here's a listing of the two pcprtma directories:
>>>> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
>>>> pcprtma.2019082400.gif pcprtma.2019082417.gif
pcprtma2.2019082410.grb2
>>>> pcprtma.2019082401.gif pcprtma.2019082418.gif
pcprtma2.2019082411.grb2
>>>> pcprtma.2019082402.gif pcprtma.2019082419.gif
pcprtma2.2019082412.grb2
>>>> pcprtma.2019082403.gif pcprtma.2019082420.gif
pcprtma2.2019082413.grb2
>>>> pcprtma.2019082404.gif pcprtma.2019082421.gif
pcprtma2.2019082414.grb2
>>>> pcprtma.2019082405.gif pcprtma.2019082422.gif
pcprtma2.2019082415.grb2
>>>> pcprtma.2019082406.gif pcprtma.2019082423.gif
pcprtma2.2019082416.grb2
>>>> pcprtma.2019082407.gif pcprtma2.2019082400.grb2
>> pcprtma2.2019082417.grb2
>>>> pcprtma.2019082408.gif pcprtma2.2019082401.grb2
>> pcprtma2.2019082418.grb2
>>>> pcprtma.2019082409.gif pcprtma2.2019082402.grb2
>> pcprtma2.2019082419.grb2
>>>> pcprtma.2019082410.gif pcprtma2.2019082403.grb2
>> pcprtma2.2019082420.grb2
>>>> pcprtma.2019082411.gif pcprtma2.2019082404.grb2
>> pcprtma2.2019082421.grb2
>>>> pcprtma.2019082412.gif pcprtma2.2019082405.grb2
>> pcprtma2.2019082422.grb2
>>>> pcprtma.2019082413.gif pcprtma2.2019082406.grb2
>> pcprtma2.2019082423.grb2
>>>> pcprtma.2019082414.gif pcprtma2.2019082407.grb2 wmo
>>>> pcprtma.2019082415.gif pcprtma2.2019082408.grb2
>>>> pcprtma.2019082416.gif pcprtma2.2019082409.grb2
>>>> m72a1 $ ls /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
>>>> pcprtma.2019082500.gif pcprtma.2019082517.gif
pcprtma2.2019082510.grb2
>>>> pcprtma.2019082501.gif pcprtma.2019082518.gif
pcprtma2.2019082511.grb2
>>>> pcprtma.2019082502.gif pcprtma.2019082519.gif
pcprtma2.2019082512.grb2
>>>> pcprtma.2019082503.gif pcprtma.2019082520.gif
pcprtma2.2019082513.grb2
>>>> pcprtma.2019082504.gif pcprtma.2019082521.gif
pcprtma2.2019082514.grb2
>>>> pcprtma.2019082505.gif pcprtma.2019082522.gif
pcprtma2.2019082515.grb2
>>>> pcprtma.2019082506.gif pcprtma.2019082523.gif
pcprtma2.2019082516.grb2
>>>> pcprtma.2019082507.gif pcprtma2.2019082500.grb2
>> pcprtma2.2019082517.grb2
>>>> pcprtma.2019082508.gif pcprtma2.2019082501.grb2
>> pcprtma2.2019082518.grb2
>>>> pcprtma.2019082509.gif pcprtma2.2019082502.grb2
>> pcprtma2.2019082519.grb2
>>>> pcprtma.2019082510.gif pcprtma2.2019082503.grb2
>> pcprtma2.2019082520.grb2
>>>> pcprtma.2019082511.gif pcprtma2.2019082504.grb2
>> pcprtma2.2019082521.grb2
>>>> pcprtma.2019082512.gif pcprtma2.2019082505.grb2
>> pcprtma2.2019082522.grb2
>>>> pcprtma.2019082513.gif pcprtma2.2019082506.grb2
>> pcprtma2.2019082523.grb2
>>>> pcprtma.2019082514.gif pcprtma2.2019082507.grb2 wmo
>>>> pcprtma.2019082515.gif pcprtma2.2019082508.grb2
>>>> pcprtma.2019082516.gif pcprtma2.2019082509.grb2
>>>>
>>>> I also placed a copy of all the *.grb2 files from above in
>>>> ftp://ftp.emc.ncep.noaa.gov/mmb/precip/pcprtma.0824-25.tar.gz, in
case
>>>> there might be something odd in the grib header. There doesn't
seem to
>>>> have been a missing file though.
>>>>
>>>> Thanks again -
>>>> Ying
>>>>
>>>>
>>>>> On 8/28/19 11:28 AM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> I noticed that one of your variables is not all upper case:
>>>>>
>>>>> # Accumulation interval available in forecast data
>>>>> fcst_PCP_COMBINE_INPUT_LEVEL = 01
>>>>>
>>>>> I don't think that would cause PcpCombine wrapper to not find
files,
>> but
>>>> it
>>>>> may help. Could you send me the directory listing for
>>>>>
>>>>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825
>>>>> and
>>>>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190824
>>>>>
>>>>> When PcpCombine wrapper cannot build an accumulation, the first
place I
>>>>> look is the input directory and template to make sure all of the
files
>> it
>>>>> will need are available. Often times there is a missing file so
it
>> can't
>>>>> build a 24 hour accumulation. Besides that, the logging doesn't
give
>> you
>>>>> much insight as to where it failed to find the required data. I
can add
>>>>> more debugging information to wrapper for this processing, but
it will
>>>>> create a lot of extra logging information.
>>>>>
>>>>> On Wed, Aug 28, 2019 at 9:11 AM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715 >
>>>>>>
>>>>>> Thanks George! I've put the masks into the pcprtma_24h.config.
If it
>>>>>> doesn't work then I'll try the "higher level" PointStatConfig.
If
>> either
>>>>>> works that'll be really useful.
>>>>>>
>>>>>> Actually the path for FCST_PCP_COMBINE_INPUT_DIR is
>>>>>> /gpfs/dell2/nco/ops/com/rtma/prod. We had the production RTMA
>>>>>> transitioned to the old phase2 (tide/gyre) machine to the
phase3/dell
>> a
>>>>>> week ago. When I ran the test_add.sh it was three weeks ago
and the
>>>>>> RTMA on the Dell was still the 'para'. It's now officially
'prod'.
>>>>>> Anyway I ran the script again with the new multiple-mask set up
in the
>>>>>> config and there is still a problem with the
pcp_combine_wrapper.py:
>>>>>>
>>>>>> 08/28 14:41:42.152 metplus.PcpCombine
(pcp_combine_wrapper.py:547)
>>>>>> ERROR: pcp_combine could not generate command for init
201908241200
>> and
>>>>>> forecast lead 24
>>>>>>
>>>>>> Not sure how to go from here. Is there a way to find out why
>>>>>> "pcp_combine could not generate command"?
>>>>>>
>>>>>> Thanks again -
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>>> On 8/27/19 6:26 PM, George McCabe via RT wrote:
>>>>>>> Hi Ying,
>>>>>>>
>>>>>>> (John, see #3)
>>>>>>>
>>>>>>> 1) It looks like your input dir for PcpCombine is slightly
wrong. You
>>>>>> have:
>>>>>>> FCST_PCP_COMBINE_INPUT_DIR =
/gpfs/dell2/nco/ops/com/rtma/*prod*
>>>>>>> and I think it should be
>>>>>>> FCST_PCP_COMBINE_INPUT_DIR =
/gpfs/dell2/nco/ops/com/rtma/*para*
>>>>>>>
>>>>>>> That is the directory you are using for the -add command in
your
>>>>>>> test_add.sh script. The current version of METplus does not
list the
>>>>>>> directory and template used in the logs when PcpCombine fails,
but
>> this
>>>>>>> will be in the next version.
>>>>>>>
>>>>>>> 2) You can change this value to a list of mask files, but you
will
>> need
>>>>>> to
>>>>>>> add []s around the values, i.e.
>>>>>>>
>>>>>>> poly = [ "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>>>>>> ..."${MASKDIR}/WGRFC.nc" ];
>>>>>>>
>>>>>>> However, you should also be able to keep the MET config file
as is
>> and
>>>>>> set
>>>>>>> these values in your METplus config file like this:
>>>>>>>
>>>>>>> POINT_STAT_POLY = [ "${MASKDIR}/ABRFC.nc",
"${MASKDIR}/CBRFC.nc",
>>>>>>> ..."${MASKDIR}/WGRFC.nc" ]
>>>>>>>
>>>>>>> I don't think passing in []s in the variable has been tested,
but in
>>>>>> theory
>>>>>>> it should work. Let me know it if does not.
>>>>>>>
>>>>>>> 3) This question may be better suited for John. I believe the
-pcpdir
>>>>>> needs
>>>>>>> to specify the directory containing the files, but I am not
sure if
>> you
>>>>>> are
>>>>>>> able to specify 2 directories to look in. You could try using
the *
>>>>>>> character, i.e. gpfs/.../pcprtma.*
>>>>>>> If you have many other directories besides those 2 dates,
PcpCombine
>>>>>> would
>>>>>>> look in all of those directories, so it would take a very long
time
>> to
>>>>>> run.
>>>>>>> I'm CC'ing John in case he has more input related to #3. Let
me know
>> if
>>>>>> my
>>>>>>> suggestions don't work or if you have any other questions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> George
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 27, 2019 at 1:18 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>>> Tue Aug 27 13:18:52 2019: Request 91715 was acted upon.
>>>>>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>>>>>> Queue: met_help
>>>>>>>> Subject: METplus PointStat; MET pcp_combine -sum
(wildcard
>> in
>>>>>> -pcpdir
>>>>>>>> argument?)
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: ying.lin at noaa.gov
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91715
>>>>>>>> Hello. I'm trying to run METplus' PointStat - want to sum up
24h
>>>>>>>> 1-hourly precip analysis (pcprtma) and verify it against
daily
>> gauges.
>>>>>>>> There are some issues I couldn't figure out myself and hope
you can
>>>> help
>>>>>>>> me with. Attached are
>>>>>>>>
>>>>>>>> pcprtma_24h.ksh: run script
>>>>>>>>
>>>>>>>> metplus_final.conf
>>>>>>>>
>>>>>>>> master_metplus.log.20190827
>>>>>>>>
>>>>>>>> pcprtma_24h.conf: my config for pcprtma
>>>>>>>>
>>>>>>>> PointStatConfig_gauges: my general PointStatConfig
>>>>>>>> test_add.sh, test_sum.sh: "manual" run of MET8.1's
>> pcp_combine
>>>> on
>>>>>>>> the pcprtma data to rule out data problem
>>>>>>>>
>>>>>>>>
>>>>>>>> Issues:
>>>>>>>>
>>>>>>>> 1) Right off the bat I could not sum up the 1-hourly
'pcprtma' into
>>>> 24h
>>>>>>>> totals: in the attached log file, there is "metplus.PointStat
>>>>>>>> (point_stat_wrapper.py:174) ERROR: Could not find file in
>>>>>>>>
>>>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket
>>>>>>>> matching template
pcprtma.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h".
>>>> For
>>>>>>>> a while I wondered whether the problem was with my pcprtma
data, so
>> I
>>>>>>>> tried out MET8.1's pcp_combine's 'add' and 'sum' on the data
>> (attached
>>>>>>>> test_add.sh and test_sum.sh) and the data seem to be OK, so
most
>>>> likely
>>>>>>>> it's error in my config files
>>>>>>>>
>>>>>>>> 2) Mask files: my pcprtma_24h.conf was originally copied over
from
>>>>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf. I
see that
>>>> in
>>>>>>>> this config file, for POINT_STAT_POLY there is
>>>>>>>>
>>>>>>>> # List of full path to poly masking files. NOTE: Only short
lists
>> of
>>>>>> poly
>>>>>>>> # files work (those that fit on one line), a long list will
result
>> in
>>>> an
>>>>>>>> # environment variable that is too long, resulting in an
error. For
>>>> long
>>>>>>>> # lists of poly masking files (i.e. all the mask files in the
>>>> NCEP_mask
>>>>>>>> # directory), define these in the MET point_stat
configuration file.
>>>>>>>>
>>>>>>>> I wanted to just test it out first so I pointed to only one
mask
>> file,
>>>>>>>> the /gpfs/..../masks/rfcs/CNSRFCS.nc (mask covering all ConUS
RFCs).
>>>>>>>> Eventually I'd like to have 13 masked regions for this. How
do I
>>>>>>>> "define these in the MET point_stat configuration file"?
Currently
>> my
>>>>>>>> PointStatConfig_gauges (cribbed from
>>>>>>>>
parm/use_cases/grid_to_obs/met_config/PointStatConfig_conus_sfc) has
>>>>>> this:
>>>>>>>> poly = ${POINT_STAT_POLY};
>>>>>>>>
>>>>>>>> Do I change this to
>>>>>>>>
>>>>>>>> poly = "${MASKDIR}/ABRFC.nc", "${MASKDIR}/CBRFC.nc",
>>>>>>>> ..."${MASKDIR}/WGRFC.nc"; ?
>>>>>>>>
>>>>>>>>
>>>>
>>
(MASKDIR=/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus/yl/masks/rfcs)
>>>>>>>> 3) This is more MET than METplus: in the attached
test_sum.sh, I
>> used
>>>>>>>> 'pcp_combine -sum' to sum up 12 1-h precip into a 12h total,
and all
>>>> the
>>>>>>>> hourly files are in
>>>> /gpfs/dell2/nco/ops/com/rtma/prod/pcprtma.20190825.
>>>>>>>> What I wanted to do initially was to sum up 24 1-h precip
into a 24h
>>>>>>>> (12Z-12Z) total and the hourly files are in
>>>> /gpfs/.../pcprtma.20190824
>>>>>>>> and /gpfs/.../pcprtma.20190825. Could you tell me how to
make the
>>>>>>>> -pcpdir argument a wildcard (like /gpfs/.../pcprtma.%Y%M%D)?
>>>>>>>>
>>>>>>>> I've been running point_stat in MET but have not run it
in
>> METplus
>>>>>>>> before, other than the use case
>>>>>>>> parm/use_cases/grid_to_obs/examples/conus_surface.conf.
Haven't
>> been
>>>>>>>> able to figure things out myself in the past few weeks. If
you
>> happen
>>>>>>>> to see anything else wrong in my set-up I would appreciate
your
>>>> pointing
>>>>>>>> them out too.
>>>>>>>>
>>>>>>>> Thank you -
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ying Lin
>>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
>
------------------------------------------------
More information about the Met_help
mailing list