[Met_help] [rt.rap.ucar.edu #91205] History for 3 METplus questions
Julie Prestopnik via RT
met_help at ucar.edu
Mon Aug 19 10:31:08 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. I have three questions about METplus2.1:
1) Running on Dell (Mars): getting a message I haven't seen on Tide/Gyre
before: metplus.GridStat (config_wrapper.py:144) ERROR: Executable
NCAP2_EXE does not exist at ncap2. Googling tells me that 'ncap2' is
likely the "netCDF arithmetic processor" and I found on Mars things like
/gpfs/dell1/usrx/local/prod/packages/ips/18.0.1/nco/4.7.0/man/ncap2.1.
Compare metplus_final.conf between the Mars run and the gyre run: I see
that
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/metplus_final.conf
has
NCAP2_EXE = /usrx/local/nco/4.2.4/bin/ncap2
While in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf,
NCAP2_EXE = ncap2
So on Mars my script doesn't know where 'ncap2' is. I don't see an
equivalent of gyre's .../bin/ncap2 under Mars' /gpfs/dell1/usrx/local.
Is there some module I should load on Mars to get the path for 'ncap2'?
The setup I have for running METplus on Mars is in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/dot.for.metplus
2) PcpCombine in METplus for GFS forecasts: what to do about the first
accumulation period when T0 file has no APCP field:
In https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/
I want to get 24h GFS precip accumulation for forecast periods of
00-24h, 12-36h, 24-48h, with
FCST_PCP_COMBINE_METHOD = SUBTRACT
i.e. using the continuously-accumulating bucket (config
file:https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/gfs_24h.conf).
The computations for the 12-36h and 24-48h had gone well, but trying to
get the 00-24h accumulation using the same pcpCombine failed:
/global/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine -subtract /gpfs/dell1/nco/ops/com/gfs/prod/gfs.20190719/12/gfs.t12z.pgrb2.0p25.f024 24 /gpfs/dell1/nco/ops/com/gfs/prod/gfs.20190719/12/gfs.t12z.pgrb2.0p25.f000 0 /meso/noscrub/Ying.Lin/metplus.out/gfs/bucket/fv3gfs.2019072012_f024_A24h -v 5
failed (No matching record found for 'APCP/A0' - https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/master_metplus.log.20190723).
It's understandable that trying to get 00-24h accumulation using
pcpCombine would fail since the zero hour model output does not contain
an APCP field. The logical thing to do is just to extract the 00-24h
accumulation contained in the gfs.t12z.pgrb2.0p25.f024 file and convert
it to netCDF. The question is, how to handle this within the framework
of METplus, e.g. how to tell METplus to run PcpCombine/SUBTRACT on all
accumulating periods except ones involving the initial hour, and for the
forecast hour that's equal to the length of accumulation, just convert
the amount to netCDF, so there'll be these netCDF files in
FCST_PCP_COMBINE_OUTPUT_DIR:
gfs.2019072012_f024_A24h (directly from gfs.t12z.pgrb2.0p25.f024)
gfs.2019072012_f036_A24h (from gfs.t00z.pgrb2.0p25.f036 and
/gfs.t00z.pgrb2.0p25.f012)
gfs.2019072012_f048_A24h (from gfs.t00z.pgrb2.0p25.f048 and
/gfs.t00z.pgrb2.0p25.f024)
3) pcp_combine -sum for verifying analysis:
In
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/gfs_24h.conf,
there is
OBS_PCP_COMBINE_METHOD = SUM
...
OBS_PCP_COMBINE_INPUT_TEMPLATE = ccpa.{valid?fmt=%Y%m%d}/{valid?fmt=%H}/ccpa.t{valid?fmt=%H}z.06h.hrap.conus.gb2
OBS_PCP_COMBINE_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}/ccpa.{valid?fmt=%Y%m%d%H}_A{level?fmt=%HH}h
(meant to sum up 6-hourly CCPAs to arrive at 24h CCPA)
This leads to the following err msg:
07/23 17:52:44.822 metplus.PcpCombine (command_runner.py:153) INFO: RUNNING: /global/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine -sum 20190719_12 -1 20190720_12 24 /meso/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx ccpa.t12z.06h.hrap.conus.gb2 -v 5 >> /meso/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190723 2>&1
ERROR :
ERROR : CommandLine::next_option() -> unrecognized command-line switch "-1"
(in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/gyre_run/master_metplus.log.20190723)
Is the setup for the OBS_QPE_COMBINE incorrect? I don't know how the
'-1' flag got into the pcp_combine and how to get rid of it.
Thank you for looking into this -
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: 3 METplus questions
From: Julie Prestopnik
Time: Wed Jul 24 10:48:11 2019
Hi Ying.
I see that you have some questions about METplus.
Regarding #1, on mars (and any of the WCOSS machines) you should be
able to
run the following:
module use /usrx/local/dev/modulefiles/
module load metplus/2.1
which should load the appropriate environment, including ncap2:
[Julie.Prestopnik at m71a3 ~]$ module use /usrx/local/dev/modulefiles/
>
> [Julie.Prestopnik at m71a3 ~]$ module load metplus/2.1
>
> [Julie.Prestopnik at m71a3 ~]$ which ncap2
>
> /usrx/local/dev/packages/ips/18.0.1/nco/4.7.0/bin/ncap2
>
I will follow up on the other two questions as soon as I am able.
Julie
------------------------------------------------
Subject: 3 METplus questions
From: Julie Prestopnik
Time: Wed Jul 24 11:03:02 2019
Hi Ying.
It seems that the last two issues are, unfortunately, bugs that we
need to
fix. George was able to quickly identify what the problems seem to
be.
Regarding #2, it sounds like we need to have a check in the subtract
mode
so if the lead time is equivalent to the desired accumulation, we will
run
in -add mode with one input file to essentially just pass through the
data
and reformat it. We will make a GitHub issue for this bug and include
it
in the 3.0 release.
Regarding #3, we believe that setting OBS_LEVEL = 0 would solve this
issue.
However, the user shouldn't have to set this to 0. For "-add" mode, if
OBS_LEVEL is not set, the code will try to figure out the accumulation
to
use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so we know
to
handle it this way. However, we use the same variable for "-subtract"
mode
and -1 does not make sense here. We need to modify the code to check
if it
is -1 and set it to 0 as a default for "-subtract" mode. We will make
a
GitHub issue and include it in the 3.0 release.
Thank you from George and the rest of the team for helping us to
identify
these bugs.
Julie
>
------------------------------------------------
Subject: 3 METplus questions
From: Ying Lin
Time: Wed Jul 24 14:17:22 2019
Hi Julie,
Thank you very much for your replies, that's really helpful.
For #1, I added the
module use/usrx/local/dev/modulefiles/
module load metplus/2.1
to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my metplus
run script) and eliminated my "load met/8.1", "load_grib_util/..."
etc.
since that's included in the metplus/2.1 module. Now the METplus run
can find the ncap2.
Thank you for adding #2 to GitHub issues and plan for this in v3.0.
For #3, do I add "OBS_LEVEL = 0" in the [config] block of
gfs_24h.conf?
I did so in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
and it does show up in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
but the "-1" still shows up below and causing an err msg:
07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117) INFO:
app_name is: pcp_combine, output sent to:
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153) INFO:
RUNNING:
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
-sum 20190719_12 -1 20190720_12 24
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
-pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
2>&1
ERROR :
ERROR : CommandLine::next_option() -> unrecognized command-line
switch "-1"
ERROR :
(seen in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724)
Could it be that I'm not placing the "OBS_LEVEL = 0" in the correct
location in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf?
Thank you again -
Ying
On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
> Hi Ying.
>
> It seems that the last two issues are, unfortunately, bugs that we
need to
> fix. George was able to quickly identify what the problems seem to
be.
>
> Regarding #2, it sounds like we need to have a check in the subtract
mode
> so if the lead time is equivalent to the desired accumulation, we
will run
> in -add mode with one input file to essentially just pass through
the data
> and reformat it. We will make a GitHub issue for this bug and
include it
> in the 3.0 release.
>
> Regarding #3, we believe that setting OBS_LEVEL = 0 would solve this
issue.
> However, the user shouldn't have to set this to 0. For "-add" mode,
if
> OBS_LEVEL is not set, the code will try to figure out the
accumulation to
> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so we
know to
> handle it this way. However, we use the same variable for "-
subtract" mode
> and -1 does not make sense here. We need to modify the code to check
if it
> is -1 and set it to 0 as a default for "-subtract" mode. We will
make a
> GitHub issue and include it in the 3.0 release.
>
> Thank you from George and the rest of the team for helping us to
identify
> these bugs.
>
> Julie
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: 3 METplus questions
From: George McCabe
Time: Wed Jul 24 14:24:33 2019
Hi Ying,
If you are using METplus 2.1 or later, try using
OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had forgotten that
I
changed this name.
On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>
> Hi Julie,
>
> Thank you very much for your replies, that's really helpful.
>
> For #1, I added the
>
> module use/usrx/local/dev/modulefiles/
> module load metplus/2.1
>
> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
> run script) and eliminated my "load met/8.1", "load_grib_util/..."
etc.
> since that's included in the metplus/2.1 module. Now the METplus
run
> can find the ncap2.
>
> Thank you for adding #2 to GitHub issues and plan for this in v3.0.
>
> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
gfs_24h.conf?
> I did so in
>
>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>
> and it does show up in
>
>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
>
> but the "-1" still shows up below and causing an err msg:
>
> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117) INFO:
> app_name is: pcp_combine, output sent to:
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153) INFO:
> RUNNING:
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> -sum 20190719_12 -1 20190720_12 24
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> 2>&1
> ERROR :
> ERROR : CommandLine::next_option() -> unrecognized command-line
switch
> "-1"
> ERROR :
> (seen in
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
> )
>
> Could it be that I'm not placing the "OBS_LEVEL = 0" in the correct
> location in
>
>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
> ?
>
> Thank you again -
> Ying
>
> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
> > Hi Ying.
> >
> > It seems that the last two issues are, unfortunately, bugs that we
need
> to
> > fix. George was able to quickly identify what the problems seem
to be.
> >
> > Regarding #2, it sounds like we need to have a check in the
subtract mode
> > so if the lead time is equivalent to the desired accumulation, we
will
> run
> > in -add mode with one input file to essentially just pass through
the
> data
> > and reformat it. We will make a GitHub issue for this bug and
include it
> > in the 3.0 release.
> >
> > Regarding #3, we believe that setting OBS_LEVEL = 0 would solve
this
> issue.
> > However, the user shouldn't have to set this to 0. For "-add"
mode, if
> > OBS_LEVEL is not set, the code will try to figure out the
accumulation to
> > use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so we
know to
> > handle it this way. However, we use the same variable for "-
subtract"
> mode
> > and -1 does not make sense here. We need to modify the code to
check if
> it
> > is -1 and set it to 0 as a default for "-subtract" mode. We will
make a
> > GitHub issue and include it in the 3.0 release.
> >
> > Thank you from George and the rest of the team for helping us to
identify
> > these bugs.
> >
> > Julie
> >
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91205] 3 METplus questions
From: Ying Lin
Time: Fri Jul 26 10:44:16 2019
Hi George,
Thanks for pointing that out. I went back to look/test run the
METplus 2.1 use_case of ruc-vs-s2grib.conf, imitated that in my gfs
conf
file for adding up the CCPA. That seems to work, when
OBS_PCP_COMBINE_METHOD is set to 'ADD'.
I remember pre-METplus, in MET itself "pcp_combine -add" requires
explicit listing of files to be added, while the '-sum' option means
MET
will search the data directory for data required. It seems that the
'ADD/SUM' options in METplus *_PCP_COMBINE_METHOD doesn't work quite
the
same way.
OBS_PCP_COMBINE_METHOD = ADD (as in the example
ruc-vs-s2grib.conf) works, with the resulting pcp_combine command
looking like this:
RUNNING:
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
-name APCP_24 -add
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/ccpa.t12z.06h.hrap.conus.gb2
6
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/06/ccpa.t06z.06h.hrap.conus.gb2
6
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/00/ccpa.t00z.06h.hrap.conus.gb2
6
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190719/18/ccpa.t18z.06h.hrap.conus.gb2
6
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
-v 5
with
OBS_PCP_COMBINE_METHOD = SUM,
there is this:
RUNNING:
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
-sum 20190719_12 06 20190720_12 24
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
-pcpdir /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12 -pcprx
ccpa.t12z.06h.hrap.conus.gb2 -v 5
Which fails because not all CCPA files required for the sum (four 6h
files into a 24h accumulation) are in
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/.
So it seems that in METplus the 'ADD' option actually got MET to
search
the directories and found the correct files, while 'SUM' doesn't mean
the same thing as 'sum' for MET.
Anyway, I'm happy with just using "OBS_PCP_COMBINE_METHOD = ADD".
Please feel free to close this ticket.
Ying
On 7/24/19 4:24 PM, George McCabe via RT wrote:
> Hi Ying,
>
> If you are using METplus 2.1 or later, try using
> OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had forgotten
that I
> changed this name.
>
> On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>
>> Hi Julie,
>>
>> Thank you very much for your replies, that's really helpful.
>>
>> For #1, I added the
>>
>> module use/usrx/local/dev/modulefiles/
>> module load metplus/2.1
>>
>> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
>> run script) and eliminated my "load met/8.1", "load_grib_util/..."
etc.
>> since that's included in the metplus/2.1 module. Now the METplus
run
>> can find the ncap2.
>>
>> Thank you for adding #2 to GitHub issues and plan for this in v3.0.
>>
>> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
gfs_24h.conf?
>> I did so in
>>
>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>>
>> and it does show up in
>>
>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
>>
>> but the "-1" still shows up below and causing an err msg:
>>
>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117) INFO:
>> app_name is: pcp_combine, output sent to:
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153) INFO:
>> RUNNING:
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>> -sum 20190719_12 -1 20190720_12 24
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
>> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>> 2>&1
>> ERROR :
>> ERROR : CommandLine::next_option() -> unrecognized command-line
switch
>> "-1"
>> ERROR :
>> (seen in
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
>> )
>>
>> Could it be that I'm not placing the "OBS_LEVEL = 0" in the correct
>> location in
>>
>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>> ?
>>
>> Thank you again -
>> Ying
>>
>> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
>>> Hi Ying.
>>>
>>> It seems that the last two issues are, unfortunately, bugs that we
need
>> to
>>> fix. George was able to quickly identify what the problems seem
to be.
>>>
>>> Regarding #2, it sounds like we need to have a check in the
subtract mode
>>> so if the lead time is equivalent to the desired accumulation, we
will
>> run
>>> in -add mode with one input file to essentially just pass through
the
>> data
>>> and reformat it. We will make a GitHub issue for this bug and
include it
>>> in the 3.0 release.
>>>
>>> Regarding #3, we believe that setting OBS_LEVEL = 0 would solve
this
>> issue.
>>> However, the user shouldn't have to set this to 0. For "-add"
mode, if
>>> OBS_LEVEL is not set, the code will try to figure out the
accumulation to
>>> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so we
know to
>>> handle it this way. However, we use the same variable for "-
subtract"
>> mode
>>> and -1 does not make sense here. We need to modify the code to
check if
>> it
>>> is -1 and set it to 0 as a default for "-subtract" mode. We will
make a
>>> GitHub issue and include it in the 3.0 release.
>>>
>>> Thank you from George and the rest of the team for helping us to
identify
>>> these bugs.
>>>
>>> Julie
>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: 3 METplus questions
From: George McCabe
Time: Fri Jul 26 10:51:21 2019
Hi Ying,
Are you saying that METplus should call pcp_combine in -sum mode just
specifying the directory, not an actual file, and it will find the
data?
Sum mode in METplus has not been used nearly as much as Add mode so it
has
not been run through as many use cases. We want METplus to run MET in
the
way you would run it without the wrappers generating the commands.
On Fri, Jul 26, 2019 at 10:44 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>
> Hi George,
>
> Thanks for pointing that out. I went back to look/test run the
> METplus 2.1 use_case of ruc-vs-s2grib.conf, imitated that in my gfs
conf
> file for adding up the CCPA. That seems to work, when
> OBS_PCP_COMBINE_METHOD is set to 'ADD'.
>
> I remember pre-METplus, in MET itself "pcp_combine -add"
requires
> explicit listing of files to be added, while the '-sum' option means
MET
> will search the data directory for data required. It seems that the
> 'ADD/SUM' options in METplus *_PCP_COMBINE_METHOD doesn't work quite
the
> same way.
> OBS_PCP_COMBINE_METHOD = ADD (as in the example
> ruc-vs-s2grib.conf) works, with the resulting pcp_combine command
> looking like this:
>
> RUNNING:
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>
> -name APCP_24 -add
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/ccpa.t12z.06h.hrap.conus.gb2
>
> 6
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/06/ccpa.t06z.06h.hrap.conus.gb2
>
> 6
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/00/ccpa.t00z.06h.hrap.conus.gb2
>
> 6
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190719/18/ccpa.t18z.06h.hrap.conus.gb2
>
> 6
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>
> -v 5
>
> with
> OBS_PCP_COMBINE_METHOD = SUM,
> there is this:
>
> RUNNING:
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>
> -sum 20190719_12 06 20190720_12 24
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>
> -pcpdir /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12 -pcprx
> ccpa.t12z.06h.hrap.conus.gb2 -v 5
>
> Which fails because not all CCPA files required for the sum (four 6h
> files into a 24h accumulation) are in
> /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/.
>
> So it seems that in METplus the 'ADD' option actually got MET to
search
> the directories and found the correct files, while 'SUM' doesn't
mean
> the same thing as 'sum' for MET.
>
> Anyway, I'm happy with just using "OBS_PCP_COMBINE_METHOD = ADD".
> Please feel free to close this ticket.
>
> Ying
>
> On 7/24/19 4:24 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > If you are using METplus 2.1 or later, try using
> > OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had forgotten
that I
> > changed this name.
> >
> > On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
> >>
> >> Hi Julie,
> >>
> >> Thank you very much for your replies, that's really
helpful.
> >>
> >> For #1, I added the
> >>
> >> module use/usrx/local/dev/modulefiles/
> >> module load metplus/2.1
> >>
> >> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
> >> run script) and eliminated my "load met/8.1",
"load_grib_util/..." etc.
> >> since that's included in the metplus/2.1 module. Now the METplus
run
> >> can find the ncap2.
> >>
> >> Thank you for adding #2 to GitHub issues and plan for this in
v3.0.
> >>
> >> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
gfs_24h.conf?
> >> I did so in
> >>
> >>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
> >>
> >> and it does show up in
> >>
> >>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
> >>
> >> but the "-1" still shows up below and causing an err msg:
> >>
> >> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117)
INFO:
> >> app_name is: pcp_combine, output sent to:
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> >> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153)
INFO:
> >> RUNNING:
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> >> -sum 20190719_12 -1 20190720_12 24
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
> >> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
> >> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> >> 2>&1
> >> ERROR :
> >> ERROR : CommandLine::next_option() -> unrecognized command-line
switch
> >> "-1"
> >> ERROR :
> >> (seen in
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
> >> )
> >>
> >> Could it be that I'm not placing the "OBS_LEVEL = 0" in the
correct
> >> location in
> >>
> >>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
> >> ?
> >>
> >> Thank you again -
> >> Ying
> >>
> >> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
> >>> Hi Ying.
> >>>
> >>> It seems that the last two issues are, unfortunately, bugs that
we need
> >> to
> >>> fix. George was able to quickly identify what the problems seem
to be.
> >>>
> >>> Regarding #2, it sounds like we need to have a check in the
subtract
> mode
> >>> so if the lead time is equivalent to the desired accumulation,
we will
> >> run
> >>> in -add mode with one input file to essentially just pass
through the
> >> data
> >>> and reformat it. We will make a GitHub issue for this bug and
include
> it
> >>> in the 3.0 release.
> >>>
> >>> Regarding #3, we believe that setting OBS_LEVEL = 0 would solve
this
> >> issue.
> >>> However, the user shouldn't have to set this to 0. For "-add"
mode, if
> >>> OBS_LEVEL is not set, the code will try to figure out the
accumulation
> to
> >>> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so
we know
> to
> >>> handle it this way. However, we use the same variable for "-
subtract"
> >> mode
> >>> and -1 does not make sense here. We need to modify the code to
check if
> >> it
> >>> is -1 and set it to 0 as a default for "-subtract" mode. We will
make a
> >>> GitHub issue and include it in the 3.0 release.
> >>>
> >>> Thank you from George and the rest of the team for helping us to
> identify
> >>> these bugs.
> >>>
> >>> Julie
> >>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91205] 3 METplus questions
From: Ying Lin
Time: Fri Jul 26 11:05:52 2019
Hi George,
I'm not saying METplus should call pcp_combine in -sum mode. I'm
saying that the METplus ADD option works - at least in the couple of
cases I'm seeing - somehow through the wrapper it told MET to look for
the correct files to add up. SUM doesn't do that. I'm happy with the
'ADD'. It just seemed a bit counter-intuitive, that's all.
Ying
On 7/26/19 12:51 PM, George McCabe via RT wrote:
> Hi Ying,
>
> Are you saying that METplus should call pcp_combine in -sum mode
just
> specifying the directory, not an actual file, and it will find the
data?
> Sum mode in METplus has not been used nearly as much as Add mode so
it has
> not been run through as many use cases. We want METplus to run MET
in the
> way you would run it without the wrappers generating the commands.
>
> On Fri, Jul 26, 2019 at 10:44 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>
>> Hi George,
>>
>> Thanks for pointing that out. I went back to look/test run
the
>> METplus 2.1 use_case of ruc-vs-s2grib.conf, imitated that in my gfs
conf
>> file for adding up the CCPA. That seems to work, when
>> OBS_PCP_COMBINE_METHOD is set to 'ADD'.
>>
>> I remember pre-METplus, in MET itself "pcp_combine -add"
requires
>> explicit listing of files to be added, while the '-sum' option
means MET
>> will search the data directory for data required. It seems that
the
>> 'ADD/SUM' options in METplus *_PCP_COMBINE_METHOD doesn't work
quite the
>> same way.
>> OBS_PCP_COMBINE_METHOD = ADD (as in the example
>> ruc-vs-s2grib.conf) works, with the resulting pcp_combine command
>> looking like this:
>>
>> RUNNING:
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>
>> -name APCP_24 -add
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/ccpa.t12z.06h.hrap.conus.gb2
>>
>> 6
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/06/ccpa.t06z.06h.hrap.conus.gb2
>>
>> 6
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/00/ccpa.t00z.06h.hrap.conus.gb2
>>
>> 6
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190719/18/ccpa.t18z.06h.hrap.conus.gb2
>>
>> 6
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>
>> -v 5
>>
>> with
>> OBS_PCP_COMBINE_METHOD = SUM,
>> there is this:
>>
>> RUNNING:
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>
>> -sum 20190719_12 06 20190720_12 24
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>
>> -pcpdir /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12 -pcprx
>> ccpa.t12z.06h.hrap.conus.gb2 -v 5
>>
>> Which fails because not all CCPA files required for the sum (four
6h
>> files into a 24h accumulation) are in
>> /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/.
>>
>> So it seems that in METplus the 'ADD' option actually got MET to
search
>> the directories and found the correct files, while 'SUM' doesn't
mean
>> the same thing as 'sum' for MET.
>>
>> Anyway, I'm happy with just using "OBS_PCP_COMBINE_METHOD = ADD".
>> Please feel free to close this ticket.
>>
>> Ying
>>
>> On 7/24/19 4:24 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> If you are using METplus 2.1 or later, try using
>>> OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had forgotten
that I
>>> changed this name.
>>>
>>> On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>>>
>>>> Hi Julie,
>>>>
>>>> Thank you very much for your replies, that's really
helpful.
>>>>
>>>> For #1, I added the
>>>>
>>>> module use/usrx/local/dev/modulefiles/
>>>> module load metplus/2.1
>>>>
>>>> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
>>>> run script) and eliminated my "load met/8.1",
"load_grib_util/..." etc.
>>>> since that's included in the metplus/2.1 module. Now the METplus
run
>>>> can find the ncap2.
>>>>
>>>> Thank you for adding #2 to GitHub issues and plan for this in
v3.0.
>>>>
>>>> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
gfs_24h.conf?
>>>> I did so in
>>>>
>>>>
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>>>> and it does show up in
>>>>
>>>>
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
>>>> but the "-1" still shows up below and causing an err msg:
>>>>
>>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117)
INFO:
>>>> app_name is: pcp_combine, output sent to:
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153)
INFO:
>>>> RUNNING:
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>>> -sum 20190719_12 -1 20190720_12 24
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>>> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
>>>> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>>>> 2>&1
>>>> ERROR :
>>>> ERROR : CommandLine::next_option() -> unrecognized command-line
switch
>>>> "-1"
>>>> ERROR :
>>>> (seen in
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
>>>> )
>>>>
>>>> Could it be that I'm not placing the "OBS_LEVEL = 0" in the
correct
>>>> location in
>>>>
>>>>
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>>>> ?
>>>>
>>>> Thank you again -
>>>> Ying
>>>>
>>>> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
>>>>> Hi Ying.
>>>>>
>>>>> It seems that the last two issues are, unfortunately, bugs that
we need
>>>> to
>>>>> fix. George was able to quickly identify what the problems seem
to be.
>>>>>
>>>>> Regarding #2, it sounds like we need to have a check in the
subtract
>> mode
>>>>> so if the lead time is equivalent to the desired accumulation,
we will
>>>> run
>>>>> in -add mode with one input file to essentially just pass
through the
>>>> data
>>>>> and reformat it. We will make a GitHub issue for this bug and
include
>> it
>>>>> in the 3.0 release.
>>>>>
>>>>> Regarding #3, we believe that setting OBS_LEVEL = 0 would solve
this
>>>> issue.
>>>>> However, the user shouldn't have to set this to 0. For "-add"
mode, if
>>>>> OBS_LEVEL is not set, the code will try to figure out the
accumulation
>> to
>>>>> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so
we know
>> to
>>>>> handle it this way. However, we use the same variable for "-
subtract"
>>>> mode
>>>>> and -1 does not make sense here. We need to modify the code to
check if
>>>> it
>>>>> is -1 and set it to 0 as a default for "-subtract" mode. We will
make a
>>>>> GitHub issue and include it in the 3.0 release.
>>>>>
>>>>> Thank you from George and the rest of the team for helping us to
>> identify
>>>>> these bugs.
>>>>>
>>>>> Julie
>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: 3 METplus questions
From: George McCabe
Time: Fri Jul 26 11:23:37 2019
I'd like to get SUM mode to call it correctly. ADD mode has a lot of
logic
to look for the files it needs to build the accumulation. Currently
SUM
mode doesn't have logic to search for files so intelligently, it just
builds the command and let's pcp_combine search. How would you have
called
SUM mode for that case? Any new use cases that show examples of
different
ways to call apps through METplus are always welcome and appreciated!
On Fri, Jul 26, 2019 at 11:05 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>
> Hi George,
>
> I'm not saying METplus should call pcp_combine in -sum mode.
I'm
> saying that the METplus ADD option works - at least in the couple of
> cases I'm seeing - somehow through the wrapper it told MET to look
for
> the correct files to add up. SUM doesn't do that. I'm happy with
the
> 'ADD'. It just seemed a bit counter-intuitive, that's all.
>
> Ying
>
> On 7/26/19 12:51 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > Are you saying that METplus should call pcp_combine in -sum mode
just
> > specifying the directory, not an actual file, and it will find the
data?
> > Sum mode in METplus has not been used nearly as much as Add mode
so it
> has
> > not been run through as many use cases. We want METplus to run MET
in the
> > way you would run it without the wrappers generating the commands.
> >
> > On Fri, Jul 26, 2019 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
> >>
> >> Hi George,
> >>
> >> Thanks for pointing that out. I went back to look/test run
the
> >> METplus 2.1 use_case of ruc-vs-s2grib.conf, imitated that in my
gfs conf
> >> file for adding up the CCPA. That seems to work, when
> >> OBS_PCP_COMBINE_METHOD is set to 'ADD'.
> >>
> >> I remember pre-METplus, in MET itself "pcp_combine -add"
requires
> >> explicit listing of files to be added, while the '-sum' option
means MET
> >> will search the data directory for data required. It seems that
the
> >> 'ADD/SUM' options in METplus *_PCP_COMBINE_METHOD doesn't work
quite the
> >> same way.
> >> OBS_PCP_COMBINE_METHOD = ADD (as in the example
> >> ruc-vs-s2grib.conf) works, with the resulting pcp_combine command
> >> looking like this:
> >>
> >> RUNNING:
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> >>
> >> -name APCP_24 -add
> >>
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/ccpa.t12z.06h.hrap.conus.gb2
> >>
> >> 6
> >>
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/06/ccpa.t06z.06h.hrap.conus.gb2
> >>
> >> 6
> >>
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/00/ccpa.t00z.06h.hrap.conus.gb2
> >>
> >> 6
> >>
>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190719/18/ccpa.t18z.06h.hrap.conus.gb2
> >>
> >> 6
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
> >>
> >> -v 5
> >>
> >> with
> >> OBS_PCP_COMBINE_METHOD = SUM,
> >> there is this:
> >>
> >> RUNNING:
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> >>
> >> -sum 20190719_12 06 20190720_12 24
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
> >>
> >> -pcpdir /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12 -pcprx
> >> ccpa.t12z.06h.hrap.conus.gb2 -v 5
> >>
> >> Which fails because not all CCPA files required for the sum (four
6h
> >> files into a 24h accumulation) are in
> >> /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/.
> >>
> >> So it seems that in METplus the 'ADD' option actually got MET to
search
> >> the directories and found the correct files, while 'SUM' doesn't
mean
> >> the same thing as 'sum' for MET.
> >>
> >> Anyway, I'm happy with just using "OBS_PCP_COMBINE_METHOD = ADD".
> >> Please feel free to close this ticket.
> >>
> >> Ying
> >>
> >> On 7/24/19 4:24 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> If you are using METplus 2.1 or later, try using
> >>> OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had
forgotten that
> I
> >>> changed this name.
> >>>
> >>> On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
> >>>>
> >>>> Hi Julie,
> >>>>
> >>>> Thank you very much for your replies, that's really
helpful.
> >>>>
> >>>> For #1, I added the
> >>>>
> >>>> module use/usrx/local/dev/modulefiles/
> >>>> module load metplus/2.1
> >>>>
> >>>> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
> >>>> run script) and eliminated my "load met/8.1",
"load_grib_util/..."
> etc.
> >>>> since that's included in the metplus/2.1 module. Now the
METplus run
> >>>> can find the ncap2.
> >>>>
> >>>> Thank you for adding #2 to GitHub issues and plan for this in
v3.0.
> >>>>
> >>>> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
> gfs_24h.conf?
> >>>> I did so in
> >>>>
> >>>>
> >>>>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
> >>>> and it does show up in
> >>>>
> >>>>
> >>>>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
> >>>> but the "-1" still shows up below and causing an err msg:
> >>>>
> >>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117)
INFO:
> >>>> app_name is: pcp_combine, output sent to:
> >>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> >>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153)
INFO:
> >>>> RUNNING:
> >>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> >>>> -sum 20190719_12 -1 20190720_12 24
> >>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
> >>>> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
> >>>> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
> >>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
> >>>> 2>&1
> >>>> ERROR :
> >>>> ERROR : CommandLine::next_option() -> unrecognized command-
line
> switch
> >>>> "-1"
> >>>> ERROR :
> >>>> (seen in
> >>>>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
> >>>> )
> >>>>
> >>>> Could it be that I'm not placing the "OBS_LEVEL = 0" in the
correct
> >>>> location in
> >>>>
> >>>>
> >>>>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
> >>>> ?
> >>>>
> >>>> Thank you again -
> >>>> Ying
> >>>>
> >>>> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
> >>>>> Hi Ying.
> >>>>>
> >>>>> It seems that the last two issues are, unfortunately, bugs
that we
> need
> >>>> to
> >>>>> fix. George was able to quickly identify what the problems
seem to
> be.
> >>>>>
> >>>>> Regarding #2, it sounds like we need to have a check in the
subtract
> >> mode
> >>>>> so if the lead time is equivalent to the desired accumulation,
we
> will
> >>>> run
> >>>>> in -add mode with one input file to essentially just pass
through the
> >>>> data
> >>>>> and reformat it. We will make a GitHub issue for this bug and
> include
> >> it
> >>>>> in the 3.0 release.
> >>>>>
> >>>>> Regarding #3, we believe that setting OBS_LEVEL = 0 would
solve this
> >>>> issue.
> >>>>> However, the user shouldn't have to set this to 0. For "-add"
mode,
> if
> >>>>> OBS_LEVEL is not set, the code will try to figure out the
> accumulation
> >> to
> >>>>> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so
we
> know
> >> to
> >>>>> handle it this way. However, we use the same variable for "-
subtract"
> >>>> mode
> >>>>> and -1 does not make sense here. We need to modify the code to
check
> if
> >>>> it
> >>>>> is -1 and set it to 0 as a default for "-subtract" mode. We
will
> make a
> >>>>> GitHub issue and include it in the 3.0 release.
> >>>>>
> >>>>> Thank you from George and the rest of the team for helping us
to
> >> identify
> >>>>> these bugs.
> >>>>>
> >>>>> Julie
> >>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91205] 3 METplus questions
From: Ying Lin
Time: Fri Jul 26 12:39:21 2019
Hi George,
Actually I didn't mean to ask for further development to make the
'SUM' option work. Initially I thought the option to use would be
'SUM'
(thinking back on the -add/-sum options for pcp_combine). When that
didn't work and seeing that 'ADD' was used in the RUC example, I
switched to ADD, and it seemed ADD worked like the functional
equivalent of '-sum' - not that it called 'pcp_combine -sum', but
somehow it gathered the correct files in different directories and
listed the files with the correct paths for "pcp_combine -add" to
work. Since as you said ADD has a lot of logic to look for the files
it needs to build the accumulation, I'm happy with ADD - seems like
it's
already functioning like what I thought SUM would have done, at least
in
the limited cases I looked at.
Thank you for your patience. Please feel free to close this
ticket
as you see fit.
Ying
On 7/26/19 1:23 PM, George McCabe via RT wrote:
> I'd like to get SUM mode to call it correctly. ADD mode has a lot of
logic
> to look for the files it needs to build the accumulation. Currently
SUM
> mode doesn't have logic to search for files so intelligently, it
just
> builds the command and let's pcp_combine search. How would you have
called
> SUM mode for that case? Any new use cases that show examples of
different
> ways to call apps through METplus are always welcome and
appreciated!
>
> On Fri, Jul 26, 2019 at 11:05 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>
>> Hi George,
>>
>> I'm not saying METplus should call pcp_combine in -sum mode.
I'm
>> saying that the METplus ADD option works - at least in the couple
of
>> cases I'm seeing - somehow through the wrapper it told MET to look
for
>> the correct files to add up. SUM doesn't do that. I'm happy with
the
>> 'ADD'. It just seemed a bit counter-intuitive, that's all.
>>
>> Ying
>>
>> On 7/26/19 12:51 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> Are you saying that METplus should call pcp_combine in -sum mode
just
>>> specifying the directory, not an actual file, and it will find the
data?
>>> Sum mode in METplus has not been used nearly as much as Add mode
so it
>> has
>>> not been run through as many use cases. We want METplus to run MET
in the
>>> way you would run it without the wrappers generating the commands.
>>>
>>> On Fri, Jul 26, 2019 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>>>
>>>> Hi George,
>>>>
>>>> Thanks for pointing that out. I went back to look/test
run the
>>>> METplus 2.1 use_case of ruc-vs-s2grib.conf, imitated that in my
gfs conf
>>>> file for adding up the CCPA. That seems to work, when
>>>> OBS_PCP_COMBINE_METHOD is set to 'ADD'.
>>>>
>>>> I remember pre-METplus, in MET itself "pcp_combine -add"
requires
>>>> explicit listing of files to be added, while the '-sum' option
means MET
>>>> will search the data directory for data required. It seems that
the
>>>> 'ADD/SUM' options in METplus *_PCP_COMBINE_METHOD doesn't work
quite the
>>>> same way.
>>>> OBS_PCP_COMBINE_METHOD = ADD (as in the example
>>>> ruc-vs-s2grib.conf) works, with the resulting pcp_combine command
>>>> looking like this:
>>>>
>>>> RUNNING:
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>>> -name APCP_24 -add
>>>>
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/ccpa.t12z.06h.hrap.conus.gb2
>>>> 6
>>>>
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/06/ccpa.t06z.06h.hrap.conus.gb2
>>>> 6
>>>>
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/00/ccpa.t00z.06h.hrap.conus.gb2
>>>> 6
>>>>
>>
/gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190719/18/ccpa.t18z.06h.hrap.conus.gb2
>>>> 6
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>>> -v 5
>>>>
>>>> with
>>>> OBS_PCP_COMBINE_METHOD = SUM,
>>>> there is this:
>>>>
>>>> RUNNING:
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>>> -sum 20190719_12 06 20190720_12 24
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>>> -pcpdir /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12 -pcprx
>>>> ccpa.t12z.06h.hrap.conus.gb2 -v 5
>>>>
>>>> Which fails because not all CCPA files required for the sum (four
6h
>>>> files into a 24h accumulation) are in
>>>> /gpfs/tp2/nco/ops/com/ccpa/prod/ccpa.20190720/12/.
>>>>
>>>> So it seems that in METplus the 'ADD' option actually got MET to
search
>>>> the directories and found the correct files, while 'SUM' doesn't
mean
>>>> the same thing as 'sum' for MET.
>>>>
>>>> Anyway, I'm happy with just using "OBS_PCP_COMBINE_METHOD = ADD".
>>>> Please feel free to close this ticket.
>>>>
>>>> Ying
>>>>
>>>> On 7/24/19 4:24 PM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> If you are using METplus 2.1 or later, try using
>>>>> OBS_PCP_COMBINE_INPUT_LEVEL instead of OBS_LEVEL. I had
forgotten that
>> I
>>>>> changed this name.
>>>>>
>>>>> On Wed, Jul 24, 2019 at 2:17 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91205 >
>>>>>>
>>>>>> Hi Julie,
>>>>>>
>>>>>> Thank you very much for your replies, that's really
helpful.
>>>>>>
>>>>>> For #1, I added the
>>>>>>
>>>>>> module use/usrx/local/dev/modulefiles/
>>>>>> module load metplus/2.1
>>>>>>
>>>>>> to my dot.for.metplus (doing ". ~/dots/dot.for.metplus" in my
metplus
>>>>>> run script) and eliminated my "load met/8.1",
"load_grib_util/..."
>> etc.
>>>>>> since that's included in the metplus/2.1 module. Now the
METplus run
>>>>>> can find the ncap2.
>>>>>>
>>>>>> Thank you for adding #2 to GitHub issues and plan for this in
v3.0.
>>>>>>
>>>>>> For #3, do I add "OBS_LEVEL = 0" in the [config] block of
>> gfs_24h.conf?
>>>>>> I did so in
>>>>>>
>>>>>>
>>>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>>>>>> and it does show up in
>>>>>>
>>>>>>
>>>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/metplus_final.conf
>>>>>> but the "-1" still shows up below and causing an err msg:
>>>>>>
>>>>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:117)
INFO:
>>>>>> app_name is: pcp_combine, output sent to:
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>>>>>> 07/24 20:07:39.792 metplus.PcpCombine (command_runner.py:153)
INFO:
>>>>>> RUNNING:
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>>>>>> -sum 20190719_12 -1 20190720_12 24
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/ccpa/bucket/20190720/ccpa.2019072012_A24h
>>>>>> -pcpdir /com2/ccpa/prod/ccpa.20190720/12 -pcprx
>>>>>> ccpa.t12z.06h.hrap.conus.gb2 -v 5 >>
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/logs/master_metplus.log.20190724
>>>>>> 2>&1
>>>>>> ERROR :
>>>>>> ERROR : CommandLine::next_option() -> unrecognized command-
line
>> switch
>>>>>> "-1"
>>>>>> ERROR :
>>>>>> (seen in
>>>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/master_metplus.log.20190724
>>>>>> )
>>>>>>
>>>>>> Could it be that I'm not placing the "OBS_LEVEL = 0" in the
correct
>>>>>> location in
>>>>>>
>>>>>>
>>>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/mars_run/gfs_24h.conf
>>>>>> ?
>>>>>>
>>>>>> Thank you again -
>>>>>> Ying
>>>>>>
>>>>>> On 7/24/19 1:03 PM, Julie Prestopnik via RT wrote:
>>>>>>> Hi Ying.
>>>>>>>
>>>>>>> It seems that the last two issues are, unfortunately, bugs
that we
>> need
>>>>>> to
>>>>>>> fix. George was able to quickly identify what the problems
seem to
>> be.
>>>>>>> Regarding #2, it sounds like we need to have a check in the
subtract
>>>> mode
>>>>>>> so if the lead time is equivalent to the desired accumulation,
we
>> will
>>>>>> run
>>>>>>> in -add mode with one input file to essentially just pass
through the
>>>>>> data
>>>>>>> and reformat it. We will make a GitHub issue for this bug and
>> include
>>>> it
>>>>>>> in the 3.0 release.
>>>>>>>
>>>>>>> Regarding #3, we believe that setting OBS_LEVEL = 0 would
solve this
>>>>>> issue.
>>>>>>> However, the user shouldn't have to set this to 0. For "-add"
mode,
>> if
>>>>>>> OBS_LEVEL is not set, the code will try to figure out the
>> accumulation
>>>> to
>>>>>>> use. The code sets OBS_LEVEL to -1, if OBS_LEVEL is not set so
we
>> know
>>>> to
>>>>>>> handle it this way. However, we use the same variable for "-
subtract"
>>>>>> mode
>>>>>>> and -1 does not make sense here. We need to modify the code to
check
>> if
>>>>>> it
>>>>>>> is -1 and set it to 0 as a default for "-subtract" mode. We
will
>> make a
>>>>>>> GitHub issue and include it in the 3.0 release.
>>>>>>>
>>>>>>> Thank you from George and the rest of the team for helping us
to
>>>> identify
>>>>>>> these bugs.
>>>>>>>
>>>>>>> Julie
>>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list