[Met_help] [rt.rap.ucar.edu #90341] History for MET Pcp-Combine error (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Wed May 29 15:41:20 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I haven't seen this type of error before. Do you have a suggestion for
resolving this? See attached observation file.
> run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
DEBUG 1: Reading input file:
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_01H_00
.00_20160209-120000.grib2
ERROR :
ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
ERROR :
ERROR : line = 2
ERROR :
ERROR : column = 1
ERROR :
ERROR : text = "(nul)"
ERROR :
ERROR :
ERROR : (nul)
ERROR : (nul)
ERROR :
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri May 24 11:25:32 2019
John,
There is most likely a syntax error in your script named
"run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't a
problem in your data. You're probably just missing a semi-colon,
single
quote, or double quote somewhere on the command line for pcp_combine.
If
you're not able to find it, just send me the contents of that script
and it
should be pretty easy to locate.
Thanks,
John
On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET Pcp-Combine error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I haven't seen this type of error before. Do you have a suggestion
for
> resolving this? See attached observation file.
>
> > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> DEBUG 1: Reading input file:
>
>
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_01H_00
> .00_20160209-120000.grib2
> ERROR :
> ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri May 24 13:05:17 2019
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for looking at the input observation file. I haven't used this
type of
file before and if that is not causing the problem then I just need to
find
the syntax error in the script. I'm using the -field option because
the
parameter name isn't APCP/A01. When I looked at the metadata, it
looked like
the parameter name was "GaugeCorrQPE01H". Is this what I should put
after -field?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, May 24, 2019 11:26 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-Combine
error
(UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
There is most likely a syntax error in your script named
"run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't a
problem
in your data. You're probably just missing a semi-colon, single
quote, or
double quote somewhere on the command line for pcp_combine. If you're
not
able to find it, just send me the contents of that script and it
should be
pretty easy to locate.
Thanks,
John
On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET Pcp-Combine error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I haven't seen this type of error before. Do you have a suggestion
for
> resolving this? See attached observation file.
>
> > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> DEBUG 1: Reading input file:
>
>
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_
> 01H_00
> .00_20160209-120000.grib2
> ERROR :
> ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue May 28 14:28:33 2019
John -
I've been trying to locate the syntax error in my run command in the
attached shell script, but unable to see it. Could you find it for me?
I've also attached the log file showing the error and some
intermediate output file produced my MET which is shown in the log
file. You already have the observation file (grib2).
Thanks.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, May 24, 2019 11:25 AM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-Combine
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
There is most likely a syntax error in your script named
"run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't a
problem in your data. You're probably just missing a semi-colon,
single
quote, or double quote somewhere on the command line for pcp_combine.
If
you're not able to find it, just send me the contents of that script
and it
should be pretty easy to locate.
Thanks,
John
On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET Pcp-Combine error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I haven't seen this type of error before. Do you have a suggestion
for
> resolving this? See attached observation file.
>
> > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> DEBUG 1: Reading input file:
>
>
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_01H_00
> .00_20160209-120000.grib2
> ERROR :
> ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 29 09:44:54 2019
John,
OK, I'm able to replicate the error message you're seeing. Here's the
first command listed in your script (with slightly modified paths):
*/usr/local/met-5.2/bin/pcp_combine \*
*-subtract \*
*GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
*GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
*MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
*-field GaugeCorrQPE01H \*
*-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
I see that you're running met-5.2. The latest available version is
8.1
which includes many bugfixes and enhancements. If you're able to
upgrade
to that, I'd encourage you to do so.
Next, I see that you're subtracting the same field from itself in this
command... which should result in 0's everywhere. That's fine, if it
makes
your scripting easier, but I just wanted to mention it.
I see that you want to extract 1-hourly precip from this file. And
running
it though wgrib2 reveals that the field is named "GaugeCorrQPE01H", as
you've already figured out. By default, pcp_combine looks for fields
of
accumulated precip, named "APCP". And so you're using the "-field"
command
line option to override that. There are 2 problems here. First, the
"-field" option needs to specify a "name" and "level". Second, this
GRIB2
file doesn't contain a 1-hour accumulation. Since it's a 0-hour
analysis
file, it contains 0-hour precip. So we'd need to use "A0" to access
it.
Here are 3 variations that all do what you're trying to do:
(1) Using the -field option as you intended (the -field option
overrides
the 01 you have elsewhere):
/usr/local/met-5.2/bin/pcp_combine -subtract
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
-field 'name="GaugeCorrQPE01H"; level="A00";' -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
(2) Instead of using "-field" just explicitly list the data you want
from
each file:
/usr/local/met-5.2/bin/pcp_combine -subtract
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 'name="GaugeCorrQPE01H";
level="A00";' GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
(3) Since you're really just running this as a pass-through for the 0-
hour
forecast, you could run a -add command for a single field instead:
/usr/local/met-5.2/bin/pcp_combine -add
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 'name="GaugeCorrQPE01H";
level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
All 3 options should give you the same result.
John
On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> John -
>
> I've been trying to locate the syntax error in my run command in the
> attached shell script, but unable to see it. Could you find it for
me? I've
> also attached the log file showing the error and some intermediate
output
> file produced my MET which is shown in the log file. You already
have the
> observation file (grib2).
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, May 24, 2019 11:25 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> There is most likely a syntax error in your script named
> "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't a
> problem in your data. You're probably just missing a semi-colon,
single
> quote, or double quote somewhere on the command line for
pcp_combine. If
> you're not able to find it, just send me the contents of that script
and it
> should be pretty easy to locate.
>
> Thanks,
> John
>
>
>
> On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I haven't seen this type of error before. Do you have a suggestion
for
> > resolving this? See attached observation file.
> >
> > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > DEBUG 1: Reading input file:
> >
> >
>
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_01H_00
> > .00_20160209-120000.grib2
> > ERROR :
> > ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
> > ERROR :
> > ERROR : line = 2
> > ERROR :
> > ERROR : column = 1
> > ERROR :
> > ERROR : text = "(nul)"
> > ERROR :
> > ERROR :
> > ERROR : (nul)
> > ERROR : (nul)
> > ERROR :
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 29 10:21:46 2019
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for the testing and diagnosis and providing 3 options for
accomplishing
the subtraction. I intentionally wanted to zero-out the accum precip
(analysis
of 1-hr precip valid at 12Z) field at 12Z in order to track the model
development of the precip specifically over the period from 12Z to 18Z
with 0
as the starting point for this case study. In my script, the
additional
commands (commented out now) I use "-sum" to accumulate for the
following
hours up to 18Z. For the -sum I assume that I will use the same
options to
describe the name and level per your examples below, correct? I'm
assuming
that if I have the same analysis fields for 13Z, 14Z, 15Z, ...., 18Z,
I can
just sum them per the other commands in my script to get the 6-hr
accum at
18Z. Will this work?
I have V8.1 running in a Singularity container on an ARL HPC
(Centennial) ,
but I haven't provisioned this HPC with all the data I need to start
using it
operationally. I plan to soon, however. Looking forward to the
enhancements.
I'm running V5.2 on the operational system (Excalibur). Unfortunately,
the
kernel is older and the implementation of the V8.1 Singularity
container has
being problematic. Working on it.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, May 29, 2019 9:45 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
OK, I'm able to replicate the error message you're seeing. Here's the
first
command listed in your script (with slightly modified paths):
*/usr/local/met-5.2/bin/pcp_combine \*
*-subtract \*
*GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
*GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
*MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
*-field GaugeCorrQPE01H \*
*-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
I see that you're running met-5.2. The latest available version is
8.1 which
includes many bugfixes and enhancements. If you're able to upgrade to
that,
I'd encourage you to do so.
Next, I see that you're subtracting the same field from itself in this
command... which should result in 0's everywhere. That's fine, if it
makes
your scripting easier, but I just wanted to mention it.
I see that you want to extract 1-hourly precip from this file. And
running it
though wgrib2 reveals that the field is named "GaugeCorrQPE01H", as
you've
already figured out. By default, pcp_combine looks for fields of
accumulated
precip, named "APCP". And so you're using the "-field" command line
option to
override that. There are 2 problems here. First, the "-field" option
needs
to specify a "name" and "level". Second, this GRIB2 file doesn't
contain a
1-hour accumulation. Since it's a 0-hour analysis file, it contains
0-hour
precip. So we'd need to use "A0" to access it.
Here are 3 variations that all do what you're trying to do:
(1) Using the -field option as you intended (the -field option
overrides the
01 you have elsewhere):
/usr/local/met-5.2/bin/pcp_combine -subtract
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
-field 'name="GaugeCorrQPE01H"; level="A00";' -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
(2) Instead of using "-field" just explicitly list the data you want
from each
file:
/usr/local/met-5.2/bin/pcp_combine -subtract
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 'name="GaugeCorrQPE01H";
level="A00";' GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
(3) Since you're really just running this as a pass-through for the 0-
hour
forecast, you could run a -add command for a single field instead:
/usr/local/met-5.2/bin/pcp_combine -add
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 'name="GaugeCorrQPE01H";
level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
MRMS_pcp_combine_obs_accum_vt12Z -v 5
All 3 options should give you the same result.
John
On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> John -
>
> I've been trying to locate the syntax error in my run command in the
> attached shell script, but unable to see it. Could you find it for
me?
> I've also attached the log file showing the error and some
> intermediate output file produced my MET which is shown in the log
> file. You already have the observation file (grib2).
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, May 24, 2019 11:25 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> There is most likely a syntax error in your script named
> "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't a
> problem in your data. You're probably just missing a semi-colon,
> single quote, or double quote somewhere on the command line for
> pcp_combine. If you're not able to find it, just send me the
contents
> of that script and it should be pretty easy to locate.
>
> Thanks,
> John
>
>
>
> On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I haven't seen this type of error before. Do you have a suggestion
> > for resolving this? See attached observation file.
> >
> > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > DEBUG 1: Reading input file:
> >
> >
>
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_
> 01H_00
> > .00_20160209-120000.grib2
> > ERROR :
> > ERROR : yyerror() -> syntax error in file "config_25586_0_.temp"
> > ERROR :
> > ERROR : line = 2
> > ERROR :
> > ERROR : column = 1
> > ERROR :
> > ERROR : text = "(nul)"
> > ERROR :
> > ERROR :
> > ERROR : (nul)
> > ERROR : (nul)
> > ERROR :
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 29 10:37:08 2019
John,
OK, let's not use "A00" or "A01" anymore. Instead, let's just use
"L0" to
indicate that the level value is 0. For some reason, the MRMS folks
don't
encode their GRIB2 files as having an accumulation interval. Instead,
the
accumulation interval is indicated by the variable name. For example,
GaugeCorrQPE01H
is a 1-hour accumulation while GaugeCorrQPE06H is a 6-hour accum.
Sorry for the confusion I may have caused in my last email.
Yes, this type of sum command should work:
/p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
20160209_150000 03
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_precip_obs_3hr_accum_vt15Z_09FEB.nc
-pcpdir /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr -field
'name="GaugeCorrQPE01H";
level="L0";' -log
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt15Z
-v 5
Although I haven't tested it with your data or with version 5.2 of MET
be
sure. Please just try running the command above to see if it works.
It
should find 3 GaugeCorr files, extract the GaugeCorrQPE01H field from
them,
and add them up.
Thanks,
John
On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for the testing and diagnosis and providing 3 options for
> accomplishing
> the subtraction. I intentionally wanted to zero-out the accum precip
> (analysis
> of 1-hr precip valid at 12Z) field at 12Z in order to track the
model
> development of the precip specifically over the period from 12Z to
18Z
> with 0
> as the starting point for this case study. In my script, the
additional
> commands (commented out now) I use "-sum" to accumulate for the
following
> hours up to 18Z. For the -sum I assume that I will use the same
options to
> describe the name and level per your examples below, correct? I'm
assuming
> that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
18Z, I
> can
> just sum them per the other commands in my script to get the 6-hr
accum at
> 18Z. Will this work?
>
> I have V8.1 running in a Singularity container on an ARL HPC
(Centennial)
> ,
> but I haven't provisioned this HPC with all the data I need to start
using
> it
> operationally. I plan to soon, however. Looking forward to the
> enhancements.
> I'm running V5.2 on the operational system (Excalibur).
Unfortunately, the
> kernel is older and the implementation of the V8.1 Singularity
container
> has
> being problematic. Working on it.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 9:45 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> OK, I'm able to replicate the error message you're seeing. Here's
the
> first
> command listed in your script (with slightly modified paths):
>
> */usr/local/met-5.2/bin/pcp_combine \*
>
> *-subtract \*
>
> *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
>
> *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
>
> *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
>
> *-field GaugeCorrQPE01H \*
>
> *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
>
>
> I see that you're running met-5.2. The latest available version is
8.1
> which
> includes many bugfixes and enhancements. If you're able to upgrade
to
> that,
> I'd encourage you to do so.
>
>
> Next, I see that you're subtracting the same field from itself in
this
> command... which should result in 0's everywhere. That's fine, if
it
> makes
> your scripting easier, but I just wanted to mention it.
>
>
> I see that you want to extract 1-hourly precip from this file. And
> running it
> though wgrib2 reveals that the field is named "GaugeCorrQPE01H", as
you've
> already figured out. By default, pcp_combine looks for fields of
> accumulated
> precip, named "APCP". And so you're using the "-field" command line
> option to
> override that. There are 2 problems here. First, the "-field"
option
> needs
> to specify a "name" and "level". Second, this GRIB2 file doesn't
contain
> a
> 1-hour accumulation. Since it's a 0-hour analysis file, it contains
> 0-hour
> precip. So we'd need to use "A0" to access it.
>
>
> Here are 3 variations that all do what you're trying to do:
>
>
> (1) Using the -field option as you intended (the -field option
overrides
> the
> 01 you have elsewhere):
>
>
> /usr/local/met-5.2/bin/pcp_combine -subtract
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> (2) Instead of using "-field" just explicitly list the data you want
from
> each
> file:
>
>
> /usr/local/met-5.2/bin/pcp_combine -subtract
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> level="A00";' GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> 'name="GaugeCorrQPE01H";
> level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> (3) Since you're really just running this as a pass-through for the
0-hour
> forecast, you could run a -add command for a single field instead:
>
>
> /usr/local/met-5.2/bin/pcp_combine -add
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> All 3 options should give you the same result.
>
>
> John
>
> On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> > John -
> >
> > I've been trying to locate the syntax error in my run command in
the
> > attached shell script, but unable to see it. Could you find it for
me?
> > I've also attached the log file showing the error and some
> > intermediate output file produced my MET which is shown in the log
> > file. You already have the observation file (grib2).
> >
> > Thanks.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Friday, May 24, 2019 11:25 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
> > error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > There is most likely a syntax error in your script named
> > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't
a
> > problem in your data. You're probably just missing a semi-colon,
> > single quote, or double quote somewhere on the command line for
> > pcp_combine. If you're not able to find it, just send me the
contents
> > of that script and it should be pretty easy to locate.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I haven't seen this type of error before. Do you have a
suggestion
> > > for resolving this? See attached observation file.
> > >
> > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > DEBUG 1: Reading input file:
> > >
> > >
> >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_
> > 01H_00
> > > .00_20160209-120000.grib2
> > > ERROR :
> > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > ERROR :
> > > ERROR : line = 2
> > > ERROR :
> > > ERROR : column = 1
> > > ERROR :
> > > ERROR : text = "(nul)"
> > > ERROR :
> > > ERROR :
> > > ERROR : (nul)
> > > ERROR : (nul)
> > > ERROR :
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 29 11:23:04 2019
John -
It worked on the first try and I believe the results are as expected.
Used the commands in the attached text file built from your email.
Also see the attached log file. The output file is 94 mb, so I'll send
a link to your email address so you can download it. I plotted it
using IDV and the range of the data values in the field (CONUS) was -9
mm to 363mm. Not sure how -9 is generated, though. Maybe from neg
reflectivity values? I'll have to look at the input obs to see why.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, May 29, 2019 10:37 AM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
OK, let's not use "A00" or "A01" anymore. Instead, let's just use
"L0" to
indicate that the level value is 0. For some reason, the MRMS folks
don't
encode their GRIB2 files as having an accumulation interval. Instead,
the
accumulation interval is indicated by the variable name. For example,
GaugeCorrQPE01H
is a 1-hour accumulation while GaugeCorrQPE06H is a 6-hour accum.
Sorry for the confusion I may have caused in my last email.
Yes, this type of sum command should work:
/p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
20160209_150000 03
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_precip_obs_3hr_accum_vt15Z_09FEB.nc
-pcpdir /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr -field
'name="GaugeCorrQPE01H";
level="L0";' -log
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt15Z
-v 5
Although I haven't tested it with your data or with version 5.2 of MET
be
sure. Please just try running the command above to see if it works.
It
should find 3 GaugeCorr files, extract the GaugeCorrQPE01H field from
them,
and add them up.
Thanks,
John
On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for the testing and diagnosis and providing 3 options for
> accomplishing
> the subtraction. I intentionally wanted to zero-out the accum precip
> (analysis
> of 1-hr precip valid at 12Z) field at 12Z in order to track the
model
> development of the precip specifically over the period from 12Z to
18Z
> with 0
> as the starting point for this case study. In my script, the
additional
> commands (commented out now) I use "-sum" to accumulate for the
following
> hours up to 18Z. For the -sum I assume that I will use the same
options to
> describe the name and level per your examples below, correct? I'm
assuming
> that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
18Z, I
> can
> just sum them per the other commands in my script to get the 6-hr
accum at
> 18Z. Will this work?
>
> I have V8.1 running in a Singularity container on an ARL HPC
(Centennial)
> ,
> but I haven't provisioned this HPC with all the data I need to start
using
> it
> operationally. I plan to soon, however. Looking forward to the
> enhancements.
> I'm running V5.2 on the operational system (Excalibur).
Unfortunately, the
> kernel is older and the implementation of the V8.1 Singularity
container
> has
> being problematic. Working on it.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 9:45 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> OK, I'm able to replicate the error message you're seeing. Here's
the
> first
> command listed in your script (with slightly modified paths):
>
> */usr/local/met-5.2/bin/pcp_combine \*
>
> *-subtract \*
>
> *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
>
> *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
>
> *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
>
> *-field GaugeCorrQPE01H \*
>
> *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
>
>
> I see that you're running met-5.2. The latest available version is
8.1
> which
> includes many bugfixes and enhancements. If you're able to upgrade
to
> that,
> I'd encourage you to do so.
>
>
> Next, I see that you're subtracting the same field from itself in
this
> command... which should result in 0's everywhere. That's fine, if
it
> makes
> your scripting easier, but I just wanted to mention it.
>
>
> I see that you want to extract 1-hourly precip from this file. And
> running it
> though wgrib2 reveals that the field is named "GaugeCorrQPE01H", as
you've
> already figured out. By default, pcp_combine looks for fields of
> accumulated
> precip, named "APCP". And so you're using the "-field" command line
> option to
> override that. There are 2 problems here. First, the "-field"
option
> needs
> to specify a "name" and "level". Second, this GRIB2 file doesn't
contain
> a
> 1-hour accumulation. Since it's a 0-hour analysis file, it contains
> 0-hour
> precip. So we'd need to use "A0" to access it.
>
>
> Here are 3 variations that all do what you're trying to do:
>
>
> (1) Using the -field option as you intended (the -field option
overrides
> the
> 01 you have elsewhere):
>
>
> /usr/local/met-5.2/bin/pcp_combine -subtract
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> (2) Instead of using "-field" just explicitly list the data you want
from
> each
> file:
>
>
> /usr/local/met-5.2/bin/pcp_combine -subtract
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> level="A00";' GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> 'name="GaugeCorrQPE01H";
> level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> (3) Since you're really just running this as a pass-through for the
0-hour
> forecast, you could run a -add command for a single field instead:
>
>
> /usr/local/met-5.2/bin/pcp_combine -add
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> MRMS_pcp_combine_obs_accum_vt12Z -v 5
>
>
> All 3 options should give you the same result.
>
>
> John
>
> On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> > John -
> >
> > I've been trying to locate the syntax error in my run command in
the
> > attached shell script, but unable to see it. Could you find it for
me?
> > I've also attached the log file showing the error and some
> > intermediate output file produced my MET which is shown in the log
> > file. You already have the observation file (grib2).
> >
> > Thanks.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Friday, May 24, 2019 11:25 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
> > error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > There is most likely a syntax error in your script named
> > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It isn't
a
> > problem in your data. You're probably just missing a semi-colon,
> > single quote, or double quote somewhere on the command line for
> > pcp_combine. If you're not able to find it, just send me the
contents
> > of that script and it should be pretty easy to locate.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I haven't seen this type of error before. Do you have a
suggestion
> > > for resolving this? See attached observation file.
> > >
> > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > DEBUG 1: Reading input file:
> > >
> > >
> >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_
> > 01H_00
> > > .00_20160209-120000.grib2
> > > ERROR :
> > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > ERROR :
> > > ERROR : line = 2
> > > ERROR :
> > > ERROR : column = 1
> > > ERROR :
> > > ERROR : text = "(nul)"
> > > ERROR :
> > > ERROR :
> > > ERROR : (nul)
> > > ERROR : (nul)
> > > ERROR :
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 29 12:03:21 2019
John,
I think you should take a step back to understand this MRMS data
better.
Try running MET's plot_data_plane tool to visualize it:
/usr/local/met-5.2/bin/plot_data_plane
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
level="L0";' -v 4
DEBUG 4: Data plane information:
DEBUG 4: plane min: -3
DEBUG 4: plane max: 175.7
I've attached a PNG file showing the resulting image.
Notice that where they have listed a value of -3, it sure looks like
it
should be missing data! Just like they don't store this data as an
accumulation interval correctly, they don't store these bad data
values
correctly either. Adding -3 together 3 times gives you a value of -9.
This cannot easily be fixed in met-5.2, but in met-8.1 you can use the
censor_thresh and censor_val options to change those values of -3 to
MET's
value for bad data, -9999:
/usr/local/met-8.1/bin/plot_data_plane
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
level="L0";' -v 4
DEBUG 3: Applying censor thresholds "==-3" and replacing with values
"-9999.00000".
DEBUG 3: Censored values for 8598569 of 24500000 grid points.
DEBUG 1: Creating postscript file:
GaugeCorr_QPE_01H_00.00_20160209-120000.ps
Make sense?
Thanks,
John
On Wed, May 29, 2019 at 11:23 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> John -
>
> It worked on the first try and I believe the results are as
expected. Used
> the commands in the attached text file built from your email. Also
see the
> attached log file. The output file is 94 mb, so I'll send a link to
your
> email address so you can download it. I plotted it using IDV and the
range
> of the data values in the field (CONUS) was -9 mm to 363mm. Not sure
how -9
> is generated, though. Maybe from neg reflectivity values? I'll have
to look
> at the input obs to see why.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 10:37 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> OK, let's not use "A00" or "A01" anymore. Instead, let's just use
"L0" to
> indicate that the level value is 0. For some reason, the MRMS folks
don't
> encode their GRIB2 files as having an accumulation interval.
Instead, the
> accumulation interval is indicated by the variable name. For
example,
> GaugeCorrQPE01H
> is a 1-hour accumulation while GaugeCorrQPE06H is a 6-hour accum.
>
> Sorry for the confusion I may have caused in my last email.
>
> Yes, this type of sum command should work:
>
> /p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
> 20160209_150000 03
>
>
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_precip_obs_3hr_accum_vt15Z_09FEB.nc
> -pcpdir /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr
-field
> 'name="GaugeCorrQPE01H";
> level="L0";' -log
>
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt15Z
-v
> 5
>
> Although I haven't tested it with your data or with version 5.2 of
MET be
> sure. Please just try running the command above to see if it works.
It
> should find 3 GaugeCorr files, extract the GaugeCorrQPE01H field
from them,
> and add them up.
>
> Thanks,
> John
>
> On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for the testing and diagnosis and providing 3 options for
> > accomplishing
> > the subtraction. I intentionally wanted to zero-out the accum
precip
> > (analysis
> > of 1-hr precip valid at 12Z) field at 12Z in order to track the
model
> > development of the precip specifically over the period from 12Z to
18Z
> > with 0
> > as the starting point for this case study. In my script, the
additional
> > commands (commented out now) I use "-sum" to accumulate for the
following
> > hours up to 18Z. For the -sum I assume that I will use the same
options
> to
> > describe the name and level per your examples below, correct? I'm
> assuming
> > that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
18Z, I
> > can
> > just sum them per the other commands in my script to get the 6-hr
accum
> at
> > 18Z. Will this work?
> >
> > I have V8.1 running in a Singularity container on an ARL HPC
(Centennial)
> > ,
> > but I haven't provisioned this HPC with all the data I need to
start
> using
> > it
> > operationally. I plan to soon, however. Looking forward to the
> > enhancements.
> > I'm running V5.2 on the operational system (Excalibur).
Unfortunately,
> the
> > kernel is older and the implementation of the V8.1 Singularity
container
> > has
> > being problematic. Working on it.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 29, 2019 9:45 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > Pcp-Combine
> > error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> > contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > OK, I'm able to replicate the error message you're seeing. Here's
the
> > first
> > command listed in your script (with slightly modified paths):
> >
> > */usr/local/met-5.2/bin/pcp_combine \*
> >
> > *-subtract \*
> >
> > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> >
> > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> >
> > *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
> >
> > *-field GaugeCorrQPE01H \*
> >
> > *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
> >
> >
> > I see that you're running met-5.2. The latest available version
is 8.1
> > which
> > includes many bugfixes and enhancements. If you're able to
upgrade to
> > that,
> > I'd encourage you to do so.
> >
> >
> > Next, I see that you're subtracting the same field from itself in
this
> > command... which should result in 0's everywhere. That's fine, if
it
> > makes
> > your scripting easier, but I just wanted to mention it.
> >
> >
> > I see that you want to extract 1-hourly precip from this file.
And
> > running it
> > though wgrib2 reveals that the field is named "GaugeCorrQPE01H",
as
> you've
> > already figured out. By default, pcp_combine looks for fields of
> > accumulated
> > precip, named "APCP". And so you're using the "-field" command
line
> > option to
> > override that. There are 2 problems here. First, the "-field"
option
> > needs
> > to specify a "name" and "level". Second, this GRIB2 file doesn't
contain
> > a
> > 1-hour accumulation. Since it's a 0-hour analysis file, it
contains
> > 0-hour
> > precip. So we'd need to use "A0" to access it.
> >
> >
> > Here are 3 variations that all do what you're trying to do:
> >
> >
> > (1) Using the -field option as you intended (the -field option
overrides
> > the
> > 01 you have elsewhere):
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -subtract
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> > -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > (2) Instead of using "-field" just explicitly list the data you
want from
> > each
> > file:
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -subtract
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> > level="A00";' GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 'name="GaugeCorrQPE01H";
> > level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > (3) Since you're really just running this as a pass-through for
the
> 0-hour
> > forecast, you could run a -add command for a single field instead:
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -add
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
'name="GaugeCorrQPE01H";
> > level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > All 3 options should give you the same result.
> >
> >
> > John
> >
> > On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > >
> > > John -
> > >
> > > I've been trying to locate the syntax error in my run command in
the
> > > attached shell script, but unable to see it. Could you find it
for me?
> > > I've also attached the log file showing the error and some
> > > intermediate output file produced my MET which is shown in the
log
> > > file. You already have the observation file (grib2).
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Friday, May 24, 2019 11:25 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
> > > error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
verify
> > > the identity of the sender, and confirm the authenticity of all
links
> > > contained within the message prior to copying and pasting the
address
> > > to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > There is most likely a syntax error in your script named
> > > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It
isn't a
> > > problem in your data. You're probably just missing a semi-
colon,
> > > single quote, or double quote somewhere on the command line for
> > > pcp_combine. If you're not able to find it, just send me the
contents
> > > of that script and it should be pretty easy to locate.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I haven't seen this type of error before. Do you have a
suggestion
> > > > for resolving this? See attached observation file.
> > > >
> > > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > > DEBUG 1: Reading input file:
> > > >
> > > >
> > >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_QPE_
> > > 01H_00
> > > > .00_20160209-120000.grib2
> > > > ERROR :
> > > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > > ERROR :
> > > > ERROR : line = 2
> > > > ERROR :
> > > > ERROR : column = 1
> > > > ERROR :
> > > > ERROR : text = "(nul)"
> > > > ERROR :
> > > > ERROR :
> > > > ERROR : (nul)
> > > > ERROR : (nul)
> > > > ERROR :
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 29 13:22:11 2019
CLASSIFICATION: UNCLASSIFIED
John -
I recall that you provided some guidance on how to replace bad data
values
(-9999) with zeros back when I was working with radar reflectivity.
See
attached email. You used Gen_Vx_Mask to change missing (bad) values to
0s. The
reason for doing that was to assure that the categorical counts and
statistics
output by MODE would not be impacted by an artificially low number of
matched
pairs caused by the presence of a large amount of missing data. Why in
this
case do you want to assign -9999 to the negative values instead of
zeros which
will result in better categorical statistics? I plan to run Grid-Stat
and
Wavelet-Stat using the output from Pcp-Combine and will rely on the
categorical and neighborhood statistics heavily. I may not be
understanding
something here or I am applying a fix for something else where it's
not
appropriate.
Also, I plan to transfer my data to the newer HPC and run the commands
you
suggest below using the censor_thresh and censor_val options in V8.1.
From
your commands below it looks like you are using the options with
plot_data_plane. I checked the User's Guide for V8.1 and I didn't see
these
options in the usage description.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, May 29, 2019 12:03 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
I think you should take a step back to understand this MRMS data
better.
Try running MET's plot_data_plane tool to visualize it:
/usr/local/met-5.2/bin/plot_data_plane
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
level="L0";' -v 4
DEBUG 4: Data plane information:
DEBUG 4: plane min: -3
DEBUG 4: plane max: 175.7
I've attached a PNG file showing the resulting image.
Notice that where they have listed a value of -3, it sure looks like
it should
be missing data! Just like they don't store this data as an
accumulation
interval correctly, they don't store these bad data values correctly
either.
Adding -3 together 3 times gives you a value of -9.
This cannot easily be fixed in met-5.2, but in met-8.1 you can use the
censor_thresh and censor_val options to change those values of -3 to
MET's
value for bad data, -9999:
/usr/local/met-8.1/bin/plot_data_plane
GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
level="L0";' -v 4
DEBUG 3: Applying censor thresholds "==-3" and replacing with values
"-9999.00000".
DEBUG 3: Censored values for 8598569 of 24500000 grid points.
DEBUG 1: Creating postscript file:
GaugeCorr_QPE_01H_00.00_20160209-120000.ps
Make sense?
Thanks,
John
On Wed, May 29, 2019 at 11:23 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> John -
>
> It worked on the first try and I believe the results are as
expected.
> Used the commands in the attached text file built from your email.
> Also see the attached log file. The output file is 94 mb, so I'll
send
> a link to your email address so you can download it. I plotted it
> using IDV and the range of the data values in the field (CONUS) was
-9
> mm to 363mm. Not sure how -9 is generated, though. Maybe from neg
> reflectivity values? I'll have to look at the input obs to see why.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 10:37 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> OK, let's not use "A00" or "A01" anymore. Instead, let's just use
> "L0" to indicate that the level value is 0. For some reason, the
MRMS
> folks don't encode their GRIB2 files as having an accumulation
> interval. Instead, the accumulation interval is indicated by the
> variable name. For example, GaugeCorrQPE01H is a 1-hour
accumulation
> while GaugeCorrQPE06H is a 6-hour accum.
>
> Sorry for the confusion I may have caused in my last email.
>
> Yes, this type of sum command should work:
>
> /p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
> 20160209_150000 03
>
>
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_precip
> _obs_3hr_accum_vt15Z_09FEB.nc -pcpdir
> /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr -field
> 'name="GaugeCorrQPE01H"; level="L0";' -log
>
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt15
> Z -v
> 5
>
> Although I haven't tested it with your data or with version 5.2 of
MET
> be sure. Please just try running the command above to see if it
> works. It should find 3 GaugeCorr files, extract the
GaugeCorrQPE01H
> field from them, and add them up.
>
> Thanks,
> John
>
> On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for the testing and diagnosis and providing 3 options for
> > accomplishing the subtraction. I intentionally wanted to zero-out
> > the accum precip (analysis of 1-hr precip valid at 12Z) field at
12Z
> > in order to track the model development of the precip specifically
> > over the period from 12Z to 18Z with 0 as the starting point for
> > this case study. In my script, the additional commands (commented
> > out now) I use "-sum" to accumulate for the following hours up to
> > 18Z. For the -sum I assume that I will use the same options
> to
> > describe the name and level per your examples below, correct? I'm
> assuming
> > that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
> > 18Z, I can just sum them per the other commands in my script to
get
> > the 6-hr accum
> at
> > 18Z. Will this work?
> >
> > I have V8.1 running in a Singularity container on an ARL HPC
> > (Centennial) , but I haven't provisioned this HPC with all the
data
> > I need to start
> using
> > it
> > operationally. I plan to soon, however. Looking forward to the
> > enhancements.
> > I'm running V5.2 on the operational system (Excalibur).
> > Unfortunately,
> the
> > kernel is older and the implementation of the V8.1 Singularity
> > container has being problematic. Working on it.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 29, 2019 9:45 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > Pcp-Combine error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify
> the
> > identity of the sender, and confirm the authenticity of all links
> > contained within the message prior to copying and pasting the
> > address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > OK, I'm able to replicate the error message you're seeing. Here's
> > the first command listed in your script (with slightly modified
> > paths):
> >
> > */usr/local/met-5.2/bin/pcp_combine \*
> >
> > *-subtract \*
> >
> > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> >
> > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> >
> > *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
> >
> > *-field GaugeCorrQPE01H \*
> >
> > *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
> >
> >
> > I see that you're running met-5.2. The latest available version
is
> > 8.1 which includes many bugfixes and enhancements. If you're able
> > to upgrade to that, I'd encourage you to do so.
> >
> >
> > Next, I see that you're subtracting the same field from itself in
> > this command... which should result in 0's everywhere. That's
fine,
> > if it makes your scripting easier, but I just wanted to mention
it.
> >
> >
> > I see that you want to extract 1-hourly precip from this file.
And
> > running it though wgrib2 reveals that the field is named
> > "GaugeCorrQPE01H", as
> you've
> > already figured out. By default, pcp_combine looks for fields of
> > accumulated precip, named "APCP". And so you're using the "-
field"
> > command line option to override that. There are 2 problems here.
> > First, the "-field" option needs to specify a "name" and "level".
> > Second, this GRIB2 file doesn't contain a 1-hour accumulation.
> > Since it's a 0-hour analysis file, it contains 0-hour precip. So
> > we'd need to use "A0" to access it.
> >
> >
> > Here are 3 variations that all do what you're trying to do:
> >
> >
> > (1) Using the -field option as you intended (the -field option
> > overrides the
> > 01 you have elsewhere):
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -subtract
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> > -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > (2) Instead of using "-field" just explicitly list the data you
want
> > from each
> > file:
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -subtract
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 'name="GaugeCorrQPE01H"; level="A00";'
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 'name="GaugeCorrQPE01H";
> > level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > (3) Since you're really just running this as a pass-through for
the
> 0-hour
> > forecast, you could run a -add command for a single field instead:
> >
> >
> > /usr/local/met-5.2/bin/pcp_combine -add
> > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > 'name="GaugeCorrQPE01H"; level="A00";'
> > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> >
> >
> > All 3 options should give you the same result.
> >
> >
> > John
> >
> > On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > >
> > > John -
> > >
> > > I've been trying to locate the syntax error in my run command in
> > > the attached shell script, but unable to see it. Could you find
it for
> > > me?
> > > I've also attached the log file showing the error and some
> > > intermediate output file produced my MET which is shown in the
log
> > > file. You already have the observation file (grib2).
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Friday, May 24, 2019 11:25 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > > Pcp-Combine error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > There is most likely a syntax error in your script named
> > > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It
isn't
> > > a problem in your data. You're probably just missing a
> > > semi-colon, single quote, or double quote somewhere on the
command
> > > line for pcp_combine. If you're not able to find it, just send
me
> > > the contents of that script and it should be pretty easy to
locate.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I haven't seen this type of error before. Do you have a
> > > > suggestion for resolving this? See attached observation file.
> > > >
> > > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > > DEBUG 1: Reading input file:
> > > >
> > > >
> > >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_
> > > QPE_
> > > 01H_00
> > > > .00_20160209-120000.grib2
> > > > ERROR :
> > > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > > ERROR :
> > > > ERROR : line = 2
> > > > ERROR :
> > > > ERROR : column = 1
> > > > ERROR :
> > > > ERROR : text = "(nul)"
> > > > ERROR :
> > > > ERROR :
> > > > ERROR : (nul)
> > > > ERROR : (nul)
> > > > ERROR :
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #81164] questions about MODE output (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 29 13:22:11 2019
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Hi John,
I see that you have a question about the categorical counts and
statistics
included in the output of MODE. You're wondering why the false alarm
counts (FY_ON) are set to 0. And looking closely at the TOTAL column
in
the cts.txt file, you'll see that the number of matched pairs is 222.
However, looking in the NetCDF file you sent, I see that the grid
dimension
is 204 x 204... so the total number of grid points *should* be 41,616!
The answer lies in the PostScript plot you sent. While all 41,616
forecast
grid points contain valid data, only 222 of the observation grid
points
contain valid data values. All that gray area in the observation field
is
missing data. MODE only computes categorical statistics at grid
points
which contain valid data in both fields.
I assume that you'd like the observation field missing data values to
actually be set to a value of 0 instead.
You could do this in one of two ways...
(1) Run the observation field through the gen_vx_mask tool in MET as a
way
of preprocessing the data and changing those bad data values to 0.
This
would be necessary if you want to also run this data through Grid-
Stat.
(2) In the MODE config file in the "obs" section, set:
raw_thresh = !=NA;
MODE will read in the observation field and any grid point not meeting
the
threshold criteria will be reset to a value of 0.
Please give that a shot and let me know how it goes.
Thanks,
John
On Tue, Jul 11, 2017 at 12:40 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Jul 11 12:40:17 2017: Request 81164 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: questions about MODE output (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81164 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I have a question about the calculation of contingency table scores
such as
> CSI which are output in the cts.txt text file. For reference I've
attached
> that file plus the postscript file and the other two output files
from a
> run
> I did using forecast and observed radar reflectivity.
>
> In the postscript file plots showing the forecast objects with
observation
> outlines, I see the red forecast object which was matched with the
blue
> outlined observation object. I see in this plot areas where there
are
> "hits"
> (FY_OY) where you can see red area overlaying the blue outlined
area. I see
> "misses" (FN_OY) where the blue outlined area does not overlap a red
area.
> Both the hits and misses have non-zero counts in their respective
columns
> in
> the cts.txt text file. I also see in the plot areas where the red
areas do
> not overlay a blue outlined area which corresponds to "false alarms"
> (FY_ON), but the cts.txt file shows that there are no false alarm
events
> with a 0 entry. The MET tutorial describes the output in the cts.txt
file
> at:
> Caution-http://www.dtcenter.org/met/users/support/online_tutorial/
> METv5.2/tutorial.p
> hp?name=mode&category=output
>
> The tutorial explains that the objects show 1 for the red areas and
0 for
> non-red and 1 for blue outlined areas and 0 for non-blue outlined
areas. If
> a simple contingency table of outcomes is generated form this, why
are
> there
> no false alarm events?
>
> Another question regarding the first page of the postscript output
file. In
> the table on the right side of the page, total interest values for
pairs
> of simple objects are listed in sorted order. My understanding from
the
> User's Guide was that those pairs of simple objects whose interest
value
> fell below the interest threshold were not "matched", but if I look
in the
> plot, I can see simple objects with red numbers (part of the merged
> forecast
> and observed objects) which also show in the table on the right with
> interest values less than the threshold which are "matched". An
example of
> this is object 6 (fcst) and object 5 (obs). I thought I read that
same
> color
> objects in both fields have been "matched". What is the relationship
> between
> the interst threshold and matching. Also note that numbered simple
objects
> which are colored royal blue which should indicate "unmatched" also
appear
> in the table of "pairs". I guess I don't quite understand the
presence of
> "pairs" in the table versus the simple numbered objects appearing in
the
> plots.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 29 14:54:22 2019
John,
If you look at the plot I sent, you'll see that the -3 values (which
show
up as white) are in areas along the edge of the domain where there
would
typically be missing data values. Those regions are beyond the range
of
the radars and are not observable. For this reason, they should be
set to
bad data and not 0. Setting them to 0 would artificially inflate the
correct negatives counts and impact the resulting stats.
Yes, you're right. You could run gen_vx_mask from met-5.2 to reset
those
-3 values to -9999. The censor_thresh and censor_val settings,
available
in more recent releases, are just more convenient.
And if you search the user's guide, you should find a reference to
them in
there. However, they are not specifically mentioned in the
pcp_combine
section. Instead, they apply broadly to all of the MET tools where
config
files are used. And the pcp_combine "-field" option is like a mini
config
file that you set on the command line.
There are a few other settings like this that apply broadly to all
config
files and are not specifically listed for each tool... like file_type,
GRIB1_ptv, GRIB1_code, GRIB2_parm, GRIB2_parm_cat... and so on.
Thanks,
John
On Wed, May 29, 2019 at 1:22 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I recall that you provided some guidance on how to replace bad data
values
> (-9999) with zeros back when I was working with radar reflectivity.
See
> attached email. You used Gen_Vx_Mask to change missing (bad) values
to 0s.
> The
> reason for doing that was to assure that the categorical counts and
> statistics
> output by MODE would not be impacted by an artificially low number
of
> matched
> pairs caused by the presence of a large amount of missing data. Why
in
> this
> case do you want to assign -9999 to the negative values instead of
zeros
> which
> will result in better categorical statistics? I plan to run Grid-
Stat and
> Wavelet-Stat using the output from Pcp-Combine and will rely on the
> categorical and neighborhood statistics heavily. I may not be
> understanding
> something here or I am applying a fix for something else where it's
not
> appropriate.
>
> Also, I plan to transfer my data to the newer HPC and run the
commands you
> suggest below using the censor_thresh and censor_val options in
V8.1. From
> your commands below it looks like you are using the options with
> plot_data_plane. I checked the User's Guide for V8.1 and I didn't
see
> these
> options in the usage description.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 12:03 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I think you should take a step back to understand this MRMS data
better.
> Try running MET's plot_data_plane tool to visualize it:
>
>
> /usr/local/met-5.2/bin/plot_data_plane
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
> level="L0";' -v 4
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: -3
>
> DEBUG 4: plane max: 175.7
>
>
> I've attached a PNG file showing the resulting image.
>
> Notice that where they have listed a value of -3, it sure looks like
it
> should
> be missing data! Just like they don't store this data as an
accumulation
> interval correctly, they don't store these bad data values correctly
> either.
> Adding -3 together 3 times gives you a value of -9.
>
> This cannot easily be fixed in met-5.2, but in met-8.1 you can use
the
> censor_thresh and censor_val options to change those values of -3 to
MET's
> value for bad data, -9999:
>
> /usr/local/met-8.1/bin/plot_data_plane
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
> level="L0";' -v 4
>
> DEBUG 3: Applying censor thresholds "==-3" and replacing with values
> "-9999.00000".
>
> DEBUG 3: Censored values for 8598569 of 24500000 grid points.
>
> DEBUG 1: Creating postscript file:
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps
>
>
> Make sense?
>
>
> Thanks,
> John
>
> On Wed, May 29, 2019 at 11:23 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> >
> > John -
> >
> > It worked on the first try and I believe the results are as
expected.
> > Used the commands in the attached text file built from your email.
> > Also see the attached log file. The output file is 94 mb, so I'll
send
> > a link to your email address so you can download it. I plotted it
> > using IDV and the range of the data values in the field (CONUS)
was -9
> > mm to 363mm. Not sure how -9 is generated, though. Maybe from neg
> > reflectivity values? I'll have to look at the input obs to see
why.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, May 29, 2019 10:37 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > Pcp-Combine error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > OK, let's not use "A00" or "A01" anymore. Instead, let's just use
> > "L0" to indicate that the level value is 0. For some reason, the
MRMS
> > folks don't encode their GRIB2 files as having an accumulation
> > interval. Instead, the accumulation interval is indicated by the
> > variable name. For example, GaugeCorrQPE01H is a 1-hour
accumulation
> > while GaugeCorrQPE06H is a 6-hour accum.
> >
> > Sorry for the confusion I may have caused in my last email.
> >
> > Yes, this type of sum command should work:
> >
> > /p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
> > 20160209_150000 03
> >
> >
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_precip
> > _obs_3hr_accum_vt15Z_09FEB.nc -pcpdir
> > /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr -field
> > 'name="GaugeCorrQPE01H"; level="L0";' -log
> >
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt15
> > Z -v
> > 5
> >
> > Although I haven't tested it with your data or with version 5.2 of
MET
> > be sure. Please just try running the command above to see if it
> > works. It should find 3 GaugeCorr files, extract the
GaugeCorrQPE01H
> > field from them, and add them up.
> >
> > Thanks,
> > John
> >
> > On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for the testing and diagnosis and providing 3 options for
> > > accomplishing the subtraction. I intentionally wanted to zero-
out
> > > the accum precip (analysis of 1-hr precip valid at 12Z) field at
12Z
> > > in order to track the model development of the precip
specifically
> > > over the period from 12Z to 18Z with 0 as the starting point for
> > > this case study. In my script, the additional commands
(commented
> > > out now) I use "-sum" to accumulate for the following hours up
to
> > > 18Z. For the -sum I assume that I will use the same options
> > to
> > > describe the name and level per your examples below, correct?
I'm
> > assuming
> > > that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
> > > 18Z, I can just sum them per the other commands in my script to
get
> > > the 6-hr accum
> > at
> > > 18Z. Will this work?
> > >
> > > I have V8.1 running in a Singularity container on an ARL HPC
> > > (Centennial) , but I haven't provisioned this HPC with all the
data
> > > I need to start
> > using
> > > it
> > > operationally. I plan to soon, however. Looking forward to the
> > > enhancements.
> > > I'm running V5.2 on the operational system (Excalibur).
> > > Unfortunately,
> > the
> > > kernel is older and the implementation of the V8.1 Singularity
> > > container has being problematic. Working on it.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, May 29, 2019 9:45 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > > Pcp-Combine error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > > contained within the message prior to copying and pasting the
> > > address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > OK, I'm able to replicate the error message you're seeing.
Here's
> > > the first command listed in your script (with slightly modified
> > > paths):
> > >
> > > */usr/local/met-5.2/bin/pcp_combine \*
> > >
> > > *-subtract \*
> > >
> > > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> > >
> > > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> > >
> > > *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
> > >
> > > *-field GaugeCorrQPE01H \*
> > >
> > > *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
> > >
> > >
> > > I see that you're running met-5.2. The latest available version
is
> > > 8.1 which includes many bugfixes and enhancements. If you're
able
> > > to upgrade to that, I'd encourage you to do so.
> > >
> > >
> > > Next, I see that you're subtracting the same field from itself
in
> > > this command... which should result in 0's everywhere. That's
fine,
> > > if it makes your scripting easier, but I just wanted to mention
it.
> > >
> > >
> > > I see that you want to extract 1-hourly precip from this file.
And
> > > running it though wgrib2 reveals that the field is named
> > > "GaugeCorrQPE01H", as
> > you've
> > > already figured out. By default, pcp_combine looks for fields
of
> > > accumulated precip, named "APCP". And so you're using the "-
field"
> > > command line option to override that. There are 2 problems
here.
> > > First, the "-field" option needs to specify a "name" and
"level".
> > > Second, this GRIB2 file doesn't contain a 1-hour accumulation.
> > > Since it's a 0-hour analysis file, it contains 0-hour precip.
So
> > > we'd need to use "A0" to access it.
> > >
> > >
> > > Here are 3 variations that all do what you're trying to do:
> > >
> > >
> > > (1) Using the -field option as you intended (the -field option
> > > overrides the
> > > 01 you have elsewhere):
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -subtract
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> > > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> > > -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > (2) Instead of using "-field" just explicitly list the data you
want
> > > from each
> > > file:
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -subtract
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H"; level="A00";'
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H";
> > > level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > (3) Since you're really just running this as a pass-through for
the
> > 0-hour
> > > forecast, you could run a -add command for a single field
instead:
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -add
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H"; level="A00";'
> > > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > All 3 options should give you the same result.
> > >
> > >
> > > John
> > >
> > > On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > > >
> > > > John -
> > > >
> > > > I've been trying to locate the syntax error in my run command
in
> > > > the attached shell script, but unable to see it. Could you
find it
> for
> > > > me?
> > > > I've also attached the log file showing the error and some
> > > > intermediate output file produced my MET which is shown in the
log
> > > > file. You already have the observation file (grib2).
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Friday, May 24, 2019 11:25 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > > > Pcp-Combine error (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > There is most likely a syntax error in your script named
> > > > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It
isn't
> > > > a problem in your data. You're probably just missing a
> > > > semi-colon, single quote, or double quote somewhere on the
command
> > > > line for pcp_combine. If you're not able to find it, just
send me
> > > > the contents of that script and it should be pretty easy to
locate.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I haven't seen this type of error before. Do you have a
> > > > > suggestion for resolving this? See attached observation
file.
> > > > >
> > > > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > > > DEBUG 1: Reading input file:
> > > > >
> > > > >
> > > >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCorr_
> > > > QPE_
> > > > 01H_00
> > > > > .00_20160209-120000.grib2
> > > > > ERROR :
> > > > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > > > ERROR :
> > > > > ERROR : line = 2
> > > > > ERROR :
> > > > > ERROR : column = 1
> > > > > ERROR :
> > > > > ERROR : text = "(nul)"
> > > > > ERROR :
> > > > > ERROR :
> > > > > ERROR : (nul)
> > > > > ERROR : (nul)
> > > > > ERROR :
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you have a question about the categorical counts and
statistics
> included in the output of MODE. You're wondering why the false
alarm
> counts (FY_ON) are set to 0. And looking closely at the TOTAL
column in
> the cts.txt file, you'll see that the number of matched pairs is
222.
> However, looking in the NetCDF file you sent, I see that the grid
dimension
> is 204 x 204... so the total number of grid points *should* be
41,616!
>
> The answer lies in the PostScript plot you sent. While all 41,616
forecast
> grid points contain valid data, only 222 of the observation grid
points
> contain valid data values. All that gray area in the observation
field is
> missing data. MODE only computes categorical statistics at grid
points
> which contain valid data in both fields.
>
> I assume that you'd like the observation field missing data values
to
> actually be set to a value of 0 instead.
>
> You could do this in one of two ways...
>
> (1) Run the observation field through the gen_vx_mask tool in MET as
a way
> of preprocessing the data and changing those bad data values to 0.
This
> would be necessary if you want to also run this data through Grid-
Stat.
>
> (2) In the MODE config file in the "obs" section, set:
> raw_thresh = !=NA;
>
> MODE will read in the observation field and any grid point not
meeting the
> threshold criteria will be reset to a value of 0.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
> On Tue, Jul 11, 2017 at 12:40 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 11 12:40:17 2017: Request 81164 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: questions about MODE output (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81164 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a question about the calculation of contingency table
scores such
> as
> > CSI which are output in the cts.txt text file. For reference I've
> attached
> > that file plus the postscript file and the other two output files
from a
> > run
> > I did using forecast and observed radar reflectivity.
> >
> > In the postscript file plots showing the forecast objects with
> observation
> > outlines, I see the red forecast object which was matched with the
blue
> > outlined observation object. I see in this plot areas where there
are
> > "hits"
> > (FY_OY) where you can see red area overlaying the blue outlined
area. I
> see
> > "misses" (FN_OY) where the blue outlined area does not overlap a
red
> area.
> > Both the hits and misses have non-zero counts in their respective
columns
> > in
> > the cts.txt text file. I also see in the plot areas where the red
areas
> do
> > not overlay a blue outlined area which corresponds to "false
alarms"
> > (FY_ON), but the cts.txt file shows that there are no false alarm
events
> > with a 0 entry. The MET tutorial describes the output in the
cts.txt file
> > at:
> > Caution-http://www.dtcenter.org/met/users/support/online_tutorial/
> > METv5.2/tutorial.p
> > hp?name=mode&category=output
> >
> > The tutorial explains that the objects show 1 for the red areas
and 0 for
> > non-red and 1 for blue outlined areas and 0 for non-blue outlined
areas.
> If
> > a simple contingency table of outcomes is generated form this, why
are
> > there
> > no false alarm events?
> >
> > Another question regarding the first page of the postscript output
file.
> In
> > the table on the right side of the page, total interest values for
pairs
> > of simple objects are listed in sorted order. My understanding
from the
> > User's Guide was that those pairs of simple objects whose interest
value
> > fell below the interest threshold were not "matched", but if I
look in
> the
> > plot, I can see simple objects with red numbers (part of the
merged
> > forecast
> > and observed objects) which also show in the table on the right
with
> > interest values less than the threshold which are "matched". An
example
> of
> > this is object 6 (fcst) and object 5 (obs). I thought I read that
same
> > color
> > objects in both fields have been "matched". What is the
relationship
> > between
> > the interst threshold and matching. Also note that numbered simple
> objects
> > which are colored royal blue which should indicate "unmatched"
also
> appear
> > in the table of "pairs". I guess I don't quite understand the
presence of
> > "pairs" in the table versus the simple numbered objects appearing
in the
> > plots.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: MET Pcp-Combine error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 29 15:23:41 2019
CLASSIFICATION: UNCLASSIFIED
Good information, John. Very helpful. Now I see the difference between
the
occurrence of neg values near the edge of the domain as being
characterized as
missing due to being located beyond the range of the radar and the
occurrences
of -9999 values well within the range of the radar which are due to
some
other reason. In that past project, I think I recall that the missing
values
were occurring within the range of the radar which is why we chose to
replace
them with 0s.
Thanks for mentioning that Gen_Vx_Mask could also be used with this
data to
re-assign the values to -9999. I would like to try using the config
capabilities of -field for accomplishing the replacement of -3 values
with -9999 which you mention below using Pcp_Combine on the HPC using
V8.1.
This is good motivation for beginning the migration now.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, May 29, 2019 2:54 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET Pcp-
Combine
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
If you look at the plot I sent, you'll see that the -3 values (which
show up
as white) are in areas along the edge of the domain where there would
typically be missing data values. Those regions are beyond the range
of the
radars and are not observable. For this reason, they should be set to
bad
data and not 0. Setting them to 0 would artificially inflate the
correct
negatives counts and impact the resulting stats.
Yes, you're right. You could run gen_vx_mask from met-5.2 to reset
those
-3 values to -9999. The censor_thresh and censor_val settings,
available in
more recent releases, are just more convenient.
And if you search the user's guide, you should find a reference to
them in
there. However, they are not specifically mentioned in the
pcp_combine
section. Instead, they apply broadly to all of the MET tools where
config
files are used. And the pcp_combine "-field" option is like a mini
config
file that you set on the command line.
There are a few other settings like this that apply broadly to all
config
files and are not specifically listed for each tool... like file_type,
GRIB1_ptv, GRIB1_code, GRIB2_parm, GRIB2_parm_cat... and so on.
Thanks,
John
On Wed, May 29, 2019 at 1:22 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90341 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I recall that you provided some guidance on how to replace bad data
> values
> (-9999) with zeros back when I was working with radar reflectivity.
> See attached email. You used Gen_Vx_Mask to change missing (bad)
values to
> 0s.
> The
> reason for doing that was to assure that the categorical counts and
> statistics output by MODE would not be impacted by an artificially
low
> number of matched pairs caused by the presence of a large amount of
> missing data. Why in this case do you want to assign -9999 to the
> negative values instead of zeros which will result in better
> categorical statistics? I plan to run Grid-Stat and Wavelet-Stat
using
> the output from Pcp-Combine and will rely on the categorical and
> neighborhood statistics heavily. I may not be understanding
something
> here or I am applying a fix for something else where it's not
> appropriate.
>
> Also, I plan to transfer my data to the newer HPC and run the
commands
> you suggest below using the censor_thresh and censor_val options in
> V8.1. From your commands below it looks like you are using the
options
> with plot_data_plane. I checked the User's Guide for V8.1 and I
didn't
> see these options in the usage description.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, May 29, 2019 12:03 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> Pcp-Combine error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I think you should take a step back to understand this MRMS data
better.
> Try running MET's plot_data_plane tool to visualize it:
>
>
> /usr/local/met-5.2/bin/plot_data_plane
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
> level="L0";' -v 4
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: -3
>
> DEBUG 4: plane max: 175.7
>
>
> I've attached a PNG file showing the resulting image.
>
> Notice that where they have listed a value of -3, it sure looks like
> it should be missing data! Just like they don't store this data as
an
> accumulation interval correctly, they don't store these bad data
> values correctly either.
> Adding -3 together 3 times gives you a value of -9.
>
> This cannot easily be fixed in met-5.2, but in met-8.1 you can use
the
> censor_thresh and censor_val options to change those values of -3 to
> MET's value for bad data, -9999:
>
> /usr/local/met-8.1/bin/plot_data_plane
> GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps 'name="GaugeCorrQPE01H";
> level="L0";' -v 4
>
> DEBUG 3: Applying censor thresholds "==-3" and replacing with values
> "-9999.00000".
>
> DEBUG 3: Censored values for 8598569 of 24500000 grid points.
>
> DEBUG 1: Creating postscript file:
> GaugeCorr_QPE_01H_00.00_20160209-120000.ps
>
>
> Make sense?
>
>
> Thanks,
> John
>
> On Wed, May 29, 2019 at 11:23 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90
> > 341 >
> >
> > John -
> >
> > It worked on the first try and I believe the results are as
expected.
> > Used the commands in the attached text file built from your email.
> > Also see the attached log file. The output file is 94 mb, so I'll
> > send a link to your email address so you can download it. I
plotted
> > it using IDV and the range of the data values in the field (CONUS)
> > was -9 mm to 363mm. Not sure how -9 is generated, though. Maybe
from
> > neg reflectivity values? I'll have to look at the input obs to see
why.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, May 29, 2019 10:37 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > Pcp-Combine error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > OK, let's not use "A00" or "A01" anymore. Instead, let's just use
> > "L0" to indicate that the level value is 0. For some reason, the
> > MRMS folks don't encode their GRIB2 files as having an
accumulation
> > interval. Instead, the accumulation interval is indicated by the
> > variable name. For example, GaugeCorrQPE01H is a 1-hour
> > accumulation while GaugeCorrQPE06H is a 6-hour accum.
> >
> > Sorry for the confusion I may have caused in my last email.
> >
> > Yes, this type of sum command should work:
> >
> > /p/home/jraby/MET/met-5.2/bin/pcp_combine -sum 00000000_000000 1
> > 20160209_150000 03
> >
> >
/p/home/jraby/MET/MET_Pcp_Combine/results_20160209_DCstudy/MRMS_prec
> > ip _obs_3hr_accum_vt15Z_09FEB.nc -pcpdir
> > /p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr -field
> > 'name="GaugeCorrQPE01H"; level="L0";' -log
> >
/p/home/jraby/MET/MET_Pcp_Combine/logs/MRMS_pcp_combine_obs_accum_vt
> > 15
> > Z -v
> > 5
> >
> > Although I haven't tested it with your data or with version 5.2 of
> > MET be sure. Please just try running the command above to see if
it
> > works. It should find 3 GaugeCorr files, extract the
> > GaugeCorrQPE01H field from them, and add them up.
> >
> > Thanks,
> > John
> >
> > On Wed, May 29, 2019 at 10:22 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90
> > 341 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for the testing and diagnosis and providing 3 options for
> > > accomplishing the subtraction. I intentionally wanted to zero-
out
> > > the accum precip (analysis of 1-hr precip valid at 12Z) field at
> > > 12Z in order to track the model development of the precip
> > > specifically over the period from 12Z to 18Z with 0 as the
> > > starting point for this case study. In my script, the additional
> > > commands (commented out now) I use "-sum" to accumulate for the
> > > following hours up to 18Z. For the -sum I assume that I will use
> > > the same options
> > to
> > > describe the name and level per your examples below, correct?
I'm
> > assuming
> > > that if I have the same analysis fields for 13Z, 14Z, 15Z, ....,
> > > 18Z, I can just sum them per the other commands in my script to
> > > get the 6-hr accum
> > at
> > > 18Z. Will this work?
> > >
> > > I have V8.1 running in a Singularity container on an ARL HPC
> > > (Centennial) , but I haven't provisioned this HPC with all the
> > > data I need to start
> > using
> > > it
> > > operationally. I plan to soon, however. Looking forward to the
> > > enhancements.
> > > I'm running V5.2 on the operational system (Excalibur).
> > > Unfortunately,
> > the
> > > kernel is older and the implementation of the V8.1 Singularity
> > > container has being problematic. Working on it.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, May 29, 2019 9:45 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > > Pcp-Combine error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > > contained within the message prior to copying and pasting the
> > > address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > OK, I'm able to replicate the error message you're seeing.
Here's
> > > the first command listed in your script (with slightly modified
> > > paths):
> > >
> > > */usr/local/met-5.2/bin/pcp_combine \*
> > >
> > > *-subtract \*
> > >
> > > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> > >
> > > *GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01 \*
> > >
> > > *MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc \*
> > >
> > > *-field GaugeCorrQPE01H \*
> > >
> > > *-log MRMS_pcp_combine_obs_accum_vt12Z -v 5*
> > >
> > >
> > > I see that you're running met-5.2. The latest available version
> > > is
> > > 8.1 which includes many bugfixes and enhancements. If you're
able
> > > to upgrade to that, I'd encourage you to do so.
> > >
> > >
> > > Next, I see that you're subtracting the same field from itself
in
> > > this command... which should result in 0's everywhere. That's
> > > fine, if it makes your scripting easier, but I just wanted to
mention
> > > it.
> > >
> > >
> > > I see that you want to extract 1-hourly precip from this file.
> > > And running it though wgrib2 reveals that the field is named
> > > "GaugeCorrQPE01H", as
> > you've
> > > already figured out. By default, pcp_combine looks for fields
of
> > > accumulated precip, named "APCP". And so you're using the "-
field"
> > > command line option to override that. There are 2 problems
here.
> > > First, the "-field" option needs to specify a "name" and
"level".
> > > Second, this GRIB2 file doesn't contain a 1-hour accumulation.
> > > Since it's a 0-hour analysis file, it contains 0-hour precip.
So
> > > we'd need to use "A0" to access it.
> > >
> > >
> > > Here are 3 variations that all do what you're trying to do:
> > >
> > >
> > > (1) Using the -field option as you intended (the -field option
> > > overrides the
> > > 01 you have elsewhere):
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -subtract
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 01 GaugeCorr_QPE_01H_00.00_20160209-120000.grib2 01
> > > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc
> > > -field 'name="GaugeCorrQPE01H"; level="A00";' -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > (2) Instead of using "-field" just explicitly list the data you
> > > want from each
> > > file:
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -subtract
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H"; level="A00";'
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H";
> > > level="A00";' MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > (3) Since you're really just running this as a pass-through for
> > > the
> > 0-hour
> > > forecast, you could run a -add command for a single field
instead:
> > >
> > >
> > > /usr/local/met-5.2/bin/pcp_combine -add
> > > GaugeCorr_QPE_01H_00.00_20160209-120000.grib2
> > > 'name="GaugeCorrQPE01H"; level="A00";'
> > > MRMS_precip_obs_0hr_accum_vt12Z_09FEB.nc -log
> > > MRMS_pcp_combine_obs_accum_vt12Z -v 5
> > >
> > >
> > > All 3 options should give you the same result.
> > >
> > >
> > > John
> > >
> > > On Tue, May 28, 2019 at 2:29 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90
> > 341 >
> > > >
> > > > John -
> > > >
> > > > I've been trying to locate the syntax error in my run command
in
> > > > the attached shell script, but unable to see it. Could you
find
> > > > it
> for
> > > > me?
> > > > I've also attached the log file showing the error and some
> > > > intermediate output file produced my MET which is shown in the
> > > > log file. You already have the observation file (grib2).
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Friday, May 24, 2019 11:25 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90341] MET
> > > > Pcp-Combine error (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity
> > > > of all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > There is most likely a syntax error in your script named
> > > > "run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh". It
> > > > isn't a problem in your data. You're probably just missing a
> > > > semi-colon, single quote, or double quote somewhere on the
> > > > command line for pcp_combine. If you're not able to find it,
> > > > just send me the contents of that script and it should be
pretty easy
> > > > to locate.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Fri, May 24, 2019 at 6:29 AM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Fri May 24 06:29:15 2019: Request 90341 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: MET Pcp-Combine error (UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90
> > 341 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I haven't seen this type of error before. Do you have a
> > > > > suggestion for resolving this? See attached observation
file.
> > > > >
> > > > > > run_pcp_combine_gen_running_accum_MRMS_obs_DCstudy.sh
> > > > > DEBUG 1: Reading input file:
> > > > >
> > > > >
> > > >
/p/home/jraby/MET_obs/MRMS_prcip_obs/20160209/GaugeCorr/GaugeCor
> > > > r_
> > > > QPE_
> > > > 01H_00
> > > > > .00_20160209-120000.grib2
> > > > > ERROR :
> > > > > ERROR : yyerror() -> syntax error in file
"config_25586_0_.temp"
> > > > > ERROR :
> > > > > ERROR : line = 2
> > > > > ERROR :
> > > > > ERROR : column = 1
> > > > > ERROR :
> > > > > ERROR : text = "(nul)"
> > > > > ERROR :
> > > > > ERROR :
> > > > > ERROR : (nul)
> > > > > ERROR : (nul)
> > > > > ERROR :
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you have a question about the categorical counts and
> statistics included in the output of MODE. You're wondering why the
> false alarm counts (FY_ON) are set to 0. And looking closely at the
> TOTAL column in the cts.txt file, you'll see that the number of
matched
> pairs is 222.
> However, looking in the NetCDF file you sent, I see that the grid
> dimension is 204 x 204... so the total number of grid points
*should* be
> 41,616!
>
> The answer lies in the PostScript plot you sent. While all 41,616
> forecast grid points contain valid data, only 222 of the observation
> grid points contain valid data values. All that gray area in the
> observation field is missing data. MODE only computes categorical
> statistics at grid points which contain valid data in both fields.
>
> I assume that you'd like the observation field missing data values
to
> actually be set to a value of 0 instead.
>
> You could do this in one of two ways...
>
> (1) Run the observation field through the gen_vx_mask tool in MET as
a
> way of preprocessing the data and changing those bad data values to
0.
> This would be necessary if you want to also run this data through
Grid-Stat.
>
> (2) In the MODE config file in the "obs" section, set:
> raw_thresh = !=NA;
>
> MODE will read in the observation field and any grid point not
meeting
> the threshold criteria will be reset to a value of 0.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
> On Tue, Jul 11, 2017 at 12:40 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 11 12:40:17 2017: Request 81164 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: questions about MODE output (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81
> > 164 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a question about the calculation of contingency table
scores
> > such
> as
> > CSI which are output in the cts.txt text file. For reference I've
> attached
> > that file plus the postscript file and the other two output files
> > from a run I did using forecast and observed radar reflectivity.
> >
> > In the postscript file plots showing the forecast objects with
> observation
> > outlines, I see the red forecast object which was matched with the
> > blue outlined observation object. I see in this plot areas where
> > there are "hits"
> > (FY_OY) where you can see red area overlaying the blue outlined
> > area. I
> see
> > "misses" (FN_OY) where the blue outlined area does not overlap a
red
> area.
> > Both the hits and misses have non-zero counts in their respective
> > columns in the cts.txt text file. I also see in the plot areas
where
> > the red areas
> do
> > not overlay a blue outlined area which corresponds to "false
alarms"
> > (FY_ON), but the cts.txt file shows that there are no false alarm
> > events with a 0 entry. The MET tutorial describes the output in
the
> > cts.txt file
> > at:
> > Caution-Caution-
http://www.dtcenter.org/met/users/support/online_tut
> > orial/
> > METv5.2/tutorial.p
> > hp?name=mode&category=output
> >
> > The tutorial explains that the objects show 1 for the red areas
and
> > 0 for non-red and 1 for blue outlined areas and 0 for non-blue
outlined
> > areas.
> If
> > a simple contingency table of outcomes is generated form this, why
> > are there no false alarm events?
> >
> > Another question regarding the first page of the postscript output
file.
> In
> > the table on the right side of the page, total interest values for
> > pairs of simple objects are listed in sorted order. My
understanding
> > from the User's Guide was that those pairs of simple objects whose
> > interest value fell below the interest threshold were not
"matched",
> > but if I look in
> the
> > plot, I can see simple objects with red numbers (part of the
merged
> > forecast and observed objects) which also show in the table on the
> > right with interest values less than the threshold which are
> > "matched". An example
> of
> > this is object 6 (fcst) and object 5 (obs). I thought I read that
> > same color objects in both fields have been "matched". What is the
> > relationship between the interst threshold and matching. Also note
> > that numbered simple
> objects
> > which are colored royal blue which should indicate "unmatched"
also
> appear
> > in the table of "pairs". I guess I don't quite understand the
> > presence of "pairs" in the table versus the simple numbered
objects
> > appearing in the plots.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list