[Met_help] [rt.rap.ucar.edu #88643] History for Different Bad Value Problem
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:07:09 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, our email is down so. I am getting a bad data error in stat analysis. Could this be an extension of the problem you found for hira processing to stat_analysis. I sent you the a file containing the stat anal command line, the output, and the met data file its reading.
Please replay to this email.
Thanks
Sent from my Verizon, Samsung Galaxy smartphone
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Different Bad Value Problem
From: John Halley Gotway
Time: Fri Jan 25 11:09:13 2019
Bob,
Yes, that could very well be the case. I presume that you're running
Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
To test, I used a sample MPR output file that I had generated. Some
of the
values in the CLIMO_MEAN column are bad data values.
I ran a "filter" job and matched 2386 lines:
*stat_analysis -lookin
out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
-line_type MPR -dump_row test.dump -interp_mthd NEAREST*
*DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
Next, I reran the same job but added this job option:
*-column_thresh CLIMO_MEAN ne-9999*
That tells Stat-Analysis to only use line where CLIMO_MEAN is *not*
bad
data. And this job uses 1821, skipping over the 565 lines where
CLIMO_MEAN
is bad data.
*DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
You could try adding that to your job command to see if the runtime
error
goes away.
Thanks,
John
On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> Transaction: Ticket created by craigr3 at cox.net
> Queue: met_help
> Subject: Different Bad Value Problem
> Owner: Nobody
> Requestors: craigr3 at cox.net
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
>
>
> John, our email is down so. I am getting a bad data error in stat
> analysis. Could this be an extension of the problem you found for
hira
> processing to stat_analysis. I sent you the a file containing the
stat
> anal command line, the output, and the met data file its reading.
> Please replay to this email.
> Thanks
> Sent from my Verizon, Samsung Galaxy smartphone
>
------------------------------------------------
Subject: Different Bad Value Problem
From: craigr3 at cox.net
Time: Fri Jan 25 13:00:10 2019
John, I tried your fix but get the same error. I placed the data file
i sent you in an editor and searched for -9999 and did not find a bad
value. There was either NA in the climo column or a valid number.
Bob
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: John Halley Gotway via RT
<met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value Problem
Bob,
Yes, that could very well be the case. I presume that you're running
Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
To test, I used a sample MPR output file that I had generated. Some
of the
values in the CLIMO_MEAN column are bad data values.
I ran a "filter" job and matched 2386 lines:
*stat_analysis -lookin
out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
-line_type MPR -dump_row test.dump -interp_mthd NEAREST*
*DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
Next, I reran the same job but added this job option:
*-column_thresh CLIMO_MEAN ne-9999*
That tells Stat-Analysis to only use line where CLIMO_MEAN is *not*
bad
data. And this job uses 1821, skipping over the 565 lines where
CLIMO_MEAN
is bad data.
*DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
You could try adding that to your job command to see if the runtime
error
goes away.
Thanks,
John
On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> Transaction: Ticket created by craigr3 at cox.net
> Queue: met_help
> Subject: Different Bad Value Problem
> Owner: Nobody
> Requestors: craigr3 at cox.net
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
>
>
> John, our email is down so. I am getting a bad data error in stat
> analysis. Could this be an extension of the problem you found for
hira
> processing to stat_analysis. I sent you the a file containing the
stat
> anal command line, the output, and the met data file its reading.
> Please replay to this email.
> Thanks
> Sent from my Verizon, Samsung Galaxy smartphone
>
------------------------------------------------
Subject: Different Bad Value Problem
From: John Halley Gotway
Time: Fri Jan 25 13:38:14 2019
Bob,
Too bad that didn't do the trick. Are you able to place the file
somewhere
I can access it? And then send me the stat_analysis command that
results
in the error?
Thanks,
John
On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> John, I tried your fix but get the same error. I placed the data
file i
> sent you in an editor and searched for -9999 and did not find a bad
value.
> There was either NA in the climo column or a valid number.
> Bob
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value
> Problem
> Bob,
>
> Yes, that could very well be the case. I presume that you're
running
> Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
>
> To test, I used a sample MPR output file that I had generated. Some
of the
> values in the CLIMO_MEAN column are bad data values.
>
> I ran a "filter" job and matched 2386 lines:
> *stat_analysis -lookin
> out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
>
> Next, I reran the same job but added this job option:
> *-column_thresh CLIMO_MEAN ne-9999*
> That tells Stat-Analysis to only use line where CLIMO_MEAN is *not*
bad
> data. And this job uses 1821, skipping over the 565 lines where
CLIMO_MEAN
> is bad data.
> *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
>
> You could try adding that to your job command to see if the runtime
error
> goes away.
>
> Thanks,
> John
>
> On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > Transaction: Ticket created by craigr3 at cox.net
> > Queue: met_help
> > Subject: Different Bad Value Problem
> > Owner: Nobody
> > Requestors: craigr3 at cox.net
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> >
> >
> > John, our email is down so. I am getting a bad data error in stat
> > analysis. Could this be an extension of the problem you found for
hira
> > processing to stat_analysis. I sent you the a file containing
the stat
> > anal command line, the output, and the met data file its reading.
> > Please replay to this email.
> > Thanks
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: craigr3 at cox.net
Time: Fri Jan 25 14:07:33 2019
John, I sent it to you in ARL safe a couple hours ago. Did you get
an email from them?
Bob
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: John Halley Gotway via RT
<met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value Problem
Bob,
Too bad that didn't do the trick. Are you able to place the file
somewhere
I can access it? And then send me the stat_analysis command that
results
in the error?
Thanks,
John
On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> John, I tried your fix but get the same error. I placed the data
file i
> sent you in an editor and searched for -9999 and did not find a bad
value.
> There was either NA in the climo column or a valid number.
> Bob
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value
> Problem
> Bob,
>
> Yes, that could very well be the case. I presume that you're
running
> Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
>
> To test, I used a sample MPR output file that I had generated. Some
of the
> values in the CLIMO_MEAN column are bad data values.
>
> I ran a "filter" job and matched 2386 lines:
> *stat_analysis -lookin
> out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
>
> Next, I reran the same job but added this job option:
> *-column_thresh CLIMO_MEAN ne-9999*
> That tells Stat-Analysis to only use line where CLIMO_MEAN is *not*
bad
> data. And this job uses 1821, skipping over the 565 lines where
CLIMO_MEAN
> is bad data.
> *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
>
> You could try adding that to your job command to see if the runtime
error
> goes away.
>
> Thanks,
> John
>
> On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > Transaction: Ticket created by craigr3 at cox.net
> > Queue: met_help
> > Subject: Different Bad Value Problem
> > Owner: Nobody
> > Requestors: craigr3 at cox.net
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> >
> >
> > John, our email is down so. I am getting a bad data error in stat
> > analysis. Could this be an extension of the problem you found for
hira
> > processing to stat_analysis. I sent you the a file containing
the stat
> > anal command line, the output, and the met data file its reading.
> > Please replay to this email.
> > Thanks
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: craigr3 at cox.net
Time: Fri Jan 25 14:09:36 2019
John, I think our email is up. I attached the data file to an earlier
email.
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: John Halley Gotway via RT
<met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value Problem
Bob,
Too bad that didn't do the trick. Are you able to place the file
somewhere
I can access it? And then send me the stat_analysis command that
results
in the error?
Thanks,
John
On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> John, I tried your fix but get the same error. I placed the data
file i
> sent you in an editor and searched for -9999 and did not find a bad
value.
> There was either NA in the climo column or a valid number.
> Bob
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value
> Problem
> Bob,
>
> Yes, that could very well be the case. I presume that you're
running
> Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
>
> To test, I used a sample MPR output file that I had generated. Some
of the
> values in the CLIMO_MEAN column are bad data values.
>
> I ran a "filter" job and matched 2386 lines:
> *stat_analysis -lookin
> out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
>
> Next, I reran the same job but added this job option:
> *-column_thresh CLIMO_MEAN ne-9999*
> That tells Stat-Analysis to only use line where CLIMO_MEAN is *not*
bad
> data. And this job uses 1821, skipping over the 565 lines where
CLIMO_MEAN
> is bad data.
> *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
>
> You could try adding that to your job command to see if the runtime
error
> goes away.
>
> Thanks,
> John
>
> On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > Transaction: Ticket created by craigr3 at cox.net
> > Queue: met_help
> > Subject: Different Bad Value Problem
> > Owner: Nobody
> > Requestors: craigr3 at cox.net
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> >
> >
> > John, our email is down so. I am getting a bad data error in stat
> > analysis. Could this be an extension of the problem you found for
hira
> > processing to stat_analysis. I sent you the a file containing
the stat
> > anal command line, the output, and the met data file its reading.
> > Please replay to this email.
> > Thanks
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: John Halley Gotway
Time: Fri Jan 25 14:17:08 2019
Bob,
No I have not received an ARL SAFE email. And I believe Matt Sittel
was
trying to send me data through there as well. But I haven't seen
anything
come through.
John
On Fri, Jan 25, 2019 at 2:10 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> John, I think our email is up. I attached the data file to an
earlier
> email.
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
craigr3 at cox.net
> Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> Bob,
>
> Too bad that didn't do the trick. Are you able to place the file
somewhere
> I can access it? And then send me the stat_analysis command that
results
> in the error?
>
> Thanks,
> John
>
> On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> > John, I tried your fix but get the same error. I placed the data
file i
> > sent you in an editor and searched for -9999 and did not find a
bad
> value.
> > There was either NA in the climo column or a valid number.
> > Bob
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> > -------- Original message --------From: John Halley Gotway via RT
<
> > met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> > craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different
Bad
> Value
> > Problem
> > Bob,
> >
> > Yes, that could very well be the case. I presume that you're
running
> > Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
> >
> > To test, I used a sample MPR output file that I had generated.
Some of
> the
> > values in the CLIMO_MEAN column are bad data values.
> >
> > I ran a "filter" job and matched 2386 lines:
> > *stat_analysis -lookin
> > out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> > -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> > *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
> >
> > Next, I reran the same job but added this job option:
> > *-column_thresh CLIMO_MEAN ne-9999*
> > That tells Stat-Analysis to only use line where CLIMO_MEAN is
*not* bad
> > data. And this job uses 1821, skipping over the 565 lines where
> CLIMO_MEAN
> > is bad data.
> > *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
> >
> > You could try adding that to your job command to see if the
runtime error
> > goes away.
> >
> > Thanks,
> > John
> >
> > On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > > Transaction: Ticket created by craigr3 at cox.net
> > > Queue: met_help
> > > Subject: Different Bad Value Problem
> > > Owner: Nobody
> > > Requestors: craigr3 at cox.net
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643
> >
> > >
> > >
> > >
> > > John, our email is down so. I am getting a bad data error in
stat
> > > analysis. Could this be an extension of the problem you found
for hira
> > > processing to stat_analysis. I sent you the a file containing
the
> stat
> > > anal command line, the output, and the met data file its
reading.
> > > Please replay to this email.
> > > Thanks
> > > Sent from my Verizon, Samsung Galaxy smartphone
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: craigr3 at cox.net
Time: Fri Jan 25 14:43:29 2019
I resent it via ARL safe in case it got got up in our email problem.
We seem to be getting email working again. I tried to attach it to my
work email but it was slightly to big. I will try to compress it down
to below 5 mb and resend via email.
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: John Halley Gotway via RT
<met_help at ucar.edu> Date: 1/25/19 3:17 PM (GMT-06:00) To:
craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value Problem
Bob,
No I have not received an ARL SAFE email. And I believe Matt Sittel
was
trying to send me data through there as well. But I haven't seen
anything
come through.
John
On Fri, Jan 25, 2019 at 2:10 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> John, I think our email is up. I attached the data file to an
earlier
> email.
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
craigr3 at cox.net
> Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> Bob,
>
> Too bad that didn't do the trick. Are you able to place the file
somewhere
> I can access it? And then send me the stat_analysis command that
results
> in the error?
>
> Thanks,
> John
>
> On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> > John, I tried your fix but get the same error. I placed the data
file i
> > sent you in an editor and searched for -9999 and did not find a
bad
> value.
> > There was either NA in the climo column or a valid number.
> > Bob
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> > -------- Original message --------From: John Halley Gotway via RT
<
> > met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> > craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different
Bad
> Value
> > Problem
> > Bob,
> >
> > Yes, that could very well be the case. I presume that you're
running
> > Stat-Analysis to read matched pair data lines (MPR) and recompute
stats.
> >
> > To test, I used a sample MPR output file that I had generated.
Some of
> the
> > values in the CLIMO_MEAN column are bad data values.
> >
> > I ran a "filter" job and matched 2386 lines:
> > *stat_analysis -lookin
> > out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> > -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> > *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
> >
> > Next, I reran the same job but added this job option:
> > *-column_thresh CLIMO_MEAN ne-9999*
> > That tells Stat-Analysis to only use line where CLIMO_MEAN is
*not* bad
> > data. And this job uses 1821, skipping over the 565 lines where
> CLIMO_MEAN
> > is bad data.
> > *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
> >
> > You could try adding that to your job command to see if the
runtime error
> > goes away.
> >
> > Thanks,
> > John
> >
> > On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > > Transaction: Ticket created by craigr3 at cox.net
> > > Queue: met_help
> > > Subject: Different Bad Value Problem
> > > Owner: Nobody
> > > Requestors: craigr3 at cox.net
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643
> >
> > >
> > >
> > >
> > > John, our email is down so. I am getting a bad data error in
stat
> > > analysis. Could this be an extension of the problem you found
for hira
> > > processing to stat_analysis. I sent you the a file containing
the
> stat
> > > anal command line, the output, and the met data file its
reading.
> > > Please replay to this email.
> > > Thanks
> > > Sent from my Verizon, Samsung Galaxy smartphone
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: John Halley Gotway
Time: Fri Jan 25 15:35:36 2019
Bob,
Great, I got the other attachment you sent. I still don't know
exactly
what Stat-Analysis job you're running, but I made an educated guess
which
did result in a runtime error:
*/usr/local/met-6.1/bin/stat_analysis \*
* -lookin point_stat_global_17km_all_120000L_20171205_120000V.stat \*
* -job aggregate_stat -line_type MPR -out_line_type PCT \*
*-out_fcst_thresh eq0.25 -out_obs_thresh gt0*
*ERROR : Nx2ContingencyTable::inc_event(double) -> bad value ...
5.07874*
The MPR output from Point-Stat contains both the raw fcst/obs matched
pair
values and the neighborhood fractional coverage values. When doing
probabilistic vx with an Nx2 contingency table, the forecast values
should
be probabilities between 0 and 1... not the raw APCP values, like
5.07874.
We can control this by telling Stat-Analysis to only use MPR lines
where
INTERP_MTHD = NBRHD_SQUARE. Looks like your data contains output for
4
different thresholds (>1, >=10, >=25, >=50) and each of those
thresholds
requires a different job. So I put multiple jobs into one config
file.
*/usr/local/met-6.1/bin/stat_analysis \*
*-lookin point_stat_global_17km_all_120000L_20171205_120000V.stat \*
*-config STATAnalysisConfig_hira*
And that generates these files:
*APCP_gt1.stat*
*APCP_ge10.statAPCP_ge25.statAPCP_ge50.stat*
Since you're running HiRA, I expected to see a larger value for
INTERP_PNTS. Since you're computing HiRA in a neighborhood of size
one,
the probability values are always either 0 (no-event) or 1 (event).
Looking at your MPR data, that's exactly what I see... 0's and 1's in
the
FCST column.
So I'd recommend increasing the HiRA neighborhood size to larger
values...
which you were probably planning to do anyway. That'll generate
smoother
probability values between 0 and 1 and produce more interesting output
from
STAT-Analysis.
In this config file, I'm dumping the PCT counts, but you can easily
change
the output line type to PSTD.
Thanks,
John
On Fri, Jan 25, 2019 at 2:44 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> I resent it via ARL safe in case it got got up in our email problem.
We
> seem to be getting email working again. I tried to attach it to my
work
> email but it was slightly to big. I will try to compress it down to
below
> 5 mb and resend via email.
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 3:17 PM (GMT-06:00) To:
craigr3 at cox.net
> Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> Bob,
>
> No I have not received an ARL SAFE email. And I believe Matt Sittel
was
> trying to send me data through there as well. But I haven't seen
anything
> come through.
>
> John
>
> On Fri, Jan 25, 2019 at 2:10 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> > John, I think our email is up. I attached the data file to an
earlier
> > email.
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> > -------- Original message --------From: John Halley Gotway via RT
<
> > met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
> craigr3 at cox.net
> > Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> > Bob,
> >
> > Too bad that didn't do the trick. Are you able to place the file
> somewhere
> > I can access it? And then send me the stat_analysis command that
results
> > in the error?
> >
> > Thanks,
> > John
> >
> > On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> > >
> > > John, I tried your fix but get the same error. I placed the
data file
> i
> > > sent you in an editor and searched for -9999 and did not find a
bad
> > value.
> > > There was either NA in the climo column or a valid number.
> > > Bob
> > >
> > >
> > > Sent from my Verizon, Samsung Galaxy smartphone
> > > -------- Original message --------From: John Halley Gotway via
RT <
> > > met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> > > craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different
Bad
> > Value
> > > Problem
> > > Bob,
> > >
> > > Yes, that could very well be the case. I presume that you're
running
> > > Stat-Analysis to read matched pair data lines (MPR) and
recompute
> stats.
> > >
> > > To test, I used a sample MPR output file that I had generated.
Some of
> > the
> > > values in the CLIMO_MEAN column are bad data values.
> > >
> > > I ran a "filter" job and matched 2386 lines:
> > > *stat_analysis -lookin
> > > out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> > > -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> > > *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
> > >
> > > Next, I reran the same job but added this job option:
> > > *-column_thresh CLIMO_MEAN ne-9999*
> > > That tells Stat-Analysis to only use line where CLIMO_MEAN is
*not* bad
> > > data. And this job uses 1821, skipping over the 565 lines where
> > CLIMO_MEAN
> > > is bad data.
> > > *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
> > >
> > > You could try adding that to your job command to see if the
runtime
> error
> > > goes away.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > > > Transaction: Ticket created by craigr3 at cox.net
> > > > Queue: met_help
> > > > Subject: Different Bad Value Problem
> > > > Owner: Nobody
> > > > Requestors: craigr3 at cox.net
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643
> > >
> > > >
> > > >
> > > >
> > > > John, our email is down so. I am getting a bad data error in
stat
> > > > analysis. Could this be an extension of the problem you found
for
> hira
> > > > processing to stat_analysis. I sent you the a file
containing the
> > stat
> > > > anal command line, the output, and the met data file its
reading.
> > > > Please replay to this email.
> > > > Thanks
> > > > Sent from my Verizon, Samsung Galaxy smartphone
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: Different Bad Value Problem
From: craigr3 at cox.net
Time: Fri Jan 25 15:52:35 2019
Thanks John. Could you send your reply to my air force email
robert.craig.2 at us.af.mil.I will try your fix. We use hira to compare
a global file with a regional file. So the global file has no
neighborhood but the regional file use a neighborhood size that
roughly matches the global file resolution.
ThanksBobSent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: John Halley Gotway via RT
<met_help at ucar.edu> Date: 1/25/19 4:35 PM (GMT-06:00) To:
craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different Bad
Value Problem
Bob,
Great, I got the other attachment you sent. I still don't know
exactly
what Stat-Analysis job you're running, but I made an educated guess
which
did result in a runtime error:
*/usr/local/met-6.1/bin/stat_analysis \*
* -lookin point_stat_global_17km_all_120000L_20171205_120000V.stat \*
* -job aggregate_stat -line_type MPR -out_line_type PCT \*
*-out_fcst_thresh eq0.25 -out_obs_thresh gt0*
*ERROR : Nx2ContingencyTable::inc_event(double) -> bad value ...
5.07874*
The MPR output from Point-Stat contains both the raw fcst/obs matched
pair
values and the neighborhood fractional coverage values. When doing
probabilistic vx with an Nx2 contingency table, the forecast values
should
be probabilities between 0 and 1... not the raw APCP values, like
5.07874.
We can control this by telling Stat-Analysis to only use MPR lines
where
INTERP_MTHD = NBRHD_SQUARE. Looks like your data contains output for
4
different thresholds (>1, >=10, >=25, >=50) and each of those
thresholds
requires a different job. So I put multiple jobs into one config
file.
*/usr/local/met-6.1/bin/stat_analysis \*
*-lookin point_stat_global_17km_all_120000L_20171205_120000V.stat \*
*-config STATAnalysisConfig_hira*
And that generates these files:
*APCP_gt1.stat*
*APCP_ge10.statAPCP_ge25.statAPCP_ge50.stat*
Since you're running HiRA, I expected to see a larger value for
INTERP_PNTS. Since you're computing HiRA in a neighborhood of size
one,
the probability values are always either 0 (no-event) or 1 (event).
Looking at your MPR data, that's exactly what I see... 0's and 1's in
the
FCST column.
So I'd recommend increasing the HiRA neighborhood size to larger
values...
which you were probably planning to do anyway. That'll generate
smoother
probability values between 0 and 1 and produce more interesting output
from
STAT-Analysis.
In this config file, I'm dumping the PCT counts, but you can easily
change
the output line type to PSTD.
Thanks,
John
On Fri, Jan 25, 2019 at 2:44 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
>
> I resent it via ARL safe in case it got got up in our email
problem. We
> seem to be getting email working again. I tried to attach it to my
work
> email but it was slightly to big. I will try to compress it down to
below
> 5 mb and resend via email.
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: John Halley Gotway via RT <
> met_help at ucar.edu> Date: 1/25/19 3:17 PM (GMT-06:00) To:
craigr3 at cox.net
> Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> Bob,
>
> No I have not received an ARL SAFE email. And I believe Matt Sittel
was
> trying to send me data through there as well. But I haven't seen
anything
> come through.
>
> John
>
> On Fri, Jan 25, 2019 at 2:10 PM craigr3 at cox.net via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> >
> > John, I think our email is up. I attached the data file to an
earlier
> > email.
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> > -------- Original message --------From: John Halley Gotway via RT
<
> > met_help at ucar.edu> Date: 1/25/19 2:38 PM (GMT-06:00) To:
> craigr3 at cox.net
> > Subject: Re: [rt.rap.ucar.edu #88643] Different Bad Value Problem
> > Bob,
> >
> > Too bad that didn't do the trick. Are you able to place the file
> somewhere
> > I can access it? And then send me the stat_analysis command that
results
> > in the error?
> >
> > Thanks,
> > John
> >
> > On Fri, Jan 25, 2019 at 1:00 PM craigr3 at cox.net via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643 >
> > >
> > > John, I tried your fix but get the same error. I placed the
data file
> i
> > > sent you in an editor and searched for -9999 and did not find a
bad
> > value.
> > > There was either NA in the climo column or a valid number.
> > > Bob
> > >
> > >
> > > Sent from my Verizon, Samsung Galaxy smartphone
> > > -------- Original message --------From: John Halley Gotway via
RT <
> > > met_help at ucar.edu> Date: 1/25/19 12:09 PM (GMT-06:00) To:
> > > craigr3 at cox.net Subject: Re: [rt.rap.ucar.edu #88643] Different
Bad
> > Value
> > > Problem
> > > Bob,
> > >
> > > Yes, that could very well be the case. I presume that you're
running
> > > Stat-Analysis to read matched pair data lines (MPR) and
recompute
> stats.
> > >
> > > To test, I used a sample MPR output file that I had generated.
Some of
> > the
> > > values in the CLIMO_MEAN column are bad data values.
> > >
> > > I ran a "filter" job and matched 2386 lines:
> > > *stat_analysis -lookin
> > > out/point_stat_korea_4_all_360000L_20171202_120000V_mpr.txt -job
filter
> > > -line_type MPR -dump_row test.dump -interp_mthd NEAREST*
> > > *DEBUG 2: Job 1 used 2386 out of 2386 STAT lines.*
> > >
> > > Next, I reran the same job but added this job option:
> > > *-column_thresh CLIMO_MEAN ne-9999*
> > > That tells Stat-Analysis to only use line where CLIMO_MEAN is
*not* bad
> > > data. And this job uses 1821, skipping over the 565 lines where
> > CLIMO_MEAN
> > > is bad data.
> > > *DEBUG 2: Job 1 used 1821 out of 1821 STAT lines.*
> > >
> > > You could try adding that to your job command to see if the
runtime
> error
> > > goes away.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jan 25, 2019 at 10:38 AM craigr3 at cox.net via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > Fri Jan 25 10:38:06 2019: Request 88643 was acted upon.
> > > > Transaction: Ticket created by craigr3 at cox.net
> > > > Queue: met_help
> > > > Subject: Different Bad Value Problem
> > > > Owner: Nobody
> > > > Requestors: craigr3 at cox.net
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88643
> > >
> > > >
> > > >
> > > >
> > > > John, our email is down so. I am getting a bad data error in
stat
> > > > analysis. Could this be an extension of the problem you found
for
> hira
> > > > processing to stat_analysis. I sent you the a file
containing the
> > stat
> > > > anal command line, the output, and the met data file its
reading.
> > > > Please replay to this email.
> > > > Thanks
> > > > Sent from my Verizon, Samsung Galaxy smartphone
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list