[Met_help] [rt.rap.ucar.edu #92376] History for Bad Value in Point Stat

John Halley Gotway via RT met_help at ucar.edu
Thu Oct 31 09:25:42 MDT 2019


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

John, I am verifying ensemble data using point stat and have the outputs set
to be PSTD and MPR.  I am also you NCEP climo for the climo field.  When I
run this, I get the following error message.  

 

DEBUG 2: 

DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type ADPSFC,
over region FULL, for interpolation method BILIN(4), using 7858 pairs.

DEBUG 3: Number of matched pairs  = 7858

DEBUG 3: Observations processed   = 665923

DEBUG 3: Rejected: SID exclusion  = 0

DEBUG 3: Rejected: obs type       = 596676

DEBUG 3: Rejected: valid time     = 0

DEBUG 3: Rejected: bad obs value  = 0

DEBUG 3: Rejected: off the grid   = 17682

DEBUG 3: Rejected: topography     = 0

DEBUG 3: Rejected: level mismatch = 26887

DEBUG 3: Rejected: quality marker = 0

DEBUG 3: Rejected: message type   = 3380

DEBUG 3: Rejected: masking region = 0

DEBUG 3: Rejected: bad fcst value = 6

DEBUG 3: Rejected: duplicates     = 13434

DEBUG 2: Computing Probabilistic Statistics.

DEBUG 2: Deriving normal approximation of climatological probabilities for
threshold >15.

ERROR  : 

ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value ... -9999

ERROR  :

 

To find out what was going on I turned off the PSTD output, reran, and got
an output .stat file (attached).  Looking at the output file, the climo data
is missing for some grid points.  As another test, I took this file and ran
it through stat_anal and got the same error.  So then I added
"-column_thresh CLIMO_MEAN !=-9999" to stat anal and I got an output file.
My stat anal job is below:

 

/h/MET/bin/stat_analysis -lookin
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_meps_none
_000000L_20190921_000000V.stat -out
/h/data/global/WXQC/data/met/summary/test -tmp_dir
/h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
-fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1 -out_obs_thresh
ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000 -by
FCST_LEAD -by fcst_init_end -dump_row
/h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh CLIMO_MEAN
!=-9999

 

So, the question is how can I get point stat not to error when there is
missing climo data and trying to calculate PSTD?  My met config file is
attached.

 

Thanks

Bob



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Mon Sep 30 09:30:03 2019

Bob,

That's a great question.  I do vaguely recall thinking about this
detail
during development - how should we handle cases where the climo mean
contains bad data.  And I assumed that if a climo field was specified,
it
would always contain valid data values over the same grid points that
the
forecast does.  It appears this was a bad assumption.  And in
particular
this is more likely to occur when climo fields are automatically
regridded
to the user's model domain - where bad data values could be
introduced,
particularly along the edge of the domain.

So the question is to you, how *should* this case be handled?
Assuming the
user has specified climo mean and/or standard deviation data, if we
encounter bad climo values, should we exclude that matched pair from
the
verification?

Thanks,
John

On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: Bad Value in Point Stat
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>
>
> John, I am verifying ensemble data using point stat and have the
outputs
> set
> to be PSTD and MPR.  I am also you NCEP climo for the climo field.
When I
> run this, I get the following error message.
>
>
>
> DEBUG 2:
>
> DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type
ADPSFC,
> over region FULL, for interpolation method BILIN(4), using 7858
pairs.
>
> DEBUG 3: Number of matched pairs  = 7858
>
> DEBUG 3: Observations processed   = 665923
>
> DEBUG 3: Rejected: SID exclusion  = 0
>
> DEBUG 3: Rejected: obs type       = 596676
>
> DEBUG 3: Rejected: valid time     = 0
>
> DEBUG 3: Rejected: bad obs value  = 0
>
> DEBUG 3: Rejected: off the grid   = 17682
>
> DEBUG 3: Rejected: topography     = 0
>
> DEBUG 3: Rejected: level mismatch = 26887
>
> DEBUG 3: Rejected: quality marker = 0
>
> DEBUG 3: Rejected: message type   = 3380
>
> DEBUG 3: Rejected: masking region = 0
>
> DEBUG 3: Rejected: bad fcst value = 6
>
> DEBUG 3: Rejected: duplicates     = 13434
>
> DEBUG 2: Computing Probabilistic Statistics.
>
> DEBUG 2: Deriving normal approximation of climatological
probabilities for
> threshold >15.
>
> ERROR  :
>
> ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
-9999
>
> ERROR  :
>
>
>
> To find out what was going on I turned off the PSTD output, reran,
and got
> an output .stat file (attached).  Looking at the output file, the
climo
> data
> is missing for some grid points.  As another test, I took this file
and ran
> it through stat_anal and got the same error.  So then I added
> "-column_thresh CLIMO_MEAN !=-9999" to stat anal and I got an output
file.
> My stat anal job is below:
>
>
>
> /h/MET/bin/stat_analysis -lookin
>
>
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_meps_none
> _000000L_20190921_000000V.stat -out
> /h/data/global/WXQC/data/met/summary/test -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
> -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
-out_obs_thresh
> ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> FCST_LEAD -by fcst_init_end -dump_row
> /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
CLIMO_MEAN
> !=-9999
>
>
>
> So, the question is how can I get point stat not to error when there
is
> missing climo data and trying to calculate PSTD?  My met config file
is
> attached.
>
>
>
> Thanks
>
> Bob
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in Point Stat
From: robert.craig.2 at us.af.mil
Time: Mon Sep 30 09:49:05 2019

John, If the climo data has missing values, then those model/ob pairs
should be excluded from the statistic calculation.  This would be
similar behavior to the column_thresh capability in stat_analysis.

Thanks
Bob

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, September 30, 2019 10:30 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
Point Stat

Bob,

That's a great question.  I do vaguely recall thinking about this
detail during development - how should we handle cases where the climo
mean contains bad data.  And I assumed that if a climo field was
specified, it would always contain valid data values over the same
grid points that the forecast does.  It appears this was a bad
assumption.  And in particular this is more likely to occur when climo
fields are automatically regridded to the user's model domain - where
bad data values could be introduced, particularly along the edge of
the domain.

So the question is to you, how *should* this case be handled?
Assuming the user has specified climo mean and/or standard deviation
data, if we encounter bad climo values, should we exclude that matched
pair from the verification?

Thanks,
John

On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: Bad Value in Point Stat
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
> >
>
>
> John, I am verifying ensemble data using point stat and have the
> outputs set to be PSTD and MPR.  I am also you NCEP climo for the
> climo field.  When I run this, I get the following error message.
>
>
>
> DEBUG 2:
>
> DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using
7858 pairs.
>
> DEBUG 3: Number of matched pairs  = 7858
>
> DEBUG 3: Observations processed   = 665923
>
> DEBUG 3: Rejected: SID exclusion  = 0
>
> DEBUG 3: Rejected: obs type       = 596676
>
> DEBUG 3: Rejected: valid time     = 0
>
> DEBUG 3: Rejected: bad obs value  = 0
>
> DEBUG 3: Rejected: off the grid   = 17682
>
> DEBUG 3: Rejected: topography     = 0
>
> DEBUG 3: Rejected: level mismatch = 26887
>
> DEBUG 3: Rejected: quality marker = 0
>
> DEBUG 3: Rejected: message type   = 3380
>
> DEBUG 3: Rejected: masking region = 0
>
> DEBUG 3: Rejected: bad fcst value = 6
>
> DEBUG 3: Rejected: duplicates     = 13434
>
> DEBUG 2: Computing Probabilistic Statistics.
>
> DEBUG 2: Deriving normal approximation of climatological
probabilities
> for threshold >15.
>
> ERROR  :
>
> ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
> -9999
>
> ERROR  :
>
>
>
> To find out what was going on I turned off the PSTD output, reran,
and
> got an output .stat file (attached).  Looking at the output file,
the
> climo data is missing for some grid points.  As another test, I took
> this file and ran it through stat_anal and got the same error.  So
> then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and I
> got an output file.
> My stat anal job is below:
>
>
>
> /h/MET/bin/stat_analysis -lookin
>
>
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
> s_none
> _000000L_20190921_000000V.stat -out
> /h/data/global/WXQC/data/met/summary/test -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
> -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
> -out_obs_thresh
> ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> FCST_LEAD -by fcst_init_end -dump_row
> /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
> CLIMO_MEAN
> !=-9999
>
>
>
> So, the question is how can I get point stat not to error when there
> is missing climo data and trying to calculate PSTD?  My met config
> file is attached.
>
>
>
> Thanks
>
> Bob
>
>
>



------------------------------------------------
Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Mon Sep 30 10:30:01 2019

Bob,

OK, I agree.  I just wanted to confirm that that's what you expected.
Seems like it'd also be convenient for Point-Stat to include those
counts
in the list of why obs were skipped.  For example:

DEBUG 3: Rejected: bad fcst value = 6
DEBUG 3: Rejected: bad climo value = 15

This will require some code changes to add these checks to Point-Stat,
Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which currently
make
use of climatology data.  Is this needed as a bugfix for met-8.1?  Or
is
adding it in met-9.0 sufficient?

Thanks,
John

On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>
> John, If the climo data has missing values, then those model/ob
pairs
> should be excluded from the statistic calculation.  This would be
similar
> behavior to the column_thresh capability in stat_analysis.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, September 30, 2019 10:30 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
Point
> Stat
>
> Bob,
>
> That's a great question.  I do vaguely recall thinking about this
detail
> during development - how should we handle cases where the climo mean
> contains bad data.  And I assumed that if a climo field was
specified, it
> would always contain valid data values over the same grid points
that the
> forecast does.  It appears this was a bad assumption.  And in
particular
> this is more likely to occur when climo fields are automatically
regridded
> to the user's model domain - where bad data values could be
introduced,
> particularly along the edge of the domain.
>
> So the question is to you, how *should* this case be handled?
Assuming
> the user has specified climo mean and/or standard deviation data, if
we
> encounter bad climo values, should we exclude that matched pair from
the
> verification?
>
> Thanks,
> John
>
> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> >        Queue: met_help
> >      Subject: Bad Value in Point Stat
> >        Owner: Nobody
> >   Requestors: robert.craig.2 at us.af.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
> > >
> >
> >
> > John, I am verifying ensemble data using point stat and have the
> > outputs set to be PSTD and MPR.  I am also you NCEP climo for the
> > climo field.  When I run this, I get the following error message.
> >
> >
> >
> > DEBUG 2:
> >
> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
7858
> pairs.
> >
> > DEBUG 3: Number of matched pairs  = 7858
> >
> > DEBUG 3: Observations processed   = 665923
> >
> > DEBUG 3: Rejected: SID exclusion  = 0
> >
> > DEBUG 3: Rejected: obs type       = 596676
> >
> > DEBUG 3: Rejected: valid time     = 0
> >
> > DEBUG 3: Rejected: bad obs value  = 0
> >
> > DEBUG 3: Rejected: off the grid   = 17682
> >
> > DEBUG 3: Rejected: topography     = 0
> >
> > DEBUG 3: Rejected: level mismatch = 26887
> >
> > DEBUG 3: Rejected: quality marker = 0
> >
> > DEBUG 3: Rejected: message type   = 3380
> >
> > DEBUG 3: Rejected: masking region = 0
> >
> > DEBUG 3: Rejected: bad fcst value = 6
> >
> > DEBUG 3: Rejected: duplicates     = 13434
> >
> > DEBUG 2: Computing Probabilistic Statistics.
> >
> > DEBUG 2: Deriving normal approximation of climatological
probabilities
> > for threshold >15.
> >
> > ERROR  :
> >
> > ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999
> >
> > ERROR  :
> >
> >
> >
> > To find out what was going on I turned off the PSTD output, reran,
and
> > got an output .stat file (attached).  Looking at the output file,
the
> > climo data is missing for some grid points.  As another test, I
took
> > this file and ran it through stat_anal and got the same error.  So
> > then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and
I
> > got an output file.
> > My stat anal job is below:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
> > s_none
> > _000000L_20190921_000000V.stat -out
> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
> > -out_obs_thresh
> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> > FCST_LEAD -by fcst_init_end -dump_row
> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
> > CLIMO_MEAN
> > !=-9999
> >
> >
> >
> > So, the question is how can I get point stat not to error when
there
> > is missing climo data and trying to calculate PSTD?  My met config
> > file is attached.
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Wed Oct 30 16:06:29 2019

Bob,

I never heard back from you about whether this fix is needed in a met-
8.1
bugfix release or whether met-9.0 would suffice?  I'd prefer the
latter
since I already fixed for met-9.0 while working on development for
issues
number 1138 (https://github.com/NCAR/MET/issues/1138).

I'd like to get the met-8.1.2 bugfix release out the door, but wanted
to
checkin with you first.

Thanks,
John

On Mon, Sep 30, 2019 at 10:29 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Bob,
>
> OK, I agree.  I just wanted to confirm that that's what you
expected.
> Seems like it'd also be convenient for Point-Stat to include those
counts
> in the list of why obs were skipped.  For example:
>
> DEBUG 3: Rejected: bad fcst value = 6
> DEBUG 3: Rejected: bad climo value = 15
>
> This will require some code changes to add these checks to Point-
Stat,
> Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which
currently make
> use of climatology data.  Is this needed as a bugfix for met-8.1?
Or is
> adding it in met-9.0 sufficient?
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>>
>> John, If the climo data has missing values, then those model/ob
pairs
>> should be excluded from the statistic calculation.  This would be
similar
>> behavior to the column_thresh capability in stat_analysis.
>>
>> Thanks
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Monday, September 30, 2019 10:30 AM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
>> Point Stat
>>
>> Bob,
>>
>> That's a great question.  I do vaguely recall thinking about this
detail
>> during development - how should we handle cases where the climo
mean
>> contains bad data.  And I assumed that if a climo field was
specified, it
>> would always contain valid data values over the same grid points
that the
>> forecast does.  It appears this was a bad assumption.  And in
particular
>> this is more likely to occur when climo fields are automatically
regridded
>> to the user's model domain - where bad data values could be
introduced,
>> particularly along the edge of the domain.
>>
>> So the question is to you, how *should* this case be handled?
Assuming
>> the user has specified climo mean and/or standard deviation data,
if we
>> encounter bad climo values, should we exclude that matched pair
from the
>> verification?
>>
>> Thanks,
>> John
>>
>> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
>> > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> >        Queue: met_help
>> >      Subject: Bad Value in Point Stat
>> >        Owner: Nobody
>> >   Requestors: robert.craig.2 at us.af.mil
>> >       Status: new
>> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
>> > >
>> >
>> >
>> > John, I am verifying ensemble data using point stat and have the
>> > outputs set to be PSTD and MPR.  I am also you NCEP climo for the
>> > climo field.  When I run this, I get the following error message.
>> >
>> >
>> >
>> > DEBUG 2:
>> >
>> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
>> > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 7858
>> pairs.
>> >
>> > DEBUG 3: Number of matched pairs  = 7858
>> >
>> > DEBUG 3: Observations processed   = 665923
>> >
>> > DEBUG 3: Rejected: SID exclusion  = 0
>> >
>> > DEBUG 3: Rejected: obs type       = 596676
>> >
>> > DEBUG 3: Rejected: valid time     = 0
>> >
>> > DEBUG 3: Rejected: bad obs value  = 0
>> >
>> > DEBUG 3: Rejected: off the grid   = 17682
>> >
>> > DEBUG 3: Rejected: topography     = 0
>> >
>> > DEBUG 3: Rejected: level mismatch = 26887
>> >
>> > DEBUG 3: Rejected: quality marker = 0
>> >
>> > DEBUG 3: Rejected: message type   = 3380
>> >
>> > DEBUG 3: Rejected: masking region = 0
>> >
>> > DEBUG 3: Rejected: bad fcst value = 6
>> >
>> > DEBUG 3: Rejected: duplicates     = 13434
>> >
>> > DEBUG 2: Computing Probabilistic Statistics.
>> >
>> > DEBUG 2: Deriving normal approximation of climatological
probabilities
>> > for threshold >15.
>> >
>> > ERROR  :
>> >
>> > ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
>> > -9999
>> >
>> > ERROR  :
>> >
>> >
>> >
>> > To find out what was going on I turned off the PSTD output,
reran, and
>> > got an output .stat file (attached).  Looking at the output file,
the
>> > climo data is missing for some grid points.  As another test, I
took
>> > this file and ran it through stat_anal and got the same error.
So
>> > then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and
I
>> > got an output file.
>> > My stat anal job is below:
>> >
>> >
>> >
>> > /h/MET/bin/stat_analysis -lookin
>> >
>> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
>> > s_none
>> > _000000L_20190921_000000V.stat -out
>> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
>> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
>> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
>> > -out_obs_thresh
>> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end
20190921_180000 -by
>> > FCST_LEAD -by fcst_init_end -dump_row
>> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
>> > CLIMO_MEAN
>> > !=-9999
>> >
>> >
>> >
>> > So, the question is how can I get point stat not to error when
there
>> > is missing climo data and trying to calculate PSTD?  My met
config
>> > file is attached.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Bob
>> >
>> >
>> >
>>
>>
>>
>>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in Point Stat
From: robert.craig.2 at us.af.mil
Time: Thu Oct 31 06:50:58 2019

John, sorry I thought I had replied.  We can wait for met 9.0 for
this.

Bob

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, October 30, 2019 5:06 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value
in Point Stat

Bob,

I never heard back from you about whether this fix is needed in a met-
8.1 bugfix release or whether met-9.0 would suffice?  I'd prefer the
latter since I already fixed for met-9.0 while working on development
for issues number 1138 (https://github.com/NCAR/MET/issues/1138).

I'd like to get the met-8.1.2 bugfix release out the door, but wanted
to checkin with you first.

Thanks,
John

On Mon, Sep 30, 2019 at 10:29 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Bob,
>
> OK, I agree.  I just wanted to confirm that that's what you
expected.
> Seems like it'd also be convenient for Point-Stat to include those
> counts in the list of why obs were skipped.  For example:
>
> DEBUG 3: Rejected: bad fcst value = 6
> DEBUG 3: Rejected: bad climo value = 15
>
> This will require some code changes to add these checks to Point-
Stat,
> Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which
currently
> make use of climatology data.  Is this needed as a bugfix for met-
8.1?
> Or is adding it in met-9.0 sufficient?
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>>
>> John, If the climo data has missing values, then those model/ob
pairs
>> should be excluded from the statistic calculation.  This would be
>> similar behavior to the column_thresh capability in stat_analysis.
>>
>> Thanks
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Monday, September 30, 2019 10:30 AM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> <robert.craig.2 at us.af.mil>
>> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
>> Point Stat
>>
>> Bob,
>>
>> That's a great question.  I do vaguely recall thinking about this
>> detail during development - how should we handle cases where the
>> climo mean contains bad data.  And I assumed that if a climo field
>> was specified, it would always contain valid data values over the
>> same grid points that the forecast does.  It appears this was a bad
>> assumption.  And in particular this is more likely to occur when
>> climo fields are automatically regridded to the user's model domain
-
>> where bad data values could be introduced, particularly along the
edge of the domain.
>>
>> So the question is to you, how *should* this case be handled?
>> Assuming the user has specified climo mean and/or standard
deviation
>> data, if we encounter bad climo values, should we exclude that
>> matched pair from the verification?
>>
>> Thanks,
>> John
>>
>> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
>> > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> >        Queue: met_help
>> >      Subject: Bad Value in Point Stat
>> >        Owner: Nobody
>> >   Requestors: robert.craig.2 at us.af.mil
>> >       Status: new
>> >  Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
>> > >
>> >
>> >
>> > John, I am verifying ensemble data using point stat and have the
>> > outputs set to be PSTD and MPR.  I am also you NCEP climo for the
>> > climo field.  When I run this, I get the following error message.
>> >
>> >
>> >
>> > DEBUG 2:
>> >
>> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
>> > ADPSFC, over region FULL, for interpolation method BILIN(4),
using
>> > 7858
>> pairs.
>> >
>> > DEBUG 3: Number of matched pairs  = 7858
>> >
>> > DEBUG 3: Observations processed   = 665923
>> >
>> > DEBUG 3: Rejected: SID exclusion  = 0
>> >
>> > DEBUG 3: Rejected: obs type       = 596676
>> >
>> > DEBUG 3: Rejected: valid time     = 0
>> >
>> > DEBUG 3: Rejected: bad obs value  = 0
>> >
>> > DEBUG 3: Rejected: off the grid   = 17682
>> >
>> > DEBUG 3: Rejected: topography     = 0
>> >
>> > DEBUG 3: Rejected: level mismatch = 26887
>> >
>> > DEBUG 3: Rejected: quality marker = 0
>> >
>> > DEBUG 3: Rejected: message type   = 3380
>> >
>> > DEBUG 3: Rejected: masking region = 0
>> >
>> > DEBUG 3: Rejected: bad fcst value = 6
>> >
>> > DEBUG 3: Rejected: duplicates     = 13434
>> >
>> > DEBUG 2: Computing Probabilistic Statistics.
>> >
>> > DEBUG 2: Deriving normal approximation of climatological
>> > probabilities for threshold >15.
>> >
>> > ERROR  :
>> >
>> > ERROR  : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
>> > -9999
>> >
>> > ERROR  :
>> >
>> >
>> >
>> > To find out what was going on I turned off the PSTD output,
reran,
>> > and got an output .stat file (attached).  Looking at the output
>> > file, the climo data is missing for some grid points.  As another
>> > test, I took this file and ran it through stat_anal and got the
>> > same error.  So then I added "-column_thresh CLIMO_MEAN !=-9999"
to
>> > stat anal and I got an output file.
>> > My stat anal job is below:
>> >
>> >
>> >
>> > /h/MET/bin/stat_analysis -lookin
>> >
>> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_
>> > mep
>> > s_none
>> > _000000L_20190921_000000V.stat -out
>> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
>> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
>> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
>> > -out_obs_thresh
>> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end
20190921_180000
>> > -by FCST_LEAD -by fcst_init_end -dump_row
>> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
>> > CLIMO_MEAN
>> > !=-9999
>> >
>> >
>> >
>> > So, the question is how can I get point stat not to error when
>> > there is missing climo data and trying to calculate PSTD?  My met
>> > config file is attached.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Bob
>> >
>> >
>> >
>>
>>
>>
>>



------------------------------------------------


More information about the Met_help mailing list