[Met_help] [rt.rap.ucar.edu #92376] History for Bad Value in Point Stat
John Halley Gotway via RT
met_help at ucar.edu
Thu Oct 31 09:25:42 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I am verifying ensemble data using point stat and have the outputs set
to be PSTD and MPR. I am also you NCEP climo for the climo field. When I
run this, I get the following error message.
DEBUG 2:
DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type ADPSFC,
over region FULL, for interpolation method BILIN(4), using 7858 pairs.
DEBUG 3: Number of matched pairs = 7858
DEBUG 3: Observations processed = 665923
DEBUG 3: Rejected: SID exclusion = 0
DEBUG 3: Rejected: obs type = 596676
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 17682
DEBUG 3: Rejected: topography = 0
DEBUG 3: Rejected: level mismatch = 26887
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 3380
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 6
DEBUG 3: Rejected: duplicates = 13434
DEBUG 2: Computing Probabilistic Statistics.
DEBUG 2: Deriving normal approximation of climatological probabilities for
threshold >15.
ERROR :
ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ... -9999
ERROR :
To find out what was going on I turned off the PSTD output, reran, and got
an output .stat file (attached). Looking at the output file, the climo data
is missing for some grid points. As another test, I took this file and ran
it through stat_anal and got the same error. So then I added
"-column_thresh CLIMO_MEAN !=-9999" to stat anal and I got an output file.
My stat anal job is below:
/h/MET/bin/stat_analysis -lookin
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_meps_none
_000000L_20190921_000000V.stat -out
/h/data/global/WXQC/data/met/summary/test -tmp_dir
/h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
-fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1 -out_obs_thresh
ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000 -by
FCST_LEAD -by fcst_init_end -dump_row
/h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh CLIMO_MEAN
!=-9999
So, the question is how can I get point stat not to error when there is
missing climo data and trying to calculate PSTD? My met config file is
attached.
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Mon Sep 30 09:30:03 2019
Bob,
That's a great question. I do vaguely recall thinking about this
detail
during development - how should we handle cases where the climo mean
contains bad data. And I assumed that if a climo field was specified,
it
would always contain valid data values over the same grid points that
the
forecast does. It appears this was a bad assumption. And in
particular
this is more likely to occur when climo fields are automatically
regridded
to the user's model domain - where bad data values could be
introduced,
particularly along the edge of the domain.
So the question is to you, how *should* this case be handled?
Assuming the
user has specified climo mean and/or standard deviation data, if we
encounter bad climo values, should we exclude that matched pair from
the
verification?
Thanks,
John
On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Bad Value in Point Stat
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>
>
> John, I am verifying ensemble data using point stat and have the
outputs
> set
> to be PSTD and MPR. I am also you NCEP climo for the climo field.
When I
> run this, I get the following error message.
>
>
>
> DEBUG 2:
>
> DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type
ADPSFC,
> over region FULL, for interpolation method BILIN(4), using 7858
pairs.
>
> DEBUG 3: Number of matched pairs = 7858
>
> DEBUG 3: Observations processed = 665923
>
> DEBUG 3: Rejected: SID exclusion = 0
>
> DEBUG 3: Rejected: obs type = 596676
>
> DEBUG 3: Rejected: valid time = 0
>
> DEBUG 3: Rejected: bad obs value = 0
>
> DEBUG 3: Rejected: off the grid = 17682
>
> DEBUG 3: Rejected: topography = 0
>
> DEBUG 3: Rejected: level mismatch = 26887
>
> DEBUG 3: Rejected: quality marker = 0
>
> DEBUG 3: Rejected: message type = 3380
>
> DEBUG 3: Rejected: masking region = 0
>
> DEBUG 3: Rejected: bad fcst value = 6
>
> DEBUG 3: Rejected: duplicates = 13434
>
> DEBUG 2: Computing Probabilistic Statistics.
>
> DEBUG 2: Deriving normal approximation of climatological
probabilities for
> threshold >15.
>
> ERROR :
>
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
-9999
>
> ERROR :
>
>
>
> To find out what was going on I turned off the PSTD output, reran,
and got
> an output .stat file (attached). Looking at the output file, the
climo
> data
> is missing for some grid points. As another test, I took this file
and ran
> it through stat_anal and got the same error. So then I added
> "-column_thresh CLIMO_MEAN !=-9999" to stat anal and I got an output
file.
> My stat anal job is below:
>
>
>
> /h/MET/bin/stat_analysis -lookin
>
>
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_meps_none
> _000000L_20190921_000000V.stat -out
> /h/data/global/WXQC/data/met/summary/test -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
> -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
-out_obs_thresh
> ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> FCST_LEAD -by fcst_init_end -dump_row
> /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
CLIMO_MEAN
> !=-9999
>
>
>
> So, the question is how can I get point stat not to error when there
is
> missing climo data and trying to calculate PSTD? My met config file
is
> attached.
>
>
>
> Thanks
>
> Bob
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in Point Stat
From: robert.craig.2 at us.af.mil
Time: Mon Sep 30 09:49:05 2019
John, If the climo data has missing values, then those model/ob pairs
should be excluded from the statistic calculation. This would be
similar behavior to the column_thresh capability in stat_analysis.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, September 30, 2019 10:30 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
Point Stat
Bob,
That's a great question. I do vaguely recall thinking about this
detail during development - how should we handle cases where the climo
mean contains bad data. And I assumed that if a climo field was
specified, it would always contain valid data values over the same
grid points that the forecast does. It appears this was a bad
assumption. And in particular this is more likely to occur when climo
fields are automatically regridded to the user's model domain - where
bad data values could be introduced, particularly along the edge of
the domain.
So the question is to you, how *should* this case be handled?
Assuming the user has specified climo mean and/or standard deviation
data, if we encounter bad climo values, should we exclude that matched
pair from the verification?
Thanks,
John
On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Bad Value in Point Stat
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
> >
>
>
> John, I am verifying ensemble data using point stat and have the
> outputs set to be PSTD and MPR. I am also you NCEP climo for the
> climo field. When I run this, I get the following error message.
>
>
>
> DEBUG 2:
>
> DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using
7858 pairs.
>
> DEBUG 3: Number of matched pairs = 7858
>
> DEBUG 3: Observations processed = 665923
>
> DEBUG 3: Rejected: SID exclusion = 0
>
> DEBUG 3: Rejected: obs type = 596676
>
> DEBUG 3: Rejected: valid time = 0
>
> DEBUG 3: Rejected: bad obs value = 0
>
> DEBUG 3: Rejected: off the grid = 17682
>
> DEBUG 3: Rejected: topography = 0
>
> DEBUG 3: Rejected: level mismatch = 26887
>
> DEBUG 3: Rejected: quality marker = 0
>
> DEBUG 3: Rejected: message type = 3380
>
> DEBUG 3: Rejected: masking region = 0
>
> DEBUG 3: Rejected: bad fcst value = 6
>
> DEBUG 3: Rejected: duplicates = 13434
>
> DEBUG 2: Computing Probabilistic Statistics.
>
> DEBUG 2: Deriving normal approximation of climatological
probabilities
> for threshold >15.
>
> ERROR :
>
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
> -9999
>
> ERROR :
>
>
>
> To find out what was going on I turned off the PSTD output, reran,
and
> got an output .stat file (attached). Looking at the output file,
the
> climo data is missing for some grid points. As another test, I took
> this file and ran it through stat_anal and got the same error. So
> then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and I
> got an output file.
> My stat anal job is below:
>
>
>
> /h/MET/bin/stat_analysis -lookin
>
>
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
> s_none
> _000000L_20190921_000000V.stat -out
> /h/data/global/WXQC/data/met/summary/test -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type MPR
> -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
> -out_obs_thresh
> ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> FCST_LEAD -by fcst_init_end -dump_row
> /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
> CLIMO_MEAN
> !=-9999
>
>
>
> So, the question is how can I get point stat not to error when there
> is missing climo data and trying to calculate PSTD? My met config
> file is attached.
>
>
>
> Thanks
>
> Bob
>
>
>
------------------------------------------------
Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Mon Sep 30 10:30:01 2019
Bob,
OK, I agree. I just wanted to confirm that that's what you expected.
Seems like it'd also be convenient for Point-Stat to include those
counts
in the list of why obs were skipped. For example:
DEBUG 3: Rejected: bad fcst value = 6
DEBUG 3: Rejected: bad climo value = 15
This will require some code changes to add these checks to Point-Stat,
Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which currently
make
use of climatology data. Is this needed as a bugfix for met-8.1? Or
is
adding it in met-9.0 sufficient?
Thanks,
John
On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>
> John, If the climo data has missing values, then those model/ob
pairs
> should be excluded from the statistic calculation. This would be
similar
> behavior to the column_thresh capability in stat_analysis.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, September 30, 2019 10:30 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
Point
> Stat
>
> Bob,
>
> That's a great question. I do vaguely recall thinking about this
detail
> during development - how should we handle cases where the climo mean
> contains bad data. And I assumed that if a climo field was
specified, it
> would always contain valid data values over the same grid points
that the
> forecast does. It appears this was a bad assumption. And in
particular
> this is more likely to occur when climo fields are automatically
regridded
> to the user's model domain - where bad data values could be
introduced,
> particularly along the edge of the domain.
>
> So the question is to you, how *should* this case be handled?
Assuming
> the user has specified climo mean and/or standard deviation data, if
we
> encounter bad climo values, should we exclude that matched pair from
the
> verification?
>
> Thanks,
> John
>
> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Bad Value in Point Stat
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
> > >
> >
> >
> > John, I am verifying ensemble data using point stat and have the
> > outputs set to be PSTD and MPR. I am also you NCEP climo for the
> > climo field. When I run this, I get the following error message.
> >
> >
> >
> > DEBUG 2:
> >
> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
7858
> pairs.
> >
> > DEBUG 3: Number of matched pairs = 7858
> >
> > DEBUG 3: Observations processed = 665923
> >
> > DEBUG 3: Rejected: SID exclusion = 0
> >
> > DEBUG 3: Rejected: obs type = 596676
> >
> > DEBUG 3: Rejected: valid time = 0
> >
> > DEBUG 3: Rejected: bad obs value = 0
> >
> > DEBUG 3: Rejected: off the grid = 17682
> >
> > DEBUG 3: Rejected: topography = 0
> >
> > DEBUG 3: Rejected: level mismatch = 26887
> >
> > DEBUG 3: Rejected: quality marker = 0
> >
> > DEBUG 3: Rejected: message type = 3380
> >
> > DEBUG 3: Rejected: masking region = 0
> >
> > DEBUG 3: Rejected: bad fcst value = 6
> >
> > DEBUG 3: Rejected: duplicates = 13434
> >
> > DEBUG 2: Computing Probabilistic Statistics.
> >
> > DEBUG 2: Deriving normal approximation of climatological
probabilities
> > for threshold >15.
> >
> > ERROR :
> >
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999
> >
> > ERROR :
> >
> >
> >
> > To find out what was going on I turned off the PSTD output, reran,
and
> > got an output .stat file (attached). Looking at the output file,
the
> > climo data is missing for some grid points. As another test, I
took
> > this file and ran it through stat_anal and got the same error. So
> > then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and
I
> > got an output file.
> > My stat anal job is below:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
> > s_none
> > _000000L_20190921_000000V.stat -out
> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
> > -out_obs_thresh
> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end 20190921_180000
-by
> > FCST_LEAD -by fcst_init_end -dump_row
> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
> > CLIMO_MEAN
> > !=-9999
> >
> >
> >
> > So, the question is how can I get point stat not to error when
there
> > is missing climo data and trying to calculate PSTD? My met config
> > file is attached.
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Bad Value in Point Stat
From: John Halley Gotway
Time: Wed Oct 30 16:06:29 2019
Bob,
I never heard back from you about whether this fix is needed in a met-
8.1
bugfix release or whether met-9.0 would suffice? I'd prefer the
latter
since I already fixed for met-9.0 while working on development for
issues
number 1138 (https://github.com/NCAR/MET/issues/1138).
I'd like to get the met-8.1.2 bugfix release out the door, but wanted
to
checkin with you first.
Thanks,
John
On Mon, Sep 30, 2019 at 10:29 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Bob,
>
> OK, I agree. I just wanted to confirm that that's what you
expected.
> Seems like it'd also be convenient for Point-Stat to include those
counts
> in the list of why obs were skipped. For example:
>
> DEBUG 3: Rejected: bad fcst value = 6
> DEBUG 3: Rejected: bad climo value = 15
>
> This will require some code changes to add these checks to Point-
Stat,
> Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which
currently make
> use of climatology data. Is this needed as a bugfix for met-8.1?
Or is
> adding it in met-9.0 sufficient?
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>>
>> John, If the climo data has missing values, then those model/ob
pairs
>> should be excluded from the statistic calculation. This would be
similar
>> behavior to the column_thresh capability in stat_analysis.
>>
>> Thanks
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Monday, September 30, 2019 10:30 AM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
>> Point Stat
>>
>> Bob,
>>
>> That's a great question. I do vaguely recall thinking about this
detail
>> during development - how should we handle cases where the climo
mean
>> contains bad data. And I assumed that if a climo field was
specified, it
>> would always contain valid data values over the same grid points
that the
>> forecast does. It appears this was a bad assumption. And in
particular
>> this is more likely to occur when climo fields are automatically
regridded
>> to the user's model domain - where bad data values could be
introduced,
>> particularly along the edge of the domain.
>>
>> So the question is to you, how *should* this case be handled?
Assuming
>> the user has specified climo mean and/or standard deviation data,
if we
>> encounter bad climo values, should we exclude that matched pair
from the
>> verification?
>>
>> Thanks,
>> John
>>
>> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
>> > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> > Queue: met_help
>> > Subject: Bad Value in Point Stat
>> > Owner: Nobody
>> > Requestors: robert.craig.2 at us.af.mil
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
>> > >
>> >
>> >
>> > John, I am verifying ensemble data using point stat and have the
>> > outputs set to be PSTD and MPR. I am also you NCEP climo for the
>> > climo field. When I run this, I get the following error message.
>> >
>> >
>> >
>> > DEBUG 2:
>> >
>> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
>> > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 7858
>> pairs.
>> >
>> > DEBUG 3: Number of matched pairs = 7858
>> >
>> > DEBUG 3: Observations processed = 665923
>> >
>> > DEBUG 3: Rejected: SID exclusion = 0
>> >
>> > DEBUG 3: Rejected: obs type = 596676
>> >
>> > DEBUG 3: Rejected: valid time = 0
>> >
>> > DEBUG 3: Rejected: bad obs value = 0
>> >
>> > DEBUG 3: Rejected: off the grid = 17682
>> >
>> > DEBUG 3: Rejected: topography = 0
>> >
>> > DEBUG 3: Rejected: level mismatch = 26887
>> >
>> > DEBUG 3: Rejected: quality marker = 0
>> >
>> > DEBUG 3: Rejected: message type = 3380
>> >
>> > DEBUG 3: Rejected: masking region = 0
>> >
>> > DEBUG 3: Rejected: bad fcst value = 6
>> >
>> > DEBUG 3: Rejected: duplicates = 13434
>> >
>> > DEBUG 2: Computing Probabilistic Statistics.
>> >
>> > DEBUG 2: Deriving normal approximation of climatological
probabilities
>> > for threshold >15.
>> >
>> > ERROR :
>> >
>> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
>> > -9999
>> >
>> > ERROR :
>> >
>> >
>> >
>> > To find out what was going on I turned off the PSTD output,
reran, and
>> > got an output .stat file (attached). Looking at the output file,
the
>> > climo data is missing for some grid points. As another test, I
took
>> > this file and ran it through stat_anal and got the same error.
So
>> > then I added "-column_thresh CLIMO_MEAN !=-9999" to stat anal and
I
>> > got an output file.
>> > My stat anal job is below:
>> >
>> >
>> >
>> > /h/MET/bin/stat_analysis -lookin
>> >
>> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_mep
>> > s_none
>> > _000000L_20190921_000000V.stat -out
>> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
>> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
>> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
>> > -out_obs_thresh
>> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end
20190921_180000 -by
>> > FCST_LEAD -by fcst_init_end -dump_row
>> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
>> > CLIMO_MEAN
>> > !=-9999
>> >
>> >
>> >
>> > So, the question is how can I get point stat not to error when
there
>> > is missing climo data and trying to calculate PSTD? My met
config
>> > file is attached.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Bob
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in Point Stat
From: robert.craig.2 at us.af.mil
Time: Thu Oct 31 06:50:58 2019
John, sorry I thought I had replied. We can wait for met 9.0 for
this.
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, October 30, 2019 5:06 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value
in Point Stat
Bob,
I never heard back from you about whether this fix is needed in a met-
8.1 bugfix release or whether met-9.0 would suffice? I'd prefer the
latter since I already fixed for met-9.0 while working on development
for issues number 1138 (https://github.com/NCAR/MET/issues/1138).
I'd like to get the met-8.1.2 bugfix release out the door, but wanted
to checkin with you first.
Thanks,
John
On Mon, Sep 30, 2019 at 10:29 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Bob,
>
> OK, I agree. I just wanted to confirm that that's what you
expected.
> Seems like it'd also be convenient for Point-Stat to include those
> counts in the list of why obs were skipped. For example:
>
> DEBUG 3: Rejected: bad fcst value = 6
> DEBUG 3: Rejected: bad climo value = 15
>
> This will require some code changes to add these checks to Point-
Stat,
> Grid-Stat, Ensemble-Stat, and Series-Analysis, all of which
currently
> make use of climatology data. Is this needed as a bugfix for met-
8.1?
> Or is adding it in met-9.0 sufficient?
>
> Thanks,
> John
>
> On Mon, Sep 30, 2019 at 9:49 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376 >
>>
>> John, If the climo data has missing values, then those model/ob
pairs
>> should be excluded from the statistic calculation. This would be
>> similar behavior to the column_thresh capability in stat_analysis.
>>
>> Thanks
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Monday, September 30, 2019 10:30 AM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> <robert.craig.2 at us.af.mil>
>> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #92376] Bad Value in
>> Point Stat
>>
>> Bob,
>>
>> That's a great question. I do vaguely recall thinking about this
>> detail during development - how should we handle cases where the
>> climo mean contains bad data. And I assumed that if a climo field
>> was specified, it would always contain valid data values over the
>> same grid points that the forecast does. It appears this was a bad
>> assumption. And in particular this is more likely to occur when
>> climo fields are automatically regridded to the user's model domain
-
>> where bad data values could be introduced, particularly along the
edge of the domain.
>>
>> So the question is to you, how *should* this case be handled?
>> Assuming the user has specified climo mean and/or standard
deviation
>> data, if we encounter bad climo values, should we exclude that
>> matched pair from the verification?
>>
>> Thanks,
>> John
>>
>> On Fri, Sep 27, 2019 at 3:13 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Fri Sep 27 15:13:21 2019: Request 92376 was acted upon.
>> > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> > Queue: met_help
>> > Subject: Bad Value in Point Stat
>> > Owner: Nobody
>> > Requestors: robert.craig.2 at us.af.mil
>> > Status: new
>> > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92376
>> > >
>> >
>> >
>> > John, I am verifying ensemble data using point stat and have the
>> > outputs set to be PSTD and MPR. I am also you NCEP climo for the
>> > climo field. When I run this, I get the following error message.
>> >
>> >
>> >
>> > DEBUG 2:
>> >
>> > DEBUG 2: Processing PWS15/Z10 versus WIND/Z10, for observation
type
>> > ADPSFC, over region FULL, for interpolation method BILIN(4),
using
>> > 7858
>> pairs.
>> >
>> > DEBUG 3: Number of matched pairs = 7858
>> >
>> > DEBUG 3: Observations processed = 665923
>> >
>> > DEBUG 3: Rejected: SID exclusion = 0
>> >
>> > DEBUG 3: Rejected: obs type = 596676
>> >
>> > DEBUG 3: Rejected: valid time = 0
>> >
>> > DEBUG 3: Rejected: bad obs value = 0
>> >
>> > DEBUG 3: Rejected: off the grid = 17682
>> >
>> > DEBUG 3: Rejected: topography = 0
>> >
>> > DEBUG 3: Rejected: level mismatch = 26887
>> >
>> > DEBUG 3: Rejected: quality marker = 0
>> >
>> > DEBUG 3: Rejected: message type = 3380
>> >
>> > DEBUG 3: Rejected: masking region = 0
>> >
>> > DEBUG 3: Rejected: bad fcst value = 6
>> >
>> > DEBUG 3: Rejected: duplicates = 13434
>> >
>> > DEBUG 2: Computing Probabilistic Statistics.
>> >
>> > DEBUG 2: Deriving normal approximation of climatological
>> > probabilities for threshold >15.
>> >
>> > ERROR :
>> >
>> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
>> > -9999
>> >
>> > ERROR :
>> >
>> >
>> >
>> > To find out what was going on I turned off the PSTD output,
reran,
>> > and got an output .stat file (attached). Looking at the output
>> > file, the climo data is missing for some grid points. As another
>> > test, I took this file and ran it through stat_anal and got the
>> > same error. So then I added "-column_thresh CLIMO_MEAN !=-9999"
to
>> > stat anal and I got an output file.
>> > My stat anal job is below:
>> >
>> >
>> >
>> > /h/MET/bin/stat_analysis -lookin
>> >
>> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/meps/WS/point_stat_
>> > mep
>> > s_none
>> > _000000L_20190921_000000V.stat -out
>> > /h/data/global/WXQC/data/met/summary/test -tmp_dir
>> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
MPR
>> > -fcst_var PWS15 -out_line_type PSTD -out_fcst_thresh ==0.1
>> > -out_obs_thresh
>> > ge15 -fcst_init_beg 20190921_000000 -fcst_init_end
20190921_180000
>> > -by FCST_LEAD -by fcst_init_end -dump_row
>> > /h/data/global/WXQC/data/met/filter_job_wd -v 4.0 -column_thresh
>> > CLIMO_MEAN
>> > !=-9999
>> >
>> >
>> >
>> > So, the question is how can I get point stat not to error when
>> > there is missing climo data and trying to calculate PSTD? My met
>> > config file is attached.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Bob
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list