[Met_help] [rt.rap.ucar.edu #75719] History for Missing values at times using point_stat
John Halley Gotway via RT
met_help at ucar.edu
Mon Apr 11 16:47:05 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear Met Help,
I'm getting some NA or scores of 0 for some fields, and I would like to make sure I'm doing things right using point_stat. Maybe I can send you an example of an observational, prepbufr and configure files to make sure we get the same scores?
Thanks for the help,
Jose
This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Thu Mar 31 12:00:55 2016
Jose,
Sure, that'd be fine. You can post some sample data to our anonymous
ftp
site following these instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Be sure to send us your forecast file, pb2nc output file, and point-
stat
configuration file. I can run the case here and send you the results.
Some of the output statistics may be NA. NA's show up when it is not
numerically possible to compute the statistic or when the user has not
requested the output (i.e. requesting no bootstrapping results in NA
for
the output bootstrap confidence interval columns).
If you do post data, please reply to this email and I'll go grab it.
Thanks,
John Halley Gotway
On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:
>
> Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> Transaction: Ticket created by GarciaJ at imsg.com
> Queue: met_help
> Subject: Missing values at times using point_stat
> Owner: Nobody
> Requestors: GarciaJ at imsg.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
>
> Dear Met Help,
>
> I'm getting some NA or scores of 0 for some fields, and I would like
to
> make sure I'm doing things right using point_stat. Maybe I can send
you an
> example of an observational, prepbufr and configure files to make
sure we
> get the same scores?
>
> Thanks for the help,
> Jose
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity to
> which it is addressed. If the reader of this e-mail is not the
intended
> recipient or his or her authorized agent, the reader is hereby
notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify
the
> sender by replying to this message and delete this e-mail
immediately.
>
>
------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016
Dear John,
Thanks for the help. The files are in 'garcia_data'.
I also attached my own results here, so you can compare them. I'm
focusing on:
GSS
CSI
FBIAS
RMSE
If you have any advice on thresholds as well (commonly used and/or
recommended, etc) that will be greatly appreciated.
Thanks for the help,
Jose
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 31, 2016 2:01 PM
To: Jose Garcia-Rivera
Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
point_stat
Jose,
Sure, that'd be fine. You can post some sample data to our anonymous
ftp site following these instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Be sure to send us your forecast file, pb2nc output file, and point-
stat configuration file. I can run the case here and send you the
results.
Some of the output statistics may be NA. NA's show up when it is not
numerically possible to compute the statistic or when the user has not
requested the output (i.e. requesting no bootstrapping results in NA
for the output bootstrap confidence interval columns).
If you do post data, please reply to this email and I'll go grab it.
Thanks,
John Halley Gotway
On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:
>
> Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> Transaction: Ticket created by GarciaJ at imsg.com
> Queue: met_help
> Subject: Missing values at times using point_stat
> Owner: Nobody
> Requestors: GarciaJ at imsg.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> >
>
>
> Dear Met Help,
>
> I'm getting some NA or scores of 0 for some fields, and I would like
> to make sure I'm doing things right using point_stat. Maybe I can
send
> you an example of an observational, prepbufr and configure files to
> make sure we get the same scores?
>
> Thanks for the help,
> Jose
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity
> to which it is addressed. If the reader of this e-mail is not the
> intended recipient or his or her authorized agent, the reader is
> hereby notified that any dissemination, distribution or copying of
> this e-mail is prohibited. If you have received this e-mail in
error,
> please notify the sender by replying to this message and delete this
e-mail immediately.
>
>
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the
intended recipient or his or her authorized agent, the reader is
hereby notified that any dissemination, distribution or copying of
this e-mail is prohibited. If you have received this e-mail in error,
please notify the sender by replying to this message and delete this
e-mail immediately.
------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU
FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL
FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV
OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL
PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES
ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV
ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL
MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU
BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10
E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL
E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR
EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL
ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU
MSESS MSESS_BCL MSESS_BCU
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP Z2 TMP Z2
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 939 278.91812 278.6563 279.17993 278.66298
279.19824 4.0934 3.91626 4.28744 3.95624 4.24214
280.85064 280.57828 281.12299 280.58551 281.13963 4.2581 4.07384
4.45995 4.10819 4.39729 0.90891 0.8971 0.91942
0.89695 0.91998 0.91562 0.75236 939 465 783
-1.93252 -2.04698 -1.81806 -2.04514 -1.81877 1.78957 1.71213
1.8744 1.69425 1.88183 0.99312 0.99272 0.99352 2.20418
2.11303 2.29832 6.93378 6.42241 7.48957 3.19914 2.86743
3.53752 2.63321 2.53425 2.73671 -4.07879 -4.22199 -3.89618 -3.10649
-3.2305 -2.97731 -2.02299 -2.13401 -1.88599 -0.81801 -0.96999
-0.60497 0.25201 0.045994 0.4576 2.28848 2.12896 2.52304
1.14102 1.06201 1.25307 NA NA NA
NA NA 3.73464 3.30792 4.18258 0.61758 0.57372
0.65528
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH Z2 RH Z2
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 908 72.12564 71.08387 73.16741 71.08168
73.10797 16.01649 15.31218 16.78921 15.5477 16.48678
64.07651 62.98601 65.16701 63.07106 65.25552 16.76564 16.0284
17.57452 16.12883 17.39894 0.7487 0.71865 0.77596
0.71288 0.77779 0.76427 0.56253 908 464 82
8.04913 7.29193 8.80633 7.27352 8.74397 11.6414 11.12949
12.20305 10.86934 12.48742 1.12562 1.11265 1.13788 11.71805
11.23768 12.21655 200.1615 182.28383 220.82803 135.37298 118.0125
155.76391 14.14784 13.50125 14.86028 -6.89396 -8.31 -5.15296
1.65724 0.52178 2.65279 10.07039 8.84892 10.62013 14.85492
14.16715 15.63692 20.35536 19.44352 21.13064 13.19768 12.27761
14.31857 6.28231 5.70165 7.02628 NA NA NA
NA NA 64.78852 52.9041 76.45705 0.2879 0.21828
0.35157
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND Z10 WIND Z10
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 936 4.59267 4.42301 4.76232 4.43297
4.76131 2.64818 2.53341 2.77393 2.47928 2.82708 3.63693
3.45551 3.81836 3.46415 3.81223 2.83191 2.70918 2.96638
2.71158 2.96677 0.71364 0.68069 0.74371 0.68037
0.74715 0.66281 0.48633 936 459 840 0.95573 0.82244
1.08902 0.82302 1.09018 2.08059 1.99042 2.17939 1.96103
2.21224 1.26279 1.22113 1.30828 1.7628 1.67329 1.85966
5.23767 4.73692 5.84173 4.32425 3.84154 4.88877 2.2886
2.17645 2.41697 -1.3576 -1.7195 -1.10904 -0.17321 -0.26662
-0.029232 0.97956 0.85551 1.1217 2.19437 1.96061 2.38548
3.35963 3.18851 3.65961 2.36758 2.14819 2.59269 1.17008 1.07611
1.28525 NA NA NA NA
NA 0.91342 0.67736 1.18849 0.3469 0.2635 0.42189
------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU
ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU
PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU
PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR
FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL
CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU
HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS ODDS_NCL
ODDS_NCU ODDS_BCL ODDS_BCU LODDS LODDS_NCL LODDS_NCU LODDS_BCL
LODDS_BCU ORSS ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL EDS_NCU EDS_BCL EDS_BCU SEDS SEDS_NCL SEDS_NCU
SEDS_BCL SEDS_BCU EDI EDI_NCL EDI_NCU EDI_BCL EDI_BCU SEDI
SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS BAGSS_BCL BAGSS_BCU
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP Z2 TMP Z2
ADPSFC FULL NEAREST 1 >=293 >=293 NA
0.05 CTS 939 0 0 0.0040743 0 0
0 0 0.0040743 0 0 1 0.99593 1
1 1 NA NA NA NA NA NA
NA NA 1 0.99593 1 1 1 0
0 0.0040743 0 0 NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH Z2 RH Z2
ADPSFC FULL NEAREST 1 >=50 >=50 NA
0.05 CTS 908 0.76322 0.73449 0.78972 0.73348 0.78965
0.91189 0.89168 0.92864 0.89317 0.93062 0.76322 0.73449
0.78972 0.73566 0.79075 1.19481 1.15374 1.24366 0.94228 0.92517
0.95567 0.92405 0.95971 0.18605 0.16208 0.21266 0.13402 0.24109
0.81395 0.78734 0.83792 0.75891 0.86598 0.21135 0.18604
0.2391 0.18539 0.24106 0.7523 0.7232 0.77928 0.7233 0.78129
0.089204 0.050183 0.13127 0.12833 0.091984 0.16467 0.073416
0.18554 0.1638 0.095571 0.23208 3.73143 2.33462 5.96395 2.37957
6.10007 1.31679 0.84785 1.78573 0.86692 1.8083 0.57729
0.42097 0.73362 0.40821 0.71831 0.63932 0.54769 0.73095 0.54235
0.7401 0.099429 0.037975 0.16088 0.056088 0.14586 0.55181
-0.064151 1.16778 0.39088 0.6903 0.27438 0.10648 0.99715
0.17561 0.36874 0.19182 0.15575 0.23327
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND Z10 WIND Z10
ADPSFC FULL NEAREST 1 >=8 >=8 NA
0.05 CTS 936 0.090812 0.074038 0.11093 0.071581 0.10897
0.13782 0.11721 0.16139 0.11645 0.16132 0.87821 0.85569
0.89763 0.85681 0.89957 1.51765 1.24981 1.88893 0.58824 0.55641
0.61934 0.49425 0.70216 0.90717 0.88688 0.92413 0.88756 0.92689
0.092832 0.075869 0.11312 0.073112 0.11244 0.6124 0.58079
0.6431 0.52828 0.69786 0.30488 0.27623 0.33512 0.23728 0.37586
0.2514 0.18538 0.32416 0.4954 0.38781 0.603 0.39934
0.60584 0.4018 0.31277 0.48961 13.96022 8.55164 22.78952 8.96456
23.31698 2.63621 2.14612 3.1263 2.19328 3.14918 0.86631
0.80517 0.92745 0.79929 0.91775 0.63775 0.53831 0.73718 0.53831
0.7448 0.49535 0.40456 0.58614 0.41692 0.57208 0.63501
0.50309 0.76692 0.54008 0.73849 0.67728 0.50502 0.76499
0.58193 0.77588 0.2282 0.16542 0.29512
------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016
Jose,
Thanks for sending your data. I ran it through met-5.1 point_stat and
compared my results to the results you sent me. And I've attached my
output files to this email.
The only differences I see show up in the columns ending in BCL and
BCU.
Those are the bootstrap confidence intervals Point-Stat computes. The
process of bootstrapping is inherently random, so I would fully expect
those values to change from run to run. So I see absolutely no
problem
here.
You are wondering why some of your statistics are NA. For example,
looking
in the CTS output file at the first line for 2-m temperature >= 293,
many
of the statistics are 0. Notice that the "BASE_RATE" column is 0
which
means that none of observations that were verified met that threshold
criteria. So all of the obs had 2-m temp values < 293. Many of the
contingency table statistics are based on at least 1 event occurring
in the
observation field... that's why many of those stats are NA.
To understand the math, you can look at the equations in Appendix C of
the
MET user's guide. You'll find that 0 observed events leads to a
'divide by
0' for many statistics. In the MET output, 'divide by 0' is written
as NA.
However, looking at your config file, let me make some suggestions:
(1) Unless you're actually using the bootstrap confidence intervals,
I'd
suggest disabling them by setting "n_rep = 0;" in the "boot" section.
Bootstrapping is inherently slow, so you should skip them unless you
have a
good reason to compute them.
(2) Same logic applies to the computation of rank correlation
statistics.
Unless you'd like to look at them, I'd suggest setting "rank_corr_flag
=
FALSE;". As the number of matched pairs increases, the computation of
rank
correlation statistics slows down.
(3) I see you've set "output_prefix = "2016032900_09hr";". If you
look at
the default output naming convention from Point-Stat, you'll see it
already
includes the model lead (050000L) and valid times (20160329_050000V).
You
are of course welcome to set that however you'd like, but it looks
somewhat
redundant with what's already being written.
(4) Lastly, I'd strongly recommend writing out the CTC and SL1L2 line
types. CTC contains contingency table counts. And the CTS line
(which
you're already writing) contains the statistics derived from those
counts.
Likewise, SL1L2 contains scalar partial sums and CNT contains the
corresponding statistics.
Suppose you run Point-Stat for 30 days. You'll have 30 CSI values,
each
computed for one of those days. You could look at a distribution of
those
30 CSI daily values (some of which may be NA)... or you could
aggregate
together the corresponding 30 daily contingency tables into one big
table
representing all 30 days. That means aggregating 30 CTC lines into
one big
CTC line. And from that aggregated table, you can compute a single
CSI
value representing performance over all 30 days.
This type of aggregation and computation of statistics is done by the
STAT-Analysis tool.
Likewise, you could aggregate 30 SL1L2 lines and compute a single RMSE
value for all 30 days.
I think you'll find CTC and SL1L2 output to be very useful.
(5) Lastly, if you'd like the finest level of detail possible, you
could
also write out the MPR (matched pair) output line type. But that
means 1
output line for each fcst/obs pair the tool finds. And that can be a
lot
of data.
Hope that helps.
Thanks,
John
On Thu, Mar 31, 2016 at 2:05 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
> Dear John,
>
> Thanks for the help. The files are in 'garcia_data'.
>
> I also attached my own results here, so you can compare them. I'm
focusing
> on:
>
> GSS
> CSI
> FBIAS
> RMSE
>
> If you have any advice on thresholds as well (commonly used and/or
> recommended, etc) that will be greatly appreciated.
>
> Thanks for the help,
> Jose
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 31, 2016 2:01 PM
> To: Jose Garcia-Rivera
> Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
> point_stat
>
> Jose,
>
> Sure, that'd be fine. You can post some sample data to our
anonymous ftp
> site following these instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Be sure to send us your forecast file, pb2nc output file, and point-
stat
> configuration file. I can run the case here and send you the
results.
>
> Some of the output statistics may be NA. NA's show up when it is
not
> numerically possible to compute the statistic or when the user has
not
> requested the output (i.e. requesting no bootstrapping results in NA
for
> the output bootstrap confidence interval columns).
>
> If you do post data, please reply to this email and I'll go grab it.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> > Transaction: Ticket created by GarciaJ at imsg.com
> > Queue: met_help
> > Subject: Missing values at times using point_stat
> > Owner: Nobody
> > Requestors: GarciaJ at imsg.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> > >
> >
> >
> > Dear Met Help,
> >
> > I'm getting some NA or scores of 0 for some fields, and I would
like
> > to make sure I'm doing things right using point_stat. Maybe I can
send
> > you an example of an observational, prepbufr and configure files
to
> > make sure we get the same scores?
> >
> > Thanks for the help,
> > Jose
> > This e-mail, including attachments, may include confidential
and/or
> > proprietary information, and may be used only by the person or
entity
> > to which it is addressed. If the reader of this e-mail is not the
> > intended recipient or his or her authorized agent, the reader is
> > hereby notified that any dissemination, distribution or copying of
> > this e-mail is prohibited. If you have received this e-mail in
error,
> > please notify the sender by replying to this message and delete
this
> e-mail immediately.
> >
> >
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity to
> which it is addressed. If the reader of this e-mail is not the
intended
> recipient or his or her authorized agent, the reader is hereby
notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify
the
> sender by replying to this message and delete this e-mail
immediately.
>
>
------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU
FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL
FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV
OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL
PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES
ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV
ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL
MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU
BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10
E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL
E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR
EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL
ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU
MSESS MSESS_BCL MSESS_BCU
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP Z2 TMP Z2
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 939 278.91812 278.6563 279.17993 278.64571
279.17018 4.0934 3.91626 4.28744 3.94621 4.24579
280.85064 280.57828 281.12299 280.58263 281.13713 4.2581 4.07384
4.45995 4.10997 4.39648 0.90891 0.8971 0.91942
0.89823 0.91917 0.91562 0.75236 939 465 783
-1.93252 -2.04698 -1.81806 -2.05275 -1.82466 1.78957 1.71213
1.8744 1.70111 1.87263 0.99312 0.9927 0.9935 2.20418
2.11911 2.29787 6.93378 6.4637 7.46741 3.19914 2.89068
3.50301 2.63321 2.54238 2.73266 -4.07879 -4.22379 -3.89775 -3.10649
-3.233 -2.97731 -2.02299 -2.13401 -1.88794 -0.81801 -0.99299
-0.60599 0.25201 0.032988 0.44901 2.28848 2.13294 2.52501
1.14102 1.06498 1.246 NA NA NA
NA NA 3.73464 3.32938 4.21377 0.61758 0.57474
0.65447
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH Z2 RH Z2
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 908 72.12564 71.08387 73.16741 71.05211
73.16202 16.01649 15.31218 16.78921 15.53791 16.50229
64.07651 62.98601 65.16701 63.02187 65.16011 16.76564 16.0284
17.57452 16.12266 17.44079 0.7487 0.71865 0.77596
0.7158 0.77776 0.76427 0.56253 908 464 82
8.04913 7.29193 8.80633 7.24383 8.78266 11.6414 11.12949
12.20305 10.87481 12.44715 1.12562 1.11227 1.13786 11.71805
11.20768 12.24184 200.1615 181.93968 220.79933 135.37298 118.13122
154.76082 14.14784 13.4885 14.85932 -6.89396 -8.31 -5.16364
1.65724 0.4222 2.63787 10.07039 8.77508 10.65085 14.85492
14.14216 15.65257 20.35536 19.34442 21.3413 13.19768 12.27272
14.41032 6.28231 5.6776 6.97012 NA NA NA
NA NA 64.78852 52.47309 77.13505 0.2879 0.21319
0.35245
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND Z10 WIND Z10
ADPSFC FULL NEAREST 1 NA NA NA
0.05 CNT 936 4.59267 4.42301 4.76232 4.44002 4.765
2.64818 2.53341 2.77393 2.49074 2.80494 3.63693
3.45551 3.81836 3.45836 3.81577 2.83191 2.70918 2.96638
2.69488 2.96167 0.71364 0.68069 0.74371 0.67948
0.74428 0.66281 0.48633 936 459 840 0.95573 0.82244
1.08902 0.82378 1.08771 2.08059 1.99042 2.17939 1.95774
2.2144 1.26279 1.21864 1.30985 1.7628 1.67671 1.85937
5.23767 4.70822 5.83044 4.32425 3.82866 4.89834 2.2886
2.16984 2.41463 -1.3576 -1.72187 -1.13599 -0.17321 -0.26851
-0.023683 0.97956 0.84307 1.11036 2.19437 1.95993 2.41586
3.35963 3.17559 3.64781 2.36758 2.12762 2.57916 1.17008 1.06827
1.27682 NA NA NA NA
NA 0.91342 0.67861 1.18311 0.3469 0.25828 0.41692
------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU
ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU
PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU
PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR
FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL
CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU
HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS ODDS_NCL
ODDS_NCU ODDS_BCL ODDS_BCU LODDS LODDS_NCL LODDS_NCU LODDS_BCL
LODDS_BCU ORSS ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL EDS_NCU EDS_BCL EDS_BCU SEDS SEDS_NCL SEDS_NCU
SEDS_BCL SEDS_BCU EDI EDI_NCL EDI_NCU EDI_BCL EDI_BCU SEDI
SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS BAGSS_BCL BAGSS_BCU
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP Z2 TMP Z2
ADPSFC FULL NEAREST 1 >=293 >=293 NA
0.05 CTS 939 0 0 0.0040743 0 0
0 0 0.0040743 0 0 1 0.99593 1
1 1 NA NA NA NA NA NA
NA NA 1 0.99593 1 1 1 0
0 0.0040743 0 0 NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH Z2 RH Z2
ADPSFC FULL NEAREST 1 >=50 >=50 NA
0.05 CTS 908 0.76322 0.73449 0.78972 0.73568 0.79185
0.91189 0.89168 0.92864 0.89317 0.92841 0.76322 0.73449
0.78972 0.73676 0.78855 1.19481 1.15223 1.23841 0.94228 0.92517
0.95567 0.92482 0.95931 0.18605 0.16208 0.21266 0.13513 0.23857
0.81395 0.78734 0.83792 0.76143 0.86487 0.21135 0.18604
0.2391 0.18599 0.23972 0.7523 0.7232 0.77928 0.72433 0.77896
0.089204 0.052284 0.1299 0.12833 0.091984 0.16467 0.07675
0.18178 0.1638 0.099372 0.22993 3.73143 2.33462 5.96395 2.34942
6.18404 1.31679 0.84785 1.78573 0.85417 1.82197 0.57729
0.42097 0.73362 0.40288 0.72161 0.63932 0.54769 0.73095 0.54216
0.73501 0.099429 0.037975 0.16088 0.058535 0.1445 0.55181
-0.064151 1.16778 0.38364 0.69596 0.27438 0.10648 0.99715
0.17683 0.3691 0.19182 0.1542 0.23377
V5.1 NMB 050000 20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND Z10 WIND Z10
ADPSFC FULL NEAREST 1 >=8 >=8 NA
0.05 CTS 936 0.090812 0.074038 0.11093 0.073718 0.109
0.13782 0.11721 0.16139 0.11752 0.16026 0.87821 0.85569
0.89763 0.85577 0.89957 1.51765 1.2576 1.88757 0.58824 0.55641
0.61934 0.48051 0.69161 0.90717 0.88688 0.92413 0.88628 0.92666
0.092832 0.075869 0.11312 0.07334 0.11372 0.6124 0.58079
0.6431 0.52174 0.69854 0.30488 0.27623 0.33512 0.23494 0.37889
0.2514 0.18311 0.32594 0.4954 0.38781 0.603 0.38698
0.60595 0.4018 0.30953 0.49164 13.96022 8.55164 22.78952 8.66514
24.26624 2.63621 2.14612 3.1263 2.15931 3.18909 0.86631
0.80517 0.92745 0.79307 0.92084 0.63775 0.53831 0.73718 0.53919
0.74416 0.49535 0.40456 0.58614 0.4086 0.57622 0.63501
0.50309 0.76692 0.52557 0.73799 0.67728 0.50502 0.76499
0.56636 0.77521 0.2282 0.16312 0.29867
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75719] Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Fri Apr 01 16:51:11 2016
Dear John,
Thank you so much for the reply. Everything makes sense. You're right,
I picked a high Temp value for verification. I made the suggested
changes. The string name is done to fit our own internal file naming
convention.
Have a great weekend!
Jose
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, April 01, 2016 2:59 PM
To: Jose Garcia-Rivera
Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
point_stat
Jose,
Thanks for sending your data. I ran it through met-5.1 point_stat and
compared my results to the results you sent me. And I've attached my
output files to this email.
The only differences I see show up in the columns ending in BCL and
BCU.
Those are the bootstrap confidence intervals Point-Stat computes. The
process of bootstrapping is inherently random, so I would fully expect
those values to change from run to run. So I see absolutely no
problem here.
You are wondering why some of your statistics are NA. For example,
looking in the CTS output file at the first line for 2-m temperature
>= 293, many of the statistics are 0. Notice that the "BASE_RATE"
column is 0 which means that none of observations that were verified
met that threshold criteria. So all of the obs had 2-m temp values <
293. Many of the contingency table statistics are based on at least 1
event occurring in the observation field... that's why many of those
stats are NA.
To understand the math, you can look at the equations in Appendix C of
the MET user's guide. You'll find that 0 observed events leads to a
'divide by 0' for many statistics. In the MET output, 'divide by 0'
is written as NA.
However, looking at your config file, let me make some suggestions:
(1) Unless you're actually using the bootstrap confidence intervals,
I'd suggest disabling them by setting "n_rep = 0;" in the "boot"
section.
Bootstrapping is inherently slow, so you should skip them unless you
have a good reason to compute them.
(2) Same logic applies to the computation of rank correlation
statistics.
Unless you'd like to look at them, I'd suggest setting "rank_corr_flag
= FALSE;". As the number of matched pairs increases, the computation
of rank correlation statistics slows down.
(3) I see you've set "output_prefix = "2016032900_09hr";". If you
look at the default output naming convention from Point-Stat, you'll
see it already includes the model lead (050000L) and valid times
(20160329_050000V). You are of course welcome to set that however
you'd like, but it looks somewhat redundant with what's already being
written.
(4) Lastly, I'd strongly recommend writing out the CTC and SL1L2 line
types. CTC contains contingency table counts. And the CTS line
(which you're already writing) contains the statistics derived from
those counts.
Likewise, SL1L2 contains scalar partial sums and CNT contains the
corresponding statistics.
Suppose you run Point-Stat for 30 days. You'll have 30 CSI values,
each computed for one of those days. You could look at a distribution
of those
30 CSI daily values (some of which may be NA)... or you could
aggregate together the corresponding 30 daily contingency tables into
one big table representing all 30 days. That means aggregating 30 CTC
lines into one big CTC line. And from that aggregated table, you can
compute a single CSI value representing performance over all 30 days.
This type of aggregation and computation of statistics is done by the
STAT-Analysis tool.
Likewise, you could aggregate 30 SL1L2 lines and compute a single RMSE
value for all 30 days.
I think you'll find CTC and SL1L2 output to be very useful.
(5) Lastly, if you'd like the finest level of detail possible, you
could also write out the MPR (matched pair) output line type. But
that means 1 output line for each fcst/obs pair the tool finds. And
that can be a lot of data.
Hope that helps.
Thanks,
John
On Thu, Mar 31, 2016 at 2:05 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
> Dear John,
>
> Thanks for the help. The files are in 'garcia_data'.
>
> I also attached my own results here, so you can compare them. I'm
> focusing
> on:
>
> GSS
> CSI
> FBIAS
> RMSE
>
> If you have any advice on thresholds as well (commonly used and/or
> recommended, etc) that will be greatly appreciated.
>
> Thanks for the help,
> Jose
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 31, 2016 2:01 PM
> To: Jose Garcia-Rivera
> Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
> point_stat
>
> Jose,
>
> Sure, that'd be fine. You can post some sample data to our
anonymous
> ftp site following these instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Be sure to send us your forecast file, pb2nc output file, and
> point-stat configuration file. I can run the case here and send you
the results.
>
> Some of the output statistics may be NA. NA's show up when it is
not
> numerically possible to compute the statistic or when the user has
not
> requested the output (i.e. requesting no bootstrapping results in NA
> for the output bootstrap confidence interval columns).
>
> If you do post data, please reply to this email and I'll go grab it.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> > Transaction: Ticket created by GarciaJ at imsg.com
> > Queue: met_help
> > Subject: Missing values at times using point_stat
> > Owner: Nobody
> > Requestors: GarciaJ at imsg.com
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> > >
> >
> >
> > Dear Met Help,
> >
> > I'm getting some NA or scores of 0 for some fields, and I would
like
> > to make sure I'm doing things right using point_stat. Maybe I can
> > send you an example of an observational, prepbufr and configure
> > files to make sure we get the same scores?
> >
> > Thanks for the help,
> > Jose
> > This e-mail, including attachments, may include confidential
and/or
> > proprietary information, and may be used only by the person or
> > entity to which it is addressed. If the reader of this e-mail is
not
> > the intended recipient or his or her authorized agent, the reader
is
> > hereby notified that any dissemination, distribution or copying of
> > this e-mail is prohibited. If you have received this e-mail in
> > error, please notify the sender by replying to this message and
> > delete this
> e-mail immediately.
> >
> >
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity
> to which it is addressed. If the reader of this e-mail is not the
> intended recipient or his or her authorized agent, the reader is
> hereby notified that any dissemination, distribution or copying of
> this e-mail is prohibited. If you have received this e-mail in
error,
> please notify the sender by replying to this message and delete this
e-mail immediately.
>
>
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the
intended recipient or his or her authorized agent, the reader is
hereby notified that any dissemination, distribution or copying of
this e-mail is prohibited. If you have received this e-mail in error,
please notify the sender by replying to this message and delete this
e-mail immediately.
------------------------------------------------
More information about the Met_help
mailing list