[Met_help] [rt.rap.ucar.edu #75719] History for Missing values at times using point_stat

John Halley Gotway via RT met_help at ucar.edu
Mon Apr 11 16:47:05 MDT 2016


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear Met Help,

I'm getting some NA or scores of 0 for some fields, and I would like to make sure I'm doing things right using point_stat. Maybe I can send you an example of an observational, prepbufr and configure files to make sure we get the same scores?

Thanks for the help,
Jose
This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Thu Mar 31 12:00:55 2016

Jose,

Sure, that'd be fine.  You can post some sample data to our anonymous
ftp
site following these instructions:
  http://www.dtcenter.org/met/users/support/met_help.php#ftp

Be sure to send us your forecast file, pb2nc output file, and point-
stat
configuration file.  I can run the case here and send you the results.

Some of the output statistics may be NA.  NA's show up when it is not
numerically possible to compute the statistic or when the user has not
requested the output (i.e. requesting no bootstrapping results in NA
for
the output bootstrap confidence interval columns).

If you do post data, please reply to this email and I'll go grab it.

Thanks,
John Halley Gotway

On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:

>
> Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> Transaction: Ticket created by GarciaJ at imsg.com
>        Queue: met_help
>      Subject: Missing values at times using point_stat
>        Owner: Nobody
>   Requestors: GarciaJ at imsg.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
>
> Dear Met Help,
>
> I'm getting some NA or scores of 0 for some fields, and I would like
to
> make sure I'm doing things right using point_stat. Maybe I can send
you an
> example of an observational, prepbufr and configure files to make
sure we
> get the same scores?
>
> Thanks for the help,
> Jose
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity to
> which it is addressed. If the reader of this e-mail is not the
intended
> recipient or his or her authorized agent, the reader is hereby
notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify
the
> sender by replying to this message and delete this e-mail
immediately.
>
>

------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016

Dear John,

Thanks for the help. The files are in 'garcia_data'.

I also attached my own results here, so you can compare them. I'm
focusing on:

GSS
CSI
FBIAS
RMSE

If you have any advice on thresholds as well (commonly used and/or
recommended, etc) that will be greatly appreciated.

Thanks for the help,
Jose






-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 31, 2016 2:01 PM
To: Jose Garcia-Rivera
Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
point_stat

Jose,

Sure, that'd be fine.  You can post some sample data to our anonymous
ftp site following these instructions:
  http://www.dtcenter.org/met/users/support/met_help.php#ftp

Be sure to send us your forecast file, pb2nc output file, and point-
stat configuration file.  I can run the case here and send you the
results.

Some of the output statistics may be NA.  NA's show up when it is not
numerically possible to compute the statistic or when the user has not
requested the output (i.e. requesting no bootstrapping results in NA
for the output bootstrap confidence interval columns).

If you do post data, please reply to this email and I'll go grab it.

Thanks,
John Halley Gotway

On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:

>
> Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> Transaction: Ticket created by GarciaJ at imsg.com
>        Queue: met_help
>      Subject: Missing values at times using point_stat
>        Owner: Nobody
>   Requestors: GarciaJ at imsg.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> >
>
>
> Dear Met Help,
>
> I'm getting some NA or scores of 0 for some fields, and I would like
> to make sure I'm doing things right using point_stat. Maybe I can
send
> you an example of an observational, prepbufr and configure files to
> make sure we get the same scores?
>
> Thanks for the help,
> Jose
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity
> to which it is addressed. If the reader of this e-mail is not the
> intended recipient or his or her authorized agent, the reader is
> hereby notified that any dissemination, distribution or copying of
> this e-mail is prohibited. If you have received this e-mail in
error,
> please notify the sender by replying to this message and delete this
e-mail immediately.
>
>

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the
intended recipient or his or her authorized agent, the reader is
hereby notified that any dissemination, distribution or copying of
this e-mail is prohibited. If you have received this e-mail in error,
please notify the sender by replying to this message and delete this
e-mail immediately.

------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR      FBAR_NCL  FBAR_NCU
FBAR_BCL  FBAR_BCU  FSTDEV   FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL
FSTDEV_BCU OBAR      OBAR_NCL  OBAR_NCU  OBAR_BCL  OBAR_BCU  OSTDEV
OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL
PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES
ORANK_TIES ME       ME_NCL   ME_NCU   ME_BCL   ME_BCU   ESTDEV
ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS   MBIAS_BCL
MBIAS_BCU MAE      MAE_BCL  MAE_BCU  MSE       MSE_BCL   MSE_BCU
BCMSE     BCMSE_BCL BCMSE_BCU RMSE     RMSE_BCL RMSE_BCU E10
E10_BCL  E10_BCU  E25      E25_BCL  E25_BCU   E50      E50_BCL
E50_BCU  E75      E75_BCL  E75_BCU  E90      E90_BCL   E90_BCU  EIQR
EIQR_BCL EIQR_BCU MAD     MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL
ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2      ME2_BCL  ME2_BCU
MSESS   MSESS_BCL MSESS_BCU
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP      Z2       TMP     Z2
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         939 278.91812 278.6563  279.17993 278.66298
279.19824  4.0934     3.91626    4.28744    3.95624    4.24214
280.85064 280.57828 281.12299 280.58551 281.13963  4.2581     4.07384
4.45995    4.10819    4.39729 0.90891     0.8971      0.91942
0.89695     0.91998 0.91562 0.75236   939        465        783
-1.93252 -2.04698 -1.81806 -2.04514 -1.81877  1.78957    1.71213
1.8744     1.69425    1.88183 0.99312   0.99272   0.99352  2.20418
2.11303  2.29832   6.93378   6.42241   7.48957   3.19914   2.86743
3.53752  2.63321  2.53425  2.73671 -4.07879 -4.22199 -3.89618 -3.10649
-3.2305  -2.97731  -2.02299 -2.13401 -1.88599 -0.81801 -0.96999
-0.60497  0.25201  0.045994  0.4576   2.28848  2.12896  2.52304
1.14102 1.06201 1.25307        NA            NA            NA
NA            NA  3.73464  3.30792  4.18258 0.61758   0.57372
0.65528
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH       Z2       RH      Z2
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         908  72.12564  71.08387  73.16741  71.08168
73.10797 16.01649   15.31218   16.78921   15.5477    16.48678
64.07651  62.98601  65.16701  63.07106  65.25552 16.76564   16.0284
17.57452   16.12883   17.39894 0.7487      0.71865     0.77596
0.71288     0.77779 0.76427 0.56253   908        464         82
8.04913  7.29193  8.80633  7.27352  8.74397 11.6414    11.12949
12.20305   10.86934   12.48742 1.12562   1.11265   1.13788 11.71805
11.23768 12.21655 200.1615  182.28383 220.82803 135.37298 118.0125
155.76391 14.14784 13.50125 14.86028 -6.89396 -8.31    -5.15296
1.65724  0.52178  2.65279  10.07039  8.84892 10.62013 14.85492
14.16715 15.63692 20.35536 19.44352  21.13064 13.19768 12.27761
14.31857 6.28231 5.70165 7.02628        NA            NA            NA
NA            NA 64.78852 52.9041  76.45705 0.2879    0.21828
0.35157
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND     Z10      WIND    Z10
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         936   4.59267   4.42301   4.76232   4.43297
4.76131  2.64818    2.53341    2.77393    2.47928    2.82708   3.63693
3.45551   3.81836   3.46415   3.81223  2.83191    2.70918    2.96638
2.71158    2.96677 0.71364     0.68069     0.74371     0.68037
0.74715 0.66281 0.48633   936        459        840  0.95573  0.82244
1.08902  0.82302  1.09018  2.08059    1.99042    2.17939    1.96103
2.21224 1.26279   1.22113   1.30828  1.7628   1.67329  1.85966
5.23767   4.73692   5.84173   4.32425   3.84154   4.88877  2.2886
2.17645  2.41697 -1.3576  -1.7195  -1.10904 -0.17321 -0.26662
-0.029232  0.97956  0.85551  1.1217   2.19437  1.96061  2.38548
3.35963  3.18851   3.65961  2.36758  2.14819  2.59269 1.17008 1.07611
1.28525        NA            NA            NA            NA
NA  0.91342  0.67736  1.18849 0.3469    0.2635    0.42189

------------------------------------------------
Subject: Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Thu Mar 31 14:05:22 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER    BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN   FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU
ACC     ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS    FBIAS_BCL FBIAS_BCU
PODY     PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN    PODN_NCL PODN_NCU
PODN_BCL PODN_BCU POFD     POFD_NCL POFD_NCU  POFD_BCL POFD_BCU FAR
FAR_NCL  FAR_NCU FAR_BCL  FAR_BCU  CSI      CSI_NCL  CSI_NCU  CSI_BCL
CSI_BCU  GSS       GSS_BCL   GSS_BCU  HK       HK_NCL    HK_NCU
HK_BCL    HK_BCU   HSS     HSS_BCL   HSS_BCU  ODDS     ODDS_NCL
ODDS_NCU ODDS_BCL ODDS_BCU LODDS    LODDS_NCL LODDS_NCU LODDS_BCL
LODDS_BCU ORSS     ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL  EDS_NCU  EDS_BCL  EDS_BCU SEDS      SEDS_NCL  SEDS_NCU
SEDS_BCL  SEDS_BCU EDI      EDI_NCL   EDI_NCU  EDI_BCL  EDI_BCU  SEDI
SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS    BAGSS_BCL BAGSS_BCU
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP      Z2       TMP     Z2
ADPSFC FULL    NEAREST     1           >=293       >=293      NA
0.05  CTS         939 0         0        0.0040743  0          0
0         0       0.0040743   0         0       1       0.99593 1
1       1       NA        NA        NA       NA       NA       NA
NA       NA       1        0.99593  1        1        1       0
0        0.0040743 0         0       NA       NA       NA      NA
NA       NA       NA       NA       NA       NA       NA        NA
NA       NA       NA        NA       NA        NA       NA      NA
NA       NA       NA       NA       NA       NA       NA        NA
NA        NA        NA       NA       NA       NA       NA       NA
NA       NA       NA       NA       NA      NA        NA        NA
NA        NA       NA       NA        NA       NA       NA       NA
NA       NA       NA       NA       NA        NA        NA
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH       Z2       RH      Z2
ADPSFC FULL    NEAREST     1           >=50        >=50       NA
0.05  CTS         908 0.76322   0.73449  0.78972    0.73348    0.78965
0.91189   0.89168 0.92864     0.89317   0.93062 0.76322 0.73449
0.78972 0.73566 0.79075  1.19481   1.15374   1.24366  0.94228  0.92517
0.95567  0.92405  0.95971 0.18605  0.16208  0.21266  0.13402  0.24109
0.81395  0.78734  0.83792   0.75891   0.86598  0.21135  0.18604
0.2391  0.18539  0.24106  0.7523   0.7232   0.77928  0.7233   0.78129
0.089204  0.050183  0.13127  0.12833  0.091984  0.16467  0.073416
0.18554  0.1638  0.095571  0.23208  3.73143  2.33462  5.96395  2.37957
6.10007  1.31679   0.84785   1.78573   0.86692   1.8083   0.57729
0.42097  0.73362  0.40821  0.71831  0.63932  0.54769  0.73095  0.54235
0.7401  0.099429  0.037975  0.16088  0.056088  0.14586  0.55181
-0.064151  1.16778  0.39088  0.6903   0.27438  0.10648  0.99715
0.17561  0.36874  0.19182   0.15575   0.23327
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND     Z10      WIND    Z10
ADPSFC FULL    NEAREST     1           >=8         >=8        NA
0.05  CTS         936 0.090812  0.074038 0.11093    0.071581   0.10897
0.13782   0.11721 0.16139     0.11645   0.16132 0.87821 0.85569
0.89763 0.85681 0.89957  1.51765   1.24981   1.88893  0.58824  0.55641
0.61934  0.49425  0.70216 0.90717  0.88688  0.92413  0.88756  0.92689
0.092832 0.075869 0.11312   0.073112  0.11244  0.6124   0.58079
0.6431  0.52828  0.69786  0.30488  0.27623  0.33512  0.23728  0.37586
0.2514    0.18538   0.32416  0.4954   0.38781   0.603    0.39934
0.60584  0.4018  0.31277   0.48961 13.96022  8.55164 22.78952  8.96456
23.31698  2.63621   2.14612   3.1263    2.19328   3.14918  0.86631
0.80517  0.92745  0.79929  0.91775  0.63775  0.53831  0.73718  0.53831
0.7448  0.49535   0.40456   0.58614  0.41692   0.57208  0.63501
0.50309   0.76692  0.54008  0.73849  0.67728  0.50502  0.76499
0.58193  0.77588  0.2282    0.16542   0.29512

------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016

Jose,

Thanks for sending your data.  I ran it through met-5.1 point_stat and
compared my results to the results you sent me.  And I've attached my
output files to this email.

The only differences I see show up in the columns ending in BCL and
BCU.
Those are the bootstrap confidence intervals Point-Stat computes.  The
process of bootstrapping is inherently random, so I would fully expect
those values to change from run to run.  So I see absolutely no
problem
here.

You are wondering why some of your statistics are NA.  For example,
looking
in the CTS output file at the first line for 2-m temperature >= 293,
many
of the statistics are 0.  Notice that the "BASE_RATE" column is 0
which
means that none of observations that were verified met that threshold
criteria.  So all of the obs had 2-m temp values < 293.  Many of the
contingency table statistics are based on at least 1 event occurring
in the
observation field... that's why many of those stats are NA.

To understand the math, you can look at the equations in Appendix C of
the
MET user's guide.  You'll find that 0 observed events leads to a
'divide by
0' for many statistics.  In the MET output, 'divide by 0' is written
as NA.

However, looking at your config file, let me make some suggestions:

(1) Unless you're actually using the bootstrap confidence intervals,
I'd
suggest disabling them by setting "n_rep = 0;" in the "boot" section.
Bootstrapping is inherently slow, so you should skip them unless you
have a
good reason to compute them.

(2) Same logic applies to the computation of rank correlation
statistics.
Unless you'd like to look at them, I'd suggest setting "rank_corr_flag
=
FALSE;".  As the number of matched pairs increases, the computation of
rank
correlation statistics slows down.

(3) I see you've set "output_prefix  = "2016032900_09hr";".  If you
look at
the default output naming convention from Point-Stat, you'll see it
already
includes the model lead (050000L) and valid times (20160329_050000V).
You
are of course welcome to set that however you'd like, but it looks
somewhat
redundant with what's already being written.

(4) Lastly, I'd strongly recommend writing out the CTC and SL1L2 line
types.  CTC contains contingency table counts.  And the CTS line
(which
you're already writing) contains the statistics derived from those
counts.
Likewise, SL1L2 contains scalar partial sums and CNT contains the
corresponding statistics.

Suppose you run Point-Stat for 30 days.  You'll have 30 CSI values,
each
computed for one of those days.  You could look at a distribution of
those
30 CSI daily values (some of which may be NA)... or you could
aggregate
together the corresponding 30 daily contingency tables into one big
table
representing all 30 days.  That means aggregating 30 CTC lines into
one big
CTC line.  And from that aggregated table, you can compute a single
CSI
value representing performance over all 30 days.

This type of aggregation and computation of statistics is done by the
STAT-Analysis tool.

Likewise, you could aggregate 30 SL1L2 lines and compute a single RMSE
value for all 30 days.

I think you'll find CTC and SL1L2 output to be very useful.

(5) Lastly, if you'd like the finest level of detail possible, you
could
also write out the MPR (matched pair) output line type.  But that
means 1
output line for each fcst/obs pair the tool finds.  And that can be a
lot
of data.

Hope that helps.

Thanks,
John



On Thu, Mar 31, 2016 at 2:05 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
> Dear John,
>
> Thanks for the help. The files are in 'garcia_data'.
>
> I also attached my own results here, so you can compare them. I'm
focusing
> on:
>
> GSS
> CSI
> FBIAS
> RMSE
>
> If you have any advice on thresholds as well (commonly used and/or
> recommended, etc) that will be greatly appreciated.
>
> Thanks for the help,
> Jose
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 31, 2016 2:01 PM
> To: Jose Garcia-Rivera
> Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
> point_stat
>
> Jose,
>
> Sure, that'd be fine.  You can post some sample data to our
anonymous ftp
> site following these instructions:
>   http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Be sure to send us your forecast file, pb2nc output file, and point-
stat
> configuration file.  I can run the case here and send you the
results.
>
> Some of the output statistics may be NA.  NA's show up when it is
not
> numerically possible to compute the statistic or when the user has
not
> requested the output (i.e. requesting no bootstrapping results in NA
for
> the output bootstrap confidence interval columns).
>
> If you do post data, please reply to this email and I'll go grab it.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> > Transaction: Ticket created by GarciaJ at imsg.com
> >        Queue: met_help
> >      Subject: Missing values at times using point_stat
> >        Owner: Nobody
> >   Requestors: GarciaJ at imsg.com
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> > >
> >
> >
> > Dear Met Help,
> >
> > I'm getting some NA or scores of 0 for some fields, and I would
like
> > to make sure I'm doing things right using point_stat. Maybe I can
send
> > you an example of an observational, prepbufr and configure files
to
> > make sure we get the same scores?
> >
> > Thanks for the help,
> > Jose
> > This e-mail, including attachments, may include confidential
and/or
> > proprietary information, and may be used only by the person or
entity
> > to which it is addressed. If the reader of this e-mail is not the
> > intended recipient or his or her authorized agent, the reader is
> > hereby notified that any dissemination, distribution or copying of
> > this e-mail is prohibited. If you have received this e-mail in
error,
> > please notify the sender by replying to this message and delete
this
> e-mail immediately.
> >
> >
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity to
> which it is addressed. If the reader of this e-mail is not the
intended
> recipient or his or her authorized agent, the reader is hereby
notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify
the
> sender by replying to this message and delete this e-mail
immediately.
>
>

------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR      FBAR_NCL  FBAR_NCU
FBAR_BCL  FBAR_BCU  FSTDEV   FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL
FSTDEV_BCU OBAR      OBAR_NCL  OBAR_NCU  OBAR_BCL  OBAR_BCU  OSTDEV
OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL
PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES
ORANK_TIES ME       ME_NCL   ME_NCU   ME_BCL   ME_BCU   ESTDEV
ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS   MBIAS_BCL
MBIAS_BCU MAE      MAE_BCL  MAE_BCU  MSE       MSE_BCL   MSE_BCU
BCMSE     BCMSE_BCL BCMSE_BCU RMSE     RMSE_BCL RMSE_BCU E10
E10_BCL  E10_BCU  E25      E25_BCL  E25_BCU   E50      E50_BCL
E50_BCU  E75      E75_BCL  E75_BCU  E90      E90_BCL   E90_BCU  EIQR
EIQR_BCL EIQR_BCU MAD     MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL
ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2      ME2_BCL  ME2_BCU
MSESS   MSESS_BCL MSESS_BCU
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP      Z2       TMP     Z2
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         939 278.91812 278.6563  279.17993 278.64571
279.17018  4.0934     3.91626    4.28744    3.94621    4.24579
280.85064 280.57828 281.12299 280.58263 281.13713  4.2581     4.07384
4.45995    4.10997    4.39648 0.90891     0.8971      0.91942
0.89823     0.91917 0.91562 0.75236   939        465        783
-1.93252 -2.04698 -1.81806 -2.05275 -1.82466  1.78957    1.71213
1.8744     1.70111    1.87263 0.99312   0.9927    0.9935   2.20418
2.11911  2.29787   6.93378   6.4637    7.46741   3.19914   2.89068
3.50301  2.63321  2.54238  2.73266 -4.07879 -4.22379 -3.89775 -3.10649
-3.233   -2.97731  -2.02299 -2.13401 -1.88794 -0.81801 -0.99299
-0.60599  0.25201  0.032988  0.44901  2.28848  2.13294  2.52501
1.14102 1.06498 1.246          NA            NA            NA
NA            NA  3.73464  3.32938  4.21377 0.61758   0.57474
0.65447
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH       Z2       RH      Z2
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         908  72.12564  71.08387  73.16741  71.05211
73.16202 16.01649   15.31218   16.78921   15.53791   16.50229
64.07651  62.98601  65.16701  63.02187  65.16011 16.76564   16.0284
17.57452   16.12266   17.44079 0.7487      0.71865     0.77596
0.7158      0.77776 0.76427 0.56253   908        464         82
8.04913  7.29193  8.80633  7.24383  8.78266 11.6414    11.12949
12.20305   10.87481   12.44715 1.12562   1.11227   1.13786 11.71805
11.20768 12.24184 200.1615  181.93968 220.79933 135.37298 118.13122
154.76082 14.14784 13.4885  14.85932 -6.89396 -8.31    -5.16364
1.65724  0.4222   2.63787  10.07039  8.77508 10.65085 14.85492
14.14216 15.65257 20.35536 19.34442  21.3413  13.19768 12.27272
14.41032 6.28231 5.6776  6.97012        NA            NA            NA
NA            NA 64.78852 52.47309 77.13505 0.2879    0.21319
0.35245
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND     Z10      WIND    Z10
ADPSFC FULL    NEAREST     1           NA          NA         NA
0.05  CNT         936   4.59267   4.42301   4.76232   4.44002   4.765
2.64818    2.53341    2.77393    2.49074    2.80494   3.63693
3.45551   3.81836   3.45836   3.81577  2.83191    2.70918    2.96638
2.69488    2.96167 0.71364     0.68069     0.74371     0.67948
0.74428 0.66281 0.48633   936        459        840  0.95573  0.82244
1.08902  0.82378  1.08771  2.08059    1.99042    2.17939    1.95774
2.2144  1.26279   1.21864   1.30985  1.7628   1.67671  1.85937
5.23767   4.70822   5.83044   4.32425   3.82866   4.89834  2.2886
2.16984  2.41463 -1.3576  -1.72187 -1.13599 -0.17321 -0.26851
-0.023683  0.97956  0.84307  1.11036  2.19437  1.95993  2.41586
3.35963  3.17559   3.64781  2.36758  2.12762  2.57916 1.17008 1.06827
1.27682        NA            NA            NA            NA
NA  0.91342  0.67861  1.18311 0.3469    0.25828   0.41692

------------------------------------------------
Subject: Missing values at times using point_stat
From: John Halley Gotway
Time: Fri Apr 01 12:58:42 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER    BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN   FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU
ACC     ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS    FBIAS_BCL FBIAS_BCU
PODY     PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN    PODN_NCL PODN_NCU
PODN_BCL PODN_BCU POFD     POFD_NCL POFD_NCU  POFD_BCL POFD_BCU FAR
FAR_NCL  FAR_NCU FAR_BCL  FAR_BCU  CSI      CSI_NCL  CSI_NCU  CSI_BCL
CSI_BCU  GSS       GSS_BCL   GSS_BCU  HK       HK_NCL    HK_NCU
HK_BCL   HK_BCU   HSS     HSS_BCL   HSS_BCU  ODDS     ODDS_NCL
ODDS_NCU ODDS_BCL ODDS_BCU LODDS    LODDS_NCL LODDS_NCU LODDS_BCL
LODDS_BCU ORSS     ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL  EDS_NCU  EDS_BCL  EDS_BCU  SEDS      SEDS_NCL  SEDS_NCU
SEDS_BCL  SEDS_BCU EDI      EDI_NCL   EDI_NCU  EDI_BCL  EDI_BCU  SEDI
SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS    BAGSS_BCL BAGSS_BCU
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 TMP      Z2       TMP     Z2
ADPSFC FULL    NEAREST     1           >=293       >=293      NA
0.05  CTS         939 0         0        0.0040743  0          0
0         0       0.0040743   0         0       1       0.99593 1
1       1       NA        NA        NA       NA       NA       NA
NA       NA       1        0.99593  1        1        1       0
0        0.0040743  0        0       NA       NA       NA      NA
NA       NA       NA       NA       NA       NA       NA        NA
NA       NA       NA        NA       NA       NA       NA      NA
NA       NA       NA       NA       NA       NA       NA        NA
NA        NA        NA       NA       NA       NA       NA       NA
NA       NA       NA       NA       NA       NA        NA        NA
NA        NA       NA       NA        NA       NA       NA       NA
NA       NA       NA       NA       NA        NA        NA
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 RH       Z2       RH      Z2
ADPSFC FULL    NEAREST     1           >=50        >=50       NA
0.05  CTS         908 0.76322   0.73449  0.78972    0.73568    0.79185
0.91189   0.89168 0.92864     0.89317   0.92841 0.76322 0.73449
0.78972 0.73676 0.78855  1.19481   1.15223   1.23841  0.94228  0.92517
0.95567  0.92482  0.95931 0.18605  0.16208  0.21266  0.13513  0.23857
0.81395  0.78734  0.83792    0.76143  0.86487  0.21135  0.18604
0.2391  0.18599  0.23972  0.7523   0.7232   0.77928  0.72433  0.77896
0.089204  0.052284  0.1299   0.12833  0.091984  0.16467  0.07675
0.18178  0.1638  0.099372  0.22993  3.73143  2.33462  5.96395  2.34942
6.18404  1.31679   0.84785   1.78573   0.85417   1.82197  0.57729
0.42097  0.73362  0.40288  0.72161  0.63932  0.54769  0.73095  0.54216
0.73501  0.099429  0.037975  0.16088  0.058535  0.1445   0.55181
-0.064151  1.16778  0.38364  0.69596  0.27438  0.10648  0.99715
0.17683  0.3691   0.19182   0.1542    0.23377
V5.1    NMB   050000    20160329_050000 20160329_050000 000000
20160329_033000 20160329_063000 WIND     Z10      WIND    Z10
ADPSFC FULL    NEAREST     1           >=8         >=8        NA
0.05  CTS         936 0.090812  0.074038 0.11093    0.073718   0.109
0.13782   0.11721 0.16139     0.11752   0.16026 0.87821 0.85569
0.89763 0.85577 0.89957  1.51765   1.2576    1.88757  0.58824  0.55641
0.61934  0.48051  0.69161 0.90717  0.88688  0.92413  0.88628  0.92666
0.092832 0.075869 0.11312    0.07334  0.11372  0.6124   0.58079
0.6431  0.52174  0.69854  0.30488  0.27623  0.33512  0.23494  0.37889
0.2514    0.18311   0.32594  0.4954   0.38781   0.603    0.38698
0.60595  0.4018  0.30953   0.49164 13.96022  8.55164 22.78952  8.66514
24.26624  2.63621   2.14612   3.1263    2.15931   3.18909  0.86631
0.80517  0.92745  0.79307  0.92084  0.63775  0.53831  0.73718  0.53919
0.74416  0.49535   0.40456   0.58614  0.4086    0.57622  0.63501
0.50309   0.76692  0.52557  0.73799  0.67728  0.50502  0.76499
0.56636  0.77521  0.2282    0.16312   0.29867

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75719] Missing values at times using point_stat
From: Jose Garcia-Rivera
Time: Fri Apr 01 16:51:11 2016

Dear John,

Thank you so much for the reply. Everything makes sense. You're right,
I picked a high Temp value for verification. I made the suggested
changes. The string name is done to fit our own internal file naming
convention.

Have a great weekend!
Jose

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, April 01, 2016 2:59 PM
To: Jose Garcia-Rivera
Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
point_stat

Jose,

Thanks for sending your data.  I ran it through met-5.1 point_stat and
compared my results to the results you sent me.  And I've attached my
output files to this email.

The only differences I see show up in the columns ending in BCL and
BCU.
Those are the bootstrap confidence intervals Point-Stat computes.  The
process of bootstrapping is inherently random, so I would fully expect
those values to change from run to run.  So I see absolutely no
problem here.

You are wondering why some of your statistics are NA.  For example,
looking in the CTS output file at the first line for 2-m temperature
>= 293, many of the statistics are 0.  Notice that the "BASE_RATE"
column is 0 which means that none of observations that were verified
met that threshold criteria.  So all of the obs had 2-m temp values <
293.  Many of the contingency table statistics are based on at least 1
event occurring in the observation field... that's why many of those
stats are NA.

To understand the math, you can look at the equations in Appendix C of
the MET user's guide.  You'll find that 0 observed events leads to a
'divide by 0' for many statistics.  In the MET output, 'divide by 0'
is written as NA.

However, looking at your config file, let me make some suggestions:

(1) Unless you're actually using the bootstrap confidence intervals,
I'd suggest disabling them by setting "n_rep = 0;" in the "boot"
section.
Bootstrapping is inherently slow, so you should skip them unless you
have a good reason to compute them.

(2) Same logic applies to the computation of rank correlation
statistics.
Unless you'd like to look at them, I'd suggest setting "rank_corr_flag
= FALSE;".  As the number of matched pairs increases, the computation
of rank correlation statistics slows down.

(3) I see you've set "output_prefix  = "2016032900_09hr";".  If you
look at the default output naming convention from Point-Stat, you'll
see it already includes the model lead (050000L) and valid times
(20160329_050000V).  You are of course welcome to set that however
you'd like, but it looks somewhat redundant with what's already being
written.

(4) Lastly, I'd strongly recommend writing out the CTC and SL1L2 line
types.  CTC contains contingency table counts.  And the CTS line
(which you're already writing) contains the statistics derived from
those counts.
Likewise, SL1L2 contains scalar partial sums and CNT contains the
corresponding statistics.

Suppose you run Point-Stat for 30 days.  You'll have 30 CSI values,
each computed for one of those days.  You could look at a distribution
of those
30 CSI daily values (some of which may be NA)... or you could
aggregate together the corresponding 30 daily contingency tables into
one big table representing all 30 days.  That means aggregating 30 CTC
lines into one big CTC line.  And from that aggregated table, you can
compute a single CSI value representing performance over all 30 days.

This type of aggregation and computation of statistics is done by the
STAT-Analysis tool.

Likewise, you could aggregate 30 SL1L2 lines and compute a single RMSE
value for all 30 days.

I think you'll find CTC and SL1L2 output to be very useful.

(5) Lastly, if you'd like the finest level of detail possible, you
could also write out the MPR (matched pair) output line type.  But
that means 1 output line for each fcst/obs pair the tool finds.  And
that can be a lot of data.

Hope that helps.

Thanks,
John



On Thu, Mar 31, 2016 at 2:05 PM, Jose Garcia-Rivera via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719 >
>
> Dear John,
>
> Thanks for the help. The files are in 'garcia_data'.
>
> I also attached my own results here, so you can compare them. I'm
> focusing
> on:
>
> GSS
> CSI
> FBIAS
> RMSE
>
> If you have any advice on thresholds as well (commonly used and/or
> recommended, etc) that will be greatly appreciated.
>
> Thanks for the help,
> Jose
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 31, 2016 2:01 PM
> To: Jose Garcia-Rivera
> Subject: Re: [rt.rap.ucar.edu #75719] Missing values at times using
> point_stat
>
> Jose,
>
> Sure, that'd be fine.  You can post some sample data to our
anonymous
> ftp site following these instructions:
>   http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Be sure to send us your forecast file, pb2nc output file, and
> point-stat configuration file.  I can run the case here and send you
the results.
>
> Some of the output statistics may be NA.  NA's show up when it is
not
> numerically possible to compute the statistic or when the user has
not
> requested the output (i.e. requesting no bootstrapping results in NA
> for the output bootstrap confidence interval columns).
>
> If you do post data, please reply to this email and I'll go grab it.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Mar 30, 2016 at 4:52 PM, Jose Garcia-Rivera via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Mar 30 16:52:07 2016: Request 75719 was acted upon.
> > Transaction: Ticket created by GarciaJ at imsg.com
> >        Queue: met_help
> >      Subject: Missing values at times using point_stat
> >        Owner: Nobody
> >   Requestors: GarciaJ at imsg.com
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75719
> > >
> >
> >
> > Dear Met Help,
> >
> > I'm getting some NA or scores of 0 for some fields, and I would
like
> > to make sure I'm doing things right using point_stat. Maybe I can
> > send you an example of an observational, prepbufr and configure
> > files to make sure we get the same scores?
> >
> > Thanks for the help,
> > Jose
> > This e-mail, including attachments, may include confidential
and/or
> > proprietary information, and may be used only by the person or
> > entity to which it is addressed. If the reader of this e-mail is
not
> > the intended recipient or his or her authorized agent, the reader
is
> > hereby notified that any dissemination, distribution or copying of
> > this e-mail is prohibited. If you have received this e-mail in
> > error, please notify the sender by replying to this message and
> > delete this
> e-mail immediately.
> >
> >
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or
entity
> to which it is addressed. If the reader of this e-mail is not the
> intended recipient or his or her authorized agent, the reader is
> hereby notified that any dissemination, distribution or copying of
> this e-mail is prohibited. If you have received this e-mail in
error,
> please notify the sender by replying to this message and delete this
e-mail immediately.
>
>

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the
intended recipient or his or her authorized agent, the reader is
hereby notified that any dissemination, distribution or copying of
this e-mail is prohibited. If you have received this e-mail in error,
please notify the sender by replying to this message and delete this
e-mail immediately.


------------------------------------------------


More information about the Met_help mailing list