[Met_help] [rt.rap.ucar.edu #77885] History for FBIAS problem

John Halley Gotway via RT met_help at ucar.edu
Tue Sep 13 14:10:27 MDT 2016


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello.  Running into a problem with FBIAS: see attached xml. Comparing 
the FBIAS numbers listed in the R data with FVS results, they match up 
well except at 3" and 4" thresholds:

threshold           Bias from FVS            From R data

3"/day                 36.333                       32.500

4"/day                   5.000                         6.000

Also attached is the VSDB records for all thresholds for this particular 
verification, which is consistent with the FVS results (detailed 
calculations omitted since this is simply a ratio of fcst/obs counts).  
Could you look into this?  The discrepancy above isn't huge, but I'm 
trying to compare the monthly scores in 
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/201607/ncep_eqts.201607.gif 
against that from the MetViewer, and over a month the differences in 
FBIAS at 3" are huge (GSS matched up well).

Thanks,

Ying

-- 
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 09:50:36 2016

Hi Ying,

Thanks for doing a careful comparison between FVS and METViewer.  I
think
that is exactly what is needed!

However, Tatiana is on travel until Sept 23rd but will return on the
26th.
Can this issue wait until then?

Thanks,
John Halley Gotway

On Tue, Sep 13, 2016 at 9:41 AM, Ying Lin via RT <met_help at ucar.edu>
wrote:

>
> Tue Sep 13 09:41:02 2016: Request 77885 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
>        Queue: met_help
>      Subject: FBIAS problem
>        Owner: Nobody
>   Requestors: ying.lin at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
>
> Hello.  Running into a problem with FBIAS: see attached xml.
Comparing
> the FBIAS numbers listed in the R data with FVS results, they match
up
> well except at 3" and 4" thresholds:
>
> threshold           Bias from FVS            From R data
>
> 3"/day                 36.333                       32.500
>
> 4"/day                   5.000                         6.000
>
> Also attached is the VSDB records for all thresholds for this
particular
> verification, which is consistent with the FVS results (detailed
> calculations omitted since this is simply a ratio of fcst/obs
counts).
> Could you look into this?  The discrepancy above isn't huge, but I'm
> trying to compare the monthly scores in
> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/
> 201607/ncep_eqts.201607.gif
> against that from the MetViewer, and over a month the differences in
> FBIAS at 3" are huge (GSS matched up well).
>
> Thanks,
>
> Ying
>
> --
> Ying Lin
> NCEP/EMC/Mesoscale Modeling Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77885] FBIAS problem
From: Ying Lin
Time: Tue Sep 13 12:04:32 2016

Hi John, yes it can wait.  Thanks for letting me know of Tatiana's
schedule.

Would like to ask to have fractions skill scores (24h and 6h) added to
the MetViewer from the VSDBs - not sure how to request this.  I'll ask
at the next DTC/EMC MET telecon.

Thanks,

Ying

On 09/13/2016 11:50 AM, John Halley Gotway via RT wrote:
> Hi Ying,
>
> Thanks for doing a careful comparison between FVS and METViewer.  I
think
> that is exactly what is needed!
>
> However, Tatiana is on travel until Sept 23rd but will return on the
26th.
> Can this issue wait until then?
>
> Thanks,
> John Halley Gotway
>
> On Tue, Sep 13, 2016 at 9:41 AM, Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Tue Sep 13 09:41:02 2016: Request 77885 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>>         Queue: met_help
>>       Subject: FBIAS problem
>>         Owner: Nobody
>>    Requestors: ying.lin at noaa.gov
>>        Status: new
>>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>>
>>
>> Hello.  Running into a problem with FBIAS: see attached xml.
Comparing
>> the FBIAS numbers listed in the R data with FVS results, they match
up
>> well except at 3" and 4" thresholds:
>>
>> threshold           Bias from FVS            From R data
>>
>> 3"/day                 36.333                       32.500
>>
>> 4"/day                   5.000                         6.000
>>
>> Also attached is the VSDB records for all thresholds for this
particular
>> verification, which is consistent with the FVS results (detailed
>> calculations omitted since this is simply a ratio of fcst/obs
counts).
>> Could you look into this?  The discrepancy above isn't huge, but
I'm
>> trying to compare the monthly scores in
>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/
>> 201607/ncep_eqts.201607.gif
>> against that from the MetViewer, and over a month the differences
in
>> FBIAS at 3" are huge (GSS matched up well).
>>
>> Thanks,
>>
>> Ying
>>
>> --
>> Ying Lin
>> NCEP/EMC/Mesoscale Modeling Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>


--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016

Hi Ying,

This is John again.  I thought I'd dig in a little on this one by
doing the
following...

(1) MET includes support for the FHO line type, not in VSDB format but
in
the MET format, where the header columns differ.  So I reformatted the
vsdb.txt file you sent into that MET format (see attached
"met_fho.txt")
file.

(2) I ran the following job through the MET STAT-Analysis tool to read
FHO
lines and derived contingency table statistics (i.e. a CTS line):

/usr/local/met-5.2/bin/stat_analysis \
   -lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
CTS \
   -by FCST_THRESH -out_stat met_cts.txt

I've attached the output file named met_cts.txt.  In that file, FBIAS
=
32.5 and 6.0 for thresholds >3.0 and >4.0, respectively.  So I'm glad
that
MET and METViewer are consistent at least!

For that >3.0 line, here's the math that's occurring in MET...

Input:
  TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003

We convert the FHO rates into integer contingency table counts.  From
MET
source code file "parse_stat_line.cc"...

FY = round(TOTAL * F_RATE) = 65
FY_OY = round(TOTAL * H_RATE) = 0
OY = round(TOTAL * O_RATE) = 2

FY_ON = FY - FY_OY = 65
FN_OY = OY - FY_OY = 2
FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603

And with those counts, we compute...
FBIAS = FY / OY = 65 / 2 = 32.5

However, I suspect FVS is computing FBIAS as follows...

FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE / O_RATE =
0.00109 / 0.00003 = 36.33

So we have two different answers... 32.5 and 36.33, and the question
is
which is more correct?

I would say the real problem here is using only 5 significant digits
for
"rare" events (rare being 24-hour precip accumulations of 3"+).  If we
had
20 decimal places, or 5 significant figure in scientific notation, we
would
not have any real difference in the results.

Assuming the counts MET computed are correct, the real rates should
be...
   F_RATE = 65 / 59670 = 0.001089324619
   O_RATE = 2 / 59670 = 0.00003351768057
And using those values...
   FBIAS = 32.5

Really that's why I like using the contingency table counts (CTC) line
in
MET much more than the rates (FHO) line.  The underlying data here is
an
integer number of counts falling into each cell of a 2x2 table.
Because of
that, I would argue that 32.5 is really more correct than 36.33.

What do you think?

Thanks,
John



On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
met_help at ucar.edu> wrote:

>
> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>        Queue: met_help
>      Subject: FBIAS problem
>        Owner: johnhg
>   Requestors: ying.lin at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
>
> This transaction appears to have no content
>

------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK  INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER      BASER_NCL  BASER_NCU
BASER_BCL BASER_BCU FMEAN      FMEAN_NCL  FMEAN_NCU  FMEAN_BCL
FMEAN_BCU ACC     ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS    FBIAS_BCL
FBIAS_BCU PODY    PODY_NCL PODY_NCU   PODY_BCL PODY_BCU PODN
PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD       POFD_NCL   POFD_NCU
POFD_BCL POFD_BCU FAR     FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI
CSI_NCL  CSI_NCU    CSI_BCL CSI_BCU GSS         GSS_BCL GSS_BCU HK
HK_NCL    HK_NCU  HK_BCL HK_BCU HSS         HSS_BCL HSS_BCU ODDS
ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU LODDS    LODDS_NCL LODDS_NCU
LODDS_BCL LODDS_BCU ORSS     ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL  EDS_NCU  EDS_BCL EDS_BCU SEDS     SEDS_NCL SEDS_NCU SEDS_BCL
SEDS_BCU EDI      EDI_NCL  EDI_NCU  EDI_BCL EDI_BCU SEDI     SEDI_NCL
SEDI_NCU SEDI_BCL SEDI_BCU BAGSS     BAGSS_BCL BAGSS_BCU
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >0.01       >0.01      NA
0.05  CTS       59670 0.43161    0.42764    0.43559           NA
NA 0.4129     0.40896    0.41686           NA        NA 0.82705
0.82399 0.83006      NA      NA  0.95667        NA        NA 0.77798
0.77462 0.78129          NA       NA 0.86431  0.86154  0.86704
NA       NA 0.13569    0.13296    0.13846          NA       NA 0.18678
0.18368 0.18993      NA      NA 0.66003  0.65622  0.66382         NA
NA  0.47673         NA      NA  0.64229     0.63556  0.64902     NA
NA  0.64565         NA      NA 22.32006 21.38554 23.29541       NA
NA  3.10549   3.06272   3.14826        NA        NA  0.91424  0.91073
0.91775       NA       NA  0.53989  0.53068  0.5491       NA      NA
0.58048  0.57103  0.58993       NA       NA  0.77668  0.76926  0.78411
NA      NA  0.79643  0.76964  0.78373       NA       NA  0.49023
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >0.10       >0.10      NA
0.05  CTS       59670 0.26481    0.26128    0.26836           NA
NA 0.25854    0.25504    0.26207           NA        NA 0.82651
0.82345 0.82953      NA      NA  0.97633        NA        NA 0.66059
0.65678 0.66438          NA       NA 0.88628  0.8837   0.8888
NA       NA 0.11372    0.1112     0.1163           NA       NA 0.32339
0.31965 0.32716      NA      NA 0.50207  0.49806  0.50608         NA
NA  0.3803          NA      NA  0.54687     0.53826  0.55548     NA
NA  0.55104         NA      NA 15.16779 14.51206 15.85314       NA
NA  2.71917   2.67498   2.76337        NA        NA  0.8763   0.87117
0.88143       NA       NA  0.52435  0.51458  0.53412      NA      NA
0.53809  0.52823  0.54795       NA       NA  0.67966  0.66971  0.6896
NA      NA  0.71748  0.67008  0.68923       NA       NA  0.38456
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >0.25       >0.25      NA
0.05  CTS       59670 0.14359    0.1408     0.14643           NA
NA 0.12903    0.12636    0.13174           NA        NA 0.85103
0.84815 0.85386      NA      NA  0.89858        NA        NA 0.43056
0.42659 0.43453          NA       NA 0.92153  0.91934  0.92366
NA       NA 0.078471   0.07634    0.080655         NA       NA 0.52085
0.51684 0.52485      NA      NA 0.29329  0.28965  0.29696         NA
NA  0.22519         NA      NA  0.35209     0.34076  0.36341     NA
NA  0.3676          NA      NA  8.87934  8.41631  9.36784       NA
NA  2.18373   2.13017   2.23728        NA        NA  0.79756  0.78781
0.8073        NA       NA  0.39451  0.38231  0.40671      NA      NA
0.43293  0.4204   0.44547       NA       NA  0.50251  0.489    0.51601
NA      NA  0.54153  0.48841  0.5166        NA       NA  0.23626
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >0.50       >0.50      NA
0.05  CTS       59670 0.065594   0.063636   0.067609          NA
NA 0.051014   0.049277   0.052808          NA        NA 0.9207  0.9185
0.92284      NA      NA  0.77772        NA        NA 0.28436  0.28076
0.288            NA       NA 0.96537  0.96387  0.9668        NA
NA 0.034633   0.033196   0.03613          NA       NA 0.63436 0.63049
0.63822      NA      NA 0.19042  0.18729  0.19359         NA      NA
0.16179         NA      NA  0.24973     0.23512  0.26435     NA     NA
0.27851         NA      NA 11.07602 10.19416 12.03417       NA
NA  2.40478   2.32181   2.48775        NA        NA  0.83438  0.82178
0.84699       NA       NA  0.36837  0.35129  0.38545      NA      NA
0.43151  0.41364  0.44937       NA       NA  0.45568  0.4377   0.47366
NA      NA  0.48189  0.43661  0.47475       NA       NA  0.18297
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >0.75       >0.75      NA
0.05  CTS       59670 0.032864   0.031463   0.034325          NA
NA 0.029864   0.028529   0.031261          NA        NA 0.9551
0.95341 0.95674      NA      NA  0.90872        NA        NA 0.27129
0.26774 0.27487          NA       NA 0.97834  0.97714  0.97948
NA       NA 0.02166    0.020523   0.02286          NA       NA 0.70146
0.69777 0.70512      NA      NA 0.16568  0.16272  0.16869         NA
NA  0.15018         NA      NA  0.24963     0.22963  0.26963     NA
NA  0.26114         NA      NA 16.81522 14.99995 18.85017       NA
NA  2.82228   2.70805   2.93652        NA        NA  0.88774  0.87563
0.89984       NA       NA  0.44721  0.42497  0.46945      NA      NA
0.46749  0.44494  0.49004       NA       NA  0.49207  0.46833  0.51581
NA      NA  0.51547  0.46705  0.5171        NA       NA  0.15705
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >1.00       >1.00      NA
0.05  CTS       59670 0.01629    0.015305   0.017337          NA
NA 0.016759   0.01576    0.01782           NA        NA 0.97369
0.97237 0.97494      NA      NA  1.02881        NA        NA 0.20679
0.20356 0.21006          NA       NA 0.98639  0.98543  0.98729
NA       NA 0.013612   0.012713   0.014574         NA       NA 0.799
0.79577 0.8022       NA      NA 0.1135   0.11097  0.11607         NA
NA  0.10527         NA      NA  0.19318     0.16753  0.21883     NA
NA  0.19048         NA      NA 18.89148 15.93487 22.39666       NA
NA  2.93871   2.76851   3.10891        NA        NA  0.89945  0.8832
0.91571       NA       NA  0.44635  0.41507  0.47763      NA      NA
0.44136  0.41019  0.47253       NA       NA  0.46328  0.42966  0.49689
NA      NA  0.48032  0.42793  0.49863       NA       NA  0.10436
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >1.50       >1.50      NA
0.05  CTS       59670 0.0039718  0.0034981  0.0045094         NA
NA 0.0082956  0.0075989  0.0090557         NA        NA 0.98948
0.98862 0.99026      NA      NA  2.08861        NA        NA 0.21941
0.21611 0.22275          NA       NA 0.99255  0.99182  0.99321
NA       NA 0.0074538  0.0067946  0.0081763        NA       NA 0.89495
0.89246 0.89738      NA      NA 0.076471 0.074365 0.07863         NA
NA  0.073793        NA      NA  0.21196     0.15939  0.26452     NA
NA  0.13744         NA      NA 37.42883 27.13762 51.62272       NA
NA  3.62244   3.30092   3.94396        NA        NA  0.94796  0.93166
0.96425       NA       NA  0.56941  0.51592  0.62291      NA      NA
0.46488  0.41495  0.51481       NA       NA  0.52717  0.46503  0.5893
NA      NA  0.54301  0.46226  0.59207       NA       NA  0.058829
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >2.00       >2.00      NA
0.05  CTS       59670 0.00067035 0.00049236 0.00091264        NA
NA 0.0041394  0.0036552  0.0046875         NA        NA 0.99536
0.99478 0.99587      NA      NA  6.175          NA        NA 0.125
0.12237 0.12768          NA       NA 0.99594  0.9954   0.99642
NA       NA 0.0040584  0.0035792  0.0046014        NA       NA 0.97976
0.9786  0.98086      NA      NA 0.01773  0.016702 0.018821        NA
NA  0.017153        NA      NA  0.12094     0.017644 0.22424     NA
NA  0.033728        NA      NA 35.05785 13.61934 90.24318       NA
NA  3.557     2.61149   4.50251        NA        NA  0.94453  0.89354
0.99552       NA       NA  0.55696  0.42097  0.69295      NA      NA
0.36302  0.24397  0.48208       NA       NA  0.4518   0.28643  0.61717
NA      NA  0.46051  0.28031  0.62329       NA       NA  0.010799
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >3.00       >3.00      NA
0.05  CTS       59670 3.3518e-05 9.1918e-06 0.00012221        NA
NA 0.0010893  0.00085483 0.001388          NA        NA 0.99888
0.99857 0.99912      NA      NA 32.5            NA        NA 0
0       6.4374e-05       NA       NA 0.99891  0.99861  0.99915
NA       NA 0.0010894  0.00085487 0.0013881        NA       NA 1
0.99994 1            NA      NA 0        0        6.4374e-05      NA
NA -3.2518e-05      NA      NA -0.0010894  -0.3299   0.32772     NA
NA -6.5039e-05      NA      NA  0       NA       NA             NA
NA NA        NA        NA              NA        NA -1       NA
NA             NA       NA NA       NA       NA            NA      NA
NA       NA       NA             NA       NA NA       NA       NA
NA      NA NA       NA       NA             NA       NA NA
NA        NA
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24      APCP    A24     CCPA
G218/RFC NEAREST     1           >4.00       >4.00      NA
0.05  CTS       59670 1.6759e-05 2.9584e-06 9.4931e-05        NA
NA 0.00010055 4.6085e-05 0.00021938        NA        NA 0.99988
0.99976 0.99994      NA      NA  6              NA        NA 0
0       6.4374e-05       NA       NA 0.9999   0.99978  0.99995
NA       NA 0.00010055 4.6086e-05 0.00021938       NA       NA 1
0.99994 1            NA      NA 0        0        6.4374e-05      NA
NA -1.4365e-05      NA      NA -0.00010055 -0.39683  0.39662     NA
NA -2.873e-05       NA      NA  0       NA       NA             NA
NA NA        NA        NA              NA        NA -1       NA
NA             NA       NA NA       NA       NA            NA      NA
NA       NA       NA             NA       NA NA       NA       NA
NA      NA NA       NA       NA             NA       NA NA
NA        NA

------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016

VERSION MODEL FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL F_RATE  H_RATE  O_RATE
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >0.01       >0.01      NA         NA
FHO       59670 0.41290 0.33578 0.43161
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >0.10       >0.10      NA         NA
FHO       59670 0.25854 0.17493 0.26481
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >0.25       >0.25      NA         NA
FHO       59670 0.12903 0.06182 0.14359
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >0.50       >0.50      NA         NA
FHO       59670 0.05101 0.01865 0.06559
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >0.75       >0.75      NA         NA
FHO       59670 0.02986 0.00892 0.03286
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >1.00       >1.00      NA         NA
FHO       59670 0.01676 0.00337 0.01629
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >1.50       >1.50      NA         NA
FHO       59670 0.00830 0.00087 0.00397
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >2.00       >2.00      NA         NA
FHO       59670 0.00414 0.00008 0.00067
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >3.00       >3.00      NA         NA
FHO       59670 0.00109 0.00000 0.00003
V5.2    NAM   240000    20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP     A24       APCP   A24     CCPA
G218/RFC  NEAREST     1         >4.00       >4.00      NA         NA
FHO       59670 0.00010 0.00000 0.00002

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77885] FBIAS problem
From: Ying Lin
Time: Tue Sep 13 12:51:32 2016

You're right John.  I believe the forecast point counts exceeding the
3"
and 4" thresholds are indeed 65 and 6, and obs points are 2 and 1, so
the numbers computed by MetViewer are indeed more accurate than that
from FVS.  It  would have been better if we had allocated more digits
after the decimal points, or, more straightforwardly, if FVS for FHO
were designed to read in the direct Fcst and Obs counts rather than
the
rates (Fcst/Total and Obs/Total), which seems to be an unnecessary
extra
step - the rates do not make the VSDB files any smaller, for one
thing!

Thank you for looking into this.

Ying

On 09/13/2016 02:30 PM, John Halley Gotway via RT wrote:
> Hi Ying,
>
> This is John again.  I thought I'd dig in a little on this one by
doing the
> following...
>
> (1) MET includes support for the FHO line type, not in VSDB format
but in
> the MET format, where the header columns differ.  So I reformatted
the
> vsdb.txt file you sent into that MET format (see attached
"met_fho.txt")nam_19920903.vsdb
> file.
>
> (2) I ran the following job through the MET STAT-Analysis tool to
read FHO
> lines and derived contingency table statistics (i.e. a CTS line):
>
> /usr/local/met-5.2/bin/stat_analysis \
>     -lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
> CTS \
>     -by FCST_THRESH -out_stat met_cts.txt
>
> I've attached the output file named met_cts.txt.  In that file,
FBIAS =
> 32.5 and 6.0 for thresholds >3.0 and >4.0, respectively.  So I'm
glad that
> MET and METViewer are consistent at least!
>
> For that >3.0 line, here's the math that's occurring in MET...
>
> Input:
>    TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003
>
> We convert the FHO rates into integer contingency table counts.
>From MET
> source code file "parse_stat_line.cc"...
>
> FY = round(TOTAL * F_RATE) = 65
> FY_OY = round(TOTAL * H_RATE) = 0
> OY = round(TOTAL * O_RATE) = 2
>
> FY_ON = FY - FY_OY = 65
> FN_OY = OY - FY_OY = 2
> FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603
>
> And with those counts, we compute...
> FBIAS = FY / OY = 65 / 2 = 32.5
>
> However, I suspect FVS is computing FBIAS as follows...
>
> FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE / O_RATE
=
> 0.00109 / 0.00003 = 36.33
>
> So we have two different answers... 32.5 and 36.33, and the question
is
> which is more correct?
>
> I would say the real problem here is using only 5 significant digits
for
> "rare" events (rare being 24-hour precip accumulations of 3"+).  If
we had
> 20 decimal places, or 5 significant figure in scientific notation,
we would
> not have any real difference in the results.
>
> Assuming the counts MET computed are correct, the real rates should
be...
>     F_RATE = 65 / 59670 = 0.001089324619
>     O_RATE = 2 / 59670 = 0.00003351768057
> And using those values...
>     FBIAS = 32.5
>
> Really that's why I like using the contingency table counts (CTC)
line in
> MET much more than the rates (FHO) line.  The underlying data here
is an
> integer number of counts falling into each cell of a 2x2 table.
Because of
> that, I would argue that 32.5 is really more correct than 36.33.
>
> What do you think?
>
> Thanks,
> John
>
>
>
> On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>>         Queue: met_help
>>       Subject: FBIAS problem
>>         Owner: johnhg
>>    Requestors: ying.lin at noaa.gov
>>        Status: new
>>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>>
>>
>> This transaction appears to have no content
>>


--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:57:14 2016

Ying,

Great, I'm glad we got this one straightened out!  Please do let us
know
what other discrepancies you find between FVS and METViewer.  That
really
helps check out the METViewer logic and hopefully builds confidence in
its
use going forward.

I'll resolve this ticket now.

Thanks,
John

On Tue, Sep 13, 2016 at 12:51 PM, Ying Lin via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
> You're right John.  I believe the forecast point counts exceeding
the 3"
> and 4" thresholds are indeed 65 and 6, and obs points are 2 and 1,
so
> the numbers computed by MetViewer are indeed more accurate than that
> from FVS.  It  would have been better if we had allocated more
digits
> after the decimal points, or, more straightforwardly, if FVS for FHO
> were designed to read in the direct Fcst and Obs counts rather than
the
> rates (Fcst/Total and Obs/Total), which seems to be an unnecessary
extra
> step - the rates do not make the VSDB files any smaller, for one
thing!
>
> Thank you for looking into this.
>
> Ying
>
> On 09/13/2016 02:30 PM, John Halley Gotway via RT wrote:
> > Hi Ying,
> >
> > This is John again.  I thought I'd dig in a little on this one by
doing
> the
> > following...
> >
> > (1) MET includes support for the FHO line type, not in VSDB format
but in
> > the MET format, where the header columns differ.  So I reformatted
the
> > vsdb.txt file you sent into that MET format (see attached
> "met_fho.txt")nam_19920903.vsdb
> > file.
> >
> > (2) I ran the following job through the MET STAT-Analysis tool to
read
> FHO
> > lines and derived contingency table statistics (i.e. a CTS line):
> >
> > /usr/local/met-5.2/bin/stat_analysis \
> >     -lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
> > CTS \
> >     -by FCST_THRESH -out_stat met_cts.txt
> >
> > I've attached the output file named met_cts.txt.  In that file,
FBIAS =
> > 32.5 and 6.0 for thresholds >3.0 and >4.0, respectively.  So I'm
glad
> that
> > MET and METViewer are consistent at least!
> >
> > For that >3.0 line, here's the math that's occurring in MET...
> >
> > Input:
> >    TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003
> >
> > We convert the FHO rates into integer contingency table counts.
>From MET
> > source code file "parse_stat_line.cc"...
> >
> > FY = round(TOTAL * F_RATE) = 65
> > FY_OY = round(TOTAL * H_RATE) = 0
> > OY = round(TOTAL * O_RATE) = 2
> >
> > FY_ON = FY - FY_OY = 65
> > FN_OY = OY - FY_OY = 2
> > FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603
> >
> > And with those counts, we compute...
> > FBIAS = FY / OY = 65 / 2 = 32.5
> >
> > However, I suspect FVS is computing FBIAS as follows...
> >
> > FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE /
O_RATE =
> > 0.00109 / 0.00003 = 36.33
> >
> > So we have two different answers... 32.5 and 36.33, and the
question is
> > which is more correct?
> >
> > I would say the real problem here is using only 5 significant
digits for
> > "rare" events (rare being 24-hour precip accumulations of 3"+).
If we
> had
> > 20 decimal places, or 5 significant figure in scientific notation,
we
> would
> > not have any real difference in the results.
> >
> > Assuming the counts MET computed are correct, the real rates
should be...
> >     F_RATE = 65 / 59670 = 0.001089324619
> >     O_RATE = 2 / 59670 = 0.00003351768057
> > And using those values...
> >     FBIAS = 32.5
> >
> > Really that's why I like using the contingency table counts (CTC)
line in
> > MET much more than the rates (FHO) line.  The underlying data here
is an
> > integer number of counts falling into each cell of a 2x2 table.
Because
> of
> > that, I would argue that 32.5 is really more correct than 36.33.
> >
> > What do you think?
> >
> > Thanks,
> > John
> >
> >
> >
> > On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >>         Queue: met_help
> >>       Subject: FBIAS problem
> >>         Owner: johnhg
> >>    Requestors: ying.lin at noaa.gov
> >>        Status: new
> >>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885
> >
> >>
> >>
> >> This transaction appears to have no content
> >>
>
>
> --
> Ying Lin
> NCEP/EMC/Mesoscale Modeling Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>

------------------------------------------------


More information about the Met_help mailing list