[Met_help] [rt.rap.ucar.edu #77885] History for FBIAS problem
John Halley Gotway via RT
met_help at ucar.edu
Tue Sep 13 14:10:27 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. Running into a problem with FBIAS: see attached xml. Comparing
the FBIAS numbers listed in the R data with FVS results, they match up
well except at 3" and 4" thresholds:
threshold Bias from FVS From R data
3"/day 36.333 32.500
4"/day 5.000 6.000
Also attached is the VSDB records for all thresholds for this particular
verification, which is consistent with the FVS results (detailed
calculations omitted since this is simply a ratio of fcst/obs counts).
Could you look into this? The discrepancy above isn't huge, but I'm
trying to compare the monthly scores in
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/201607/ncep_eqts.201607.gif
against that from the MetViewer, and over a month the differences in
FBIAS at 3" are huge (GSS matched up well).
Thanks,
Ying
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 09:50:36 2016
Hi Ying,
Thanks for doing a careful comparison between FVS and METViewer. I
think
that is exactly what is needed!
However, Tatiana is on travel until Sept 23rd but will return on the
26th.
Can this issue wait until then?
Thanks,
John Halley Gotway
On Tue, Sep 13, 2016 at 9:41 AM, Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Tue Sep 13 09:41:02 2016: Request 77885 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: FBIAS problem
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
>
> Hello. Running into a problem with FBIAS: see attached xml.
Comparing
> the FBIAS numbers listed in the R data with FVS results, they match
up
> well except at 3" and 4" thresholds:
>
> threshold Bias from FVS From R data
>
> 3"/day 36.333 32.500
>
> 4"/day 5.000 6.000
>
> Also attached is the VSDB records for all thresholds for this
particular
> verification, which is consistent with the FVS results (detailed
> calculations omitted since this is simply a ratio of fcst/obs
counts).
> Could you look into this? The discrepancy above isn't huge, but I'm
> trying to compare the monthly scores in
> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/
> 201607/ncep_eqts.201607.gif
> against that from the MetViewer, and over a month the differences in
> FBIAS at 3" are huge (GSS matched up well).
>
> Thanks,
>
> Ying
>
> --
> Ying Lin
> NCEP/EMC/Mesoscale Modeling Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77885] FBIAS problem
From: Ying Lin
Time: Tue Sep 13 12:04:32 2016
Hi John, yes it can wait. Thanks for letting me know of Tatiana's
schedule.
Would like to ask to have fractions skill scores (24h and 6h) added to
the MetViewer from the VSDBs - not sure how to request this. I'll ask
at the next DTC/EMC MET telecon.
Thanks,
Ying
On 09/13/2016 11:50 AM, John Halley Gotway via RT wrote:
> Hi Ying,
>
> Thanks for doing a careful comparison between FVS and METViewer. I
think
> that is exactly what is needed!
>
> However, Tatiana is on travel until Sept 23rd but will return on the
26th.
> Can this issue wait until then?
>
> Thanks,
> John Halley Gotway
>
> On Tue, Sep 13, 2016 at 9:41 AM, Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Tue Sep 13 09:41:02 2016: Request 77885 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: FBIAS problem
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>>
>>
>> Hello. Running into a problem with FBIAS: see attached xml.
Comparing
>> the FBIAS numbers listed in the R data with FVS results, they match
up
>> well except at 3" and 4" thresholds:
>>
>> threshold Bias from FVS From R data
>>
>> 3"/day 36.333 32.500
>>
>> 4"/day 5.000 6.000
>>
>> Also attached is the VSDB records for all thresholds for this
particular
>> verification, which is consistent with the FVS results (detailed
>> calculations omitted since this is simply a ratio of fcst/obs
counts).
>> Could you look into this? The discrepancy above isn't huge, but
I'm
>> trying to compare the monthly scores in
>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2016/
>> 201607/ncep_eqts.201607.gif
>> against that from the MetViewer, and over a month the differences
in
>> FBIAS at 3" are huge (GSS matched up well).
>>
>> Thanks,
>>
>> Ying
>>
>> --
>> Ying Lin
>> NCEP/EMC/Mesoscale Modeling Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016
Hi Ying,
This is John again. I thought I'd dig in a little on this one by
doing the
following...
(1) MET includes support for the FHO line type, not in VSDB format but
in
the MET format, where the header columns differ. So I reformatted the
vsdb.txt file you sent into that MET format (see attached
"met_fho.txt")
file.
(2) I ran the following job through the MET STAT-Analysis tool to read
FHO
lines and derived contingency table statistics (i.e. a CTS line):
/usr/local/met-5.2/bin/stat_analysis \
-lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
CTS \
-by FCST_THRESH -out_stat met_cts.txt
I've attached the output file named met_cts.txt. In that file, FBIAS
=
32.5 and 6.0 for thresholds >3.0 and >4.0, respectively. So I'm glad
that
MET and METViewer are consistent at least!
For that >3.0 line, here's the math that's occurring in MET...
Input:
TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003
We convert the FHO rates into integer contingency table counts. From
MET
source code file "parse_stat_line.cc"...
FY = round(TOTAL * F_RATE) = 65
FY_OY = round(TOTAL * H_RATE) = 0
OY = round(TOTAL * O_RATE) = 2
FY_ON = FY - FY_OY = 65
FN_OY = OY - FY_OY = 2
FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603
And with those counts, we compute...
FBIAS = FY / OY = 65 / 2 = 32.5
However, I suspect FVS is computing FBIAS as follows...
FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE / O_RATE =
0.00109 / 0.00003 = 36.33
So we have two different answers... 32.5 and 36.33, and the question
is
which is more correct?
I would say the real problem here is using only 5 significant digits
for
"rare" events (rare being 24-hour precip accumulations of 3"+). If we
had
20 decimal places, or 5 significant figure in scientific notation, we
would
not have any real difference in the results.
Assuming the counts MET computed are correct, the real rates should
be...
F_RATE = 65 / 59670 = 0.001089324619
O_RATE = 2 / 59670 = 0.00003351768057
And using those values...
FBIAS = 32.5
Really that's why I like using the contingency table counts (CTC) line
in
MET much more than the rates (FHO) line. The underlying data here is
an
integer number of counts falling into each cell of a 2x2 table.
Because of
that, I would argue that 32.5 is really more correct than 36.33.
What do you think?
Thanks,
John
On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: FBIAS problem
> Owner: johnhg
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL
FMEAN_BCU ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL
FBIAS_BCU PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN
PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU
POFD_BCL POFD_BCU FAR FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI
CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS GSS_BCL GSS_BCU HK
HK_NCL HK_NCU HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS
ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU LODDS LODDS_NCL LODDS_NCU
LODDS_BCL LODDS_BCU ORSS ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS
EDS_NCL EDS_NCU EDS_BCL EDS_BCU SEDS SEDS_NCL SEDS_NCU SEDS_BCL
SEDS_BCU EDI EDI_NCL EDI_NCU EDI_BCL EDI_BCU SEDI SEDI_NCL
SEDI_NCU SEDI_BCL SEDI_BCU BAGSS BAGSS_BCL BAGSS_BCU
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.01 >0.01 NA
0.05 CTS 59670 0.43161 0.42764 0.43559 NA
NA 0.4129 0.40896 0.41686 NA NA 0.82705
0.82399 0.83006 NA NA 0.95667 NA NA 0.77798
0.77462 0.78129 NA NA 0.86431 0.86154 0.86704
NA NA 0.13569 0.13296 0.13846 NA NA 0.18678
0.18368 0.18993 NA NA 0.66003 0.65622 0.66382 NA
NA 0.47673 NA NA 0.64229 0.63556 0.64902 NA
NA 0.64565 NA NA 22.32006 21.38554 23.29541 NA
NA 3.10549 3.06272 3.14826 NA NA 0.91424 0.91073
0.91775 NA NA 0.53989 0.53068 0.5491 NA NA
0.58048 0.57103 0.58993 NA NA 0.77668 0.76926 0.78411
NA NA 0.79643 0.76964 0.78373 NA NA 0.49023
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.10 >0.10 NA
0.05 CTS 59670 0.26481 0.26128 0.26836 NA
NA 0.25854 0.25504 0.26207 NA NA 0.82651
0.82345 0.82953 NA NA 0.97633 NA NA 0.66059
0.65678 0.66438 NA NA 0.88628 0.8837 0.8888
NA NA 0.11372 0.1112 0.1163 NA NA 0.32339
0.31965 0.32716 NA NA 0.50207 0.49806 0.50608 NA
NA 0.3803 NA NA 0.54687 0.53826 0.55548 NA
NA 0.55104 NA NA 15.16779 14.51206 15.85314 NA
NA 2.71917 2.67498 2.76337 NA NA 0.8763 0.87117
0.88143 NA NA 0.52435 0.51458 0.53412 NA NA
0.53809 0.52823 0.54795 NA NA 0.67966 0.66971 0.6896
NA NA 0.71748 0.67008 0.68923 NA NA 0.38456
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.25 >0.25 NA
0.05 CTS 59670 0.14359 0.1408 0.14643 NA
NA 0.12903 0.12636 0.13174 NA NA 0.85103
0.84815 0.85386 NA NA 0.89858 NA NA 0.43056
0.42659 0.43453 NA NA 0.92153 0.91934 0.92366
NA NA 0.078471 0.07634 0.080655 NA NA 0.52085
0.51684 0.52485 NA NA 0.29329 0.28965 0.29696 NA
NA 0.22519 NA NA 0.35209 0.34076 0.36341 NA
NA 0.3676 NA NA 8.87934 8.41631 9.36784 NA
NA 2.18373 2.13017 2.23728 NA NA 0.79756 0.78781
0.8073 NA NA 0.39451 0.38231 0.40671 NA NA
0.43293 0.4204 0.44547 NA NA 0.50251 0.489 0.51601
NA NA 0.54153 0.48841 0.5166 NA NA 0.23626
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.50 >0.50 NA
0.05 CTS 59670 0.065594 0.063636 0.067609 NA
NA 0.051014 0.049277 0.052808 NA NA 0.9207 0.9185
0.92284 NA NA 0.77772 NA NA 0.28436 0.28076
0.288 NA NA 0.96537 0.96387 0.9668 NA
NA 0.034633 0.033196 0.03613 NA NA 0.63436 0.63049
0.63822 NA NA 0.19042 0.18729 0.19359 NA NA
0.16179 NA NA 0.24973 0.23512 0.26435 NA NA
0.27851 NA NA 11.07602 10.19416 12.03417 NA
NA 2.40478 2.32181 2.48775 NA NA 0.83438 0.82178
0.84699 NA NA 0.36837 0.35129 0.38545 NA NA
0.43151 0.41364 0.44937 NA NA 0.45568 0.4377 0.47366
NA NA 0.48189 0.43661 0.47475 NA NA 0.18297
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.75 >0.75 NA
0.05 CTS 59670 0.032864 0.031463 0.034325 NA
NA 0.029864 0.028529 0.031261 NA NA 0.9551
0.95341 0.95674 NA NA 0.90872 NA NA 0.27129
0.26774 0.27487 NA NA 0.97834 0.97714 0.97948
NA NA 0.02166 0.020523 0.02286 NA NA 0.70146
0.69777 0.70512 NA NA 0.16568 0.16272 0.16869 NA
NA 0.15018 NA NA 0.24963 0.22963 0.26963 NA
NA 0.26114 NA NA 16.81522 14.99995 18.85017 NA
NA 2.82228 2.70805 2.93652 NA NA 0.88774 0.87563
0.89984 NA NA 0.44721 0.42497 0.46945 NA NA
0.46749 0.44494 0.49004 NA NA 0.49207 0.46833 0.51581
NA NA 0.51547 0.46705 0.5171 NA NA 0.15705
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >1.00 >1.00 NA
0.05 CTS 59670 0.01629 0.015305 0.017337 NA
NA 0.016759 0.01576 0.01782 NA NA 0.97369
0.97237 0.97494 NA NA 1.02881 NA NA 0.20679
0.20356 0.21006 NA NA 0.98639 0.98543 0.98729
NA NA 0.013612 0.012713 0.014574 NA NA 0.799
0.79577 0.8022 NA NA 0.1135 0.11097 0.11607 NA
NA 0.10527 NA NA 0.19318 0.16753 0.21883 NA
NA 0.19048 NA NA 18.89148 15.93487 22.39666 NA
NA 2.93871 2.76851 3.10891 NA NA 0.89945 0.8832
0.91571 NA NA 0.44635 0.41507 0.47763 NA NA
0.44136 0.41019 0.47253 NA NA 0.46328 0.42966 0.49689
NA NA 0.48032 0.42793 0.49863 NA NA 0.10436
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >1.50 >1.50 NA
0.05 CTS 59670 0.0039718 0.0034981 0.0045094 NA
NA 0.0082956 0.0075989 0.0090557 NA NA 0.98948
0.98862 0.99026 NA NA 2.08861 NA NA 0.21941
0.21611 0.22275 NA NA 0.99255 0.99182 0.99321
NA NA 0.0074538 0.0067946 0.0081763 NA NA 0.89495
0.89246 0.89738 NA NA 0.076471 0.074365 0.07863 NA
NA 0.073793 NA NA 0.21196 0.15939 0.26452 NA
NA 0.13744 NA NA 37.42883 27.13762 51.62272 NA
NA 3.62244 3.30092 3.94396 NA NA 0.94796 0.93166
0.96425 NA NA 0.56941 0.51592 0.62291 NA NA
0.46488 0.41495 0.51481 NA NA 0.52717 0.46503 0.5893
NA NA 0.54301 0.46226 0.59207 NA NA 0.058829
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >2.00 >2.00 NA
0.05 CTS 59670 0.00067035 0.00049236 0.00091264 NA
NA 0.0041394 0.0036552 0.0046875 NA NA 0.99536
0.99478 0.99587 NA NA 6.175 NA NA 0.125
0.12237 0.12768 NA NA 0.99594 0.9954 0.99642
NA NA 0.0040584 0.0035792 0.0046014 NA NA 0.97976
0.9786 0.98086 NA NA 0.01773 0.016702 0.018821 NA
NA 0.017153 NA NA 0.12094 0.017644 0.22424 NA
NA 0.033728 NA NA 35.05785 13.61934 90.24318 NA
NA 3.557 2.61149 4.50251 NA NA 0.94453 0.89354
0.99552 NA NA 0.55696 0.42097 0.69295 NA NA
0.36302 0.24397 0.48208 NA NA 0.4518 0.28643 0.61717
NA NA 0.46051 0.28031 0.62329 NA NA 0.010799
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >3.00 >3.00 NA
0.05 CTS 59670 3.3518e-05 9.1918e-06 0.00012221 NA
NA 0.0010893 0.00085483 0.001388 NA NA 0.99888
0.99857 0.99912 NA NA 32.5 NA NA 0
0 6.4374e-05 NA NA 0.99891 0.99861 0.99915
NA NA 0.0010894 0.00085487 0.0013881 NA NA 1
0.99994 1 NA NA 0 0 6.4374e-05 NA
NA -3.2518e-05 NA NA -0.0010894 -0.3299 0.32772 NA
NA -6.5039e-05 NA NA 0 NA NA NA
NA NA NA NA NA NA -1 NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >4.00 >4.00 NA
0.05 CTS 59670 1.6759e-05 2.9584e-06 9.4931e-05 NA
NA 0.00010055 4.6085e-05 0.00021938 NA NA 0.99988
0.99976 0.99994 NA NA 6 NA NA 0
0 6.4374e-05 NA NA 0.9999 0.99978 0.99995
NA NA 0.00010055 4.6086e-05 0.00021938 NA NA 1
0.99994 1 NA NA 0 0 6.4374e-05 NA
NA -1.4365e-05 NA NA -0.00010055 -0.39683 0.39662 NA
NA -2.873e-05 NA NA 0 NA NA NA
NA NA NA NA NA NA -1 NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA
NA NA
------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:30:08 2016
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL F_RATE H_RATE O_RATE
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.01 >0.01 NA NA
FHO 59670 0.41290 0.33578 0.43161
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.10 >0.10 NA NA
FHO 59670 0.25854 0.17493 0.26481
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.25 >0.25 NA NA
FHO 59670 0.12903 0.06182 0.14359
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.50 >0.50 NA NA
FHO 59670 0.05101 0.01865 0.06559
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >0.75 >0.75 NA NA
FHO 59670 0.02986 0.00892 0.03286
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >1.00 >1.00 NA NA
FHO 59670 0.01676 0.00337 0.01629
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >1.50 >1.50 NA NA
FHO 59670 0.00830 0.00087 0.00397
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >2.00 >2.00 NA NA
FHO 59670 0.00414 0.00008 0.00067
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >3.00 >3.00 NA NA
FHO 59670 0.00109 0.00000 0.00003
V5.2 NAM 240000 20160709_120000 20160709_120000 000000
20160709_120000 20160709_120000 APCP A24 APCP A24 CCPA
G218/RFC NEAREST 1 >4.00 >4.00 NA NA
FHO 59670 0.00010 0.00000 0.00002
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77885] FBIAS problem
From: Ying Lin
Time: Tue Sep 13 12:51:32 2016
You're right John. I believe the forecast point counts exceeding the
3"
and 4" thresholds are indeed 65 and 6, and obs points are 2 and 1, so
the numbers computed by MetViewer are indeed more accurate than that
from FVS. It would have been better if we had allocated more digits
after the decimal points, or, more straightforwardly, if FVS for FHO
were designed to read in the direct Fcst and Obs counts rather than
the
rates (Fcst/Total and Obs/Total), which seems to be an unnecessary
extra
step - the rates do not make the VSDB files any smaller, for one
thing!
Thank you for looking into this.
Ying
On 09/13/2016 02:30 PM, John Halley Gotway via RT wrote:
> Hi Ying,
>
> This is John again. I thought I'd dig in a little on this one by
doing the
> following...
>
> (1) MET includes support for the FHO line type, not in VSDB format
but in
> the MET format, where the header columns differ. So I reformatted
the
> vsdb.txt file you sent into that MET format (see attached
"met_fho.txt")nam_19920903.vsdb
> file.
>
> (2) I ran the following job through the MET STAT-Analysis tool to
read FHO
> lines and derived contingency table statistics (i.e. a CTS line):
>
> /usr/local/met-5.2/bin/stat_analysis \
> -lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
> CTS \
> -by FCST_THRESH -out_stat met_cts.txt
>
> I've attached the output file named met_cts.txt. In that file,
FBIAS =
> 32.5 and 6.0 for thresholds >3.0 and >4.0, respectively. So I'm
glad that
> MET and METViewer are consistent at least!
>
> For that >3.0 line, here's the math that's occurring in MET...
>
> Input:
> TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003
>
> We convert the FHO rates into integer contingency table counts.
>From MET
> source code file "parse_stat_line.cc"...
>
> FY = round(TOTAL * F_RATE) = 65
> FY_OY = round(TOTAL * H_RATE) = 0
> OY = round(TOTAL * O_RATE) = 2
>
> FY_ON = FY - FY_OY = 65
> FN_OY = OY - FY_OY = 2
> FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603
>
> And with those counts, we compute...
> FBIAS = FY / OY = 65 / 2 = 32.5
>
> However, I suspect FVS is computing FBIAS as follows...
>
> FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE / O_RATE
=
> 0.00109 / 0.00003 = 36.33
>
> So we have two different answers... 32.5 and 36.33, and the question
is
> which is more correct?
>
> I would say the real problem here is using only 5 significant digits
for
> "rare" events (rare being 24-hour precip accumulations of 3"+). If
we had
> 20 decimal places, or 5 significant figure in scientific notation,
we would
> not have any real difference in the results.
>
> Assuming the counts MET computed are correct, the real rates should
be...
> F_RATE = 65 / 59670 = 0.001089324619
> O_RATE = 2 / 59670 = 0.00003351768057
> And using those values...
> FBIAS = 32.5
>
> Really that's why I like using the contingency table counts (CTC)
line in
> MET much more than the rates (FHO) line. The underlying data here
is an
> integer number of counts falling into each cell of a 2x2 table.
Because of
> that, I would argue that 32.5 is really more correct than 36.33.
>
> What do you think?
>
> Thanks,
> John
>
>
>
> On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: FBIAS problem
>> Owner: johnhg
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>>
>>
>> This transaction appears to have no content
>>
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: FBIAS problem
From: John Halley Gotway
Time: Tue Sep 13 12:57:14 2016
Ying,
Great, I'm glad we got this one straightened out! Please do let us
know
what other discrepancies you find between FVS and METViewer. That
really
helps check out the METViewer logic and hopefully builds confidence in
its
use going forward.
I'll resolve this ticket now.
Thanks,
John
On Tue, Sep 13, 2016 at 12:51 PM, Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885 >
>
> You're right John. I believe the forecast point counts exceeding
the 3"
> and 4" thresholds are indeed 65 and 6, and obs points are 2 and 1,
so
> the numbers computed by MetViewer are indeed more accurate than that
> from FVS. It would have been better if we had allocated more
digits
> after the decimal points, or, more straightforwardly, if FVS for FHO
> were designed to read in the direct Fcst and Obs counts rather than
the
> rates (Fcst/Total and Obs/Total), which seems to be an unnecessary
extra
> step - the rates do not make the VSDB files any smaller, for one
thing!
>
> Thank you for looking into this.
>
> Ying
>
> On 09/13/2016 02:30 PM, John Halley Gotway via RT wrote:
> > Hi Ying,
> >
> > This is John again. I thought I'd dig in a little on this one by
doing
> the
> > following...
> >
> > (1) MET includes support for the FHO line type, not in VSDB format
but in
> > the MET format, where the header columns differ. So I reformatted
the
> > vsdb.txt file you sent into that MET format (see attached
> "met_fho.txt")nam_19920903.vsdb
> > file.
> >
> > (2) I ran the following job through the MET STAT-Analysis tool to
read
> FHO
> > lines and derived contingency table statistics (i.e. a CTS line):
> >
> > /usr/local/met-5.2/bin/stat_analysis \
> > -lookin met_fho.txt -job aggregate_stat -line_type FHO
-out_line_type
> > CTS \
> > -by FCST_THRESH -out_stat met_cts.txt
> >
> > I've attached the output file named met_cts.txt. In that file,
FBIAS =
> > 32.5 and 6.0 for thresholds >3.0 and >4.0, respectively. So I'm
glad
> that
> > MET and METViewer are consistent at least!
> >
> > For that >3.0 line, here's the math that's occurring in MET...
> >
> > Input:
> > TOTAL = 59670, F_RATE = 0.00109, H_RATE = 0, O_RATE = 0.00003
> >
> > We convert the FHO rates into integer contingency table counts.
>From MET
> > source code file "parse_stat_line.cc"...
> >
> > FY = round(TOTAL * F_RATE) = 65
> > FY_OY = round(TOTAL * H_RATE) = 0
> > OY = round(TOTAL * O_RATE) = 2
> >
> > FY_ON = FY - FY_OY = 65
> > FN_OY = OY - FY_OY = 2
> > FN_ON = TOTAL - FY_OY - FY_ON - FN_OY = 59603
> >
> > And with those counts, we compute...
> > FBIAS = FY / OY = 65 / 2 = 32.5
> >
> > However, I suspect FVS is computing FBIAS as follows...
> >
> > FBIAS = FY / OY = (F_RATE*TOTAL) / (O_RATE*TOTAL) = F_RATE /
O_RATE =
> > 0.00109 / 0.00003 = 36.33
> >
> > So we have two different answers... 32.5 and 36.33, and the
question is
> > which is more correct?
> >
> > I would say the real problem here is using only 5 significant
digits for
> > "rare" events (rare being 24-hour precip accumulations of 3"+).
If we
> had
> > 20 decimal places, or 5 significant figure in scientific notation,
we
> would
> > not have any real difference in the results.
> >
> > Assuming the counts MET computed are correct, the real rates
should be...
> > F_RATE = 65 / 59670 = 0.001089324619
> > O_RATE = 2 / 59670 = 0.00003351768057
> > And using those values...
> > FBIAS = 32.5
> >
> > Really that's why I like using the contingency table counts (CTC)
line in
> > MET much more than the rates (FHO) line. The underlying data here
is an
> > integer number of counts falling into each cell of a 2x2 table.
Because
> of
> > that, I would argue that 32.5 is really more correct than 36.33.
> >
> > What do you think?
> >
> > Thanks,
> > John
> >
> >
> >
> > On Tue, Sep 13, 2016 at 9:50 AM, The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Tue Sep 13 09:50:36 2016: Request 77885 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >> Queue: met_help
> >> Subject: FBIAS problem
> >> Owner: johnhg
> >> Requestors: ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77885
> >
> >>
> >>
> >> This transaction appears to have no content
> >>
>
>
> --
> Ying Lin
> NCEP/EMC/Mesoscale Modeling Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list