[Met_help] [rt.rap.ucar.edu #77783] History for GSS computation when there are few data points
John Halley Gotway via RT
met_help at ucar.edu
Wed Jul 10 16:43:25 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
[Don't mean to ask you to look into this before the holiday weekend -
after Labor Day is fine. Just want to get this out when all the details
are fresh on my mind.]
Another issue with GSS computation - when I tried plotting the GSS on
the 12km grid 218, the scores look quite wrong - see attached xml, png,
and the scores (top panel, ETS=GSS) plotted from FVS. The R data window
shows that the scores computed for MetViewer for most thresholds
actually look to be what's on the FVS plot, only the scores for the
3.0"/day threshold is far off. The FVS plot shows that for this
threshold, there are only 3 data points where the verifying analysis has
values >=3"/day. Could this have blown the calculation for MetViewer?
Scores for this day for FBIAS/CSI/FAR on g218 look fine. All three of
them have much simpler formulas than GSS.
Thanks,
Ying
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: GSS computation when there are few data points
From: Tatiana Burek
Time: Tue Sep 06 14:10:22 2016
Ying,
Looking at the data in database and simplifying filtering to
obs_thresh ='>3.0'
there is only 1 records in the database for each model and threshold.
For example,
for NAM and obs_thresh='>3.0':
| model | fcst_init_beg | fcst_valid_beg | fcst_lead |
obs_thresh | fcst_var | vx_mask | stat_name | stat_value |
stat_ncl | stat_ncu | stat_bcl | stat_bcu |
| NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24 | >3.0
| APCP/24 | G218/RFC | 59670 | 0 | 34 | 3 | 59633 |
Using the formula:
c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
GSS = -4.6202
Tatiana
On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
> [Don't mean to ask you to look into this before the holiday weekend
-
> after Labor Day is fine. Just want to get this out when all the
details
> are fresh on my mind.]
>
> Another issue with GSS computation - when I tried plotting the GSS
on
> the 12km grid 218, the scores look quite wrong - see attached xml,
png,
> and the scores (top panel, ETS=GSS) plotted from FVS. The R data
window
> shows that the scores computed for MetViewer for most thresholds
> actually look to be what's on the FVS plot, only the scores for the
> 3.0"/day threshold is far off. The FVS plot shows that for this
> threshold, there are only 3 data points where the verifying analysis
has
> values >=3"/day. Could this have blown the calculation for
MetViewer?
>
> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All three
of
> them have much simpler formulas than GSS.
>
> Thanks,
>
> Ying
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77783] GSS computation when there are few data points
From: Ying Lin
Time: Fri Sep 09 09:42:14 2016
Hi Tatiana,
I plugged in the numbers for NAM at 3" and got GSS=-4.6202E-05,
which
is exactly 5 orders of magnitude smaller than the MET calculation.
Here's the computation, for thresholds of 3" and 1". Curious that the
GSS at 1"/day is quite close to the MET result:
At 3"/day: Fcst=34, Hit=0, Obs=3, Total=59670
Chance =(Fcst*Obs)/Total
=0.0017094
GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
= (-0.0017094)/(34+3-0.0017094)
= 0.000046202
At 1"/day: Fcst=1267, Hit=115, Obs=652
Chance = (Fcst*Obs)/Total
= 13.8442
GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
= (115-13.8442)/(1267+652-115-13.8442)
= 0.05651
The FVS definitions above compared to that in your formula:
Fcst=fy+fn
Obs=oy+on
Hit=fy_oy
VSDB records for NAM 24h forecast, 24h accumulation ending at 12Z 1
Jul,
over the whole ConUS:
V01 NAM 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
0.50241
0.39447 0.47419
V01 NAM 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
0.32717
0.20198 0.28927
V01 NAM 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
0.18653
0.07300 0.14465
V01 NAM 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
0.07716
0.01438 0.05267
V01 NAM 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
0.03883
0.00439 0.02283
V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
0.02123
0.00194 0.01093
V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
0.00605
0.00039 0.00313
V01 NAM 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
0.00243
0.00002 0.00057
V01 NAM 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
0.00057
0.00000 0.00005
V01 NAM 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
0.00010
0.00000 0.00000
Could you double check the calculations for 3" and 1" on your end? I
double-checked my formula and yours against that in equation (7.18) in
the Wilks book (2nd ed) and don't see a problem.
Ying
On 09/06/2016 04:10 PM, Tatiana Burek via RT wrote:
> Ying,
> Looking at the data in database and simplifying filtering to
obs_thresh ='>3.0'
> there is only 1 records in the database for each model and
threshold. For example,
> for NAM and obs_thresh='>3.0':
> | model | fcst_init_beg | fcst_valid_beg | fcst_lead |
obs_thresh | fcst_var | vx_mask | stat_name | stat_value |
stat_ncl | stat_ncu | stat_bcl | stat_bcu |
>
> | NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24 |
>3.0 | APCP/24 | G218/RFC | 59670 | 0 | 34 | 3 |
59633 |
> Using the formula:
> c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
> gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
> GSS = -4.6202
>
> Tatiana
>
> On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
>> [Don't mean to ask you to look into this before the holiday weekend
-
>> after Labor Day is fine. Just want to get this out when all the
details
>> are fresh on my mind.]
>>
>> Another issue with GSS computation - when I tried plotting the GSS
on
>> the 12km grid 218, the scores look quite wrong - see attached xml,
png,
>> and the scores (top panel, ETS=GSS) plotted from FVS. The R data
window
>> shows that the scores computed for MetViewer for most thresholds
>> actually look to be what's on the FVS plot, only the scores for the
>> 3.0"/day threshold is far off. The FVS plot shows that for this
>> threshold, there are only 3 data points where the verifying
analysis has
>> values >=3"/day. Could this have blown the calculation for
MetViewer?
>>
>> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All three
of
>> them have much simpler formulas than GSS.
>>
>> Thanks,
>>
>> Ying
>>
>
>
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: GSS computation when there are few data points
From: Tatiana Burek
Time: Fri Sep 09 13:42:13 2016
Ying,
I checked our calculations and they are correct.
The problem was in the MySQL functions for calculating stats where we
used REAL datatype to store intermediate results. Somehow this
datatype does not working well with small numbers like 0.0000462
(4.62e-05) and returned 4 instead of the actual value. The fix is
available for GSS calculation in mv_pcp_jul database.
Tatiana
On Fri Sep 09 09:42:14 2016, ying.lin at noaa.gov wrote:
> Hi Tatiana,
>
> I plugged in the numbers for NAM at 3" and got GSS=-4.6202E-05,
which
> is exactly 5 orders of magnitude smaller than the MET calculation.
> Here's the computation, for thresholds of 3" and 1". Curious that
the
> GSS at 1"/day is quite close to the MET result:
>
> At 3"/day: Fcst=34, Hit=0, Obs=3, Total=59670
> Chance =(Fcst*Obs)/Total
> =0.0017094
> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
> = (-0.0017094)/(34+3-0.0017094)
> = 0.000046202
>
> At 1"/day: Fcst=1267, Hit=115, Obs=652
> Chance = (Fcst*Obs)/Total
> = 13.8442
> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
> = (115-13.8442)/(1267+652-115-13.8442)
> = 0.05651
>
> The FVS definitions above compared to that in your formula:
> Fcst=fy+fn
> Obs=oy+on
> Hit=fy_oy
>
> VSDB records for NAM 24h forecast, 24h accumulation ending at 12Z 1
> Jul,
> over the whole ConUS:
>
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
> 0.50241
> 0.39447 0.47419
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
> 0.32717
> 0.20198 0.28927
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
> 0.18653
> 0.07300 0.14465
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
> 0.07716
> 0.01438 0.05267
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
> 0.03883
> 0.00439 0.02283
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
> 0.02123
> 0.00194 0.01093
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
> 0.00605
> 0.00039 0.00313
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
> 0.00243
> 0.00002 0.00057
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
> 0.00057
> 0.00000 0.00005
> V01 NAM 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
> 0.00010
> 0.00000 0.00000
>
> Could you double check the calculations for 3" and 1" on your end?
I
> double-checked my formula and yours against that in equation (7.18)
in
> the Wilks book (2nd ed) and don't see a problem.
>
> Ying
>
> On 09/06/2016 04:10 PM, Tatiana Burek via RT wrote:
> > Ying,
> > Looking at the data in database and simplifying filtering to
> > obs_thresh ='>3.0'
> > there is only 1 records in the database for each model and
> > threshold. For example,
> > for NAM and obs_thresh='>3.0':
> > | model | fcst_init_beg | fcst_valid_beg | fcst_lead |
> > obs_thresh | fcst_var | vx_mask | stat_name | stat_value |
> > stat_ncl | stat_ncu | stat_bcl | stat_bcu |
> >
> > | NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24 |
> > >3.0 | APCP/24 | G218/RFC | 59670 | 0 | 34 | 3 |
> > 59633 |
> > Using the formula:
> > c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
> > gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
> > GSS = -4.6202
> >
> > Tatiana
> >
> > On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
> >> [Don't mean to ask you to look into this before the holiday
weekend
> >> -
> >> after Labor Day is fine. Just want to get this out when all the
> >> details
> >> are fresh on my mind.]
> >>
> >> Another issue with GSS computation - when I tried plotting the
GSS
> >> on
> >> the 12km grid 218, the scores look quite wrong - see attached
xml,
> >> png,
> >> and the scores (top panel, ETS=GSS) plotted from FVS. The R data
> >> window
> >> shows that the scores computed for MetViewer for most thresholds
> >> actually look to be what's on the FVS plot, only the scores for
the
> >> 3.0"/day threshold is far off. The FVS plot shows that for this
> >> threshold, there are only 3 data points where the verifying
analysis
> >> has
> >> values >=3"/day. Could this have blown the calculation for
> >> MetViewer?
> >>
> >> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All
three
> >> of
> >> them have much simpler formulas than GSS.
> >>
> >> Thanks,
> >>
> >> Ying
> >>
> >
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77783] GSS computation when there are few data points
From: Ying Lin
Time: Fri Sep 09 14:17:03 2016
Tatiana,
Thanks for fixing that. Now the numbers and plot look more
reasonable.
Now that the obvious problem is fixed, there seems also to be an issue
with MV GSS when there are many data points: the numbers for both
models still seem off at 0.01"/day. I computed this for GFS:
At 0.01"/day for GFS:
Fcst=34763, Hit=25125, Obs=28295. Total=59670
Chance = (Fcst*Obs)/Total
= 16484.32
GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
= (25125-16484.32)/(34763+28295-25125-16484.32)
= 0.40705
I rounded the Fcst/Hit/Obs to whole numbers, so the result is slightly
different from the FVS result of 0.402868.
The number from MetViewer R data is 0.54147. NAM's GSS from MetViewer
at 0.01"/day is also higher than what FVS produces.
Here are the VSDB records for NAM 24h forecast, 24h accumulation
ending
at 12Z 1 Jul, over the whole ConUS:
V01 GFS 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
0.58259
0.42107 0.47419
V01 GFS 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
0.35122
0.21770 0.28927
V01 GFS 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
0.19881
0.08570 0.14465
V01 GFS 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
0.07404
0.01689 0.05267
V01 GFS 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
0.02903
0.00514 0.02283
V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
0.01316
0.00196 0.01093
V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
0.00429
0.00017 0.00313
V01 GFS 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
0.00184
0.00002 0.00057
V01 GFS 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
0.00010
0.00000 0.00005
V01 GFS 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
0.00000
0.00000 0.00000
Would you check the calculation (or MySQL) at 0.01"? Thanks!
Ying
On 09/09/2016 03:42 PM, Tatiana Burek via RT wrote:
> Ying,
> I checked our calculations and they are correct.
> The problem was in the MySQL functions for calculating stats where
we used REAL datatype to store intermediate results. Somehow this
datatype does not working well with small numbers like 0.0000462
(4.62e-05) and returned 4 instead of the actual value. The fix is
available for GSS calculation in mv_pcp_jul database.
>
> Tatiana
>
>
> On Fri Sep 09 09:42:14 2016, ying.lin at noaa.gov wrote:
>> Hi Tatiana,
>>
>> I plugged in the numbers for NAM at 3" and got GSS=-4.6202E-05,
which
>> is exactly 5 orders of magnitude smaller than the MET calculation.
>> Here's the computation, for thresholds of 3" and 1". Curious that
the
>> GSS at 1"/day is quite close to the MET result:
>>
>> At 3"/day: Fcst=34, Hit=0, Obs=3, Total=59670
>> Chance =(Fcst*Obs)/Total
>> =0.0017094
>> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
>> = (-0.0017094)/(34+3-0.0017094)
>> = 0.000046202
>>
>> At 1"/day: Fcst=1267, Hit=115, Obs=652
>> Chance = (Fcst*Obs)/Total
>> = 13.8442
>> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
>> = (115-13.8442)/(1267+652-115-13.8442)
>> = 0.05651
>>
>> The FVS definitions above compared to that in your formula:
>> Fcst=fy+fn
>> Obs=oy+on
>> Hit=fy_oy
>>
>> VSDB records for NAM 24h forecast, 24h accumulation ending at 12Z 1
>> Jul,
>> over the whole ConUS:
>>
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
>> 0.50241
>> 0.39447 0.47419
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
>> 0.32717
>> 0.20198 0.28927
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
>> 0.18653
>> 0.07300 0.14465
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
>> 0.07716
>> 0.01438 0.05267
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
>> 0.03883
>> 0.00439 0.02283
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
>> 0.02123
>> 0.00194 0.01093
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
>> 0.00605
>> 0.00039 0.00313
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
>> 0.00243
>> 0.00002 0.00057
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
>> 0.00057
>> 0.00000 0.00005
>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
>> 0.00010
>> 0.00000 0.00000
>>
>> Could you double check the calculations for 3" and 1" on your end?
I
>> double-checked my formula and yours against that in equation (7.18)
in
>> the Wilks book (2nd ed) and don't see a problem.
>>
>> Ying
>>
>> On 09/06/2016 04:10 PM, Tatiana Burek via RT wrote:
>>> Ying,
>>> Looking at the data in database and simplifying filtering to
>>> obs_thresh ='>3.0'
>>> there is only 1 records in the database for each model and
>>> threshold. For example,
>>> for NAM and obs_thresh='>3.0':
>>> | model | fcst_init_beg | fcst_valid_beg | fcst_lead |
>>> obs_thresh | fcst_var | vx_mask | stat_name | stat_value |
>>> stat_ncl | stat_ncu | stat_bcl | stat_bcu |
>>>
>>> | NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24 |
>>>> 3.0 | APCP/24 | G218/RFC | 59670 | 0 | 34 | 3 |
>>> 59633 |
>>> Using the formula:
>>> c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
>>> gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
>>> GSS = -4.6202
>>>
>>> Tatiana
>>>
>>> On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
>>>> [Don't mean to ask you to look into this before the holiday
weekend
>>>> -
>>>> after Labor Day is fine. Just want to get this out when all the
>>>> details
>>>> are fresh on my mind.]
>>>>
>>>> Another issue with GSS computation - when I tried plotting the
GSS
>>>> on
>>>> the 12km grid 218, the scores look quite wrong - see attached
xml,
>>>> png,
>>>> and the scores (top panel, ETS=GSS) plotted from FVS. The R data
>>>> window
>>>> shows that the scores computed for MetViewer for most thresholds
>>>> actually look to be what's on the FVS plot, only the scores for
the
>>>> 3.0"/day threshold is far off. The FVS plot shows that for this
>>>> threshold, there are only 3 data points where the verifying
analysis
>>>> has
>>>> values >=3"/day. Could this have blown the calculation for
>>>> MetViewer?
>>>>
>>>> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All
three
>>>> of
>>>> them have much simpler formulas than GSS.
>>>>
>>>> Thanks,
>>>>
>>>> Ying
>>>>
>>>
>
>
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: GSS computation when there are few data points
From: Tatiana Burek
Time: Fri Sep 09 15:15:45 2016
Ying,
After adjusting MySQL functions I got GSS values:
for NAM 0.454294 and GFS 0.402854
Please check if it is close to your calculations.
Tatiana
On Fri Sep 09 14:17:03 2016, ying.lin at noaa.gov wrote:
> Tatiana,
>
> Thanks for fixing that. Now the numbers and plot look more
> reasonable.
>
> Now that the obvious problem is fixed, there seems also to be an
issue
> with MV GSS when there are many data points: the numbers for both
> models still seem off at 0.01"/day. I computed this for GFS:
>
> At 0.01"/day for GFS:
>
> Fcst=34763, Hit=25125, Obs=28295. Total=59670
> Chance = (Fcst*Obs)/Total
> = 16484.32
> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
> = (25125-16484.32)/(34763+28295-25125-16484.32)
> = 0.40705
>
> I rounded the Fcst/Hit/Obs to whole numbers, so the result is
slightly
> different from the FVS result of 0.402868.
>
> The number from MetViewer R data is 0.54147. NAM's GSS from
MetViewer
> at 0.01"/day is also higher than what FVS produces.
>
> Here are the VSDB records for NAM 24h forecast, 24h accumulation
> ending
> at 12Z 1 Jul, over the whole ConUS:
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
> 0.58259
> 0.42107 0.47419
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
> 0.35122
> 0.21770 0.28927
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
> 0.19881
> 0.08570 0.14465
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
> 0.07404
> 0.01689 0.05267
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
> 0.02903
> 0.00514 0.02283
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
> 0.01316
> 0.00196 0.01093
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
> 0.00429
> 0.00017 0.00313
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
> 0.00184
> 0.00002 0.00057
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
> 0.00010
> 0.00000 0.00005
> V01 GFS 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
> 0.00000
> 0.00000 0.00000
>
> Would you check the calculation (or MySQL) at 0.01"? Thanks!
>
> Ying
>
> On 09/09/2016 03:42 PM, Tatiana Burek via RT wrote:
> > Ying,
> > I checked our calculations and they are correct.
> > The problem was in the MySQL functions for calculating stats where
we
> > used REAL datatype to store intermediate results. Somehow this
> > datatype does not working well with small numbers like 0.0000462
> > (4.62e-05) and returned 4 instead of the actual value. The fix is
> > available for GSS calculation in mv_pcp_jul database.
> >
> > Tatiana
> >
> >
> > On Fri Sep 09 09:42:14 2016, ying.lin at noaa.gov wrote:
> >> Hi Tatiana,
> >>
> >> I plugged in the numbers for NAM at 3" and got GSS=-4.6202E-05,
> >> which
> >> is exactly 5 orders of magnitude smaller than the MET
calculation.
> >> Here's the computation, for thresholds of 3" and 1". Curious
that
> >> the
> >> GSS at 1"/day is quite close to the MET result:
> >>
> >> At 3"/day: Fcst=34, Hit=0, Obs=3, Total=59670
> >> Chance =(Fcst*Obs)/Total
> >> =0.0017094
> >> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
> >> = (-0.0017094)/(34+3-0.0017094)
> >> = 0.000046202
> >>
> >> At 1"/day: Fcst=1267, Hit=115, Obs=652
> >> Chance = (Fcst*Obs)/Total
> >> = 13.8442
> >> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
> >> = (115-13.8442)/(1267+652-115-13.8442)
> >> = 0.05651
> >>
> >> The FVS definitions above compared to that in your formula:
> >> Fcst=fy+fn
> >> Obs=oy+on
> >> Hit=fy_oy
> >>
> >> VSDB records for NAM 24h forecast, 24h accumulation ending at 12Z
1
> >> Jul,
> >> over the whole ConUS:
> >>
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
> >> 0.50241
> >> 0.39447 0.47419
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
> >> 0.32717
> >> 0.20198 0.28927
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
> >> 0.18653
> >> 0.07300 0.14465
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
> >> 0.07716
> >> 0.01438 0.05267
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
> >> 0.03883
> >> 0.00439 0.02283
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
> >> 0.02123
> >> 0.00194 0.01093
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
> >> 0.00605
> >> 0.00039 0.00313
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
> >> 0.00243
> >> 0.00002 0.00057
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
> >> 0.00057
> >> 0.00000 0.00005
> >> V01 NAM 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
> >> 0.00010
> >> 0.00000 0.00000
> >>
> >> Could you double check the calculations for 3" and 1" on your
end?
> >> I
> >> double-checked my formula and yours against that in equation
(7.18)
> >> in
> >> the Wilks book (2nd ed) and don't see a problem.
> >>
> >> Ying
> >>
> >> On 09/06/2016 04:10 PM, Tatiana Burek via RT wrote:
> >>> Ying,
> >>> Looking at the data in database and simplifying filtering to
> >>> obs_thresh ='>3.0'
> >>> there is only 1 records in the database for each model and
> >>> threshold. For example,
> >>> for NAM and obs_thresh='>3.0':
> >>> | model | fcst_init_beg | fcst_valid_beg | fcst_lead
|
> >>> obs_thresh | fcst_var | vx_mask | stat_name | stat_value
|
> >>> stat_ncl | stat_ncu | stat_bcl | stat_bcu |
> >>>
> >>> | NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24
|
> >>>> 3.0 | APCP/24 | G218/RFC | 59670 | 0 | 34 | 3
|
> >>> 59633 |
> >>> Using the formula:
> >>> c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
> >>> gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
> >>> GSS = -4.6202
> >>>
> >>> Tatiana
> >>>
> >>> On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
> >>>> [Don't mean to ask you to look into this before the holiday
> >>>> weekend
> >>>> -
> >>>> after Labor Day is fine. Just want to get this out when all
the
> >>>> details
> >>>> are fresh on my mind.]
> >>>>
> >>>> Another issue with GSS computation - when I tried plotting the
GSS
> >>>> on
> >>>> the 12km grid 218, the scores look quite wrong - see attached
xml,
> >>>> png,
> >>>> and the scores (top panel, ETS=GSS) plotted from FVS. The R
data
> >>>> window
> >>>> shows that the scores computed for MetViewer for most
thresholds
> >>>> actually look to be what's on the FVS plot, only the scores for
> >>>> the
> >>>> 3.0"/day threshold is far off. The FVS plot shows that for this
> >>>> threshold, there are only 3 data points where the verifying
> >>>> analysis
> >>>> has
> >>>> values >=3"/day. Could this have blown the calculation for
> >>>> MetViewer?
> >>>>
> >>>> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All
> >>>> three
> >>>> of
> >>>> them have much simpler formulas than GSS.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Ying
> >>>>
> >>>
> >
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #77783] GSS computation when there are few data points
From: Ying Lin
Time: Fri Sep 09 17:09:29 2016
Tatiana,
They now look quite close to that generated from FVS. Thank you very
much for your help.
Ying
On 09/09/2016 05:15 PM, Tatiana Burek via RT wrote:
> Ying,
>
> After adjusting MySQL functions I got GSS values:
> for NAM 0.454294 and GFS 0.402854
> Please check if it is close to your calculations.
>
> Tatiana
>
> On Fri Sep 09 14:17:03 2016, ying.lin at noaa.gov wrote:
>> Tatiana,
>>
>> Thanks for fixing that. Now the numbers and plot look more
>> reasonable.
>>
>> Now that the obvious problem is fixed, there seems also to be an
issue
>> with MV GSS when there are many data points: the numbers for both
>> models still seem off at 0.01"/day. I computed this for GFS:
>>
>> At 0.01"/day for GFS:
>>
>> Fcst=34763, Hit=25125, Obs=28295. Total=59670
>> Chance = (Fcst*Obs)/Total
>> = 16484.32
>> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
>> = (25125-16484.32)/(34763+28295-25125-16484.32)
>> = 0.40705
>>
>> I rounded the Fcst/Hit/Obs to whole numbers, so the result is
slightly
>> different from the FVS result of 0.402868.
>>
>> The number from MetViewer R data is 0.54147. NAM's GSS from
MetViewer
>> at 0.01"/day is also higher than what FVS produces.
>>
>> Here are the VSDB records for NAM 24h forecast, 24h accumulation
>> ending
>> at 12Z 1 Jul, over the whole ConUS:
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
>> 0.58259
>> 0.42107 0.47419
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
>> 0.35122
>> 0.21770 0.28927
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
>> 0.19881
>> 0.08570 0.14465
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
>> 0.07404
>> 0.01689 0.05267
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
>> 0.02903
>> 0.00514 0.02283
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
>> 0.01316
>> 0.00196 0.01093
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
>> 0.00429
>> 0.00017 0.00313
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
>> 0.00184
>> 0.00002 0.00057
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
>> 0.00010
>> 0.00000 0.00005
>> V01 GFS 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
>> 0.00000
>> 0.00000 0.00000
>>
>> Would you check the calculation (or MySQL) at 0.01"? Thanks!
>>
>> Ying
>>
>> On 09/09/2016 03:42 PM, Tatiana Burek via RT wrote:
>>> Ying,
>>> I checked our calculations and they are correct.
>>> The problem was in the MySQL functions for calculating stats where
we
>>> used REAL datatype to store intermediate results. Somehow this
>>> datatype does not working well with small numbers like 0.0000462
>>> (4.62e-05) and returned 4 instead of the actual value. The fix is
>>> available for GSS calculation in mv_pcp_jul database.
>>>
>>> Tatiana
>>>
>>>
>>> On Fri Sep 09 09:42:14 2016, ying.lin at noaa.gov wrote:
>>>> Hi Tatiana,
>>>>
>>>> I plugged in the numbers for NAM at 3" and got GSS=-4.6202E-05,
>>>> which
>>>> is exactly 5 orders of magnitude smaller than the MET
calculation.
>>>> Here's the computation, for thresholds of 3" and 1". Curious
that
>>>> the
>>>> GSS at 1"/day is quite close to the MET result:
>>>>
>>>> At 3"/day: Fcst=34, Hit=0, Obs=3, Total=59670
>>>> Chance =(Fcst*Obs)/Total
>>>> =0.0017094
>>>> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
>>>> = (-0.0017094)/(34+3-0.0017094)
>>>> = 0.000046202
>>>>
>>>> At 1"/day: Fcst=1267, Hit=115, Obs=652
>>>> Chance = (Fcst*Obs)/Total
>>>> = 13.8442
>>>> GSS = (Hit-Chance)/(Fcst+Obs-Hit-Chance)
>>>> = (115-13.8442)/(1267+652-115-13.8442)
>>>> = 0.05651
>>>>
>>>> The FVS definitions above compared to that in your formula:
>>>> Fcst=fy+fn
>>>> Obs=oy+on
>>>> Hit=fy_oy
>>>>
>>>> VSDB records for NAM 24h forecast, 24h accumulation ending at 12Z
1
>>>> Jul,
>>>> over the whole ConUS:
>>>>
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.01 APCP/24 SFC = 59670
>>>> 0.50241
>>>> 0.39447 0.47419
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.10 APCP/24 SFC = 59670
>>>> 0.32717
>>>> 0.20198 0.28927
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.25 APCP/24 SFC = 59670
>>>> 0.18653
>>>> 0.07300 0.14465
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.50 APCP/24 SFC = 59670
>>>> 0.07716
>>>> 0.01438 0.05267
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>.75 APCP/24 SFC = 59670
>>>> 0.03883
>>>> 0.00439 0.02283
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.0 APCP/24 SFC = 59670
>>>> 0.02123
>>>> 0.00194 0.01093
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>1.5 APCP/24 SFC = 59670
>>>> 0.00605
>>>> 0.00039 0.00313
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>2.0 APCP/24 SFC = 59670
>>>> 0.00243
>>>> 0.00002 0.00057
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>3.0 APCP/24 SFC = 59670
>>>> 0.00057
>>>> 0.00000 0.00005
>>>> V01 NAM 24 2016070112 CCPA G218/RFC FHO>4.0 APCP/24 SFC = 59670
>>>> 0.00010
>>>> 0.00000 0.00000
>>>>
>>>> Could you double check the calculations for 3" and 1" on your
end?
>>>> I
>>>> double-checked my formula and yours against that in equation
(7.18)
>>>> in
>>>> the Wilks book (2nd ed) and don't see a problem.
>>>>
>>>> Ying
>>>>
>>>> On 09/06/2016 04:10 PM, Tatiana Burek via RT wrote:
>>>>> Ying,
>>>>> Looking at the data in database and simplifying filtering to
>>>>> obs_thresh ='>3.0'
>>>>> there is only 1 records in the database for each model and
>>>>> threshold. For example,
>>>>> for NAM and obs_thresh='>3.0':
>>>>> | model | fcst_init_beg | fcst_valid_beg | fcst_lead
|
>>>>> obs_thresh | fcst_var | vx_mask | stat_name | stat_value
|
>>>>> stat_ncl | stat_ncu | stat_bcl | stat_bcu |
>>>>>
>>>>> | NAM | 2016-06-30 12:00:00 | 2016-07-01 12:00:00 | 24
|
>>>>>> 3.0 | APCP/24 | G218/RFC | 59670 | 0 | 34 | 3
|
>>>>> 59633 |
>>>>> Using the formula:
>>>>> c = ( (fy_oy + fy_on) / total ) * (fy_oy + fn_oy);
>>>>> gss= (fy_oy - c) / (fy_oy + fy_on + fn_oy - c)
>>>>> GSS = -4.6202
>>>>>
>>>>> Tatiana
>>>>>
>>>>> On Fri Sep 02 13:56:13 2016, ying.lin at noaa.gov wrote:
>>>>>> [Don't mean to ask you to look into this before the holiday
>>>>>> weekend
>>>>>> -
>>>>>> after Labor Day is fine. Just want to get this out when all
the
>>>>>> details
>>>>>> are fresh on my mind.]
>>>>>>
>>>>>> Another issue with GSS computation - when I tried plotting the
GSS
>>>>>> on
>>>>>> the 12km grid 218, the scores look quite wrong - see attached
xml,
>>>>>> png,
>>>>>> and the scores (top panel, ETS=GSS) plotted from FVS. The R
data
>>>>>> window
>>>>>> shows that the scores computed for MetViewer for most
thresholds
>>>>>> actually look to be what's on the FVS plot, only the scores for
>>>>>> the
>>>>>> 3.0"/day threshold is far off. The FVS plot shows that for this
>>>>>> threshold, there are only 3 data points where the verifying
>>>>>> analysis
>>>>>> has
>>>>>> values >=3"/day. Could this have blown the calculation for
>>>>>> MetViewer?
>>>>>>
>>>>>> Scores for this day for FBIAS/CSI/FAR on g218 look fine. All
>>>>>> three
>>>>>> of
>>>>>> them have much simpler formulas than GSS.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ying
>>>>>>
>>>
>
>
--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list