[Met_help] [rt.rap.ucar.edu #56762] History for Questions about go index
John Halley Gotway via RT
met_help at ucar.edu
Thu Jun 14 07:47:12 MDT 2012
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
met_help,
I tried the go index in MET. I have some questions about it.
- go index computes the weighted average of RMSE. In the log file, I see
Skill Score Index Term 14: fcst_var = WIND, fcst_lev = Z10, fcst_lead_sec =
86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref = 1, fcst =
2.16095, ref = 2.17051, skill = 0.00878351, weight = 6
In my understanding, RMSE statistic is available in point_stat cnt file,
why line_type SL1L2 is used here?
- When n_fcst >1, there are should be more than RMSE statistic for
reference or forecast model, how to assign ref or fcst value here? Is it
the average of several RMSE? e.g.
Skill Score Index Term 12: fcst_var = WIND, fcst_lev = P850, fcst_lead_sec
= 172800, line_type = SL1L2, column = RMSE, n_fcst = 2, n_ref = 2, fcst =
3.1197, ref = 3.16026, skill = 0.0255049, weight = 1
- In the log file, I found Term information are repeated, then go index
is computed . e.g.
DEBUG 3: Skill Score Index Term 1: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
DEBUG 3: Skill Score Index Term 2: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
DEBUG 3: Skill Score Index Term 3: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
DEBUG 3: Skill Score Index Term 4: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
DEBUG 3: Skill Score Index Term 5: fcst_var = WIND, fcst_lev = P400,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
.
.
.
Start from Term 49, Term 1,2,3,4,5 are repeated. Why?
DEBUG 3: Skill Score Index Term 49: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
DEBUG 3: Skill Score Index Term 50: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
DEBUG 3: Skill Score Index Term 51: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
DEBUG 3: Skill Score Index Term 52: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
DEBUG 3: Skill Score Index Term 53: fcst_var = WIND, fcst_lev = P400,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
Could you help me with these question?
The detailed log information is on bluefire:
/ptmp/lir/Go_index/Verify/Run/stat_analysis.html
Running scripts: /ptmp/lir/Go_index%met_verif_wrapper.ksh
/ptmp/lir/Go_index%met_run_stat_analysis.ksh
Thank you,
Ruifang
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Questions about go index
From: John Halley Gotway
Time: Mon Jun 04 11:50:45 2012
Ruifang,
I created a met_help ticket for your question.
Answers are inline below. Just let me know if you have any more
questions.
Thanks,
John Halley Gotway
On Mon Jun 04 11:43:09 2012, johnhg wrote:
> met_help,
>
> I tried the go index in MET. I have some questions about it.
>
>
> - go index computes the weighted average of RMSE. In the log
file, I see
>
> Skill Score Index Term 14: fcst_var = WIND, fcst_lev = Z10,
fcst_lead_sec =
> 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref = 1, fcst
=
> 2.16095, ref = 2.17051, skill = 0.00878351, weight = 6
>
> In my understanding, RMSE statistic is available in point_stat cnt
file,
> why line_type SL1L2 is used here?
>
You are correct. The RMSE statistic is included in the CNT line.
However, it can also be derived from the SL1L2 line, which we do in
this case. Using the SL1L2 line enables us to aggregate together
multiple lines, and from them compute a single RMSE score. For
example, suppose you've done your verification over 2 subdomains. You
could compute a GO Index score for either subdomain - OR you could
compute a GO Index for both by aggregating together the SL1L2 lines.
>
> - When n_fcst >1, there are should be more than RMSE statistic
for
> reference or forecast model, how to assign ref or fcst value
here? Is it
> the average of several RMSE? e.g.
>
> Skill Score Index Term 12: fcst_var = WIND, fcst_lev = P850,
fcst_lead_sec
> = 172800, line_type = SL1L2, column = RMSE, n_fcst = 2, n_ref = 2,
fcst =
> 3.1197, ref = 3.16026, skill = 0.0255049, weight = 1
>
As mentioned above, for each term that goes into the GO Index, we
aggregate together the SL1L2 lines, compute a single RMSE value for
the forecast, and a single RMSE value for the observation. And with
those we compute a skill score for that term.
>
> - In the log file, I found Term information are repeated, then
go index
> is computed . e.g.
>
> DEBUG 3: Skill Score Index Term 1: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
> DEBUG 3: Skill Score Index Term 2: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
> DEBUG 3: Skill Score Index Term 3: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
> DEBUG 3: Skill Score Index Term 4: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
> DEBUG 3: Skill Score Index Term 5: fcst_var = WIND, fcst_lev = P400,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
> .
> .
> .
> Start from Term 49, Term 1,2,3,4,5 are repeated. Why?
>
Thanks for pointing this out! This is actually a bug in the code that
we're resolving for the next release of MET. The good new is that
since this is a weighted average of terms the resulting GO Index value
is correct. But to be clear, the GO Index is defined over 48 terms -
not 96, as the log file seems to indicate.
> DEBUG 3: Skill Score Index Term 49: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
> DEBUG 3: Skill Score Index Term 50: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
> DEBUG 3: Skill Score Index Term 51: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
> DEBUG 3: Skill Score Index Term 52: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
> DEBUG 3: Skill Score Index Term 53: fcst_var = WIND, fcst_lev =
P400,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
>
>
> Could you help me with these question?
> The detailed log information is on bluefire:
> /ptmp/lir/Go_index/Verify/Run/stat_analysis.html
> Running scripts: /ptmp/lir/Go_index%met_verif_wrapper.ksh
>
/ptmp/lir/Go_index%met_run_stat_analysis.ksh
>
> Thank you,
>
> Ruifang
>
------------------------------------------------
More information about the Met_help
mailing list