[Met_help] [rt.rap.ucar.edu #56762] History for Questions about go index

John Halley Gotway via RT met_help at ucar.edu
Thu Jun 14 10:30:35 MDT 2012


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

met_help,

I tried the go index in MET. I have some questions about it.


   - go index computes the weighted average of RMSE. In the log file, I see

Skill Score Index Term 14: fcst_var = WIND, fcst_lev = Z10, fcst_lead_sec =
86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref = 1, fcst =
2.16095, ref = 2.17051, skill = 0.00878351, weight = 6

In my understanding, RMSE statistic is available in point_stat cnt  file,
why line_type SL1L2 is used here?


   - When n_fcst >1, there are should be more than RMSE statistic for
   reference or forecast model, how to assign ref or fcst value here? Is it
   the average of several RMSE? e.g.

Skill Score Index Term 12: fcst_var = WIND, fcst_lev = P850, fcst_lead_sec
= 172800, line_type = SL1L2, column = RMSE, n_fcst = 2, n_ref = 2, fcst =
3.1197, ref = 3.16026, skill = 0.0255049, weight = 1


   - In the log file, I found  Term information are repeated, then go index
   is computed . e.g.

DEBUG 3: Skill Score Index Term 1: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
DEBUG 3: Skill Score Index Term 2: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
DEBUG 3: Skill Score Index Term 3: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
DEBUG 3: Skill Score Index Term 4: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
DEBUG 3: Skill Score Index Term 5: fcst_var = WIND, fcst_lev = P400,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
.
.
.
Start from Term 49, Term 1,2,3,4,5 are repeated. Why?

DEBUG 3: Skill Score Index Term 49: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
DEBUG 3: Skill Score Index Term 50: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
DEBUG 3: Skill Score Index Term 51: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
DEBUG 3: Skill Score Index Term 52: fcst_var = WIND, fcst_lev = P250,
fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
DEBUG 3: Skill Score Index Term 53: fcst_var = WIND, fcst_lev = P400,
fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref
= 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4


Could you help me with these question?
The detailed log information is on bluefire:
/ptmp/lir/Go_index/Verify/Run/stat_analysis.html
Running scripts: /ptmp/lir/Go_index%met_verif_wrapper.ksh
                           /ptmp/lir/Go_index%met_run_stat_analysis.ksh

Thank you,

Ruifang



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Questions about go index
From: John Halley Gotway
Time: Mon Jun 04 11:50:45 2012

Ruifang,

I created a met_help ticket for your question.

Answers are inline below.  Just let me know if you have any more
questions.

Thanks,
John Halley Gotway

On Mon Jun 04 11:43:09 2012, johnhg wrote:
> met_help,
>
> I tried the go index in MET. I have some questions about it.
>
>
>    - go index computes the weighted average of RMSE. In the log
file, I see
>
> Skill Score Index Term 14: fcst_var = WIND, fcst_lev = Z10,
fcst_lead_sec =
> 86400, line_type = SL1L2, column = RMSE, n_fcst = 1, n_ref = 1, fcst
=
> 2.16095, ref = 2.17051, skill = 0.00878351, weight = 6
>
> In my understanding, RMSE statistic is available in point_stat cnt
file,
> why line_type SL1L2 is used here?
>

You are correct.  The RMSE statistic is included in the CNT line.
However, it can also be derived from the SL1L2 line, which we do in
this case.  Using the SL1L2 line enables us to aggregate together
multiple lines, and from them compute a single RMSE score.  For
example, suppose you've done your verification over 2 subdomains.  You
could compute a GO Index score for either subdomain - OR you could
compute a GO Index for both by aggregating together the SL1L2 lines.

>
>    - When n_fcst >1, there are should be more than RMSE statistic
for
>    reference or forecast model, how to assign ref or fcst value
here? Is it
>    the average of several RMSE? e.g.
>
> Skill Score Index Term 12: fcst_var = WIND, fcst_lev = P850,
fcst_lead_sec
> = 172800, line_type = SL1L2, column = RMSE, n_fcst = 2, n_ref = 2,
fcst =
> 3.1197, ref = 3.16026, skill = 0.0255049, weight = 1
>

As mentioned above, for each term that goes into the GO Index, we
aggregate together the SL1L2 lines, compute a single RMSE value for
the forecast, and a single RMSE value for the observation.  And with
those we compute a skill score for that term.

>
>    - In the log file, I found  Term information are repeated, then
go index
>    is computed . e.g.
>
> DEBUG 3: Skill Score Index Term 1: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
> DEBUG 3: Skill Score Index Term 2: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
> DEBUG 3: Skill Score Index Term 3: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
> DEBUG 3: Skill Score Index Term 4: fcst_var = WIND, fcst_lev = P250,
> fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
> DEBUG 3: Skill Score Index Term 5: fcst_var = WIND, fcst_lev = P400,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
> .
> .
> .
> Start from Term 49, Term 1,2,3,4,5 are repeated. Why?
>

Thanks for pointing this out!  This is actually a bug in the code that
we're resolving for the next release of MET.  The good new is that
since this is a weighted average of terms the resulting GO Index value
is correct.  But to be clear, the GO Index is defined over 48 terms -
not 96, as the log file seems to indicate.

> DEBUG 3: Skill Score Index Term 49: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 3.55266, ref = 3.48565, skill = -0.0388138, weight = 4
> DEBUG 3: Skill Score Index Term 50: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 86400, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 4.22806, ref = 4.19348, skill = -0.0165622, weight = 3
> DEBUG 3: Skill Score Index Term 51: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 129600, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 4.543, ref = 4.4668, skill = -0.0344075, weight = 2
> DEBUG 3: Skill Score Index Term 52: fcst_var = WIND, fcst_lev =
P250,
> fcst_lead_sec = 172800, line_type = SL1L2, column = RMSE, n_fcst =
1, n_ref
> = 1, fcst = 5.16767, ref = 5.0109, skill = -0.0635514, weight = 1
> DEBUG 3: Skill Score Index Term 53: fcst_var = WIND, fcst_lev =
P400,
> fcst_lead_sec = 43200, line_type = SL1L2, column = RMSE, n_fcst = 1,
n_ref
> = 1, fcst = 2.55755, ref = 2.57513, skill = 0.0136112, weight = 4
>
>
> Could you help me with these question?
> The detailed log information is on bluefire:
> /ptmp/lir/Go_index/Verify/Run/stat_analysis.html
> Running scripts: /ptmp/lir/Go_index%met_verif_wrapper.ksh
>
/ptmp/lir/Go_index%met_run_stat_analysis.ksh
>
> Thank you,
>
> Ruifang
>



------------------------------------------------
Subject: Questions about go index
From: lir at ucar.edu
Time: Thu Jun 14 09:05:24 2012

John,
Thank you very much for your help.
Ruifang

On Thu, Jun 14, 2012 at 7:47 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>



--
Ruifang

Mesoscale & Microscale Meteorology Division
Phone: 303-497-8938
Office:  FL3-2125A
Email:  lir at ucar.edu

------------------------------------------------


More information about the Met_help mailing list