[Met_help] [rt.rap.ucar.edu #87959] History for METviewer: (event) equalizer
John Halley Gotway via RT
met_help at ucar.edu
Mon Dec 10 15:04:17 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. A question about METviewe event equalizer: in an interactive
METviewer plot such as for the attached xml, what I'd like to do is to
make sure that only stats for the forecast cycles where all models' data
are present are used. NAM/GFS stats for FSS06h are computed out to 84h,
CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
"equalize" the FCST_LEAD. Looks like I cannot check the "Equalize"
button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need to check
"Event Equalizer", which then turns on all the "Equalize" buttons. I
can then un-check the "Equalize" box next to the "FCST_LEAD", but
whether this box is checked or not, the score plot will only go out to
36h (the forecast lead time where all four models have in common).
Question: does "event equalizer" button override all other equalizer
buttons - when it's on, all the fixed values/independent variables must
equalize? On the interactive METviewer, there is no way to turn off
"event equalizer" while leaving some other equalizer on. I did try to
1) click on the "event equalizer", 2) un-select the "equalizer" next to
"FCST_LEAD", then save the xml, edit it to set event_equal to "false"
while leaving field equalize="true" for fcst_valid_beg_0, then upload
the xml. It seems to work - the plot produced would show NAM/GFS out to
84h, CONUSNEST to 60h and HRRR to 36h, but then all the "equalize"
buttons would be unchecked, even the one next to FCST_VALID_BEGIN, so I
don't know that the resulting plots are from forecast cycles where all
models are present. What's the proper way to make sure that the
consistency check is done for model run cycles (fcst_valid_begin), but
allow the plot to show different forecast lead times?
Thanks!
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: METviewer: (event) equalizer
From: John Halley Gotway
Time: Wed Dec 05 09:38:12 2018
Ying,
In theory, the event equalization logic is very simple. But in
practice
(as you've found) it's pretty darn confusing! Thanks for sending your
XML... that really helps to illustrate.
First, we defined the event equalization logic to subset the data down
to
the set of "cases" that are common for all "series" on the plot. I
put
"case" and "series" in quotes to emphasize that they have a specific
meaning in this context.
The "series" are just the lines (typically) on the plot. Very often
the
series are defined as different models. The "case" consists of the
set of
fields which define the keys to be equalized across the series. For
each
independent variable values (for example, forecast hour = 24), we
define
the cases for each series and subset the data so that we only keep the
cases common to all series.
What you're requesting is a partial form of event equalization.
Basically,
apply the equalization logic out to a certain forecast hour, but not
beyond
that.
Unfortunately, METviewer does not support that logic. Event
equalization
is either on or off. I understand what you want to do in your
example, but
the difficulty is in defining a clear rule set in the code to handle
it.
Thanks,
John
On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: METviewer: (event) equalizer
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>
>
> Hello. A question about METviewe event equalizer: in an interactive
> METviewer plot such as for the attached xml, what I'd like to do is
to
> make sure that only stats for the forecast cycles where all models'
data
> are present are used. NAM/GFS stats for FSS06h are computed out to
84h,
> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
> "equalize" the FCST_LEAD. Looks like I cannot check the "Equalize"
> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need to
check
> "Event Equalizer", which then turns on all the "Equalize" buttons.
I
> can then un-check the "Equalize" box next to the "FCST_LEAD", but
> whether this box is checked or not, the score plot will only go out
to
> 36h (the forecast lead time where all four models have in common).
>
> Question: does "event equalizer" button override all other equalizer
> buttons - when it's on, all the fixed values/independent variables
must
> equalize? On the interactive METviewer, there is no way to turn off
> "event equalizer" while leaving some other equalizer on. I did try
to
> 1) click on the "event equalizer", 2) un-select the "equalizer" next
to
> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
> the xml. It seems to work - the plot produced would show NAM/GFS
out to
> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the "equalize"
> buttons would be unchecked, even the one next to FCST_VALID_BEGIN,
so I
> don't know that the resulting plots are from forecast cycles where
all
> models are present. What's the proper way to make sure that the
> consistency check is done for model run cycles (fcst_valid_begin),
but
> allow the plot to show different forecast lead times?
>
> Thanks!
>
> Ying
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87959] METviewer: (event) equalizer
From: Ying Lin
Time: Thu Dec 06 10:10:28 2018
Hi John,
Thank you for the very helpful and thoughtful reply. Before we
close this, I want to make sure I understand this correctly - so when
making plots such as I described earlier (scores as a function of
model
lead time, Model A & B go out to 84h, Model C goes out to 36h):
1) Should only do that - if one really insists - if one is reasonably
certain there are no missing model cycles (e.g. comparing operational
models) Turn event equalizer off.
2) When there are missing model cycles (e.g. comparing parallel models
against opnl models), event equalizer is necessary; should limit
comparisons to forecasts lengths that all models have in common (e.g.
36h).
Thanks again,
Ying
On 12/5/18 11:38 AM, John Halley Gotway via RT wrote:
> Ying,
>
> In theory, the event equalization logic is very simple. But in
practice
> (as you've found) it's pretty darn confusing! Thanks for sending
your
> XML... that really helps to illustrate.
>
> First, we defined the event equalization logic to subset the data
down to
> the set of "cases" that are common for all "series" on the plot. I
put
> "case" and "series" in quotes to emphasize that they have a specific
> meaning in this context.
> The "series" are just the lines (typically) on the plot. Very often
the
> series are defined as different models. The "case" consists of the
set of
> fields which define the keys to be equalized across the series. For
each
> independent variable values (for example, forecast hour = 24), we
define
> the cases for each series and subset the data so that we only keep
the
> cases common to all series.
>
> What you're requesting is a partial form of event equalization.
Basically,
> apply the equalization logic out to a certain forecast hour, but not
beyond
> that.
>
> Unfortunately, METviewer does not support that logic. Event
equalization
> is either on or off. I understand what you want to do in your
example, but
> the difficulty is in defining a clear rule set in the code to handle
it.
>
> Thanks,
> John
>
>
>
> On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: METviewer: (event) equalizer
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>>
>>
>> Hello. A question about METviewe event equalizer: in an
interactive
>> METviewer plot such as for the attached xml, what I'd like to do is
to
>> make sure that only stats for the forecast cycles where all models'
data
>> are present are used. NAM/GFS stats for FSS06h are computed out to
84h,
>> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
>> "equalize" the FCST_LEAD. Looks like I cannot check the "Equalize"
>> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need to
check
>> "Event Equalizer", which then turns on all the "Equalize" buttons.
I
>> can then un-check the "Equalize" box next to the "FCST_LEAD", but
>> whether this box is checked or not, the score plot will only go out
to
>> 36h (the forecast lead time where all four models have in common).
>>
>> Question: does "event equalizer" button override all other
equalizer
>> buttons - when it's on, all the fixed values/independent variables
must
>> equalize? On the interactive METviewer, there is no way to turn
off
>> "event equalizer" while leaving some other equalizer on. I did try
to
>> 1) click on the "event equalizer", 2) un-select the "equalizer"
next to
>> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
>> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
>> the xml. It seems to work - the plot produced would show NAM/GFS
out to
>> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the "equalize"
>> buttons would be unchecked, even the one next to FCST_VALID_BEGIN,
so I
>> don't know that the resulting plots are from forecast cycles where
all
>> models are present. What's the proper way to make sure that the
>> consistency check is done for model run cycles (fcst_valid_begin),
but
>> allow the plot to show different forecast lead times?
>>
>> Thanks!
>>
>> Ying
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: (event) equalizer
From: John Halley Gotway
Time: Fri Dec 07 14:48:00 2018
Ying,
Yes, both of your statements are correct.
With event equalized turned off, you can include multiple models on
the
same plot and see generally how the lines compare. But you should be
cognizant of the data being compared. For example, perhaps HRRR is
initialized every hour while GFS is initialized every 6 hours. You
might
consider defining "INIT_HOUR = 00, 06, 12, and 18" in the fixed values
section to the comparison better.
If you really want to test whether one model is doing better than
another,
we recommend defining a pairwise difference curve. When doing a
pairwise
difference, event equalization is turned on automatically.
In METviewer, event equalization is either turned ON or OFF. And what
I
*think* you'd like to see is a 3rd option...
For each independent variable value (e.g. forecast lead = 48 hours),
equalize across the series that actually contain data for that
independent
variable value.
So if model A goes to 24 hours, model B to 36, and model C to 48...
- Equalize A, B, C from 0 to 24 hours.
- Equalize B and C from 25 to 36 hours.
- Equalize C by itself from 37 to 48.
I *think* this is the functionality you'd like to see for your
example.
But with event equalization, things can get very confusing very
quickly!
Thanks,
John
On Thu, Dec 6, 2018 at 10:11 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>
> Hi John,
>
> Thank you for the very helpful and thoughtful reply. Before we
> close this, I want to make sure I understand this correctly - so
when
> making plots such as I described earlier (scores as a function of
model
> lead time, Model A & B go out to 84h, Model C goes out to 36h):
>
> 1) Should only do that - if one really insists - if one is
reasonably
> certain there are no missing model cycles (e.g. comparing
operational
> models) Turn event equalizer off.
>
> 2) When there are missing model cycles (e.g. comparing parallel
models
> against opnl models), event equalizer is necessary; should limit
> comparisons to forecasts lengths that all models have in common
(e.g. 36h).
>
> Thanks again,
>
> Ying
>
> On 12/5/18 11:38 AM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > In theory, the event equalization logic is very simple. But in
practice
> > (as you've found) it's pretty darn confusing! Thanks for sending
your
> > XML... that really helps to illustrate.
> >
> > First, we defined the event equalization logic to subset the data
down to
> > the set of "cases" that are common for all "series" on the plot.
I put
> > "case" and "series" in quotes to emphasize that they have a
specific
> > meaning in this context.
> > The "series" are just the lines (typically) on the plot. Very
often the
> > series are defined as different models. The "case" consists of
the set
> of
> > fields which define the keys to be equalized across the series.
For each
> > independent variable values (for example, forecast hour = 24), we
define
> > the cases for each series and subset the data so that we only keep
the
> > cases common to all series.
> >
> > What you're requesting is a partial form of event equalization.
> Basically,
> > apply the equalization logic out to a certain forecast hour, but
not
> beyond
> > that.
> >
> > Unfortunately, METviewer does not support that logic. Event
equalization
> > is either on or off. I understand what you want to do in your
example,
> but
> > the difficulty is in defining a clear rule set in the code to
handle it.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
> >> Transaction: Ticket created by ying.lin at noaa.gov
> >> Queue: met_help
> >> Subject: METviewer: (event) equalizer
> >> Owner: Nobody
> >> Requestors: ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959
> >
> >>
> >>
> >> Hello. A question about METviewe event equalizer: in an
interactive
> >> METviewer plot such as for the attached xml, what I'd like to do
is to
> >> make sure that only stats for the forecast cycles where all
models' data
> >> are present are used. NAM/GFS stats for FSS06h are computed out
to 84h,
> >> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
> >> "equalize" the FCST_LEAD. Looks like I cannot check the
"Equalize"
> >> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need
to check
> >> "Event Equalizer", which then turns on all the "Equalize"
buttons. I
> >> can then un-check the "Equalize" box next to the "FCST_LEAD", but
> >> whether this box is checked or not, the score plot will only go
out to
> >> 36h (the forecast lead time where all four models have in
common).
> >>
> >> Question: does "event equalizer" button override all other
equalizer
> >> buttons - when it's on, all the fixed values/independent
variables must
> >> equalize? On the interactive METviewer, there is no way to turn
off
> >> "event equalizer" while leaving some other equalizer on. I did
try to
> >> 1) click on the "event equalizer", 2) un-select the "equalizer"
next to
> >> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
> >> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
> >> the xml. It seems to work - the plot produced would show NAM/GFS
out to
> >> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the
"equalize"
> >> buttons would be unchecked, even the one next to
FCST_VALID_BEGIN, so I
> >> don't know that the resulting plots are from forecast cycles
where all
> >> models are present. What's the proper way to make sure that the
> >> consistency check is done for model run cycles
(fcst_valid_begin), but
> >> allow the plot to show different forecast lead times?
> >>
> >> Thanks!
> >>
> >> Ying
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87959] METviewer: (event) equalizer
From: Ying Lin
Time: Fri Dec 07 17:22:43 2018
Thank you John. I've been learning a lot here. Please close this
ticket at your convenience.
Yes, definitely should specify INIT_HOUR if/when event equalizer is
turned off (I'm only verifying HRRR's 00/06/12/18Z cycles, but still.
Right now one of the regional FV3 runs is only on the 00Z cycle, so
this
is pretty important).
Good tip about pairwise difference curve, thanks!
On your model A/B/C example ... I haven't really thought of it in
terms
of equalizing between two models or among N models, what I was
thinking
was along the line of "compute/display model stats for the run cycles
that all models have in common, out to each model's own forecast range
(or the range of verification), even when models' have different
forecast ranges. I _think_ that's equivalent to what you're saying in
the A/B/C example. The concept does get harder to grasp here (if a
model forecast cycle crashed or somehow verif for one model did not
run
to completion, then ...)
Anyway. Have a great weekend.
Ying
On 12/7/18 4:48 PM, John Halley Gotway via RT wrote:
> Ying,
>
> Yes, both of your statements are correct.
>
> With event equalized turned off, you can include multiple models on
the
> same plot and see generally how the lines compare. But you should
be
> cognizant of the data being compared. For example, perhaps HRRR is
> initialized every hour while GFS is initialized every 6 hours. You
might
> consider defining "INIT_HOUR = 00, 06, 12, and 18" in the fixed
values
> section to the comparison better.
>
> If you really want to test whether one model is doing better than
another,
> we recommend defining a pairwise difference curve. When doing a
pairwise
> difference, event equalization is turned on automatically.
>
> In METviewer, event equalization is either turned ON or OFF. And
what I
> *think* you'd like to see is a 3rd option...
> For each independent variable value (e.g. forecast lead = 48 hours),
> equalize across the series that actually contain data for that
independent
> variable value.
>
> So if model A goes to 24 hours, model B to 36, and model C to 48...
> - Equalize A, B, C from 0 to 24 hours.
> - Equalize B and C from 25 to 36 hours.
> - Equalize C by itself from 37 to 48.
>
> I *think* this is the functionality you'd like to see for your
example.
> But with event equalization, things can get very confusing very
quickly!
>
> Thanks,
> John
>
>
>
> On Thu, Dec 6, 2018 at 10:11 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>>
>> Hi John,
>>
>> Thank you for the very helpful and thoughtful reply. Before
we
>> close this, I want to make sure I understand this correctly - so
when
>> making plots such as I described earlier (scores as a function of
model
>> lead time, Model A & B go out to 84h, Model C goes out to 36h):
>>
>> 1) Should only do that - if one really insists - if one is
reasonably
>> certain there are no missing model cycles (e.g. comparing
operational
>> models) Turn event equalizer off.
>>
>> 2) When there are missing model cycles (e.g. comparing parallel
models
>> against opnl models), event equalizer is necessary; should limit
>> comparisons to forecasts lengths that all models have in common
(e.g. 36h).
>>
>> Thanks again,
>>
>> Ying
>>
>> On 12/5/18 11:38 AM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> In theory, the event equalization logic is very simple. But in
practice
>>> (as you've found) it's pretty darn confusing! Thanks for sending
your
>>> XML... that really helps to illustrate.
>>>
>>> First, we defined the event equalization logic to subset the data
down to
>>> the set of "cases" that are common for all "series" on the plot.
I put
>>> "case" and "series" in quotes to emphasize that they have a
specific
>>> meaning in this context.
>>> The "series" are just the lines (typically) on the plot. Very
often the
>>> series are defined as different models. The "case" consists of
the set
>> of
>>> fields which define the keys to be equalized across the series.
For each
>>> independent variable values (for example, forecast hour = 24), we
define
>>> the cases for each series and subset the data so that we only keep
the
>>> cases common to all series.
>>>
>>> What you're requesting is a partial form of event equalization.
>> Basically,
>>> apply the equalization logic out to a certain forecast hour, but
not
>> beyond
>>> that.
>>>
>>> Unfortunately, METviewer does not support that logic. Event
equalization
>>> is either on or off. I understand what you want to do in your
example,
>> but
>>> the difficulty is in defining a clear rule set in the code to
handle it.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>>
>>> On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>> Queue: met_help
>>>> Subject: METviewer: (event) equalizer
>>>> Owner: Nobody
>>>> Requestors: ying.lin at noaa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959
>>>>
>>>> Hello. A question about METviewe event equalizer: in an
interactive
>>>> METviewer plot such as for the attached xml, what I'd like to do
is to
>>>> make sure that only stats for the forecast cycles where all
models' data
>>>> are present are used. NAM/GFS stats for FSS06h are computed out
to 84h,
>>>> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
>>>> "equalize" the FCST_LEAD. Looks like I cannot check the
"Equalize"
>>>> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need
to check
>>>> "Event Equalizer", which then turns on all the "Equalize"
buttons. I
>>>> can then un-check the "Equalize" box next to the "FCST_LEAD", but
>>>> whether this box is checked or not, the score plot will only go
out to
>>>> 36h (the forecast lead time where all four models have in
common).
>>>>
>>>> Question: does "event equalizer" button override all other
equalizer
>>>> buttons - when it's on, all the fixed values/independent
variables must
>>>> equalize? On the interactive METviewer, there is no way to turn
off
>>>> "event equalizer" while leaving some other equalizer on. I did
try to
>>>> 1) click on the "event equalizer", 2) un-select the "equalizer"
next to
>>>> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
>>>> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
>>>> the xml. It seems to work - the plot produced would show NAM/GFS
out to
>>>> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the
"equalize"
>>>> buttons would be unchecked, even the one next to
FCST_VALID_BEGIN, so I
>>>> don't know that the resulting plots are from forecast cycles
where all
>>>> models are present. What's the proper way to make sure that the
>>>> consistency check is done for model run cycles
(fcst_valid_begin), but
>>>> allow the plot to show different forecast lead times?
>>>>
>>>> Thanks!
>>>>
>>>> Ying
>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: (event) equalizer
From: John Halley Gotway
Time: Mon Dec 10 13:59:07 2018
Ying,
I should have mentioned... I used the request you made and create a
new
GitHub issue for METviewer. We can consider adding this feature in a
future version. Please take a look to make sure I described the
request
accurately:
https://github.com/NCAR/METviewer/issues/106
I think the logic is pretty well-defined and makes sense.
Thanks,
John
On Fri, Dec 7, 2018 at 5:23 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>
> Thank you John. I've been learning a lot here. Please close this
> ticket at your convenience.
>
> Yes, definitely should specify INIT_HOUR if/when event equalizer is
> turned off (I'm only verifying HRRR's 00/06/12/18Z cycles, but
still.
> Right now one of the regional FV3 runs is only on the 00Z cycle, so
this
> is pretty important).
>
> Good tip about pairwise difference curve, thanks!
>
> On your model A/B/C example ... I haven't really thought of it in
terms
> of equalizing between two models or among N models, what I was
thinking
> was along the line of "compute/display model stats for the run
cycles
> that all models have in common, out to each model's own forecast
range
> (or the range of verification), even when models' have different
> forecast ranges. I _think_ that's equivalent to what you're saying
in
> the A/B/C example. The concept does get harder to grasp here (if a
> model forecast cycle crashed or somehow verif for one model did not
run
> to completion, then ...)
>
> Anyway. Have a great weekend.
>
> Ying
>
> On 12/7/18 4:48 PM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > Yes, both of your statements are correct.
> >
> > With event equalized turned off, you can include multiple models
on the
> > same plot and see generally how the lines compare. But you should
be
> > cognizant of the data being compared. For example, perhaps HRRR
is
> > initialized every hour while GFS is initialized every 6 hours.
You might
> > consider defining "INIT_HOUR = 00, 06, 12, and 18" in the fixed
values
> > section to the comparison better.
> >
> > If you really want to test whether one model is doing better than
> another,
> > we recommend defining a pairwise difference curve. When doing a
pairwise
> > difference, event equalization is turned on automatically.
> >
> > In METviewer, event equalization is either turned ON or OFF. And
what I
> > *think* you'd like to see is a 3rd option...
> > For each independent variable value (e.g. forecast lead = 48
hours),
> > equalize across the series that actually contain data for that
> independent
> > variable value.
> >
> > So if model A goes to 24 hours, model B to 36, and model C to
48...
> > - Equalize A, B, C from 0 to 24 hours.
> > - Equalize B and C from 25 to 36 hours.
> > - Equalize C by itself from 37 to 48.
> >
> > I *think* this is the functionality you'd like to see for your
example.
> > But with event equalization, things can get very confusing very
quickly!
> >
> > Thanks,
> > John
> >
> >
> >
> > On Thu, Dec 6, 2018 at 10:11 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
> >>
> >> Hi John,
> >>
> >> Thank you for the very helpful and thoughtful reply.
Before we
> >> close this, I want to make sure I understand this correctly - so
when
> >> making plots such as I described earlier (scores as a function of
model
> >> lead time, Model A & B go out to 84h, Model C goes out to 36h):
> >>
> >> 1) Should only do that - if one really insists - if one is
reasonably
> >> certain there are no missing model cycles (e.g. comparing
operational
> >> models) Turn event equalizer off.
> >>
> >> 2) When there are missing model cycles (e.g. comparing parallel
models
> >> against opnl models), event equalizer is necessary; should limit
> >> comparisons to forecasts lengths that all models have in common
(e.g.
> 36h).
> >>
> >> Thanks again,
> >>
> >> Ying
> >>
> >> On 12/5/18 11:38 AM, John Halley Gotway via RT wrote:
> >>> Ying,
> >>>
> >>> In theory, the event equalization logic is very simple. But in
> practice
> >>> (as you've found) it's pretty darn confusing! Thanks for
sending your
> >>> XML... that really helps to illustrate.
> >>>
> >>> First, we defined the event equalization logic to subset the
data down
> to
> >>> the set of "cases" that are common for all "series" on the plot.
I put
> >>> "case" and "series" in quotes to emphasize that they have a
specific
> >>> meaning in this context.
> >>> The "series" are just the lines (typically) on the plot. Very
often
> the
> >>> series are defined as different models. The "case" consists of
the set
> >> of
> >>> fields which define the keys to be equalized across the series.
For
> each
> >>> independent variable values (for example, forecast hour = 24),
we
> define
> >>> the cases for each series and subset the data so that we only
keep the
> >>> cases common to all series.
> >>>
> >>> What you're requesting is a partial form of event equalization.
> >> Basically,
> >>> apply the equalization logic out to a certain forecast hour, but
not
> >> beyond
> >>> that.
> >>>
> >>> Unfortunately, METviewer does not support that logic. Event
> equalization
> >>> is either on or off. I understand what you want to do in your
example,
> >> but
> >>> the difficulty is in defining a clear rule set in the code to
handle
> it.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>>
> >>>
> >>> On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
> >>>> Transaction: Ticket created by ying.lin at noaa.gov
> >>>> Queue: met_help
> >>>> Subject: METviewer: (event) equalizer
> >>>> Owner: Nobody
> >>>> Requestors: ying.lin at noaa.gov
> >>>> Status: new
> >>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959
> >>>>
> >>>> Hello. A question about METviewe event equalizer: in an
interactive
> >>>> METviewer plot such as for the attached xml, what I'd like to
do is to
> >>>> make sure that only stats for the forecast cycles where all
models'
> data
> >>>> are present are used. NAM/GFS stats for FSS06h are computed
out to
> 84h,
> >>>> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
> >>>> "equalize" the FCST_LEAD. Looks like I cannot check the
"Equalize"
> >>>> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need
to
> check
> >>>> "Event Equalizer", which then turns on all the "Equalize"
buttons. I
> >>>> can then un-check the "Equalize" box next to the "FCST_LEAD",
but
> >>>> whether this box is checked or not, the score plot will only go
out to
> >>>> 36h (the forecast lead time where all four models have in
common).
> >>>>
> >>>> Question: does "event equalizer" button override all other
equalizer
> >>>> buttons - when it's on, all the fixed values/independent
variables
> must
> >>>> equalize? On the interactive METviewer, there is no way to
turn off
> >>>> "event equalizer" while leaving some other equalizer on. I did
try to
> >>>> 1) click on the "event equalizer", 2) un-select the "equalizer"
next
> to
> >>>> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
> >>>> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
> >>>> the xml. It seems to work - the plot produced would show
NAM/GFS out
> to
> >>>> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the
"equalize"
> >>>> buttons would be unchecked, even the one next to
FCST_VALID_BEGIN, so
> I
> >>>> don't know that the resulting plots are from forecast cycles
where all
> >>>> models are present. What's the proper way to make sure that
the
> >>>> consistency check is done for model run cycles
(fcst_valid_begin), but
> >>>> allow the plot to show different forecast lead times?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Ying
> >>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87959] METviewer: (event) equalizer
From: Ying Lin
Time: Mon Dec 10 14:07:43 2018
Hi John,
Yes, that's it. Thank you for the comprehensive answer to my
original questions and for creating the GitHub issue.
Ying
On 12/10/18 3:59 PM, John Halley Gotway via RT wrote:
> Ying,
>
> I should have mentioned... I used the request you made and create a
new
> GitHub issue for METviewer. We can consider adding this feature in
a
> future version. Please take a look to make sure I described the
request
> accurately:
> https://github.com/NCAR/METviewer/issues/106
>
> I think the logic is pretty well-defined and makes sense.
>
> Thanks,
> John
>
> On Fri, Dec 7, 2018 at 5:23 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>>
>> Thank you John. I've been learning a lot here. Please close this
>> ticket at your convenience.
>>
>> Yes, definitely should specify INIT_HOUR if/when event equalizer is
>> turned off (I'm only verifying HRRR's 00/06/12/18Z cycles, but
still.
>> Right now one of the regional FV3 runs is only on the 00Z cycle, so
this
>> is pretty important).
>>
>> Good tip about pairwise difference curve, thanks!
>>
>> On your model A/B/C example ... I haven't really thought of it in
terms
>> of equalizing between two models or among N models, what I was
thinking
>> was along the line of "compute/display model stats for the run
cycles
>> that all models have in common, out to each model's own forecast
range
>> (or the range of verification), even when models' have different
>> forecast ranges. I _think_ that's equivalent to what you're saying
in
>> the A/B/C example. The concept does get harder to grasp here (if a
>> model forecast cycle crashed or somehow verif for one model did not
run
>> to completion, then ...)
>>
>> Anyway. Have a great weekend.
>>
>> Ying
>>
>> On 12/7/18 4:48 PM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> Yes, both of your statements are correct.
>>>
>>> With event equalized turned off, you can include multiple models
on the
>>> same plot and see generally how the lines compare. But you should
be
>>> cognizant of the data being compared. For example, perhaps HRRR
is
>>> initialized every hour while GFS is initialized every 6 hours.
You might
>>> consider defining "INIT_HOUR = 00, 06, 12, and 18" in the fixed
values
>>> section to the comparison better.
>>>
>>> If you really want to test whether one model is doing better than
>> another,
>>> we recommend defining a pairwise difference curve. When doing a
pairwise
>>> difference, event equalization is turned on automatically.
>>>
>>> In METviewer, event equalization is either turned ON or OFF. And
what I
>>> *think* you'd like to see is a 3rd option...
>>> For each independent variable value (e.g. forecast lead = 48
hours),
>>> equalize across the series that actually contain data for that
>> independent
>>> variable value.
>>>
>>> So if model A goes to 24 hours, model B to 36, and model C to
48...
>>> - Equalize A, B, C from 0 to 24 hours.
>>> - Equalize B and C from 25 to 36 hours.
>>> - Equalize C by itself from 37 to 48.
>>>
>>> I *think* this is the functionality you'd like to see for your
example.
>>> But with event equalization, things can get very confusing very
quickly!
>>>
>>> Thanks,
>>> John
>>>
>>>
>>>
>>> On Thu, Dec 6, 2018 at 10:11 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959 >
>>>>
>>>> Hi John,
>>>>
>>>> Thank you for the very helpful and thoughtful reply.
Before we
>>>> close this, I want to make sure I understand this correctly - so
when
>>>> making plots such as I described earlier (scores as a function of
model
>>>> lead time, Model A & B go out to 84h, Model C goes out to 36h):
>>>>
>>>> 1) Should only do that - if one really insists - if one is
reasonably
>>>> certain there are no missing model cycles (e.g. comparing
operational
>>>> models) Turn event equalizer off.
>>>>
>>>> 2) When there are missing model cycles (e.g. comparing parallel
models
>>>> against opnl models), event equalizer is necessary; should limit
>>>> comparisons to forecasts lengths that all models have in common
(e.g.
>> 36h).
>>>> Thanks again,
>>>>
>>>> Ying
>>>>
>>>> On 12/5/18 11:38 AM, John Halley Gotway via RT wrote:
>>>>> Ying,
>>>>>
>>>>> In theory, the event equalization logic is very simple. But in
>> practice
>>>>> (as you've found) it's pretty darn confusing! Thanks for
sending your
>>>>> XML... that really helps to illustrate.
>>>>>
>>>>> First, we defined the event equalization logic to subset the
data down
>> to
>>>>> the set of "cases" that are common for all "series" on the plot.
I put
>>>>> "case" and "series" in quotes to emphasize that they have a
specific
>>>>> meaning in this context.
>>>>> The "series" are just the lines (typically) on the plot. Very
often
>> the
>>>>> series are defined as different models. The "case" consists of
the set
>>>> of
>>>>> fields which define the keys to be equalized across the series.
For
>> each
>>>>> independent variable values (for example, forecast hour = 24),
we
>> define
>>>>> the cases for each series and subset the data so that we only
keep the
>>>>> cases common to all series.
>>>>>
>>>>> What you're requesting is a partial form of event equalization.
>>>> Basically,
>>>>> apply the equalization logic out to a certain forecast hour, but
not
>>>> beyond
>>>>> that.
>>>>>
>>>>> Unfortunately, METviewer does not support that logic. Event
>> equalization
>>>>> is either on or off. I understand what you want to do in your
example,
>>>> but
>>>>> the difficulty is in defining a clear rule set in the code to
handle
>> it.
>>>>> Thanks,
>>>>> John
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 29, 2018 at 3:17 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> Thu Nov 29 15:17:17 2018: Request 87959 was acted upon.
>>>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>>>> Queue: met_help
>>>>>> Subject: METviewer: (event) equalizer
>>>>>> Owner: Nobody
>>>>>> Requestors: ying.lin at noaa.gov
>>>>>> Status: new
>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87959
>>>>>> Hello. A question about METviewe event equalizer: in an
interactive
>>>>>> METviewer plot such as for the attached xml, what I'd like to
do is to
>>>>>> make sure that only stats for the forecast cycles where all
models'
>> data
>>>>>> are present are used. NAM/GFS stats for FSS06h are computed
out to
>> 84h,
>>>>>> CONUSNEST is out to 60h and HRRR out to 36h, so I don't want to
>>>>>> "equalize" the FCST_LEAD. Looks like I cannot check the
"Equalize"
>>>>>> button next to FCST_VALID_BEGIN and "FCST_LEAD" directly - need
to
>> check
>>>>>> "Event Equalizer", which then turns on all the "Equalize"
buttons. I
>>>>>> can then un-check the "Equalize" box next to the "FCST_LEAD",
but
>>>>>> whether this box is checked or not, the score plot will only go
out to
>>>>>> 36h (the forecast lead time where all four models have in
common).
>>>>>>
>>>>>> Question: does "event equalizer" button override all other
equalizer
>>>>>> buttons - when it's on, all the fixed values/independent
variables
>> must
>>>>>> equalize? On the interactive METviewer, there is no way to
turn off
>>>>>> "event equalizer" while leaving some other equalizer on. I did
try to
>>>>>> 1) click on the "event equalizer", 2) un-select the "equalizer"
next
>> to
>>>>>> "FCST_LEAD", then save the xml, edit it to set event_equal to
"false"
>>>>>> while leaving field equalize="true" for fcst_valid_beg_0, then
upload
>>>>>> the xml. It seems to work - the plot produced would show
NAM/GFS out
>> to
>>>>>> 84h, CONUSNEST to 60h and HRRR to 36h, but then all the
"equalize"
>>>>>> buttons would be unchecked, even the one next to
FCST_VALID_BEGIN, so
>> I
>>>>>> don't know that the resulting plots are from forecast cycles
where all
>>>>>> models are present. What's the proper way to make sure that
the
>>>>>> consistency check is done for model run cycles
(fcst_valid_begin), but
>>>>>> allow the plot to show different forecast lead times?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list