[Met_help] [rt.rap.ucar.edu #79563] History for AUC plots in METViewer - possible issue with vx_mask
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 22 09:13:29 MST 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I think something is incorrect with my ROC_AUC plots in METViewer.
I recently generated some statistics using the Grid Stat tool over the
entire CONUS grid ("FULL"), and also over the 14 NCEP verification regions
(using the *.poly files in /share/met/poly). When I made a plot of AUC as
a function of forecast lead time using the full grid, I got reasonable mean
values for AUC, ~0.95 (hrefv2_auc12.png). However, when I made a plot of
AUC using the 14 NCEP subregions, the sum of which should be close to the
scores from the full grid, I get extremely low mean values of AUC
(auc_seriesplot.png).
I took a look at the values METViewer was plotting in the R data tab and I
also made the attached box plot. Most of the AUC values are 0, with all
other values being much higher. The mean value at each lead time is
indicated by the black star. I'm not sure why MET is producing 0's for
AUC; when I use the "FULL" grid there are no 0's. What do you think could
be the issue here? My database on the NCEP instance of METViewer is
mv_met_bblake_prob. I've also attached a GridStat config file which
specifies how the vx_mask is defined.
Thanks,
Ben Blake
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: John Halley Gotway
Time: Fri Feb 17 12:36:06 2017
Hi Ben,
I see you have some issues regarding the computation of the area under
a
ROC curve in the output of MET. First, thanks for sending the
METViewer
plots... they really help a lot to illustrate your point. I have a
pretty
good guess as to what might be going on, but couldn't be positive
without
seeing your METViewer XML's.
My guess is that you're seeing the difference between plotting the
summary
"mean" of several statistics versus plotting an "aggregated" statistic
value.
Let's step back and think about a receiver operating characteristic
(ROC).
It's a plot showing the relationship between probability of detection
and
false alarm rate for several probability thresholds. I looked at some
output of MET's regression test and see that when there are no actual
events observed, the AUC is written out as 0. Perhaps that should be
a
missing value instead of zero... I should ask a statistician here? As
you
increase the threshold, the event becomes more "rare" and the more
likely
you are to have 0 events... especially in the smaller subregions.
Comparing your box plot of subregions (auc_boxplot.png) to the series
plot
(auc_seriesplot.png), I see the mean *'s from the boxplot match the
dots of
the time-series. That tells me that the time-series is plotting the
MEAN
of the AUC values for each sub-region. And that's why the values are
so
low. Instead, I suspect what you really intended to do was AGGREGATE
the
underlying probabilistic contingency tables (PCT) together first and
then
compute an aggregated AUC score.
This logic is controlled in the METViewer GUI in the "Statistics"
section.
I suspect you had "Summary" selected with "Plot Statistic = Mean".
And
that results in the MEAN AUC of the 14 subregions being plotted.
Instead, I'd recommend using "Aggregation Statistics" with
"Probability
methods output (PCT)". That should aggregate together the probability
contingency tables (PCT lines) for the 14 subregions first and derive
a
single AUC value for the group.
All of this being said, unfortunately, the logic I recommend using
isn't
working in METViewer right now. I've sent Tatiana and XML which
demonstrates the issue and she looking into the problem. I'll let you
know
when we have a fix/update for METViewer.
Does this all make sense or do you have more questions?
Thanks,
John
On Fri, Feb 17, 2017 at 9:07 AM, Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Fri Feb 17 09:07:14 2017: Request 79563 was acted upon.
> Transaction: Ticket created by benjamin.blake at noaa.gov
> Queue: met_help
> Subject: AUC plots in METViewer - possible issue with vx_mask
> Owner: Nobody
> Requestors: benjamin.blake at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
>
>
> Hi,
>
> I think something is incorrect with my ROC_AUC plots in METViewer.
>
> I recently generated some statistics using the Grid Stat tool over
the
> entire CONUS grid ("FULL"), and also over the 14 NCEP verification
regions
> (using the *.poly files in /share/met/poly). When I made a plot of
AUC as
> a function of forecast lead time using the full grid, I got
reasonable mean
> values for AUC, ~0.95 (hrefv2_auc12.png). However, when I made a
plot of
> AUC using the 14 NCEP subregions, the sum of which should be close
to the
> scores from the full grid, I get extremely low mean values of AUC
> (auc_seriesplot.png).
>
> I took a look at the values METViewer was plotting in the R data tab
and I
> also made the attached box plot. Most of the AUC values are 0, with
all
> other values being much higher. The mean value at each lead time is
> indicated by the black star. I'm not sure why MET is producing 0's
for
> AUC; when I use the "FULL" grid there are no 0's. What do you think
could
> be the issue here? My database on the NCEP instance of METViewer is
> mv_met_bblake_prob. I've also attached a GridStat config file which
> specifies how the vx_mask is defined.
>
> Thanks,
> Ben Blake
>
>
------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: John Halley Gotway
Time: Fri Feb 17 14:46:55 2017
Ben,
FYI, I just confirmed with one of the statisticians here that the area
under the ROC curve should be undefined, not zero, when there are 0
events. With 0 events, the PODY score (y-axis of ROC curve) is 0/0
which
is undefined.
I updated the logic in MET for the upcoming met-6.0 release to correct
this
issue.
We're still working on debugging the METViewer aggregation of PCT
lines
issue.
Thanks,
John
On Fri, Feb 17, 2017 at 12:36 PM, The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Fri Feb 17 12:36:06 2017: Request 79563 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: AUC plots in METViewer - possible issue with vx_mask
> Owner: johnhg
> Requestors: benjamin.blake at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: Benjamin Blake - NOAA Affiliate
Time: Tue Feb 21 05:41:51 2017
Hi John,
That all makes sense. You are correct, what I had intended to do was
plot
the aggregated AUC score and not include the 0's or undefined values.
I
found where I can select aggregation statistics with probability
methods
output (PCT), so once that is working I can go ahead and use that.
For a
short term fix I could output the CONUS grid in addition to the 14
subregions - I may want to do that anyway.
And yes I believe that's correct - the AUC should be undefined when
there
are 0 events. Glad to know that the logic will be updated in the next
release.
Thanks for the help!
Ben
On Fri, Feb 17, 2017 at 4:46 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ben,
>
> FYI, I just confirmed with one of the statisticians here that the
area
> under the ROC curve should be undefined, not zero, when there are 0
> events. With 0 events, the PODY score (y-axis of ROC curve) is 0/0
which
> is undefined.
>
> I updated the logic in MET for the upcoming met-6.0 release to
correct this
> issue.
>
> We're still working on debugging the METViewer aggregation of PCT
lines
> issue.
>
> Thanks,
> John
>
> On Fri, Feb 17, 2017 at 12:36 PM, The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Feb 17 12:36:06 2017: Request 79563 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > Queue: met_help
> > Subject: AUC plots in METViewer - possible issue with vx_mask
> > Owner: johnhg
> > Requestors: benjamin.blake at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
> >
> >
> > This transaction appears to have no content
> >
>
>
------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: John Halley Gotway
Time: Tue Feb 21 09:37:10 2017
Ben,
Sounds good. I'll keep this ticket open and let you know when
METViewer
has been updated to correctly do the aggregation of AUC.
Thanks,
John
On Tue, Feb 21, 2017 at 5:41 AM, Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
>
> Hi John,
>
> That all makes sense. You are correct, what I had intended to do
was plot
> the aggregated AUC score and not include the 0's or undefined
values. I
> found where I can select aggregation statistics with probability
methods
> output (PCT), so once that is working I can go ahead and use that.
For a
> short term fix I could output the CONUS grid in addition to the 14
> subregions - I may want to do that anyway.
>
> And yes I believe that's correct - the AUC should be undefined when
there
> are 0 events. Glad to know that the logic will be updated in the
next
> release.
>
> Thanks for the help!
> Ben
>
> On Fri, Feb 17, 2017 at 4:46 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ben,
> >
> > FYI, I just confirmed with one of the statisticians here that the
area
> > under the ROC curve should be undefined, not zero, when there are
0
> > events. With 0 events, the PODY score (y-axis of ROC curve) is
0/0 which
> > is undefined.
> >
> > I updated the logic in MET for the upcoming met-6.0 release to
correct
> this
> > issue.
> >
> > We're still working on debugging the METViewer aggregation of PCT
lines
> > issue.
> >
> > Thanks,
> > John
> >
> > On Fri, Feb 17, 2017 at 12:36 PM, The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Feb 17 12:36:06 2017: Request 79563 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > > Queue: met_help
> > > Subject: AUC plots in METViewer - possible issue with
vx_mask
> > > Owner: johnhg
> > > Requestors: benjamin.blake at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
>
>
------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: John Halley Gotway
Time: Tue Feb 21 14:24:33 2017
Ben,
Good news. Tatiana just updated the build of METViewer at EMC to
correctly
plot the ROC_AUC aggregations.
In my testing I made the attached plot/xml which includes 4 lines:
CONUS,
EAST, WEST, and EAST + WEST (i.e. East and West aggregated together).
You
only see 3 lines, because CONUS and EAST + WEST are on top of
eachother.
If you'd like to play around with this example, use:
http://www.dtcenter.org/met/metviewer/servlet?jsp=new
Please give it a shot with your data at EMC and let me know if you run
into
any more problems.
Thanks,
John
On Tue, Feb 21, 2017 at 9:36 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ben,
>
> Sounds good. I'll keep this ticket open and let you know when
METViewer
> has been updated to correctly do the aggregation of AUC.
>
> Thanks,
> John
>
> On Tue, Feb 21, 2017 at 5:41 AM, Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
>>
>> Hi John,
>>
>> That all makes sense. You are correct, what I had intended to do
was plot
>> the aggregated AUC score and not include the 0's or undefined
values. I
>> found where I can select aggregation statistics with probability
methods
>> output (PCT), so once that is working I can go ahead and use that.
For a
>> short term fix I could output the CONUS grid in addition to the 14
>> subregions - I may want to do that anyway.
>>
>> And yes I believe that's correct - the AUC should be undefined when
there
>> are 0 events. Glad to know that the logic will be updated in the
next
>> release.
>>
>> Thanks for the help!
>> Ben
>>
>> On Fri, Feb 17, 2017 at 4:46 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Ben,
>> >
>> > FYI, I just confirmed with one of the statisticians here that the
area
>> > under the ROC curve should be undefined, not zero, when there are
0
>> > events. With 0 events, the PODY score (y-axis of ROC curve) is
0/0
>> which
>> > is undefined.
>> >
>> > I updated the logic in MET for the upcoming met-6.0 release to
correct
>> this
>> > issue.
>> >
>> > We're still working on debugging the METViewer aggregation of PCT
lines
>> > issue.
>> >
>> > Thanks,
>> > John
>> >
>> > On Fri, Feb 17, 2017 at 12:36 PM, The RT System itself via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Fri Feb 17 12:36:06 2017: Request 79563 was acted upon.
>> > > Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> > > Queue: met_help
>> > > Subject: AUC plots in METViewer - possible issue with
vx_mask
>> > > Owner: johnhg
>> > > Requestors: benjamin.blake at noaa.gov
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563
>> >
>> > >
>> > >
>> > > This transaction appears to have no content
>> > >
>> >
>> >
>>
>>
>
------------------------------------------------
Subject: AUC plots in METViewer - possible issue with vx_mask
From: Benjamin Blake - NOAA Affiliate
Time: Wed Feb 22 05:40:57 2017
Hi John,
Yes it seems to be working now. Thank you and Tatiana for fixing it
so
quickly!
Ben
On Tue, Feb 21, 2017 at 4:24 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ben,
>
> Good news. Tatiana just updated the build of METViewer at EMC to
correctly
> plot the ROC_AUC aggregations.
>
> In my testing I made the attached plot/xml which includes 4 lines:
CONUS,
> EAST, WEST, and EAST + WEST (i.e. East and West aggregated
together). You
> only see 3 lines, because CONUS and EAST + WEST are on top of
eachother.
> If you'd like to play around with this example, use:
> http://www.dtcenter.org/met/metviewer/servlet?jsp=new
>
> Please give it a shot with your data at EMC and let me know if you
run into
> any more problems.
>
> Thanks,
> John
>
>
>
> On Tue, Feb 21, 2017 at 9:36 AM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Ben,
> >
> > Sounds good. I'll keep this ticket open and let you know when
METViewer
> > has been updated to correctly do the aggregation of AUC.
> >
> > Thanks,
> > John
> >
> > On Tue, Feb 21, 2017 at 5:41 AM, Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79563 >
> >>
> >> Hi John,
> >>
> >> That all makes sense. You are correct, what I had intended to do
was
> plot
> >> the aggregated AUC score and not include the 0's or undefined
values. I
> >> found where I can select aggregation statistics with probability
methods
> >> output (PCT), so once that is working I can go ahead and use
that. For
> a
> >> short term fix I could output the CONUS grid in addition to the
14
> >> subregions - I may want to do that anyway.
> >>
> >> And yes I believe that's correct - the AUC should be undefined
when
> there
> >> are 0 events. Glad to know that the logic will be updated in the
next
> >> release.
> >>
> >> Thanks for the help!
> >> Ben
> >>
> >> On Fri, Feb 17, 2017 at 4:46 PM, John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> > Ben,
> >> >
> >> > FYI, I just confirmed with one of the statisticians here that
the area
> >> > under the ROC curve should be undefined, not zero, when there
are 0
> >> > events. With 0 events, the PODY score (y-axis of ROC curve) is
0/0
> >> which
> >> > is undefined.
> >> >
> >> > I updated the logic in MET for the upcoming met-6.0 release to
correct
> >> this
> >> > issue.
> >> >
> >> > We're still working on debugging the METViewer aggregation of
PCT
> lines
> >> > issue.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Fri, Feb 17, 2017 at 12:36 PM, The RT System itself via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Fri Feb 17 12:36:06 2017: Request 79563 was acted upon.
> >> > > Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> >> > > Queue: met_help
> >> > > Subject: AUC plots in METViewer - possible issue with
vx_mask
> >> > > Owner: johnhg
> >> > > Requestors: benjamin.blake at noaa.gov
> >> > > Status: new
> >> > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=79563
> >> >
> >> > >
> >> > >
> >> > > This transaction appears to have no content
> >> > >
> >> >
> >> >
> >>
> >>
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list