[Met_help] [rt.rap.ucar.edu #78097] History for Reliability Diagrams in METViewer
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 7 16:38:30 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I think there may be a possible issue with METViewer's reliability diagrams.
Here is a ROC curve plot and a reliability diagram for probability
forecasts I generated from the HREF and HREFx. The ROC curves look fine
but the reliability diagram says that all of the models are way below the
no skill line which seems odd to me.
Thanks!
Ben Blake
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Reliability Diagrams in METViewer
From: John Halley Gotway
Time: Fri Sep 30 10:50:49 2016
Hi Ben,
Sorry for the delay in responding to you on this issue. We've had
some
internal emails about this issue, but I forgot to write you back about
that.
Tatiana took a look at the plot and grabbed the corresponding plot
data
file. So far we haven't found any obvious reason to believe that the
plot
is "wrong" but Tara agrees that it's odd the reliability values are so
low.
So we'll keep digging. Tatiana is going to look in the database for
the
path to your data files and try to pull them back here.
I first want to test that the underlying aggregation that METViewer is
doing matches the aggregation do by the MET STAT-Analysis tool. From
there, we could either look upstream (i.e. at the Point-Stat or Grid-
Stat
runs themselves) or downstream (i.e. how METViewer uses these counts
to
make the plot).
One thing to check is units. In your Point-Stat or Grid-Stat runs, I
see
that your observation threshold is ">=12.7". If the units of your
observations were actually inches, then the stats would be way off.
John
On Tue, Sep 27, 2016 at 9:48 AM, Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Tue Sep 27 09:48:46 2016: Request 78097 was acted upon.
> Transaction: Ticket created by benjamin.blake at noaa.gov
> Queue: met_help
> Subject: Reliability Diagrams in METViewer
> Owner: Nobody
> Requestors: benjamin.blake at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78097 >
>
>
> Hi,
>
> I think there may be a possible issue with METViewer's reliability
> diagrams.
>
> Here is a ROC curve plot and a reliability diagram for probability
> forecasts I generated from the HREF and HREFx. The ROC curves look
fine
> but the reliability diagram says that all of the models are way
below the
> no skill line which seems odd to me.
>
> Thanks!
> Ben Blake
>
>
------------------------------------------------
Subject: Reliability Diagrams in METViewer
From: Benjamin Blake - NOAA Affiliate
Time: Fri Sep 30 11:16:32 2016
Hi John,
No problem, thanks for looking into it!
I created the mv_met_probexp database where I loaded the .stat files I
generated from MET's GridStat tool for the HREF and HREFx
(experimental
version of the HREF). For each model, I'm testing two different
methods
for generating probabiltiies. I expect the GSD probabilities will be
much
more reliable from the data that I've loaded into the database so far,
but
like you said the reliability diagram indicates that both methods have
very
low reliability.
I am using CCPA grib2 files as the verifying precipitation
observations for
GridStat. I did a wgrib2 -V on one of the CCPA files I used and it
said
that the total precipitation was in units of kg/m2 (equivalent to mm).
Thanks,
Ben
On Fri, Sep 30, 2016 at 12:50 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Ben,
>
> Sorry for the delay in responding to you on this issue. We've had
some
> internal emails about this issue, but I forgot to write you back
about
> that.
>
> Tatiana took a look at the plot and grabbed the corresponding plot
data
> file. So far we haven't found any obvious reason to believe that
the plot
> is "wrong" but Tara agrees that it's odd the reliability values are
so low.
>
> So we'll keep digging. Tatiana is going to look in the database for
the
> path to your data files and try to pull them back here.
>
> I first want to test that the underlying aggregation that METViewer
is
> doing matches the aggregation do by the MET STAT-Analysis tool.
From
> there, we could either look upstream (i.e. at the Point-Stat or
Grid-Stat
> runs themselves) or downstream (i.e. how METViewer uses these counts
to
> make the plot).
>
> One thing to check is units. In your Point-Stat or Grid-Stat runs,
I see
> that your observation threshold is ">=12.7". If the units of your
> observations were actually inches, then the stats would be way off.
>
> John
>
> On Tue, Sep 27, 2016 at 9:48 AM, Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Sep 27 09:48:46 2016: Request 78097 was acted upon.
> > Transaction: Ticket created by benjamin.blake at noaa.gov
> > Queue: met_help
> > Subject: Reliability Diagrams in METViewer
> > Owner: Nobody
> > Requestors: benjamin.blake at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78097 >
> >
> >
> > Hi,
> >
> > I think there may be a possible issue with METViewer's reliability
> > diagrams.
> >
> > Here is a ROC curve plot and a reliability diagram for probability
> > forecasts I generated from the HREF and HREFx. The ROC curves
look fine
> > but the reliability diagram says that all of the models are way
below the
> > no skill line which seems odd to me.
> >
> > Thanks!
> > Ben Blake
> >
> >
>
>
------------------------------------------------
Subject: Reliability Diagrams in METViewer
From: John Halley Gotway
Time: Fri Sep 30 12:23:07 2016
Tatiana,
Great, thanks for pulling down data for the HREFX-AAS model. I
grabbed the
data and ran it through the following STAT-Analysis command:
/usr/local/met-5.2/bin/stat_analysis -lookin *.stat -job aggregate
-line_type PCT -by MODEL
The resulting OY and ON counts exactly match the output the METViewer
sums
in the file "sum_data.txt".
COL_NAME: MODEL TOTAL N_THRESH THRESH_1 OY_1 ON_1 THRESH_2
OY_2
ON_2 THRESH_3 OY_3 ON_3 THRESH_4 OY_4 ON_4 THRESH_5 OY_5 ON_5
THRESH_6 OY_6 ON_6 THRESH_7 OY_7 ON_7 THRESH_8 OY_8 ON_8 THRESH_9
OY_9 ON_9 THRESH_10 OY_10 ON_10 THRESH_11
PCT: HREFX-AAS 6953149 11 0 20953 5308301 0.1
5931
337405 0.2 6911 304127 0.3 8071 262509 0.4 13869 231710
0.5 14852 171885 0.6 7988 93891 0.7 7478 64573 0.8 6043
41130 0.9 13757 31765 1
So there is no problem with the aggregation. I really don't think
there's
a problem in the plotting logic either. Each Y-value represents the
observed frequency (i.e. OY/(OY+ON)). Reading those OY and ON sums
into R
and doing the math yields:
0.0039 0.0173 0.0222 0.0298 0.0565 0.0795 0.0784 0.1038 0.1281
0.3022
And those values correspond nicely with the plot. If there is a
problem, I
really don't think it's with METViewer's aggregation or plotting.
Ben, I'd suggest stepping back to your Grid-Stat runs and looking for
issues there. If I had your data and Grid-Stat config file in front
of me,
I'd plot it using MET's plot_data_plane tool and then loop through
several
cases to try to identify issues.
You checked on the units. What about the accumulation intervals?
What is
the accumulation interval is used for the probability forecasts? Does
that
match the CCPA accumulation interval?
Thanks,
John
On Fri, Sep 30, 2016 at 11:46 AM, Tatiana Burek <tatiana at ucar.edu>
wrote:
> I put a tar file with MET data that Ben used to build the
reliability
> diagram ( 'HREFX-AAS' model only):
> dakota : d3/projects/METViewer/rely
>
> I also added an image with one mode (plot_20160930_172724.png)l and
data
> that is coming from database and used by the Rscript to build the
plot
> (sum_data.txt).
> I noticed that the SQL query uses sum function to aggregate oy_i and
on_i
> by thresh_i and model. I added a file with DB data without
aggregation (
> real_data.txt ) .
>
> Tatiana
>
> On Tue, Sep 27, 2016 at 10:04 AM, Tara Jensen <jensen at ucar.edu>
wrote:
>
>> The diagrams do look a little strange. The diagram is more likely
to be
>> between the two lines. First question that comes to my mind is:
Is he
>> computing from VSDB or MET output... If VSDB - error could be
coming in
>> during mapping of VSDB to METV tables. If MET - maybe an error is
coming
>> in during aggregation. A second question: How many cases are
included in
>> the aggregation.
>>
>>
>> ---------- Forwarded message ----------
>> From: Benjamin Blake - NOAA Affiliate via RT <met_help at ucar.edu>
>> Date: Tue, Sep 27, 2016 at 9:48 AM
>> Subject: [rt.rap.ucar.edu #78097] Reliability Diagrams in METViewer
>> To:
>>
>>
>>
>> Tue Sep 27 09:48:46 2016: Request 78097 was acted upon.
>> Transaction: Ticket created by benjamin.blake at noaa.gov
>> Queue: met_help
>> Subject: Reliability Diagrams in METViewer
>> Owner: Nobody
>> Requestors: benjamin.blake at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78097 >
>>
>>
>> Hi,
>>
>> I think there may be a possible issue with METViewer's reliability
>> diagrams.
>>
>> Here is a ROC curve plot and a reliability diagram for probability
>> forecasts I generated from the HREF and HREFx. The ROC curves look
fine
>> but the reliability diagram says that all of the models are way
below the
>> no skill line which seems odd to me.
>>
>> Thanks!
>> Ben Blake
>>
>>
>>
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Tara Jensen
>> Project Manager II
>> NCAR RAL and DTC
>> PO Box 3000, Boulder, Colorado 80307 USA
>> +1 303-497-8479 jensen at ucar.edu
>>
>
>
------------------------------------------------
Subject: Reliability Diagrams in METViewer
From: Benjamin Blake - NOAA Affiliate
Time: Fri Sep 30 12:34:07 2016
Hi John,
It's good to know that METViewer is plotting the data correctly, so
the
issue must be with my GridStat runs. The accumulation intervals
should
both be 6 hours covering the same time period - I will double check to
make
sure though. That's a good idea for using the plot_data_plane tool,
I'll
try using that on individual cases next week.
I saw Tatiana asked about how many cases were included in the
aggregation.
I only had about 3 weeks of cases, and I know for probability
forecasts it
is better to have data over a longer time period.
Matt Pyle also had a similar issue where his reliability diagrams for
the
HREF looked very similar to mine. However, my Grid Stat config files
and
scripts are based off of his - so, if that is where the issue is, it
would
make sense that we are both getting low reliabilities.
Thanks,
Ben
On Fri, Sep 30, 2016 at 2:23 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Tatiana,
>
> Great, thanks for pulling down data for the HREFX-AAS model. I
grabbed the
> data and ran it through the following STAT-Analysis command:
>
> /usr/local/met-5.2/bin/stat_analysis -lookin *.stat -job aggregate
> -line_type PCT -by MODEL
>
> The resulting OY and ON counts exactly match the output the
METViewer sums
> in the file "sum_data.txt".
>
> COL_NAME: MODEL TOTAL N_THRESH THRESH_1 OY_1 ON_1 THRESH_2
OY_2
> ON_2 THRESH_3 OY_3 ON_3 THRESH_4 OY_4 ON_4 THRESH_5 OY_5 ON_5
> THRESH_6 OY_6 ON_6 THRESH_7 OY_7 ON_7 THRESH_8 OY_8 ON_8
THRESH_9
> OY_9 ON_9 THRESH_10 OY_10 ON_10 THRESH_11
> PCT: HREFX-AAS 6953149 11 0 20953 5308301 0.1
5931
> 337405 0.2 6911 304127 0.3 8071 262509 0.4 13869
231710
> 0.5 14852 171885 0.6 7988 93891 0.7 7478 64573 0.8
6043
> 41130 0.9 13757 31765 1
>
> So there is no problem with the aggregation. I really don't think
there's
> a problem in the plotting logic either. Each Y-value represents the
> observed frequency (i.e. OY/(OY+ON)). Reading those OY and ON sums
into R
> and doing the math yields:
> 0.0039 0.0173 0.0222 0.0298 0.0565 0.0795 0.0784 0.1038 0.1281
0.3022
>
> And those values correspond nicely with the plot. If there is a
problem, I
> really don't think it's with METViewer's aggregation or plotting.
>
> Ben, I'd suggest stepping back to your Grid-Stat runs and looking
for
> issues there. If I had your data and Grid-Stat config file in front
of me,
> I'd plot it using MET's plot_data_plane tool and then loop through
several
> cases to try to identify issues.
>
> You checked on the units. What about the accumulation intervals?
What is
> the accumulation interval is used for the probability forecasts?
Does that
> match the CCPA accumulation interval?
>
> Thanks,
> John
>
>
>
>
>
> On Fri, Sep 30, 2016 at 11:46 AM, Tatiana Burek <tatiana at ucar.edu>
wrote:
>
> > I put a tar file with MET data that Ben used to build the
reliability
> > diagram ( 'HREFX-AAS' model only):
> > dakota : d3/projects/METViewer/rely
> >
> > I also added an image with one mode (plot_20160930_172724.png)l
and data
> > that is coming from database and used by the Rscript to build the
plot
> > (sum_data.txt).
> > I noticed that the SQL query uses sum function to aggregate oy_i
and on_i
> > by thresh_i and model. I added a file with DB data without
aggregation (
> > real_data.txt ) .
> >
> > Tatiana
> >
> > On Tue, Sep 27, 2016 at 10:04 AM, Tara Jensen <jensen at ucar.edu>
wrote:
> >
> >> The diagrams do look a little strange. The diagram is more
likely to be
> >> between the two lines. First question that comes to my mind is:
Is he
> >> computing from VSDB or MET output... If VSDB - error could be
coming in
> >> during mapping of VSDB to METV tables. If MET - maybe an error
is
> coming
> >> in during aggregation. A second question: How many cases are
included
> in
> >> the aggregation.
> >>
> >>
> >> ---------- Forwarded message ----------
> >> From: Benjamin Blake - NOAA Affiliate via RT <met_help at ucar.edu>
> >> Date: Tue, Sep 27, 2016 at 9:48 AM
> >> Subject: [rt.rap.ucar.edu #78097] Reliability Diagrams in
METViewer
> >> To:
> >>
> >>
> >>
> >> Tue Sep 27 09:48:46 2016: Request 78097 was acted upon.
> >> Transaction: Ticket created by benjamin.blake at noaa.gov
> >> Queue: met_help
> >> Subject: Reliability Diagrams in METViewer
> >> Owner: Nobody
> >> Requestors: benjamin.blake at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78097 >
> >>
> >>
> >> Hi,
> >>
> >> I think there may be a possible issue with METViewer's
reliability
> >> diagrams.
> >>
> >> Here is a ROC curve plot and a reliability diagram for
probability
> >> forecasts I generated from the HREF and HREFx. The ROC curves
look fine
> >> but the reliability diagram says that all of the models are way
below
> the
> >> no skill line which seems odd to me.
> >>
> >> Thanks!
> >> Ben Blake
> >>
> >>
> >>
> >>
> >> --
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> Tara Jensen
> >> Project Manager II
> >> NCAR RAL and DTC
> >> PO Box 3000, Boulder, Colorado 80307 USA
> >> +1 303-497-8479 jensen at ucar.edu
> >>
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list