[Met_help] [rt.rap.ucar.edu #91999] History for Generating reliability diagram from ensemble_stat output

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello MET helpdesk,

I am working to compare the NSSL Warn-on-Forecast 18-member, 6-hour accumulated precipitation against Stage-IV-observed accumulated precipitation using MET tools  “pcp_combine”, “gen_vx_mask”, “ensemble_stat”, and “stat_analysis”. The goal is to be able to take the statistical output from ensemble_stat to generate a reliability diagram for accumulated precipitation thresholded for 6-hour totals >1". In my current understanding, “ensemble_stat” outputs the relative frequency of ensemble members matching or exceeding the threshold, but this is where I am stuck. I don’t see values that would be useful for plotting on a reliability diagram. My guess is that I would have to pass the netCDF ensemble file (e.g., ensemble_stat_20180720_090000V_ens.nc) through to the grid_stat tool, but I am not sure if this is the right approach. What should the next appropriate step be to generate statistics that would be useful in creating the reliability diagram?

I am still spinning up on my proficiency with MET and perhaps I am missing something along the way so before I proceed without making any missteps, any advice/guidance would be greatly appreciated.

Thanks!

-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory

Phone: (405) 325-1688



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Generating reliability diagram from ensemble_stat output
From: John Halley Gotway
Time: Tue Sep 10 19:30:24 2019

Brian,

Yes, you've got it exactly right.  You can run Ensemble-Stat to
compute
ensemble relative frequencies (i.e. probability forecasts) which are
written to the NetCDF output file. And those probability forecasts can
be
evaluated using Point-Stat or Grid-Stat.  Both Point-Stat and Grid-
Stat
compute an Nx2 probabilistic contingency table from which
probabilistic
statistics, like reliability, are computed.  Those Nx2 tables are
written
to the PCT output line type.

We designed MET this way because there are many ways of deriving
probabilistic forecasts.  Computing relative frequencies in Ensemble-
Stat
are the simplest way, but there's many ways of doing ensemble
post-processing to derive better calibrated and more reliable
probabilistic
forecasts.  And they all can be evaluated using Grid-Stat or Point-
Stat.

Practically speaking, we run Ensemble-Stat to derive simple
probabilities,
verify them using Grid-Stat or Point-Stat, load the PCT output lines
into
METviewer, and then use METviewer to create reliability diagrams.

Hope that helps clarify.

Thanks,
John Halley Gotway

On Tue, Sep 10, 2019 at 4:55 PM Matilla, Brian C. via RT
<met_help at ucar.edu>
wrote:

>
> Tue Sep 10 16:54:45 2019: Request 91999 was acted upon.
> Transaction: Ticket created by brian.matilla at ou.edu
>        Queue: met_help
>      Subject: Generating reliability diagram from ensemble_stat
output
>        Owner: Nobody
>   Requestors: brian.matilla at ou.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91999 >
>
>
> Hello MET helpdesk,
>
> I am working to compare the NSSL Warn-on-Forecast 18-member, 6-hour
> accumulated precipitation against Stage-IV-observed accumulated
> precipitation using MET tools  “pcp_combine”, “gen_vx_mask”,
> “ensemble_stat”, and “stat_analysis”. The goal is to be able to take
the
> statistical output from ensemble_stat to generate a reliability
diagram for
> accumulated precipitation thresholded for 6-hour totals >1". In my
current
> understanding, “ensemble_stat” outputs the relative frequency of
ensemble
> members matching or exceeding the threshold, but this is where I am
stuck.
> I don’t see values that would be useful for plotting on a
reliability
> diagram. My guess is that I would have to pass the netCDF ensemble
file
> (e.g., ensemble_stat_20180720_090000V_ens.nc) through to the
grid_stat
> tool, but I am not sure if this is the right approach. What should
the next
> appropriate step be to generate statistics that would be useful in
creating
> the reliability diagram?
>
> I am still spinning up on my proficiency with MET and perhaps I am
missing
> something along the way so before I proceed without making any
missteps,
> any advice/guidance would be greatly appreciated.
>
> Thanks!
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> Phone: (405) 325-1688
>
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91999] Generating reliability diagram from ensemble_stat output
From: Brian Matilla - NOAA Affiliate
Time: Wed Sep 11 16:19:01 2019

Hi John,

Thanks so much for the prompt reply and feedback! I’m happy to know
I’m on the right track with it and can try to take it from here for
the time being. I’ll keep working along and should something else come
up, I’ll open up a separate ticket.

Thanks again!

-Brian
> On Sep 10, 2019, at 8:30 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Brian,
>
> Yes, you've got it exactly right.  You can run Ensemble-Stat to
compute
> ensemble relative frequencies (i.e. probability forecasts) which are
> written to the NetCDF output file. And those probability forecasts
can be
> evaluated using Point-Stat or Grid-Stat.  Both Point-Stat and Grid-
Stat
> compute an Nx2 probabilistic contingency table from which
probabilistic
> statistics, like reliability, are computed.  Those Nx2 tables are
written
> to the PCT output line type.
>
> We designed MET this way because there are many ways of deriving
> probabilistic forecasts.  Computing relative frequencies in
Ensemble-Stat
> are the simplest way, but there's many ways of doing ensemble
> post-processing to derive better calibrated and more reliable
probabilistic
> forecasts.  And they all can be evaluated using Grid-Stat or Point-
Stat.
>
> Practically speaking, we run Ensemble-Stat to derive simple
probabilities,
> verify them using Grid-Stat or Point-Stat, load the PCT output lines
into
> METviewer, and then use METviewer to create reliability diagrams.
>
> Hope that helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Tue, Sep 10, 2019 at 4:55 PM Matilla, Brian C. via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> Tue Sep 10 16:54:45 2019: Request 91999 was acted upon.
>> Transaction: Ticket created by brian.matilla at ou.edu
>>       Queue: met_help
>>     Subject: Generating reliability diagram from ensemble_stat
output
>>       Owner: Nobody
>>  Requestors: brian.matilla at ou.edu
>>      Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91999 >
>>
>>
>> Hello MET helpdesk,
>>
>> I am working to compare the NSSL Warn-on-Forecast 18-member, 6-hour
>> accumulated precipitation against Stage-IV-observed accumulated
>> precipitation using MET tools  “pcp_combine”, “gen_vx_mask”,
>> “ensemble_stat”, and “stat_analysis”. The goal is to be able to
take the
>> statistical output from ensemble_stat to generate a reliability
diagram for
>> accumulated precipitation thresholded for 6-hour totals >1". In my
current
>> understanding, “ensemble_stat” outputs the relative frequency of
ensemble
>> members matching or exceeding the threshold, but this is where I am
stuck.
>> I don’t see values that would be useful for plotting on a
reliability
>> diagram. My guess is that I would have to pass the netCDF ensemble
file
>> (e.g., ensemble_stat_20180720_090000V_ens.nc) through to the
grid_stat
>> tool, but I am not sure if this is the right approach. What should
the next
>> appropriate step be to generate statistics that would be useful in
creating
>> the reliability diagram?
>>
>> I am still spinning up on my proficiency with MET and perhaps I am
missing
>> something along the way so before I proceed without making any
missteps,
>> any advice/guidance would be greatly appreciated.
>>
>> Thanks!
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>> Phone: (405) 325-1688
>>
>>
>>
>



------------------------------------------------