[Met_help] [rt.rap.ucar.edu #93133] History for Question about percentile choices in ensemble_stat

Mon Nov 18 09:57:07 MST 2019

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi again,

I’m following up on a previous ticket regarding the use of percentile thresholds in ensemble_stat and grid_stat. So currently I am looking to examine the 75th and 90th percentile for each of the models I am working with and compute probabilities based on the percentiles for each forecast hour over x members, but when I do apply those thresholds (ex. >SFP75, >SFP90) in the ensemble_stat config file, I get an error stating that the percentile thresholds were used before they were set. I am a bit perplexed about this message and was wondering if I could receive some advice on how to correct that.

Thanks!

-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory

Phone: (405) 325-1688

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Question about percentile choices in ensemble_stat
From: John Halley Gotway
Time: Thu Nov 14 10:26:15 2019

Brian,

The percentile thresholds, such as >SFP75, should work just find in
Grid-Stat.

However, they will not work in Ensemble-Stat in met-8.1.  This all
comes
down to the data over which those percentiles are computed.  In met-
8.1,
they are computed from a single field of data.  However, the logic
does not
currently support the computation of percentiles over all of the
ensemble
members.

One of the main motivations for adding percentile thresholds to MET
was
enabling easy bias-correction.  And that drove our design choices.  Of
course, mathematically it would be possible to compute them over all
ensemble members, but that doesn't currently exist.  I assume that
you'd
like to see that functionality added?

For now, you could compute the percentiles outside of MET.  For
example,
compute the 75th percentile across the ensemble members as being
">2.5".
And then specify that in the MET config file as ">USP75(2.5)", where
USP
means User-Specified-Percentile.  That should apply the >2.5 threshold
and
set "FCST_THRESH = >USP75(2.5)" in the output files.

Thanks,
John

On Thu, Nov 14, 2019 at 10:04 AM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> Thu Nov 14 10:04:35 2019: Request 93133 was acted upon.
> Transaction: Ticket created by brian.matilla at noaa.gov
>        Queue: met_help
>      Subject: Question about percentile choices in ensemble_stat
>        Owner: Nobody
>   Requestors: brian.matilla at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93133 >
>
>
> Hi again,
>
> I’m following up on a previous ticket regarding the use of
percentile
> thresholds in ensemble_stat and grid_stat. So currently I am looking
to
> examine the 75th and 90th percentile for each of the models I am
working
> with and compute probabilities based on the percentiles for each
forecast
> hour over x members, but when I do apply those thresholds (ex.
>SFP75,
> >SFP90) in the ensemble_stat config file, I get an error stating
that the
> percentile thresholds were used before they were set. I am a bit
perplexed
> about this message and was wondering if I could receive some advice
on how
> to correct that.
>
> Thanks!
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> Phone: (405) 325-1688
>
>
>
>

------------------------------------------------
Subject: Question about percentile choices in ensemble_stat
From: John Halley Gotway
Time: Thu Nov 14 10:30:22 2019

Brian,

FYI, here's a development issue that already existed related to this
request:
https://github.com/NCAR/MET/issues/1112

It originally said Series-Analysis and MTD, but I just added Ensemble-
Stat
to that list as well.

John

On Thu, Nov 14, 2019 at 10:26 AM The RT System itself via RT <
met_help at ucar.edu> wrote:

>
> Thu Nov 14 10:26:15 2019: Request 93133 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>        Queue: met_help
>      Subject: Question about percentile choices in ensemble_stat
>        Owner: johnhg
>   Requestors: brian.matilla at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93133 >
>
>
> This transaction appears to have no content
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #93133] Question about percentile choices in ensemble_stat
From: Brian Matilla - NOAA Affiliate
Time: Fri Nov 15 17:54:55 2019

Hi John,

Thank you so much for sending me the development issue on Github. Yes,
I do think such a feature would be useful in future versions of MET.

So I was able to generate the percentile thresholds for each member
with Grid-Stat and I back-checked them with Python and all seems to
check out nicely. Sorry to pester you with so many recent questions on
this… I learn a lot more each day with what MET can do.

Thanks as always,

-Brian
> On Nov 14, 2019, at 11:26 AM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Brian,
>
> The percentile thresholds, such as >SFP75, should work just find in
> Grid-Stat.
>
> However, they will not work in Ensemble-Stat in met-8.1.  This all
comes
> down to the data over which those percentiles are computed.  In met-
8.1,
> they are computed from a single field of data.  However, the logic
does not
> currently support the computation of percentiles over all of the
ensemble
> members.
>
> One of the main motivations for adding percentile thresholds to MET
was
> enabling easy bias-correction.  And that drove our design choices.
Of
> course, mathematically it would be possible to compute them over all
> ensemble members, but that doesn't currently exist.  I assume that
you'd
> like to see that functionality added?
>
> For now, you could compute the percentiles outside of MET.  For
example,
> compute the 75th percentile across the ensemble members as being
">2.5".
> And then specify that in the MET config file as ">USP75(2.5)", where
USP
> means User-Specified-Percentile.  That should apply the >2.5
threshold and
> set "FCST_THRESH = >USP75(2.5)" in the output files.
>
> Thanks,
> John
>
>
>
> On Thu, Nov 14, 2019 at 10:04 AM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> Thu Nov 14 10:04:35 2019: Request 93133 was acted upon.
>> Transaction: Ticket created by brian.matilla at noaa.gov
>>       Queue: met_help
>>     Subject: Question about percentile choices in ensemble_stat
>>       Owner: Nobody
>>  Requestors: brian.matilla at noaa.gov
>>      Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93133 >
>>
>>
>> Hi again,
>>
>> I’m following up on a previous ticket regarding the use of
percentile
>> thresholds in ensemble_stat and grid_stat. So currently I am
looking to
>> examine the 75th and 90th percentile for each of the models I am
working
>> with and compute probabilities based on the percentiles for each
forecast
>> hour over x members, but when I do apply those thresholds (ex.
>SFP75,
>>> SFP90) in the ensemble_stat config file, I get an error stating
that the
>> percentile thresholds were used before they were set. I am a bit
perplexed
>> about this message and was wondering if I could receive some advice
on how
>> to correct that.
>>
>> Thanks!
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>> Phone: (405) 325-1688
>>
>>
>>
>>
>

------------------------------------------------