[Met_help] [rt.rap.ucar.edu #95665] History for Brier Skill Score
John Halley Gotway via RT
met_help at ucar.edu
Tue Jun 23 12:06:20 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
MET helpers, I need help to compare the ensemble forecast and
deterministic runs (non-ensemble). I had set up the scripts and configure
files which seemed to work for ensemble verifications. But I am not sure
how to change the configure files so I can compare the ensemble forecast
against the non-ensembles runs.
1. Is the Brier Skill Score the best plot that I can use METViewer to do
the comparison?
2. How to change my configure file so I can make SSC plot using METViewer?
My script is on Hera:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
I had set the non-ensemble runs that I want to compare with in the script
(DETER1 and DETER2)
The configure files could be found from the script above.
Let me know if anything is not clear and thank you for helping me.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Brier Skill Score
From: Minna Win
Time: Thu Jun 18 15:48:21 2020
Hello Binyu,
It looks like you have a lot of questions pertaining to setting up
your
ensemble stat config files and comparing ensemble data with non-
ensemble
data. I'm assigning this to John Halley Gotway, please allow a few
business days for a full response.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: Brier Skill Score
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
>
>
> Hello,
>
> MET helpers, I need help to compare the ensemble forecast and
> deterministic runs (non-ensemble). I had set up the scripts and
configure
> files which seemed to work for ensemble verifications. But I am not
sure
> how to change the configure files so I can compare the ensemble
forecast
> against the non-ensembles runs.
>
> 1. Is the Brier Skill Score the best plot that I can use METViewer
to do
> the comparison?
> 2. How to change my configure file so I can make SSC plot using
METViewer?
>
> My script is on Hera:
>
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> I had set the non-ensemble runs that I want to compare with in the
script
> (DETER1 and DETER2)
> The configure files could be found from the script above.
>
> Let me know if anything is not clear and thank you for helping me.
>
> Binyu
>
>
------------------------------------------------
Subject: Brier Skill Score
From: John Halley Gotway
Time: Thu Jun 18 15:59:34 2020
Binyu,
I'm having a difficult time understanding your question.
Are you asking for help to create a plot with METviewer to compare
ensemble
statistics to those produced for individual models?
Or are you asking for help modifying your shell script and/or Grid-
Stat or
Point-Stat config files to verify individual models?
Thanks,
John
On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
>
> Hello Binyu,
>
> It looks like you have a lot of questions pertaining to setting up
your
> ensemble stat config files and comparing ensemble data with non-
ensemble
> data. I'm assigning this to John Halley Gotway, please allow a few
> business days for a full response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: Brier Skill Score
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> >
> >
> > Hello,
> >
> > MET helpers, I need help to compare the ensemble forecast and
> > deterministic runs (non-ensemble). I had set up the scripts and
configure
> > files which seemed to work for ensemble verifications. But I am
not sure
> > how to change the configure files so I can compare the ensemble
forecast
> > against the non-ensembles runs.
> >
> > 1. Is the Brier Skill Score the best plot that I can use METViewer
to do
> > the comparison?
> > 2. How to change my configure file so I can make SSC plot using
> METViewer?
> >
> > My script is on Hera:
> >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > I had set the non-ensemble runs that I want to compare with in the
script
> > (DETER1 and DETER2)
> > The configure files could be found from the script above.
> >
> > Let me know if anything is not clear and thank you for helping me.
> >
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: binyu.wang at noaa.gov
Time: Thu Jun 18 19:59:57 2020
Thank you for your reply. John.
I need help to modify the shell script and ensemble-stat/Grid-Stat
config
files, so I could compare my ensemble model vs. the non-ensemble runs
by
looking at the BSS using METViwer.
I am not sure how to change the configure files to do that.
Binyu
On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I'm having a difficult time understanding your question.
>
> Are you asking for help to create a plot with METviewer to compare
ensemble
> statistics to those produced for individual models?
>
> Or are you asking for help modifying your shell script and/or Grid-
Stat or
> Point-Stat config files to verify individual models?
>
> Thanks,
> John
>
>
> On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> >
> > Hello Binyu,
> >
> > It looks like you have a lot of questions pertaining to setting up
your
> > ensemble stat config files and comparing ensemble data with non-
ensemble
> > data. I'm assigning this to John Halley Gotway, please allow a
few
> > business days for a full response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: Brier Skill Score
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> >
> > >
> > >
> > > Hello,
> > >
> > > MET helpers, I need help to compare the ensemble forecast and
> > > deterministic runs (non-ensemble). I had set up the scripts and
> configure
> > > files which seemed to work for ensemble verifications. But I am
not
> sure
> > > how to change the configure files so I can compare the ensemble
> forecast
> > > against the non-ensembles runs.
> > >
> > > 1. Is the Brier Skill Score the best plot that I can use
METViewer to
> do
> > > the comparison?
> > > 2. How to change my configure file so I can make SSC plot using
> > METViewer?
> > >
> > > My script is on Hera:
> > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > I had set the non-ensemble runs that I want to compare with in
the
> script
> > > (DETER1 and DETER2)
> > > The configure files could be found from the script above.
> > >
> > > Let me know if anything is not clear and thank you for helping
me.
> > >
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: John Halley Gotway
Time: Fri Jun 19 10:45:15 2020
Binyu,
One obvious point of comparison is comparing statistics for the
ensemble
mean field to statistics for individual deterministic members. It's
often
the case that the ensemble mean outperforms the members mainly because
it's
a much smoother forecast and smoother forecasts usually outperform
sharper
ones when measured with traditional statistics, such as RMSE.
However, you mentioned wanting to use the Brier Skill Score for the
comparison. MET computes Brier Score and Brier Skill Score when
evaluating
probabilistic fields, not deterministic ones. So you'll have BRIER and
BSS
for your ensemble-derived probabilities, but you won't have it for the
deterministic models. It is not obvious to me how to do a "fair"
comparison
between ensemble probabilities and deterministic model output. When
evaluating against point observations, Point-Stat does compute
probabilistic line types using the HiRA methodology. However those
probabilities are defined using a spatial distribution within each
field
rather than deriving them from multiple ensemble members. So while
technically you could compare an ensemble-derived BSS to a HiRA-
derived
BSS, I really wouldn't know how to interpret the results.
Perhaps ensemble probabilities are compared to deterministic models
routinely some way at EMC, and I'm just not familiar with it.
I'd recommend asking other folks at EMC for advice on exactly what
processing logic should be applied. If you can provide a clear
description
of how you'd like to process the data, hopefully I can advise on which
tools to run and how to configure them.
Thanks,
John
On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
>
> Thank you for your reply. John.
>
> I need help to modify the shell script and ensemble-stat/Grid-Stat
config
> files, so I could compare my ensemble model vs. the non-ensemble
runs by
> looking at the BSS using METViwer.
> I am not sure how to change the configure files to do that.
>
> Binyu
>
> On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I'm having a difficult time understanding your question.
> >
> > Are you asking for help to create a plot with METviewer to compare
> ensemble
> > statistics to those produced for individual models?
> >
> > Or are you asking for help modifying your shell script and/or
Grid-Stat
> or
> > Point-Stat config files to verify individual models?
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > >
> > > Hello Binyu,
> > >
> > > It looks like you have a lot of questions pertaining to setting
up your
> > > ensemble stat config files and comparing ensemble data with
> non-ensemble
> > > data. I'm assigning this to John Halley Gotway, please allow a
few
> > > business days for a full response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax: 303-497-8401
> > >
> > >
> > >
> > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: Brier Skill Score
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > MET helpers, I need help to compare the ensemble forecast and
> > > > deterministic runs (non-ensemble). I had set up the scripts
and
> > configure
> > > > files which seemed to work for ensemble verifications. But I
am not
> > sure
> > > > how to change the configure files so I can compare the
ensemble
> > forecast
> > > > against the non-ensembles runs.
> > > >
> > > > 1. Is the Brier Skill Score the best plot that I can use
METViewer to
> > do
> > > > the comparison?
> > > > 2. How to change my configure file so I can make SSC plot
using
> > > METViewer?
> > > >
> > > > My script is on Hera:
> > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > I had set the non-ensemble runs that I want to compare with in
the
> > script
> > > > (DETER1 and DETER2)
> > > > The configure files could be found from the script above.
> > > >
> > > > Let me know if anything is not clear and thank you for helping
me.
> > > >
> > > > Binyu
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: binyu.wang at noaa.gov
Time: Fri Jun 19 11:00:01 2020
Hello,
John, thank you for your suggestion.
If you look at the plot at section V from the link below: (I also
attached
the plot to this email)
https://www.wpc.ncep.noaa.gov/ensembletraining/ (by the way, Binbin
said
the y-axis is wrong, it should be "BSS" instead of "BS")
Anyhow, this is the kind of plot I want to make (compare BSS b/t
ensemble
and deterministic runs), can we use MET/METviewer to make the same
plot?
If so how to write the configure files? Thank you.
Binyu
On Fri, Jun 19, 2020 at 12:45 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> One obvious point of comparison is comparing statistics for the
ensemble
> mean field to statistics for individual deterministic members. It's
often
> the case that the ensemble mean outperforms the members mainly
because it's
> a much smoother forecast and smoother forecasts usually outperform
sharper
> ones when measured with traditional statistics, such as RMSE.
>
> However, you mentioned wanting to use the Brier Skill Score for the
> comparison. MET computes Brier Score and Brier Skill Score when
evaluating
> probabilistic fields, not deterministic ones. So you'll have BRIER
and BSS
> for your ensemble-derived probabilities, but you won't have it for
the
> deterministic models. It is not obvious to me how to do a "fair"
comparison
> between ensemble probabilities and deterministic model output. When
> evaluating against point observations, Point-Stat does compute
> probabilistic line types using the HiRA methodology. However those
> probabilities are defined using a spatial distribution within each
field
> rather than deriving them from multiple ensemble members. So while
> technically you could compare an ensemble-derived BSS to a HiRA-
derived
> BSS, I really wouldn't know how to interpret the results.
>
> Perhaps ensemble probabilities are compared to deterministic models
> routinely some way at EMC, and I'm just not familiar with it.
>
> I'd recommend asking other folks at EMC for advice on exactly what
> processing logic should be applied. If you can provide a clear
description
> of how you'd like to process the data, hopefully I can advise on
which
> tools to run and how to configure them.
>
> Thanks,
> John
>
> On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> >
> > Thank you for your reply. John.
> >
> > I need help to modify the shell script and ensemble-stat/Grid-
Stat
> config
> > files, so I could compare my ensemble model vs. the non-ensemble
runs by
> > looking at the BSS using METViwer.
> > I am not sure how to change the configure files to do that.
> >
> > Binyu
> >
> > On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I'm having a difficult time understanding your question.
> > >
> > > Are you asking for help to create a plot with METviewer to
compare
> > ensemble
> > > statistics to those produced for individual models?
> > >
> > > Or are you asking for help modifying your shell script and/or
Grid-Stat
> > or
> > > Point-Stat config files to verify individual models?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
>
> > > >
> > > > Hello Binyu,
> > > >
> > > > It looks like you have a lot of questions pertaining to
setting up
> your
> > > > ensemble stat config files and comparing ensemble data with
> > non-ensemble
> > > > data. I'm assigning this to John Halley Gotway, please allow
a few
> > > > business days for a full response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax: 303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Brier Skill Score
> > > > > Owner: Nobody
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > MET helpers, I need help to compare the ensemble forecast
and
> > > > > deterministic runs (non-ensemble). I had set up the scripts
and
> > > configure
> > > > > files which seemed to work for ensemble verifications. But I
am not
> > > sure
> > > > > how to change the configure files so I can compare the
ensemble
> > > forecast
> > > > > against the non-ensembles runs.
> > > > >
> > > > > 1. Is the Brier Skill Score the best plot that I can use
METViewer
> to
> > > do
> > > > > the comparison?
> > > > > 2. How to change my configure file so I can make SSC plot
using
> > > > METViewer?
> > > > >
> > > > > My script is on Hera:
> > > > >
> > > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > > I had set the non-ensemble runs that I want to compare with
in the
> > > script
> > > > > (DETER1 and DETER2)
> > > > > The configure files could be found from the script above.
> > > > >
> > > > > Let me know if anything is not clear and thank you for
helping me.
> > > > >
> > > > > Binyu
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: John Halley Gotway
Time: Fri Jun 19 12:01:23 2020
Binyu,
Reading the caption of that graphic, it is clear to me that this
involves
the application of binned climatologies. Yes, it is true that the
Point-Stat, Grid-Stat, and Ensemble-Stat tools do include logic for
handling climatological bins, as defined by the climo_mean,
climo_stdev,
and climo_cdf config file entries. However, I suspect that these tools
are
not doing everything Binbin would like/need them to do with this data.
Currently, in Grid-Stat, when the user specifies the climo_mean,
climo_stdev, and climo_cdf config file entries, it can subset the
global
matched pairs into bins based on where the observation falls relative
to
the climatological distribution at the point, compute stats separately
for
each of the climo bins, and then report the mean of the statistics
across
those bins.
Additionally, Grid-Stat can define thresholds relative to the climo
distribution, like ">CDP90", meaning greater than the 90-th percentile
of
the climatological distribution at each point.
However, to my knowledge, none of this functionality produces a Brier
Score
or Brier Skill Score from deterministic model data.
Based on that, my answer is, no, I do not think that you're currently
able
to use MET/METviewer to replicate the plot you sent.
I've cc'ed John Opatz and Tara Jensen on this email, a scientist and
project manager for METplus. I'd recommend John Opatz engaging with
you and
Binbin to define what additional functionality should be added to
Grid-Stat
to support this.
Thanks,
John
On Fri, Jun 19, 2020 at 11:00 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
>
> Hello,
>
> John, thank you for your suggestion.
> If you look at the plot at section V from the link below: (I also
attached
> the plot to this email)
> https://www.wpc.ncep.noaa.gov/ensembletraining/ (by the way, Binbin
said
> the y-axis is wrong, it should be "BSS" instead of "BS")
> Anyhow, this is the kind of plot I want to make (compare BSS b/t
ensemble
> and deterministic runs), can we use MET/METviewer to make the same
plot?
> If so how to write the configure files? Thank you.
>
> Binyu
>
> On Fri, Jun 19, 2020 at 12:45 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Binyu,
> >
> > One obvious point of comparison is comparing statistics for the
ensemble
> > mean field to statistics for individual deterministic members.
It's often
> > the case that the ensemble mean outperforms the members mainly
because
> it's
> > a much smoother forecast and smoother forecasts usually outperform
> sharper
> > ones when measured with traditional statistics, such as RMSE.
> >
> > However, you mentioned wanting to use the Brier Skill Score for
the
> > comparison. MET computes Brier Score and Brier Skill Score when
> evaluating
> > probabilistic fields, not deterministic ones. So you'll have BRIER
and
> BSS
> > for your ensemble-derived probabilities, but you won't have it for
the
> > deterministic models. It is not obvious to me how to do a "fair"
> comparison
> > between ensemble probabilities and deterministic model output.
When
> > evaluating against point observations, Point-Stat does compute
> > probabilistic line types using the HiRA methodology. However those
> > probabilities are defined using a spatial distribution within each
field
> > rather than deriving them from multiple ensemble members. So while
> > technically you could compare an ensemble-derived BSS to a HiRA-
derived
> > BSS, I really wouldn't know how to interpret the results.
> >
> > Perhaps ensemble probabilities are compared to deterministic
models
> > routinely some way at EMC, and I'm just not familiar with it.
> >
> > I'd recommend asking other folks at EMC for advice on exactly what
> > processing logic should be applied. If you can provide a clear
> description
> > of how you'd like to process the data, hopefully I can advise on
which
> > tools to run and how to configure them.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > >
> > > Thank you for your reply. John.
> > >
> > > I need help to modify the shell script and ensemble-stat/Grid-
Stat
> > config
> > > files, so I could compare my ensemble model vs. the non-ensemble
runs
> by
> > > looking at the BSS using METViwer.
> > > I am not sure how to change the configure files to do that.
> > >
> > > Binyu
> > >
> > > On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I'm having a difficult time understanding your question.
> > > >
> > > > Are you asking for help to create a plot with METviewer to
compare
> > > ensemble
> > > > statistics to those produced for individual models?
> > > >
> > > > Or are you asking for help modifying your shell script and/or
> Grid-Stat
> > > or
> > > > Point-Stat config files to verify individual models?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT
<met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > > > >
> > > > > Hello Binyu,
> > > > >
> > > > > It looks like you have a lot of questions pertaining to
setting up
> > your
> > > > > ensemble stat config files and comparing ensemble data with
> > > non-ensemble
> > > > > data. I'm assigning this to John Halley Gotway, please
allow a few
> > > > > business days for a full response.
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > > ---------------
> > > > > Minna Win
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax: 303-497-8401
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Brier Skill Score
> > > > > > Owner: Nobody
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > MET helpers, I need help to compare the ensemble forecast
and
> > > > > > deterministic runs (non-ensemble). I had set up the
scripts and
> > > > configure
> > > > > > files which seemed to work for ensemble verifications. But
I am
> not
> > > > sure
> > > > > > how to change the configure files so I can compare the
ensemble
> > > > forecast
> > > > > > against the non-ensembles runs.
> > > > > >
> > > > > > 1. Is the Brier Skill Score the best plot that I can use
> METViewer
> > to
> > > > do
> > > > > > the comparison?
> > > > > > 2. How to change my configure file so I can make SSC plot
using
> > > > > METViewer?
> > > > > >
> > > > > > My script is on Hera:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > > > I had set the non-ensemble runs that I want to compare
with in
> the
> > > > script
> > > > > > (DETER1 and DETER2)
> > > > > > The configure files could be found from the script above.
> > > > > >
> > > > > > Let me know if anything is not clear and thank you for
helping
> me.
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: binyu.wang at noaa.gov
Time: Sun Jun 21 09:00:11 2020
John,
Thank you. Currently MET is not able to do any probabilistic
calculation/evaluation is ONLY for ensemble, NOT for deterministic
runs, is
that correct?
Binyu
On Fri, Jun 19, 2020 at 2:01 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> Reading the caption of that graphic, it is clear to me that this
involves
> the application of binned climatologies. Yes, it is true that the
> Point-Stat, Grid-Stat, and Ensemble-Stat tools do include logic for
> handling climatological bins, as defined by the climo_mean,
climo_stdev,
> and climo_cdf config file entries. However, I suspect that these
tools are
> not doing everything Binbin would like/need them to do with this
data.
>
> Currently, in Grid-Stat, when the user specifies the climo_mean,
> climo_stdev, and climo_cdf config file entries, it can subset the
global
> matched pairs into bins based on where the observation falls
relative to
> the climatological distribution at the point, compute stats
separately for
> each of the climo bins, and then report the mean of the statistics
across
> those bins.
>
> Additionally, Grid-Stat can define thresholds relative to the climo
> distribution, like ">CDP90", meaning greater than the 90-th
percentile of
> the climatological distribution at each point.
>
> However, to my knowledge, none of this functionality produces a
Brier Score
> or Brier Skill Score from deterministic model data.
>
> Based on that, my answer is, no, I do not think that you're
currently able
> to use MET/METviewer to replicate the plot you sent.
>
> I've cc'ed John Opatz and Tara Jensen on this email, a scientist and
> project manager for METplus. I'd recommend John Opatz engaging with
you and
> Binbin to define what additional functionality should be added to
Grid-Stat
> to support this.
>
> Thanks,
> John
>
>
> On Fri, Jun 19, 2020 at 11:00 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> >
> > Hello,
> >
> > John, thank you for your suggestion.
> > If you look at the plot at section V from the link below: (I also
> attached
> > the plot to this email)
> > https://www.wpc.ncep.noaa.gov/ensembletraining/ (by the way,
Binbin
> said
> > the y-axis is wrong, it should be "BSS" instead of "BS")
> > Anyhow, this is the kind of plot I want to make (compare BSS b/t
> ensemble
> > and deterministic runs), can we use MET/METviewer to make the same
plot?
> > If so how to write the configure files? Thank you.
> >
> > Binyu
> >
> > On Fri, Jun 19, 2020 at 12:45 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Binyu,
> > >
> > > One obvious point of comparison is comparing statistics for the
> ensemble
> > > mean field to statistics for individual deterministic members.
It's
> often
> > > the case that the ensemble mean outperforms the members mainly
because
> > it's
> > > a much smoother forecast and smoother forecasts usually
outperform
> > sharper
> > > ones when measured with traditional statistics, such as RMSE.
> > >
> > > However, you mentioned wanting to use the Brier Skill Score for
the
> > > comparison. MET computes Brier Score and Brier Skill Score when
> > evaluating
> > > probabilistic fields, not deterministic ones. So you'll have
BRIER and
> > BSS
> > > for your ensemble-derived probabilities, but you won't have it
for the
> > > deterministic models. It is not obvious to me how to do a "fair"
> > comparison
> > > between ensemble probabilities and deterministic model output.
When
> > > evaluating against point observations, Point-Stat does compute
> > > probabilistic line types using the HiRA methodology. However
those
> > > probabilities are defined using a spatial distribution within
each
> field
> > > rather than deriving them from multiple ensemble members. So
while
> > > technically you could compare an ensemble-derived BSS to a HiRA-
derived
> > > BSS, I really wouldn't know how to interpret the results.
> > >
> > > Perhaps ensemble probabilities are compared to deterministic
models
> > > routinely some way at EMC, and I'm just not familiar with it.
> > >
> > > I'd recommend asking other folks at EMC for advice on exactly
what
> > > processing logic should be applied. If you can provide a clear
> > description
> > > of how you'd like to process the data, hopefully I can advise on
which
> > > tools to run and how to configure them.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
>
> > > >
> > > > Thank you for your reply. John.
> > > >
> > > > I need help to modify the shell script and ensemble-
stat/Grid-Stat
> > > config
> > > > files, so I could compare my ensemble model vs. the non-
ensemble runs
> > by
> > > > looking at the BSS using METViwer.
> > > > I am not sure how to change the configure files to do that.
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > I'm having a difficult time understanding your question.
> > > > >
> > > > > Are you asking for help to create a plot with METviewer to
compare
> > > > ensemble
> > > > > statistics to those produced for individual models?
> > > > >
> > > > > Or are you asking for help modifying your shell script
and/or
> > Grid-Stat
> > > > or
> > > > > Point-Stat config files to verify individual models?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT <
> met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > > > > >
> > > > > > Hello Binyu,
> > > > > >
> > > > > > It looks like you have a lot of questions pertaining to
setting
> up
> > > your
> > > > > > ensemble stat config files and comparing ensemble data
with
> > > > non-ensemble
> > > > > > data. I'm assigning this to John Halley Gotway, please
allow a
> few
> > > > > > business days for a full response.
> > > > > >
> > > > > > Regards,
> > > > > > Minna
> > > > > > ---------------
> > > > > > Minna Win
> > > > > > National Center for Atmospheric Research
> > > > > > Developmental Testbed Center
> > > > > > Phone: 303-497-8423
> > > > > > Fax: 303-497-8401
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted upon.
> > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: Brier Skill Score
> > > > > > > Owner: Nobody
> > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > MET helpers, I need help to compare the ensemble
forecast and
> > > > > > > deterministic runs (non-ensemble). I had set up the
scripts and
> > > > > configure
> > > > > > > files which seemed to work for ensemble verifications.
But I am
> > not
> > > > > sure
> > > > > > > how to change the configure files so I can compare the
ensemble
> > > > > forecast
> > > > > > > against the non-ensembles runs.
> > > > > > >
> > > > > > > 1. Is the Brier Skill Score the best plot that I can use
> > METViewer
> > > to
> > > > > do
> > > > > > > the comparison?
> > > > > > > 2. How to change my configure file so I can make SSC
plot using
> > > > > > METViewer?
> > > > > > >
> > > > > > > My script is on Hera:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > > > > I had set the non-ensemble runs that I want to compare
with in
> > the
> > > > > script
> > > > > > > (DETER1 and DETER2)
> > > > > > > The configure files could be found from the script
above.
> > > > > > >
> > > > > > > Let me know if anything is not clear and thank you for
helping
> > me.
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: John Halley Gotway
Time: Mon Jun 22 09:46:21 2020
Binyu,
Yes, that is basically correct. MET only computes probabilistic
statistics
in the PSTD, PRC, PJC, and PCT line types when evaluating
probabilistic
forecasts, not deterministic ones.
The only exception to that is the HiRA methodology in Point-Stat. It
reads
a deterministic forecast, derives a probability value at each grid
point
based on the event frequency within the surrounding spatial
neighborhood,
and computes probabilistic statistics from that.
If there is additional functionality required, I'd recommend engaging
with
John Optaz and Tara Jensen to define it.
Hope that helps clarify.
John
On Sun, Jun 21, 2020 at 9:00 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
>
> John,
>
> Thank you. Currently MET is not able to do any probabilistic
> calculation/evaluation is ONLY for ensemble, NOT for deterministic
runs, is
> that correct?
>
> Binyu
>
> On Fri, Jun 19, 2020 at 2:01 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > Reading the caption of that graphic, it is clear to me that this
involves
> > the application of binned climatologies. Yes, it is true that the
> > Point-Stat, Grid-Stat, and Ensemble-Stat tools do include logic
for
> > handling climatological bins, as defined by the climo_mean,
climo_stdev,
> > and climo_cdf config file entries. However, I suspect that these
tools
> are
> > not doing everything Binbin would like/need them to do with this
data.
> >
> > Currently, in Grid-Stat, when the user specifies the climo_mean,
> > climo_stdev, and climo_cdf config file entries, it can subset the
global
> > matched pairs into bins based on where the observation falls
relative to
> > the climatological distribution at the point, compute stats
separately
> for
> > each of the climo bins, and then report the mean of the statistics
across
> > those bins.
> >
> > Additionally, Grid-Stat can define thresholds relative to the
climo
> > distribution, like ">CDP90", meaning greater than the 90-th
percentile of
> > the climatological distribution at each point.
> >
> > However, to my knowledge, none of this functionality produces a
Brier
> Score
> > or Brier Skill Score from deterministic model data.
> >
> > Based on that, my answer is, no, I do not think that you're
currently
> able
> > to use MET/METviewer to replicate the plot you sent.
> >
> > I've cc'ed John Opatz and Tara Jensen on this email, a scientist
and
> > project manager for METplus. I'd recommend John Opatz engaging
with you
> and
> > Binbin to define what additional functionality should be added to
> Grid-Stat
> > to support this.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 19, 2020 at 11:00 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > >
> > > Hello,
> > >
> > > John, thank you for your suggestion.
> > > If you look at the plot at section V from the link below: (I
also
> > attached
> > > the plot to this email)
> > > https://www.wpc.ncep.noaa.gov/ensembletraining/ (by the way,
Binbin
> > said
> > > the y-axis is wrong, it should be "BSS" instead of "BS")
> > > Anyhow, this is the kind of plot I want to make (compare BSS
b/t
> > ensemble
> > > and deterministic runs), can we use MET/METviewer to make the
same
> plot?
> > > If so how to write the configure files? Thank you.
> > >
> > > Binyu
> > >
> > > On Fri, Jun 19, 2020 at 12:45 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Binyu,
> > > >
> > > > One obvious point of comparison is comparing statistics for
the
> > ensemble
> > > > mean field to statistics for individual deterministic members.
It's
> > often
> > > > the case that the ensemble mean outperforms the members mainly
> because
> > > it's
> > > > a much smoother forecast and smoother forecasts usually
outperform
> > > sharper
> > > > ones when measured with traditional statistics, such as RMSE.
> > > >
> > > > However, you mentioned wanting to use the Brier Skill Score
for the
> > > > comparison. MET computes Brier Score and Brier Skill Score
when
> > > evaluating
> > > > probabilistic fields, not deterministic ones. So you'll have
BRIER
> and
> > > BSS
> > > > for your ensemble-derived probabilities, but you won't have it
for
> the
> > > > deterministic models. It is not obvious to me how to do a
"fair"
> > > comparison
> > > > between ensemble probabilities and deterministic model output.
When
> > > > evaluating against point observations, Point-Stat does compute
> > > > probabilistic line types using the HiRA methodology. However
those
> > > > probabilities are defined using a spatial distribution within
each
> > field
> > > > rather than deriving them from multiple ensemble members. So
while
> > > > technically you could compare an ensemble-derived BSS to a
> HiRA-derived
> > > > BSS, I really wouldn't know how to interpret the results.
> > > >
> > > > Perhaps ensemble probabilities are compared to deterministic
models
> > > > routinely some way at EMC, and I'm just not familiar with it.
> > > >
> > > > I'd recommend asking other folks at EMC for advice on exactly
what
> > > > processing logic should be applied. If you can provide a clear
> > > description
> > > > of how you'd like to process the data, hopefully I can advise
on
> which
> > > > tools to run and how to configure them.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > > > >
> > > > > Thank you for your reply. John.
> > > > >
> > > > > I need help to modify the shell script and ensemble-
stat/Grid-Stat
> > > > config
> > > > > files, so I could compare my ensemble model vs. the non-
ensemble
> runs
> > > by
> > > > > looking at the BSS using METViwer.
> > > > > I am not sure how to change the configure files to do that.
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > I'm having a difficult time understanding your question.
> > > > > >
> > > > > > Are you asking for help to create a plot with METviewer to
> compare
> > > > > ensemble
> > > > > > statistics to those produced for individual models?
> > > > > >
> > > > > > Or are you asking for help modifying your shell script
and/or
> > > Grid-Stat
> > > > > or
> > > > > > Point-Stat config files to verify individual models?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT <
> > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> >
> > > > > > >
> > > > > > > Hello Binyu,
> > > > > > >
> > > > > > > It looks like you have a lot of questions pertaining to
setting
> > up
> > > > your
> > > > > > > ensemble stat config files and comparing ensemble data
with
> > > > > non-ensemble
> > > > > > > data. I'm assigning this to John Halley Gotway, please
allow a
> > few
> > > > > > > business days for a full response.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Minna
> > > > > > > ---------------
> > > > > > > Minna Win
> > > > > > > National Center for Atmospheric Research
> > > > > > > Developmental Testbed Center
> > > > > > > Phone: 303-497-8423
> > > > > > > Fax: 303-497-8401
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted
upon.
> > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: Brier Skill Score
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > MET helpers, I need help to compare the ensemble
forecast and
> > > > > > > > deterministic runs (non-ensemble). I had set up the
scripts
> and
> > > > > > configure
> > > > > > > > files which seemed to work for ensemble verifications.
But I
> am
> > > not
> > > > > > sure
> > > > > > > > how to change the configure files so I can compare the
> ensemble
> > > > > > forecast
> > > > > > > > against the non-ensembles runs.
> > > > > > > >
> > > > > > > > 1. Is the Brier Skill Score the best plot that I can
use
> > > METViewer
> > > > to
> > > > > > do
> > > > > > > > the comparison?
> > > > > > > > 2. How to change my configure file so I can make SSC
plot
> using
> > > > > > > METViewer?
> > > > > > > >
> > > > > > > > My script is on Hera:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > > > > > I had set the non-ensemble runs that I want to compare
with
> in
> > > the
> > > > > > script
> > > > > > > > (DETER1 and DETER2)
> > > > > > > > The configure files could be found from the script
above.
> > > > > > > >
> > > > > > > > Let me know if anything is not clear and thank you for
> helping
> > > me.
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Brier Skill Score
From: binyu.wang at noaa.gov
Time: Tue Jun 23 07:08:15 2020
John,
Thank you for the information.
Binyu
On Mon, Jun 22, 2020 at 11:46 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> Yes, that is basically correct. MET only computes probabilistic
statistics
> in the PSTD, PRC, PJC, and PCT line types when evaluating
probabilistic
> forecasts, not deterministic ones.
>
> The only exception to that is the HiRA methodology in Point-Stat. It
reads
> a deterministic forecast, derives a probability value at each grid
point
> based on the event frequency within the surrounding spatial
neighborhood,
> and computes probabilistic statistics from that.
>
> If there is additional functionality required, I'd recommend
engaging with
> John Optaz and Tara Jensen to define it.
>
> Hope that helps clarify.
>
> John
>
> On Sun, Jun 21, 2020 at 9:00 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> >
> > John,
> >
> > Thank you. Currently MET is not able to do any probabilistic
> > calculation/evaluation is ONLY for ensemble, NOT for deterministic
runs,
> is
> > that correct?
> >
> > Binyu
> >
> > On Fri, Jun 19, 2020 at 2:01 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > Reading the caption of that graphic, it is clear to me that this
> involves
> > > the application of binned climatologies. Yes, it is true that
the
> > > Point-Stat, Grid-Stat, and Ensemble-Stat tools do include logic
for
> > > handling climatological bins, as defined by the climo_mean,
> climo_stdev,
> > > and climo_cdf config file entries. However, I suspect that these
tools
> > are
> > > not doing everything Binbin would like/need them to do with this
data.
> > >
> > > Currently, in Grid-Stat, when the user specifies the climo_mean,
> > > climo_stdev, and climo_cdf config file entries, it can subset
the
> global
> > > matched pairs into bins based on where the observation falls
relative
> to
> > > the climatological distribution at the point, compute stats
separately
> > for
> > > each of the climo bins, and then report the mean of the
statistics
> across
> > > those bins.
> > >
> > > Additionally, Grid-Stat can define thresholds relative to the
climo
> > > distribution, like ">CDP90", meaning greater than the 90-th
percentile
> of
> > > the climatological distribution at each point.
> > >
> > > However, to my knowledge, none of this functionality produces a
Brier
> > Score
> > > or Brier Skill Score from deterministic model data.
> > >
> > > Based on that, my answer is, no, I do not think that you're
currently
> > able
> > > to use MET/METviewer to replicate the plot you sent.
> > >
> > > I've cc'ed John Opatz and Tara Jensen on this email, a scientist
and
> > > project manager for METplus. I'd recommend John Opatz engaging
with you
> > and
> > > Binbin to define what additional functionality should be added
to
> > Grid-Stat
> > > to support this.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Jun 19, 2020 at 11:00 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
>
> > > >
> > > > Hello,
> > > >
> > > > John, thank you for your suggestion.
> > > > If you look at the plot at section V from the link below: (I
also
> > > attached
> > > > the plot to this email)
> > > > https://www.wpc.ncep.noaa.gov/ensembletraining/ (by the way,
Binbin
> > > said
> > > > the y-axis is wrong, it should be "BSS" instead of "BS")
> > > > Anyhow, this is the kind of plot I want to make (compare BSS
b/t
> > > ensemble
> > > > and deterministic runs), can we use MET/METviewer to make the
same
> > plot?
> > > > If so how to write the configure files? Thank you.
> > > >
> > > > Binyu
> > > >
> > > > On Fri, Jun 19, 2020 at 12:45 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > One obvious point of comparison is comparing statistics for
the
> > > ensemble
> > > > > mean field to statistics for individual deterministic
members. It's
> > > often
> > > > > the case that the ensemble mean outperforms the members
mainly
> > because
> > > > it's
> > > > > a much smoother forecast and smoother forecasts usually
outperform
> > > > sharper
> > > > > ones when measured with traditional statistics, such as
RMSE.
> > > > >
> > > > > However, you mentioned wanting to use the Brier Skill Score
for the
> > > > > comparison. MET computes Brier Score and Brier Skill Score
when
> > > > evaluating
> > > > > probabilistic fields, not deterministic ones. So you'll have
BRIER
> > and
> > > > BSS
> > > > > for your ensemble-derived probabilities, but you won't have
it for
> > the
> > > > > deterministic models. It is not obvious to me how to do a
"fair"
> > > > comparison
> > > > > between ensemble probabilities and deterministic model
output. When
> > > > > evaluating against point observations, Point-Stat does
compute
> > > > > probabilistic line types using the HiRA methodology. However
those
> > > > > probabilities are defined using a spatial distribution
within each
> > > field
> > > > > rather than deriving them from multiple ensemble members. So
while
> > > > > technically you could compare an ensemble-derived BSS to a
> > HiRA-derived
> > > > > BSS, I really wouldn't know how to interpret the results.
> > > > >
> > > > > Perhaps ensemble probabilities are compared to deterministic
models
> > > > > routinely some way at EMC, and I'm just not familiar with
it.
> > > > >
> > > > > I'd recommend asking other folks at EMC for advice on
exactly what
> > > > > processing logic should be applied. If you can provide a
clear
> > > > description
> > > > > of how you'd like to process the data, hopefully I can
advise on
> > which
> > > > > tools to run and how to configure them.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jun 18, 2020 at 8:00 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665 >
> > > > > >
> > > > > > Thank you for your reply. John.
> > > > > >
> > > > > > I need help to modify the shell script and
> ensemble-stat/Grid-Stat
> > > > > config
> > > > > > files, so I could compare my ensemble model vs. the non-
ensemble
> > runs
> > > > by
> > > > > > looking at the BSS using METViwer.
> > > > > > I am not sure how to change the configure files to do
that.
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Thu, Jun 18, 2020 at 5:59 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > I'm having a difficult time understanding your question.
> > > > > > >
> > > > > > > Are you asking for help to create a plot with METviewer
to
> > compare
> > > > > > ensemble
> > > > > > > statistics to those produced for individual models?
> > > > > > >
> > > > > > > Or are you asking for help modifying your shell script
and/or
> > > > Grid-Stat
> > > > > > or
> > > > > > > Point-Stat config files to verify individual models?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 18, 2020 at 3:48 PM Minna Win via RT <
> > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > >
> > > > > > > >
> > > > > > > > Hello Binyu,
> > > > > > > >
> > > > > > > > It looks like you have a lot of questions pertaining
to
> setting
> > > up
> > > > > your
> > > > > > > > ensemble stat config files and comparing ensemble data
with
> > > > > > non-ensemble
> > > > > > > > data. I'm assigning this to John Halley Gotway,
please
> allow a
> > > few
> > > > > > > > business days for a full response.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Minna
> > > > > > > > ---------------
> > > > > > > > Minna Win
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Developmental Testbed Center
> > > > > > > > Phone: 303-497-8423
> > > > > > > > Fax: 303-497-8401
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 18, 2020 at 3:12 PM binyu.wang at noaa.gov
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thu Jun 18 15:12:04 2020: Request 95665 was acted
upon.
> > > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: Brier Skill Score
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95665
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > MET helpers, I need help to compare the ensemble
forecast
> and
> > > > > > > > > deterministic runs (non-ensemble). I had set up the
scripts
> > and
> > > > > > > configure
> > > > > > > > > files which seemed to work for ensemble
verifications. But
> I
> > am
> > > > not
> > > > > > > sure
> > > > > > > > > how to change the configure files so I can compare
the
> > ensemble
> > > > > > > forecast
> > > > > > > > > against the non-ensembles runs.
> > > > > > > > >
> > > > > > > > > 1. Is the Brier Skill Score the best plot that I can
use
> > > > METViewer
> > > > > to
> > > > > > > do
> > > > > > > > > the comparison?
> > > > > > > > > 2. How to change my configure file so I can make SSC
plot
> > using
> > > > > > > > METViewer?
> > > > > > > > >
> > > > > > > > > My script is on Hera:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_kasat.sh
> > > > > > > > > I had set the non-ensemble runs that I want to
compare with
> > in
> > > > the
> > > > > > > script
> > > > > > > > > (DETER1 and DETER2)
> > > > > > > > > The configure files could be found from the script
above.
> > > > > > > > >
> > > > > > > > > Let me know if anything is not clear and thank you
for
> > helping
> > > > me.
> > > > > > > > >
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list