[Met_help] [rt.rap.ucar.edu #100255] History for reliability calculation

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello,

I was trying to use ensemble forecasts and satellite obs. to make a
reliability plot (see attachment 1).  The plot indicates that no matter
what the forecast frequency is, the observed frequency is near 1. But
actually I see lots of 'misses' - areas where the model has ash but the
observations don't. I have 31 members (no missing file). I think the way
that MET use to calculate the probability is "ONLY if all the ensemble
members have overlap with the grid that obs, exist,  then it will be
counted as forecast "catch" the observation", is that correct? Which means
if one grid was caught only by 30 members, the grid will be treaded as
missed by the forecast

The plot I got using MET/METviewer (attachment 1) is much different than
using python (see attachment 2), and here is the way that was used:
---------------------------------------------
For the reliability curve, at each point, I count how many ensemble members
are above the threshold. That number determines the x bin value. So if I
use the 31 bins and 20 members are above threshold, then I am in the 20th
bin/x value. Then I see if the obs is above threshold. If it is, then a 1
goes into that bin. If it isn't then a 0 goes into that bin.

After I have gone through all the points, to get the y values I sum the
values in the bins and divide by the total number of them to get the
fraction observed.
-------------------------------------------

So can I use the similar way in MET? I mean to calculate probability
without using all ensemble members?  I did set "vld_thresh = 0.1" in my
config file.

Here is the ensemble config file that I use:

/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5


My obs:

/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/


My forecast:

/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS

Thank you.


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: reliability calculation
From: Minna Win
Time: Thu Jun 17 08:25:18 2021

Hi Binyu,

It looks like you have multiple questions that will require more than
one
person to respond.:

1) Your expected reliability plot in METviewer using MET ensemble
output on
satellite obs data is *not* what is expected and you want to know how
MET
is calculating the probability.

2)  You want to know how METviewer implements the generation of the
reliability diagram, so you can reproduce it in Python?

Let me know if I've captured your questions correctly, so I can find
the
appropriate people to contact.

Regards,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
>        Queue: met_help
>      Subject: reliability calculation
>        Owner: Nobody
>   Requestors: binyu.wang at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>
>
> Hello,
>
> I was trying to use ensemble forecasts and satellite obs. to make a
> reliability plot (see attachment 1).  The plot indicates that no
matter
> what the forecast frequency is, the observed frequency is near 1.
But
> actually I see lots of 'misses' - areas where the model has ash but
the
> observations don't. I have 31 members (no missing file). I think the
way
> that MET use to calculate the probability is "ONLY if all the
ensemble
> members have overlap with the grid that obs, exist,  then it will be
> counted as forecast "catch" the observation", is that correct? Which
means
> if one grid was caught only by 30 members, the grid will be treaded
as
> missed by the forecast
>
> The plot I got using MET/METviewer (attachment 1) is much different
than
> using python (see attachment 2), and here is the way that was used:
> ---------------------------------------------
> For the reliability curve, at each point, I count how many ensemble
members
> are above the threshold. That number determines the x bin value. So
if I
> use the 31 bins and 20 members are above threshold, then I am in the
20th
> bin/x value. Then I see if the obs is above threshold. If it is,
then a 1
> goes into that bin. If it isn't then a 0 goes into that bin.
>
> After I have gone through all the points, to get the y values I sum
the
> values in the bins and divide by the total number of them to get the
> fraction observed.
> -------------------------------------------
>
> So can I use the similar way in MET? I mean to calculate probability
> without using all ensemble members?  I did set "vld_thresh = 0.1" in
my
> config file.
>
> Here is the ensemble config file that I use:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
>
>
> My obs:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
>
>
> My forecast:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
>
> Thank you.
>
>

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Thu Jun 17 08:45:39 2021

Hello Minna,

Thank you for your quick response. In order not to give extra work to
too
many people, I think at this moment I only want to find an expert to
understand how MET
is calculating the probability and make sure the config files that I
am
using are correct. I think the way that Python is used is pretty
similar to
MET, however the plot is a big difference.
Best
Binyu

On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT <met_help at ucar.edu>
wrote:

> Hi Binyu,
>
> It looks like you have multiple questions that will require more
than one
> person to respond.:
>
> 1) Your expected reliability plot in METviewer using MET ensemble
output on
> satellite obs data is *not* what is expected and you want to know
how MET
> is calculating the probability.
>
> 2)  You want to know how METviewer implements the generation of the
> reliability diagram, so you can reproduce it in Python?
>
> Let me know if I've captured your questions correctly, so I can find
the
> appropriate people to contact.
>
> Regards,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
> ---------------
>
>
>
> On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> >        Queue: met_help
> >      Subject: reliability calculation
> >        Owner: Nobody
> >   Requestors: binyu.wang at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> >
> >
> > Hello,
> >
> > I was trying to use ensemble forecasts and satellite obs. to make
a
> > reliability plot (see attachment 1).  The plot indicates that no
matter
> > what the forecast frequency is, the observed frequency is near 1.
But
> > actually I see lots of 'misses' - areas where the model has ash
but the
> > observations don't. I have 31 members (no missing file). I think
the way
> > that MET use to calculate the probability is "ONLY if all the
ensemble
> > members have overlap with the grid that obs, exist,  then it will
be
> > counted as forecast "catch" the observation", is that correct?
Which
> means
> > if one grid was caught only by 30 members, the grid will be
treaded as
> > missed by the forecast
> >
> > The plot I got using MET/METviewer (attachment 1) is much
different than
> > using python (see attachment 2), and here is the way that was
used:
> > ---------------------------------------------
> > For the reliability curve, at each point, I count how many
ensemble
> members
> > are above the threshold. That number determines the x bin value.
So if I
> > use the 31 bins and 20 members are above threshold, then I am in
the 20th
> > bin/x value. Then I see if the obs is above threshold. If it is,
then a 1
> > goes into that bin. If it isn't then a 0 goes into that bin.
> >
> > After I have gone through all the points, to get the y values I
sum the
> > values in the bins and divide by the total number of them to get
the
> > fraction observed.
> > -------------------------------------------
> >
> > So can I use the similar way in MET? I mean to calculate
probability
> > without using all ensemble members?  I did set "vld_thresh = 0.1"
in my
> > config file.
> >
> > Here is the ensemble config file that I use:
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> >
> >
> > My obs:
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> >
> >
> > My forecast:
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> >
> > Thank you.
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: John Opatz
Time: Thu Jun 17 10:00:52 2021

I think the most useful information to this ticket is here
https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file

Take a look at the entry for vld_thresh. This is the ratio of valid
data values to ensemble members for each grid point. If it falls below
the user's vld_thresh, ensemble-stat will write out bad data.

ens_thresh has a similar quality control behavior, but it's for the
entire ensemble field vs each grid point. So if there are 10 ensemble
members and 1 is a bad member but your ens_thresh is set to 1.0,
ensemble_stat will quit.

I think the default for both of these is 1.0. Maybe they can try to
set it to 0.75, or 0.5 and rerun their results?

------------------------------------------------
Subject: reliability calculation
From: Minna Win
Time: Thu Jun 17 10:01:34 2021

Hi Binyu,

Here's an explanation from John Halley Gotway (who is supposed to be
on
vacation):

For each field listed in the ens dictionary of the config file and for
each
cat_thresh threshold defined for that field, ensemble-stat derives an
ensemble probability as the ratio of ensemble members that meet the
cat_thresh threshold criteria. For example, for 2-m temperature and a
threshold of <273 over an ensemble of size 10, if 6 of the 10 members
have
freezing temperatures at some grid point, then the probability there
is
just 6/10=0.6. The vld_thresh defines how to handle missing data.

I hope that helps.

Regards,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>
> Hello Minna,
>
> Thank you for your quick response. In order not to give extra work
to too
> many people, I think at this moment I only want to find an expert to
> understand how MET
> is calculating the probability and make sure the config files that I
am
> using are correct. I think the way that Python is used is pretty
similar to
> MET, however the plot is a big difference.
> Best
> Binyu
>
> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Binyu,
> >
> > It looks like you have multiple questions that will require more
than one
> > person to respond.:
> >
> > 1) Your expected reliability plot in METviewer using MET ensemble
output
> on
> > satellite obs data is *not* what is expected and you want to know
how MET
> > is calculating the probability.
> >
> > 2)  You want to know how METviewer implements the generation of
the
> > reliability diagram, so you can reproduce it in Python?
> >
> > Let me know if I've captured your questions correctly, so I can
find the
> > appropriate people to contact.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > Pronouns: she/her
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> > ---------------
> >
> >
> >
> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > >        Queue: met_help
> > >      Subject: reliability calculation
> > >        Owner: Nobody
> > >   Requestors: binyu.wang at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
> >
> > >
> > >
> > > Hello,
> > >
> > > I was trying to use ensemble forecasts and satellite obs. to
make a
> > > reliability plot (see attachment 1).  The plot indicates that no
matter
> > > what the forecast frequency is, the observed frequency is near
1. But
> > > actually I see lots of 'misses' - areas where the model has ash
but the
> > > observations don't. I have 31 members (no missing file). I think
the
> way
> > > that MET use to calculate the probability is "ONLY if all the
ensemble
> > > members have overlap with the grid that obs, exist,  then it
will be
> > > counted as forecast "catch" the observation", is that correct?
Which
> > means
> > > if one grid was caught only by 30 members, the grid will be
treaded as
> > > missed by the forecast
> > >
> > > The plot I got using MET/METviewer (attachment 1) is much
different
> than
> > > using python (see attachment 2), and here is the way that was
used:
> > > ---------------------------------------------
> > > For the reliability curve, at each point, I count how many
ensemble
> > members
> > > are above the threshold. That number determines the x bin value.
So if
> I
> > > use the 31 bins and 20 members are above threshold, then I am in
the
> 20th
> > > bin/x value. Then I see if the obs is above threshold. If it is,
then
> a 1
> > > goes into that bin. If it isn't then a 0 goes into that bin.
> > >
> > > After I have gone through all the points, to get the y values I
sum the
> > > values in the bins and divide by the total number of them to get
the
> > > fraction observed.
> > > -------------------------------------------
> > >
> > > So can I use the similar way in MET? I mean to calculate
probability
> > > without using all ensemble members?  I did set "vld_thresh =
0.1" in my
> > > config file.
> > >
> > > Here is the ensemble config file that I use:
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > >
> > >
> > > My obs:
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > >
> > >
> > > My forecast:
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > >
> > > Thank you.
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: Minna Win
Time: Thu Jun 17 10:05:40 2021

Hi Binyu,

John Opatz also has some good information:

----------------
I think the most useful information to this ticket is here
https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file

Take a look at the entry for vld_thresh. This is the ratio of valid
data
values to ensemble members for each grid point. If it falls below the
user's vld_thresh, ensemble-stat will write out bad data.

ens_thresh has a similar quality control behavior, but it's for the
entire
ensemble field vs each grid point. So if there are 10 ensemble members
and
1 is a bad member but your ens_thresh is set to 1.0, ensemble_stat
will
quit.

I think the default for both of these is 1.0. Maybe they can try to
set it
to 0.75, or 0.5 and rerun their results?

--------------

Regards,
Minna

---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister
<minnawin at ucar.edu>
wrote:

> Hi Binyu,
>
> Here's an explanation from John Halley Gotway (who is supposed to be
on
> vacation):
>
> For each field listed in the ens dictionary of the config file and
for
> each cat_thresh threshold defined for that field, ensemble-stat
derives an
> ensemble probability as the ratio of ensemble members that meet the
> cat_thresh threshold criteria. For example, for 2-m temperature and
a
> threshold of <273 over an ensemble of size 10, if 6 of the 10
members have
> freezing temperatures at some grid point, then the probability there
is
> just 6/10=0.6. The vld_thresh defines how to handle missing data.
>
> I hope that helps.
>
> Regards,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
> ---------------
>
>
>
> On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>>
>> Hello Minna,
>>
>> Thank you for your quick response. In order not to give extra work
to too
>> many people, I think at this moment I only want to find an expert
to
>> understand how MET
>> is calculating the probability and make sure the config files that
I am
>> using are correct. I think the way that Python is used is pretty
similar
>> to
>> MET, however the plot is a big difference.
>> Best
>> Binyu
>>
>> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Binyu,
>> >
>> > It looks like you have multiple questions that will require more
than
>> one
>> > person to respond.:
>> >
>> > 1) Your expected reliability plot in METviewer using MET ensemble
>> output on
>> > satellite obs data is *not* what is expected and you want to know
how
>> MET
>> > is calculating the probability.
>> >
>> > 2)  You want to know how METviewer implements the generation of
the
>> > reliability diagram, so you can reproduce it in Python?
>> >
>> > Let me know if I've captured your questions correctly, so I can
find the
>> > appropriate people to contact.
>> >
>> > Regards,
>> > Minna
>> > ---------------
>> > Minna Win
>> > Pronouns: she/her
>> > National Center for Atmospheric Research
>> > Developmental Testbed Center
>> > Phone: 303-497-8423
>> > Fax:   303-497-8401
>> > ---------------
>> >
>> >
>> >
>> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
>> > > Transaction: Ticket created by binyu.wang at noaa.gov
>> > >        Queue: met_help
>> > >      Subject: reliability calculation
>> > >        Owner: Nobody
>> > >   Requestors: binyu.wang at noaa.gov
>> > >       Status: new
>> > >  Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>> > >
>> > >
>> > > Hello,
>> > >
>> > > I was trying to use ensemble forecasts and satellite obs. to
make a
>> > > reliability plot (see attachment 1).  The plot indicates that
no
>> matter
>> > > what the forecast frequency is, the observed frequency is near
1. But
>> > > actually I see lots of 'misses' - areas where the model has ash
but
>> the
>> > > observations don't. I have 31 members (no missing file). I
think the
>> way
>> > > that MET use to calculate the probability is "ONLY if all the
ensemble
>> > > members have overlap with the grid that obs, exist,  then it
will be
>> > > counted as forecast "catch" the observation", is that correct?
Which
>> > means
>> > > if one grid was caught only by 30 members, the grid will be
treaded as
>> > > missed by the forecast
>> > >
>> > > The plot I got using MET/METviewer (attachment 1) is much
different
>> than
>> > > using python (see attachment 2), and here is the way that was
used:
>> > > ---------------------------------------------
>> > > For the reliability curve, at each point, I count how many
ensemble
>> > members
>> > > are above the threshold. That number determines the x bin
value. So
>> if I
>> > > use the 31 bins and 20 members are above threshold, then I am
in the
>> 20th
>> > > bin/x value. Then I see if the obs is above threshold. If it
is, then
>> a 1
>> > > goes into that bin. If it isn't then a 0 goes into that bin.
>> > >
>> > > After I have gone through all the points, to get the y values I
sum
>> the
>> > > values in the bins and divide by the total number of them to
get the
>> > > fraction observed.
>> > > -------------------------------------------
>> > >
>> > > So can I use the similar way in MET? I mean to calculate
probability
>> > > without using all ensemble members?  I did set "vld_thresh =
0.1" in
>> my
>> > > config file.
>> > >
>> > > Here is the ensemble config file that I use:
>> > >
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
>> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
>> > >
>> > >
>> > > My obs:
>> > >
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
>> > >
>> > >
>> > > My forecast:
>> > >
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
>> > >
>> > > Thank you.
>> > >
>> > >
>> >
>> >
>>
>>

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Fri Jun 18 16:19:57 2021

Hello Minna,

Thank you for forwarding this information to me. Based on what John
said
before: ens_thresh is used to specify the required ratio of ensemble
members that must be
present for ensemble_stat to run at all. I have no missing ensemble
members so there is no impact to change "ens_thresh" in my test.

Attached are the output from ensemble_stat and grid_stat (same
dataset).
The weird thing is: there are "9 pairs" from the first file, but "100
pairs" from the second one.
Based on the MET document, "9" in the first file (ECNT) refers to
"Count of
observations", and "100" from the second file (PCT) refers to  "Total
number of matched pairs", that should be the same thing, right? But
the
value is different, why?

Thank you.


On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT <met_help at ucar.edu>
wrote:

> Hi Binyu,
>
> John Opatz also has some good information:
>
> ----------------
> I think the most useful information to this ticket is here
>
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
>
> Take a look at the entry for vld_thresh. This is the ratio of valid
data
> values to ensemble members for each grid point. If it falls below
the
> user's vld_thresh, ensemble-stat will write out bad data.
>
> ens_thresh has a similar quality control behavior, but it's for the
entire
> ensemble field vs each grid point. So if there are 10 ensemble
members and
> 1 is a bad member but your ens_thresh is set to 1.0, ensemble_stat
will
> quit.
>
> I think the default for both of these is 1.0. Maybe they can try to
set it
> to 0.75, or 0.5 and rerun their results?
>
> --------------
>
> Regards,
> Minna
>
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
> ---------------
>
>
>
> On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> minnawin at ucar.edu>
> wrote:
>
> > Hi Binyu,
> >
> > Here's an explanation from John Halley Gotway (who is supposed to
be on
> > vacation):
> >
> > For each field listed in the ens dictionary of the config file and
for
> > each cat_thresh threshold defined for that field, ensemble-stat
derives
> an
> > ensemble probability as the ratio of ensemble members that meet
the
> > cat_thresh threshold criteria. For example, for 2-m temperature
and a
> > threshold of <273 over an ensemble of size 10, if 6 of the 10
members
> have
> > freezing temperatures at some grid point, then the probability
there is
> > just 6/10=0.6. The vld_thresh defines how to handle missing data.
> >
> > I hope that helps.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > Pronouns: she/her
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> > ---------------
> >
> >
> >
> > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> >>
> >> Hello Minna,
> >>
> >> Thank you for your quick response. In order not to give extra
work to
> too
> >> many people, I think at this moment I only want to find an expert
to
> >> understand how MET
> >> is calculating the probability and make sure the config files
that I am
> >> using are correct. I think the way that Python is used is pretty
similar
> >> to
> >> MET, however the plot is a big difference.
> >> Best
> >> Binyu
> >>
> >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hi Binyu,
> >> >
> >> > It looks like you have multiple questions that will require
more than
> >> one
> >> > person to respond.:
> >> >
> >> > 1) Your expected reliability plot in METviewer using MET
ensemble
> >> output on
> >> > satellite obs data is *not* what is expected and you want to
know how
> >> MET
> >> > is calculating the probability.
> >> >
> >> > 2)  You want to know how METviewer implements the generation of
the
> >> > reliability diagram, so you can reproduce it in Python?
> >> >
> >> > Let me know if I've captured your questions correctly, so I can
find
> the
> >> > appropriate people to contact.
> >> >
> >> > Regards,
> >> > Minna
> >> > ---------------
> >> > Minna Win
> >> > Pronouns: she/her
> >> > National Center for Atmospheric Research
> >> > Developmental Testbed Center
> >> > Phone: 303-497-8423
> >> > Fax:   303-497-8401
> >> > ---------------
> >> >
> >> >
> >> >
> >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> >> > >        Queue: met_help
> >> > >      Subject: reliability calculation
> >> > >        Owner: Nobody
> >> > >   Requestors: binyu.wang at noaa.gov
> >> > >       Status: new
> >> > >  Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> >> > >
> >> > >
> >> > > Hello,
> >> > >
> >> > > I was trying to use ensemble forecasts and satellite obs. to
make a
> >> > > reliability plot (see attachment 1).  The plot indicates that
no
> >> matter
> >> > > what the forecast frequency is, the observed frequency is
near 1.
> But
> >> > > actually I see lots of 'misses' - areas where the model has
ash but
> >> the
> >> > > observations don't. I have 31 members (no missing file). I
think the
> >> way
> >> > > that MET use to calculate the probability is "ONLY if all the
> ensemble
> >> > > members have overlap with the grid that obs, exist,  then it
will be
> >> > > counted as forecast "catch" the observation", is that
correct? Which
> >> > means
> >> > > if one grid was caught only by 30 members, the grid will be
treaded
> as
> >> > > missed by the forecast
> >> > >
> >> > > The plot I got using MET/METviewer (attachment 1) is much
different
> >> than
> >> > > using python (see attachment 2), and here is the way that was
used:
> >> > > ---------------------------------------------
> >> > > For the reliability curve, at each point, I count how many
ensemble
> >> > members
> >> > > are above the threshold. That number determines the x bin
value. So
> >> if I
> >> > > use the 31 bins and 20 members are above threshold, then I am
in the
> >> 20th
> >> > > bin/x value. Then I see if the obs is above threshold. If it
is,
> then
> >> a 1
> >> > > goes into that bin. If it isn't then a 0 goes into that bin.
> >> > >
> >> > > After I have gone through all the points, to get the y values
I sum
> >> the
> >> > > values in the bins and divide by the total number of them to
get the
> >> > > fraction observed.
> >> > > -------------------------------------------
> >> > >
> >> > > So can I use the similar way in MET? I mean to calculate
probability
> >> > > without using all ensemble members?  I did set "vld_thresh =
0.1" in
> >> my
> >> > > config file.
> >> > >
> >> > > Here is the ensemble config file that I use:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> >> > >
> >> > >
> >> > > My obs:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> >> > >
> >> > >
> >> > > My forecast:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> >> > >
> >> > > Thank you.
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>

------------------------------------------------
Subject: reliability calculation
From: Minna Win
Time: Fri Jun 18 16:34:54 2021

Hi Binyu,

It looks like you are trying to make sense of your grid-stat and
ensemble-stat output, which produce different output results.  I don't
have
any familiarity with either tool, I'll need to ask someone if they
know
what this means.  John Halley Gotway will be out of the office until
July 6
so I'll need to find someone else.

Regards,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Fri, Jun 18, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>
> Hello Minna,
>
> Thank you for forwarding this information to me. Based on what John
said
> before: ens_thresh is used to specify the required ratio of ensemble
> members that must be
> present for ensemble_stat to run at all. I have no missing ensemble
> members so there is no impact to change "ens_thresh" in my test.
>
> Attached are the output from ensemble_stat and grid_stat (same
dataset).
> The weird thing is: there are "9 pairs" from the first file, but
"100
> pairs" from the second one.
> Based on the MET document, "9" in the first file (ECNT) refers to
"Count of
> observations", and "100" from the second file (PCT) refers to
"Total
> number of matched pairs", that should be the same thing, right? But
the
> value is different, why?
>
> Thank you.
>
>
> On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Binyu,
> >
> > John Opatz also has some good information:
> >
> > ----------------
> > I think the most useful information to this ticket is here
> >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> >
> > Take a look at the entry for vld_thresh. This is the ratio of
valid data
> > values to ensemble members for each grid point. If it falls below
the
> > user's vld_thresh, ensemble-stat will write out bad data.
> >
> > ens_thresh has a similar quality control behavior, but it's for
the
> entire
> > ensemble field vs each grid point. So if there are 10 ensemble
members
> and
> > 1 is a bad member but your ens_thresh is set to 1.0, ensemble_stat
will
> > quit.
> >
> > I think the default for both of these is 1.0. Maybe they can try
to set
> it
> > to 0.75, or 0.5 and rerun their results?
> >
> > --------------
> >
> > Regards,
> > Minna
> >
> > ---------------
> > Minna Win
> > Pronouns: she/her
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> > ---------------
> >
> >
> >
> > On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> > minnawin at ucar.edu>
> > wrote:
> >
> > > Hi Binyu,
> > >
> > > Here's an explanation from John Halley Gotway (who is supposed
to be on
> > > vacation):
> > >
> > > For each field listed in the ens dictionary of the config file
and for
> > > each cat_thresh threshold defined for that field, ensemble-stat
derives
> > an
> > > ensemble probability as the ratio of ensemble members that meet
the
> > > cat_thresh threshold criteria. For example, for 2-m temperature
and a
> > > threshold of <273 over an ensemble of size 10, if 6 of the 10
members
> > have
> > > freezing temperatures at some grid point, then the probability
there is
> > > just 6/10=0.6. The vld_thresh defines how to handle missing
data.
> > >
> > > I hope that helps.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > Pronouns: she/her
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > > ---------------
> > >
> > >
> > >
> > > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
>
> > >>
> > >> Hello Minna,
> > >>
> > >> Thank you for your quick response. In order not to give extra
work to
> > too
> > >> many people, I think at this moment I only want to find an
expert to
> > >> understand how MET
> > >> is calculating the probability and make sure the config files
that I
> am
> > >> using are correct. I think the way that Python is used is
pretty
> similar
> > >> to
> > >> MET, however the plot is a big difference.
> > >> Best
> > >> Binyu
> > >>
> > >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT
<met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Hi Binyu,
> > >> >
> > >> > It looks like you have multiple questions that will require
more
> than
> > >> one
> > >> > person to respond.:
> > >> >
> > >> > 1) Your expected reliability plot in METviewer using MET
ensemble
> > >> output on
> > >> > satellite obs data is *not* what is expected and you want to
know
> how
> > >> MET
> > >> > is calculating the probability.
> > >> >
> > >> > 2)  You want to know how METviewer implements the generation
of the
> > >> > reliability diagram, so you can reproduce it in Python?
> > >> >
> > >> > Let me know if I've captured your questions correctly, so I
can find
> > the
> > >> > appropriate people to contact.
> > >> >
> > >> > Regards,
> > >> > Minna
> > >> > ---------------
> > >> > Minna Win
> > >> > Pronouns: she/her
> > >> > National Center for Atmospheric Research
> > >> > Developmental Testbed Center
> > >> > Phone: 303-497-8423
> > >> > Fax:   303-497-8401
> > >> > ---------------
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > >> > >        Queue: met_help
> > >> > >      Subject: reliability calculation
> > >> > >        Owner: Nobody
> > >> > >   Requestors: binyu.wang at noaa.gov
> > >> > >       Status: new
> > >> > >  Ticket <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > >> > >
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > > I was trying to use ensemble forecasts and satellite obs.
to make
> a
> > >> > > reliability plot (see attachment 1).  The plot indicates
that no
> > >> matter
> > >> > > what the forecast frequency is, the observed frequency is
near 1.
> > But
> > >> > > actually I see lots of 'misses' - areas where the model has
ash
> but
> > >> the
> > >> > > observations don't. I have 31 members (no missing file). I
think
> the
> > >> way
> > >> > > that MET use to calculate the probability is "ONLY if all
the
> > ensemble
> > >> > > members have overlap with the grid that obs, exist,  then
it will
> be
> > >> > > counted as forecast "catch" the observation", is that
correct?
> Which
> > >> > means
> > >> > > if one grid was caught only by 30 members, the grid will be
> treaded
> > as
> > >> > > missed by the forecast
> > >> > >
> > >> > > The plot I got using MET/METviewer (attachment 1) is much
> different
> > >> than
> > >> > > using python (see attachment 2), and here is the way that
was
> used:
> > >> > > ---------------------------------------------
> > >> > > For the reliability curve, at each point, I count how many
> ensemble
> > >> > members
> > >> > > are above the threshold. That number determines the x bin
value.
> So
> > >> if I
> > >> > > use the 31 bins and 20 members are above threshold, then I
am in
> the
> > >> 20th
> > >> > > bin/x value. Then I see if the obs is above threshold. If
it is,
> > then
> > >> a 1
> > >> > > goes into that bin. If it isn't then a 0 goes into that
bin.
> > >> > >
> > >> > > After I have gone through all the points, to get the y
values I
> sum
> > >> the
> > >> > > values in the bins and divide by the total number of them
to get
> the
> > >> > > fraction observed.
> > >> > > -------------------------------------------
> > >> > >
> > >> > > So can I use the similar way in MET? I mean to calculate
> probability
> > >> > > without using all ensemble members?  I did set "vld_thresh
= 0.1"
> in
> > >> my
> > >> > > config file.
> > >> > >
> > >> > > Here is the ensemble config file that I use:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > >> > >
> > >> > >
> > >> > > My obs:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > >> > >
> > >> > >
> > >> > > My forecast:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > >> > >
> > >> > > Thank you.
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: Minna Win
Time: Fri Jun 18 16:48:16 2021

Hi Binyu,

I took a look at the user documentation, and the ECNT line type
(ensemble
stat) is different from the PCT linetype (grid-stat).  Since ensemble
stat
and grid stat are doing different things, at first glance I wouldn't
expect
these two values to be the same.   I'll have to verify with a
scientist.

Regards,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Fri, Jun 18, 2021 at 4:20 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>
> Hello Minna,
>
> Thank you for forwarding this information to me. Based on what John
said
> before: ens_thresh is used to specify the required ratio of ensemble
> members that must be
> present for ensemble_stat to run at all. I have no missing ensemble
> members so there is no impact to change "ens_thresh" in my test.
>
> Attached are the output from ensemble_stat and grid_stat (same
dataset).
> The weird thing is: there are "9 pairs" from the first file, but
"100
> pairs" from the second one.
> Based on the MET document, "9" in the first file (ECNT) refers to
"Count of
> observations", and "100" from the second file (PCT) refers to
"Total
> number of matched pairs", that should be the same thing, right? But
the
> value is different, why?
>
> Thank you.
>
>
> On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Binyu,
> >
> > John Opatz also has some good information:
> >
> > ----------------
> > I think the most useful information to this ticket is here
> >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> >
> > Take a look at the entry for vld_thresh. This is the ratio of
valid data
> > values to ensemble members for each grid point. If it falls below
the
> > user's vld_thresh, ensemble-stat will write out bad data.
> >
> > ens_thresh has a similar quality control behavior, but it's for
the
> entire
> > ensemble field vs each grid point. So if there are 10 ensemble
members
> and
> > 1 is a bad member but your ens_thresh is set to 1.0, ensemble_stat
will
> > quit.
> >
> > I think the default for both of these is 1.0. Maybe they can try
to set
> it
> > to 0.75, or 0.5 and rerun their results?
> >
> > --------------
> >
> > Regards,
> > Minna
> >
> > ---------------
> > Minna Win
> > Pronouns: she/her
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> > ---------------
> >
> >
> >
> > On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> > minnawin at ucar.edu>
> > wrote:
> >
> > > Hi Binyu,
> > >
> > > Here's an explanation from John Halley Gotway (who is supposed
to be on
> > > vacation):
> > >
> > > For each field listed in the ens dictionary of the config file
and for
> > > each cat_thresh threshold defined for that field, ensemble-stat
derives
> > an
> > > ensemble probability as the ratio of ensemble members that meet
the
> > > cat_thresh threshold criteria. For example, for 2-m temperature
and a
> > > threshold of <273 over an ensemble of size 10, if 6 of the 10
members
> > have
> > > freezing temperatures at some grid point, then the probability
there is
> > > just 6/10=0.6. The vld_thresh defines how to handle missing
data.
> > >
> > > I hope that helps.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > Pronouns: she/her
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > > ---------------
> > >
> > >
> > >
> > > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
>
> > >>
> > >> Hello Minna,
> > >>
> > >> Thank you for your quick response. In order not to give extra
work to
> > too
> > >> many people, I think at this moment I only want to find an
expert to
> > >> understand how MET
> > >> is calculating the probability and make sure the config files
that I
> am
> > >> using are correct. I think the way that Python is used is
pretty
> similar
> > >> to
> > >> MET, however the plot is a big difference.
> > >> Best
> > >> Binyu
> > >>
> > >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT
<met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Hi Binyu,
> > >> >
> > >> > It looks like you have multiple questions that will require
more
> than
> > >> one
> > >> > person to respond.:
> > >> >
> > >> > 1) Your expected reliability plot in METviewer using MET
ensemble
> > >> output on
> > >> > satellite obs data is *not* what is expected and you want to
know
> how
> > >> MET
> > >> > is calculating the probability.
> > >> >
> > >> > 2)  You want to know how METviewer implements the generation
of the
> > >> > reliability diagram, so you can reproduce it in Python?
> > >> >
> > >> > Let me know if I've captured your questions correctly, so I
can find
> > the
> > >> > appropriate people to contact.
> > >> >
> > >> > Regards,
> > >> > Minna
> > >> > ---------------
> > >> > Minna Win
> > >> > Pronouns: she/her
> > >> > National Center for Atmospheric Research
> > >> > Developmental Testbed Center
> > >> > Phone: 303-497-8423
> > >> > Fax:   303-497-8401
> > >> > ---------------
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > >> > >        Queue: met_help
> > >> > >      Subject: reliability calculation
> > >> > >        Owner: Nobody
> > >> > >   Requestors: binyu.wang at noaa.gov
> > >> > >       Status: new
> > >> > >  Ticket <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > >> > >
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > > I was trying to use ensemble forecasts and satellite obs.
to make
> a
> > >> > > reliability plot (see attachment 1).  The plot indicates
that no
> > >> matter
> > >> > > what the forecast frequency is, the observed frequency is
near 1.
> > But
> > >> > > actually I see lots of 'misses' - areas where the model has
ash
> but
> > >> the
> > >> > > observations don't. I have 31 members (no missing file). I
think
> the
> > >> way
> > >> > > that MET use to calculate the probability is "ONLY if all
the
> > ensemble
> > >> > > members have overlap with the grid that obs, exist,  then
it will
> be
> > >> > > counted as forecast "catch" the observation", is that
correct?
> Which
> > >> > means
> > >> > > if one grid was caught only by 30 members, the grid will be
> treaded
> > as
> > >> > > missed by the forecast
> > >> > >
> > >> > > The plot I got using MET/METviewer (attachment 1) is much
> different
> > >> than
> > >> > > using python (see attachment 2), and here is the way that
was
> used:
> > >> > > ---------------------------------------------
> > >> > > For the reliability curve, at each point, I count how many
> ensemble
> > >> > members
> > >> > > are above the threshold. That number determines the x bin
value.
> So
> > >> if I
> > >> > > use the 31 bins and 20 members are above threshold, then I
am in
> the
> > >> 20th
> > >> > > bin/x value. Then I see if the obs is above threshold. If
it is,
> > then
> > >> a 1
> > >> > > goes into that bin. If it isn't then a 0 goes into that
bin.
> > >> > >
> > >> > > After I have gone through all the points, to get the y
values I
> sum
> > >> the
> > >> > > values in the bins and divide by the total number of them
to get
> the
> > >> > > fraction observed.
> > >> > > -------------------------------------------
> > >> > >
> > >> > > So can I use the similar way in MET? I mean to calculate
> probability
> > >> > > without using all ensemble members?  I did set "vld_thresh
= 0.1"
> in
> > >> my
> > >> > > config file.
> > >> > >
> > >> > > Here is the ensemble config file that I use:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > >> > >
> > >> > >
> > >> > > My obs:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > >> > >
> > >> > >
> > >> > > My forecast:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > >> > >
> > >> > > Thank you.
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Sat Jun 19 21:36:57 2021

Thank you for your help, Minna.

Binyu

On Fri, Jun 18, 2021 at 6:48 PM Minna Win via RT <met_help at ucar.edu>
wrote:

> Hi Binyu,
>
> I took a look at the user documentation, and the ECNT line type
(ensemble
> stat) is different from the PCT linetype (grid-stat).  Since
ensemble stat
> and grid stat are doing different things, at first glance I wouldn't
expect
> these two values to be the same.   I'll have to verify with a
scientist.
>
> Regards,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
> ---------------
>
>
>
> On Fri, Jun 18, 2021 at 4:20 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> >
> > Hello Minna,
> >
> > Thank you for forwarding this information to me. Based on what
John said
> > before: ens_thresh is used to specify the required ratio of
ensemble
> > members that must be
> > present for ensemble_stat to run at all. I have no missing
ensemble
> > members so there is no impact to change "ens_thresh" in my test.
> >
> > Attached are the output from ensemble_stat and grid_stat (same
dataset).
> > The weird thing is: there are "9 pairs" from the first file, but
"100
> > pairs" from the second one.
> > Based on the MET document, "9" in the first file (ECNT) refers to
"Count
> of
> > observations", and "100" from the second file (PCT) refers to
"Total
> > number of matched pairs", that should be the same thing, right?
But the
> > value is different, why?
> >
> > Thank you.
> >
> >
> > On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Binyu,
> > >
> > > John Opatz also has some good information:
> > >
> > > ----------------
> > > I think the most useful information to this ticket is here
> > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> > >
> > > Take a look at the entry for vld_thresh. This is the ratio of
valid
> data
> > > values to ensemble members for each grid point. If it falls
below the
> > > user's vld_thresh, ensemble-stat will write out bad data.
> > >
> > > ens_thresh has a similar quality control behavior, but it's for
the
> > entire
> > > ensemble field vs each grid point. So if there are 10 ensemble
members
> > and
> > > 1 is a bad member but your ens_thresh is set to 1.0,
ensemble_stat will
> > > quit.
> > >
> > > I think the default for both of these is 1.0. Maybe they can try
to set
> > it
> > > to 0.75, or 0.5 and rerun their results?
> > >
> > > --------------
> > >
> > > Regards,
> > > Minna
> > >
> > > ---------------
> > > Minna Win
> > > Pronouns: she/her
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > > ---------------
> > >
> > >
> > >
> > > On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> > > minnawin at ucar.edu>
> > > wrote:
> > >
> > > > Hi Binyu,
> > > >
> > > > Here's an explanation from John Halley Gotway (who is supposed
to be
> on
> > > > vacation):
> > > >
> > > > For each field listed in the ens dictionary of the config file
and
> for
> > > > each cat_thresh threshold defined for that field, ensemble-
stat
> derives
> > > an
> > > > ensemble probability as the ratio of ensemble members that
meet the
> > > > cat_thresh threshold criteria. For example, for 2-m
temperature and a
> > > > threshold of <273 over an ensemble of size 10, if 6 of the 10
members
> > > have
> > > > freezing temperatures at some grid point, then the probability
there
> is
> > > > just 6/10=0.6. The vld_thresh defines how to handle missing
data.
> > > >
> > > > I hope that helps.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > Pronouns: she/her
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax:   303-497-8401
> > > > ---------------
> > > >
> > > >
> > > >
> > > > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > > >>
> > > >> Hello Minna,
> > > >>
> > > >> Thank you for your quick response. In order not to give extra
work
> to
> > > too
> > > >> many people, I think at this moment I only want to find an
expert to
> > > >> understand how MET
> > > >> is calculating the probability and make sure the config files
that I
> > am
> > > >> using are correct. I think the way that Python is used is
pretty
> > similar
> > > >> to
> > > >> MET, however the plot is a big difference.
> > > >> Best
> > > >> Binyu
> > > >>
> > > >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT <
> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> > Hi Binyu,
> > > >> >
> > > >> > It looks like you have multiple questions that will require
more
> > than
> > > >> one
> > > >> > person to respond.:
> > > >> >
> > > >> > 1) Your expected reliability plot in METviewer using MET
ensemble
> > > >> output on
> > > >> > satellite obs data is *not* what is expected and you want
to know
> > how
> > > >> MET
> > > >> > is calculating the probability.
> > > >> >
> > > >> > 2)  You want to know how METviewer implements the
generation of
> the
> > > >> > reliability diagram, so you can reproduce it in Python?
> > > >> >
> > > >> > Let me know if I've captured your questions correctly, so I
can
> find
> > > the
> > > >> > appropriate people to contact.
> > > >> >
> > > >> > Regards,
> > > >> > Minna
> > > >> > ---------------
> > > >> > Minna Win
> > > >> > Pronouns: she/her
> > > >> > National Center for Atmospheric Research
> > > >> > Developmental Testbed Center
> > > >> > Phone: 303-497-8423
> > > >> > Fax:   303-497-8401
> > > >> > ---------------
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via RT
<
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted upon.
> > > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > >> > >        Queue: met_help
> > > >> > >      Subject: reliability calculation
> > > >> > >        Owner: Nobody
> > > >> > >   Requestors: binyu.wang at noaa.gov
> > > >> > >       Status: new
> > > >> > >  Ticket <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > > >> > >
> > > >> > >
> > > >> > > Hello,
> > > >> > >
> > > >> > > I was trying to use ensemble forecasts and satellite obs.
to
> make
> > a
> > > >> > > reliability plot (see attachment 1).  The plot indicates
that no
> > > >> matter
> > > >> > > what the forecast frequency is, the observed frequency is
near
> 1.
> > > But
> > > >> > > actually I see lots of 'misses' - areas where the model
has ash
> > but
> > > >> the
> > > >> > > observations don't. I have 31 members (no missing file).
I think
> > the
> > > >> way
> > > >> > > that MET use to calculate the probability is "ONLY if all
the
> > > ensemble
> > > >> > > members have overlap with the grid that obs, exist,  then
it
> will
> > be
> > > >> > > counted as forecast "catch" the observation", is that
correct?
> > Which
> > > >> > means
> > > >> > > if one grid was caught only by 30 members, the grid will
be
> > treaded
> > > as
> > > >> > > missed by the forecast
> > > >> > >
> > > >> > > The plot I got using MET/METviewer (attachment 1) is much
> > different
> > > >> than
> > > >> > > using python (see attachment 2), and here is the way that
was
> > used:
> > > >> > > ---------------------------------------------
> > > >> > > For the reliability curve, at each point, I count how
many
> > ensemble
> > > >> > members
> > > >> > > are above the threshold. That number determines the x bin
value.
> > So
> > > >> if I
> > > >> > > use the 31 bins and 20 members are above threshold, then
I am in
> > the
> > > >> 20th
> > > >> > > bin/x value. Then I see if the obs is above threshold. If
it is,
> > > then
> > > >> a 1
> > > >> > > goes into that bin. If it isn't then a 0 goes into that
bin.
> > > >> > >
> > > >> > > After I have gone through all the points, to get the y
values I
> > sum
> > > >> the
> > > >> > > values in the bins and divide by the total number of them
to get
> > the
> > > >> > > fraction observed.
> > > >> > > -------------------------------------------
> > > >> > >
> > > >> > > So can I use the similar way in MET? I mean to calculate
> > probability
> > > >> > > without using all ensemble members?  I did set
"vld_thresh =
> 0.1"
> > in
> > > >> my
> > > >> > > config file.
> > > >> > >
> > > >> > > Here is the ensemble config file that I use:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > > >> > >
> > > >> > >
> > > >> > > My obs:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > > >> > >
> > > >> > >
> > > >> > > My forecast:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > > >> > >
> > > >> > > Thank you.
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: Minna Win
Time: Tue Jun 22 14:17:24 2021

Hi Binyu,

I posed your question to one of our scientists, and he's interested in
looking at the log files from your ensemble-stat and grid-stat runs.
Can
you please provide those?

Thanks,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
---------------



On Sat, Jun 19, 2021 at 9:36 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
>
> Thank you for your help, Minna.
>
> Binyu
>
> On Fri, Jun 18, 2021 at 6:48 PM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> > Hi Binyu,
> >
> > I took a look at the user documentation, and the ECNT line type
(ensemble
> > stat) is different from the PCT linetype (grid-stat).  Since
ensemble
> stat
> > and grid stat are doing different things, at first glance I
wouldn't
> expect
> > these two values to be the same.   I'll have to verify with a
scientist.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > Pronouns: she/her
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> > ---------------
> >
> >
> >
> > On Fri, Jun 18, 2021 at 4:20 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > >
> > > Hello Minna,
> > >
> > > Thank you for forwarding this information to me. Based on what
John
> said
> > > before: ens_thresh is used to specify the required ratio of
ensemble
> > > members that must be
> > > present for ensemble_stat to run at all. I have no missing
ensemble
> > > members so there is no impact to change "ens_thresh" in my test.
> > >
> > > Attached are the output from ensemble_stat and grid_stat (same
> dataset).
> > > The weird thing is: there are "9 pairs" from the first file, but
"100
> > > pairs" from the second one.
> > > Based on the MET document, "9" in the first file (ECNT) refers
to
> "Count
> > of
> > > observations", and "100" from the second file (PCT) refers to
"Total
> > > number of matched pairs", that should be the same thing, right?
But the
> > > value is different, why?
> > >
> > > Thank you.
> > >
> > >
> > > On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Binyu,
> > > >
> > > > John Opatz also has some good information:
> > > >
> > > > ----------------
> > > > I think the most useful information to this ticket is here
> > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> > > >
> > > > Take a look at the entry for vld_thresh. This is the ratio of
valid
> > data
> > > > values to ensemble members for each grid point. If it falls
below the
> > > > user's vld_thresh, ensemble-stat will write out bad data.
> > > >
> > > > ens_thresh has a similar quality control behavior, but it's
for the
> > > entire
> > > > ensemble field vs each grid point. So if there are 10 ensemble
> members
> > > and
> > > > 1 is a bad member but your ens_thresh is set to 1.0,
ensemble_stat
> will
> > > > quit.
> > > >
> > > > I think the default for both of these is 1.0. Maybe they can
try to
> set
> > > it
> > > > to 0.75, or 0.5 and rerun their results?
> > > >
> > > > --------------
> > > >
> > > > Regards,
> > > > Minna
> > > >
> > > > ---------------
> > > > Minna Win
> > > > Pronouns: she/her
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax:   303-497-8401
> > > > ---------------
> > > >
> > > >
> > > >
> > > > On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> > > > minnawin at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Binyu,
> > > > >
> > > > > Here's an explanation from John Halley Gotway (who is
supposed to
> be
> > on
> > > > > vacation):
> > > > >
> > > > > For each field listed in the ens dictionary of the config
file and
> > for
> > > > > each cat_thresh threshold defined for that field, ensemble-
stat
> > derives
> > > > an
> > > > > ensemble probability as the ratio of ensemble members that
meet the
> > > > > cat_thresh threshold criteria. For example, for 2-m
temperature
> and a
> > > > > threshold of <273 over an ensemble of size 10, if 6 of the
10
> members
> > > > have
> > > > > freezing temperatures at some grid point, then the
probability
> there
> > is
> > > > > just 6/10=0.6. The vld_thresh defines how to handle missing
data.
> > > > >
> > > > > I hope that helps.
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > > ---------------
> > > > > Minna Win
> > > > > Pronouns: she/her
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax:   303-497-8401
> > > > > ---------------
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > > > >>
> > > > >> Hello Minna,
> > > > >>
> > > > >> Thank you for your quick response. In order not to give
extra work
> > to
> > > > too
> > > > >> many people, I think at this moment I only want to find an
expert
> to
> > > > >> understand how MET
> > > > >> is calculating the probability and make sure the config
files
> that I
> > > am
> > > > >> using are correct. I think the way that Python is used is
pretty
> > > similar
> > > > >> to
> > > > >> MET, however the plot is a big difference.
> > > > >> Best
> > > > >> Binyu
> > > > >>
> > > > >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT <
> > met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Binyu,
> > > > >> >
> > > > >> > It looks like you have multiple questions that will
require more
> > > than
> > > > >> one
> > > > >> > person to respond.:
> > > > >> >
> > > > >> > 1) Your expected reliability plot in METviewer using MET
> ensemble
> > > > >> output on
> > > > >> > satellite obs data is *not* what is expected and you want
to
> know
> > > how
> > > > >> MET
> > > > >> > is calculating the probability.
> > > > >> >
> > > > >> > 2)  You want to know how METviewer implements the
generation of
> > the
> > > > >> > reliability diagram, so you can reproduce it in Python?
> > > > >> >
> > > > >> > Let me know if I've captured your questions correctly, so
I can
> > find
> > > > the
> > > > >> > appropriate people to contact.
> > > > >> >
> > > > >> > Regards,
> > > > >> > Minna
> > > > >> > ---------------
> > > > >> > Minna Win
> > > > >> > Pronouns: she/her
> > > > >> > National Center for Atmospheric Research
> > > > >> > Developmental Testbed Center
> > > > >> > Phone: 303-497-8423
> > > > >> > Fax:   303-497-8401
> > > > >> > ---------------
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via
RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted
upon.
> > > > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > >> > >        Queue: met_help
> > > > >> > >      Subject: reliability calculation
> > > > >> > >        Owner: Nobody
> > > > >> > >   Requestors: binyu.wang at noaa.gov
> > > > >> > >       Status: new
> > > > >> > >  Ticket <URL:
> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> > > > >> > >
> > > > >> > >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > I was trying to use ensemble forecasts and satellite
obs. to
> > make
> > > a
> > > > >> > > reliability plot (see attachment 1).  The plot
indicates that
> no
> > > > >> matter
> > > > >> > > what the forecast frequency is, the observed frequency
is near
> > 1.
> > > > But
> > > > >> > > actually I see lots of 'misses' - areas where the model
has
> ash
> > > but
> > > > >> the
> > > > >> > > observations don't. I have 31 members (no missing
file). I
> think
> > > the
> > > > >> way
> > > > >> > > that MET use to calculate the probability is "ONLY if
all the
> > > > ensemble
> > > > >> > > members have overlap with the grid that obs, exist,
then it
> > will
> > > be
> > > > >> > > counted as forecast "catch" the observation", is that
correct?
> > > Which
> > > > >> > means
> > > > >> > > if one grid was caught only by 30 members, the grid
will be
> > > treaded
> > > > as
> > > > >> > > missed by the forecast
> > > > >> > >
> > > > >> > > The plot I got using MET/METviewer (attachment 1) is
much
> > > different
> > > > >> than
> > > > >> > > using python (see attachment 2), and here is the way
that was
> > > used:
> > > > >> > > ---------------------------------------------
> > > > >> > > For the reliability curve, at each point, I count how
many
> > > ensemble
> > > > >> > members
> > > > >> > > are above the threshold. That number determines the x
bin
> value.
> > > So
> > > > >> if I
> > > > >> > > use the 31 bins and 20 members are above threshold,
then I am
> in
> > > the
> > > > >> 20th
> > > > >> > > bin/x value. Then I see if the obs is above threshold.
If it
> is,
> > > > then
> > > > >> a 1
> > > > >> > > goes into that bin. If it isn't then a 0 goes into that
bin.
> > > > >> > >
> > > > >> > > After I have gone through all the points, to get the y
values
> I
> > > sum
> > > > >> the
> > > > >> > > values in the bins and divide by the total number of
them to
> get
> > > the
> > > > >> > > fraction observed.
> > > > >> > > -------------------------------------------
> > > > >> > >
> > > > >> > > So can I use the similar way in MET? I mean to
calculate
> > > probability
> > > > >> > > without using all ensemble members?  I did set
"vld_thresh =
> > 0.1"
> > > in
> > > > >> my
> > > > >> > > config file.
> > > > >> > >
> > > > >> > > Here is the ensemble config file that I use:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > > > >> > >
> > > > >> > >
> > > > >> > > My obs:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > > > >> > >
> > > > >> > >
> > > > >> > > My forecast:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > > > >> > >
> > > > >> > > Thank you.
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Tue Jun 22 17:44:54 2021

Hello Minna,

WCOSS is currently down and will be back on Thursday. I will send you
the
information on Thursday.  Thank you for your help.

Binyu

On Tue, Jun 22, 2021 at 4:17 PM Minna Win via RT <met_help at ucar.edu>
wrote:

> Hi Binyu,
>
> I posed your question to one of our scientists, and he's interested
in
> looking at the log files from your ensemble-stat and grid-stat runs.
Can
> you please provide those?
>
> Thanks,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
> ---------------
>
>
>
> On Sat, Jun 19, 2021 at 9:36 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255 >
> >
> > Thank you for your help, Minna.
> >
> > Binyu
> >
> > On Fri, Jun 18, 2021 at 6:48 PM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Binyu,
> > >
> > > I took a look at the user documentation, and the ECNT line type
> (ensemble
> > > stat) is different from the PCT linetype (grid-stat).  Since
ensemble
> > stat
> > > and grid stat are doing different things, at first glance I
wouldn't
> > expect
> > > these two values to be the same.   I'll have to verify with a
> scientist.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > Pronouns: she/her
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > > ---------------
> > >
> > >
> > >
> > > On Fri, Jun 18, 2021 at 4:20 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
>
> > > >
> > > > Hello Minna,
> > > >
> > > > Thank you for forwarding this information to me. Based on what
John
> > said
> > > > before: ens_thresh is used to specify the required ratio of
ensemble
> > > > members that must be
> > > > present for ensemble_stat to run at all. I have no missing
ensemble
> > > > members so there is no impact to change "ens_thresh" in my
test.
> > > >
> > > > Attached are the output from ensemble_stat and grid_stat (same
> > dataset).
> > > > The weird thing is: there are "9 pairs" from the first file,
but "100
> > > > pairs" from the second one.
> > > > Based on the MET document, "9" in the first file (ECNT) refers
to
> > "Count
> > > of
> > > > observations", and "100" from the second file (PCT) refers to
"Total
> > > > number of matched pairs", that should be the same thing,
right? But
> the
> > > > value is different, why?
> > > >
> > > > Thank you.
> > > >
> > > >
> > > > On Thu, Jun 17, 2021 at 12:05 PM Minna Win via RT
<met_help at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > Hi Binyu,
> > > > >
> > > > > John Opatz also has some good information:
> > > > >
> > > > > ----------------
> > > > > I think the most useful information to this ticket is here
> > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> > > > >
> > > > > Take a look at the entry for vld_thresh. This is the ratio
of valid
> > > data
> > > > > values to ensemble members for each grid point. If it falls
below
> the
> > > > > user's vld_thresh, ensemble-stat will write out bad data.
> > > > >
> > > > > ens_thresh has a similar quality control behavior, but it's
for the
> > > > entire
> > > > > ensemble field vs each grid point. So if there are 10
ensemble
> > members
> > > > and
> > > > > 1 is a bad member but your ens_thresh is set to 1.0,
ensemble_stat
> > will
> > > > > quit.
> > > > >
> > > > > I think the default for both of these is 1.0. Maybe they can
try to
> > set
> > > > it
> > > > > to 0.75, or 0.5 and rerun their results?
> > > > >
> > > > > --------------
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > >
> > > > > ---------------
> > > > > Minna Win
> > > > > Pronouns: she/her
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax:   303-497-8401
> > > > > ---------------
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 17, 2021 at 10:00 AM Minna Win-Gildenmeister <
> > > > > minnawin at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Binyu,
> > > > > >
> > > > > > Here's an explanation from John Halley Gotway (who is
supposed to
> > be
> > > on
> > > > > > vacation):
> > > > > >
> > > > > > For each field listed in the ens dictionary of the config
file
> and
> > > for
> > > > > > each cat_thresh threshold defined for that field,
ensemble-stat
> > > derives
> > > > > an
> > > > > > ensemble probability as the ratio of ensemble members that
meet
> the
> > > > > > cat_thresh threshold criteria. For example, for 2-m
temperature
> > and a
> > > > > > threshold of <273 over an ensemble of size 10, if 6 of the
10
> > members
> > > > > have
> > > > > > freezing temperatures at some grid point, then the
probability
> > there
> > > is
> > > > > > just 6/10=0.6. The vld_thresh defines how to handle
missing data.
> > > > > >
> > > > > > I hope that helps.
> > > > > >
> > > > > > Regards,
> > > > > > Minna
> > > > > > ---------------
> > > > > > Minna Win
> > > > > > Pronouns: she/her
> > > > > > National Center for Atmospheric Research
> > > > > > Developmental Testbed Center
> > > > > > Phone: 303-497-8423
> > > > > > Fax:   303-497-8401
> > > > > > ---------------
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 17, 2021 at 8:46 AM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
> >
> > > > > >>
> > > > > >> Hello Minna,
> > > > > >>
> > > > > >> Thank you for your quick response. In order not to give
extra
> work
> > > to
> > > > > too
> > > > > >> many people, I think at this moment I only want to find
an
> expert
> > to
> > > > > >> understand how MET
> > > > > >> is calculating the probability and make sure the config
files
> > that I
> > > > am
> > > > > >> using are correct. I think the way that Python is used is
pretty
> > > > similar
> > > > > >> to
> > > > > >> MET, however the plot is a big difference.
> > > > > >> Best
> > > > > >> Binyu
> > > > > >>
> > > > > >> On Thu, Jun 17, 2021 at 10:25 AM Minna Win via RT <
> > > met_help at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Binyu,
> > > > > >> >
> > > > > >> > It looks like you have multiple questions that will
require
> more
> > > > than
> > > > > >> one
> > > > > >> > person to respond.:
> > > > > >> >
> > > > > >> > 1) Your expected reliability plot in METviewer using
MET
> > ensemble
> > > > > >> output on
> > > > > >> > satellite obs data is *not* what is expected and you
want to
> > know
> > > > how
> > > > > >> MET
> > > > > >> > is calculating the probability.
> > > > > >> >
> > > > > >> > 2)  You want to know how METviewer implements the
generation
> of
> > > the
> > > > > >> > reliability diagram, so you can reproduce it in Python?
> > > > > >> >
> > > > > >> > Let me know if I've captured your questions correctly,
so I
> can
> > > find
> > > > > the
> > > > > >> > appropriate people to contact.
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> > Minna
> > > > > >> > ---------------
> > > > > >> > Minna Win
> > > > > >> > Pronouns: she/her
> > > > > >> > National Center for Atmospheric Research
> > > > > >> > Developmental Testbed Center
> > > > > >> > Phone: 303-497-8423
> > > > > >> > Fax:   303-497-8401
> > > > > >> > ---------------
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Jun 16, 2021 at 7:48 PM binyu.wang at noaa.gov via
RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > Wed Jun 16 19:48:11 2021: Request 100255 was acted
upon.
> > > > > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > >> > >        Queue: met_help
> > > > > >> > >      Subject: reliability calculation
> > > > > >> > >        Owner: Nobody
> > > > > >> > >   Requestors: binyu.wang at noaa.gov
> > > > > >> > >       Status: new
> > > > > >> > >  Ticket <URL:
> > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100255
>
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Hello,
> > > > > >> > >
> > > > > >> > > I was trying to use ensemble forecasts and satellite
obs. to
> > > make
> > > > a
> > > > > >> > > reliability plot (see attachment 1).  The plot
indicates
> that
> > no
> > > > > >> matter
> > > > > >> > > what the forecast frequency is, the observed
frequency is
> near
> > > 1.
> > > > > But
> > > > > >> > > actually I see lots of 'misses' - areas where the
model has
> > ash
> > > > but
> > > > > >> the
> > > > > >> > > observations don't. I have 31 members (no missing
file). I
> > think
> > > > the
> > > > > >> way
> > > > > >> > > that MET use to calculate the probability is "ONLY if
all
> the
> > > > > ensemble
> > > > > >> > > members have overlap with the grid that obs, exist,
then it
> > > will
> > > > be
> > > > > >> > > counted as forecast "catch" the observation", is that
> correct?
> > > > Which
> > > > > >> > means
> > > > > >> > > if one grid was caught only by 30 members, the grid
will be
> > > > treaded
> > > > > as
> > > > > >> > > missed by the forecast
> > > > > >> > >
> > > > > >> > > The plot I got using MET/METviewer (attachment 1) is
much
> > > > different
> > > > > >> than
> > > > > >> > > using python (see attachment 2), and here is the way
that
> was
> > > > used:
> > > > > >> > > ---------------------------------------------
> > > > > >> > > For the reliability curve, at each point, I count how
many
> > > > ensemble
> > > > > >> > members
> > > > > >> > > are above the threshold. That number determines the x
bin
> > value.
> > > > So
> > > > > >> if I
> > > > > >> > > use the 31 bins and 20 members are above threshold,
then I
> am
> > in
> > > > the
> > > > > >> 20th
> > > > > >> > > bin/x value. Then I see if the obs is above
threshold. If it
> > is,
> > > > > then
> > > > > >> a 1
> > > > > >> > > goes into that bin. If it isn't then a 0 goes into
that bin.
> > > > > >> > >
> > > > > >> > > After I have gone through all the points, to get the
y
> values
> > I
> > > > sum
> > > > > >> the
> > > > > >> > > values in the bins and divide by the total number of
them to
> > get
> > > > the
> > > > > >> > > fraction observed.
> > > > > >> > > -------------------------------------------
> > > > > >> > >
> > > > > >> > > So can I use the similar way in MET? I mean to
calculate
> > > > probability
> > > > > >> > > without using all ensemble members?  I did set
"vld_thresh =
> > > 0.1"
> > > > in
> > > > > >> my
> > > > > >> > > config file.
> > > > > >> > >
> > > > > >> > > Here is the ensemble config file that I use:
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > > >> > > verf_g2g_ens_stat_regn_config_Bezy.vld_thresh.0p5
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > My obs:
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Bezymianny/Satellite/From_Alice_regridM/Hourly_mean/
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > My forecast:
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Hysplit/Bezy/Res_0p1/DIFF.EMIS
> > > > > >> > >
> > > > > >> > > Thank you.
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: reliability calculation
From: John Halley Gotway
Time: Mon Jul 12 11:50:05 2021

Binyu,

I see this existing met_help at ucar.edu ticket from a few weeks ago. I'm
wondering if you have any remaining questions or topics you'd like to
discuss.

Please note that the METplus team recently moved our support over to
GitHub discussions:
https://github.com/dtcenter/METplus/discussions

So please post future questions or discussion topics to the "Incoming"
category there.

Thanks,
John Halley Gotway

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Mon Jul 12 14:37:50 2021

/lfs/h1/emc/physics/noscrub/Binyu.Wang/Transition2/sorc/hysplit_gfslr2arl_g2.fd

On Mon, Jul 12, 2021 at 1:57 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binyu,
>
> I see this existing met_help at ucar.edu ticket from a few weeks ago.
I'm
> wondering if you have any remaining questions or topics you'd like
to
> discuss.
>
> Please note that the METplus team recently moved our support over to
> GitHub discussions:
> https://github.com/dtcenter/METplus/discussions
>
> So please post future questions or discussion topics to the
"Incoming"
> category there.
>
> Thanks,
> John Halley Gotway
>

------------------------------------------------
Subject: reliability calculation
From: binyu.wang at noaa.gov
Time: Mon Jul 12 14:47:53 2021

Sorry, wrong email.

On Mon, Jul 12, 2021 at 4:37 PM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:

>
>
/lfs/h1/emc/physics/noscrub/Binyu.Wang/Transition2/sorc/hysplit_gfslr2arl_g2.fd
>
> On Mon, Jul 12, 2021 at 1:57 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binyu,
>>
>> I see this existing met_help at ucar.edu ticket from a few weeks ago.
I'm
>> wondering if you have any remaining questions or topics you'd like
to
>> discuss.
>>
>> Please note that the METplus team recently moved our support over
to
>> GitHub discussions:
>> https://github.com/dtcenter/METplus/discussions
>>
>> So please post future questions or discussion topics to the
"Incoming"
>> category there.
>>
>> Thanks,
>> John Halley Gotway
>>
>

------------------------------------------------