[Met_help] [rt.rap.ucar.edu #99940] History for ensemble verification
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 13 12:53:01 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I was using MET for ensemble verification and I had a question about using
the cat_thresh. Please look at the config below, and the full version is
under:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy_thr0p05
I am not sure if my config is right or not, I want to make such plots
including:
(1).rank plot by setting different threshold (like both obs. value and all
ensemble members values should larger than 0.1/0.05/0/01 ).
(2).Rely plots for different threshold [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7, >=10 ];
My config file is only setup for threshold=0.1, but I am not sure if I
should move the two lines below from "fcst" entry to "ens" entry?
censor_thresh=[<0.1];
censor_val=[-9999]; desc="GE0.1";
--------------------------------------------------
fcst = {
ens_thresh = 1.0;
vld_thresh = 1.0;
field = [
{
name = "VAFTD";
convert(x) = 21335*(10^(x));
level = [ "L0-21335" ];
censor_thresh=[<0.1];
censor_val=[-9999]; desc="GE0.1";
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=3,>=5,>=7,
>=10 ];
lead_time = "${LEAD_HR}";
valid_time ="${EDATE}_${EHOUR}";
}
];
};
ens = {
field = [
{
name = "VAFTD";
convert(x) = 21335*(10^(x));
level = [ "L0-21335" ];
lead_time = "${LEAD_HR}";
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=3,>=5,>=7,>=10
];
}
];
};
obs = {
file_type = NETCDF_MET;
field = [
{
name = "ash_mass_loading";
level = ["(*,*)"];
cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,>=10 ];
}
];
---------------------------------
If my config file is correct, is it possible that I can use only one run to
get the stat to make rank plots with multiple thresholds?
Thank you for help.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: ensemble verification
From: George McCabe
Time: Wed May 19 16:42:30 2021
Hi Binyu,
I assigned this ticket to John HG. Please allow a few business days
for a
response.
Thanks,
George
On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: ensemble verification
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
>
>
> Hello,
>
>
> I was using MET for ensemble verification and I had a question about
using
> the cat_thresh. Please look at the config below, and the full
version is
> under:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy_thr0p05
>
>
> I am not sure if my config is right or not, I want to make such
plots
> including:
>
>
> (1).rank plot by setting different threshold (like both obs. value
and all
> ensemble members values should larger than 0.1/0.05/0/01 ).
>
> (2).Rely plots for different threshold [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=3,>=5,>=7, >=10 ];
>
>
> My config file is only setup for threshold=0.1, but I am not sure if
I
> should move the two lines below from "fcst" entry to "ens" entry?
>
>
> censor_thresh=[<0.1];
>
> censor_val=[-9999]; desc="GE0.1";
>
>
> --------------------------------------------------
>
>
> fcst = {
>
> ens_thresh = 1.0;
>
> vld_thresh = 1.0;
>
>
> field = [
>
> {
>
> name = "VAFTD";
>
> convert(x) = 21335*(10^(x));
>
> level = [ "L0-21335" ];
>
> censor_thresh=[<0.1];
>
> censor_val=[-9999]; desc="GE0.1";
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> >=10 ];
>
> lead_time = "${LEAD_HR}";
>
> valid_time ="${EDATE}_${EHOUR}";
>
> }
>
> ];
>
> };
>
>
> ens = {
>
> field = [
>
> {
>
> name = "VAFTD";
>
> convert(x) = 21335*(10^(x));
>
> level = [ "L0-21335" ];
>
> lead_time = "${LEAD_HR}";
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,>=10
> ];
>
>
> }
>
> ];
>
> };
>
>
> obs = {
>
>
> file_type = NETCDF_MET;
>
> field = [
>
> {
>
> name = "ash_mass_loading";
>
> level = ["(*,*)"];
>
> cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=3,>=5,>=7,>=10 ];
>
> }
>
> ];
>
> ---------------------------------
>
>
> If my config file is correct, is it possible that I can use only one
run to
> get the stat to make rank plots with multiple thresholds?
>
>
> Thank you for help.
>
>
> Binyu
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: ensemble verification
From: John Halley Gotway
Time: Thu May 20 16:43:22 2021
Binyu,
I don't fully understand your question.
By "rank plot" do you mean Ranked Histograms? Thresholds do NOT apply
to
ranked histogram plots. Ranked histograms are computed on raw ensemble
and
observation values.
I'm just guessing here... perhaps you really mean that you want to do
conditional verification. And you want Ranked Histograms conditioned
on
observation values being...
greater than 0.01, 0.05, 0.1, and so on.
Is that correct?
John
On Wed, May 19, 2021 at 4:43 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
>
> Hi Binyu,
>
> I assigned this ticket to John HG. Please allow a few business days
for a
> response.
>
> Thanks,
> George
>
> On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: ensemble verification
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> >
> >
> > Hello,
> >
> >
> > I was using MET for ensemble verification and I had a question
about
> using
> > the cat_thresh. Please look at the config below, and the full
version is
> > under:
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> >
> >
> > I am not sure if my config is right or not, I want to make such
plots
> > including:
> >
> >
> > (1).rank plot by setting different threshold (like both obs. value
and
> all
> > ensemble members values should larger than 0.1/0.05/0/01 ).
> >
> > (2).Rely plots for different threshold [>=0.01, >=0.05,
>=0.1,>=0.3,
> > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> >
> >
> > My config file is only setup for threshold=0.1, but I am not sure
if I
> > should move the two lines below from "fcst" entry to "ens" entry?
> >
> >
> > censor_thresh=[<0.1];
> >
> > censor_val=[-9999]; desc="GE0.1";
> >
> >
> > --------------------------------------------------
> >
> >
> > fcst = {
> >
> > ens_thresh = 1.0;
> >
> > vld_thresh = 1.0;
> >
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > convert(x) = 21335*(10^(x));
> >
> > level = [ "L0-21335" ];
> >
> > censor_thresh=[<0.1];
> >
> > censor_val=[-9999]; desc="GE0.1";
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> > >=10 ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > valid_time ="${EDATE}_${EHOUR}";
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > ens = {
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > convert(x) = 21335*(10^(x));
> >
> > level = [ "L0-21335" ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=3,>=5,>=7,>=10
> > ];
> >
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > obs = {
> >
> >
> > file_type = NETCDF_MET;
> >
> > field = [
> >
> > {
> >
> > name = "ash_mass_loading";
> >
> > level = ["(*,*)"];
> >
> > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> >
> > }
> >
> > ];
> >
> > ---------------------------------
> >
> >
> > If my config file is correct, is it possible that I can use only
one run
> to
> > get the stat to make rank plots with multiple thresholds?
> >
> >
> > Thank you for help.
> >
> >
> > Binyu
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: ensemble verification
From: binyu.wang at noaa.gov
Time: Thu May 20 21:30:33 2021
Hello John,
Yes both of your guesses are correct. Rank plots means "Ranked
histograms"
and I do want the Ranked Histograms conditioned on
observation values being greater than 0.01, 0.05, 0.1, and so on. BUT
except for observation, I want the Histograms conditioned on both
obs. and
forecast. (e.g: if the obs. detection limit is 0.1, then I also want
to
remove the value less than 0,1 from the forecast too). Or my
understanding is not correct (we can ONLY make the Histograms
conditioned
on obs?)
Is that possible to do that using ONLY one configuration? Or we must
run
multiple times with different configurations?
Suppose we can only create the required *stat file for one condition
0.05,
I am not sure where to put the two lines:
censor_thresh=[<0.05];
censor_val=[-9999]; desc="GE0.05";
Should they be in the fcst entry or ens entry or both?
I tried 3 different runs (the first time I put the two lines in "fcst"
entry and the second time I put the lines in the "ens" entry, and
third
time I removed the lines), I got the same *stat files. I am not sure
what
the problem is here?
You can find the 3 config files:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.1
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.2
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.3
Another question:
Setting those censor_thresh will not affect "cat_thresh", right? In
other
words, even if I set censor_thresh=[<0.05]; I still can set a value
less
than 0.05 ( e.g: 0.01 in my config below) as one of the cat_thresh (
which
will be used for the RELY plot), right?
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=3,>=5,>=7, >=10
];
Thank you.
Binyu
On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I don't fully understand your question.
>
> By "rank plot" do you mean Ranked Histograms? Thresholds do NOT
apply to
> ranked histogram plots. Ranked histograms are computed on raw
ensemble and
> observation values.
>
> I'm just guessing here... perhaps you really mean that you want to
do
> conditional verification. And you want Ranked Histograms conditioned
on
> observation values being...
> greater than 0.01, 0.05, 0.1, and so on.
>
> Is that correct?
>
> John
>
> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> >
> > Hi Binyu,
> >
> > I assigned this ticket to John HG. Please allow a few business
days for a
> > response.
> >
> > Thanks,
> > George
> >
> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: ensemble verification
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
> >
> > >
> > >
> > > Hello,
> > >
> > >
> > > I was using MET for ensemble verification and I had a question
about
> > using
> > > the cat_thresh. Please look at the config below, and the full
version
> is
> > > under:
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> > >
> > >
> > > I am not sure if my config is right or not, I want to make such
plots
> > > including:
> > >
> > >
> > > (1).rank plot by setting different threshold (like both obs.
value and
> > all
> > > ensemble members values should larger than 0.1/0.05/0/01 ).
> > >
> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> > >
> > >
> > > My config file is only setup for threshold=0.1, but I am not
sure if I
> > > should move the two lines below from "fcst" entry to "ens"
entry?
> > >
> > >
> > > censor_thresh=[<0.1];
> > >
> > > censor_val=[-9999]; desc="GE0.1";
> > >
> > >
> > > --------------------------------------------------
> > >
> > >
> > > fcst = {
> > >
> > > ens_thresh = 1.0;
> > >
> > > vld_thresh = 1.0;
> > >
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > convert(x) = 21335*(10^(x));
> > >
> > > level = [ "L0-21335" ];
> > >
> > > censor_thresh=[<0.1];
> > >
> > > censor_val=[-9999]; desc="GE0.1";
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=3,>=5,>=7,
> > > >=10 ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > valid_time ="${EDATE}_${EHOUR}";
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > ens = {
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > convert(x) = 21335*(10^(x));
> > >
> > > level = [ "L0-21335" ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=3,>=5,>=7,>=10
> > > ];
> > >
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > obs = {
> > >
> > >
> > > file_type = NETCDF_MET;
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "ash_mass_loading";
> > >
> > > level = ["(*,*)"];
> > >
> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> > >
> > > }
> > >
> > > ];
> > >
> > > ---------------------------------
> > >
> > >
> > > If my config file is correct, is it possible that I can use only
one
> run
> > to
> > > get the stat to make rank plots with multiple thresholds?
> > >
> > >
> > > Thank you for help.
> > >
> > >
> > > Binyu
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: ensemble verification
From: binyu.wang at noaa.gov
Time: Fri May 21 07:45:37 2021
Hello John,
Just want to make it clear, so I reorganize the questions:
1. Will setting "censor_thresh" affect "cat_thresh"? e.g:
field = [
{
name = "VAFTD";
convert(x) = 21335*(10^(x));
level = [ "L0-21335" ];
censor_thresh=[<0.1];
censor_val=[-9999]; desc="GE0.1";
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
>=10 ];
}
];
};
I set 0.1 as "sensor_thresh", so can I still set a value less than 0.1
like
0.01/0.05 for the "cat_thresh"? Or only values higher than 0.1 will be
used
for cat_thresh?
2.Suppose I only want to create the required *stat file for Histogram
plot
based one condition 0.05, I am not sure where to put the two lines:
censor_thresh=[<0.05];
censor_val=[-9999]; desc="GE0.05";
Should they be in the fcst entry or ens entry or both?
I tried 3 different runs (the first time I put the two lines in "fcst"
entry and the second time I put the lines in the "ens" entry, and
third
time I removed the lines), I got the same *stat files. I am not sure
what
the problem is here?
You can find the 3 config files:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.1
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.2
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.3
3.
Yes both of your guesses are correct. Rank plots means "Ranked
histograms"
and I do want the Ranked Histograms conditioned on
observation values being greater than 0.01, 0.05, 0.1, and so on. BUT
except for observation, I want the Histograms conditioned on both
obs. and
forecast. (e.g: if the obs. detection limit is 0.1, then I also want
to
remove the value less than 0,1 from the forecast too). Or my
understanding is not correct (we can ONLY make the Histograms
conditioned
on obs?)
Is that possible to do that using ONLY one configuration? Or we must
run
multiple times with different configurations?
Thank you.
Binyu
On Thu, May 20, 2021 at 11:30 PM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:
> Hello John,
>
> Yes both of your guesses are correct. Rank plots means "Ranked
histograms"
> and I do want the Ranked Histograms conditioned on
> observation values being greater than 0.01, 0.05, 0.1, and so on.
BUT
> except for observation, I want the Histograms conditioned on both
obs. and
> forecast. (e.g: if the obs. detection limit is 0.1, then I also want
to
> remove the value less than 0,1 from the forecast too). Or my
> understanding is not correct (we can ONLY make the Histograms
conditioned
> on obs?)
> Is that possible to do that using ONLY one configuration? Or we must
run
> multiple times with different configurations?
>
> Suppose we can only create the required *stat file for one condition
0.05,
> I am not sure where to put the two lines:
> censor_thresh=[<0.05];
> censor_val=[-9999]; desc="GE0.05";
> Should they be in the fcst entry or ens entry or both?
>
> I tried 3 different runs (the first time I put the two lines in
"fcst"
> entry and the second time I put the lines in the "ens" entry, and
third
> time I removed the lines), I got the same *stat files. I am not sure
what
> the problem is here?
> You can find the 3 config files:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.1
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.2
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.3
>
>
> Another question:
> Setting those censor_thresh will not affect "cat_thresh", right? In
other
> words, even if I set censor_thresh=[<0.05]; I still can set a
value
> less than 0.05 ( e.g: 0.01 in my config below) as one of the
cat_thresh (
> which will be used for the RELY plot), right?
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=3,>=5,>=7,
>=10 ];
> Thank you.
> Binyu
>
> On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binyu,
>>
>> I don't fully understand your question.
>>
>> By "rank plot" do you mean Ranked Histograms? Thresholds do NOT
apply to
>> ranked histogram plots. Ranked histograms are computed on raw
ensemble and
>> observation values.
>>
>> I'm just guessing here... perhaps you really mean that you want to
do
>> conditional verification. And you want Ranked Histograms
conditioned on
>> observation values being...
>> greater than 0.01, 0.05, 0.1, and so on.
>>
>> Is that correct?
>>
>> John
>>
>> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
>> >
>> > Hi Binyu,
>> >
>> > I assigned this ticket to John HG. Please allow a few business
days for
>> a
>> > response.
>> >
>> > Thanks,
>> > George
>> >
>> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
>> > > Transaction: Ticket created by binyu.wang at noaa.gov
>> > > Queue: met_help
>> > > Subject: ensemble verification
>> > > Owner: Nobody
>> > > Requestors: binyu.wang at noaa.gov
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
>> >
>> > >
>> > >
>> > > Hello,
>> > >
>> > >
>> > > I was using MET for ensemble verification and I had a question
about
>> > using
>> > > the cat_thresh. Please look at the config below, and the full
version
>> is
>> > > under:
>> > >
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
>> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
>> > >
>> > >
>> > > I am not sure if my config is right or not, I want to make such
plots
>> > > including:
>> > >
>> > >
>> > > (1).rank plot by setting different threshold (like both obs.
value and
>> > all
>> > > ensemble members values should larger than 0.1/0.05/0/01 ).
>> > >
>> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
>=0.1,>=0.3,
>> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
>> > >
>> > >
>> > > My config file is only setup for threshold=0.1, but I am not
sure if I
>> > > should move the two lines below from "fcst" entry to "ens"
entry?
>> > >
>> > >
>> > > censor_thresh=[<0.1];
>> > >
>> > > censor_val=[-9999]; desc="GE0.1";
>> > >
>> > >
>> > > --------------------------------------------------
>> > >
>> > >
>> > > fcst = {
>> > >
>> > > ens_thresh = 1.0;
>> > >
>> > > vld_thresh = 1.0;
>> > >
>> > >
>> > > field = [
>> > >
>> > > {
>> > >
>> > > name = "VAFTD";
>> > >
>> > > convert(x) = 21335*(10^(x));
>> > >
>> > > level = [ "L0-21335" ];
>> > >
>> > > censor_thresh=[<0.1];
>> > >
>> > > censor_val=[-9999]; desc="GE0.1";
>> > >
>> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>> >=0.5,>=1,>=3,>=5,>=7,
>> > > >=10 ];
>> > >
>> > > lead_time = "${LEAD_HR}";
>> > >
>> > > valid_time ="${EDATE}_${EHOUR}";
>> > >
>> > > }
>> > >
>> > > ];
>> > >
>> > > };
>> > >
>> > >
>> > > ens = {
>> > >
>> > > field = [
>> > >
>> > > {
>> > >
>> > > name = "VAFTD";
>> > >
>> > > convert(x) = 21335*(10^(x));
>> > >
>> > > level = [ "L0-21335" ];
>> > >
>> > > lead_time = "${LEAD_HR}";
>> > >
>> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>> > >=0.5,>=1,>=3,>=5,>=7,>=10
>> > > ];
>> > >
>> > >
>> > > }
>> > >
>> > > ];
>> > >
>> > > };
>> > >
>> > >
>> > > obs = {
>> > >
>> > >
>> > > file_type = NETCDF_MET;
>> > >
>> > > field = [
>> > >
>> > > {
>> > >
>> > > name = "ash_mass_loading";
>> > >
>> > > level = ["(*,*)"];
>> > >
>> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
>> > >
>> > > }
>> > >
>> > > ];
>> > >
>> > > ---------------------------------
>> > >
>> > >
>> > > If my config file is correct, is it possible that I can use
only one
>> run
>> > to
>> > > get the stat to make rank plots with multiple thresholds?
>> > >
>> > >
>> > > Thank you for help.
>> > >
>> > >
>> > > Binyu
>> > >
>> > >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
------------------------------------------------
Subject: ensemble verification
From: John Halley Gotway
Time: Fri May 21 12:05:45 2021
Binyu,
Your use of "cat_thresh" will not work the way you want it to.
You want to do conditional verification in Ensemble-Stat and it is the
"obs_thresh" option which enables you to filter the data that is
included
in the output. Listed below is some info from the user's guide:
https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
obs_thresh = [ NA ];
The *obs_thresh* entry is an array of thresholds for filtering
observation
values prior to applying ensemble verification logic. The default
setting
of *NA* means that no observations should be filtered out.
Verification
output will be computed separately for each threshold specified. This
option may be set separately for each *obs.field* entry.
So this enables you to get separate rank histogram output (RHIST line
type)
for each obs_thresh setting you define. However, applying the same
filtering logic to the ensemble member values is not really supported.
You
mention using the "censor_thresh" and "censor_val" options to reset
the
ensemble member values to bad data. But that will break the rank
histogram
logic. For some points, all ensemble members would have valid data
values
while for others there would be bad data present. The observation rank
is
not well defined if the ensemble members have bad data.
Personally I think that filtering based on the observation value makes
sense as is easy to understand. However, trying to condition of the
ensemble member values gets very confusing.
So please try using obs_thresh in this way.
Thanks,
John
On Fri, May 21, 2021 at 7:46 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
>
> Hello John,
>
> Just want to make it clear, so I reorganize the questions:
>
> 1. Will setting "censor_thresh" affect "cat_thresh"? e.g:
> field = [
>
> {
>
> name = "VAFTD";
>
> convert(x) = 21335*(10^(x));
>
> level = [ "L0-21335" ];
>
> censor_thresh=[<0.1];
>
> censor_val=[-9999]; desc="GE0.1";
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> >=10 ];
>
> }
>
> ];
>
> };
>
> I set 0.1 as "sensor_thresh", so can I still set a value less than
0.1 like
> 0.01/0.05 for the "cat_thresh"? Or only values higher than 0.1 will
be used
> for cat_thresh?
>
> 2.Suppose I only want to create the required *stat file for
Histogram plot
> based one condition 0.05, I am not sure where to put the two lines:
> censor_thresh=[<0.05];
> censor_val=[-9999]; desc="GE0.05";
> Should they be in the fcst entry or ens entry or both?
>
> I tried 3 different runs (the first time I put the two lines in
"fcst"
> entry and the second time I put the lines in the "ens" entry, and
third
> time I removed the lines), I got the same *stat files. I am not sure
what
> the problem is here?
> You can find the 3 config files:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.1
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.2
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> verf_g2g_ens_stat_regn_config_Bezy.3
>
> 3.
> Yes both of your guesses are correct. Rank plots means "Ranked
histograms"
> and I do want the Ranked Histograms conditioned on
> observation values being greater than 0.01, 0.05, 0.1, and so on.
BUT
> except for observation, I want the Histograms conditioned on both
obs. and
> forecast. (e.g: if the obs. detection limit is 0.1, then I also want
to
> remove the value less than 0,1 from the forecast too). Or my
> understanding is not correct (we can ONLY make the Histograms
conditioned
> on obs?)
> Is that possible to do that using ONLY one configuration? Or we must
run
> multiple times with different configurations?
>
> Thank you.
>
> Binyu
>
> On Thu, May 20, 2021 at 11:30 PM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > Hello John,
> >
> > Yes both of your guesses are correct. Rank plots means "Ranked
> histograms"
> > and I do want the Ranked Histograms conditioned on
> > observation values being greater than 0.01, 0.05, 0.1, and so on.
BUT
> > except for observation, I want the Histograms conditioned on both
obs.
> and
> > forecast. (e.g: if the obs. detection limit is 0.1, then I also
want to
> > remove the value less than 0,1 from the forecast too). Or my
> > understanding is not correct (we can ONLY make the Histograms
conditioned
> > on obs?)
> > Is that possible to do that using ONLY one configuration? Or we
must run
> > multiple times with different configurations?
> >
> > Suppose we can only create the required *stat file for one
condition
> 0.05,
> > I am not sure where to put the two lines:
> > censor_thresh=[<0.05];
> > censor_val=[-9999]; desc="GE0.05";
> > Should they be in the fcst entry or ens entry or both?
> >
> > I tried 3 different runs (the first time I put the two lines in
"fcst"
> > entry and the second time I put the lines in the "ens" entry, and
third
> > time I removed the lines), I got the same *stat files. I am not
sure what
> > the problem is here?
> > You can find the 3 config files:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.1
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.2
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.3
> >
> >
> > Another question:
> > Setting those censor_thresh will not affect "cat_thresh", right?
In other
> > words, even if I set censor_thresh=[<0.05]; I still can set a
value
> > less than 0.05 ( e.g: 0.01 in my config below) as one of the
cat_thresh (
> > which will be used for the RELY plot), right?
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=3,>=5,>=7,
>=10 ];
> > Thank you.
> > Binyu
> >
> > On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Binyu,
> >>
> >> I don't fully understand your question.
> >>
> >> By "rank plot" do you mean Ranked Histograms? Thresholds do NOT
apply to
> >> ranked histogram plots. Ranked histograms are computed on raw
ensemble
> and
> >> observation values.
> >>
> >> I'm just guessing here... perhaps you really mean that you want
to do
> >> conditional verification. And you want Ranked Histograms
conditioned on
> >> observation values being...
> >> greater than 0.01, 0.05, 0.1, and so on.
> >>
> >> Is that correct?
> >>
> >> John
> >>
> >> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> >> >
> >> > Hi Binyu,
> >> >
> >> > I assigned this ticket to John HG. Please allow a few business
days
> for
> >> a
> >> > response.
> >> >
> >> > Thanks,
> >> > George
> >> >
> >> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> >> > > Queue: met_help
> >> > > Subject: ensemble verification
> >> > > Owner: Nobody
> >> > > Requestors: binyu.wang at noaa.gov
> >> > > Status: new
> >> > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
> >> >
> >> > >
> >> > >
> >> > > Hello,
> >> > >
> >> > >
> >> > > I was using MET for ensemble verification and I had a
question about
> >> > using
> >> > > the cat_thresh. Please look at the config below, and the full
> version
> >> is
> >> > > under:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> >> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> >> > >
> >> > >
> >> > > I am not sure if my config is right or not, I want to make
such
> plots
> >> > > including:
> >> > >
> >> > >
> >> > > (1).rank plot by setting different threshold (like both obs.
value
> and
> >> > all
> >> > > ensemble members values should larger than 0.1/0.05/0/01 ).
> >> > >
> >> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
>=0.1,>=0.3,
> >> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> >> > >
> >> > >
> >> > > My config file is only setup for threshold=0.1, but I am not
sure
> if I
> >> > > should move the two lines below from "fcst" entry to "ens"
entry?
> >> > >
> >> > >
> >> > > censor_thresh=[<0.1];
> >> > >
> >> > > censor_val=[-9999]; desc="GE0.1";
> >> > >
> >> > >
> >> > > --------------------------------------------------
> >> > >
> >> > >
> >> > > fcst = {
> >> > >
> >> > > ens_thresh = 1.0;
> >> > >
> >> > > vld_thresh = 1.0;
> >> > >
> >> > >
> >> > > field = [
> >> > >
> >> > > {
> >> > >
> >> > > name = "VAFTD";
> >> > >
> >> > > convert(x) = 21335*(10^(x));
> >> > >
> >> > > level = [ "L0-21335" ];
> >> > >
> >> > > censor_thresh=[<0.1];
> >> > >
> >> > > censor_val=[-9999]; desc="GE0.1";
> >> > >
> >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >> >=0.5,>=1,>=3,>=5,>=7,
> >> > > >=10 ];
> >> > >
> >> > > lead_time = "${LEAD_HR}";
> >> > >
> >> > > valid_time ="${EDATE}_${EHOUR}";
> >> > >
> >> > > }
> >> > >
> >> > > ];
> >> > >
> >> > > };
> >> > >
> >> > >
> >> > > ens = {
> >> > >
> >> > > field = [
> >> > >
> >> > > {
> >> > >
> >> > > name = "VAFTD";
> >> > >
> >> > > convert(x) = 21335*(10^(x));
> >> > >
> >> > > level = [ "L0-21335" ];
> >> > >
> >> > > lead_time = "${LEAD_HR}";
> >> > >
> >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >> > >=0.5,>=1,>=3,>=5,>=7,>=10
> >> > > ];
> >> > >
> >> > >
> >> > > }
> >> > >
> >> > > ];
> >> > >
> >> > > };
> >> > >
> >> > >
> >> > > obs = {
> >> > >
> >> > >
> >> > > file_type = NETCDF_MET;
> >> > >
> >> > > field = [
> >> > >
> >> > > {
> >> > >
> >> > > name = "ash_mass_loading";
> >> > >
> >> > > level = ["(*,*)"];
> >> > >
> >> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> >> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> >> > >
> >> > > }
> >> > >
> >> > > ];
> >> > >
> >> > > ---------------------------------
> >> > >
> >> > >
> >> > > If my config file is correct, is it possible that I can use
only one
> >> run
> >> > to
> >> > > get the stat to make rank plots with multiple thresholds?
> >> > >
> >> > >
> >> > > Thank you for help.
> >> > >
> >> > >
> >> > > Binyu
> >> > >
> >> > >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: ensemble verification
From: binyu.wang at noaa.gov
Time: Fri May 21 14:38:31 2021
So the magnitude of "obs_thresh" will NOT affect the "cat_thresh",
Correct?
"obs_thresh" ONLY affect ensemble verification output,
""cat_thresh" is used for categorical statistics, they are independent
of
each other.
Thank you.
On Fri, May 21, 2021 at 2:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> Your use of "cat_thresh" will not work the way you want it to.
>
> You want to do conditional verification in Ensemble-Stat and it is
the
> "obs_thresh" option which enables you to filter the data that is
included
> in the output. Listed below is some info from the user's guide:
>
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
>
> obs_thresh = [ NA ];
>
> The *obs_thresh* entry is an array of thresholds for filtering
observation
> values prior to applying ensemble verification logic. The default
setting
> of *NA* means that no observations should be filtered out.
Verification
> output will be computed separately for each threshold specified.
This
> option may be set separately for each *obs.field* entry.
>
>
> So this enables you to get separate rank histogram output (RHIST
line type)
> for each obs_thresh setting you define. However, applying the same
> filtering logic to the ensemble member values is not really
supported. You
> mention using the "censor_thresh" and "censor_val" options to reset
the
> ensemble member values to bad data. But that will break the rank
histogram
> logic. For some points, all ensemble members would have valid data
values
> while for others there would be bad data present. The observation
rank is
> not well defined if the ensemble members have bad data.
>
> Personally I think that filtering based on the observation value
makes
> sense as is easy to understand. However, trying to condition of the
> ensemble member values gets very confusing.
>
> So please try using obs_thresh in this way.
>
> Thanks,
> John
>
>
> On Fri, May 21, 2021 at 7:46 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> >
> > Hello John,
> >
> > Just want to make it clear, so I reorganize the questions:
> >
> > 1. Will setting "censor_thresh" affect "cat_thresh"? e.g:
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > convert(x) = 21335*(10^(x));
> >
> > level = [ "L0-21335" ];
> >
> > censor_thresh=[<0.1];
> >
> > censor_val=[-9999]; desc="GE0.1";
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> > >=10 ];
> >
> > }
> >
> > ];
> >
> > };
> >
> > I set 0.1 as "sensor_thresh", so can I still set a value less than
0.1
> like
> > 0.01/0.05 for the "cat_thresh"? Or only values higher than 0.1
will be
> used
> > for cat_thresh?
> >
> > 2.Suppose I only want to create the required *stat file for
Histogram
> plot
> > based one condition 0.05, I am not sure where to put the two
lines:
> > censor_thresh=[<0.05];
> > censor_val=[-9999]; desc="GE0.05";
> > Should they be in the fcst entry or ens entry or both?
> >
> > I tried 3 different runs (the first time I put the two lines in
"fcst"
> > entry and the second time I put the lines in the "ens" entry, and
third
> > time I removed the lines), I got the same *stat files. I am not
sure what
> > the problem is here?
> > You can find the 3 config files:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.1
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.2
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > verf_g2g_ens_stat_regn_config_Bezy.3
> >
> > 3.
> > Yes both of your guesses are correct. Rank plots means "Ranked
> histograms"
> > and I do want the Ranked Histograms conditioned on
> > observation values being greater than 0.01, 0.05, 0.1, and so on.
BUT
> > except for observation, I want the Histograms conditioned on both
obs.
> and
> > forecast. (e.g: if the obs. detection limit is 0.1, then I also
want to
> > remove the value less than 0,1 from the forecast too). Or my
> > understanding is not correct (we can ONLY make the Histograms
conditioned
> > on obs?)
> > Is that possible to do that using ONLY one configuration? Or we
must run
> > multiple times with different configurations?
> >
> > Thank you.
> >
> > Binyu
> >
> > On Thu, May 20, 2021 at 11:30 PM Binyu Wang - NOAA Affiliate <
> > binyu.wang at noaa.gov> wrote:
> >
> > > Hello John,
> > >
> > > Yes both of your guesses are correct. Rank plots means "Ranked
> > histograms"
> > > and I do want the Ranked Histograms conditioned on
> > > observation values being greater than 0.01, 0.05, 0.1, and so
on. BUT
> > > except for observation, I want the Histograms conditioned on
both obs.
> > and
> > > forecast. (e.g: if the obs. detection limit is 0.1, then I also
want to
> > > remove the value less than 0,1 from the forecast too). Or my
> > > understanding is not correct (we can ONLY make the Histograms
> conditioned
> > > on obs?)
> > > Is that possible to do that using ONLY one configuration? Or we
must
> run
> > > multiple times with different configurations?
> > >
> > > Suppose we can only create the required *stat file for one
condition
> > 0.05,
> > > I am not sure where to put the two lines:
> > > censor_thresh=[<0.05];
> > > censor_val=[-9999]; desc="GE0.05";
> > > Should they be in the fcst entry or ens entry or both?
> > >
> > > I tried 3 different runs (the first time I put the two lines in
"fcst"
> > > entry and the second time I put the lines in the "ens" entry,
and third
> > > time I removed the lines), I got the same *stat files. I am not
sure
> what
> > > the problem is here?
> > > You can find the 3 config files:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.1
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.2
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.3
> > >
> > >
> > > Another question:
> > > Setting those censor_thresh will not affect "cat_thresh", right?
In
> other
> > > words, even if I set censor_thresh=[<0.05]; I still can set a
value
> > > less than 0.05 ( e.g: 0.01 in my config below) as one of the
> cat_thresh (
> > > which will be used for the RELY plot), right?
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7, >=10
> ];
> > > Thank you.
> > > Binyu
> > >
> > > On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Binyu,
> > >>
> > >> I don't fully understand your question.
> > >>
> > >> By "rank plot" do you mean Ranked Histograms? Thresholds do NOT
apply
> to
> > >> ranked histogram plots. Ranked histograms are computed on raw
ensemble
> > and
> > >> observation values.
> > >>
> > >> I'm just guessing here... perhaps you really mean that you want
to do
> > >> conditional verification. And you want Ranked Histograms
conditioned
> on
> > >> observation values being...
> > >> greater than 0.01, 0.05, 0.1, and so on.
> > >>
> > >> Is that correct?
> > >>
> > >> John
> > >>
> > >> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
>
> > >> >
> > >> > Hi Binyu,
> > >> >
> > >> > I assigned this ticket to John HG. Please allow a few
business days
> > for
> > >> a
> > >> > response.
> > >> >
> > >> > Thanks,
> > >> > George
> > >> >
> > >> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > >> > > Queue: met_help
> > >> > > Subject: ensemble verification
> > >> > > Owner: Nobody
> > >> > > Requestors: binyu.wang at noaa.gov
> > >> > > Status: new
> > >> > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
> > >> >
> > >> > >
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > >
> > >> > > I was using MET for ensemble verification and I had a
question
> about
> > >> > using
> > >> > > the cat_thresh. Please look at the config below, and the
full
> > version
> > >> is
> > >> > > under:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > >> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> > >> > >
> > >> > >
> > >> > > I am not sure if my config is right or not, I want to make
such
> > plots
> > >> > > including:
> > >> > >
> > >> > >
> > >> > > (1).rank plot by setting different threshold (like both
obs. value
> > and
> > >> > all
> > >> > > ensemble members values should larger than 0.1/0.05/0/01 ).
> > >> > >
> > >> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
> >=0.1,>=0.3,
> > >> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> > >> > >
> > >> > >
> > >> > > My config file is only setup for threshold=0.1, but I am
not sure
> > if I
> > >> > > should move the two lines below from "fcst" entry to "ens"
entry?
> > >> > >
> > >> > >
> > >> > > censor_thresh=[<0.1];
> > >> > >
> > >> > > censor_val=[-9999]; desc="GE0.1";
> > >> > >
> > >> > >
> > >> > > --------------------------------------------------
> > >> > >
> > >> > >
> > >> > > fcst = {
> > >> > >
> > >> > > ens_thresh = 1.0;
> > >> > >
> > >> > > vld_thresh = 1.0;
> > >> > >
> > >> > >
> > >> > > field = [
> > >> > >
> > >> > > {
> > >> > >
> > >> > > name = "VAFTD";
> > >> > >
> > >> > > convert(x) = 21335*(10^(x));
> > >> > >
> > >> > > level = [ "L0-21335" ];
> > >> > >
> > >> > > censor_thresh=[<0.1];
> > >> > >
> > >> > > censor_val=[-9999]; desc="GE0.1";
> > >> > >
> > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >> >=0.5,>=1,>=3,>=5,>=7,
> > >> > > >=10 ];
> > >> > >
> > >> > > lead_time = "${LEAD_HR}";
> > >> > >
> > >> > > valid_time ="${EDATE}_${EHOUR}";
> > >> > >
> > >> > > }
> > >> > >
> > >> > > ];
> > >> > >
> > >> > > };
> > >> > >
> > >> > >
> > >> > > ens = {
> > >> > >
> > >> > > field = [
> > >> > >
> > >> > > {
> > >> > >
> > >> > > name = "VAFTD";
> > >> > >
> > >> > > convert(x) = 21335*(10^(x));
> > >> > >
> > >> > > level = [ "L0-21335" ];
> > >> > >
> > >> > > lead_time = "${LEAD_HR}";
> > >> > >
> > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >> > >=0.5,>=1,>=3,>=5,>=7,>=10
> > >> > > ];
> > >> > >
> > >> > >
> > >> > > }
> > >> > >
> > >> > > ];
> > >> > >
> > >> > > };
> > >> > >
> > >> > >
> > >> > > obs = {
> > >> > >
> > >> > >
> > >> > > file_type = NETCDF_MET;
> > >> > >
> > >> > > field = [
> > >> > >
> > >> > > {
> > >> > >
> > >> > > name = "ash_mass_loading";
> > >> > >
> > >> > > level = ["(*,*)"];
> > >> > >
> > >> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > >> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> > >> > >
> > >> > > }
> > >> > >
> > >> > > ];
> > >> > >
> > >> > > ---------------------------------
> > >> > >
> > >> > >
> > >> > > If my config file is correct, is it possible that I can use
only
> one
> > >> run
> > >> > to
> > >> > > get the stat to make rank plots with multiple thresholds?
> > >> > >
> > >> > >
> > >> > > Thank you for help.
> > >> > >
> > >> > >
> > >> > > Binyu
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: ensemble verification
From: John Halley Gotway
Time: Fri May 21 15:01:51 2021
Binyu,
Yes, in the ensemble-stat config file...
- set obs_thresh in the obs.field entries to specify how to do
conditional
verification... filtering the pairs based on each obs threshold.
- the cat_thresh in the ens.field entries defines the thresholds for
which
you want probability forecasts defined and written to gridded NetCDF
output
file.
So yes, obs_thresh in the obs dictionary operates independently of the
cat_thresh in the ens dictionary.
John
On Fri, May 21, 2021 at 2:38 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
>
> So the magnitude of "obs_thresh" will NOT affect the "cat_thresh",
> Correct?
> "obs_thresh" ONLY affect ensemble verification output,
> ""cat_thresh" is used for categorical statistics, they are
independent of
> each other.
>
> Thank you.
>
>
>
> On Fri, May 21, 2021 at 2:05 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > Your use of "cat_thresh" will not work the way you want it to.
> >
> > You want to do conditional verification in Ensemble-Stat and it is
the
> > "obs_thresh" option which enables you to filter the data that is
included
> > in the output. Listed below is some info from the user's guide:
> >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> >
> > obs_thresh = [ NA ];
> >
> > The *obs_thresh* entry is an array of thresholds for filtering
> observation
> > values prior to applying ensemble verification logic. The default
setting
> > of *NA* means that no observations should be filtered out.
Verification
> > output will be computed separately for each threshold specified.
This
> > option may be set separately for each *obs.field* entry.
> >
> >
> > So this enables you to get separate rank histogram output (RHIST
line
> type)
> > for each obs_thresh setting you define. However, applying the same
> > filtering logic to the ensemble member values is not really
supported.
> You
> > mention using the "censor_thresh" and "censor_val" options to
reset the
> > ensemble member values to bad data. But that will break the rank
> histogram
> > logic. For some points, all ensemble members would have valid data
values
> > while for others there would be bad data present. The observation
rank is
> > not well defined if the ensemble members have bad data.
> >
> > Personally I think that filtering based on the observation value
makes
> > sense as is easy to understand. However, trying to condition of
the
> > ensemble member values gets very confusing.
> >
> > So please try using obs_thresh in this way.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, May 21, 2021 at 7:46 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> > >
> > > Hello John,
> > >
> > > Just want to make it clear, so I reorganize the questions:
> > >
> > > 1. Will setting "censor_thresh" affect "cat_thresh"? e.g:
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > convert(x) = 21335*(10^(x));
> > >
> > > level = [ "L0-21335" ];
> > >
> > > censor_thresh=[<0.1];
> > >
> > > censor_val=[-9999]; desc="GE0.1";
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=3,>=5,>=7,
> > > >=10 ];
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > > I set 0.1 as "sensor_thresh", so can I still set a value less
than 0.1
> > like
> > > 0.01/0.05 for the "cat_thresh"? Or only values higher than 0.1
will be
> > used
> > > for cat_thresh?
> > >
> > > 2.Suppose I only want to create the required *stat file for
Histogram
> > plot
> > > based one condition 0.05, I am not sure where to put the two
lines:
> > > censor_thresh=[<0.05];
> > > censor_val=[-9999]; desc="GE0.05";
> > > Should they be in the fcst entry or ens entry or both?
> > >
> > > I tried 3 different runs (the first time I put the two lines in
"fcst"
> > > entry and the second time I put the lines in the "ens" entry,
and third
> > > time I removed the lines), I got the same *stat files. I am not
sure
> what
> > > the problem is here?
> > > You can find the 3 config files:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.1
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.2
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > verf_g2g_ens_stat_regn_config_Bezy.3
> > >
> > > 3.
> > > Yes both of your guesses are correct. Rank plots means "Ranked
> > histograms"
> > > and I do want the Ranked Histograms conditioned on
> > > observation values being greater than 0.01, 0.05, 0.1, and so
on. BUT
> > > except for observation, I want the Histograms conditioned on
both obs.
> > and
> > > forecast. (e.g: if the obs. detection limit is 0.1, then I also
want to
> > > remove the value less than 0,1 from the forecast too). Or my
> > > understanding is not correct (we can ONLY make the Histograms
> conditioned
> > > on obs?)
> > > Is that possible to do that using ONLY one configuration? Or we
must
> run
> > > multiple times with different configurations?
> > >
> > > Thank you.
> > >
> > > Binyu
> > >
> > > On Thu, May 20, 2021 at 11:30 PM Binyu Wang - NOAA Affiliate <
> > > binyu.wang at noaa.gov> wrote:
> > >
> > > > Hello John,
> > > >
> > > > Yes both of your guesses are correct. Rank plots means "Ranked
> > > histograms"
> > > > and I do want the Ranked Histograms conditioned on
> > > > observation values being greater than 0.01, 0.05, 0.1, and so
on. BUT
> > > > except for observation, I want the Histograms conditioned on
both
> obs.
> > > and
> > > > forecast. (e.g: if the obs. detection limit is 0.1, then I
also want
> to
> > > > remove the value less than 0,1 from the forecast too). Or my
> > > > understanding is not correct (we can ONLY make the Histograms
> > conditioned
> > > > on obs?)
> > > > Is that possible to do that using ONLY one configuration? Or
we must
> > run
> > > > multiple times with different configurations?
> > > >
> > > > Suppose we can only create the required *stat file for one
condition
> > > 0.05,
> > > > I am not sure where to put the two lines:
> > > > censor_thresh=[<0.05];
> > > > censor_val=[-9999]; desc="GE0.05";
> > > > Should they be in the fcst entry or ens entry or both?
> > > >
> > > > I tried 3 different runs (the first time I put the two lines
in
> "fcst"
> > > > entry and the second time I put the lines in the "ens" entry,
and
> third
> > > > time I removed the lines), I got the same *stat files. I am
not sure
> > what
> > > > the problem is here?
> > > > You can find the 3 config files:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.1
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.2
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.3
> > > >
> > > >
> > > > Another question:
> > > > Setting those censor_thresh will not affect "cat_thresh",
right? In
> > other
> > > > words, even if I set censor_thresh=[<0.05]; I still can set
a
> value
> > > > less than 0.05 ( e.g: 0.01 in my config below) as one of the
> > cat_thresh (
> > > > which will be used for the RELY plot), right?
> > > >
> > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> >=10
> > ];
> > > > Thank you.
> > > > Binyu
> > > >
> > > > On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Binyu,
> > > >>
> > > >> I don't fully understand your question.
> > > >>
> > > >> By "rank plot" do you mean Ranked Histograms? Thresholds do
NOT
> apply
> > to
> > > >> ranked histogram plots. Ranked histograms are computed on raw
> ensemble
> > > and
> > > >> observation values.
> > > >>
> > > >> I'm just guessing here... perhaps you really mean that you
want to
> do
> > > >> conditional verification. And you want Ranked Histograms
conditioned
> > on
> > > >> observation values being...
> > > >> greater than 0.01, 0.05, 0.1, and so on.
> > > >>
> > > >> Is that correct?
> > > >>
> > > >> John
> > > >>
> > > >> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> > > >> >
> > > >> > Hi Binyu,
> > > >> >
> > > >> > I assigned this ticket to John HG. Please allow a few
business
> days
> > > for
> > > >> a
> > > >> > response.
> > > >> >
> > > >> > Thanks,
> > > >> > George
> > > >> >
> > > >> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via RT
<
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> > > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > >> > > Queue: met_help
> > > >> > > Subject: ensemble verification
> > > >> > > Owner: Nobody
> > > >> > > Requestors: binyu.wang at noaa.gov
> > > >> > > Status: new
> > > >> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
> > > >> >
> > > >> > >
> > > >> > >
> > > >> > > Hello,
> > > >> > >
> > > >> > >
> > > >> > > I was using MET for ensemble verification and I had a
question
> > about
> > > >> > using
> > > >> > > the cat_thresh. Please look at the config below, and the
full
> > > version
> > > >> is
> > > >> > > under:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > >> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> > > >> > >
> > > >> > >
> > > >> > > I am not sure if my config is right or not, I want to
make such
> > > plots
> > > >> > > including:
> > > >> > >
> > > >> > >
> > > >> > > (1).rank plot by setting different threshold (like both
obs.
> value
> > > and
> > > >> > all
> > > >> > > ensemble members values should larger than 0.1/0.05/0/01
).
> > > >> > >
> > > >> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
> > >=0.1,>=0.3,
> > > >> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> > > >> > >
> > > >> > >
> > > >> > > My config file is only setup for threshold=0.1, but I am
not
> sure
> > > if I
> > > >> > > should move the two lines below from "fcst" entry to
"ens"
> entry?
> > > >> > >
> > > >> > >
> > > >> > > censor_thresh=[<0.1];
> > > >> > >
> > > >> > > censor_val=[-9999]; desc="GE0.1";
> > > >> > >
> > > >> > >
> > > >> > > --------------------------------------------------
> > > >> > >
> > > >> > >
> > > >> > > fcst = {
> > > >> > >
> > > >> > > ens_thresh = 1.0;
> > > >> > >
> > > >> > > vld_thresh = 1.0;
> > > >> > >
> > > >> > >
> > > >> > > field = [
> > > >> > >
> > > >> > > {
> > > >> > >
> > > >> > > name = "VAFTD";
> > > >> > >
> > > >> > > convert(x) = 21335*(10^(x));
> > > >> > >
> > > >> > > level = [ "L0-21335" ];
> > > >> > >
> > > >> > > censor_thresh=[<0.1];
> > > >> > >
> > > >> > > censor_val=[-9999]; desc="GE0.1";
> > > >> > >
> > > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >> >=0.5,>=1,>=3,>=5,>=7,
> > > >> > > >=10 ];
> > > >> > >
> > > >> > > lead_time = "${LEAD_HR}";
> > > >> > >
> > > >> > > valid_time ="${EDATE}_${EHOUR}";
> > > >> > >
> > > >> > > }
> > > >> > >
> > > >> > > ];
> > > >> > >
> > > >> > > };
> > > >> > >
> > > >> > >
> > > >> > > ens = {
> > > >> > >
> > > >> > > field = [
> > > >> > >
> > > >> > > {
> > > >> > >
> > > >> > > name = "VAFTD";
> > > >> > >
> > > >> > > convert(x) = 21335*(10^(x));
> > > >> > >
> > > >> > > level = [ "L0-21335" ];
> > > >> > >
> > > >> > > lead_time = "${LEAD_HR}";
> > > >> > >
> > > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >> > >=0.5,>=1,>=3,>=5,>=7,>=10
> > > >> > > ];
> > > >> > >
> > > >> > >
> > > >> > > }
> > > >> > >
> > > >> > > ];
> > > >> > >
> > > >> > > };
> > > >> > >
> > > >> > >
> > > >> > > obs = {
> > > >> > >
> > > >> > >
> > > >> > > file_type = NETCDF_MET;
> > > >> > >
> > > >> > > field = [
> > > >> > >
> > > >> > > {
> > > >> > >
> > > >> > > name = "ash_mass_loading";
> > > >> > >
> > > >> > > level = ["(*,*)"];
> > > >> > >
> > > >> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > >> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> > > >> > >
> > > >> > > }
> > > >> > >
> > > >> > > ];
> > > >> > >
> > > >> > > ---------------------------------
> > > >> > >
> > > >> > >
> > > >> > > If my config file is correct, is it possible that I can
use only
> > one
> > > >> run
> > > >> > to
> > > >> > > get the stat to make rank plots with multiple thresholds?
> > > >> > >
> > > >> > >
> > > >> > > Thank you for help.
> > > >> > >
> > > >> > >
> > > >> > > Binyu
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > George McCabe - Software Engineer III
> > > >> > National Center for Atmospheric Research
> > > >> > Research Applications Laboratory
> > > >> > 303-497-2768
> > > >> > ---
> > > >> > My working day may not be your working day. Please do not
feel
> > obliged
> > > >> to
> > > >> > reply to this email outside of your normal working hours.
> > > >> >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ensemble verification
From: binyu.wang at noaa.gov
Time: Fri May 21 15:31:10 2021
There is another thing I want to confirm:
Are the two config files the same? I mean will they provide the same
output?
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.1
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.2
To set the filter (>0.05) for both obs. and forecast for ensemble
verification, I should use the config file below for ensemble, is
that correct?
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
verf_g2g_ens_stat_regn_config_Bezy.3
Or is it not reasonable to set both filters for obs. and forecast?
Thank you.
On Fri, May 21, 2021 at 5:01 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> Yes, in the ensemble-stat config file...
> - set obs_thresh in the obs.field entries to specify how to do
conditional
> verification... filtering the pairs based on each obs threshold.
> - the cat_thresh in the ens.field entries defines the thresholds for
which
> you want probability forecasts defined and written to gridded NetCDF
output
> file.
>
> So yes, obs_thresh in the obs dictionary operates independently of
the
> cat_thresh in the ens dictionary.
>
> John
>
> On Fri, May 21, 2021 at 2:38 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> >
> > So the magnitude of "obs_thresh" will NOT affect the
"cat_thresh",
> > Correct?
> > "obs_thresh" ONLY affect ensemble verification output,
> > ""cat_thresh" is used for categorical statistics, they are
independent of
> > each other.
> >
> > Thank you.
> >
> >
> >
> > On Fri, May 21, 2021 at 2:05 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > Your use of "cat_thresh" will not work the way you want it to.
> > >
> > > You want to do conditional verification in Ensemble-Stat and it
is the
> > > "obs_thresh" option which enables you to filter the data that is
> included
> > > in the output. Listed below is some info from the user's guide:
> > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#ensemble-stat-configuration-file
> > >
> > > obs_thresh = [ NA ];
> > >
> > > The *obs_thresh* entry is an array of thresholds for filtering
> > observation
> > > values prior to applying ensemble verification logic. The
default
> setting
> > > of *NA* means that no observations should be filtered out.
Verification
> > > output will be computed separately for each threshold specified.
This
> > > option may be set separately for each *obs.field* entry.
> > >
> > >
> > > So this enables you to get separate rank histogram output (RHIST
line
> > type)
> > > for each obs_thresh setting you define. However, applying the
same
> > > filtering logic to the ensemble member values is not really
supported.
> > You
> > > mention using the "censor_thresh" and "censor_val" options to
reset the
> > > ensemble member values to bad data. But that will break the rank
> > histogram
> > > logic. For some points, all ensemble members would have valid
data
> values
> > > while for others there would be bad data present. The
observation rank
> is
> > > not well defined if the ensemble members have bad data.
> > >
> > > Personally I think that filtering based on the observation value
makes
> > > sense as is easy to understand. However, trying to condition of
the
> > > ensemble member values gets very confusing.
> > >
> > > So please try using obs_thresh in this way.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, May 21, 2021 at 7:46 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
>
> > > >
> > > > Hello John,
> > > >
> > > > Just want to make it clear, so I reorganize the questions:
> > > >
> > > > 1. Will setting "censor_thresh" affect "cat_thresh"? e.g:
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "VAFTD";
> > > >
> > > > convert(x) = 21335*(10^(x));
> > > >
> > > > level = [ "L0-21335" ];
> > > >
> > > > censor_thresh=[<0.1];
> > > >
> > > > censor_val=[-9999]; desc="GE0.1";
> > > >
> > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=3,>=5,>=7,
> > > > >=10 ];
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > };
> > > >
> > > > I set 0.1 as "sensor_thresh", so can I still set a value less
than
> 0.1
> > > like
> > > > 0.01/0.05 for the "cat_thresh"? Or only values higher than 0.1
will
> be
> > > used
> > > > for cat_thresh?
> > > >
> > > > 2.Suppose I only want to create the required *stat file for
> Histogram
> > > plot
> > > > based one condition 0.05, I am not sure where to put the two
lines:
> > > > censor_thresh=[<0.05];
> > > > censor_val=[-9999]; desc="GE0.05";
> > > > Should they be in the fcst entry or ens entry or both?
> > > >
> > > > I tried 3 different runs (the first time I put the two lines
in
> "fcst"
> > > > entry and the second time I put the lines in the "ens" entry,
and
> third
> > > > time I removed the lines), I got the same *stat files. I am
not sure
> > what
> > > > the problem is here?
> > > > You can find the 3 config files:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.1
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.2
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > verf_g2g_ens_stat_regn_config_Bezy.3
> > > >
> > > > 3.
> > > > Yes both of your guesses are correct. Rank plots means "Ranked
> > > histograms"
> > > > and I do want the Ranked Histograms conditioned on
> > > > observation values being greater than 0.01, 0.05, 0.1, and so
on. BUT
> > > > except for observation, I want the Histograms conditioned on
both
> obs.
> > > and
> > > > forecast. (e.g: if the obs. detection limit is 0.1, then I
also want
> to
> > > > remove the value less than 0,1 from the forecast too). Or my
> > > > understanding is not correct (we can ONLY make the Histograms
> > conditioned
> > > > on obs?)
> > > > Is that possible to do that using ONLY one configuration? Or
we must
> > run
> > > > multiple times with different configurations?
> > > >
> > > > Thank you.
> > > >
> > > > Binyu
> > > >
> > > > On Thu, May 20, 2021 at 11:30 PM Binyu Wang - NOAA Affiliate <
> > > > binyu.wang at noaa.gov> wrote:
> > > >
> > > > > Hello John,
> > > > >
> > > > > Yes both of your guesses are correct. Rank plots means
"Ranked
> > > > histograms"
> > > > > and I do want the Ranked Histograms conditioned on
> > > > > observation values being greater than 0.01, 0.05, 0.1, and
so on.
> BUT
> > > > > except for observation, I want the Histograms conditioned
on both
> > obs.
> > > > and
> > > > > forecast. (e.g: if the obs. detection limit is 0.1, then I
also
> want
> > to
> > > > > remove the value less than 0,1 from the forecast too). Or my
> > > > > understanding is not correct (we can ONLY make the
Histograms
> > > conditioned
> > > > > on obs?)
> > > > > Is that possible to do that using ONLY one configuration? Or
we
> must
> > > run
> > > > > multiple times with different configurations?
> > > > >
> > > > > Suppose we can only create the required *stat file for one
> condition
> > > > 0.05,
> > > > > I am not sure where to put the two lines:
> > > > > censor_thresh=[<0.05];
> > > > > censor_val=[-9999]; desc="GE0.05";
> > > > > Should they be in the fcst entry or ens entry or both?
> > > > >
> > > > > I tried 3 different runs (the first time I put the two lines
in
> > "fcst"
> > > > > entry and the second time I put the lines in the "ens"
entry, and
> > third
> > > > > time I removed the lines), I got the same *stat files. I am
not
> sure
> > > what
> > > > > the problem is here?
> > > > > You can find the 3 config files:
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > > verf_g2g_ens_stat_regn_config_Bezy.1
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > > verf_g2g_ens_stat_regn_config_Bezy.2
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > > verf_g2g_ens_stat_regn_config_Bezy.3
> > > > >
> > > > >
> > > > > Another question:
> > > > > Setting those censor_thresh will not affect "cat_thresh",
right? In
> > > other
> > > > > words, even if I set censor_thresh=[<0.05]; I still can
set a
> > value
> > > > > less than 0.05 ( e.g: 0.01 in my config below) as one of the
> > > cat_thresh (
> > > > > which will be used for the RELY plot), right?
> > > > >
> > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=3,>=5,>=7,
> > >=10
> > > ];
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > > On Thu, May 20, 2021 at 6:46 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Binyu,
> > > > >>
> > > > >> I don't fully understand your question.
> > > > >>
> > > > >> By "rank plot" do you mean Ranked Histograms? Thresholds do
NOT
> > apply
> > > to
> > > > >> ranked histogram plots. Ranked histograms are computed on
raw
> > ensemble
> > > > and
> > > > >> observation values.
> > > > >>
> > > > >> I'm just guessing here... perhaps you really mean that you
want to
> > do
> > > > >> conditional verification. And you want Ranked Histograms
> conditioned
> > > on
> > > > >> observation values being...
> > > > >> greater than 0.01, 0.05, 0.1, and so on.
> > > > >>
> > > > >> Is that correct?
> > > > >>
> > > > >> John
> > > > >>
> > > > >> On Wed, May 19, 2021 at 4:43 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940 >
> > > > >> >
> > > > >> > Hi Binyu,
> > > > >> >
> > > > >> > I assigned this ticket to John HG. Please allow a few
business
> > days
> > > > for
> > > > >> a
> > > > >> > response.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > George
> > > > >> >
> > > > >> > On Wed, May 19, 2021 at 4:19 PM binyu.wang at noaa.gov via
RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > Wed May 19 16:19:08 2021: Request 99940 was acted upon.
> > > > >> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > >> > > Queue: met_help
> > > > >> > > Subject: ensemble verification
> > > > >> > > Owner: Nobody
> > > > >> > > Requestors: binyu.wang at noaa.gov
> > > > >> > > Status: new
> > > > >> > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99940
> > > > >> >
> > > > >> > >
> > > > >> > >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > >
> > > > >> > > I was using MET for ensemble verification and I had a
question
> > > about
> > > > >> > using
> > > > >> > > the cat_thresh. Please look at the config below, and
the full
> > > > version
> > > > >> is
> > > > >> > > under:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/
> > > > >> > > verf_g2g_ens_stat_regn_config_Bezy_thr0p05
> > > > >> > >
> > > > >> > >
> > > > >> > > I am not sure if my config is right or not, I want to
make
> such
> > > > plots
> > > > >> > > including:
> > > > >> > >
> > > > >> > >
> > > > >> > > (1).rank plot by setting different threshold (like both
obs.
> > value
> > > > and
> > > > >> > all
> > > > >> > > ensemble members values should larger than
0.1/0.05/0/01 ).
> > > > >> > >
> > > > >> > > (2).Rely plots for different threshold [>=0.01, >=0.05,
> > > >=0.1,>=0.3,
> > > > >> > > >=0.5,>=1,>=3,>=5,>=7, >=10 ];
> > > > >> > >
> > > > >> > >
> > > > >> > > My config file is only setup for threshold=0.1, but I
am not
> > sure
> > > > if I
> > > > >> > > should move the two lines below from "fcst" entry to
"ens"
> > entry?
> > > > >> > >
> > > > >> > >
> > > > >> > > censor_thresh=[<0.1];
> > > > >> > >
> > > > >> > > censor_val=[-9999]; desc="GE0.1";
> > > > >> > >
> > > > >> > >
> > > > >> > > --------------------------------------------------
> > > > >> > >
> > > > >> > >
> > > > >> > > fcst = {
> > > > >> > >
> > > > >> > > ens_thresh = 1.0;
> > > > >> > >
> > > > >> > > vld_thresh = 1.0;
> > > > >> > >
> > > > >> > >
> > > > >> > > field = [
> > > > >> > >
> > > > >> > > {
> > > > >> > >
> > > > >> > > name = "VAFTD";
> > > > >> > >
> > > > >> > > convert(x) = 21335*(10^(x));
> > > > >> > >
> > > > >> > > level = [ "L0-21335" ];
> > > > >> > >
> > > > >> > > censor_thresh=[<0.1];
> > > > >> > >
> > > > >> > > censor_val=[-9999]; desc="GE0.1";
> > > > >> > >
> > > > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >> >=0.5,>=1,>=3,>=5,>=7,
> > > > >> > > >=10 ];
> > > > >> > >
> > > > >> > > lead_time = "${LEAD_HR}";
> > > > >> > >
> > > > >> > > valid_time ="${EDATE}_${EHOUR}";
> > > > >> > >
> > > > >> > > }
> > > > >> > >
> > > > >> > > ];
> > > > >> > >
> > > > >> > > };
> > > > >> > >
> > > > >> > >
> > > > >> > > ens = {
> > > > >> > >
> > > > >> > > field = [
> > > > >> > >
> > > > >> > > {
> > > > >> > >
> > > > >> > > name = "VAFTD";
> > > > >> > >
> > > > >> > > convert(x) = 21335*(10^(x));
> > > > >> > >
> > > > >> > > level = [ "L0-21335" ];
> > > > >> > >
> > > > >> > > lead_time = "${LEAD_HR}";
> > > > >> > >
> > > > >> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >> > >=0.5,>=1,>=3,>=5,>=7,>=10
> > > > >> > > ];
> > > > >> > >
> > > > >> > >
> > > > >> > > }
> > > > >> > >
> > > > >> > > ];
> > > > >> > >
> > > > >> > > };
> > > > >> > >
> > > > >> > >
> > > > >> > > obs = {
> > > > >> > >
> > > > >> > >
> > > > >> > > file_type = NETCDF_MET;
> > > > >> > >
> > > > >> > > field = [
> > > > >> > >
> > > > >> > > {
> > > > >> > >
> > > > >> > > name = "ash_mass_loading";
> > > > >> > >
> > > > >> > > level = ["(*,*)"];
> > > > >> > >
> > > > >> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > >> > > >=0.5,>=1,>=3,>=5,>=7,>=10 ];
> > > > >> > >
> > > > >> > > }
> > > > >> > >
> > > > >> > > ];
> > > > >> > >
> > > > >> > > ---------------------------------
> > > > >> > >
> > > > >> > >
> > > > >> > > If my config file is correct, is it possible that I can
use
> only
> > > one
> > > > >> run
> > > > >> > to
> > > > >> > > get the stat to make rank plots with multiple
thresholds?
> > > > >> > >
> > > > >> > >
> > > > >> > > Thank you for help.
> > > > >> > >
> > > > >> > >
> > > > >> > > Binyu
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > George McCabe - Software Engineer III
> > > > >> > National Center for Atmospheric Research
> > > > >> > Research Applications Laboratory
> > > > >> > 303-497-2768
> > > > >> > ---
> > > > >> > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > >> to
> > > > >> > reply to this email outside of your normal working hours.
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ensemble verification
From: John Halley Gotway
Time: Mon Jul 12 11:34:25 2021
Binyu,
The METplus team recently switched from providing support via
met_help at ucar.edu to doing so via GitHub Discussions:
https://github.com/dtcenter/METplus/discussions
I'm going through existing tickets in the MET-Help queue and realize
that I never responded to this question you asked. Do you have any
remaining questions about ensemble-stat before I close this ticket?
Thanks,
John HG
------------------------------------------------
Subject: ensemble verification
From: binyu.wang at noaa.gov
Time: Tue Jul 13 07:26:24 2021
Hello John,
Yes you can close the ticket now, thank you.
Binyu
On Mon, Jul 12, 2021 at 1:47 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> The METplus team recently switched from providing support via
> met_help at ucar.edu to doing so via GitHub Discussions:
> https://github.com/dtcenter/METplus/discussions
>
> I'm going through existing tickets in the MET-Help queue and realize
that
> I never responded to this question you asked. Do you have any
remaining
> questions about ensemble-stat before I close this ticket?
>
> Thanks,
> John HG
>
------------------------------------------------
More information about the Met_help
mailing list