[Met_help] [rt.rap.ucar.edu #99973] History for config files
John Halley Gotway via RT
met_help at ucar.edu
Mon Jun 7 08:45:31 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I have two ensemble config files as below (the other parts of the config
are the same), will they give exactly the same output?
The only difference lies in there is no "cat_thresh" in fcst and obs
entries. Based on my tests, it looks like the ensemble results are the same
with the two config files below, but I am afraid I may miss some output
flags.
1.
fcst = {
ens_thresh = 1.0;
vld_thresh = 1.0;
field = [
{
name = "VAFTD";
level = [ "L0-21335" ];
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
lead_time = "${LEAD_HR}";
}
];
};
ens = {
field = [
{
name = "VAFTD";
level = [ "L0-21335" ];
lead_time = "${LEAD_HR}";
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
}
];
};
obs = {
field = [
{
name = "ash_mass_loading";
level = ["(*,*)"];
cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
}
];
---------------------------------------------
2.
fcst = {
ens_thresh = 1.0;
vld_thresh = 1.0;
field = [
{
name = "VAFTD";
level = [ "L0-21335" ];
// cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
lead_time = "${LEAD_HR}";
}
];
};
ens = {
field = [
{
name = "VAFTD";
level = [ "L0-21335" ];
lead_time = "${LEAD_HR}";
cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
}
];
};
obs = {
field = [
{
name = "ash_mass_loading";
level = ["(*,*)"];
// cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10 ];
}
];
So if they will lead to different outputs, what will be the difference?
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Mon May 24 16:15:45 2021
Binyu,
No that's correct. In ensemble_stat the cat_thresh setting applies to
the
"ens" dictionary and defines the thresholds that to be used to in
defining
probabilities written to the NetCDF output file.
The cat_thresh option does not apply to the fcst and obs dictionaries.
None
of the output stat line types produced by ensemble_stat (listed below)
are
based on the application of a categorical threshold defined by
cat_thresh.
Thanks,
John
output_flag = {
ecnt = NONE;
rps = NONE;
rhist = NONE;
phist = NONE;
orank = NONE;
ssvar = NONE;
relp = NONE;
}
On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: config files
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
>
> Hello,
>
>
> I have two ensemble config files as below (the other parts of the
config
> are the same), will they give exactly the same output?
>
>
> The only difference lies in there is no "cat_thresh" in fcst and obs
> entries. Based on my tests, it looks like the ensemble results are
the same
> with the two config files below, but I am afraid I may miss some
output
> flags.
>
>
> 1.
>
> fcst = {
>
> ens_thresh = 1.0;
>
> vld_thresh = 1.0;
>
>
> field = [
>
> {
>
> name = "VAFTD";
>
> level = [ "L0-21335" ];
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
>
> lead_time = "${LEAD_HR}";
>
> }
>
> ];
>
> };
>
>
> ens = {
>
> field = [
>
> {
>
> name = "VAFTD";
>
> level = [ "L0-21335" ];
>
> lead_time = "${LEAD_HR}";
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10
];
>
>
> }
>
> ];
>
> };
>
>
> obs = {
>
> field = [
>
> {
>
> name = "ash_mass_loading";
>
> level = ["(*,*)"];
>
> cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
>
> }
>
> ];
>
> ---------------------------------------------
>
> 2.
>
> fcst = {
>
> ens_thresh = 1.0;
>
> vld_thresh = 1.0;
>
>
> field = [
>
> {
>
> name = "VAFTD";
>
> level = [ "L0-21335" ];
>
> // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
>
> lead_time = "${LEAD_HR}";
>
> }
>
> ];
>
> };
>
>
> ens = {
>
> field = [
>
> {
>
> name = "VAFTD";
>
> level = [ "L0-21335" ];
>
> lead_time = "${LEAD_HR}";
>
> cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3, >=0.5,>=1,>=5,>=10
];
>
> }
>
> ];
>
> };
>
>
> obs = {
>
> field = [
>
> {
>
> name = "ash_mass_loading";
>
> level = ["(*,*)"];
>
> // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
>
> }
>
> ];
>
>
> So if they will lead to different outputs, what will be the
difference?
> Thank you.
>
> Binyu
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Mon May 24 16:50:59 2021
Thank you, John.
Looking at the config file, I had another question:
fcst = {
ens_thresh = 1.0;
vld_thresh = 1.0;
I understand the meaning of ens_thresh. I got very few matching
using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the count of
number
for ECNT are the same, the number of ensemble values are the same as
well,
they are all 31 (totally there are 31 ensemble members),
Here is the *stat file:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
Any thing wrong here? Thank you.
Binyu
On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> No that's correct. In ensemble_stat the cat_thresh setting applies
to the
> "ens" dictionary and defines the thresholds that to be used to in
defining
> probabilities written to the NetCDF output file.
>
> The cat_thresh option does not apply to the fcst and obs
dictionaries. None
> of the output stat line types produced by ensemble_stat (listed
below) are
> based on the application of a categorical threshold defined by
cat_thresh.
>
> Thanks,
> John
>
> output_flag = {
> ecnt = NONE;
> rps = NONE;
> rhist = NONE;
> phist = NONE;
> orank = NONE;
> ssvar = NONE;
> relp = NONE;
> }
>
> On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: config files
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> >
> > Hello,
> >
> >
> > I have two ensemble config files as below (the other parts of the
config
> > are the same), will they give exactly the same output?
> >
> >
> > The only difference lies in there is no "cat_thresh" in fcst and
obs
> > entries. Based on my tests, it looks like the ensemble results are
the
> same
> > with the two config files below, but I am afraid I may miss some
output
> > flags.
> >
> >
> > 1.
> >
> > fcst = {
> >
> > ens_thresh = 1.0;
> >
> > vld_thresh = 1.0;
> >
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > level = [ "L0-21335" ];
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > ens = {
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > level = [ "L0-21335" ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> >
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > obs = {
> >
> > field = [
> >
> > {
> >
> > name = "ash_mass_loading";
> >
> > level = ["(*,*)"];
> >
> > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> >
> > }
> >
> > ];
> >
> > ---------------------------------------------
> >
> > 2.
> >
> > fcst = {
> >
> > ens_thresh = 1.0;
> >
> > vld_thresh = 1.0;
> >
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > level = [ "L0-21335" ];
> >
> > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > ens = {
> >
> > field = [
> >
> > {
> >
> > name = "VAFTD";
> >
> > level = [ "L0-21335" ];
> >
> > lead_time = "${LEAD_HR}";
> >
> > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> >
> > }
> >
> > ];
> >
> > };
> >
> >
> > obs = {
> >
> > field = [
> >
> > {
> >
> > name = "ash_mass_loading";
> >
> > level = ["(*,*)"];
> >
> > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> >
> > }
> >
> > ];
> >
> >
> > So if they will lead to different outputs, what will be the
difference?
> > Thank you.
> >
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Tue May 25 08:57:34 2021
Binyu,
The ens_thresh and vld_thresh options tell ensemble_stat how to handle
missing data... ens_thresh for missing ensemble member inputs and
vld_thresh for missing data within the input ensemble member data.
They are
as described in the user's guide:
https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
----------------
When summarizing the ensemble, compute a ratio of the number of valid
ensemble fields to the total number of ensemble members. If this ratio
is
less than the *ens_thresh*, then quit with an error. This threshold
must be
between 0 and 1. Setting this threshold to 1 will require that all
ensemble
members be present to be processed.
When summarizing the ensemble, for each grid point compute a ratio of
the
number of valid data values to the number of ensemble members. If that
ratio is less than *vld_thresh*, write out bad data. This threshold
must be
between 0 and 1. Setting this threshold to 1 will require each grid
point
to contain valid data for all ensemble members.
----------------
If you have no missing ensemble members and none have any missing data
values, then changing these settings would have no impact on your
results.
John
On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> Thank you, John.
>
> Looking at the config file, I had another question:
>
> fcst = {
>
> ens_thresh = 1.0;
>
> vld_thresh = 1.0;
>
>
> I understand the meaning of ens_thresh. I got very few matching
> using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the count of
number
> for ECNT are the same, the number of ensemble values are the same as
well,
> they are all 31 (totally there are 31 ensemble members),
>
>
> Here is the *stat file:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
>
>
> Any thing wrong here? Thank you.
>
>
> Binyu
>
> On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > No that's correct. In ensemble_stat the cat_thresh setting applies
to the
> > "ens" dictionary and defines the thresholds that to be used to in
> defining
> > probabilities written to the NetCDF output file.
> >
> > The cat_thresh option does not apply to the fcst and obs
dictionaries.
> None
> > of the output stat line types produced by ensemble_stat (listed
below)
> are
> > based on the application of a categorical threshold defined by
> cat_thresh.
> >
> > Thanks,
> > John
> >
> > output_flag = {
> > ecnt = NONE;
> > rps = NONE;
> > rhist = NONE;
> > phist = NONE;
> > orank = NONE;
> > ssvar = NONE;
> > relp = NONE;
> > }
> >
> > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: config files
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> > >
> > >
> > > Hello,
> > >
> > >
> > > I have two ensemble config files as below (the other parts of
the
> config
> > > are the same), will they give exactly the same output?
> > >
> > >
> > > The only difference lies in there is no "cat_thresh" in fcst and
obs
> > > entries. Based on my tests, it looks like the ensemble results
are the
> > same
> > > with the two config files below, but I am afraid I may miss some
output
> > > flags.
> > >
> > >
> > > 1.
> > >
> > > fcst = {
> > >
> > > ens_thresh = 1.0;
> > >
> > > vld_thresh = 1.0;
> > >
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > level = [ "L0-21335" ];
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > ens = {
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > level = [ "L0-21335" ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> > >
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > obs = {
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "ash_mass_loading";
> > >
> > > level = ["(*,*)"];
> > >
> > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> > >
> > > }
> > >
> > > ];
> > >
> > > ---------------------------------------------
> > >
> > > 2.
> > >
> > > fcst = {
> > >
> > > ens_thresh = 1.0;
> > >
> > > vld_thresh = 1.0;
> > >
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > level = [ "L0-21335" ];
> > >
> > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > ens = {
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "VAFTD";
> > >
> > > level = [ "L0-21335" ];
> > >
> > > lead_time = "${LEAD_HR}";
> > >
> > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10 ];
> > >
> > > }
> > >
> > > ];
> > >
> > > };
> > >
> > >
> > > obs = {
> > >
> > > field = [
> > >
> > > {
> > >
> > > name = "ash_mass_loading";
> > >
> > > level = ["(*,*)"];
> > >
> > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > >
> > > }
> > >
> > > ];
> > >
> > >
> > > So if they will lead to different outputs, what will be the
difference?
> > > Thank you.
> > >
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Wed May 26 10:04:22 2021
Hello John,
I am confused about the "missing" ensemble members and data
definition. For example, there are 10 ensemble members overall.
ens_thresh=0.5, vld_thresh=0.5
Ensemble member missing means:
1. There is only input for 5 members, or
2. all 10 member files exist, but only 5 of them have overlap with
obs.
I guess 1 is correct?
The "missing" data refers to
3. all 10 members have data, but some data are not valid
4. all 10 forecasts have data, but no overlap?
Is 3 correct?
It seems "missing" does NOT mean "no overlap". If this is the case, is
there a way to get ensemble statistics if I don't need all ensemble
members
to overlap with obs? I.e: is it possible to get the RELY stat. by
setting more than 5 has overlap instead of 10?
I hope it makes sense. Thank you.
Binyu
On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> The ens_thresh and vld_thresh options tell ensemble_stat how to
handle
> missing data... ens_thresh for missing ensemble member inputs and
> vld_thresh for missing data within the input ensemble member data.
They are
> as described in the user's guide:
>
>
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
>
> ----------------
>
> When summarizing the ensemble, compute a ratio of the number of
valid
> ensemble fields to the total number of ensemble members. If this
ratio is
> less than the *ens_thresh*, then quit with an error. This threshold
must be
> between 0 and 1. Setting this threshold to 1 will require that all
ensemble
> members be present to be processed.
>
> When summarizing the ensemble, for each grid point compute a ratio
of the
> number of valid data values to the number of ensemble members. If
that
> ratio is less than *vld_thresh*, write out bad data. This threshold
must be
> between 0 and 1. Setting this threshold to 1 will require each grid
point
> to contain valid data for all ensemble members.
>
> ----------------
>
> If you have no missing ensemble members and none have any missing
data
> values, then changing these settings would have no impact on your
results.
>
> John
>
> On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > Thank you, John.
> >
> > Looking at the config file, I had another question:
> >
> > fcst = {
> >
> > ens_thresh = 1.0;
> >
> > vld_thresh = 1.0;
> >
> >
> > I understand the meaning of ens_thresh. I got very few matching
> > using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the count
of
> number
> > for ECNT are the same, the number of ensemble values are the same
as
> well,
> > they are all 31 (totally there are 31 ensemble members),
> >
> >
> > Here is the *stat file:
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> >
> >
> > Any thing wrong here? Thank you.
> >
> >
> > Binyu
> >
> > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > No that's correct. In ensemble_stat the cat_thresh setting
applies to
> the
> > > "ens" dictionary and defines the thresholds that to be used to
in
> > defining
> > > probabilities written to the NetCDF output file.
> > >
> > > The cat_thresh option does not apply to the fcst and obs
dictionaries.
> > None
> > > of the output stat line types produced by ensemble_stat (listed
below)
> > are
> > > based on the application of a categorical threshold defined by
> > cat_thresh.
> > >
> > > Thanks,
> > > John
> > >
> > > output_flag = {
> > > ecnt = NONE;
> > > rps = NONE;
> > > rhist = NONE;
> > > phist = NONE;
> > > orank = NONE;
> > > ssvar = NONE;
> > > relp = NONE;
> > > }
> > >
> > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: config files
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > >
> > > > I have two ensemble config files as below (the other parts of
the
> > config
> > > > are the same), will they give exactly the same output?
> > > >
> > > >
> > > > The only difference lies in there is no "cat_thresh" in fcst
and obs
> > > > entries. Based on my tests, it looks like the ensemble results
are
> the
> > > same
> > > > with the two config files below, but I am afraid I may miss
some
> output
> > > > flags.
> > > >
> > > >
> > > > 1.
> > > >
> > > > fcst = {
> > > >
> > > > ens_thresh = 1.0;
> > > >
> > > > vld_thresh = 1.0;
> > > >
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "VAFTD";
> > > >
> > > > level = [ "L0-21335" ];
> > > >
> > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > > >
> > > > lead_time = "${LEAD_HR}";
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > };
> > > >
> > > >
> > > > ens = {
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "VAFTD";
> > > >
> > > > level = [ "L0-21335" ];
> > > >
> > > > lead_time = "${LEAD_HR}";
> > > >
> > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> > > >
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > };
> > > >
> > > >
> > > > obs = {
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "ash_mass_loading";
> > > >
> > > > level = ["(*,*)"];
> > > >
> > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > ---------------------------------------------
> > > >
> > > > 2.
> > > >
> > > > fcst = {
> > > >
> > > > ens_thresh = 1.0;
> > > >
> > > > vld_thresh = 1.0;
> > > >
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "VAFTD";
> > > >
> > > > level = [ "L0-21335" ];
> > > >
> > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > >
> > > > lead_time = "${LEAD_HR}";
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > };
> > > >
> > > >
> > > > ens = {
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "VAFTD";
> > > >
> > > > level = [ "L0-21335" ];
> > > >
> > > > lead_time = "${LEAD_HR}";
> > > >
> > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> ];
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > > };
> > > >
> > > >
> > > > obs = {
> > > >
> > > > field = [
> > > >
> > > > {
> > > >
> > > > name = "ash_mass_loading";
> > > >
> > > > level = ["(*,*)"];
> > > >
> > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > >
> > > > }
> > > >
> > > > ];
> > > >
> > > >
> > > > So if they will lead to different outputs, what will be the
> difference?
> > > > Thank you.
> > > >
> > > > Binyu
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Wed May 26 10:24:05 2021
Binyu,
OK, so assuming you have 10 input ensemble members and ens_thresh =
0.5 and
vld_thresh = 0.5.
So you pass in 10 input files, 1 for each ensemble member. If 6 of
those
files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then ensemble_stat
will
error out and produce no output. So this is applied once and ensemble-
stat
either will run or error out.
One of the things that ensemble-stat does is compute an ensemble mean
field. At a particular grid point, if 6 of the 10 members do not have
a
valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the ensemble mean
value will be computed as bad data. This applies when deriving
ensemble
fields that are written to the NetCDF output file. So this is applied
once
for each grid point and affects the values written to the NetCDF
output.
So ens_thresh specifies the required ratio of ensemble members that
must be
present for ensemble_stat to run at all.
And vld_thresh specifies the required ratio of valid data for output
to be
created at each grid point.
I get the idea of what you mean by "overlap" with the obs, but the
details
matter a lot. When an ensemble member does NOT overlap with the obs,
is
that encoded as bad data or as some legitimate value, like 0?
If you have more questions, I'm guessing it'd be more informative to
look
at a specific example.
John
On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> Hello John,
>
> I am confused about the "missing" ensemble members and data
> definition. For example, there are 10 ensemble members overall.
> ens_thresh=0.5, vld_thresh=0.5
> Ensemble member missing means:
> 1. There is only input for 5 members, or
> 2. all 10 member files exist, but only 5 of them have overlap with
obs.
> I guess 1 is correct?
>
> The "missing" data refers to
> 3. all 10 members have data, but some data are not valid
> 4. all 10 forecasts have data, but no overlap?
> Is 3 correct?
>
> It seems "missing" does NOT mean "no overlap". If this is the case,
is
> there a way to get ensemble statistics if I don't need all ensemble
members
> to overlap with obs? I.e: is it possible to get the RELY stat. by
> setting more than 5 has overlap instead of 10?
>
> I hope it makes sense. Thank you.
> Binyu
>
> On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Binyu,
> >
> > The ens_thresh and vld_thresh options tell ensemble_stat how to
handle
> > missing data... ens_thresh for missing ensemble member inputs and
> > vld_thresh for missing data within the input ensemble member data.
They
> are
> > as described in the user's guide:
> >
> >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> >
> > ----------------
> >
> > When summarizing the ensemble, compute a ratio of the number of
valid
> > ensemble fields to the total number of ensemble members. If this
ratio is
> > less than the *ens_thresh*, then quit with an error. This
threshold must
> be
> > between 0 and 1. Setting this threshold to 1 will require that all
> ensemble
> > members be present to be processed.
> >
> > When summarizing the ensemble, for each grid point compute a ratio
of the
> > number of valid data values to the number of ensemble members. If
that
> > ratio is less than *vld_thresh*, write out bad data. This
threshold must
> be
> > between 0 and 1. Setting this threshold to 1 will require each
grid point
> > to contain valid data for all ensemble members.
> >
> > ----------------
> >
> > If you have no missing ensemble members and none have any missing
data
> > values, then changing these settings would have no impact on your
> results.
> >
> > John
> >
> > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > Thank you, John.
> > >
> > > Looking at the config file, I had another question:
> > >
> > > fcst = {
> > >
> > > ens_thresh = 1.0;
> > >
> > > vld_thresh = 1.0;
> > >
> > >
> > > I understand the meaning of ens_thresh. I got very few matching
> > > using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the
count of
> > number
> > > for ECNT are the same, the number of ensemble values are the
same as
> > well,
> > > they are all 31 (totally there are 31 ensemble members),
> > >
> > >
> > > Here is the *stat file:
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > >
> > >
> > > Any thing wrong here? Thank you.
> > >
> > >
> > > Binyu
> > >
> > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > No that's correct. In ensemble_stat the cat_thresh setting
applies to
> > the
> > > > "ens" dictionary and defines the thresholds that to be used to
in
> > > defining
> > > > probabilities written to the NetCDF output file.
> > > >
> > > > The cat_thresh option does not apply to the fcst and obs
> dictionaries.
> > > None
> > > > of the output stat line types produced by ensemble_stat
(listed
> below)
> > > are
> > > > based on the application of a categorical threshold defined by
> > > cat_thresh.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > output_flag = {
> > > > ecnt = NONE;
> > > > rps = NONE;
> > > > rhist = NONE;
> > > > phist = NONE;
> > > > orank = NONE;
> > > > ssvar = NONE;
> > > > relp = NONE;
> > > > }
> > > >
> > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: config files
> > > > > Owner: Nobody
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > >
> > > > > I have two ensemble config files as below (the other parts
of the
> > > config
> > > > > are the same), will they give exactly the same output?
> > > > >
> > > > >
> > > > > The only difference lies in there is no "cat_thresh" in fcst
and
> obs
> > > > > entries. Based on my tests, it looks like the ensemble
results are
> > the
> > > > same
> > > > > with the two config files below, but I am afraid I may miss
some
> > output
> > > > > flags.
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > fcst = {
> > > > >
> > > > > ens_thresh = 1.0;
> > > > >
> > > > > vld_thresh = 1.0;
> > > > >
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "VAFTD";
> > > > >
> > > > > level = [ "L0-21335" ];
> > > > >
> > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > > >
> > > > > lead_time = "${LEAD_HR}";
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > > };
> > > > >
> > > > >
> > > > > ens = {
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "VAFTD";
> > > > >
> > > > > level = [ "L0-21335" ];
> > > > >
> > > > > lead_time = "${LEAD_HR}";
> > > > >
> > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > > > >
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > > };
> > > > >
> > > > >
> > > > > obs = {
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "ash_mass_loading";
> > > > >
> > > > > level = ["(*,*)"];
> > > > >
> > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > > ---------------------------------------------
> > > > >
> > > > > 2.
> > > > >
> > > > > fcst = {
> > > > >
> > > > > ens_thresh = 1.0;
> > > > >
> > > > > vld_thresh = 1.0;
> > > > >
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "VAFTD";
> > > > >
> > > > > level = [ "L0-21335" ];
> > > > >
> > > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > >
> > > > > lead_time = "${LEAD_HR}";
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > > };
> > > > >
> > > > >
> > > > > ens = {
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "VAFTD";
> > > > >
> > > > > level = [ "L0-21335" ];
> > > > >
> > > > > lead_time = "${LEAD_HR}";
> > > > >
> > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
>=0.5,>=1,>=5,>=10
> > ];
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > > };
> > > > >
> > > > >
> > > > > obs = {
> > > > >
> > > > > field = [
> > > > >
> > > > > {
> > > > >
> > > > > name = "ash_mass_loading";
> > > > >
> > > > > level = ["(*,*)"];
> > > > >
> > > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > >
> > > > > }
> > > > >
> > > > > ];
> > > > >
> > > > >
> > > > > So if they will lead to different outputs, what will be the
> > difference?
> > > > > Thank you.
> > > > >
> > > > > Binyu
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Wed May 26 14:39:44 2021
John,
That is very clear. I really appreciate your detailed explanation. I
had
misunderstood "missing/invalid value" with "not overlap" before.
Here is what I am doing: I have 31 ensemble inputs, and I am using
satellite observed volcano ash for model verification. Based on the
current
results, there are only 3 matching pairs (which means only 3 grid
cells
over which all 31 members have ash overlap with obs). So I am
"thinking" if
there is a way to look at the performance when only part of the
members
(like 20 out of 31) has overlap? Maybe this thought is unrealistic.
Below
is my current config file:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
I added "NEP" and "NMEP" in the config file, this may help make the
verification look better, right?
Thank you.
Binyu
On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> OK, so assuming you have 10 input ensemble members and ens_thresh =
0.5 and
> vld_thresh = 0.5.
>
> So you pass in 10 input files, 1 for each ensemble member. If 6 of
those
> files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then ensemble_stat
will
> error out and produce no output. So this is applied once and
ensemble-stat
> either will run or error out.
>
> One of the things that ensemble-stat does is compute an ensemble
mean
> field. At a particular grid point, if 6 of the 10 members do not
have a
> valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the ensemble
mean
> value will be computed as bad data. This applies when deriving
ensemble
> fields that are written to the NetCDF output file. So this is
applied once
> for each grid point and affects the values written to the NetCDF
output.
>
> So ens_thresh specifies the required ratio of ensemble members that
must be
> present for ensemble_stat to run at all.
> And vld_thresh specifies the required ratio of valid data for output
to be
> created at each grid point.
>
> I get the idea of what you mean by "overlap" with the obs, but the
details
> matter a lot. When an ensemble member does NOT overlap with the obs,
is
> that encoded as bad data or as some legitimate value, like 0?
>
> If you have more questions, I'm guessing it'd be more informative to
look
> at a specific example.
>
> John
>
> On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > Hello John,
> >
> > I am confused about the "missing" ensemble members and data
> > definition. For example, there are 10 ensemble members overall.
> > ens_thresh=0.5, vld_thresh=0.5
> > Ensemble member missing means:
> > 1. There is only input for 5 members, or
> > 2. all 10 member files exist, but only 5 of them have overlap with
obs.
> > I guess 1 is correct?
> >
> > The "missing" data refers to
> > 3. all 10 members have data, but some data are not valid
> > 4. all 10 forecasts have data, but no overlap?
> > Is 3 correct?
> >
> > It seems "missing" does NOT mean "no overlap". If this is the
case, is
> > there a way to get ensemble statistics if I don't need all
ensemble
> members
> > to overlap with obs? I.e: is it possible to get the RELY stat. by
> > setting more than 5 has overlap instead of 10?
> >
> > I hope it makes sense. Thank you.
> > Binyu
> >
> > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Binyu,
> > >
> > > The ens_thresh and vld_thresh options tell ensemble_stat how to
handle
> > > missing data... ens_thresh for missing ensemble member inputs
and
> > > vld_thresh for missing data within the input ensemble member
data. They
> > are
> > > as described in the user's guide:
> > >
> > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > >
> > > ----------------
> > >
> > > When summarizing the ensemble, compute a ratio of the number of
valid
> > > ensemble fields to the total number of ensemble members. If this
ratio
> is
> > > less than the *ens_thresh*, then quit with an error. This
threshold
> must
> > be
> > > between 0 and 1. Setting this threshold to 1 will require that
all
> > ensemble
> > > members be present to be processed.
> > >
> > > When summarizing the ensemble, for each grid point compute a
ratio of
> the
> > > number of valid data values to the number of ensemble members.
If that
> > > ratio is less than *vld_thresh*, write out bad data. This
threshold
> must
> > be
> > > between 0 and 1. Setting this threshold to 1 will require each
grid
> point
> > > to contain valid data for all ensemble members.
> > >
> > > ----------------
> > >
> > > If you have no missing ensemble members and none have any
missing data
> > > values, then changing these settings would have no impact on
your
> > results.
> > >
> > > John
> > >
> > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > Thank you, John.
> > > >
> > > > Looking at the config file, I had another question:
> > > >
> > > > fcst = {
> > > >
> > > > ens_thresh = 1.0;
> > > >
> > > > vld_thresh = 1.0;
> > > >
> > > >
> > > > I understand the meaning of ens_thresh. I got very few
matching
> > > > using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the
count of
> > > number
> > > > for ECNT are the same, the number of ensemble values are the
same as
> > > well,
> > > > they are all 31 (totally there are 31 ensemble members),
> > > >
> > > >
> > > > Here is the *stat file:
> > > >
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > >
> > > >
> > > > Any thing wrong here? Thank you.
> > > >
> > > >
> > > > Binyu
> > > >
> > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > No that's correct. In ensemble_stat the cat_thresh setting
applies
> to
> > > the
> > > > > "ens" dictionary and defines the thresholds that to be used
to in
> > > > defining
> > > > > probabilities written to the NetCDF output file.
> > > > >
> > > > > The cat_thresh option does not apply to the fcst and obs
> > dictionaries.
> > > > None
> > > > > of the output stat line types produced by ensemble_stat
(listed
> > below)
> > > > are
> > > > > based on the application of a categorical threshold defined
by
> > > > cat_thresh.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > output_flag = {
> > > > > ecnt = NONE;
> > > > > rps = NONE;
> > > > > rhist = NONE;
> > > > > phist = NONE;
> > > > > orank = NONE;
> > > > > ssvar = NONE;
> > > > > relp = NONE;
> > > > > }
> > > > >
> > > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: config files
> > > > > > Owner: Nobody
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > >
> > > > > > I have two ensemble config files as below (the other
parts of
> the
> > > > config
> > > > > > are the same), will they give exactly the same output?
> > > > > >
> > > > > >
> > > > > > The only difference lies in there is no "cat_thresh" in
fcst and
> > obs
> > > > > > entries. Based on my tests, it looks like the ensemble
results
> are
> > > the
> > > > > same
> > > > > > with the two config files below, but I am afraid I may
miss some
> > > output
> > > > > > flags.
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > fcst = {
> > > > > >
> > > > > > ens_thresh = 1.0;
> > > > > >
> > > > > > vld_thresh = 1.0;
> > > > > >
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "VAFTD";
> > > > > >
> > > > > > level = [ "L0-21335" ];
> > > > > >
> > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > > >
> > > > > > lead_time = "${LEAD_HR}";
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > > };
> > > > > >
> > > > > >
> > > > > > ens = {
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "VAFTD";
> > > > > >
> > > > > > level = [ "L0-21335" ];
> > > > > >
> > > > > > lead_time = "${LEAD_HR}";
> > > > > >
> > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > > > >
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > > };
> > > > > >
> > > > > >
> > > > > > obs = {
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "ash_mass_loading";
> > > > > >
> > > > > > level = ["(*,*)"];
> > > > > >
> > > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > > ---------------------------------------------
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > fcst = {
> > > > > >
> > > > > > ens_thresh = 1.0;
> > > > > >
> > > > > > vld_thresh = 1.0;
> > > > > >
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "VAFTD";
> > > > > >
> > > > > > level = [ "L0-21335" ];
> > > > > >
> > > > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > >
> > > > > > lead_time = "${LEAD_HR}";
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > > };
> > > > > >
> > > > > >
> > > > > > ens = {
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "VAFTD";
> > > > > >
> > > > > > level = [ "L0-21335" ];
> > > > > >
> > > > > > lead_time = "${LEAD_HR}";
> > > > > >
> > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> >=0.5,>=1,>=5,>=10
> > > ];
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > > };
> > > > > >
> > > > > >
> > > > > > obs = {
> > > > > >
> > > > > > field = [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > name = "ash_mass_loading";
> > > > > >
> > > > > > level = ["(*,*)"];
> > > > > >
> > > > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ];
> > > > > >
> > > > > >
> > > > > > So if they will lead to different outputs, what will be
the
> > > difference?
> > > > > > Thank you.
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Wed May 26 15:27:21 2021
Binyu,
When the ensemble member does not predict the ash at a grid point, is
that
stored as a missing data value or some real data value, like 0?
If it's a bad data value instead of 0, that'll cause problems.
Ensemble-Stat only computes an observation rank value when all of the
ensemble members contain valid data at that point.
You only getting 3 pairs in the output may mean that there are only 3
grid
points at which all 31 ensemble member contains valid data at those
points.
If it is bad data, you could switch it to 0, using the censor_thresh
and
censor_val options:
censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
John
On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> John,
>
> That is very clear. I really appreciate your detailed explanation. I
had
> misunderstood "missing/invalid value" with "not overlap" before.
>
> Here is what I am doing: I have 31 ensemble inputs, and I am using
> satellite observed volcano ash for model verification. Based on the
current
> results, there are only 3 matching pairs (which means only 3 grid
cells
> over which all 31 members have ash overlap with obs). So I am
"thinking" if
> there is a way to look at the performance when only part of the
members
> (like 20 out of 31) has overlap? Maybe this thought is unrealistic.
Below
> is my current config file:
>
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
>
>
> I added "NEP" and "NMEP" in the config file, this may help make the
> verification look better, right?
>
>
> Thank you.
>
> Binyu
>
> On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Binyu,
> >
> > OK, so assuming you have 10 input ensemble members and ens_thresh
= 0.5
> and
> > vld_thresh = 0.5.
> >
> > So you pass in 10 input files, 1 for each ensemble member. If 6 of
those
> > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
ensemble_stat will
> > error out and produce no output. So this is applied once and
> ensemble-stat
> > either will run or error out.
> >
> > One of the things that ensemble-stat does is compute an ensemble
mean
> > field. At a particular grid point, if 6 of the 10 members do not
have a
> > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the ensemble
mean
> > value will be computed as bad data. This applies when deriving
ensemble
> > fields that are written to the NetCDF output file. So this is
applied
> once
> > for each grid point and affects the values written to the NetCDF
output.
> >
> > So ens_thresh specifies the required ratio of ensemble members
that must
> be
> > present for ensemble_stat to run at all.
> > And vld_thresh specifies the required ratio of valid data for
output to
> be
> > created at each grid point.
> >
> > I get the idea of what you mean by "overlap" with the obs, but the
> details
> > matter a lot. When an ensemble member does NOT overlap with the
obs, is
> > that encoded as bad data or as some legitimate value, like 0?
> >
> > If you have more questions, I'm guessing it'd be more informative
to look
> > at a specific example.
> >
> > John
> >
> > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > Hello John,
> > >
> > > I am confused about the "missing" ensemble members and data
> > > definition. For example, there are 10 ensemble members
overall.
> > > ens_thresh=0.5, vld_thresh=0.5
> > > Ensemble member missing means:
> > > 1. There is only input for 5 members, or
> > > 2. all 10 member files exist, but only 5 of them have overlap
with obs.
> > > I guess 1 is correct?
> > >
> > > The "missing" data refers to
> > > 3. all 10 members have data, but some data are not valid
> > > 4. all 10 forecasts have data, but no overlap?
> > > Is 3 correct?
> > >
> > > It seems "missing" does NOT mean "no overlap". If this is the
case, is
> > > there a way to get ensemble statistics if I don't need all
ensemble
> > members
> > > to overlap with obs? I.e: is it possible to get the RELY stat.
by
> > > setting more than 5 has overlap instead of 10?
> > >
> > > I hope it makes sense. Thank you.
> > > Binyu
> > >
> > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Binyu,
> > > >
> > > > The ens_thresh and vld_thresh options tell ensemble_stat how
to
> handle
> > > > missing data... ens_thresh for missing ensemble member inputs
and
> > > > vld_thresh for missing data within the input ensemble member
data.
> They
> > > are
> > > > as described in the user's guide:
> > > >
> > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > >
> > > > ----------------
> > > >
> > > > When summarizing the ensemble, compute a ratio of the number
of valid
> > > > ensemble fields to the total number of ensemble members. If
this
> ratio
> > is
> > > > less than the *ens_thresh*, then quit with an error. This
threshold
> > must
> > > be
> > > > between 0 and 1. Setting this threshold to 1 will require that
all
> > > ensemble
> > > > members be present to be processed.
> > > >
> > > > When summarizing the ensemble, for each grid point compute a
ratio of
> > the
> > > > number of valid data values to the number of ensemble members.
If
> that
> > > > ratio is less than *vld_thresh*, write out bad data. This
threshold
> > must
> > > be
> > > > between 0 and 1. Setting this threshold to 1 will require each
grid
> > point
> > > > to contain valid data for all ensemble members.
> > > >
> > > > ----------------
> > > >
> > > > If you have no missing ensemble members and none have any
missing
> data
> > > > values, then changing these settings would have no impact on
your
> > > results.
> > > >
> > > > John
> > > >
> > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > >
> > > > > Thank you, John.
> > > > >
> > > > > Looking at the config file, I had another question:
> > > > >
> > > > > fcst = {
> > > > >
> > > > > ens_thresh = 1.0;
> > > > >
> > > > > vld_thresh = 1.0;
> > > > >
> > > > >
> > > > > I understand the meaning of ens_thresh. I got very few
matching
> > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5, however, the
count
> of
> > > > number
> > > > > for ECNT are the same, the number of ensemble values are the
same
> as
> > > > well,
> > > > > they are all 31 (totally there are 31 ensemble members),
> > > > >
> > > > >
> > > > > Here is the *stat file:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > >
> > > > >
> > > > > Any thing wrong here? Thank you.
> > > > >
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > No that's correct. In ensemble_stat the cat_thresh setting
> applies
> > to
> > > > the
> > > > > > "ens" dictionary and defines the thresholds that to be
used to in
> > > > > defining
> > > > > > probabilities written to the NetCDF output file.
> > > > > >
> > > > > > The cat_thresh option does not apply to the fcst and obs
> > > dictionaries.
> > > > > None
> > > > > > of the output stat line types produced by ensemble_stat
(listed
> > > below)
> > > > > are
> > > > > > based on the application of a categorical threshold
defined by
> > > > > cat_thresh.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > output_flag = {
> > > > > > ecnt = NONE;
> > > > > > rps = NONE;
> > > > > > rhist = NONE;
> > > > > > phist = NONE;
> > > > > > orank = NONE;
> > > > > > ssvar = NONE;
> > > > > > relp = NONE;
> > > > > > }
> > > > > >
> > > > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Mon May 24 15:55:23 2021: Request 99973 was acted upon.
> > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: config files
> > > > > > > Owner: Nobody
> > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > >
> > > > > > > I have two ensemble config files as below (the other
parts of
> > the
> > > > > config
> > > > > > > are the same), will they give exactly the same output?
> > > > > > >
> > > > > > >
> > > > > > > The only difference lies in there is no "cat_thresh" in
fcst
> and
> > > obs
> > > > > > > entries. Based on my tests, it looks like the ensemble
results
> > are
> > > > the
> > > > > > same
> > > > > > > with the two config files below, but I am afraid I may
miss
> some
> > > > output
> > > > > > > flags.
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > fcst = {
> > > > > > >
> > > > > > > ens_thresh = 1.0;
> > > > > > >
> > > > > > > vld_thresh = 1.0;
> > > > > > >
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "VAFTD";
> > > > > > >
> > > > > > > level = [ "L0-21335" ];
> > > > > > >
> > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > > >
> > > > > > > lead_time = "${LEAD_HR}";
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > > ens = {
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "VAFTD";
> > > > > > >
> > > > > > > level = [ "L0-21335" ];
> > > > > > >
> > > > > > > lead_time = "${LEAD_HR}";
> > > > > > >
> > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > > > >
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > > obs = {
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "ash_mass_loading";
> > > > > > >
> > > > > > > level = ["(*,*)"];
> > > > > > >
> > > > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > > ---------------------------------------------
> > > > > > >
> > > > > > > 2.
> > > > > > >
> > > > > > > fcst = {
> > > > > > >
> > > > > > > ens_thresh = 1.0;
> > > > > > >
> > > > > > > vld_thresh = 1.0;
> > > > > > >
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "VAFTD";
> > > > > > >
> > > > > > > level = [ "L0-21335" ];
> > > > > > >
> > > > > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > >
> > > > > > > lead_time = "${LEAD_HR}";
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > > ens = {
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "VAFTD";
> > > > > > >
> > > > > > > level = [ "L0-21335" ];
> > > > > > >
> > > > > > > lead_time = "${LEAD_HR}";
> > > > > > >
> > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > >=0.5,>=1,>=5,>=10
> > > > ];
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > > obs = {
> > > > > > >
> > > > > > > field = [
> > > > > > >
> > > > > > > {
> > > > > > >
> > > > > > > name = "ash_mass_loading";
> > > > > > >
> > > > > > > level = ["(*,*)"];
> > > > > > >
> > > > > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > ];
> > > > > > >
> > > > > > >
> > > > > > > So if they will lead to different outputs, what will be
the
> > > > difference?
> > > > > > > Thank you.
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Wed May 26 20:23:06 2021
John,
Actually the non-signal grid has "-99" in my ensemble member (the
original
data was converted using LogX), and it needs to be anti-log in order
to
compare with observation, so I use "convert(x) =(10^(x))" in the
config
file. I guess the converted value (10^(-99)=1E(-99)) is treated as
valid
data, this is why there is no difference in my results when I change
"vld_thresh=1" to "vld_thresh=0.5", because there is no invalid data
in all
the members.
If my guess above is correct, is there a way to look at the ensemble
performance when only part of the members (like 20 out of 31) has
overlap?
Binyu
On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> When the ensemble member does not predict the ash at a grid point,
is that
> stored as a missing data value or some real data value, like 0?
>
> If it's a bad data value instead of 0, that'll cause problems.
> Ensemble-Stat only computes an observation rank value when all of
the
> ensemble members contain valid data at that point.
>
> You only getting 3 pairs in the output may mean that there are only
3 grid
> points at which all 31 ensemble member contains valid data at those
points.
>
> If it is bad data, you could switch it to 0, using the censor_thresh
and
> censor_val options:
> censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
>
> John
>
>
>
> On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > John,
> >
> > That is very clear. I really appreciate your detailed explanation.
I had
> > misunderstood "missing/invalid value" with "not overlap" before.
> >
> > Here is what I am doing: I have 31 ensemble inputs, and I am using
> > satellite observed volcano ash for model verification. Based on
the
> current
> > results, there are only 3 matching pairs (which means only 3 grid
cells
> > over which all 31 members have ash overlap with obs). So I am
"thinking"
> if
> > there is a way to look at the performance when only part of the
members
> > (like 20 out of 31) has overlap? Maybe this thought is
unrealistic. Below
> > is my current config file:
> >
> >
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> >
> >
> > I added "NEP" and "NMEP" in the config file, this may help make
the
> > verification look better, right?
> >
> >
> > Thank you.
> >
> > Binyu
> >
> > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Binyu,
> > >
> > > OK, so assuming you have 10 input ensemble members and
ens_thresh = 0.5
> > and
> > > vld_thresh = 0.5.
> > >
> > > So you pass in 10 input files, 1 for each ensemble member. If 6
of
> those
> > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
ensemble_stat
> will
> > > error out and produce no output. So this is applied once and
> > ensemble-stat
> > > either will run or error out.
> > >
> > > One of the things that ensemble-stat does is compute an ensemble
mean
> > > field. At a particular grid point, if 6 of the 10 members do not
have a
> > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the
ensemble mean
> > > value will be computed as bad data. This applies when deriving
ensemble
> > > fields that are written to the NetCDF output file. So this is
applied
> > once
> > > for each grid point and affects the values written to the NetCDF
> output.
> > >
> > > So ens_thresh specifies the required ratio of ensemble members
that
> must
> > be
> > > present for ensemble_stat to run at all.
> > > And vld_thresh specifies the required ratio of valid data for
output to
> > be
> > > created at each grid point.
> > >
> > > I get the idea of what you mean by "overlap" with the obs, but
the
> > details
> > > matter a lot. When an ensemble member does NOT overlap with the
obs, is
> > > that encoded as bad data or as some legitimate value, like 0?
> > >
> > > If you have more questions, I'm guessing it'd be more
informative to
> look
> > > at a specific example.
> > >
> > > John
> > >
> > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > Hello John,
> > > >
> > > > I am confused about the "missing" ensemble members and data
> > > > definition. For example, there are 10 ensemble members
overall.
> > > > ens_thresh=0.5, vld_thresh=0.5
> > > > Ensemble member missing means:
> > > > 1. There is only input for 5 members, or
> > > > 2. all 10 member files exist, but only 5 of them have overlap
with
> obs.
> > > > I guess 1 is correct?
> > > >
> > > > The "missing" data refers to
> > > > 3. all 10 members have data, but some data are not valid
> > > > 4. all 10 forecasts have data, but no overlap?
> > > > Is 3 correct?
> > > >
> > > > It seems "missing" does NOT mean "no overlap". If this is the
case,
> is
> > > > there a way to get ensemble statistics if I don't need all
ensemble
> > > members
> > > > to overlap with obs? I.e: is it possible to get the RELY
stat. by
> > > > setting more than 5 has overlap instead of 10?
> > > >
> > > > I hope it makes sense. Thank you.
> > > > Binyu
> > > >
> > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > The ens_thresh and vld_thresh options tell ensemble_stat how
to
> > handle
> > > > > missing data... ens_thresh for missing ensemble member
inputs and
> > > > > vld_thresh for missing data within the input ensemble member
data.
> > They
> > > > are
> > > > > as described in the user's guide:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > >
> > > > > ----------------
> > > > >
> > > > > When summarizing the ensemble, compute a ratio of the number
of
> valid
> > > > > ensemble fields to the total number of ensemble members. If
this
> > ratio
> > > is
> > > > > less than the *ens_thresh*, then quit with an error. This
threshold
> > > must
> > > > be
> > > > > between 0 and 1. Setting this threshold to 1 will require
that all
> > > > ensemble
> > > > > members be present to be processed.
> > > > >
> > > > > When summarizing the ensemble, for each grid point compute a
ratio
> of
> > > the
> > > > > number of valid data values to the number of ensemble
members. If
> > that
> > > > > ratio is less than *vld_thresh*, write out bad data. This
threshold
> > > must
> > > > be
> > > > > between 0 and 1. Setting this threshold to 1 will require
each grid
> > > point
> > > > > to contain valid data for all ensemble members.
> > > > >
> > > > > ----------------
> > > > >
> > > > > If you have no missing ensemble members and none have any
missing
> > data
> > > > > values, then changing these settings would have no impact on
your
> > > > results.
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > > >
> > > > > > Thank you, John.
> > > > > >
> > > > > > Looking at the config file, I had another question:
> > > > > >
> > > > > > fcst = {
> > > > > >
> > > > > > ens_thresh = 1.0;
> > > > > >
> > > > > > vld_thresh = 1.0;
> > > > > >
> > > > > >
> > > > > > I understand the meaning of ens_thresh. I got very few
matching
> > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5, however,
the count
> > of
> > > > > number
> > > > > > for ECNT are the same, the number of ensemble values are
the same
> > as
> > > > > well,
> > > > > > they are all 31 (totally there are 31 ensemble members),
> > > > > >
> > > > > >
> > > > > > Here is the *stat file:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > >
> > > > > >
> > > > > > Any thing wrong here? Thank you.
> > > > > >
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > No that's correct. In ensemble_stat the cat_thresh
setting
> > applies
> > > to
> > > > > the
> > > > > > > "ens" dictionary and defines the thresholds that to be
used to
> in
> > > > > > defining
> > > > > > > probabilities written to the NetCDF output file.
> > > > > > >
> > > > > > > The cat_thresh option does not apply to the fcst and obs
> > > > dictionaries.
> > > > > > None
> > > > > > > of the output stat line types produced by ensemble_stat
(listed
> > > > below)
> > > > > > are
> > > > > > > based on the application of a categorical threshold
defined by
> > > > > > cat_thresh.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > output_flag = {
> > > > > > > ecnt = NONE;
> > > > > > > rps = NONE;
> > > > > > > rhist = NONE;
> > > > > > > phist = NONE;
> > > > > > > orank = NONE;
> > > > > > > ssvar = NONE;
> > > > > > > relp = NONE;
> > > > > > > }
> > > > > > >
> > > > > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was acted
upon.
> > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: config files
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > >
> > > > > > > > I have two ensemble config files as below (the other
parts
> of
> > > the
> > > > > > config
> > > > > > > > are the same), will they give exactly the same output?
> > > > > > > >
> > > > > > > >
> > > > > > > > The only difference lies in there is no "cat_thresh"
in fcst
> > and
> > > > obs
> > > > > > > > entries. Based on my tests, it looks like the ensemble
> results
> > > are
> > > > > the
> > > > > > > same
> > > > > > > > with the two config files below, but I am afraid I may
miss
> > some
> > > > > output
> > > > > > > > flags.
> > > > > > > >
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > fcst = {
> > > > > > > >
> > > > > > > > ens_thresh = 1.0;
> > > > > > > >
> > > > > > > > vld_thresh = 1.0;
> > > > > > > >
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "VAFTD";
> > > > > > > >
> > > > > > > > level = [ "L0-21335" ];
> > > > > > > >
> > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > > >
> > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > > };
> > > > > > > >
> > > > > > > >
> > > > > > > > ens = {
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "VAFTD";
> > > > > > > >
> > > > > > > > level = [ "L0-21335" ];
> > > > > > > >
> > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > >
> > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > > > >
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > > };
> > > > > > > >
> > > > > > > >
> > > > > > > > obs = {
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "ash_mass_loading";
> > > > > > > >
> > > > > > > > level = ["(*,*)"];
> > > > > > > >
> > > > > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > > ---------------------------------------------
> > > > > > > >
> > > > > > > > 2.
> > > > > > > >
> > > > > > > > fcst = {
> > > > > > > >
> > > > > > > > ens_thresh = 1.0;
> > > > > > > >
> > > > > > > > vld_thresh = 1.0;
> > > > > > > >
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "VAFTD";
> > > > > > > >
> > > > > > > > level = [ "L0-21335" ];
> > > > > > > >
> > > > > > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > >
> > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > > };
> > > > > > > >
> > > > > > > >
> > > > > > > > ens = {
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "VAFTD";
> > > > > > > >
> > > > > > > > level = [ "L0-21335" ];
> > > > > > > >
> > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > >
> > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > >=0.5,>=1,>=5,>=10
> > > > > ];
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > > };
> > > > > > > >
> > > > > > > >
> > > > > > > > obs = {
> > > > > > > >
> > > > > > > > field = [
> > > > > > > >
> > > > > > > > {
> > > > > > > >
> > > > > > > > name = "ash_mass_loading";
> > > > > > > >
> > > > > > > > level = ["(*,*)"];
> > > > > > > >
> > > > > > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > >
> > > > > > > > }
> > > > > > > >
> > > > > > > > ];
> > > > > > > >
> > > > > > > >
> > > > > > > > So if they will lead to different outputs, what will
be the
> > > > > difference?
> > > > > > > > Thank you.
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Thu May 27 16:24:01 2021
Binyu,
So you're saying "ensemble performance" but I need you to be more
explicit
than that. Do you mean, for example, that that point needs to be
included
in the ranked histogram (RHIST) output line type?
Remember that vld_thresh applies in the computation of the ensemble
mean
and probabilities written to the NetCDF output file. It does not apply
to
the computation of ranked histograms.
John
On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> John,
>
> Actually the non-signal grid has "-99" in my ensemble member (the
original
> data was converted using LogX), and it needs to be anti-log in order
to
> compare with observation, so I use "convert(x) =(10^(x))" in the
config
> file. I guess the converted value (10^(-99)=1E(-99)) is treated as
valid
> data, this is why there is no difference in my results when I change
> "vld_thresh=1" to "vld_thresh=0.5", because there is no invalid data
in all
> the members.
>
> If my guess above is correct, is there a way to look at the
ensemble
> performance when only part of the members (like 20 out of 31) has
overlap?
>
> Binyu
>
> On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > When the ensemble member does not predict the ash at a grid point,
is
> that
> > stored as a missing data value or some real data value, like 0?
> >
> > If it's a bad data value instead of 0, that'll cause problems.
> > Ensemble-Stat only computes an observation rank value when all of
the
> > ensemble members contain valid data at that point.
> >
> > You only getting 3 pairs in the output may mean that there are
only 3
> grid
> > points at which all 31 ensemble member contains valid data at
those
> points.
> >
> > If it is bad data, you could switch it to 0, using the
censor_thresh and
> > censor_val options:
> > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> >
> > John
> >
> >
> >
> > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > John,
> > >
> > > That is very clear. I really appreciate your detailed
explanation. I
> had
> > > misunderstood "missing/invalid value" with "not overlap" before.
> > >
> > > Here is what I am doing: I have 31 ensemble inputs, and I am
using
> > > satellite observed volcano ash for model verification. Based on
the
> > current
> > > results, there are only 3 matching pairs (which means only 3
grid cells
> > > over which all 31 members have ash overlap with obs). So I am
> "thinking"
> > if
> > > there is a way to look at the performance when only part of the
members
> > > (like 20 out of 31) has overlap? Maybe this thought is
unrealistic.
> Below
> > > is my current config file:
> > >
> > >
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > >
> > >
> > > I added "NEP" and "NMEP" in the config file, this may help make
the
> > > verification look better, right?
> > >
> > >
> > > Thank you.
> > >
> > > Binyu
> > >
> > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Binyu,
> > > >
> > > > OK, so assuming you have 10 input ensemble members and
ens_thresh =
> 0.5
> > > and
> > > > vld_thresh = 0.5.
> > > >
> > > > So you pass in 10 input files, 1 for each ensemble member. If
6 of
> > those
> > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
ensemble_stat
> > will
> > > > error out and produce no output. So this is applied once and
> > > ensemble-stat
> > > > either will run or error out.
> > > >
> > > > One of the things that ensemble-stat does is compute an
ensemble mean
> > > > field. At a particular grid point, if 6 of the 10 members do
not
> have a
> > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the
ensemble
> mean
> > > > value will be computed as bad data. This applies when deriving
> ensemble
> > > > fields that are written to the NetCDF output file. So this is
applied
> > > once
> > > > for each grid point and affects the values written to the
NetCDF
> > output.
> > > >
> > > > So ens_thresh specifies the required ratio of ensemble members
that
> > must
> > > be
> > > > present for ensemble_stat to run at all.
> > > > And vld_thresh specifies the required ratio of valid data for
output
> to
> > > be
> > > > created at each grid point.
> > > >
> > > > I get the idea of what you mean by "overlap" with the obs, but
the
> > > details
> > > > matter a lot. When an ensemble member does NOT overlap with
the obs,
> is
> > > > that encoded as bad data or as some legitimate value, like 0?
> > > >
> > > > If you have more questions, I'm guessing it'd be more
informative to
> > look
> > > > at a specific example.
> > > >
> > > > John
> > > >
> > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > >
> > > > > Hello John,
> > > > >
> > > > > I am confused about the "missing" ensemble members and data
> > > > > definition. For example, there are 10 ensemble members
overall.
> > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > Ensemble member missing means:
> > > > > 1. There is only input for 5 members, or
> > > > > 2. all 10 member files exist, but only 5 of them have
overlap with
> > obs.
> > > > > I guess 1 is correct?
> > > > >
> > > > > The "missing" data refers to
> > > > > 3. all 10 members have data, but some data are not valid
> > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > Is 3 correct?
> > > > >
> > > > > It seems "missing" does NOT mean "no overlap". If this is
the case,
> > is
> > > > > there a way to get ensemble statistics if I don't need all
ensemble
> > > > members
> > > > > to overlap with obs? I.e: is it possible to get the RELY
stat. by
> > > > > setting more than 5 has overlap instead of 10?
> > > > >
> > > > > I hope it makes sense. Thank you.
> > > > > Binyu
> > > > >
> > > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > The ens_thresh and vld_thresh options tell ensemble_stat
how to
> > > handle
> > > > > > missing data... ens_thresh for missing ensemble member
inputs and
> > > > > > vld_thresh for missing data within the input ensemble
member
> data.
> > > They
> > > > > are
> > > > > > as described in the user's guide:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > >
> > > > > > ----------------
> > > > > >
> > > > > > When summarizing the ensemble, compute a ratio of the
number of
> > valid
> > > > > > ensemble fields to the total number of ensemble members.
If this
> > > ratio
> > > > is
> > > > > > less than the *ens_thresh*, then quit with an error. This
> threshold
> > > > must
> > > > > be
> > > > > > between 0 and 1. Setting this threshold to 1 will require
that
> all
> > > > > ensemble
> > > > > > members be present to be processed.
> > > > > >
> > > > > > When summarizing the ensemble, for each grid point compute
a
> ratio
> > of
> > > > the
> > > > > > number of valid data values to the number of ensemble
members. If
> > > that
> > > > > > ratio is less than *vld_thresh*, write out bad data. This
> threshold
> > > > must
> > > > > be
> > > > > > between 0 and 1. Setting this threshold to 1 will require
each
> grid
> > > > point
> > > > > > to contain valid data for all ensemble members.
> > > > > >
> > > > > > ----------------
> > > > > >
> > > > > > If you have no missing ensemble members and none have any
missing
> > > data
> > > > > > values, then changing these settings would have no impact
on your
> > > > > results.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> > > > > > >
> > > > > > > Thank you, John.
> > > > > > >
> > > > > > > Looking at the config file, I had another question:
> > > > > > >
> > > > > > > fcst = {
> > > > > > >
> > > > > > > ens_thresh = 1.0;
> > > > > > >
> > > > > > > vld_thresh = 1.0;
> > > > > > >
> > > > > > >
> > > > > > > I understand the meaning of ens_thresh. I got very few
matching
> > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5, however,
the
> count
> > > of
> > > > > > number
> > > > > > > for ECNT are the same, the number of ensemble values are
the
> same
> > > as
> > > > > > well,
> > > > > > > they are all 31 (totally there are 31 ensemble members),
> > > > > > >
> > > > > > >
> > > > > > > Here is the *stat file:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > >
> > > > > > >
> > > > > > > Any thing wrong here? Thank you.
> > > > > > >
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Binyu,
> > > > > > > >
> > > > > > > > No that's correct. In ensemble_stat the cat_thresh
setting
> > > applies
> > > > to
> > > > > > the
> > > > > > > > "ens" dictionary and defines the thresholds that to be
used
> to
> > in
> > > > > > > defining
> > > > > > > > probabilities written to the NetCDF output file.
> > > > > > > >
> > > > > > > > The cat_thresh option does not apply to the fcst and
obs
> > > > > dictionaries.
> > > > > > > None
> > > > > > > > of the output stat line types produced by
ensemble_stat
> (listed
> > > > > below)
> > > > > > > are
> > > > > > > > based on the application of a categorical threshold
defined
> by
> > > > > > > cat_thresh.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > output_flag = {
> > > > > > > > ecnt = NONE;
> > > > > > > > rps = NONE;
> > > > > > > > rhist = NONE;
> > > > > > > > phist = NONE;
> > > > > > > > orank = NONE;
> > > > > > > > ssvar = NONE;
> > > > > > > > relp = NONE;
> > > > > > > > }
> > > > > > > >
> > > > > > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was acted
upon.
> > > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: config files
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I have two ensemble config files as below (the
other parts
> > of
> > > > the
> > > > > > > config
> > > > > > > > > are the same), will they give exactly the same
output?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The only difference lies in there is no "cat_thresh"
in
> fcst
> > > and
> > > > > obs
> > > > > > > > > entries. Based on my tests, it looks like the
ensemble
> > results
> > > > are
> > > > > > the
> > > > > > > > same
> > > > > > > > > with the two config files below, but I am afraid I
may miss
> > > some
> > > > > > output
> > > > > > > > > flags.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > fcst = {
> > > > > > > > >
> > > > > > > > > ens_thresh = 1.0;
> > > > > > > > >
> > > > > > > > > vld_thresh = 1.0;
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "VAFTD";
> > > > > > > > >
> > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > >
> > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > > >
> > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ens = {
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "VAFTD";
> > > > > > > > >
> > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > >
> > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > >
> > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > obs = {
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "ash_mass_loading";
> > > > > > > > >
> > > > > > > > > level = ["(*,*)"];
> > > > > > > > >
> > > > > > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > ---------------------------------------------
> > > > > > > > >
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > > fcst = {
> > > > > > > > >
> > > > > > > > > ens_thresh = 1.0;
> > > > > > > > >
> > > > > > > > > vld_thresh = 1.0;
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "VAFTD";
> > > > > > > > >
> > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > >
> > > > > > > > > // cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > >
> > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ens = {
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "VAFTD";
> > > > > > > > >
> > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > >
> > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > >
> > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > >=0.5,>=1,>=5,>=10
> > > > > > ];
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > obs = {
> > > > > > > > >
> > > > > > > > > field = [
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > >
> > > > > > > > > name = "ash_mass_loading";
> > > > > > > > >
> > > > > > > > > level = ["(*,*)"];
> > > > > > > > >
> > > > > > > > > // cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > >
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > So if they will lead to different outputs, what will
be the
> > > > > > difference?
> > > > > > > > > Thank you.
> > > > > > > > >
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Fri May 28 08:51:09 2021
John,
Currently, "vld_thresh" only sets thresh to INVALID data, is there a
way
to compute the ensemble mean and probabilities using only part of
ensemble
members?
e.g:
16 out of 31 have overlap with obs, the other 15 members are valid but
no
overlap.
Does that sound right?
Thank you.
Binyu
On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> So you're saying "ensemble performance" but I need you to be more
explicit
> than that. Do you mean, for example, that that point needs to be
included
> in the ranked histogram (RHIST) output line type?
>
> Remember that vld_thresh applies in the computation of the ensemble
mean
> and probabilities written to the NetCDF output file. It does not
apply to
> the computation of ranked histograms.
>
> John
>
> On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > John,
> >
> > Actually the non-signal grid has "-99" in my ensemble member (the
> original
> > data was converted using LogX), and it needs to be anti-log in
order to
> > compare with observation, so I use "convert(x) =(10^(x))" in the
config
> > file. I guess the converted value (10^(-99)=1E(-99)) is treated
as valid
> > data, this is why there is no difference in my results when I
change
> > "vld_thresh=1" to "vld_thresh=0.5", because there is no invalid
data in
> all
> > the members.
> >
> > If my guess above is correct, is there a way to look at the
ensemble
> > performance when only part of the members (like 20 out of 31) has
> overlap?
> >
> > Binyu
> >
> > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > When the ensemble member does not predict the ash at a grid
point, is
> > that
> > > stored as a missing data value or some real data value, like 0?
> > >
> > > If it's a bad data value instead of 0, that'll cause problems.
> > > Ensemble-Stat only computes an observation rank value when all
of the
> > > ensemble members contain valid data at that point.
> > >
> > > You only getting 3 pairs in the output may mean that there are
only 3
> > grid
> > > points at which all 31 ensemble member contains valid data at
those
> > points.
> > >
> > > If it is bad data, you could switch it to 0, using the
censor_thresh
> and
> > > censor_val options:
> > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > >
> > > John
> > >
> > >
> > >
> > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > John,
> > > >
> > > > That is very clear. I really appreciate your detailed
explanation. I
> > had
> > > > misunderstood "missing/invalid value" with "not overlap"
before.
> > > >
> > > > Here is what I am doing: I have 31 ensemble inputs, and I am
using
> > > > satellite observed volcano ash for model verification. Based
on the
> > > current
> > > > results, there are only 3 matching pairs (which means only 3
grid
> cells
> > > > over which all 31 members have ash overlap with obs). So I am
> > "thinking"
> > > if
> > > > there is a way to look at the performance when only part of
the
> members
> > > > (like 20 out of 31) has overlap? Maybe this thought is
unrealistic.
> > Below
> > > > is my current config file:
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > >
> > > >
> > > > I added "NEP" and "NMEP" in the config file, this may help
make the
> > > > verification look better, right?
> > > >
> > > >
> > > > Thank you.
> > > >
> > > > Binyu
> > > >
> > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > OK, so assuming you have 10 input ensemble members and
ens_thresh =
> > 0.5
> > > > and
> > > > > vld_thresh = 0.5.
> > > > >
> > > > > So you pass in 10 input files, 1 for each ensemble member.
If 6 of
> > > those
> > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
ensemble_stat
> > > will
> > > > > error out and produce no output. So this is applied once and
> > > > ensemble-stat
> > > > > either will run or error out.
> > > > >
> > > > > One of the things that ensemble-stat does is compute an
ensemble
> mean
> > > > > field. At a particular grid point, if 6 of the 10 members do
not
> > have a
> > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the
ensemble
> > mean
> > > > > value will be computed as bad data. This applies when
deriving
> > ensemble
> > > > > fields that are written to the NetCDF output file. So this
is
> applied
> > > > once
> > > > > for each grid point and affects the values written to the
NetCDF
> > > output.
> > > > >
> > > > > So ens_thresh specifies the required ratio of ensemble
members that
> > > must
> > > > be
> > > > > present for ensemble_stat to run at all.
> > > > > And vld_thresh specifies the required ratio of valid data
for
> output
> > to
> > > > be
> > > > > created at each grid point.
> > > > >
> > > > > I get the idea of what you mean by "overlap" with the obs,
but the
> > > > details
> > > > > matter a lot. When an ensemble member does NOT overlap with
the
> obs,
> > is
> > > > > that encoded as bad data or as some legitimate value, like
0?
> > > > >
> > > > > If you have more questions, I'm guessing it'd be more
informative
> to
> > > look
> > > > > at a specific example.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > I am confused about the "missing" ensemble members and
data
> > > > > > definition. For example, there are 10 ensemble members
overall.
> > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > Ensemble member missing means:
> > > > > > 1. There is only input for 5 members, or
> > > > > > 2. all 10 member files exist, but only 5 of them have
overlap
> with
> > > obs.
> > > > > > I guess 1 is correct?
> > > > > >
> > > > > > The "missing" data refers to
> > > > > > 3. all 10 members have data, but some data are not valid
> > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > Is 3 correct?
> > > > > >
> > > > > > It seems "missing" does NOT mean "no overlap". If this is
the
> case,
> > > is
> > > > > > there a way to get ensemble statistics if I don't need all
> ensemble
> > > > > members
> > > > > > to overlap with obs? I.e: is it possible to get the RELY
stat.
> by
> > > > > > setting more than 5 has overlap instead of 10?
> > > > > >
> > > > > > I hope it makes sense. Thank you.
> > > > > > Binyu
> > > > > >
> > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > The ens_thresh and vld_thresh options tell ensemble_stat
how to
> > > > handle
> > > > > > > missing data... ens_thresh for missing ensemble member
inputs
> and
> > > > > > > vld_thresh for missing data within the input ensemble
member
> > data.
> > > > They
> > > > > > are
> > > > > > > as described in the user's guide:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > >
> > > > > > > ----------------
> > > > > > >
> > > > > > > When summarizing the ensemble, compute a ratio of the
number of
> > > valid
> > > > > > > ensemble fields to the total number of ensemble members.
If
> this
> > > > ratio
> > > > > is
> > > > > > > less than the *ens_thresh*, then quit with an error.
This
> > threshold
> > > > > must
> > > > > > be
> > > > > > > between 0 and 1. Setting this threshold to 1 will
require that
> > all
> > > > > > ensemble
> > > > > > > members be present to be processed.
> > > > > > >
> > > > > > > When summarizing the ensemble, for each grid point
compute a
> > ratio
> > > of
> > > > > the
> > > > > > > number of valid data values to the number of ensemble
members.
> If
> > > > that
> > > > > > > ratio is less than *vld_thresh*, write out bad data.
This
> > threshold
> > > > > must
> > > > > > be
> > > > > > > between 0 and 1. Setting this threshold to 1 will
require each
> > grid
> > > > > point
> > > > > > > to contain valid data for all ensemble members.
> > > > > > >
> > > > > > > ----------------
> > > > > > >
> > > > > > > If you have no missing ensemble members and none have
any
> missing
> > > > data
> > > > > > > values, then changing these settings would have no
impact on
> your
> > > > > > results.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > >
> > > > > > > >
> > > > > > > > Thank you, John.
> > > > > > > >
> > > > > > > > Looking at the config file, I had another question:
> > > > > > > >
> > > > > > > > fcst = {
> > > > > > > >
> > > > > > > > ens_thresh = 1.0;
> > > > > > > >
> > > > > > > > vld_thresh = 1.0;
> > > > > > > >
> > > > > > > >
> > > > > > > > I understand the meaning of ens_thresh. I got very few
> matching
> > > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5,
however, the
> > count
> > > > of
> > > > > > > number
> > > > > > > > for ECNT are the same, the number of ensemble values
are the
> > same
> > > > as
> > > > > > > well,
> > > > > > > > they are all 31 (totally there are 31 ensemble
members),
> > > > > > > >
> > > > > > > >
> > > > > > > > Here is the *stat file:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > >
> > > > > > > >
> > > > > > > > Any thing wrong here? Thank you.
> > > > > > > >
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Binyu,
> > > > > > > > >
> > > > > > > > > No that's correct. In ensemble_stat the cat_thresh
setting
> > > > applies
> > > > > to
> > > > > > > the
> > > > > > > > > "ens" dictionary and defines the thresholds that to
be used
> > to
> > > in
> > > > > > > > defining
> > > > > > > > > probabilities written to the NetCDF output file.
> > > > > > > > >
> > > > > > > > > The cat_thresh option does not apply to the fcst and
obs
> > > > > > dictionaries.
> > > > > > > > None
> > > > > > > > > of the output stat line types produced by
ensemble_stat
> > (listed
> > > > > > below)
> > > > > > > > are
> > > > > > > > > based on the application of a categorical threshold
defined
> > by
> > > > > > > > cat_thresh.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > output_flag = {
> > > > > > > > > ecnt = NONE;
> > > > > > > > > rps = NONE;
> > > > > > > > > rhist = NONE;
> > > > > > > > > phist = NONE;
> > > > > > > > > orank = NONE;
> > > > > > > > > ssvar = NONE;
> > > > > > > > > relp = NONE;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > On Mon, May 24, 2021 at 3:55 PM binyu.wang at noaa.gov
via
> RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: config files
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I have two ensemble config files as below (the
other
> parts
> > > of
> > > > > the
> > > > > > > > config
> > > > > > > > > > are the same), will they give exactly the same
output?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The only difference lies in there is no
"cat_thresh" in
> > fcst
> > > > and
> > > > > > obs
> > > > > > > > > > entries. Based on my tests, it looks like the
ensemble
> > > results
> > > > > are
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > with the two config files below, but I am afraid I
may
> miss
> > > > some
> > > > > > > output
> > > > > > > > > > flags.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > fcst = {
> > > > > > > > > >
> > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "VAFTD";
> > > > > > > > > >
> > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > >
> > > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ens = {
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "VAFTD";
> > > > > > > > > >
> > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > >
> > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > >
> > > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > obs = {
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > >
> > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > >
> > > > > > > > > > cat_thresh = [ >=0.01,>=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > ---------------------------------------------
> > > > > > > > > >
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > fcst = {
> > > > > > > > > >
> > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "VAFTD";
> > > > > > > > > >
> > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > >
> > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ens = {
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "VAFTD";
> > > > > > > > > >
> > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > >
> > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > >
> > > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > >=0.5,>=1,>=5,>=10
> > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > obs = {
> > > > > > > > > >
> > > > > > > > > > field = [
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > >
> > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > >
> > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > >
> > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > So if they will lead to different outputs, what
will be
> the
> > > > > > > difference?
> > > > > > > > > > Thank you.
> > > > > > > > > >
> > > > > > > > > > Binyu
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Fri May 28 10:12:49 2021
Binyu,
Correct, setting vld_thresh = 0.5 will result in an ensemble mean
being
computed at any grid point which contains valid data for at least 50%
of
the ensemble members.
MET stores bad data values internally as -9999.
This usually is NOT a concern because most of the time ensemble
forecasts
do not include any bata data values. But I haven't worked with your
ensemble data before so I don't know if/how many bad data values it
contains.
John
On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> John,
>
> Currently, "vld_thresh" only sets thresh to INVALID data, is there
a way
> to compute the ensemble mean and probabilities using only part of
ensemble
> members?
> e.g:
>
> 16 out of 31 have overlap with obs, the other 15 members are valid
but no
> overlap.
>
> Does that sound right?
>
> Thank you.
> Binyu
>
>
>
> On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > So you're saying "ensemble performance" but I need you to be more
> explicit
> > than that. Do you mean, for example, that that point needs to be
included
> > in the ranked histogram (RHIST) output line type?
> >
> > Remember that vld_thresh applies in the computation of the
ensemble mean
> > and probabilities written to the NetCDF output file. It does not
apply to
> > the computation of ranked histograms.
> >
> > John
> >
> > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > John,
> > >
> > > Actually the non-signal grid has "-99" in my ensemble member
(the
> > original
> > > data was converted using LogX), and it needs to be anti-log in
order to
> > > compare with observation, so I use "convert(x) =(10^(x))" in the
config
> > > file. I guess the converted value (10^(-99)=1E(-99)) is treated
as
> valid
> > > data, this is why there is no difference in my results when I
change
> > > "vld_thresh=1" to "vld_thresh=0.5", because there is no invalid
data in
> > all
> > > the members.
> > >
> > > If my guess above is correct, is there a way to look at the
ensemble
> > > performance when only part of the members (like 20 out of 31)
has
> > overlap?
> > >
> > > Binyu
> > >
> > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > When the ensemble member does not predict the ash at a grid
point, is
> > > that
> > > > stored as a missing data value or some real data value, like
0?
> > > >
> > > > If it's a bad data value instead of 0, that'll cause problems.
> > > > Ensemble-Stat only computes an observation rank value when all
of the
> > > > ensemble members contain valid data at that point.
> > > >
> > > > You only getting 3 pairs in the output may mean that there are
only 3
> > > grid
> > > > points at which all 31 ensemble member contains valid data at
those
> > > points.
> > > >
> > > > If it is bad data, you could switch it to 0, using the
censor_thresh
> > and
> > > > censor_val options:
> > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > >
> > > > > John,
> > > > >
> > > > > That is very clear. I really appreciate your detailed
explanation.
> I
> > > had
> > > > > misunderstood "missing/invalid value" with "not overlap"
before.
> > > > >
> > > > > Here is what I am doing: I have 31 ensemble inputs, and I am
using
> > > > > satellite observed volcano ash for model verification. Based
on the
> > > > current
> > > > > results, there are only 3 matching pairs (which means only 3
grid
> > cells
> > > > > over which all 31 members have ash overlap with obs). So I
am
> > > "thinking"
> > > > if
> > > > > there is a way to look at the performance when only part of
the
> > members
> > > > > (like 20 out of 31) has overlap? Maybe this thought is
unrealistic.
> > > Below
> > > > > is my current config file:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > >
> > > > >
> > > > > I added "NEP" and "NMEP" in the config file, this may help
make the
> > > > > verification look better, right?
> > > > >
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > OK, so assuming you have 10 input ensemble members and
> ens_thresh =
> > > 0.5
> > > > > and
> > > > > > vld_thresh = 0.5.
> > > > > >
> > > > > > So you pass in 10 input files, 1 for each ensemble member.
If 6
> of
> > > > those
> > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
> ensemble_stat
> > > > will
> > > > > > error out and produce no output. So this is applied once
and
> > > > > ensemble-stat
> > > > > > either will run or error out.
> > > > > >
> > > > > > One of the things that ensemble-stat does is compute an
ensemble
> > mean
> > > > > > field. At a particular grid point, if 6 of the 10 members
do not
> > > have a
> > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the
ensemble
> > > mean
> > > > > > value will be computed as bad data. This applies when
deriving
> > > ensemble
> > > > > > fields that are written to the NetCDF output file. So this
is
> > applied
> > > > > once
> > > > > > for each grid point and affects the values written to the
NetCDF
> > > > output.
> > > > > >
> > > > > > So ens_thresh specifies the required ratio of ensemble
members
> that
> > > > must
> > > > > be
> > > > > > present for ensemble_stat to run at all.
> > > > > > And vld_thresh specifies the required ratio of valid data
for
> > output
> > > to
> > > > > be
> > > > > > created at each grid point.
> > > > > >
> > > > > > I get the idea of what you mean by "overlap" with the obs,
but
> the
> > > > > details
> > > > > > matter a lot. When an ensemble member does NOT overlap
with the
> > obs,
> > > is
> > > > > > that encoded as bad data or as some legitimate value, like
0?
> > > > > >
> > > > > > If you have more questions, I'm guessing it'd be more
informative
> > to
> > > > look
> > > > > > at a specific example.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> > > > > > >
> > > > > > > Hello John,
> > > > > > >
> > > > > > > I am confused about the "missing" ensemble members and
data
> > > > > > > definition. For example, there are 10 ensemble members
> overall.
> > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > Ensemble member missing means:
> > > > > > > 1. There is only input for 5 members, or
> > > > > > > 2. all 10 member files exist, but only 5 of them have
overlap
> > with
> > > > obs.
> > > > > > > I guess 1 is correct?
> > > > > > >
> > > > > > > The "missing" data refers to
> > > > > > > 3. all 10 members have data, but some data are not valid
> > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > Is 3 correct?
> > > > > > >
> > > > > > > It seems "missing" does NOT mean "no overlap". If this
is the
> > case,
> > > > is
> > > > > > > there a way to get ensemble statistics if I don't need
all
> > ensemble
> > > > > > members
> > > > > > > to overlap with obs? I.e: is it possible to get the
RELY stat.
> > by
> > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > >
> > > > > > > I hope it makes sense. Thank you.
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Binyu,
> > > > > > > >
> > > > > > > > The ens_thresh and vld_thresh options tell
ensemble_stat how
> to
> > > > > handle
> > > > > > > > missing data... ens_thresh for missing ensemble member
inputs
> > and
> > > > > > > > vld_thresh for missing data within the input ensemble
member
> > > data.
> > > > > They
> > > > > > > are
> > > > > > > > as described in the user's guide:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > >
> > > > > > > > ----------------
> > > > > > > >
> > > > > > > > When summarizing the ensemble, compute a ratio of the
number
> of
> > > > valid
> > > > > > > > ensemble fields to the total number of ensemble
members. If
> > this
> > > > > ratio
> > > > > > is
> > > > > > > > less than the *ens_thresh*, then quit with an error.
This
> > > threshold
> > > > > > must
> > > > > > > be
> > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> that
> > > all
> > > > > > > ensemble
> > > > > > > > members be present to be processed.
> > > > > > > >
> > > > > > > > When summarizing the ensemble, for each grid point
compute a
> > > ratio
> > > > of
> > > > > > the
> > > > > > > > number of valid data values to the number of ensemble
> members.
> > If
> > > > > that
> > > > > > > > ratio is less than *vld_thresh*, write out bad data.
This
> > > threshold
> > > > > > must
> > > > > > > be
> > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> each
> > > grid
> > > > > > point
> > > > > > > > to contain valid data for all ensemble members.
> > > > > > > >
> > > > > > > > ----------------
> > > > > > > >
> > > > > > > > If you have no missing ensemble members and none have
any
> > missing
> > > > > data
> > > > > > > > values, then changing these settings would have no
impact on
> > your
> > > > > > > results.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > >
> > > > > > > > >
> > > > > > > > > Thank you, John.
> > > > > > > > >
> > > > > > > > > Looking at the config file, I had another question:
> > > > > > > > >
> > > > > > > > > fcst = {
> > > > > > > > >
> > > > > > > > > ens_thresh = 1.0;
> > > > > > > > >
> > > > > > > > > vld_thresh = 1.0;
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I understand the meaning of ens_thresh. I got very
few
> > matching
> > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5,
however, the
> > > count
> > > > > of
> > > > > > > > number
> > > > > > > > > for ECNT are the same, the number of ensemble values
are
> the
> > > same
> > > > > as
> > > > > > > > well,
> > > > > > > > > they are all 31 (totally there are 31 ensemble
members),
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Here is the *stat file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Binyu,
> > > > > > > > > >
> > > > > > > > > > No that's correct. In ensemble_stat the cat_thresh
> setting
> > > > > applies
> > > > > > to
> > > > > > > > the
> > > > > > > > > > "ens" dictionary and defines the thresholds that
to be
> used
> > > to
> > > > in
> > > > > > > > > defining
> > > > > > > > > > probabilities written to the NetCDF output file.
> > > > > > > > > >
> > > > > > > > > > The cat_thresh option does not apply to the fcst
and obs
> > > > > > > dictionaries.
> > > > > > > > > None
> > > > > > > > > > of the output stat line types produced by
ensemble_stat
> > > (listed
> > > > > > > below)
> > > > > > > > > are
> > > > > > > > > > based on the application of a categorical
threshold
> defined
> > > by
> > > > > > > > > cat_thresh.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > output_flag = {
> > > > > > > > > > ecnt = NONE;
> > > > > > > > > > rps = NONE;
> > > > > > > > > > rhist = NONE;
> > > > > > > > > > phist = NONE;
> > > > > > > > > > orank = NONE;
> > > > > > > > > > ssvar = NONE;
> > > > > > > > > > relp = NONE;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
binyu.wang at noaa.gov via
> > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was
acted upon.
> > > > > > > > > > > Transaction: Ticket created by
binyu.wang at noaa.gov
> > > > > > > > > > > Queue: met_help
> > > > > > > > > > > Subject: config files
> > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > Status: new
> > > > > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I have two ensemble config files as below (the
other
> > parts
> > > > of
> > > > > > the
> > > > > > > > > config
> > > > > > > > > > > are the same), will they give exactly the same
output?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The only difference lies in there is no
"cat_thresh" in
> > > fcst
> > > > > and
> > > > > > > obs
> > > > > > > > > > > entries. Based on my tests, it looks like the
ensemble
> > > > results
> > > > > > are
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > with the two config files below, but I am afraid
I may
> > miss
> > > > > some
> > > > > > > > output
> > > > > > > > > > > flags.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > fcst = {
> > > > > > > > > > >
> > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > >
> > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > >
> > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ens = {
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > >
> > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > >
> > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > >
> > > > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > obs = {
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > >
> > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > >
> > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > ---------------------------------------------
> > > > > > > > > > >
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > fcst = {
> > > > > > > > > > >
> > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > >
> > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > >
> > > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ens = {
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > >
> > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > >
> > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > >
> > > > > > > > > > > cat_thresh = [>=0.01, >=0.05, >=0.1,>=0.3,
> > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > obs = {
> > > > > > > > > > >
> > > > > > > > > > > field = [
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >
> > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > >
> > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > >
> > > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > So if they will lead to different outputs, what
will be
> > the
> > > > > > > > difference?
> > > > > > > > > > > Thank you.
> > > > > > > > > > >
> > > > > > > > > > > Binyu
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Fri May 28 13:53:47 2021
Hello John,
1. Based on what you said,
e.g: in my example (31 members) if I set vld_thresh=0.5, this will
result
in ensemble mean and probability being written into the NetCDF at any
grid
when 16 or more ensemble members have overlap with obs? I had thought
the
function of vld_thresh=0.5 is to throw away all the grids when half of
the
members have with invalid data.
2 . So there is no way that we can set some thresh to change the rule
of computation
of ranked histograms? I mean we must use ALL members for that?
Thank you and have a great long weekend.
Binyu
On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> Correct, setting vld_thresh = 0.5 will result in an ensemble mean
being
> computed at any grid point which contains valid data for at least
50% of
> the ensemble members.
>
> MET stores bad data values internally as -9999.
>
> This usually is NOT a concern because most of the time ensemble
forecasts
> do not include any bata data values. But I haven't worked with your
> ensemble data before so I don't know if/how many bad data values it
> contains.
>
> John
>
> On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > John,
> >
> > Currently, "vld_thresh" only sets thresh to INVALID data, is
there a way
> > to compute the ensemble mean and probabilities using only part of
> ensemble
> > members?
> > e.g:
> >
> > 16 out of 31 have overlap with obs, the other 15 members are valid
but no
> > overlap.
> >
> > Does that sound right?
> >
> > Thank you.
> > Binyu
> >
> >
> >
> > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > So you're saying "ensemble performance" but I need you to be
more
> > explicit
> > > than that. Do you mean, for example, that that point needs to be
> included
> > > in the ranked histogram (RHIST) output line type?
> > >
> > > Remember that vld_thresh applies in the computation of the
ensemble
> mean
> > > and probabilities written to the NetCDF output file. It does not
apply
> to
> > > the computation of ranked histograms.
> > >
> > > John
> > >
> > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > John,
> > > >
> > > > Actually the non-signal grid has "-99" in my ensemble member
(the
> > > original
> > > > data was converted using LogX), and it needs to be anti-log in
order
> to
> > > > compare with observation, so I use "convert(x) =(10^(x))" in
the
> config
> > > > file. I guess the converted value (10^(-99)=1E(-99)) is
treated as
> > valid
> > > > data, this is why there is no difference in my results when I
change
> > > > "vld_thresh=1" to "vld_thresh=0.5", because there is no
invalid data
> in
> > > all
> > > > the members.
> > > >
> > > > If my guess above is correct, is there a way to look at the
ensemble
> > > > performance when only part of the members (like 20 out of 31)
has
> > > overlap?
> > > >
> > > > Binyu
> > > >
> > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > When the ensemble member does not predict the ash at a grid
point,
> is
> > > > that
> > > > > stored as a missing data value or some real data value, like
0?
> > > > >
> > > > > If it's a bad data value instead of 0, that'll cause
problems.
> > > > > Ensemble-Stat only computes an observation rank value when
all of
> the
> > > > > ensemble members contain valid data at that point.
> > > > >
> > > > > You only getting 3 pairs in the output may mean that there
are
> only 3
> > > > grid
> > > > > points at which all 31 ensemble member contains valid data
at those
> > > > points.
> > > > >
> > > > > If it is bad data, you could switch it to 0, using the
> censor_thresh
> > > and
> > > > > censor_val options:
> > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > That is very clear. I really appreciate your detailed
> explanation.
> > I
> > > > had
> > > > > > misunderstood "missing/invalid value" with "not overlap"
before.
> > > > > >
> > > > > > Here is what I am doing: I have 31 ensemble inputs, and I
am
> using
> > > > > > satellite observed volcano ash for model verification.
Based on
> the
> > > > > current
> > > > > > results, there are only 3 matching pairs (which means only
3 grid
> > > cells
> > > > > > over which all 31 members have ash overlap with obs). So I
am
> > > > "thinking"
> > > > > if
> > > > > > there is a way to look at the performance when only part
of the
> > > members
> > > > > > (like 20 out of 31) has overlap? Maybe this thought is
> unrealistic.
> > > > Below
> > > > > > is my current config file:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > > >
> > > > > >
> > > > > > I added "NEP" and "NMEP" in the config file, this may help
make
> the
> > > > > > verification look better, right?
> > > > > >
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > OK, so assuming you have 10 input ensemble members and
> > ens_thresh =
> > > > 0.5
> > > > > > and
> > > > > > > vld_thresh = 0.5.
> > > > > > >
> > > > > > > So you pass in 10 input files, 1 for each ensemble
member. If 6
> > of
> > > > > those
> > > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
> > ensemble_stat
> > > > > will
> > > > > > > error out and produce no output. So this is applied once
and
> > > > > > ensemble-stat
> > > > > > > either will run or error out.
> > > > > > >
> > > > > > > One of the things that ensemble-stat does is compute an
> ensemble
> > > mean
> > > > > > > field. At a particular grid point, if 6 of the 10
members do
> not
> > > > have a
> > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then the
> ensemble
> > > > mean
> > > > > > > value will be computed as bad data. This applies when
deriving
> > > > ensemble
> > > > > > > fields that are written to the NetCDF output file. So
this is
> > > applied
> > > > > > once
> > > > > > > for each grid point and affects the values written to
the
> NetCDF
> > > > > output.
> > > > > > >
> > > > > > > So ens_thresh specifies the required ratio of ensemble
members
> > that
> > > > > must
> > > > > > be
> > > > > > > present for ensemble_stat to run at all.
> > > > > > > And vld_thresh specifies the required ratio of valid
data for
> > > output
> > > > to
> > > > > > be
> > > > > > > created at each grid point.
> > > > > > >
> > > > > > > I get the idea of what you mean by "overlap" with the
obs, but
> > the
> > > > > > details
> > > > > > > matter a lot. When an ensemble member does NOT overlap
with the
> > > obs,
> > > > is
> > > > > > > that encoded as bad data or as some legitimate value,
like 0?
> > > > > > >
> > > > > > > If you have more questions, I'm guessing it'd be more
> informative
> > > to
> > > > > look
> > > > > > > at a specific example.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > >
> > > > > > > >
> > > > > > > > Hello John,
> > > > > > > >
> > > > > > > > I am confused about the "missing" ensemble members and
data
> > > > > > > > definition. For example, there are 10 ensemble
members
> > overall.
> > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > > Ensemble member missing means:
> > > > > > > > 1. There is only input for 5 members, or
> > > > > > > > 2. all 10 member files exist, but only 5 of them have
overlap
> > > with
> > > > > obs.
> > > > > > > > I guess 1 is correct?
> > > > > > > >
> > > > > > > > The "missing" data refers to
> > > > > > > > 3. all 10 members have data, but some data are not
valid
> > > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > > Is 3 correct?
> > > > > > > >
> > > > > > > > It seems "missing" does NOT mean "no overlap". If this
is the
> > > case,
> > > > > is
> > > > > > > > there a way to get ensemble statistics if I don't need
all
> > > ensemble
> > > > > > > members
> > > > > > > > to overlap with obs? I.e: is it possible to get the
RELY
> stat.
> > > by
> > > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > > >
> > > > > > > > I hope it makes sense. Thank you.
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Binyu,
> > > > > > > > >
> > > > > > > > > The ens_thresh and vld_thresh options tell
ensemble_stat
> how
> > to
> > > > > > handle
> > > > > > > > > missing data... ens_thresh for missing ensemble
member
> inputs
> > > and
> > > > > > > > > vld_thresh for missing data within the input
ensemble
> member
> > > > data.
> > > > > > They
> > > > > > > > are
> > > > > > > > > as described in the user's guide:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > > >
> > > > > > > > > ----------------
> > > > > > > > >
> > > > > > > > > When summarizing the ensemble, compute a ratio of
the
> number
> > of
> > > > > valid
> > > > > > > > > ensemble fields to the total number of ensemble
members. If
> > > this
> > > > > > ratio
> > > > > > > is
> > > > > > > > > less than the *ens_thresh*, then quit with an error.
This
> > > > threshold
> > > > > > > must
> > > > > > > > be
> > > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> > that
> > > > all
> > > > > > > > ensemble
> > > > > > > > > members be present to be processed.
> > > > > > > > >
> > > > > > > > > When summarizing the ensemble, for each grid point
compute
> a
> > > > ratio
> > > > > of
> > > > > > > the
> > > > > > > > > number of valid data values to the number of
ensemble
> > members.
> > > If
> > > > > > that
> > > > > > > > > ratio is less than *vld_thresh*, write out bad data.
This
> > > > threshold
> > > > > > > must
> > > > > > > > be
> > > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> > each
> > > > grid
> > > > > > > point
> > > > > > > > > to contain valid data for all ensemble members.
> > > > > > > > >
> > > > > > > > > ----------------
> > > > > > > > >
> > > > > > > > > If you have no missing ensemble members and none
have any
> > > missing
> > > > > > data
> > > > > > > > > values, then changing these settings would have no
impact
> on
> > > your
> > > > > > > > results.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, May 24, 2021 at 4:51 PM binyu.wang at noaa.gov
via
> RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > >
> > > > > > > > > >
> > > > > > > > > > Thank you, John.
> > > > > > > > > >
> > > > > > > > > > Looking at the config file, I had another
question:
> > > > > > > > > >
> > > > > > > > > > fcst = {
> > > > > > > > > >
> > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I understand the meaning of ens_thresh. I got very
few
> > > matching
> > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5,
however,
> the
> > > > count
> > > > > > of
> > > > > > > > > number
> > > > > > > > > > for ECNT are the same, the number of ensemble
values are
> > the
> > > > same
> > > > > > as
> > > > > > > > > well,
> > > > > > > > > > they are all 31 (totally there are 31 ensemble
members),
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Here is the *stat file:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Binyu
> > > > > > > > > >
> > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley Gotway
via
> RT <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Binyu,
> > > > > > > > > > >
> > > > > > > > > > > No that's correct. In ensemble_stat the
cat_thresh
> > setting
> > > > > > applies
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > "ens" dictionary and defines the thresholds that
to be
> > used
> > > > to
> > > > > in
> > > > > > > > > > defining
> > > > > > > > > > > probabilities written to the NetCDF output file.
> > > > > > > > > > >
> > > > > > > > > > > The cat_thresh option does not apply to the fcst
and
> obs
> > > > > > > > dictionaries.
> > > > > > > > > > None
> > > > > > > > > > > of the output stat line types produced by
ensemble_stat
> > > > (listed
> > > > > > > > below)
> > > > > > > > > > are
> > > > > > > > > > > based on the application of a categorical
threshold
> > defined
> > > > by
> > > > > > > > > > cat_thresh.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > output_flag = {
> > > > > > > > > > > ecnt = NONE;
> > > > > > > > > > > rps = NONE;
> > > > > > > > > > > rhist = NONE;
> > > > > > > > > > > phist = NONE;
> > > > > > > > > > > orank = NONE;
> > > > > > > > > > > ssvar = NONE;
> > > > > > > > > > > relp = NONE;
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
binyu.wang at noaa.gov
> via
> > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was
acted
> upon.
> > > > > > > > > > > > Transaction: Ticket created by
binyu.wang at noaa.gov
> > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > Subject: config files
> > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > > Status: new
> > > > > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I have two ensemble config files as below
(the other
> > > parts
> > > > > of
> > > > > > > the
> > > > > > > > > > config
> > > > > > > > > > > > are the same), will they give exactly the same
> output?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The only difference lies in there is no
"cat_thresh"
> in
> > > > fcst
> > > > > > and
> > > > > > > > obs
> > > > > > > > > > > > entries. Based on my tests, it looks like the
> ensemble
> > > > > results
> > > > > > > are
> > > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > with the two config files below, but I am
afraid I
> may
> > > miss
> > > > > > some
> > > > > > > > > output
> > > > > > > > > > > > flags.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > > fcst = {
> > > > > > > > > > > >
> > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > >
> > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > >
> > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > ens = {
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > >
> > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > >
> > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > >
> > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > obs = {
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > >
> > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > >
> > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > ---------------------------------------------
> > > > > > > > > > > >
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > > fcst = {
> > > > > > > > > > > >
> > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > >
> > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > >
> > > > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > ens = {
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > >
> > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > >
> > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > >
> > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > obs = {
> > > > > > > > > > > >
> > > > > > > > > > > > field = [
> > > > > > > > > > > >
> > > > > > > > > > > > {
> > > > > > > > > > > >
> > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > >
> > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > >
> > > > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > ];
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > So if they will lead to different outputs,
what will
> be
> > > the
> > > > > > > > > difference?
> > > > > > > > > > > > Thank you.
> > > > > > > > > > > >
> > > > > > > > > > > > Binyu
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Fri May 28 15:05:38 2021
Binyu,
There is no way to change the computation of the ranked histogram
logic. If
we let the number of valid ensemble members vary from grid point to
grid
point, then the resulting observation rank values cannot be combined
into a
ranked histogram.
For example, let's say you compute the observation rank as...
- At grid point 1, obs rank is 5 out of 10 valid ensemble member
values.
- At grid point 2, obs rank is 5 out of 15 valid ensemble member
values.
- At grid point 3, obs rank is 5 out of 20 valid ensemble member
values.
While all 3 points have the same rank of 5, they cannot be combined
into a
rank histogram because of the varying number of valid ensemble member
values.
But again this only matters if your ensemble forecasts contain missing
data
values. If they do, you could always use the "censor_thresh" and
"censor_val" logic to replace the missing data value with some minimum
value.
John
On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> Hello John,
>
> 1. Based on what you said,
> e.g: in my example (31 members) if I set vld_thresh=0.5, this will
result
> in ensemble mean and probability being written into the NetCDF at
any grid
> when 16 or more ensemble members have overlap with obs? I had
thought the
> function of vld_thresh=0.5 is to throw away all the grids when half
of the
> members have with invalid data.
>
>
> 2 . So there is no way that we can set some thresh to change the
rule
> of computation
> of ranked histograms? I mean we must use ALL members for that?
>
>
>
> Thank you and have a great long weekend.
>
> Binyu
>
> On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Binyu,
> >
> > Correct, setting vld_thresh = 0.5 will result in an ensemble mean
being
> > computed at any grid point which contains valid data for at least
50% of
> > the ensemble members.
> >
> > MET stores bad data values internally as -9999.
> >
> > This usually is NOT a concern because most of the time ensemble
forecasts
> > do not include any bata data values. But I haven't worked with
your
> > ensemble data before so I don't know if/how many bad data values
it
> > contains.
> >
> > John
> >
> > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > John,
> > >
> > > Currently, "vld_thresh" only sets thresh to INVALID data, is
there a
> way
> > > to compute the ensemble mean and probabilities using only part
of
> > ensemble
> > > members?
> > > e.g:
> > >
> > > 16 out of 31 have overlap with obs, the other 15 members are
valid but
> no
> > > overlap.
> > >
> > > Does that sound right?
> > >
> > > Thank you.
> > > Binyu
> > >
> > >
> > >
> > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > So you're saying "ensemble performance" but I need you to be
more
> > > explicit
> > > > than that. Do you mean, for example, that that point needs to
be
> > included
> > > > in the ranked histogram (RHIST) output line type?
> > > >
> > > > Remember that vld_thresh applies in the computation of the
ensemble
> > mean
> > > > and probabilities written to the NetCDF output file. It does
not
> apply
> > to
> > > > the computation of ranked histograms.
> > > >
> > > > John
> > > >
> > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > >
> > > > > John,
> > > > >
> > > > > Actually the non-signal grid has "-99" in my ensemble member
(the
> > > > original
> > > > > data was converted using LogX), and it needs to be anti-log
in
> order
> > to
> > > > > compare with observation, so I use "convert(x) =(10^(x))" in
the
> > config
> > > > > file. I guess the converted value (10^(-99)=1E(-99)) is
treated as
> > > valid
> > > > > data, this is why there is no difference in my results when
I
> change
> > > > > "vld_thresh=1" to "vld_thresh=0.5", because there is no
invalid
> data
> > in
> > > > all
> > > > > the members.
> > > > >
> > > > > If my guess above is correct, is there a way to look at the
> ensemble
> > > > > performance when only part of the members (like 20 out of
31) has
> > > > overlap?
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > When the ensemble member does not predict the ash at a
grid
> point,
> > is
> > > > > that
> > > > > > stored as a missing data value or some real data value,
like 0?
> > > > > >
> > > > > > If it's a bad data value instead of 0, that'll cause
problems.
> > > > > > Ensemble-Stat only computes an observation rank value when
all of
> > the
> > > > > > ensemble members contain valid data at that point.
> > > > > >
> > > > > > You only getting 3 pairs in the output may mean that there
are
> > only 3
> > > > > grid
> > > > > > points at which all 31 ensemble member contains valid data
at
> those
> > > > > points.
> > > > > >
> > > > > > If it is bad data, you could switch it to 0, using the
> > censor_thresh
> > > > and
> > > > > > censor_val options:
> > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> > > > > > >
> > > > > > > John,
> > > > > > >
> > > > > > > That is very clear. I really appreciate your detailed
> > explanation.
> > > I
> > > > > had
> > > > > > > misunderstood "missing/invalid value" with "not overlap"
> before.
> > > > > > >
> > > > > > > Here is what I am doing: I have 31 ensemble inputs, and
I am
> > using
> > > > > > > satellite observed volcano ash for model verification.
Based on
> > the
> > > > > > current
> > > > > > > results, there are only 3 matching pairs (which means
only 3
> grid
> > > > cells
> > > > > > > over which all 31 members have ash overlap with obs). So
I am
> > > > > "thinking"
> > > > > > if
> > > > > > > there is a way to look at the performance when only part
of the
> > > > members
> > > > > > > (like 20 out of 31) has overlap? Maybe this thought is
> > unrealistic.
> > > > > Below
> > > > > > > is my current config file:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > > > >
> > > > > > >
> > > > > > > I added "NEP" and "NMEP" in the config file, this may
help make
> > the
> > > > > > > verification look better, right?
> > > > > > >
> > > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Binyu,
> > > > > > > >
> > > > > > > > OK, so assuming you have 10 input ensemble members and
> > > ens_thresh =
> > > > > 0.5
> > > > > > > and
> > > > > > > > vld_thresh = 0.5.
> > > > > > > >
> > > > > > > > So you pass in 10 input files, 1 for each ensemble
member.
> If 6
> > > of
> > > > > > those
> > > > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh), then
> > > ensemble_stat
> > > > > > will
> > > > > > > > error out and produce no output. So this is applied
once and
> > > > > > > ensemble-stat
> > > > > > > > either will run or error out.
> > > > > > > >
> > > > > > > > One of the things that ensemble-stat does is compute
an
> > ensemble
> > > > mean
> > > > > > > > field. At a particular grid point, if 6 of the 10
members do
> > not
> > > > > have a
> > > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then
the
> > ensemble
> > > > > mean
> > > > > > > > value will be computed as bad data. This applies when
> deriving
> > > > > ensemble
> > > > > > > > fields that are written to the NetCDF output file. So
this is
> > > > applied
> > > > > > > once
> > > > > > > > for each grid point and affects the values written to
the
> > NetCDF
> > > > > > output.
> > > > > > > >
> > > > > > > > So ens_thresh specifies the required ratio of ensemble
> members
> > > that
> > > > > > must
> > > > > > > be
> > > > > > > > present for ensemble_stat to run at all.
> > > > > > > > And vld_thresh specifies the required ratio of valid
data for
> > > > output
> > > > > to
> > > > > > > be
> > > > > > > > created at each grid point.
> > > > > > > >
> > > > > > > > I get the idea of what you mean by "overlap" with the
obs,
> but
> > > the
> > > > > > > details
> > > > > > > > matter a lot. When an ensemble member does NOT overlap
with
> the
> > > > obs,
> > > > > is
> > > > > > > > that encoded as bad data or as some legitimate value,
like 0?
> > > > > > > >
> > > > > > > > If you have more questions, I'm guessing it'd be more
> > informative
> > > > to
> > > > > > look
> > > > > > > > at a specific example.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov
via RT
> <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > >
> > > > > > > > >
> > > > > > > > > Hello John,
> > > > > > > > >
> > > > > > > > > I am confused about the "missing" ensemble members
and data
> > > > > > > > > definition. For example, there are 10 ensemble
members
> > > overall.
> > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > > > Ensemble member missing means:
> > > > > > > > > 1. There is only input for 5 members, or
> > > > > > > > > 2. all 10 member files exist, but only 5 of them
have
> overlap
> > > > with
> > > > > > obs.
> > > > > > > > > I guess 1 is correct?
> > > > > > > > >
> > > > > > > > > The "missing" data refers to
> > > > > > > > > 3. all 10 members have data, but some data are not
valid
> > > > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > > > Is 3 correct?
> > > > > > > > >
> > > > > > > > > It seems "missing" does NOT mean "no overlap". If
this is
> the
> > > > case,
> > > > > > is
> > > > > > > > > there a way to get ensemble statistics if I don't
need all
> > > > ensemble
> > > > > > > > members
> > > > > > > > > to overlap with obs? I.e: is it possible to get the
RELY
> > stat.
> > > > by
> > > > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > > > >
> > > > > > > > > I hope it makes sense. Thank you.
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Binyu,
> > > > > > > > > >
> > > > > > > > > > The ens_thresh and vld_thresh options tell
ensemble_stat
> > how
> > > to
> > > > > > > handle
> > > > > > > > > > missing data... ens_thresh for missing ensemble
member
> > inputs
> > > > and
> > > > > > > > > > vld_thresh for missing data within the input
ensemble
> > member
> > > > > data.
> > > > > > > They
> > > > > > > > > are
> > > > > > > > > > as described in the user's guide:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > > > >
> > > > > > > > > > ----------------
> > > > > > > > > >
> > > > > > > > > > When summarizing the ensemble, compute a ratio of
the
> > number
> > > of
> > > > > > valid
> > > > > > > > > > ensemble fields to the total number of ensemble
members.
> If
> > > > this
> > > > > > > ratio
> > > > > > > > is
> > > > > > > > > > less than the *ens_thresh*, then quit with an
error. This
> > > > > threshold
> > > > > > > > must
> > > > > > > > > be
> > > > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> > > that
> > > > > all
> > > > > > > > > ensemble
> > > > > > > > > > members be present to be processed.
> > > > > > > > > >
> > > > > > > > > > When summarizing the ensemble, for each grid point
> compute
> > a
> > > > > ratio
> > > > > > of
> > > > > > > > the
> > > > > > > > > > number of valid data values to the number of
ensemble
> > > members.
> > > > If
> > > > > > > that
> > > > > > > > > > ratio is less than *vld_thresh*, write out bad
data. This
> > > > > threshold
> > > > > > > > must
> > > > > > > > > be
> > > > > > > > > > between 0 and 1. Setting this threshold to 1 will
require
> > > each
> > > > > grid
> > > > > > > > point
> > > > > > > > > > to contain valid data for all ensemble members.
> > > > > > > > > >
> > > > > > > > > > ----------------
> > > > > > > > > >
> > > > > > > > > > If you have no missing ensemble members and none
have any
> > > > missing
> > > > > > > data
> > > > > > > > > > values, then changing these settings would have no
impact
> > on
> > > > your
> > > > > > > > > results.
> > > > > > > > > >
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
binyu.wang at noaa.gov via
> > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thank you, John.
> > > > > > > > > > >
> > > > > > > > > > > Looking at the config file, I had another
question:
> > > > > > > > > > >
> > > > > > > > > > > fcst = {
> > > > > > > > > > >
> > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I understand the meaning of ens_thresh. I got
very few
> > > > matching
> > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5,
however,
> > the
> > > > > count
> > > > > > > of
> > > > > > > > > > number
> > > > > > > > > > > for ECNT are the same, the number of ensemble
values
> are
> > > the
> > > > > same
> > > > > > > as
> > > > > > > > > > well,
> > > > > > > > > > > they are all 31 (totally there are 31 ensemble
> members),
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Here is the *stat file:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > > > ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Binyu
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley
Gotway via
> > RT <
> > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Binyu,
> > > > > > > > > > > >
> > > > > > > > > > > > No that's correct. In ensemble_stat the
cat_thresh
> > > setting
> > > > > > > applies
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > "ens" dictionary and defines the thresholds
that to
> be
> > > used
> > > > > to
> > > > > > in
> > > > > > > > > > > defining
> > > > > > > > > > > > probabilities written to the NetCDF output
file.
> > > > > > > > > > > >
> > > > > > > > > > > > The cat_thresh option does not apply to the
fcst and
> > obs
> > > > > > > > > dictionaries.
> > > > > > > > > > > None
> > > > > > > > > > > > of the output stat line types produced by
> ensemble_stat
> > > > > (listed
> > > > > > > > > below)
> > > > > > > > > > > are
> > > > > > > > > > > > based on the application of a categorical
threshold
> > > defined
> > > > > by
> > > > > > > > > > > cat_thresh.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > output_flag = {
> > > > > > > > > > > > ecnt = NONE;
> > > > > > > > > > > > rps = NONE;
> > > > > > > > > > > > rhist = NONE;
> > > > > > > > > > > > phist = NONE;
> > > > > > > > > > > > orank = NONE;
> > > > > > > > > > > > ssvar = NONE;
> > > > > > > > > > > > relp = NONE;
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
binyu.wang at noaa.gov
> > via
> > > > RT <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973 was
acted
> > upon.
> > > > > > > > > > > > > Transaction: Ticket created by
binyu.wang at noaa.gov
> > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > Subject: config files
> > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > Ticket <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hello,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have two ensemble config files as below
(the
> other
> > > > parts
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > config
> > > > > > > > > > > > > are the same), will they give exactly the
same
> > output?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The only difference lies in there is no
> "cat_thresh"
> > in
> > > > > fcst
> > > > > > > and
> > > > > > > > > obs
> > > > > > > > > > > > > entries. Based on my tests, it looks like
the
> > ensemble
> > > > > > results
> > > > > > > > are
> > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > with the two config files below, but I am
afraid I
> > may
> > > > miss
> > > > > > > some
> > > > > > > > > > output
> > > > > > > > > > > > > flags.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1.
> > > > > > > > > > > > >
> > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > ens = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > >
> > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > obs = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > >
> > > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > >
---------------------------------------------
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2.
> > > > > > > > > > > > >
> > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > ens = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > >
> > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > obs = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > field = [
> > > > > > > > > > > > >
> > > > > > > > > > > > > {
> > > > > > > > > > > > >
> > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > >
> > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > >
> > > > > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > ];
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > So if they will lead to different outputs,
what
> will
> > be
> > > > the
> > > > > > > > > > difference?
> > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Binyu
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Tue Jun 01 15:02:27 2021
Hello John,
We can use "censor_thresh" and "censor_val" logic to replace the
missing
data value with some minimum
value, but how about the opposite way?
E.g: the non-signal grid has value 10^(-99)=1E(-99) in my forecast, so
they
are all valid data. What if I want to lower the standard vld_thresh
from 1
to 0.6? Can I set the non-signal grid to invalid data? I mean I don't
need
all members to overlap. Does that sound right?
Thank you.
Binyu
On Fri, May 28, 2021 at 5:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> There is no way to change the computation of the ranked histogram
logic. If
> we let the number of valid ensemble members vary from grid point to
grid
> point, then the resulting observation rank values cannot be combined
into a
> ranked histogram.
>
> For example, let's say you compute the observation rank as...
> - At grid point 1, obs rank is 5 out of 10 valid ensemble member
values.
> - At grid point 2, obs rank is 5 out of 15 valid ensemble member
values.
> - At grid point 3, obs rank is 5 out of 20 valid ensemble member
values.
>
> While all 3 points have the same rank of 5, they cannot be combined
into a
> rank histogram because of the varying number of valid ensemble
member
> values.
>
> But again this only matters if your ensemble forecasts contain
missing data
> values. If they do, you could always use the "censor_thresh" and
> "censor_val" logic to replace the missing data value with some
minimum
> value.
>
> John
>
> On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > Hello John,
> >
> > 1. Based on what you said,
> > e.g: in my example (31 members) if I set vld_thresh=0.5, this will
result
> > in ensemble mean and probability being written into the NetCDF at
any
> grid
> > when 16 or more ensemble members have overlap with obs? I had
thought the
> > function of vld_thresh=0.5 is to throw away all the grids when
half of
> the
> > members have with invalid data.
> >
> >
> > 2 . So there is no way that we can set some thresh to change the
rule
> > of computation
> > of ranked histograms? I mean we must use ALL members for that?
> >
> >
> >
> > Thank you and have a great long weekend.
> >
> > Binyu
> >
> > On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Binyu,
> > >
> > > Correct, setting vld_thresh = 0.5 will result in an ensemble
mean being
> > > computed at any grid point which contains valid data for at
least 50%
> of
> > > the ensemble members.
> > >
> > > MET stores bad data values internally as -9999.
> > >
> > > This usually is NOT a concern because most of the time ensemble
> forecasts
> > > do not include any bata data values. But I haven't worked with
your
> > > ensemble data before so I don't know if/how many bad data values
it
> > > contains.
> > >
> > > John
> > >
> > > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > John,
> > > >
> > > > Currently, "vld_thresh" only sets thresh to INVALID data, is
there a
> > way
> > > > to compute the ensemble mean and probabilities using only part
of
> > > ensemble
> > > > members?
> > > > e.g:
> > > >
> > > > 16 out of 31 have overlap with obs, the other 15 members are
valid
> but
> > no
> > > > overlap.
> > > >
> > > > Does that sound right?
> > > >
> > > > Thank you.
> > > > Binyu
> > > >
> > > >
> > > >
> > > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > So you're saying "ensemble performance" but I need you to be
more
> > > > explicit
> > > > > than that. Do you mean, for example, that that point needs
to be
> > > included
> > > > > in the ranked histogram (RHIST) output line type?
> > > > >
> > > > > Remember that vld_thresh applies in the computation of the
ensemble
> > > mean
> > > > > and probabilities written to the NetCDF output file. It does
not
> > apply
> > > to
> > > > > the computation of ranked histograms.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > Actually the non-signal grid has "-99" in my ensemble
member (the
> > > > > original
> > > > > > data was converted using LogX), and it needs to be anti-
log in
> > order
> > > to
> > > > > > compare with observation, so I use "convert(x) =(10^(x))"
in the
> > > config
> > > > > > file. I guess the converted value (10^(-99)=1E(-99)) is
treated
> as
> > > > valid
> > > > > > data, this is why there is no difference in my results
when I
> > change
> > > > > > "vld_thresh=1" to "vld_thresh=0.5", because there is no
invalid
> > data
> > > in
> > > > > all
> > > > > > the members.
> > > > > >
> > > > > > If my guess above is correct, is there a way to look at
the
> > ensemble
> > > > > > performance when only part of the members (like 20 out of
31) has
> > > > > overlap?
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > When the ensemble member does not predict the ash at a
grid
> > point,
> > > is
> > > > > > that
> > > > > > > stored as a missing data value or some real data value,
like 0?
> > > > > > >
> > > > > > > If it's a bad data value instead of 0, that'll cause
problems.
> > > > > > > Ensemble-Stat only computes an observation rank value
when all
> of
> > > the
> > > > > > > ensemble members contain valid data at that point.
> > > > > > >
> > > > > > > You only getting 3 pairs in the output may mean that
there are
> > > only 3
> > > > > > grid
> > > > > > > points at which all 31 ensemble member contains valid
data at
> > those
> > > > > > points.
> > > > > > >
> > > > > > > If it is bad data, you could switch it to 0, using the
> > > censor_thresh
> > > > > and
> > > > > > > censor_val options:
> > > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > >
> > > > > > > >
> > > > > > > > John,
> > > > > > > >
> > > > > > > > That is very clear. I really appreciate your detailed
> > > explanation.
> > > > I
> > > > > > had
> > > > > > > > misunderstood "missing/invalid value" with "not
overlap"
> > before.
> > > > > > > >
> > > > > > > > Here is what I am doing: I have 31 ensemble inputs,
and I am
> > > using
> > > > > > > > satellite observed volcano ash for model verification.
Based
> on
> > > the
> > > > > > > current
> > > > > > > > results, there are only 3 matching pairs (which means
only 3
> > grid
> > > > > cells
> > > > > > > > over which all 31 members have ash overlap with obs).
So I am
> > > > > > "thinking"
> > > > > > > if
> > > > > > > > there is a way to look at the performance when only
part of
> the
> > > > > members
> > > > > > > > (like 20 out of 31) has overlap? Maybe this thought is
> > > unrealistic.
> > > > > > Below
> > > > > > > > is my current config file:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > > > > >
> > > > > > > >
> > > > > > > > I added "NEP" and "NMEP" in the config file, this may
help
> make
> > > the
> > > > > > > > verification look better, right?
> > > > > > > >
> > > > > > > >
> > > > > > > > Thank you.
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Binyu,
> > > > > > > > >
> > > > > > > > > OK, so assuming you have 10 input ensemble members
and
> > > > ens_thresh =
> > > > > > 0.5
> > > > > > > > and
> > > > > > > > > vld_thresh = 0.5.
> > > > > > > > >
> > > > > > > > > So you pass in 10 input files, 1 for each ensemble
member.
> > If 6
> > > > of
> > > > > > > those
> > > > > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh),
then
> > > > ensemble_stat
> > > > > > > will
> > > > > > > > > error out and produce no output. So this is applied
once
> and
> > > > > > > > ensemble-stat
> > > > > > > > > either will run or error out.
> > > > > > > > >
> > > > > > > > > One of the things that ensemble-stat does is compute
an
> > > ensemble
> > > > > mean
> > > > > > > > > field. At a particular grid point, if 6 of the 10
members
> do
> > > not
> > > > > > have a
> > > > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh), then
the
> > > ensemble
> > > > > > mean
> > > > > > > > > value will be computed as bad data. This applies
when
> > deriving
> > > > > > ensemble
> > > > > > > > > fields that are written to the NetCDF output file.
So this
> is
> > > > > applied
> > > > > > > > once
> > > > > > > > > for each grid point and affects the values written
to the
> > > NetCDF
> > > > > > > output.
> > > > > > > > >
> > > > > > > > > So ens_thresh specifies the required ratio of
ensemble
> > members
> > > > that
> > > > > > > must
> > > > > > > > be
> > > > > > > > > present for ensemble_stat to run at all.
> > > > > > > > > And vld_thresh specifies the required ratio of valid
data
> for
> > > > > output
> > > > > > to
> > > > > > > > be
> > > > > > > > > created at each grid point.
> > > > > > > > >
> > > > > > > > > I get the idea of what you mean by "overlap" with
the obs,
> > but
> > > > the
> > > > > > > > details
> > > > > > > > > matter a lot. When an ensemble member does NOT
overlap with
> > the
> > > > > obs,
> > > > > > is
> > > > > > > > > that encoded as bad data or as some legitimate
value, like
> 0?
> > > > > > > > >
> > > > > > > > > If you have more questions, I'm guessing it'd be
more
> > > informative
> > > > > to
> > > > > > > look
> > > > > > > > > at a specific example.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Wed, May 26, 2021 at 10:04 AM binyu.wang at noaa.gov
via
> RT
> > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hello John,
> > > > > > > > > >
> > > > > > > > > > I am confused about the "missing" ensemble members
and
> data
> > > > > > > > > > definition. For example, there are 10 ensemble
members
> > > > overall.
> > > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > > > > Ensemble member missing means:
> > > > > > > > > > 1. There is only input for 5 members, or
> > > > > > > > > > 2. all 10 member files exist, but only 5 of them
have
> > overlap
> > > > > with
> > > > > > > obs.
> > > > > > > > > > I guess 1 is correct?
> > > > > > > > > >
> > > > > > > > > > The "missing" data refers to
> > > > > > > > > > 3. all 10 members have data, but some data are not
valid
> > > > > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > > > > Is 3 correct?
> > > > > > > > > >
> > > > > > > > > > It seems "missing" does NOT mean "no overlap". If
this is
> > the
> > > > > case,
> > > > > > > is
> > > > > > > > > > there a way to get ensemble statistics if I don't
need
> all
> > > > > ensemble
> > > > > > > > > members
> > > > > > > > > > to overlap with obs? I.e: is it possible to get
the RELY
> > > stat.
> > > > > by
> > > > > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > > > > >
> > > > > > > > > > I hope it makes sense. Thank you.
> > > > > > > > > > Binyu
> > > > > > > > > >
> > > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Binyu,
> > > > > > > > > > >
> > > > > > > > > > > The ens_thresh and vld_thresh options tell
> ensemble_stat
> > > how
> > > > to
> > > > > > > > handle
> > > > > > > > > > > missing data... ens_thresh for missing ensemble
member
> > > inputs
> > > > > and
> > > > > > > > > > > vld_thresh for missing data within the input
ensemble
> > > member
> > > > > > data.
> > > > > > > > They
> > > > > > > > > > are
> > > > > > > > > > > as described in the user's guide:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > > > > >
> > > > > > > > > > > ----------------
> > > > > > > > > > >
> > > > > > > > > > > When summarizing the ensemble, compute a ratio
of the
> > > number
> > > > of
> > > > > > > valid
> > > > > > > > > > > ensemble fields to the total number of ensemble
> members.
> > If
> > > > > this
> > > > > > > > ratio
> > > > > > > > > is
> > > > > > > > > > > less than the *ens_thresh*, then quit with an
error.
> This
> > > > > > threshold
> > > > > > > > > must
> > > > > > > > > > be
> > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> require
> > > > that
> > > > > > all
> > > > > > > > > > ensemble
> > > > > > > > > > > members be present to be processed.
> > > > > > > > > > >
> > > > > > > > > > > When summarizing the ensemble, for each grid
point
> > compute
> > > a
> > > > > > ratio
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > number of valid data values to the number of
ensemble
> > > > members.
> > > > > If
> > > > > > > > that
> > > > > > > > > > > ratio is less than *vld_thresh*, write out bad
data.
> This
> > > > > > threshold
> > > > > > > > > must
> > > > > > > > > > be
> > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> require
> > > > each
> > > > > > grid
> > > > > > > > > point
> > > > > > > > > > > to contain valid data for all ensemble members.
> > > > > > > > > > >
> > > > > > > > > > > ----------------
> > > > > > > > > > >
> > > > > > > > > > > If you have no missing ensemble members and none
have
> any
> > > > > missing
> > > > > > > > data
> > > > > > > > > > > values, then changing these settings would have
no
> impact
> > > on
> > > > > your
> > > > > > > > > > results.
> > > > > > > > > > >
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
binyu.wang at noaa.gov
> via
> > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you, John.
> > > > > > > > > > > >
> > > > > > > > > > > > Looking at the config file, I had another
question:
> > > > > > > > > > > >
> > > > > > > > > > > > fcst = {
> > > > > > > > > > > >
> > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I understand the meaning of ens_thresh. I got
very
> few
> > > > > matching
> > > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to 0.5,
> however,
> > > the
> > > > > > count
> > > > > > > > of
> > > > > > > > > > > number
> > > > > > > > > > > > for ECNT are the same, the number of ensemble
values
> > are
> > > > the
> > > > > > same
> > > > > > > > as
> > > > > > > > > > > well,
> > > > > > > > > > > > they are all 31 (totally there are 31 ensemble
> > members),
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Here is the *stat file:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > > > >
ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Binyu
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley
Gotway
> via
> > > RT <
> > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Binyu,
> > > > > > > > > > > > >
> > > > > > > > > > > > > No that's correct. In ensemble_stat the
cat_thresh
> > > > setting
> > > > > > > > applies
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > "ens" dictionary and defines the thresholds
that to
> > be
> > > > used
> > > > > > to
> > > > > > > in
> > > > > > > > > > > > defining
> > > > > > > > > > > > > probabilities written to the NetCDF output
file.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The cat_thresh option does not apply to the
fcst
> and
> > > obs
> > > > > > > > > > dictionaries.
> > > > > > > > > > > > None
> > > > > > > > > > > > > of the output stat line types produced by
> > ensemble_stat
> > > > > > (listed
> > > > > > > > > > below)
> > > > > > > > > > > > are
> > > > > > > > > > > > > based on the application of a categorical
threshold
> > > > defined
> > > > > > by
> > > > > > > > > > > > cat_thresh.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > output_flag = {
> > > > > > > > > > > > > ecnt = NONE;
> > > > > > > > > > > > > rps = NONE;
> > > > > > > > > > > > > rhist = NONE;
> > > > > > > > > > > > > phist = NONE;
> > > > > > > > > > > > > orank = NONE;
> > > > > > > > > > > > > ssvar = NONE;
> > > > > > > > > > > > > relp = NONE;
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
> binyu.wang at noaa.gov
> > > via
> > > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973
was acted
> > > upon.
> > > > > > > > > > > > > > Transaction: Ticket created by
> binyu.wang at noaa.gov
> > > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > > Subject: config files
> > > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > > Ticket <URL:
> > > > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have two ensemble config files as below
(the
> > other
> > > > > parts
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > config
> > > > > > > > > > > > > > are the same), will they give exactly the
same
> > > output?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The only difference lies in there is no
> > "cat_thresh"
> > > in
> > > > > > fcst
> > > > > > > > and
> > > > > > > > > > obs
> > > > > > > > > > > > > > entries. Based on my tests, it looks like
the
> > > ensemble
> > > > > > > results
> > > > > > > > > are
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > with the two config files below, but I am
afraid
> I
> > > may
> > > > > miss
> > > > > > > > some
> > > > > > > > > > > output
> > > > > > > > > > > > > > flags.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > };
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > };
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
---------------------------------------------
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > };
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>=0.1,>=0.3,
> > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > };
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So if they will lead to different outputs,
what
> > will
> > > be
> > > > > the
> > > > > > > > > > > difference?
> > > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Binyu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Tue Jun 01 16:19:51 2021
Binyu,
I feel like we're talking in circles. And doing so without crystal
clear
examples in data is not useful.
Yes, you can use censor_thresh and censor_val to replace good data
values
with bad data values (-9999).
Or you can use them to replace bad data values (-9999) with good data
values.
It's up to you.
Whether it'll have the exact desired effect you seek, I cannot say
with
confidence.
I'd recommend that you run the plot_data_plane tool on your data to
test
out the censor_thresh and censor_val settings you'd like to use. That
way
you can see the effect they have.
And then test the ensemble_stat tool to see if your configuration has
the
desired effect.
If you have specific questions, please send me the command you ran,
the
path to the output file, and your question(s). And I'll do my best to
answer them.
John
On Tue, Jun 1, 2021 at 3:03 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> Hello John,
>
> We can use "censor_thresh" and "censor_val" logic to replace the
missing
> data value with some minimum
> value, but how about the opposite way?
> E.g: the non-signal grid has value 10^(-99)=1E(-99) in my forecast,
so they
> are all valid data. What if I want to lower the standard vld_thresh
from 1
> to 0.6? Can I set the non-signal grid to invalid data? I mean I
don't need
> all members to overlap. Does that sound right?
>
> Thank you.
> Binyu
>
> On Fri, May 28, 2021 at 5:05 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > There is no way to change the computation of the ranked histogram
logic.
> If
> > we let the number of valid ensemble members vary from grid point
to grid
> > point, then the resulting observation rank values cannot be
combined
> into a
> > ranked histogram.
> >
> > For example, let's say you compute the observation rank as...
> > - At grid point 1, obs rank is 5 out of 10 valid ensemble member
values.
> > - At grid point 2, obs rank is 5 out of 15 valid ensemble member
values.
> > - At grid point 3, obs rank is 5 out of 20 valid ensemble member
values.
> >
> > While all 3 points have the same rank of 5, they cannot be
combined into
> a
> > rank histogram because of the varying number of valid ensemble
member
> > values.
> >
> > But again this only matters if your ensemble forecasts contain
missing
> data
> > values. If they do, you could always use the "censor_thresh" and
> > "censor_val" logic to replace the missing data value with some
minimum
> > value.
> >
> > John
> >
> > On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > >
> > > Hello John,
> > >
> > > 1. Based on what you said,
> > > e.g: in my example (31 members) if I set vld_thresh=0.5, this
will
> result
> > > in ensemble mean and probability being written into the NetCDF
at any
> > grid
> > > when 16 or more ensemble members have overlap with obs? I had
thought
> the
> > > function of vld_thresh=0.5 is to throw away all the grids when
half of
> > the
> > > members have with invalid data.
> > >
> > >
> > > 2 . So there is no way that we can set some thresh to change
the rule
> > > of computation
> > > of ranked histograms? I mean we must use ALL members for that?
> > >
> > >
> > >
> > > Thank you and have a great long weekend.
> > >
> > > Binyu
> > >
> > > On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Binyu,
> > > >
> > > > Correct, setting vld_thresh = 0.5 will result in an ensemble
mean
> being
> > > > computed at any grid point which contains valid data for at
least 50%
> > of
> > > > the ensemble members.
> > > >
> > > > MET stores bad data values internally as -9999.
> > > >
> > > > This usually is NOT a concern because most of the time
ensemble
> > forecasts
> > > > do not include any bata data values. But I haven't worked with
your
> > > > ensemble data before so I don't know if/how many bad data
values it
> > > > contains.
> > > >
> > > > John
> > > >
> > > > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > >
> > > > > John,
> > > > >
> > > > > Currently, "vld_thresh" only sets thresh to INVALID data,
is
> there a
> > > way
> > > > > to compute the ensemble mean and probabilities using only
part of
> > > > ensemble
> > > > > members?
> > > > > e.g:
> > > > >
> > > > > 16 out of 31 have overlap with obs, the other 15 members are
valid
> > but
> > > no
> > > > > overlap.
> > > > >
> > > > > Does that sound right?
> > > > >
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > >
> > > > >
> > > > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > So you're saying "ensemble performance" but I need you to
be more
> > > > > explicit
> > > > > > than that. Do you mean, for example, that that point needs
to be
> > > > included
> > > > > > in the ranked histogram (RHIST) output line type?
> > > > > >
> > > > > > Remember that vld_thresh applies in the computation of the
> ensemble
> > > > mean
> > > > > > and probabilities written to the NetCDF output file. It
does not
> > > apply
> > > > to
> > > > > > the computation of ranked histograms.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> > > > > > >
> > > > > > > John,
> > > > > > >
> > > > > > > Actually the non-signal grid has "-99" in my ensemble
member
> (the
> > > > > > original
> > > > > > > data was converted using LogX), and it needs to be anti-
log in
> > > order
> > > > to
> > > > > > > compare with observation, so I use "convert(x)
=(10^(x))" in
> the
> > > > config
> > > > > > > file. I guess the converted value (10^(-99)=1E(-99)) is
> treated
> > as
> > > > > valid
> > > > > > > data, this is why there is no difference in my results
when I
> > > change
> > > > > > > "vld_thresh=1" to "vld_thresh=0.5", because there is no
invalid
> > > data
> > > > in
> > > > > > all
> > > > > > > the members.
> > > > > > >
> > > > > > > If my guess above is correct, is there a way to look at
the
> > > ensemble
> > > > > > > performance when only part of the members (like 20 out
of 31)
> has
> > > > > > overlap?
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Binyu,
> > > > > > > >
> > > > > > > > When the ensemble member does not predict the ash at a
grid
> > > point,
> > > > is
> > > > > > > that
> > > > > > > > stored as a missing data value or some real data
value, like
> 0?
> > > > > > > >
> > > > > > > > If it's a bad data value instead of 0, that'll cause
> problems.
> > > > > > > > Ensemble-Stat only computes an observation rank value
when
> all
> > of
> > > > the
> > > > > > > > ensemble members contain valid data at that point.
> > > > > > > >
> > > > > > > > You only getting 3 pairs in the output may mean that
there
> are
> > > > only 3
> > > > > > > grid
> > > > > > > > points at which all 31 ensemble member contains valid
data at
> > > those
> > > > > > > points.
> > > > > > > >
> > > > > > > > If it is bad data, you could switch it to 0, using the
> > > > censor_thresh
> > > > > > and
> > > > > > > > censor_val options:
> > > > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > >
> > > > > > > > >
> > > > > > > > > John,
> > > > > > > > >
> > > > > > > > > That is very clear. I really appreciate your
detailed
> > > > explanation.
> > > > > I
> > > > > > > had
> > > > > > > > > misunderstood "missing/invalid value" with "not
overlap"
> > > before.
> > > > > > > > >
> > > > > > > > > Here is what I am doing: I have 31 ensemble inputs,
and I
> am
> > > > using
> > > > > > > > > satellite observed volcano ash for model
verification.
> Based
> > on
> > > > the
> > > > > > > > current
> > > > > > > > > results, there are only 3 matching pairs (which
means only
> 3
> > > grid
> > > > > > cells
> > > > > > > > > over which all 31 members have ash overlap with
obs). So I
> am
> > > > > > > "thinking"
> > > > > > > > if
> > > > > > > > > there is a way to look at the performance when only
part of
> > the
> > > > > > members
> > > > > > > > > (like 20 out of 31) has overlap? Maybe this thought
is
> > > > unrealistic.
> > > > > > > Below
> > > > > > > > > is my current config file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I added "NEP" and "NMEP" in the config file, this
may help
> > make
> > > > the
> > > > > > > > > verification look better, right?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thank you.
> > > > > > > > >
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Binyu,
> > > > > > > > > >
> > > > > > > > > > OK, so assuming you have 10 input ensemble members
and
> > > > > ens_thresh =
> > > > > > > 0.5
> > > > > > > > > and
> > > > > > > > > > vld_thresh = 0.5.
> > > > > > > > > >
> > > > > > > > > > So you pass in 10 input files, 1 for each ensemble
> member.
> > > If 6
> > > > > of
> > > > > > > > those
> > > > > > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh),
then
> > > > > ensemble_stat
> > > > > > > > will
> > > > > > > > > > error out and produce no output. So this is
applied once
> > and
> > > > > > > > > ensemble-stat
> > > > > > > > > > either will run or error out.
> > > > > > > > > >
> > > > > > > > > > One of the things that ensemble-stat does is
compute an
> > > > ensemble
> > > > > > mean
> > > > > > > > > > field. At a particular grid point, if 6 of the 10
members
> > do
> > > > not
> > > > > > > have a
> > > > > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh),
then the
> > > > ensemble
> > > > > > > mean
> > > > > > > > > > value will be computed as bad data. This applies
when
> > > deriving
> > > > > > > ensemble
> > > > > > > > > > fields that are written to the NetCDF output file.
So
> this
> > is
> > > > > > applied
> > > > > > > > > once
> > > > > > > > > > for each grid point and affects the values written
to the
> > > > NetCDF
> > > > > > > > output.
> > > > > > > > > >
> > > > > > > > > > So ens_thresh specifies the required ratio of
ensemble
> > > members
> > > > > that
> > > > > > > > must
> > > > > > > > > be
> > > > > > > > > > present for ensemble_stat to run at all.
> > > > > > > > > > And vld_thresh specifies the required ratio of
valid data
> > for
> > > > > > output
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > created at each grid point.
> > > > > > > > > >
> > > > > > > > > > I get the idea of what you mean by "overlap" with
the
> obs,
> > > but
> > > > > the
> > > > > > > > > details
> > > > > > > > > > matter a lot. When an ensemble member does NOT
overlap
> with
> > > the
> > > > > > obs,
> > > > > > > is
> > > > > > > > > > that encoded as bad data or as some legitimate
value,
> like
> > 0?
> > > > > > > > > >
> > > > > > > > > > If you have more questions, I'm guessing it'd be
more
> > > > informative
> > > > > > to
> > > > > > > > look
> > > > > > > > > > at a specific example.
> > > > > > > > > >
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Wed, May 26, 2021 at 10:04 AM
binyu.wang at noaa.gov via
> > RT
> > > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hello John,
> > > > > > > > > > >
> > > > > > > > > > > I am confused about the "missing" ensemble
members and
> > data
> > > > > > > > > > > definition. For example, there are 10 ensemble
> members
> > > > > overall.
> > > > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > > > > > Ensemble member missing means:
> > > > > > > > > > > 1. There is only input for 5 members, or
> > > > > > > > > > > 2. all 10 member files exist, but only 5 of them
have
> > > overlap
> > > > > > with
> > > > > > > > obs.
> > > > > > > > > > > I guess 1 is correct?
> > > > > > > > > > >
> > > > > > > > > > > The "missing" data refers to
> > > > > > > > > > > 3. all 10 members have data, but some data are
not
> valid
> > > > > > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > > > > > Is 3 correct?
> > > > > > > > > > >
> > > > > > > > > > > It seems "missing" does NOT mean "no overlap".
If this
> is
> > > the
> > > > > > case,
> > > > > > > > is
> > > > > > > > > > > there a way to get ensemble statistics if I
don't need
> > all
> > > > > > ensemble
> > > > > > > > > > members
> > > > > > > > > > > to overlap with obs? I.e: is it possible to get
the
> RELY
> > > > stat.
> > > > > > by
> > > > > > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > > > > > >
> > > > > > > > > > > I hope it makes sense. Thank you.
> > > > > > > > > > > Binyu
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Binyu,
> > > > > > > > > > > >
> > > > > > > > > > > > The ens_thresh and vld_thresh options tell
> > ensemble_stat
> > > > how
> > > > > to
> > > > > > > > > handle
> > > > > > > > > > > > missing data... ens_thresh for missing
ensemble
> member
> > > > inputs
> > > > > > and
> > > > > > > > > > > > vld_thresh for missing data within the input
ensemble
> > > > member
> > > > > > > data.
> > > > > > > > > They
> > > > > > > > > > > are
> > > > > > > > > > > > as described in the user's guide:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > > > > > >
> > > > > > > > > > > > ----------------
> > > > > > > > > > > >
> > > > > > > > > > > > When summarizing the ensemble, compute a ratio
of the
> > > > number
> > > > > of
> > > > > > > > valid
> > > > > > > > > > > > ensemble fields to the total number of
ensemble
> > members.
> > > If
> > > > > > this
> > > > > > > > > ratio
> > > > > > > > > > is
> > > > > > > > > > > > less than the *ens_thresh*, then quit with an
error.
> > This
> > > > > > > threshold
> > > > > > > > > > must
> > > > > > > > > > > be
> > > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> > require
> > > > > that
> > > > > > > all
> > > > > > > > > > > ensemble
> > > > > > > > > > > > members be present to be processed.
> > > > > > > > > > > >
> > > > > > > > > > > > When summarizing the ensemble, for each grid
point
> > > compute
> > > > a
> > > > > > > ratio
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > number of valid data values to the number of
ensemble
> > > > > members.
> > > > > > If
> > > > > > > > > that
> > > > > > > > > > > > ratio is less than *vld_thresh*, write out bad
data.
> > This
> > > > > > > threshold
> > > > > > > > > > must
> > > > > > > > > > > be
> > > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> > require
> > > > > each
> > > > > > > grid
> > > > > > > > > > point
> > > > > > > > > > > > to contain valid data for all ensemble
members.
> > > > > > > > > > > >
> > > > > > > > > > > > ----------------
> > > > > > > > > > > >
> > > > > > > > > > > > If you have no missing ensemble members and
none have
> > any
> > > > > > missing
> > > > > > > > > data
> > > > > > > > > > > > values, then changing these settings would
have no
> > impact
> > > > on
> > > > > > your
> > > > > > > > > > > results.
> > > > > > > > > > > >
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
binyu.wang at noaa.gov
> > via
> > > > RT <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you, John.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking at the config file, I had another
question:
> > > > > > > > > > > > >
> > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > >
> > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I understand the meaning of ens_thresh. I
got very
> > few
> > > > > > matching
> > > > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to
0.5,
> > however,
> > > > the
> > > > > > > count
> > > > > > > > > of
> > > > > > > > > > > > number
> > > > > > > > > > > > > for ECNT are the same, the number of
ensemble
> values
> > > are
> > > > > the
> > > > > > > same
> > > > > > > > > as
> > > > > > > > > > > > well,
> > > > > > > > > > > > > they are all 31 (totally there are 31
ensemble
> > > members),
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here is the *stat file:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > > > > >
ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Binyu
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John Halley
Gotway
> > via
> > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Binyu,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > No that's correct. In ensemble_stat the
> cat_thresh
> > > > > setting
> > > > > > > > > applies
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > "ens" dictionary and defines the
thresholds that
> to
> > > be
> > > > > used
> > > > > > > to
> > > > > > > > in
> > > > > > > > > > > > > defining
> > > > > > > > > > > > > > probabilities written to the NetCDF output
file.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The cat_thresh option does not apply to
the fcst
> > and
> > > > obs
> > > > > > > > > > > dictionaries.
> > > > > > > > > > > > > None
> > > > > > > > > > > > > > of the output stat line types produced by
> > > ensemble_stat
> > > > > > > (listed
> > > > > > > > > > > below)
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > based on the application of a categorical
> threshold
> > > > > defined
> > > > > > > by
> > > > > > > > > > > > > cat_thresh.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > John
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > output_flag = {
> > > > > > > > > > > > > > ecnt = NONE;
> > > > > > > > > > > > > > rps = NONE;
> > > > > > > > > > > > > > rhist = NONE;
> > > > > > > > > > > > > > phist = NONE;
> > > > > > > > > > > > > > orank = NONE;
> > > > > > > > > > > > > > ssvar = NONE;
> > > > > > > > > > > > > > relp = NONE;
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
> > binyu.wang at noaa.gov
> > > > via
> > > > > > RT <
> > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request 99973
was
> acted
> > > > upon.
> > > > > > > > > > > > > > > Transaction: Ticket created by
> > binyu.wang at noaa.gov
> > > > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > > > Subject: config files
> > > > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > > > Ticket <URL:
> > > > > > > > > > > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have two ensemble config files as
below (the
> > > other
> > > > > > parts
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > are the same), will they give exactly
the same
> > > > output?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The only difference lies in there is no
> > > "cat_thresh"
> > > > in
> > > > > > > fcst
> > > > > > > > > and
> > > > > > > > > > > obs
> > > > > > > > > > > > > > > entries. Based on my tests, it looks
like the
> > > > ensemble
> > > > > > > > results
> > > > > > > > > > are
> > > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > with the two config files below, but I
am
> afraid
> > I
> > > > may
> > > > > > miss
> > > > > > > > > some
> > > > > > > > > > > > output
> > > > > > > > > > > > > > > flags.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
---------------------------------------------
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > // cat_thresh = [>=0.01, >=0.05,
> > > >=0.1,>=0.3,
> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> >=0.1,>=0.3,
> > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > // cat_thresh = [ >=0.01,>=0.05,
> > > >=0.1,>=0.3,
> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So if they will lead to different
outputs, what
> > > will
> > > > be
> > > > > > the
> > > > > > > > > > > > difference?
> > > > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Binyu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Thu Jun 03 08:30:47 2021
John,
Thank you for your suggestions and sorry for taking so much of your
time.
I recalled that you know a command that can easily convert a text file
(which is obtained using ncdump *.nc) back to NetCDF format, I forgot
what
that is (my old computer is broken and I lost all the records).
eg.
$ ncdump test.nc > test.nc.txt
I can modify the header file of "test.nc.txt", and then convert the
txt
file back to NetCDF format.
Thank you.
Binyu
On Tue, Jun 1, 2021 at 6:19 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I feel like we're talking in circles. And doing so without crystal
clear
> examples in data is not useful.
>
> Yes, you can use censor_thresh and censor_val to replace good data
values
> with bad data values (-9999).
> Or you can use them to replace bad data values (-9999) with good
data
> values.
> It's up to you.
>
> Whether it'll have the exact desired effect you seek, I cannot say
with
> confidence.
>
> I'd recommend that you run the plot_data_plane tool on your data to
test
> out the censor_thresh and censor_val settings you'd like to use.
That way
> you can see the effect they have.
> And then test the ensemble_stat tool to see if your configuration
has the
> desired effect.
>
> If you have specific questions, please send me the command you ran,
the
> path to the output file, and your question(s). And I'll do my best
to
> answer them.
>
> John
>
> On Tue, Jun 1, 2021 at 3:03 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >
> > Hello John,
> >
> > We can use "censor_thresh" and "censor_val" logic to replace the
missing
> > data value with some minimum
> > value, but how about the opposite way?
> > E.g: the non-signal grid has value 10^(-99)=1E(-99) in my
forecast, so
> they
> > are all valid data. What if I want to lower the standard
vld_thresh
> from 1
> > to 0.6? Can I set the non-signal grid to invalid data? I mean I
don't
> need
> > all members to overlap. Does that sound right?
> >
> > Thank you.
> > Binyu
> >
> > On Fri, May 28, 2021 at 5:05 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > There is no way to change the computation of the ranked
histogram
> logic.
> > If
> > > we let the number of valid ensemble members vary from grid point
to
> grid
> > > point, then the resulting observation rank values cannot be
combined
> > into a
> > > ranked histogram.
> > >
> > > For example, let's say you compute the observation rank as...
> > > - At grid point 1, obs rank is 5 out of 10 valid ensemble member
> values.
> > > - At grid point 2, obs rank is 5 out of 15 valid ensemble member
> values.
> > > - At grid point 3, obs rank is 5 out of 20 valid ensemble member
> values.
> > >
> > > While all 3 points have the same rank of 5, they cannot be
combined
> into
> > a
> > > rank histogram because of the varying number of valid ensemble
member
> > > values.
> > >
> > > But again this only matters if your ensemble forecasts contain
missing
> > data
> > > values. If they do, you could always use the "censor_thresh" and
> > > "censor_val" logic to replace the missing data value with some
minimum
> > > value.
> > >
> > > John
> > >
> > > On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
> > > >
> > > > Hello John,
> > > >
> > > > 1. Based on what you said,
> > > > e.g: in my example (31 members) if I set vld_thresh=0.5, this
will
> > result
> > > > in ensemble mean and probability being written into the NetCDF
at
> any
> > > grid
> > > > when 16 or more ensemble members have overlap with obs? I had
thought
> > the
> > > > function of vld_thresh=0.5 is to throw away all the grids when
half
> of
> > > the
> > > > members have with invalid data.
> > > >
> > > >
> > > > 2 . So there is no way that we can set some thresh to change
the
> rule
> > > > of computation
> > > > of ranked histograms? I mean we must use ALL members for that?
> > > >
> > > >
> > > >
> > > > Thank you and have a great long weekend.
> > > >
> > > > Binyu
> > > >
> > > > On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > Correct, setting vld_thresh = 0.5 will result in an ensemble
mean
> > being
> > > > > computed at any grid point which contains valid data for at
least
> 50%
> > > of
> > > > > the ensemble members.
> > > > >
> > > > > MET stores bad data values internally as -9999.
> > > > >
> > > > > This usually is NOT a concern because most of the time
ensemble
> > > forecasts
> > > > > do not include any bata data values. But I haven't worked
with your
> > > > > ensemble data before so I don't know if/how many bad data
values it
> > > > > contains.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > Currently, "vld_thresh" only sets thresh to INVALID data,
is
> > there a
> > > > way
> > > > > > to compute the ensemble mean and probabilities using only
part of
> > > > > ensemble
> > > > > > members?
> > > > > > e.g:
> > > > > >
> > > > > > 16 out of 31 have overlap with obs, the other 15 members
are
> valid
> > > but
> > > > no
> > > > > > overlap.
> > > > > >
> > > > > > Does that sound right?
> > > > > >
> > > > > > Thank you.
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > So you're saying "ensemble performance" but I need you
to be
> more
> > > > > > explicit
> > > > > > > than that. Do you mean, for example, that that point
needs to
> be
> > > > > included
> > > > > > > in the ranked histogram (RHIST) output line type?
> > > > > > >
> > > > > > > Remember that vld_thresh applies in the computation of
the
> > ensemble
> > > > > mean
> > > > > > > and probabilities written to the NetCDF output file. It
does
> not
> > > > apply
> > > > > to
> > > > > > > the computation of ranked histograms.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > >
> > > > > > > >
> > > > > > > > John,
> > > > > > > >
> > > > > > > > Actually the non-signal grid has "-99" in my ensemble
member
> > (the
> > > > > > > original
> > > > > > > > data was converted using LogX), and it needs to be
anti-log
> in
> > > > order
> > > > > to
> > > > > > > > compare with observation, so I use "convert(x)
=(10^(x))" in
> > the
> > > > > config
> > > > > > > > file. I guess the converted value (10^(-99)=1E(-99))
is
> > treated
> > > as
> > > > > > valid
> > > > > > > > data, this is why there is no difference in my results
when I
> > > > change
> > > > > > > > "vld_thresh=1" to "vld_thresh=0.5", because there is
no
> invalid
> > > > data
> > > > > in
> > > > > > > all
> > > > > > > > the members.
> > > > > > > >
> > > > > > > > If my guess above is correct, is there a way to look
at the
> > > > ensemble
> > > > > > > > performance when only part of the members (like 20 out
of 31)
> > has
> > > > > > > overlap?
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Binyu,
> > > > > > > > >
> > > > > > > > > When the ensemble member does not predict the ash at
a grid
> > > > point,
> > > > > is
> > > > > > > > that
> > > > > > > > > stored as a missing data value or some real data
value,
> like
> > 0?
> > > > > > > > >
> > > > > > > > > If it's a bad data value instead of 0, that'll cause
> > problems.
> > > > > > > > > Ensemble-Stat only computes an observation rank
value when
> > all
> > > of
> > > > > the
> > > > > > > > > ensemble members contain valid data at that point.
> > > > > > > > >
> > > > > > > > > You only getting 3 pairs in the output may mean that
there
> > are
> > > > > only 3
> > > > > > > > grid
> > > > > > > > > points at which all 31 ensemble member contains
valid data
> at
> > > > those
> > > > > > > > points.
> > > > > > > > >
> > > > > > > > > If it is bad data, you could switch it to 0, using
the
> > > > > censor_thresh
> > > > > > > and
> > > > > > > > > censor_val options:
> > > > > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov
via
> RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > >
> > > > > > > > > >
> > > > > > > > > > John,
> > > > > > > > > >
> > > > > > > > > > That is very clear. I really appreciate your
detailed
> > > > > explanation.
> > > > > > I
> > > > > > > > had
> > > > > > > > > > misunderstood "missing/invalid value" with "not
overlap"
> > > > before.
> > > > > > > > > >
> > > > > > > > > > Here is what I am doing: I have 31 ensemble
inputs, and I
> > am
> > > > > using
> > > > > > > > > > satellite observed volcano ash for model
verification.
> > Based
> > > on
> > > > > the
> > > > > > > > > current
> > > > > > > > > > results, there are only 3 matching pairs (which
means
> only
> > 3
> > > > grid
> > > > > > > cells
> > > > > > > > > > over which all 31 members have ash overlap with
obs). So
> I
> > am
> > > > > > > > "thinking"
> > > > > > > > > if
> > > > > > > > > > there is a way to look at the performance when
only part
> of
> > > the
> > > > > > > members
> > > > > > > > > > (like 20 out of 31) has overlap? Maybe this
thought is
> > > > > unrealistic.
> > > > > > > > Below
> > > > > > > > > > is my current config file:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I added "NEP" and "NMEP" in the config file, this
may
> help
> > > make
> > > > > the
> > > > > > > > > > verification look better, right?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thank you.
> > > > > > > > > >
> > > > > > > > > > Binyu
> > > > > > > > > >
> > > > > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Binyu,
> > > > > > > > > > >
> > > > > > > > > > > OK, so assuming you have 10 input ensemble
members and
> > > > > > ens_thresh =
> > > > > > > > 0.5
> > > > > > > > > > and
> > > > > > > > > > > vld_thresh = 0.5.
> > > > > > > > > > >
> > > > > > > > > > > So you pass in 10 input files, 1 for each
ensemble
> > member.
> > > > If 6
> > > > > > of
> > > > > > > > > those
> > > > > > > > > > > files don't exist (6/10 = 0.6 > 0.5 ens_thresh),
then
> > > > > > ensemble_stat
> > > > > > > > > will
> > > > > > > > > > > error out and produce no output. So this is
applied
> once
> > > and
> > > > > > > > > > ensemble-stat
> > > > > > > > > > > either will run or error out.
> > > > > > > > > > >
> > > > > > > > > > > One of the things that ensemble-stat does is
compute an
> > > > > ensemble
> > > > > > > mean
> > > > > > > > > > > field. At a particular grid point, if 6 of the
10
> members
> > > do
> > > > > not
> > > > > > > > have a
> > > > > > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh),
then
> the
> > > > > ensemble
> > > > > > > > mean
> > > > > > > > > > > value will be computed as bad data. This applies
when
> > > > deriving
> > > > > > > > ensemble
> > > > > > > > > > > fields that are written to the NetCDF output
file. So
> > this
> > > is
> > > > > > > applied
> > > > > > > > > > once
> > > > > > > > > > > for each grid point and affects the values
written to
> the
> > > > > NetCDF
> > > > > > > > > output.
> > > > > > > > > > >
> > > > > > > > > > > So ens_thresh specifies the required ratio of
ensemble
> > > > members
> > > > > > that
> > > > > > > > > must
> > > > > > > > > > be
> > > > > > > > > > > present for ensemble_stat to run at all.
> > > > > > > > > > > And vld_thresh specifies the required ratio of
valid
> data
> > > for
> > > > > > > output
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > created at each grid point.
> > > > > > > > > > >
> > > > > > > > > > > I get the idea of what you mean by "overlap"
with the
> > obs,
> > > > but
> > > > > > the
> > > > > > > > > > details
> > > > > > > > > > > matter a lot. When an ensemble member does NOT
overlap
> > with
> > > > the
> > > > > > > obs,
> > > > > > > > is
> > > > > > > > > > > that encoded as bad data or as some legitimate
value,
> > like
> > > 0?
> > > > > > > > > > >
> > > > > > > > > > > If you have more questions, I'm guessing it'd be
more
> > > > > informative
> > > > > > > to
> > > > > > > > > look
> > > > > > > > > > > at a specific example.
> > > > > > > > > > >
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 26, 2021 at 10:04 AM
binyu.wang at noaa.gov
> via
> > > RT
> > > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hello John,
> > > > > > > > > > > >
> > > > > > > > > > > > I am confused about the "missing" ensemble
members
> and
> > > data
> > > > > > > > > > > > definition. For example, there are 10
ensemble
> > members
> > > > > > overall.
> > > > > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> > > > > > > > > > > > Ensemble member missing means:
> > > > > > > > > > > > 1. There is only input for 5 members, or
> > > > > > > > > > > > 2. all 10 member files exist, but only 5 of
them have
> > > > overlap
> > > > > > > with
> > > > > > > > > obs.
> > > > > > > > > > > > I guess 1 is correct?
> > > > > > > > > > > >
> > > > > > > > > > > > The "missing" data refers to
> > > > > > > > > > > > 3. all 10 members have data, but some data are
not
> > valid
> > > > > > > > > > > > 4. all 10 forecasts have data, but no overlap?
> > > > > > > > > > > > Is 3 correct?
> > > > > > > > > > > >
> > > > > > > > > > > > It seems "missing" does NOT mean "no overlap".
If
> this
> > is
> > > > the
> > > > > > > case,
> > > > > > > > > is
> > > > > > > > > > > > there a way to get ensemble statistics if I
don't
> need
> > > all
> > > > > > > ensemble
> > > > > > > > > > > members
> > > > > > > > > > > > to overlap with obs? I.e: is it possible to
get the
> > RELY
> > > > > stat.
> > > > > > > by
> > > > > > > > > > > > setting more than 5 has overlap instead of 10?
> > > > > > > > > > > >
> > > > > > > > > > > > I hope it makes sense. Thank you.
> > > > > > > > > > > > Binyu
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Binyu,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The ens_thresh and vld_thresh options tell
> > > ensemble_stat
> > > > > how
> > > > > > to
> > > > > > > > > > handle
> > > > > > > > > > > > > missing data... ens_thresh for missing
ensemble
> > member
> > > > > inputs
> > > > > > > and
> > > > > > > > > > > > > vld_thresh for missing data within the input
> ensemble
> > > > > member
> > > > > > > > data.
> > > > > > > > > > They
> > > > > > > > > > > > are
> > > > > > > > > > > > > as described in the user's guide:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> > > > > > > > > > > > >
> > > > > > > > > > > > > ----------------
> > > > > > > > > > > > >
> > > > > > > > > > > > > When summarizing the ensemble, compute a
ratio of
> the
> > > > > number
> > > > > > of
> > > > > > > > > valid
> > > > > > > > > > > > > ensemble fields to the total number of
ensemble
> > > members.
> > > > If
> > > > > > > this
> > > > > > > > > > ratio
> > > > > > > > > > > is
> > > > > > > > > > > > > less than the *ens_thresh*, then quit with
an
> error.
> > > This
> > > > > > > > threshold
> > > > > > > > > > > must
> > > > > > > > > > > > be
> > > > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> > > require
> > > > > > that
> > > > > > > > all
> > > > > > > > > > > > ensemble
> > > > > > > > > > > > > members be present to be processed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > When summarizing the ensemble, for each grid
point
> > > > compute
> > > > > a
> > > > > > > > ratio
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > number of valid data values to the number of
> ensemble
> > > > > > members.
> > > > > > > If
> > > > > > > > > > that
> > > > > > > > > > > > > ratio is less than *vld_thresh*, write out
bad
> data.
> > > This
> > > > > > > > threshold
> > > > > > > > > > > must
> > > > > > > > > > > > be
> > > > > > > > > > > > > between 0 and 1. Setting this threshold to 1
will
> > > require
> > > > > > each
> > > > > > > > grid
> > > > > > > > > > > point
> > > > > > > > > > > > > to contain valid data for all ensemble
members.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ----------------
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you have no missing ensemble members and
none
> have
> > > any
> > > > > > > missing
> > > > > > > > > > data
> > > > > > > > > > > > > values, then changing these settings would
have no
> > > impact
> > > > > on
> > > > > > > your
> > > > > > > > > > > > results.
> > > > > > > > > > > > >
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
> binyu.wang at noaa.gov
> > > via
> > > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you, John.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looking at the config file, I had another
> question:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I understand the meaning of ens_thresh. I
got
> very
> > > few
> > > > > > > matching
> > > > > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to
0.5,
> > > however,
> > > > > the
> > > > > > > > count
> > > > > > > > > > of
> > > > > > > > > > > > > number
> > > > > > > > > > > > > > for ECNT are the same, the number of
ensemble
> > values
> > > > are
> > > > > > the
> > > > > > > > same
> > > > > > > > > > as
> > > > > > > > > > > > > well,
> > > > > > > > > > > > > > they are all 31 (totally there are 31
ensemble
> > > > members),
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here is the *stat file:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> > > > > > > > > > > > > >
ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any thing wrong here? Thank you.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Binyu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John
Halley
> Gotway
> > > via
> > > > > RT <
> > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Binyu,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > No that's correct. In ensemble_stat the
> > cat_thresh
> > > > > > setting
> > > > > > > > > > applies
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > "ens" dictionary and defines the
thresholds
> that
> > to
> > > > be
> > > > > > used
> > > > > > > > to
> > > > > > > > > in
> > > > > > > > > > > > > > defining
> > > > > > > > > > > > > > > probabilities written to the NetCDF
output
> file.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The cat_thresh option does not apply to
the
> fcst
> > > and
> > > > > obs
> > > > > > > > > > > > dictionaries.
> > > > > > > > > > > > > > None
> > > > > > > > > > > > > > > of the output stat line types produced
by
> > > > ensemble_stat
> > > > > > > > (listed
> > > > > > > > > > > > below)
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > based on the application of a
categorical
> > threshold
> > > > > > defined
> > > > > > > > by
> > > > > > > > > > > > > > cat_thresh.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > output_flag = {
> > > > > > > > > > > > > > > ecnt = NONE;
> > > > > > > > > > > > > > > rps = NONE;
> > > > > > > > > > > > > > > rhist = NONE;
> > > > > > > > > > > > > > > phist = NONE;
> > > > > > > > > > > > > > > orank = NONE;
> > > > > > > > > > > > > > > ssvar = NONE;
> > > > > > > > > > > > > > > relp = NONE;
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
> > > binyu.wang at noaa.gov
> > > > > via
> > > > > > > RT <
> > > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request
99973 was
> > acted
> > > > > upon.
> > > > > > > > > > > > > > > > Transaction: Ticket created by
> > > binyu.wang at noaa.gov
> > > > > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > > > > Subject: config files
> > > > > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > > > > Ticket <URL:
> > > > > > > > > > > > >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have two ensemble config files as
below
> (the
> > > > other
> > > > > > > parts
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > are the same), will they give exactly
the
> same
> > > > > output?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The only difference lies in there is
no
> > > > "cat_thresh"
> > > > > in
> > > > > > > > fcst
> > > > > > > > > > and
> > > > > > > > > > > > obs
> > > > > > > > > > > > > > > > entries. Based on my tests, it looks
like the
> > > > > ensemble
> > > > > > > > > results
> > > > > > > > > > > are
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > with the two config files below, but I
am
> > afraid
> > > I
> > > > > may
> > > > > > > miss
> > > > > > > > > > some
> > > > > > > > > > > > > output
> > > > > > > > > > > > > > > > flags.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> > > >=0.1,>=0.3,
> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
> > > >=0.1,>=0.3,
> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
---------------------------------------------
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > fcst = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ens_thresh = 1.0;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > // cat_thresh = [>=0.01,
>=0.05,
> > > > >=0.1,>=0.3,
> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ens = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "VAFTD";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> > >=0.1,>=0.3,
> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > };
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > obs = {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > field = [
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > name = "ash_mass_loading";
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > level = ["(*,*)"];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > // cat_thresh = [
>=0.01,>=0.05,
> > > > >=0.1,>=0.3,
> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ];
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So if they will lead to different
outputs,
> what
> > > > will
> > > > > be
> > > > > > > the
> > > > > > > > > > > > > difference?
> > > > > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Binyu
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: config files
From: binyu.wang at noaa.gov
Time: Thu Jun 03 08:47:52 2021
I think I found it, is it "ncgen"? Thank you.
Binyu
On Thu, Jun 3, 2021 at 10:30 AM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:
> John,
>
> Thank you for your suggestions and sorry for taking so much of your
time.
>
> I recalled that you know a command that can easily convert a text
file
> (which is obtained using ncdump *.nc) back to NetCDF format, I
forgot what
> that is (my old computer is broken and I lost all the records).
>
> eg.
> $ ncdump test.nc > test.nc.txt
> I can modify the header file of "test.nc.txt", and then convert the
txt
> file back to NetCDF format.
>
> Thank you.
> Binyu
>
> On Tue, Jun 1, 2021 at 6:19 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binyu,
>>
>> I feel like we're talking in circles. And doing so without crystal
clear
>> examples in data is not useful.
>>
>> Yes, you can use censor_thresh and censor_val to replace good data
values
>> with bad data values (-9999).
>> Or you can use them to replace bad data values (-9999) with good
data
>> values.
>> It's up to you.
>>
>> Whether it'll have the exact desired effect you seek, I cannot say
with
>> confidence.
>>
>> I'd recommend that you run the plot_data_plane tool on your data to
test
>> out the censor_thresh and censor_val settings you'd like to use.
That way
>> you can see the effect they have.
>> And then test the ensemble_stat tool to see if your configuration
has the
>> desired effect.
>>
>> If you have specific questions, please send me the command you ran,
the
>> path to the output file, and your question(s). And I'll do my best
to
>> answer them.
>>
>> John
>>
>> On Tue, Jun 1, 2021 at 3:03 PM binyu.wang at noaa.gov via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>> >
>> > Hello John,
>> >
>> > We can use "censor_thresh" and "censor_val" logic to replace the
missing
>> > data value with some minimum
>> > value, but how about the opposite way?
>> > E.g: the non-signal grid has value 10^(-99)=1E(-99) in my
forecast, so
>> they
>> > are all valid data. What if I want to lower the standard
vld_thresh
>> from 1
>> > to 0.6? Can I set the non-signal grid to invalid data? I mean I
don't
>> need
>> > all members to overlap. Does that sound right?
>> >
>> > Thank you.
>> > Binyu
>> >
>> > On Fri, May 28, 2021 at 5:05 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Binyu,
>> > >
>> > > There is no way to change the computation of the ranked
histogram
>> logic.
>> > If
>> > > we let the number of valid ensemble members vary from grid
point to
>> grid
>> > > point, then the resulting observation rank values cannot be
combined
>> > into a
>> > > ranked histogram.
>> > >
>> > > For example, let's say you compute the observation rank as...
>> > > - At grid point 1, obs rank is 5 out of 10 valid ensemble
member
>> values.
>> > > - At grid point 2, obs rank is 5 out of 15 valid ensemble
member
>> values.
>> > > - At grid point 3, obs rank is 5 out of 20 valid ensemble
member
>> values.
>> > >
>> > > While all 3 points have the same rank of 5, they cannot be
combined
>> into
>> > a
>> > > rank histogram because of the varying number of valid ensemble
member
>> > > values.
>> > >
>> > > But again this only matters if your ensemble forecasts contain
missing
>> > data
>> > > values. If they do, you could always use the "censor_thresh"
and
>> > > "censor_val" logic to replace the missing data value with some
minimum
>> > > value.
>> > >
>> > > John
>> > >
>> > > On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>
>> > > >
>> > > > Hello John,
>> > > >
>> > > > 1. Based on what you said,
>> > > > e.g: in my example (31 members) if I set vld_thresh=0.5, this
will
>> > result
>> > > > in ensemble mean and probability being written into the
NetCDF at
>> any
>> > > grid
>> > > > when 16 or more ensemble members have overlap with obs? I had
>> thought
>> > the
>> > > > function of vld_thresh=0.5 is to throw away all the grids
when half
>> of
>> > > the
>> > > > members have with invalid data.
>> > > >
>> > > >
>> > > > 2 . So there is no way that we can set some thresh to change
the
>> rule
>> > > > of computation
>> > > > of ranked histograms? I mean we must use ALL members for
that?
>> > > >
>> > > >
>> > > >
>> > > > Thank you and have a great long weekend.
>> > > >
>> > > > Binyu
>> > > >
>> > > > On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Binyu,
>> > > > >
>> > > > > Correct, setting vld_thresh = 0.5 will result in an
ensemble mean
>> > being
>> > > > > computed at any grid point which contains valid data for at
least
>> 50%
>> > > of
>> > > > > the ensemble members.
>> > > > >
>> > > > > MET stores bad data values internally as -9999.
>> > > > >
>> > > > > This usually is NOT a concern because most of the time
ensemble
>> > > forecasts
>> > > > > do not include any bata data values. But I haven't worked
with
>> your
>> > > > > ensemble data before so I don't know if/how many bad data
values
>> it
>> > > > > contains.
>> > > > >
>> > > > > John
>> > > > >
>> > > > > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via RT
<
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>> > > > > >
>> > > > > > John,
>> > > > > >
>> > > > > > Currently, "vld_thresh" only sets thresh to INVALID data,
is
>> > there a
>> > > > way
>> > > > > > to compute the ensemble mean and probabilities using only
part
>> of
>> > > > > ensemble
>> > > > > > members?
>> > > > > > e.g:
>> > > > > >
>> > > > > > 16 out of 31 have overlap with obs, the other 15 members
are
>> valid
>> > > but
>> > > > no
>> > > > > > overlap.
>> > > > > >
>> > > > > > Does that sound right?
>> > > > > >
>> > > > > > Thank you.
>> > > > > > Binyu
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via RT
<
>> > > > > > met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Binyu,
>> > > > > > >
>> > > > > > > So you're saying "ensemble performance" but I need you
to be
>> more
>> > > > > > explicit
>> > > > > > > than that. Do you mean, for example, that that point
needs to
>> be
>> > > > > included
>> > > > > > > in the ranked histogram (RHIST) output line type?
>> > > > > > >
>> > > > > > > Remember that vld_thresh applies in the computation of
the
>> > ensemble
>> > > > > mean
>> > > > > > > and probabilities written to the NetCDF output file. It
does
>> not
>> > > > apply
>> > > > > to
>> > > > > > > the computation of ranked histograms.
>> > > > > > >
>> > > > > > > John
>> > > > > > >
>> > > > > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov via
RT <
>> > > > > > > met_help at ucar.edu> wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>> > >
>> > > > > > > >
>> > > > > > > > John,
>> > > > > > > >
>> > > > > > > > Actually the non-signal grid has "-99" in my ensemble
member
>> > (the
>> > > > > > > original
>> > > > > > > > data was converted using LogX), and it needs to be
anti-log
>> in
>> > > > order
>> > > > > to
>> > > > > > > > compare with observation, so I use "convert(x)
=(10^(x))" in
>> > the
>> > > > > config
>> > > > > > > > file. I guess the converted value (10^(-99)=1E(-99))
is
>> > treated
>> > > as
>> > > > > > valid
>> > > > > > > > data, this is why there is no difference in my
results when
>> I
>> > > > change
>> > > > > > > > "vld_thresh=1" to "vld_thresh=0.5", because there is
no
>> invalid
>> > > > data
>> > > > > in
>> > > > > > > all
>> > > > > > > > the members.
>> > > > > > > >
>> > > > > > > > If my guess above is correct, is there a way to look
at the
>> > > > ensemble
>> > > > > > > > performance when only part of the members (like 20
out of
>> 31)
>> > has
>> > > > > > > overlap?
>> > > > > > > >
>> > > > > > > > Binyu
>> > > > > > > >
>> > > > > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway
via RT <
>> > > > > > > > met_help at ucar.edu>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Binyu,
>> > > > > > > > >
>> > > > > > > > > When the ensemble member does not predict the ash
at a
>> grid
>> > > > point,
>> > > > > is
>> > > > > > > > that
>> > > > > > > > > stored as a missing data value or some real data
value,
>> like
>> > 0?
>> > > > > > > > >
>> > > > > > > > > If it's a bad data value instead of 0, that'll
cause
>> > problems.
>> > > > > > > > > Ensemble-Stat only computes an observation rank
value when
>> > all
>> > > of
>> > > > > the
>> > > > > > > > > ensemble members contain valid data at that point.
>> > > > > > > > >
>> > > > > > > > > You only getting 3 pairs in the output may mean
that there
>> > are
>> > > > > only 3
>> > > > > > > > grid
>> > > > > > > > > points at which all 31 ensemble member contains
valid
>> data at
>> > > > those
>> > > > > > > > points.
>> > > > > > > > >
>> > > > > > > > > If it is bad data, you could switch it to 0, using
the
>> > > > > censor_thresh
>> > > > > > > and
>> > > > > > > > > censor_val options:
>> > > > > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
>> > > > > > > > >
>> > > > > > > > > John
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, May 26, 2021 at 2:47 PM binyu.wang at noaa.gov
via
>> RT <
>> > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > John,
>> > > > > > > > > >
>> > > > > > > > > > That is very clear. I really appreciate your
detailed
>> > > > > explanation.
>> > > > > > I
>> > > > > > > > had
>> > > > > > > > > > misunderstood "missing/invalid value" with "not
overlap"
>> > > > before.
>> > > > > > > > > >
>> > > > > > > > > > Here is what I am doing: I have 31 ensemble
inputs, and
>> I
>> > am
>> > > > > using
>> > > > > > > > > > satellite observed volcano ash for model
verification.
>> > Based
>> > > on
>> > > > > the
>> > > > > > > > > current
>> > > > > > > > > > results, there are only 3 matching pairs (which
means
>> only
>> > 3
>> > > > grid
>> > > > > > > cells
>> > > > > > > > > > over which all 31 members have ash overlap with
obs).
>> So I
>> > am
>> > > > > > > > "thinking"
>> > > > > > > > > if
>> > > > > > > > > > there is a way to look at the performance when
only
>> part of
>> > > the
>> > > > > > > members
>> > > > > > > > > > (like 20 out of 31) has overlap? Maybe this
thought is
>> > > > > unrealistic.
>> > > > > > > > Below
>> > > > > > > > > > is my current config file:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > I added "NEP" and "NMEP" in the config file, this
may
>> help
>> > > make
>> > > > > the
>> > > > > > > > > > verification look better, right?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Thank you.
>> > > > > > > > > >
>> > > > > > > > > > Binyu
>> > > > > > > > > >
>> > > > > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley
Gotway via
>> RT
>> > <
>> > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Binyu,
>> > > > > > > > > > >
>> > > > > > > > > > > OK, so assuming you have 10 input ensemble
members and
>> > > > > > ens_thresh =
>> > > > > > > > 0.5
>> > > > > > > > > > and
>> > > > > > > > > > > vld_thresh = 0.5.
>> > > > > > > > > > >
>> > > > > > > > > > > So you pass in 10 input files, 1 for each
ensemble
>> > member.
>> > > > If 6
>> > > > > > of
>> > > > > > > > > those
>> > > > > > > > > > > files don't exist (6/10 = 0.6 > 0.5
ens_thresh), then
>> > > > > > ensemble_stat
>> > > > > > > > > will
>> > > > > > > > > > > error out and produce no output. So this is
applied
>> once
>> > > and
>> > > > > > > > > > ensemble-stat
>> > > > > > > > > > > either will run or error out.
>> > > > > > > > > > >
>> > > > > > > > > > > One of the things that ensemble-stat does is
compute
>> an
>> > > > > ensemble
>> > > > > > > mean
>> > > > > > > > > > > field. At a particular grid point, if 6 of the
10
>> members
>> > > do
>> > > > > not
>> > > > > > > > have a
>> > > > > > > > > > > valid data value (6/10 = 0.6 > 0.5 vld_thresh),
then
>> the
>> > > > > ensemble
>> > > > > > > > mean
>> > > > > > > > > > > value will be computed as bad data. This
applies when
>> > > > deriving
>> > > > > > > > ensemble
>> > > > > > > > > > > fields that are written to the NetCDF output
file. So
>> > this
>> > > is
>> > > > > > > applied
>> > > > > > > > > > once
>> > > > > > > > > > > for each grid point and affects the values
written to
>> the
>> > > > > NetCDF
>> > > > > > > > > output.
>> > > > > > > > > > >
>> > > > > > > > > > > So ens_thresh specifies the required ratio of
ensemble
>> > > > members
>> > > > > > that
>> > > > > > > > > must
>> > > > > > > > > > be
>> > > > > > > > > > > present for ensemble_stat to run at all.
>> > > > > > > > > > > And vld_thresh specifies the required ratio of
valid
>> data
>> > > for
>> > > > > > > output
>> > > > > > > > to
>> > > > > > > > > > be
>> > > > > > > > > > > created at each grid point.
>> > > > > > > > > > >
>> > > > > > > > > > > I get the idea of what you mean by "overlap"
with the
>> > obs,
>> > > > but
>> > > > > > the
>> > > > > > > > > > details
>> > > > > > > > > > > matter a lot. When an ensemble member does NOT
overlap
>> > with
>> > > > the
>> > > > > > > obs,
>> > > > > > > > is
>> > > > > > > > > > > that encoded as bad data or as some legitimate
value,
>> > like
>> > > 0?
>> > > > > > > > > > >
>> > > > > > > > > > > If you have more questions, I'm guessing it'd
be more
>> > > > > informative
>> > > > > > > to
>> > > > > > > > > look
>> > > > > > > > > > > at a specific example.
>> > > > > > > > > > >
>> > > > > > > > > > > John
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, May 26, 2021 at 10:04 AM
binyu.wang at noaa.gov
>> via
>> > > RT
>> > > > <
>> > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>> > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hello John,
>> > > > > > > > > > > >
>> > > > > > > > > > > > I am confused about the "missing" ensemble
members
>> and
>> > > data
>> > > > > > > > > > > > definition. For example, there are 10
ensemble
>> > members
>> > > > > > overall.
>> > > > > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
>> > > > > > > > > > > > Ensemble member missing means:
>> > > > > > > > > > > > 1. There is only input for 5 members, or
>> > > > > > > > > > > > 2. all 10 member files exist, but only 5 of
them
>> have
>> > > > overlap
>> > > > > > > with
>> > > > > > > > > obs.
>> > > > > > > > > > > > I guess 1 is correct?
>> > > > > > > > > > > >
>> > > > > > > > > > > > The "missing" data refers to
>> > > > > > > > > > > > 3. all 10 members have data, but some data
are not
>> > valid
>> > > > > > > > > > > > 4. all 10 forecasts have data, but no
overlap?
>> > > > > > > > > > > > Is 3 correct?
>> > > > > > > > > > > >
>> > > > > > > > > > > > It seems "missing" does NOT mean "no
overlap". If
>> this
>> > is
>> > > > the
>> > > > > > > case,
>> > > > > > > > > is
>> > > > > > > > > > > > there a way to get ensemble statistics if I
don't
>> need
>> > > all
>> > > > > > > ensemble
>> > > > > > > > > > > members
>> > > > > > > > > > > > to overlap with obs? I.e: is it possible to
get the
>> > RELY
>> > > > > stat.
>> > > > > > > by
>> > > > > > > > > > > > setting more than 5 has overlap instead of
10?
>> > > > > > > > > > > >
>> > > > > > > > > > > > I hope it makes sense. Thank you.
>> > > > > > > > > > > > Binyu
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John Halley
Gotway
>> via
>> > > RT
>> > > > <
>> > > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Binyu,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The ens_thresh and vld_thresh options tell
>> > > ensemble_stat
>> > > > > how
>> > > > > > to
>> > > > > > > > > > handle
>> > > > > > > > > > > > > missing data... ens_thresh for missing
ensemble
>> > member
>> > > > > inputs
>> > > > > > > and
>> > > > > > > > > > > > > vld_thresh for missing data within the
input
>> ensemble
>> > > > > member
>> > > > > > > > data.
>> > > > > > > > > > They
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > as described in the user's guide:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > ----------------
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > When summarizing the ensemble, compute a
ratio of
>> the
>> > > > > number
>> > > > > > of
>> > > > > > > > > valid
>> > > > > > > > > > > > > ensemble fields to the total number of
ensemble
>> > > members.
>> > > > If
>> > > > > > > this
>> > > > > > > > > > ratio
>> > > > > > > > > > > is
>> > > > > > > > > > > > > less than the *ens_thresh*, then quit with
an
>> error.
>> > > This
>> > > > > > > > threshold
>> > > > > > > > > > > must
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > between 0 and 1. Setting this threshold to
1 will
>> > > require
>> > > > > > that
>> > > > > > > > all
>> > > > > > > > > > > > ensemble
>> > > > > > > > > > > > > members be present to be processed.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > When summarizing the ensemble, for each
grid point
>> > > > compute
>> > > > > a
>> > > > > > > > ratio
>> > > > > > > > > of
>> > > > > > > > > > > the
>> > > > > > > > > > > > > number of valid data values to the number
of
>> ensemble
>> > > > > > members.
>> > > > > > > If
>> > > > > > > > > > that
>> > > > > > > > > > > > > ratio is less than *vld_thresh*, write out
bad
>> data.
>> > > This
>> > > > > > > > threshold
>> > > > > > > > > > > must
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > between 0 and 1. Setting this threshold to
1 will
>> > > require
>> > > > > > each
>> > > > > > > > grid
>> > > > > > > > > > > point
>> > > > > > > > > > > > > to contain valid data for all ensemble
members.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > ----------------
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > If you have no missing ensemble members and
none
>> have
>> > > any
>> > > > > > > missing
>> > > > > > > > > > data
>> > > > > > > > > > > > > values, then changing these settings would
have no
>> > > impact
>> > > > > on
>> > > > > > > your
>> > > > > > > > > > > > results.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > John
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
>> binyu.wang at noaa.gov
>> > > via
>> > > > > RT <
>> > > > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > <URL:
>> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>> > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you, John.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Looking at the config file, I had another
>> question:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > fcst = {
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > ens_thresh = 1.0;
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > vld_thresh = 1.0;
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I understand the meaning of ens_thresh. I
got
>> very
>> > > few
>> > > > > > > matching
>> > > > > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0 to
0.5,
>> > > however,
>> > > > > the
>> > > > > > > > count
>> > > > > > > > > > of
>> > > > > > > > > > > > > number
>> > > > > > > > > > > > > > for ECNT are the same, the number of
ensemble
>> > values
>> > > > are
>> > > > > > the
>> > > > > > > > same
>> > > > > > > > > > as
>> > > > > > > > > > > > > well,
>> > > > > > > > > > > > > > they are all 31 (totally there are 31
ensemble
>> > > > members),
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Here is the *stat file:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
>> > > > > > > > > > > > > >
ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Any thing wrong here? Thank you.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Binyu
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John
Halley
>> Gotway
>> > > via
>> > > > > RT <
>> > > > > > > > > > > > > > met_help at ucar.edu>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Binyu,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > No that's correct. In ensemble_stat the
>> > cat_thresh
>> > > > > > setting
>> > > > > > > > > > applies
>> > > > > > > > > > > to
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > "ens" dictionary and defines the
thresholds
>> that
>> > to
>> > > > be
>> > > > > > used
>> > > > > > > > to
>> > > > > > > > > in
>> > > > > > > > > > > > > > defining
>> > > > > > > > > > > > > > > probabilities written to the NetCDF
output
>> file.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > The cat_thresh option does not apply to
the
>> fcst
>> > > and
>> > > > > obs
>> > > > > > > > > > > > dictionaries.
>> > > > > > > > > > > > > > None
>> > > > > > > > > > > > > > > of the output stat line types produced
by
>> > > > ensemble_stat
>> > > > > > > > (listed
>> > > > > > > > > > > > below)
>> > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > based on the application of a
categorical
>> > threshold
>> > > > > > defined
>> > > > > > > > by
>> > > > > > > > > > > > > > cat_thresh.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > John
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > output_flag = {
>> > > > > > > > > > > > > > > ecnt = NONE;
>> > > > > > > > > > > > > > > rps = NONE;
>> > > > > > > > > > > > > > > rhist = NONE;
>> > > > > > > > > > > > > > > phist = NONE;
>> > > > > > > > > > > > > > > orank = NONE;
>> > > > > > > > > > > > > > > ssvar = NONE;
>> > > > > > > > > > > > > > > relp = NONE;
>> > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
>> > > binyu.wang at noaa.gov
>> > > > > via
>> > > > > > > RT <
>> > > > > > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request
99973 was
>> > acted
>> > > > > upon.
>> > > > > > > > > > > > > > > > Transaction: Ticket created by
>> > > binyu.wang at noaa.gov
>> > > > > > > > > > > > > > > > Queue: met_help
>> > > > > > > > > > > > > > > > Subject: config files
>> > > > > > > > > > > > > > > > Owner: Nobody
>> > > > > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
>> > > > > > > > > > > > > > > > Status: new
>> > > > > > > > > > > > > > > > Ticket <URL:
>> > > > > > > > > > > > >
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hello,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I have two ensemble config files as
below
>> (the
>> > > > other
>> > > > > > > parts
>> > > > > > > > > of
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > config
>> > > > > > > > > > > > > > > > are the same), will they give exactly
the
>> same
>> > > > > output?
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > The only difference lies in there is
no
>> > > > "cat_thresh"
>> > > > > in
>> > > > > > > > fcst
>> > > > > > > > > > and
>> > > > > > > > > > > > obs
>> > > > > > > > > > > > > > > > entries. Based on my tests, it looks
like
>> the
>> > > > > ensemble
>> > > > > > > > > results
>> > > > > > > > > > > are
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > > with the two config files below, but
I am
>> > afraid
>> > > I
>> > > > > may
>> > > > > > > miss
>> > > > > > > > > > some
>> > > > > > > > > > > > > output
>> > > > > > > > > > > > > > > > flags.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > 1.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > fcst = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ens_thresh = 1.0;
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > vld_thresh = 1.0;
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "VAFTD";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>> > > >=0.1,>=0.3,
>> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > };
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ens = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "VAFTD";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>> > >=0.1,>=0.3,
>> > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > };
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > obs = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "ash_mass_loading";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = ["(*,*)"];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > cat_thresh = [ >=0.01,>=0.05,
>> > > >=0.1,>=0.3,
>> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> ---------------------------------------------
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > 2.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > fcst = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ens_thresh = 1.0;
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > vld_thresh = 1.0;
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "VAFTD";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > // cat_thresh = [>=0.01,
>=0.05,
>> > > > >=0.1,>=0.3,
>> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > };
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ens = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "VAFTD";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
>> > >=0.1,>=0.3,
>> > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > };
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > obs = {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > field = [
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > {
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > name = "ash_mass_loading";
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > level = ["(*,*)"];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > // cat_thresh = [
>=0.01,>=0.05,
>> > > > >=0.1,>=0.3,
>> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
>> > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > ];
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > So if they will lead to different
outputs,
>> what
>> > > > will
>> > > > > be
>> > > > > > > the
>> > > > > > > > > > > > > difference?
>> > > > > > > > > > > > > > > > Thank you.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Binyu
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: config files
From: John Halley Gotway
Time: Thu Jun 03 10:05:31 2021
Binyu,
Sure thing. Yes, the companion to ncdump is ncgen.
For example you can run:
ncdump test.nc > test.ncdump
# Then edit test.ncdump however you'd like... and then regenerate a
new
NetCDF file from it
ncgen -o test_new.nc test.ncdump
John
On Thu, Jun 3, 2021 at 8:48 AM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
>
> I think I found it, is it "ncgen"? Thank you.
>
> Binyu
>
> On Thu, Jun 3, 2021 at 10:30 AM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > John,
> >
> > Thank you for your suggestions and sorry for taking so much of
your time.
> >
> > I recalled that you know a command that can easily convert a text
file
> > (which is obtained using ncdump *.nc) back to NetCDF format, I
forgot
> what
> > that is (my old computer is broken and I lost all the records).
> >
> > eg.
> > $ ncdump test.nc > test.nc.txt
> > I can modify the header file of "test.nc.txt", and then convert
the txt
> > file back to NetCDF format.
> >
> > Thank you.
> > Binyu
> >
> > On Tue, Jun 1, 2021 at 6:19 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Binyu,
> >>
> >> I feel like we're talking in circles. And doing so without
crystal clear
> >> examples in data is not useful.
> >>
> >> Yes, you can use censor_thresh and censor_val to replace good
data
> values
> >> with bad data values (-9999).
> >> Or you can use them to replace bad data values (-9999) with good
data
> >> values.
> >> It's up to you.
> >>
> >> Whether it'll have the exact desired effect you seek, I cannot
say with
> >> confidence.
> >>
> >> I'd recommend that you run the plot_data_plane tool on your data
to test
> >> out the censor_thresh and censor_val settings you'd like to use.
That
> way
> >> you can see the effect they have.
> >> And then test the ensemble_stat tool to see if your configuration
has
> the
> >> desired effect.
> >>
> >> If you have specific questions, please send me the command you
ran, the
> >> path to the output file, and your question(s). And I'll do my
best to
> >> answer them.
> >>
> >> John
> >>
> >> On Tue, Jun 1, 2021 at 3:03 PM binyu.wang at noaa.gov via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >> >
> >> > Hello John,
> >> >
> >> > We can use "censor_thresh" and "censor_val" logic to replace
the
> missing
> >> > data value with some minimum
> >> > value, but how about the opposite way?
> >> > E.g: the non-signal grid has value 10^(-99)=1E(-99) in my
forecast, so
> >> they
> >> > are all valid data. What if I want to lower the standard
vld_thresh
> >> from 1
> >> > to 0.6? Can I set the non-signal grid to invalid data? I mean I
don't
> >> need
> >> > all members to overlap. Does that sound right?
> >> >
> >> > Thank you.
> >> > Binyu
> >> >
> >> > On Fri, May 28, 2021 at 5:05 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Binyu,
> >> > >
> >> > > There is no way to change the computation of the ranked
histogram
> >> logic.
> >> > If
> >> > > we let the number of valid ensemble members vary from grid
point to
> >> grid
> >> > > point, then the resulting observation rank values cannot be
combined
> >> > into a
> >> > > ranked histogram.
> >> > >
> >> > > For example, let's say you compute the observation rank as...
> >> > > - At grid point 1, obs rank is 5 out of 10 valid ensemble
member
> >> values.
> >> > > - At grid point 2, obs rank is 5 out of 15 valid ensemble
member
> >> values.
> >> > > - At grid point 3, obs rank is 5 out of 20 valid ensemble
member
> >> values.
> >> > >
> >> > > While all 3 points have the same rank of 5, they cannot be
combined
> >> into
> >> > a
> >> > > rank histogram because of the varying number of valid
ensemble
> member
> >> > > values.
> >> > >
> >> > > But again this only matters if your ensemble forecasts
contain
> missing
> >> > data
> >> > > values. If they do, you could always use the "censor_thresh"
and
> >> > > "censor_val" logic to replace the missing data value with
some
> minimum
> >> > > value.
> >> > >
> >> > > John
> >> > >
> >> > > On Fri, May 28, 2021 at 1:53 PM binyu.wang at noaa.gov via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973 >
> >> > > >
> >> > > > Hello John,
> >> > > >
> >> > > > 1. Based on what you said,
> >> > > > e.g: in my example (31 members) if I set vld_thresh=0.5,
this will
> >> > result
> >> > > > in ensemble mean and probability being written into the
NetCDF at
> >> any
> >> > > grid
> >> > > > when 16 or more ensemble members have overlap with obs? I
had
> >> thought
> >> > the
> >> > > > function of vld_thresh=0.5 is to throw away all the grids
when
> half
> >> of
> >> > > the
> >> > > > members have with invalid data.
> >> > > >
> >> > > >
> >> > > > 2 . So there is no way that we can set some thresh to
change the
> >> rule
> >> > > > of computation
> >> > > > of ranked histograms? I mean we must use ALL members for
that?
> >> > > >
> >> > > >
> >> > > >
> >> > > > Thank you and have a great long weekend.
> >> > > >
> >> > > > Binyu
> >> > > >
> >> > > > On Fri, May 28, 2021 at 12:12 PM John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Binyu,
> >> > > > >
> >> > > > > Correct, setting vld_thresh = 0.5 will result in an
ensemble
> mean
> >> > being
> >> > > > > computed at any grid point which contains valid data for
at
> least
> >> 50%
> >> > > of
> >> > > > > the ensemble members.
> >> > > > >
> >> > > > > MET stores bad data values internally as -9999.
> >> > > > >
> >> > > > > This usually is NOT a concern because most of the time
ensemble
> >> > > forecasts
> >> > > > > do not include any bata data values. But I haven't worked
with
> >> your
> >> > > > > ensemble data before so I don't know if/how many bad data
values
> >> it
> >> > > > > contains.
> >> > > > >
> >> > > > > John
> >> > > > >
> >> > > > > On Fri, May 28, 2021 at 8:51 AM binyu.wang at noaa.gov via
RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >
> >> > > > > >
> >> > > > > > John,
> >> > > > > >
> >> > > > > > Currently, "vld_thresh" only sets thresh to INVALID
data, is
> >> > there a
> >> > > > way
> >> > > > > > to compute the ensemble mean and probabilities using
only part
> >> of
> >> > > > > ensemble
> >> > > > > > members?
> >> > > > > > e.g:
> >> > > > > >
> >> > > > > > 16 out of 31 have overlap with obs, the other 15
members are
> >> valid
> >> > > but
> >> > > > no
> >> > > > > > overlap.
> >> > > > > >
> >> > > > > > Does that sound right?
> >> > > > > >
> >> > > > > > Thank you.
> >> > > > > > Binyu
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, May 27, 2021 at 6:24 PM John Halley Gotway via
RT <
> >> > > > > > met_help at ucar.edu>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Binyu,
> >> > > > > > >
> >> > > > > > > So you're saying "ensemble performance" but I need
you to be
> >> more
> >> > > > > > explicit
> >> > > > > > > than that. Do you mean, for example, that that point
needs
> to
> >> be
> >> > > > > included
> >> > > > > > > in the ranked histogram (RHIST) output line type?
> >> > > > > > >
> >> > > > > > > Remember that vld_thresh applies in the computation
of the
> >> > ensemble
> >> > > > > mean
> >> > > > > > > and probabilities written to the NetCDF output file.
It does
> >> not
> >> > > > apply
> >> > > > > to
> >> > > > > > > the computation of ranked histograms.
> >> > > > > > >
> >> > > > > > > John
> >> > > > > > >
> >> > > > > > > On Wed, May 26, 2021 at 8:23 PM binyu.wang at noaa.gov
via RT
> <
> >> > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >> > >
> >> > > > > > > >
> >> > > > > > > > John,
> >> > > > > > > >
> >> > > > > > > > Actually the non-signal grid has "-99" in my
ensemble
> member
> >> > (the
> >> > > > > > > original
> >> > > > > > > > data was converted using LogX), and it needs to be
> anti-log
> >> in
> >> > > > order
> >> > > > > to
> >> > > > > > > > compare with observation, so I use "convert(x)
=(10^(x))"
> in
> >> > the
> >> > > > > config
> >> > > > > > > > file. I guess the converted value (10^(-99)=1E(-
99)) is
> >> > treated
> >> > > as
> >> > > > > > valid
> >> > > > > > > > data, this is why there is no difference in my
results
> when
> >> I
> >> > > > change
> >> > > > > > > > "vld_thresh=1" to "vld_thresh=0.5", because there
is no
> >> invalid
> >> > > > data
> >> > > > > in
> >> > > > > > > all
> >> > > > > > > > the members.
> >> > > > > > > >
> >> > > > > > > > If my guess above is correct, is there a way to
look at
> the
> >> > > > ensemble
> >> > > > > > > > performance when only part of the members (like 20
out of
> >> 31)
> >> > has
> >> > > > > > > overlap?
> >> > > > > > > >
> >> > > > > > > > Binyu
> >> > > > > > > >
> >> > > > > > > > On Wed, May 26, 2021 at 5:27 PM John Halley Gotway
via RT
> <
> >> > > > > > > > met_help at ucar.edu>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Binyu,
> >> > > > > > > > >
> >> > > > > > > > > When the ensemble member does not predict the ash
at a
> >> grid
> >> > > > point,
> >> > > > > is
> >> > > > > > > > that
> >> > > > > > > > > stored as a missing data value or some real data
value,
> >> like
> >> > 0?
> >> > > > > > > > >
> >> > > > > > > > > If it's a bad data value instead of 0, that'll
cause
> >> > problems.
> >> > > > > > > > > Ensemble-Stat only computes an observation rank
value
> when
> >> > all
> >> > > of
> >> > > > > the
> >> > > > > > > > > ensemble members contain valid data at that
point.
> >> > > > > > > > >
> >> > > > > > > > > You only getting 3 pairs in the output may mean
that
> there
> >> > are
> >> > > > > only 3
> >> > > > > > > > grid
> >> > > > > > > > > points at which all 31 ensemble member contains
valid
> >> data at
> >> > > > those
> >> > > > > > > > points.
> >> > > > > > > > >
> >> > > > > > > > > If it is bad data, you could switch it to 0,
using the
> >> > > > > censor_thresh
> >> > > > > > > and
> >> > > > > > > > > censor_val options:
> >> > > > > > > > > censor_thresh = [ ==-9999 ]; censor_val = [ 0 ];
> >> > > > > > > > >
> >> > > > > > > > > John
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, May 26, 2021 at 2:47 PM
binyu.wang at noaa.gov via
> >> RT <
> >> > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > John,
> >> > > > > > > > > >
> >> > > > > > > > > > That is very clear. I really appreciate your
detailed
> >> > > > > explanation.
> >> > > > > > I
> >> > > > > > > > had
> >> > > > > > > > > > misunderstood "missing/invalid value" with "not
> overlap"
> >> > > > before.
> >> > > > > > > > > >
> >> > > > > > > > > > Here is what I am doing: I have 31 ensemble
inputs,
> and
> >> I
> >> > am
> >> > > > > using
> >> > > > > > > > > > satellite observed volcano ash for model
verification.
> >> > Based
> >> > > on
> >> > > > > the
> >> > > > > > > > > current
> >> > > > > > > > > > results, there are only 3 matching pairs (which
means
> >> only
> >> > 3
> >> > > > grid
> >> > > > > > > cells
> >> > > > > > > > > > over which all 31 members have ash overlap with
obs).
> >> So I
> >> > am
> >> > > > > > > > "thinking"
> >> > > > > > > > > if
> >> > > > > > > > > > there is a way to look at the performance when
only
> >> part of
> >> > > the
> >> > > > > > > members
> >> > > > > > > > > > (like 20 out of 31) has overlap? Maybe this
thought is
> >> > > > > unrealistic.
> >> > > > > > > > Below
> >> > > > > > > > > > is my current config file:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Bezy_thr0p05.nep
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > I added "NEP" and "NMEP" in the config file,
this may
> >> help
> >> > > make
> >> > > > > the
> >> > > > > > > > > > verification look better, right?
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Thank you.
> >> > > > > > > > > >
> >> > > > > > > > > > Binyu
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, May 26, 2021 at 12:24 PM John Halley
Gotway
> via
> >> RT
> >> > <
> >> > > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Binyu,
> >> > > > > > > > > > >
> >> > > > > > > > > > > OK, so assuming you have 10 input ensemble
members
> and
> >> > > > > > ens_thresh =
> >> > > > > > > > 0.5
> >> > > > > > > > > > and
> >> > > > > > > > > > > vld_thresh = 0.5.
> >> > > > > > > > > > >
> >> > > > > > > > > > > So you pass in 10 input files, 1 for each
ensemble
> >> > member.
> >> > > > If 6
> >> > > > > > of
> >> > > > > > > > > those
> >> > > > > > > > > > > files don't exist (6/10 = 0.6 > 0.5
ens_thresh),
> then
> >> > > > > > ensemble_stat
> >> > > > > > > > > will
> >> > > > > > > > > > > error out and produce no output. So this is
applied
> >> once
> >> > > and
> >> > > > > > > > > > ensemble-stat
> >> > > > > > > > > > > either will run or error out.
> >> > > > > > > > > > >
> >> > > > > > > > > > > One of the things that ensemble-stat does is
compute
> >> an
> >> > > > > ensemble
> >> > > > > > > mean
> >> > > > > > > > > > > field. At a particular grid point, if 6 of
the 10
> >> members
> >> > > do
> >> > > > > not
> >> > > > > > > > have a
> >> > > > > > > > > > > valid data value (6/10 = 0.6 > 0.5
vld_thresh), then
> >> the
> >> > > > > ensemble
> >> > > > > > > > mean
> >> > > > > > > > > > > value will be computed as bad data. This
applies
> when
> >> > > > deriving
> >> > > > > > > > ensemble
> >> > > > > > > > > > > fields that are written to the NetCDF output
file.
> So
> >> > this
> >> > > is
> >> > > > > > > applied
> >> > > > > > > > > > once
> >> > > > > > > > > > > for each grid point and affects the values
written
> to
> >> the
> >> > > > > NetCDF
> >> > > > > > > > > output.
> >> > > > > > > > > > >
> >> > > > > > > > > > > So ens_thresh specifies the required ratio of
> ensemble
> >> > > > members
> >> > > > > > that
> >> > > > > > > > > must
> >> > > > > > > > > > be
> >> > > > > > > > > > > present for ensemble_stat to run at all.
> >> > > > > > > > > > > And vld_thresh specifies the required ratio
of valid
> >> data
> >> > > for
> >> > > > > > > output
> >> > > > > > > > to
> >> > > > > > > > > > be
> >> > > > > > > > > > > created at each grid point.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I get the idea of what you mean by "overlap"
with
> the
> >> > obs,
> >> > > > but
> >> > > > > > the
> >> > > > > > > > > > details
> >> > > > > > > > > > > matter a lot. When an ensemble member does
NOT
> overlap
> >> > with
> >> > > > the
> >> > > > > > > obs,
> >> > > > > > > > is
> >> > > > > > > > > > > that encoded as bad data or as some
legitimate
> value,
> >> > like
> >> > > 0?
> >> > > > > > > > > > >
> >> > > > > > > > > > > If you have more questions, I'm guessing it'd
be
> more
> >> > > > > informative
> >> > > > > > > to
> >> > > > > > > > > look
> >> > > > > > > > > > > at a specific example.
> >> > > > > > > > > > >
> >> > > > > > > > > > > John
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, May 26, 2021 at 10:04 AM
> binyu.wang at noaa.gov
> >> via
> >> > > RT
> >> > > > <
> >> > > > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > <URL:
> >> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >> > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Hello John,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I am confused about the "missing" ensemble
members
> >> and
> >> > > data
> >> > > > > > > > > > > > definition. For example, there are 10
ensemble
> >> > members
> >> > > > > > overall.
> >> > > > > > > > > > > > ens_thresh=0.5, vld_thresh=0.5
> >> > > > > > > > > > > > Ensemble member missing means:
> >> > > > > > > > > > > > 1. There is only input for 5 members, or
> >> > > > > > > > > > > > 2. all 10 member files exist, but only 5 of
them
> >> have
> >> > > > overlap
> >> > > > > > > with
> >> > > > > > > > > obs.
> >> > > > > > > > > > > > I guess 1 is correct?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > The "missing" data refers to
> >> > > > > > > > > > > > 3. all 10 members have data, but some data
are not
> >> > valid
> >> > > > > > > > > > > > 4. all 10 forecasts have data, but no
overlap?
> >> > > > > > > > > > > > Is 3 correct?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > It seems "missing" does NOT mean "no
overlap". If
> >> this
> >> > is
> >> > > > the
> >> > > > > > > case,
> >> > > > > > > > > is
> >> > > > > > > > > > > > there a way to get ensemble statistics if I
don't
> >> need
> >> > > all
> >> > > > > > > ensemble
> >> > > > > > > > > > > members
> >> > > > > > > > > > > > to overlap with obs? I.e: is it possible
to get
> the
> >> > RELY
> >> > > > > stat.
> >> > > > > > > by
> >> > > > > > > > > > > > setting more than 5 has overlap instead of
10?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I hope it makes sense. Thank you.
> >> > > > > > > > > > > > Binyu
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, May 25, 2021 at 10:57 AM John
Halley
> Gotway
> >> via
> >> > > RT
> >> > > > <
> >> > > > > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Binyu,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > The ens_thresh and vld_thresh options
tell
> >> > > ensemble_stat
> >> > > > > how
> >> > > > > > to
> >> > > > > > > > > > handle
> >> > > > > > > > > > > > > missing data... ens_thresh for missing
ensemble
> >> > member
> >> > > > > inputs
> >> > > > > > > and
> >> > > > > > > > > > > > > vld_thresh for missing data within the
input
> >> ensemble
> >> > > > > member
> >> > > > > > > > data.
> >> > > > > > > > > > They
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > > as described in the user's guide:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://met.readthedocs.io/en/latest/Users_Guide/ensemble-
stat.html#practical-information
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > ----------------
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > When summarizing the ensemble, compute a
ratio
> of
> >> the
> >> > > > > number
> >> > > > > > of
> >> > > > > > > > > valid
> >> > > > > > > > > > > > > ensemble fields to the total number of
ensemble
> >> > > members.
> >> > > > If
> >> > > > > > > this
> >> > > > > > > > > > ratio
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > > less than the *ens_thresh*, then quit
with an
> >> error.
> >> > > This
> >> > > > > > > > threshold
> >> > > > > > > > > > > must
> >> > > > > > > > > > > > be
> >> > > > > > > > > > > > > between 0 and 1. Setting this threshold
to 1
> will
> >> > > require
> >> > > > > > that
> >> > > > > > > > all
> >> > > > > > > > > > > > ensemble
> >> > > > > > > > > > > > > members be present to be processed.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > When summarizing the ensemble, for each
grid
> point
> >> > > > compute
> >> > > > > a
> >> > > > > > > > ratio
> >> > > > > > > > > of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > number of valid data values to the number
of
> >> ensemble
> >> > > > > > members.
> >> > > > > > > If
> >> > > > > > > > > > that
> >> > > > > > > > > > > > > ratio is less than *vld_thresh*, write
out bad
> >> data.
> >> > > This
> >> > > > > > > > threshold
> >> > > > > > > > > > > must
> >> > > > > > > > > > > > be
> >> > > > > > > > > > > > > between 0 and 1. Setting this threshold
to 1
> will
> >> > > require
> >> > > > > > each
> >> > > > > > > > grid
> >> > > > > > > > > > > point
> >> > > > > > > > > > > > > to contain valid data for all ensemble
members.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > ----------------
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > If you have no missing ensemble members
and none
> >> have
> >> > > any
> >> > > > > > > missing
> >> > > > > > > > > > data
> >> > > > > > > > > > > > > values, then changing these settings
would have
> no
> >> > > impact
> >> > > > > on
> >> > > > > > > your
> >> > > > > > > > > > > > results.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > John
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Mon, May 24, 2021 at 4:51 PM
> >> binyu.wang at noaa.gov
> >> > > via
> >> > > > > RT <
> >> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > <URL:
> >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >> > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thank you, John.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Looking at the config file, I had
another
> >> question:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > fcst = {
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > ens_thresh = 1.0;
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > vld_thresh = 1.0;
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I understand the meaning of ens_thresh.
I got
> >> very
> >> > > few
> >> > > > > > > matching
> >> > > > > > > > > > > > > > using ens_thresh=1.0, so I changed 1.0
to 0.5,
> >> > > however,
> >> > > > > the
> >> > > > > > > > count
> >> > > > > > > > > > of
> >> > > > > > > > > > > > > number
> >> > > > > > > > > > > > > > for ECNT are the same, the number of
ensemble
> >> > values
> >> > > > are
> >> > > > > > the
> >> > > > > > > > same
> >> > > > > > > > > > as
> >> > > > > > > > > > > > > well,
> >> > > > > > > > > > > > > > they are all 31 (totally there are 31
ensemble
> >> > > > members),
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Here is the *stat file:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Bezy_31.res0p1.thresh0p05.vld0p5.ensthresh.0p5/
> >> > > > > > > > > > > > > >
> ensemble_stat_Bezy_31_hr04_20201021_230000V.stat
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Any thing wrong here? Thank you.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Binyu
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Mon, May 24, 2021 at 6:15 PM John
Halley
> >> Gotway
> >> > > via
> >> > > > > RT <
> >> > > > > > > > > > > > > > met_help at ucar.edu>
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Binyu,
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > No that's correct. In ensemble_stat
the
> >> > cat_thresh
> >> > > > > > setting
> >> > > > > > > > > > applies
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > "ens" dictionary and defines the
thresholds
> >> that
> >> > to
> >> > > > be
> >> > > > > > used
> >> > > > > > > > to
> >> > > > > > > > > in
> >> > > > > > > > > > > > > > defining
> >> > > > > > > > > > > > > > > probabilities written to the NetCDF
output
> >> file.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > The cat_thresh option does not apply
to the
> >> fcst
> >> > > and
> >> > > > > obs
> >> > > > > > > > > > > > dictionaries.
> >> > > > > > > > > > > > > > None
> >> > > > > > > > > > > > > > > of the output stat line types
produced by
> >> > > > ensemble_stat
> >> > > > > > > > (listed
> >> > > > > > > > > > > > below)
> >> > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > based on the application of a
categorical
> >> > threshold
> >> > > > > > defined
> >> > > > > > > > by
> >> > > > > > > > > > > > > > cat_thresh.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > John
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > output_flag = {
> >> > > > > > > > > > > > > > > ecnt = NONE;
> >> > > > > > > > > > > > > > > rps = NONE;
> >> > > > > > > > > > > > > > > rhist = NONE;
> >> > > > > > > > > > > > > > > phist = NONE;
> >> > > > > > > > > > > > > > > orank = NONE;
> >> > > > > > > > > > > > > > > ssvar = NONE;
> >> > > > > > > > > > > > > > > relp = NONE;
> >> > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Mon, May 24, 2021 at 3:55 PM
> >> > > binyu.wang at noaa.gov
> >> > > > > via
> >> > > > > > > RT <
> >> > > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Mon May 24 15:55:23 2021: Request
99973
> was
> >> > acted
> >> > > > > upon.
> >> > > > > > > > > > > > > > > > Transaction: Ticket created by
> >> > > binyu.wang at noaa.gov
> >> > > > > > > > > > > > > > > > Queue: met_help
> >> > > > > > > > > > > > > > > > Subject: config files
> >> > > > > > > > > > > > > > > > Owner: Nobody
> >> > > > > > > > > > > > > > > > Requestors: binyu.wang at noaa.gov
> >> > > > > > > > > > > > > > > > Status: new
> >> > > > > > > > > > > > > > > > Ticket <URL:
> >> > > > > > > > > > > > >
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99973
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Hello,
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > I have two ensemble config files
as below
> >> (the
> >> > > > other
> >> > > > > > > parts
> >> > > > > > > > > of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > > config
> >> > > > > > > > > > > > > > > > are the same), will they give
exactly the
> >> same
> >> > > > > output?
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > The only difference lies in there
is no
> >> > > > "cat_thresh"
> >> > > > > in
> >> > > > > > > > fcst
> >> > > > > > > > > > and
> >> > > > > > > > > > > > obs
> >> > > > > > > > > > > > > > > > entries. Based on my tests, it
looks like
> >> the
> >> > > > > ensemble
> >> > > > > > > > > results
> >> > > > > > > > > > > are
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > same
> >> > > > > > > > > > > > > > > > with the two config files below,
but I am
> >> > afraid
> >> > > I
> >> > > > > may
> >> > > > > > > miss
> >> > > > > > > > > > some
> >> > > > > > > > > > > > > output
> >> > > > > > > > > > > > > > > > flags.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > 1.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > fcst = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ens_thresh = 1.0;
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > vld_thresh = 1.0;
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "VAFTD";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > cat_thresh = [>=0.01,
>=0.05,
> >> > > >=0.1,>=0.3,
> >> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > };
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ens = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "VAFTD";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> >> > >=0.1,>=0.3,
> >> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > };
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > obs = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "ash_mass_loading";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = ["(*,*)"];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > cat_thresh = [
>=0.01,>=0.05,
> >> > > >=0.1,>=0.3,
> >> > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> ---------------------------------------------
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > 2.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > fcst = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ens_thresh = 1.0;
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > vld_thresh = 1.0;
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "VAFTD";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > // cat_thresh = [>=0.01,
>=0.05,
> >> > > > >=0.1,>=0.3,
> >> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > };
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ens = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "VAFTD";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = [ "L0-21335" ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > lead_time = "${LEAD_HR}";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > cat_thresh = [>=0.01, >=0.05,
> >> > >=0.1,>=0.3,
> >> > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > };
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > obs = {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > field = [
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > {
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > name = "ash_mass_loading";
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > level = ["(*,*)"];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > // cat_thresh = [
>=0.01,>=0.05,
> >> > > > >=0.1,>=0.3,
> >> > > > > > > > > > > > > >=0.5,>=1,>=5,>=10
> >> > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > }
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > ];
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > So if they will lead to different
outputs,
> >> what
> >> > > > will
> >> > > > > be
> >> > > > > > > the
> >> > > > > > > > > > > > > difference?
> >> > > > > > > > > > > > > > > > Thank you.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Binyu
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
More information about the Met_help
mailing list