[Met_help] [rt.rap.ucar.edu #96427] History for Threshold not work?
John Halley Gotway via RT
met_help at ucar.edu
Tue Aug 25 16:53:11 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
Could you please help to take a look at the config file below: I try to
remove the low values less than 0.1 by adding the line ( fcst_thresh = [
>=0.1 ];) in the config file , but the outputs have no difference: they
have the same OBAR and FBAR.
Here is the config file:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2o_ens_stat_regn_configHe_Reventador
output without thresh:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador_no.thresh/ensemble_stat_Reventador_hr02_20190225_180000V.stat
output with thresh.
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador/ensemble_stat_Reventador_hr02_20190225_180000V.stat
So what should I do to remove the value less than a threshold? I want to
remove those low values before I get OBAR/FBAR, Rank Hist plot. etc.
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Threshold not work?
From: John Halley Gotway
Time: Mon Aug 24 14:33:44 2020
Binyu,
I see that you're running the Ensemble-Stat tool and would like to
exclude
some data from your analysis based on a filtering threshold. And
you've
setup your config file to specify:
fcst_thresh = [ >=0.1 ];
But that setting has no impact on the output.
When you look at the default Ensemble-Stat config file (
https://github.com/dtcenter/MET/blob/master_v9.1/met/data/config/EnsembleStatConfig_default)
you'll find that there is no entry for "fcst_thresh". However, there
is an
entry for "obs_thresh" and a description of it can be found in this
chapter
of the user's guide:
https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html
So you could try setting "obs_thresh = [ >=0.1 ];" and then only obs
which
meet that threshold criteria would be included in the output.
No "fcst_thresh" option is provided because the logic would be
confusing.
Ensemble-Stat reads multiple input ensemble members. So for each obs
value,
we have N forecast ensemble member values. We could check that
threshold
for each ensemble value, but must ALL ensemble members meet that
threshold?
Or only some fraction of them? Or perhaps their min, max, median, or
average value? We did not add this option because the answers to those
questions were not immediately obvious. Let me know if you think this
functionality would be useful and we could work together to define a
GitHub
issue to describe this feature request and answer those questions.
For now, you do have another option. When reading the forecast data,
you
could use the censor_thresh and censor_val options to "censor" your
forecast data. You could simply replace any forecast values <0.1 with
a
missing data value of -9999. For example:
fcst = {
field = [ { name="APCP"; level="A3"; censor_thresh=[<0.1];
censor_val=[-9999]; desc="GE0.1" } ];
}
In this example, we read 3-hourly precip and replace any values less
than
0.1 with MET's internal bad data value of -9999. Notice that I also
used
the "desc" option to set the DESC (for "description") output column.
That
indicates that this output is only for data greater than or equal to
0.1.
Otherwise, there would be no way to indicate that fact in the output.
Hope that helps clarify.
Thanks,
John Halley Gotway
On Mon, Aug 24, 2020 at 11:37 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 24 11:37:56 2020: Request 96427 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: Threshold not work?
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96427 >
>
>
> Hello,
> Could you please help to take a look at the config file below: I try
to
> remove the low values less than 0.1 by adding the line (
fcst_thresh = [
> >=0.1 ];) in the config file , but the outputs have no difference:
they
> have the same OBAR and FBAR.
>
> Here is the config file:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2o_ens_stat_regn_configHe_Reventador
>
> output without thresh:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador_no.thresh/ensemble_stat_Reventador_hr02_20190225_180000V.stat
>
> output with thresh.
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador/ensemble_stat_Reventador_hr02_20190225_180000V.stat
>
> So what should I do to remove the value less than a threshold? I
want to
> remove those low values before I get OBAR/FBAR, Rank Hist plot. etc.
>
> Thank you.
> Binyu
>
>
------------------------------------------------
Subject: Threshold not work?
From: binyu.wang at noaa.gov
Time: Tue Aug 25 12:55:56 2020
Thank you very much, John. It works. I think " fcst_thresh " is very
useful
because frequently we want to remove the very low values, and usually
this
needs to apply for all members.
Binyu
On Mon, Aug 24, 2020 at 4:33 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I see that you're running the Ensemble-Stat tool and would like to
exclude
> some data from your analysis based on a filtering threshold. And
you've
> setup your config file to specify:
> fcst_thresh = [ >=0.1 ];
> But that setting has no impact on the output.
>
> When you look at the default Ensemble-Stat config file (
>
>
https://github.com/dtcenter/MET/blob/master_v9.1/met/data/config/EnsembleStatConfig_default
> )
> you'll find that there is no entry for "fcst_thresh". However, there
is an
> entry for "obs_thresh" and a description of it can be found in this
chapter
> of the user's guide:
> https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html
>
> So you could try setting "obs_thresh = [ >=0.1 ];" and then only obs
which
> meet that threshold criteria would be included in the output.
>
> No "fcst_thresh" option is provided because the logic would be
confusing.
> Ensemble-Stat reads multiple input ensemble members. So for each obs
value,
> we have N forecast ensemble member values. We could check that
threshold
> for each ensemble value, but must ALL ensemble members meet that
threshold?
> Or only some fraction of them? Or perhaps their min, max, median, or
> average value? We did not add this option because the answers to
those
> questions were not immediately obvious. Let me know if you think
this
> functionality would be useful and we could work together to define a
GitHub
> issue to describe this feature request and answer those questions.
>
> For now, you do have another option. When reading the forecast data,
you
> could use the censor_thresh and censor_val options to "censor" your
> forecast data. You could simply replace any forecast values <0.1
with a
> missing data value of -9999. For example:
>
> fcst = {
> field = [ { name="APCP"; level="A3"; censor_thresh=[<0.1];
> censor_val=[-9999]; desc="GE0.1" } ];
> }
>
> In this example, we read 3-hourly precip and replace any values less
than
> 0.1 with MET's internal bad data value of -9999. Notice that I also
used
> the "desc" option to set the DESC (for "description") output column.
That
> indicates that this output is only for data greater than or equal to
0.1.
> Otherwise, there would be no way to indicate that fact in the
output.
>
> Hope that helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Aug 24, 2020 at 11:37 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 24 11:37:56 2020: Request 96427 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: Threshold not work?
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96427 >
> >
> >
> > Hello,
> > Could you please help to take a look at the config file below: I
try to
> > remove the low values less than 0.1 by adding the line (
fcst_thresh
> = [
> > >=0.1 ];) in the config file , but the outputs have no difference:
they
> > have the same OBAR and FBAR.
> >
> > Here is the config file:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2o_ens_stat_regn_configHe_Reventador
> >
> > output without thresh:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador_no.thresh/ensemble_stat_Reventador_hr02_20190225_180000V.stat
> >
> > output with thresh.
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador/ensemble_stat_Reventador_hr02_20190225_180000V.stat
> >
> > So what should I do to remove the value less than a threshold? I
want to
> > remove those low values before I get OBAR/FBAR, Rank Hist plot.
etc.
> >
> > Thank you.
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: Threshold not work?
From: John Halley Gotway
Time: Tue Aug 25 16:52:55 2020
Binyu,
Oops, looks like I responded on the wrong thread. Here's the GitHub
development issue about adding fcst_thresh to ensemble_stat:
https://github.com/dtcenter/MET/issues/999
I'll go ahead and resolve this ticket, but I realize I still need to
work
on your other one.
John
On Tue, Aug 25, 2020 at 12:56 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96427 >
>
> Thank you very much, John. It works. I think " fcst_thresh " is very
useful
> because frequently we want to remove the very low values, and
usually this
> needs to apply for all members.
>
> Binyu
>
> On Mon, Aug 24, 2020 at 4:33 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I see that you're running the Ensemble-Stat tool and would like to
> exclude
> > some data from your analysis based on a filtering threshold. And
you've
> > setup your config file to specify:
> > fcst_thresh = [ >=0.1 ];
> > But that setting has no impact on the output.
> >
> > When you look at the default Ensemble-Stat config file (
> >
> >
>
https://github.com/dtcenter/MET/blob/master_v9.1/met/data/config/EnsembleStatConfig_default
> > )
> > you'll find that there is no entry for "fcst_thresh". However,
there is
> an
> > entry for "obs_thresh" and a description of it can be found in
this
> chapter
> > of the user's guide:
> > https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html
> >
> > So you could try setting "obs_thresh = [ >=0.1 ];" and then only
obs
> which
> > meet that threshold criteria would be included in the output.
> >
> > No "fcst_thresh" option is provided because the logic would be
confusing.
> > Ensemble-Stat reads multiple input ensemble members. So for each
obs
> value,
> > we have N forecast ensemble member values. We could check that
threshold
> > for each ensemble value, but must ALL ensemble members meet that
> threshold?
> > Or only some fraction of them? Or perhaps their min, max, median,
or
> > average value? We did not add this option because the answers to
those
> > questions were not immediately obvious. Let me know if you think
this
> > functionality would be useful and we could work together to define
a
> GitHub
> > issue to describe this feature request and answer those questions.
> >
> > For now, you do have another option. When reading the forecast
data, you
> > could use the censor_thresh and censor_val options to "censor"
your
> > forecast data. You could simply replace any forecast values <0.1
with a
> > missing data value of -9999. For example:
> >
> > fcst = {
> > field = [ { name="APCP"; level="A3"; censor_thresh=[<0.1];
> > censor_val=[-9999]; desc="GE0.1" } ];
> > }
> >
> > In this example, we read 3-hourly precip and replace any values
less than
> > 0.1 with MET's internal bad data value of -9999. Notice that I
also used
> > the "desc" option to set the DESC (for "description") output
column. That
> > indicates that this output is only for data greater than or equal
to 0.1.
> > Otherwise, there would be no way to indicate that fact in the
output.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Mon, Aug 24, 2020 at 11:37 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Aug 24 11:37:56 2020: Request 96427 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: Threshold not work?
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96427
> >
> > >
> > >
> > > Hello,
> > > Could you please help to take a look at the config file below: I
try to
> > > remove the low values less than 0.1 by adding the line (
fcst_thresh
> > = [
> > > >=0.1 ];) in the config file , but the outputs have no
difference: they
> > > have the same OBAR and FBAR.
> > >
> > > Here is the config file:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2o_ens_stat_regn_configHe_Reventador
> > >
> > > output without thresh:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador_no.thresh/ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > >
> > > output with thresh.
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Reventador/ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > >
> > > So what should I do to remove the value less than a threshold? I
want
> to
> > > remove those low values before I get OBAR/FBAR, Rank Hist plot.
etc.
> > >
> > > Thank you.
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list