[Met_help] [rt.rap.ucar.edu #87466] History for question about series_analysis_config
John Halley Gotway via RT
met_help at ucar.edu
Wed Oct 24 10:35:20 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
I am using MET : series analysis tool to do monthly analysis between urma
and other forecast models.
I saw column "FCST_THRESH" in the header information, but if I added
"FCST_THRESH" to series_analysis_config, I got error " unsupported column
name requested "FCST_THRESH""
Would you please explain why this happens?
Thank you
Guohui1
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question about series_analysis_config
From: John Halley Gotway
Time: Mon Oct 22 11:05:27 2018
Guohui,
I see that you have a question about the Series-Analysis tool in MET.
To
answer your question, let me clarify a couple of details.
The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat, Ensemble-
Stat,
and Wavelet-Stat) all write output files named ".stat". The ".stat"
output
files contain multiple line types, each containing a different type of
output verification statistics.
These lines consists of 2 parts... The first 22 columns are the header
columns and are common to all line types. They contain metadata about
the
forecast and observation data being compared, timing information, and
threshold information. This is metadata about the verification
comparison.
Columns 23 to the end of the line are specific to each line type.
This is
the actual verification data. For example, the CNT line type contains
the
RMSE (Root-Mean-Squared-Error) column.
The Grid-Stat and Point-Stat tools are run once per verification time
and
compute statistics that are computed over some spatial area. The
Series-Analysis tool, on the other hand, computes statistics for each
individual grid point... and those stats are typically computed over a
time
series of data.
In the Series-Analysis config file, the "output_stats" config file
entry
defines the statistics that should be computed for each grid point.
There
is a separate entry for each line type. You simply specify the name
of the
column for the statistic(s) you'd like to see computed in the output.
So
these should be the column names for columns 23 and beyond.
As you found, if you request columns 1 through 22, you'll get an
error.
Those contain metadata rather than the actual computed statistics in
columns 23 and beyond.
Hopefully that clarifies the issue.
If not, please let me know.
Thanks,
John Halley Gotway
On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> Transaction: Ticket created by guohui.wang at noaa.gov
> Queue: met_help
> Subject: question about series_analysis_config
> Owner: Nobody
> Requestors: guohui.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
>
>
> Good morning,
>
> I am using MET : series analysis tool to do monthly analysis between
urma
> and other forecast models.
>
> I saw column "FCST_THRESH" in the header information, but if I added
> "FCST_THRESH" to series_analysis_config, I got error " unsupported
column
> name requested "FCST_THRESH""
>
> Would you please explain why this happens?
>
> Thank you
>
> Guohui1
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: Guohui Wang - NOAA Affiliate
Time: Mon Oct 22 11:39:39 2018
Good afternoon, John,
Thank you for replying back quickly.
You can see that I defined cat_thresh and cat_thresh from line 55 and
56
in the config file. So for series analysis, there is no need to
specify
cnt_thresh and cat_thresh as we did in grid_stat?
Guohui
On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> I see that you have a question about the Series-Analysis tool in
MET. To
> answer your question, let me clarify a couple of details.
>
> The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
Ensemble-Stat,
> and Wavelet-Stat) all write output files named ".stat". The ".stat"
output
> files contain multiple line types, each containing a different type
of
> output verification statistics.
>
> These lines consists of 2 parts... The first 22 columns are the
header
> columns and are common to all line types. They contain metadata
about the
> forecast and observation data being compared, timing information,
and
> threshold information. This is metadata about the verification
comparison.
>
> Columns 23 to the end of the line are specific to each line type.
This is
> the actual verification data. For example, the CNT line type
contains the
> RMSE (Root-Mean-Squared-Error) column.
>
> The Grid-Stat and Point-Stat tools are run once per verification
time and
> compute statistics that are computed over some spatial area. The
> Series-Analysis tool, on the other hand, computes statistics for
each
> individual grid point... and those stats are typically computed over
a time
> series of data.
>
> In the Series-Analysis config file, the "output_stats" config file
entry
> defines the statistics that should be computed for each grid point.
There
> is a separate entry for each line type. You simply specify the name
of the
> column for the statistic(s) you'd like to see computed in the
output. So
> these should be the column names for columns 23 and beyond.
>
> As you found, if you request columns 1 through 22, you'll get an
error.
> Those contain metadata rather than the actual computed statistics in
> columns 23 and beyond.
>
> Hopefully that clarifies the issue.
>
> If not, please let me know.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > Transaction: Ticket created by guohui.wang at noaa.gov
> > Queue: met_help
> > Subject: question about series_analysis_config
> > Owner: Nobody
> > Requestors: guohui.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
> >
> >
> > Good morning,
> >
> > I am using MET : series analysis tool to do monthly analysis
between urma
> > and other forecast models.
> >
> > I saw column "FCST_THRESH" in the header information, but if I
added
> > "FCST_THRESH" to series_analysis_config, I got error " unsupported
column
> > name requested "FCST_THRESH""
> >
> > Would you please explain why this happens?
> >
> > Thank you
> >
> > Guohui1
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: John Halley Gotway
Time: Mon Oct 22 13:07:32 2018
Guohui,
Let's say that you're running Series-Analysis to verify 3-hourly
accumulated precip. So in the config file, you've set:
fcst = {
field = [ { name = "APCP"; level = [ "A03" ]; } ];
}
obs = fcst;
And you want to compute CSI and GSS (which is equitable threat score).
In
the config file, you'd request those in the "output_stats" entry:
output_stats = {
...
cts = [ "TOTAL", "CSI", "GSS" ];
...
}
Note that the TOTAL column is just the number of matched pairs in the
time
series. I recommend that people look at that to make sure that they
don't
have any unexpected areas of missing data.
Since these are categorical statistics, you do need to define the
threshold(s) you'd like to use to define them. Just like in Grid-Stat
and
Point-Stat, you define the categorical thresholds using the
"cat_thresh"
config file entry:
cat_thresh = [ >0, >2.54, >25.4 ];
Just fill in whatever categorical thresholds you'd like to use.
Make sense?
Thanks,
John
On Mon, Oct 22, 2018 at 11:40 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
>
> Good afternoon, John,
>
> Thank you for replying back quickly.
>
> You can see that I defined cat_thresh and cat_thresh from line 55
and 56
> in the config file. So for series analysis, there is no need to
specify
> cnt_thresh and cat_thresh as we did in grid_stat?
>
> Guohui
>
> On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Guohui,
> >
> > I see that you have a question about the Series-Analysis tool in
MET. To
> > answer your question, let me clarify a couple of details.
> >
> > The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
> Ensemble-Stat,
> > and Wavelet-Stat) all write output files named ".stat". The
".stat"
> output
> > files contain multiple line types, each containing a different
type of
> > output verification statistics.
> >
> > These lines consists of 2 parts... The first 22 columns are the
header
> > columns and are common to all line types. They contain metadata
about
> the
> > forecast and observation data being compared, timing information,
and
> > threshold information. This is metadata about the verification
> comparison.
> >
> > Columns 23 to the end of the line are specific to each line type.
This
> is
> > the actual verification data. For example, the CNT line type
contains
> the
> > RMSE (Root-Mean-Squared-Error) column.
> >
> > The Grid-Stat and Point-Stat tools are run once per verification
time and
> > compute statistics that are computed over some spatial area. The
> > Series-Analysis tool, on the other hand, computes statistics for
each
> > individual grid point... and those stats are typically computed
over a
> time
> > series of data.
> >
> > In the Series-Analysis config file, the "output_stats" config file
entry
> > defines the statistics that should be computed for each grid
point.
> There
> > is a separate entry for each line type. You simply specify the
name of
> the
> > column for the statistic(s) you'd like to see computed in the
output. So
> > these should be the column names for columns 23 and beyond.
> >
> > As you found, if you request columns 1 through 22, you'll get an
error.
> > Those contain metadata rather than the actual computed statistics
in
> > columns 23 and beyond.
> >
> > Hopefully that clarifies the issue.
> >
> > If not, please let me know.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > Queue: met_help
> > > Subject: question about series_analysis_config
> > > Owner: Nobody
> > > Requestors: guohui.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
> >
> > >
> > >
> > > Good morning,
> > >
> > > I am using MET : series analysis tool to do monthly analysis
between
> urma
> > > and other forecast models.
> > >
> > > I saw column "FCST_THRESH" in the header information, but if I
added
> > > "FCST_THRESH" to series_analysis_config, I got error "
unsupported
> column
> > > name requested "FCST_THRESH""
> > >
> > > Would you please explain why this happens?
> > >
> > > Thank you
> > >
> > > Guohui1
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: Guohui Wang - NOAA Affiliate
Time: Tue Oct 23 08:31:01 2018
Good morning, John,
I saw there are two different place to define cat_thresh and cnt
thresh:
one is in fcst definition and one is out. I would think that the one
in
fcst definition will cover the one outside. But if I define my
cat_thresh
as follows,
*************************************************************************************************************************************************
cat_thresh = [ NA ];
cnt_thresh = [ NA ];
cnt_logic = UNION;
/////////// from grid_stat_config
wind_thresh = [ NA ];
wind_logic = UNION;
//
// Forecast and observation fields to be verified
//
fcst = {
field = [
{
name = "APCP_06";
level = "(*,*)";
cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
>=6.35&&<12.7, >=12.7&&<25.4, >=25.4&&<76.2, >=76.2&&<127, >=127 ];
cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=25.4,
>=50.8,
>=76.2, >=127 ];
}
];
}
obs = fcst;
*************************************************************************************************************************************************
I saw the fcst_thresh of variables in output netcdf file is "NA". Do
you
know why it is like this? Or, how can I check variable values per
thresh-hold value?
float series_cnt_RMSE_BCU(lat, lon) ;
series_cnt_RMSE_BCU:_FillValue = -9999.f ;
series_cnt_RMSE_BCU:name = "RMSE_BCU" ;
series_cnt_RMSE_BCU:long_name = "Root mean squared
error
bootstrap confidence interval upper limit" ;
series_cnt_RMSE_BCU:fcst_thresh = "NA" ;
series_cnt_RMSE_BCU:obs_thresh = "NA" ;
Thank you
Guohui
On Mon, Oct 22, 2018 at 3:07 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> Let's say that you're running Series-Analysis to verify 3-hourly
> accumulated precip. So in the config file, you've set:
>
> fcst = {
> field = [ { name = "APCP"; level = [ "A03" ]; } ];
> }
> obs = fcst;
>
> And you want to compute CSI and GSS (which is equitable threat
score). In
> the config file, you'd request those in the "output_stats" entry:
>
> output_stats = {
> ...
> cts = [ "TOTAL", "CSI", "GSS" ];
> ...
> }
>
> Note that the TOTAL column is just the number of matched pairs in
the time
> series. I recommend that people look at that to make sure that they
don't
> have any unexpected areas of missing data.
>
> Since these are categorical statistics, you do need to define the
> threshold(s) you'd like to use to define them. Just like in Grid-
Stat and
> Point-Stat, you define the categorical thresholds using the
"cat_thresh"
> config file entry:
>
> cat_thresh = [ >0, >2.54, >25.4 ];
>
> Just fill in whatever categorical thresholds you'd like to use.
>
> Make sense?
>
> Thanks,
> John
>
> On Mon, Oct 22, 2018 at 11:40 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
> >
> > Good afternoon, John,
> >
> > Thank you for replying back quickly.
> >
> > You can see that I defined cat_thresh and cat_thresh from line 55
and 56
> > in the config file. So for series analysis, there is no need to
specify
> > cnt_thresh and cat_thresh as we did in grid_stat?
> >
> > Guohui
> >
> > On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Guohui,
> > >
> > > I see that you have a question about the Series-Analysis tool in
MET.
> To
> > > answer your question, let me clarify a couple of details.
> > >
> > > The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
> > Ensemble-Stat,
> > > and Wavelet-Stat) all write output files named ".stat". The
".stat"
> > output
> > > files contain multiple line types, each containing a different
type of
> > > output verification statistics.
> > >
> > > These lines consists of 2 parts... The first 22 columns are the
header
> > > columns and are common to all line types. They contain metadata
about
> > the
> > > forecast and observation data being compared, timing
information, and
> > > threshold information. This is metadata about the verification
> > comparison.
> > >
> > > Columns 23 to the end of the line are specific to each line
type. This
> > is
> > > the actual verification data. For example, the CNT line type
contains
> > the
> > > RMSE (Root-Mean-Squared-Error) column.
> > >
> > > The Grid-Stat and Point-Stat tools are run once per verification
time
> and
> > > compute statistics that are computed over some spatial area.
The
> > > Series-Analysis tool, on the other hand, computes statistics for
each
> > > individual grid point... and those stats are typically computed
over a
> > time
> > > series of data.
> > >
> > > In the Series-Analysis config file, the "output_stats" config
file
> entry
> > > defines the statistics that should be computed for each grid
point.
> > There
> > > is a separate entry for each line type. You simply specify the
name of
> > the
> > > column for the statistic(s) you'd like to see computed in the
output.
> So
> > > these should be the column names for columns 23 and beyond.
> > >
> > > As you found, if you request columns 1 through 22, you'll get an
error.
> > > Those contain metadata rather than the actual computed
statistics in
> > > columns 23 and beyond.
> > >
> > > Hopefully that clarifies the issue.
> > >
> > > If not, please let me know.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > > On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: question about series_analysis_config
> > > > Owner: Nobody
> > > > Requestors: guohui.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
> > >
> > > >
> > > >
> > > > Good morning,
> > > >
> > > > I am using MET : series analysis tool to do monthly analysis
between
> > urma
> > > > and other forecast models.
> > > >
> > > > I saw column "FCST_THRESH" in the header information, but if I
added
> > > > "FCST_THRESH" to series_analysis_config, I got error "
unsupported
> > column
> > > > name requested "FCST_THRESH""
> > > >
> > > > Would you please explain why this happens?
> > > >
> > > > Thank you
> > > >
> > > > Guohui1
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: John Halley Gotway
Time: Tue Oct 23 09:53:40 2018
Guohui,
Yes, I see your confusion. Please define these thresholds at the
higher
level of context, like this:
//
// Forecast and observation fields to be verified
//
fcst = {
cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
>=6.35&&<12.7,
>=12.7&&<25.4, >=25.4&&<76.2, >=76.2&&<127, >=127 ];
cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=25.4, >=50.8,
>=76.2,
>=127 ];
field = [
{
name = "APCP_06";
level = "(*,*)";
}
];
}
obs = fcst;
Since the Grid-Stat and Point-Stat tools can be run to evaluate
multiple
fields at a time, the cnt_thresh and cat_thresh settings are parsed
separately for each entry in the "field" array. Series-Analysis, on
the
other hand, can only be used to define one series at a time. For that
reason, the code parses them from a higher level of context. While
they
can be set differently for "fcst" and "obs", they can't be set
differently
for each entry in the "field" array.
So you should either define them inside the "fcst" dictionary (as
shown
above) or at the global level of context (meaning, outside the "fcst"
dictionary).
Please give that a shot and let me know if you have more questions.
Thanks,
John
On Tue, Oct 23, 2018 at 8:31 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
>
> Good morning, John,
>
> I saw there are two different place to define cat_thresh and cnt
thresh:
> one is in fcst definition and one is out. I would think that the
one in
> fcst definition will cover the one outside. But if I define my
cat_thresh
> as follows,
>
>
>
*************************************************************************************************************************************************
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
>
> /////////// from grid_stat_config
> wind_thresh = [ NA ];
> wind_logic = UNION;
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> field = [
> {
> name = "APCP_06";
> level = "(*,*)";
> cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> >=6.35&&<12.7, >=12.7&&<25.4, >=25.4&&<76.2, >=76.2&&<127, >=127 ];
> cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=25.4,
>=50.8,
> >=76.2, >=127 ];
> }
> ];
>
> }
> obs = fcst;
>
>
*************************************************************************************************************************************************
> I saw the fcst_thresh of variables in output netcdf file is "NA".
Do you
> know why it is like this? Or, how can I check variable values per
> thresh-hold value?
>
> float series_cnt_RMSE_BCU(lat, lon) ;
> series_cnt_RMSE_BCU:_FillValue = -9999.f ;
> series_cnt_RMSE_BCU:name = "RMSE_BCU" ;
> series_cnt_RMSE_BCU:long_name = "Root mean squared
error
> bootstrap confidence interval upper limit" ;
> series_cnt_RMSE_BCU:fcst_thresh = "NA" ;
> series_cnt_RMSE_BCU:obs_thresh = "NA" ;
>
> Thank you
>
> Guohui
>
> On Mon, Oct 22, 2018 at 3:07 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Guohui,
> >
> > Let's say that you're running Series-Analysis to verify 3-hourly
> > accumulated precip. So in the config file, you've set:
> >
> > fcst = {
> > field = [ { name = "APCP"; level = [ "A03" ]; } ];
> > }
> > obs = fcst;
> >
> > And you want to compute CSI and GSS (which is equitable threat
score).
> In
> > the config file, you'd request those in the "output_stats" entry:
> >
> > output_stats = {
> > ...
> > cts = [ "TOTAL", "CSI", "GSS" ];
> > ...
> > }
> >
> > Note that the TOTAL column is just the number of matched pairs in
the
> time
> > series. I recommend that people look at that to make sure that
they
> don't
> > have any unexpected areas of missing data.
> >
> > Since these are categorical statistics, you do need to define the
> > threshold(s) you'd like to use to define them. Just like in Grid-
Stat
> and
> > Point-Stat, you define the categorical thresholds using the
"cat_thresh"
> > config file entry:
> >
> > cat_thresh = [ >0, >2.54, >25.4 ];
> >
> > Just fill in whatever categorical thresholds you'd like to use.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> > On Mon, Oct 22, 2018 at 11:40 AM Guohui Wang - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
> > >
> > > Good afternoon, John,
> > >
> > > Thank you for replying back quickly.
> > >
> > > You can see that I defined cat_thresh and cat_thresh from line
55 and
> 56
> > > in the config file. So for series analysis, there is no need to
specify
> > > cnt_thresh and cat_thresh as we did in grid_stat?
> > >
> > > Guohui
> > >
> > > On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Guohui,
> > > >
> > > > I see that you have a question about the Series-Analysis tool
in MET.
> > To
> > > > answer your question, let me clarify a couple of details.
> > > >
> > > > The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
> > > Ensemble-Stat,
> > > > and Wavelet-Stat) all write output files named ".stat". The
".stat"
> > > output
> > > > files contain multiple line types, each containing a different
type
> of
> > > > output verification statistics.
> > > >
> > > > These lines consists of 2 parts... The first 22 columns are
the
> header
> > > > columns and are common to all line types. They contain
metadata
> about
> > > the
> > > > forecast and observation data being compared, timing
information, and
> > > > threshold information. This is metadata about the
verification
> > > comparison.
> > > >
> > > > Columns 23 to the end of the line are specific to each line
type.
> This
> > > is
> > > > the actual verification data. For example, the CNT line type
> contains
> > > the
> > > > RMSE (Root-Mean-Squared-Error) column.
> > > >
> > > > The Grid-Stat and Point-Stat tools are run once per
verification time
> > and
> > > > compute statistics that are computed over some spatial area.
The
> > > > Series-Analysis tool, on the other hand, computes statistics
for each
> > > > individual grid point... and those stats are typically
computed over
> a
> > > time
> > > > series of data.
> > > >
> > > > In the Series-Analysis config file, the "output_stats" config
file
> > entry
> > > > defines the statistics that should be computed for each grid
point.
> > > There
> > > > is a separate entry for each line type. You simply specify
the name
> of
> > > the
> > > > column for the statistic(s) you'd like to see computed in the
output.
> > So
> > > > these should be the column names for columns 23 and beyond.
> > > >
> > > > As you found, if you request columns 1 through 22, you'll get
an
> error.
> > > > Those contain metadata rather than the actual computed
statistics in
> > > > columns 23 and beyond.
> > > >
> > > > Hopefully that clarifies the issue.
> > > >
> > > > If not, please let me know.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > > On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: question about series_analysis_config
> > > > > Owner: Nobody
> > > > > Requestors: guohui.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
> > > >
> > > > >
> > > > >
> > > > > Good morning,
> > > > >
> > > > > I am using MET : series analysis tool to do monthly analysis
> between
> > > urma
> > > > > and other forecast models.
> > > > >
> > > > > I saw column "FCST_THRESH" in the header information, but if
I
> added
> > > > > "FCST_THRESH" to series_analysis_config, I got error "
unsupported
> > > column
> > > > > name requested "FCST_THRESH""
> > > > >
> > > > > Would you please explain why this happens?
> > > > >
> > > > > Thank you
> > > > >
> > > > > Guohui1
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: Guohui Wang - NOAA Affiliate
Time: Tue Oct 23 10:01:01 2018
Got it. I will try and keep you updated
Thank you
Guohui
On Tue, Oct 23, 2018 at 11:53 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Guohui,
>
> Yes, I see your confusion. Please define these thresholds at the
higher
> level of context, like this:
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
>=6.35&&<12.7,
> >=12.7&&<25.4, >=25.4&&<76.2, >=76.2&&<127, >=127 ];
> cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=25.4, >=50.8,
>=76.2,
> >=127 ];
> field = [
> {
> name = "APCP_06";
> level = "(*,*)";
> }
> ];
>
> }
> obs = fcst;
>
> Since the Grid-Stat and Point-Stat tools can be run to evaluate
multiple
> fields at a time, the cnt_thresh and cat_thresh settings are parsed
> separately for each entry in the "field" array. Series-Analysis, on
the
> other hand, can only be used to define one series at a time. For
that
> reason, the code parses them from a higher level of context. While
they
> can be set differently for "fcst" and "obs", they can't be set
differently
> for each entry in the "field" array.
>
> So you should either define them inside the "fcst" dictionary (as
shown
> above) or at the global level of context (meaning, outside the
"fcst"
> dictionary).
>
> Please give that a shot and let me know if you have more questions.
>
> Thanks,
> John
>
> On Tue, Oct 23, 2018 at 8:31 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
> >
> > Good morning, John,
> >
> > I saw there are two different place to define cat_thresh and cnt
thresh:
> > one is in fcst definition and one is out. I would think that the
one in
> > fcst definition will cover the one outside. But if I define my
cat_thresh
> > as follows,
> >
> >
> >
>
*************************************************************************************************************************************************
> > cat_thresh = [ NA ];
> > cnt_thresh = [ NA ];
> > cnt_logic = UNION;
> >
> > /////////// from grid_stat_config
> > wind_thresh = [ NA ];
> > wind_logic = UNION;
> >
> > //
> > // Forecast and observation fields to be verified
> > //
> > fcst = {
> > field = [
> > {
> > name = "APCP_06";
> > level = "(*,*)";
> > cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > >=6.35&&<12.7, >=12.7&&<25.4, >=25.4&&<76.2, >=76.2&&<127, >=127
];
> > cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=25.4,
>=50.8,
> > >=76.2, >=127 ];
> > }
> > ];
> >
> > }
> > obs = fcst;
> >
> >
>
*************************************************************************************************************************************************
> > I saw the fcst_thresh of variables in output netcdf file is "NA".
Do you
> > know why it is like this? Or, how can I check variable values per
> > thresh-hold value?
> >
> > float series_cnt_RMSE_BCU(lat, lon) ;
> > series_cnt_RMSE_BCU:_FillValue = -9999.f ;
> > series_cnt_RMSE_BCU:name = "RMSE_BCU" ;
> > series_cnt_RMSE_BCU:long_name = "Root mean squared
error
> > bootstrap confidence interval upper limit" ;
> > series_cnt_RMSE_BCU:fcst_thresh = "NA" ;
> > series_cnt_RMSE_BCU:obs_thresh = "NA" ;
> >
> > Thank you
> >
> > Guohui
> >
> > On Mon, Oct 22, 2018 at 3:07 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Guohui,
> > >
> > > Let's say that you're running Series-Analysis to verify 3-hourly
> > > accumulated precip. So in the config file, you've set:
> > >
> > > fcst = {
> > > field = [ { name = "APCP"; level = [ "A03" ]; } ];
> > > }
> > > obs = fcst;
> > >
> > > And you want to compute CSI and GSS (which is equitable threat
score).
> > In
> > > the config file, you'd request those in the "output_stats"
entry:
> > >
> > > output_stats = {
> > > ...
> > > cts = [ "TOTAL", "CSI", "GSS" ];
> > > ...
> > > }
> > >
> > > Note that the TOTAL column is just the number of matched pairs
in the
> > time
> > > series. I recommend that people look at that to make sure that
they
> > don't
> > > have any unexpected areas of missing data.
> > >
> > > Since these are categorical statistics, you do need to define
the
> > > threshold(s) you'd like to use to define them. Just like in
Grid-Stat
> > and
> > > Point-Stat, you define the categorical thresholds using the
> "cat_thresh"
> > > config file entry:
> > >
> > > cat_thresh = [ >0, >2.54, >25.4 ];
> > >
> > > Just fill in whatever categorical thresholds you'd like to use.
> > >
> > > Make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Oct 22, 2018 at 11:40 AM Guohui Wang - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
>
> > > >
> > > > Good afternoon, John,
> > > >
> > > > Thank you for replying back quickly.
> > > >
> > > > You can see that I defined cat_thresh and cat_thresh from
line 55
> and
> > 56
> > > > in the config file. So for series analysis, there is no need
to
> specify
> > > > cnt_thresh and cat_thresh as we did in grid_stat?
> > > >
> > > > Guohui
> > > >
> > > > On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Guohui,
> > > > >
> > > > > I see that you have a question about the Series-Analysis
tool in
> MET.
> > > To
> > > > > answer your question, let me clarify a couple of details.
> > > > >
> > > > > The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
> > > > Ensemble-Stat,
> > > > > and Wavelet-Stat) all write output files named ".stat". The
> ".stat"
> > > > output
> > > > > files contain multiple line types, each containing a
different type
> > of
> > > > > output verification statistics.
> > > > >
> > > > > These lines consists of 2 parts... The first 22 columns are
the
> > header
> > > > > columns and are common to all line types. They contain
metadata
> > about
> > > > the
> > > > > forecast and observation data being compared, timing
information,
> and
> > > > > threshold information. This is metadata about the
verification
> > > > comparison.
> > > > >
> > > > > Columns 23 to the end of the line are specific to each line
type.
> > This
> > > > is
> > > > > the actual verification data. For example, the CNT line
type
> > contains
> > > > the
> > > > > RMSE (Root-Mean-Squared-Error) column.
> > > > >
> > > > > The Grid-Stat and Point-Stat tools are run once per
verification
> time
> > > and
> > > > > compute statistics that are computed over some spatial area.
The
> > > > > Series-Analysis tool, on the other hand, computes statistics
for
> each
> > > > > individual grid point... and those stats are typically
computed
> over
> > a
> > > > time
> > > > > series of data.
> > > > >
> > > > > In the Series-Analysis config file, the "output_stats"
config file
> > > entry
> > > > > defines the statistics that should be computed for each grid
point.
> > > > There
> > > > > is a separate entry for each line type. You simply specify
the
> name
> > of
> > > > the
> > > > > column for the statistic(s) you'd like to see computed in
the
> output.
> > > So
> > > > > these should be the column names for columns 23 and beyond.
> > > > >
> > > > > As you found, if you request columns 1 through 22, you'll
get an
> > error.
> > > > > Those contain metadata rather than the actual computed
statistics
> in
> > > > > columns 23 and beyond.
> > > > >
> > > > > Hopefully that clarifies the issue.
> > > > >
> > > > > If not, please let me know.
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > >
> > > > >
> > > > > On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate
via
> RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: question about series_analysis_config
> > > > > > Owner: Nobody
> > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning,
> > > > > >
> > > > > > I am using MET : series analysis tool to do monthly
analysis
> > between
> > > > urma
> > > > > > and other forecast models.
> > > > > >
> > > > > > I saw column "FCST_THRESH" in the header information, but
if I
> > added
> > > > > > "FCST_THRESH" to series_analysis_config, I got error "
> unsupported
> > > > column
> > > > > > name requested "FCST_THRESH""
> > > > > >
> > > > > > Would you please explain why this happens?
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > Guohui1
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: Guohui Wang - NOAA Affiliate
Time: Wed Oct 24 08:55:57 2018
Good morning, John,
I moved the cnt_thresh and cat_thresh to upper level and got correct
output
now. Thank you very much for your instruction!
Another confusion I got is that you think "The tools in MET that end
in
"-Stat" (Grid-Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat) all
write
output files named ".stat". The ".stat" outputfiles contain multiple
line
types, each containing a different type of output verification
statistics."
But I read from MET manual that "The output of Grid-Stat is one or
more
ASCII files containing statistics summarizing the verification
performed
and a NetCDF file containing difference fields", that's also what I
got as
output of Grid_Stat tool. Are these ".stat" outputfiles the same
files as
"grid_stat_xxxxxx.txt"?
Thanks
Guohui
On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> I see that you have a question about the Series-Analysis tool in
MET. To
> answer your question, let me clarify a couple of details.
>
> The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
Ensemble-Stat,
> and Wavelet-Stat) all write output files named ".stat". The ".stat"
output
> files contain multiple line types, each containing a different type
of
> output verification statistics.
>
> These lines consists of 2 parts... The first 22 columns are the
header
> columns and are common to all line types. They contain metadata
about the
> forecast and observation data being compared, timing information,
and
> threshold information. This is metadata about the verification
comparison.
>
> Columns 23 to the end of the line are specific to each line type.
This is
> the actual verification data. For example, the CNT line type
contains the
> RMSE (Root-Mean-Squared-Error) column.
>
> The Grid-Stat and Point-Stat tools are run once per verification
time and
> compute statistics that are computed over some spatial area. The
> Series-Analysis tool, on the other hand, computes statistics for
each
> individual grid point... and those stats are typically computed over
a time
> series of data.
>
> In the Series-Analysis config file, the "output_stats" config file
entry
> defines the statistics that should be computed for each grid point.
There
> is a separate entry for each line type. You simply specify the name
of the
> column for the statistic(s) you'd like to see computed in the
output. So
> these should be the column names for columns 23 and beyond.
>
> As you found, if you request columns 1 through 22, you'll get an
error.
> Those contain metadata rather than the actual computed statistics in
> columns 23 and beyond.
>
> Hopefully that clarifies the issue.
>
> If not, please let me know.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> > Transaction: Ticket created by guohui.wang at noaa.gov
> > Queue: met_help
> > Subject: question about series_analysis_config
> > Owner: Nobody
> > Requestors: guohui.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
> >
> >
> > Good morning,
> >
> > I am using MET : series analysis tool to do monthly analysis
between urma
> > and other forecast models.
> >
> > I saw column "FCST_THRESH" in the header information, but if I
added
> > "FCST_THRESH" to series_analysis_config, I got error " unsupported
column
> > name requested "FCST_THRESH""
> >
> > Would you please explain why this happens?
> >
> > Thank you
> >
> > Guohui1
> >
> >
>
>
------------------------------------------------
Subject: question about series_analysis_config
From: Guohui Wang - NOAA Affiliate
Time: Wed Oct 24 09:31:07 2018
John,
Please ignore my last email. I found stat files in our grid_stat
output
folder.
Thank you
Guohui
On Wed, Oct 24, 2018 at 10:55 AM Guohui Wang - NOAA Affiliate <
guohui.wang at noaa.gov> wrote:
> Good morning, John,
>
> I moved the cnt_thresh and cat_thresh to upper level and got correct
> output now. Thank you very much for your instruction!
>
> Another confusion I got is that you think "The tools in MET that end
in
> "-Stat" (Grid-Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat) all
write
> output files named ".stat". The ".stat" outputfiles contain
multiple line
> types, each containing a different type of output verification
statistics."
> But I read from MET manual that "The output of Grid-Stat is one or
more
> ASCII files containing statistics summarizing the verification
performed
> and a NetCDF file containing difference fields", that's also what I
got as
> output of Grid_Stat tool. Are these ".stat" outputfiles the same
files as
> "grid_stat_xxxxxx.txt"?
>
> Thanks
>
> Guohui
>
> On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Guohui,
>>
>> I see that you have a question about the Series-Analysis tool in
MET. To
>> answer your question, let me clarify a couple of details.
>>
>> The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
>> Ensemble-Stat,
>> and Wavelet-Stat) all write output files named ".stat". The
".stat"
>> output
>> files contain multiple line types, each containing a different type
of
>> output verification statistics.
>>
>> These lines consists of 2 parts... The first 22 columns are the
header
>> columns and are common to all line types. They contain metadata
about the
>> forecast and observation data being compared, timing information,
and
>> threshold information. This is metadata about the verification
>> comparison.
>>
>> Columns 23 to the end of the line are specific to each line type.
This is
>> the actual verification data. For example, the CNT line type
contains the
>> RMSE (Root-Mean-Squared-Error) column.
>>
>> The Grid-Stat and Point-Stat tools are run once per verification
time and
>> compute statistics that are computed over some spatial area. The
>> Series-Analysis tool, on the other hand, computes statistics for
each
>> individual grid point... and those stats are typically computed
over a
>> time
>> series of data.
>>
>> In the Series-Analysis config file, the "output_stats" config file
entry
>> defines the statistics that should be computed for each grid point.
There
>> is a separate entry for each line type. You simply specify the
name of
>> the
>> column for the statistic(s) you'd like to see computed in the
output. So
>> these should be the column names for columns 23 and beyond.
>>
>> As you found, if you request columns 1 through 22, you'll get an
error.
>> Those contain metadata rather than the actual computed statistics
in
>> columns 23 and beyond.
>>
>> Hopefully that clarifies the issue.
>>
>> If not, please let me know.
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
>> > Transaction: Ticket created by guohui.wang at noaa.gov
>> > Queue: met_help
>> > Subject: question about series_analysis_config
>> > Owner: Nobody
>> > Requestors: guohui.wang at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
>> >
>> >
>> > Good morning,
>> >
>> > I am using MET : series analysis tool to do monthly analysis
between
>> urma
>> > and other forecast models.
>> >
>> > I saw column "FCST_THRESH" in the header information, but if I
added
>> > "FCST_THRESH" to series_analysis_config, I got error "
unsupported
>> column
>> > name requested "FCST_THRESH""
>> >
>> > Would you please explain why this happens?
>> >
>> > Thank you
>> >
>> > Guohui1
>> >
>> >
>>
>>
------------------------------------------------
Subject: question about series_analysis_config
From: John Halley Gotway
Time: Wed Oct 24 09:39:57 2018
Guohui,
Just to clarify how the output of Grid-Stat is organized.
You setup the "output_flag" setting in the Grid-Stat config file.
That
tells Grid-Stat which line types you'd like it to compute, settings
them to
NONE, STAT, or BOTH.
All of the line types set to STAT or BOTH are written to the
"grid_stat_*.stat" output file.
In addition, line types set to BOTH are also written out the
corresponding
"grid_stat_*_LTY.txt" where LTY is the current line type.
When you set a line type to BOTH the output is redundant since the
same
thing is written to 2 output files. However the "grid_stat_*_LTY.txt"
text
files are much more human readable. All the data is in the
"grid_stat_*.stat" file but it's difficult to read since all the line
types
of varying lengths are jumbled together.
Hope that helps clarify.
Thanks,
John
On Wed, Oct 24, 2018 at 9:31 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466 >
>
> John,
>
> Please ignore my last email. I found stat files in our grid_stat
output
> folder.
>
> Thank you
>
> Guohui
>
> On Wed, Oct 24, 2018 at 10:55 AM Guohui Wang - NOAA Affiliate <
> guohui.wang at noaa.gov> wrote:
>
> > Good morning, John,
> >
> > I moved the cnt_thresh and cat_thresh to upper level and got
correct
> > output now. Thank you very much for your instruction!
> >
> > Another confusion I got is that you think "The tools in MET that
end in
> > "-Stat" (Grid-Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat)
all
> write
> > output files named ".stat". The ".stat" outputfiles contain
multiple
> line
> > types, each containing a different type of output verification
> statistics."
> > But I read from MET manual that "The output of Grid-Stat is one or
more
> > ASCII files containing statistics summarizing the verification
performed
> > and a NetCDF file containing difference fields", that's also what
I got
> as
> > output of Grid_Stat tool. Are these ".stat" outputfiles the same
files
> as
> > "grid_stat_xxxxxx.txt"?
> >
> > Thanks
> >
> > Guohui
> >
> > On Mon, Oct 22, 2018 at 1:05 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Guohui,
> >>
> >> I see that you have a question about the Series-Analysis tool in
MET.
> To
> >> answer your question, let me clarify a couple of details.
> >>
> >> The tools in MET that end in "-Stat" (Grid-Stat, Point-Stat,
> >> Ensemble-Stat,
> >> and Wavelet-Stat) all write output files named ".stat". The
".stat"
> >> output
> >> files contain multiple line types, each containing a different
type of
> >> output verification statistics.
> >>
> >> These lines consists of 2 parts... The first 22 columns are the
header
> >> columns and are common to all line types. They contain metadata
about
> the
> >> forecast and observation data being compared, timing information,
and
> >> threshold information. This is metadata about the verification
> >> comparison.
> >>
> >> Columns 23 to the end of the line are specific to each line type.
This
> is
> >> the actual verification data. For example, the CNT line type
contains
> the
> >> RMSE (Root-Mean-Squared-Error) column.
> >>
> >> The Grid-Stat and Point-Stat tools are run once per verification
time
> and
> >> compute statistics that are computed over some spatial area. The
> >> Series-Analysis tool, on the other hand, computes statistics for
each
> >> individual grid point... and those stats are typically computed
over a
> >> time
> >> series of data.
> >>
> >> In the Series-Analysis config file, the "output_stats" config
file entry
> >> defines the statistics that should be computed for each grid
point.
> There
> >> is a separate entry for each line type. You simply specify the
name of
> >> the
> >> column for the statistic(s) you'd like to see computed in the
output.
> So
> >> these should be the column names for columns 23 and beyond.
> >>
> >> As you found, if you request columns 1 through 22, you'll get an
error.
> >> Those contain metadata rather than the actual computed statistics
in
> >> columns 23 and beyond.
> >>
> >> Hopefully that clarifies the issue.
> >>
> >> If not, please let me know.
> >>
> >> Thanks,
> >> John Halley Gotway
> >>
> >>
> >> On Mon, Oct 22, 2018 at 9:33 AM Guohui Wang - NOAA Affiliate via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > Mon Oct 22 09:33:45 2018: Request 87466 was acted upon.
> >> > Transaction: Ticket created by guohui.wang at noaa.gov
> >> > Queue: met_help
> >> > Subject: question about series_analysis_config
> >> > Owner: Nobody
> >> > Requestors: guohui.wang at noaa.gov
> >> > Status: new
> >> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87466
> >
> >> >
> >> >
> >> > Good morning,
> >> >
> >> > I am using MET : series analysis tool to do monthly analysis
between
> >> urma
> >> > and other forecast models.
> >> >
> >> > I saw column "FCST_THRESH" in the header information, but if I
added
> >> > "FCST_THRESH" to series_analysis_config, I got error "
unsupported
> >> column
> >> > name requested "FCST_THRESH""
> >> >
> >> > Would you please explain why this happens?
> >> >
> >> > Thank you
> >> >
> >> > Guohui1
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
More information about the Met_help
mailing list