[Met_help] [rt.rap.ucar.edu #87558] History for Question about stat_analysis tool
John Halley Gotway via RT
met_help at ucar.edu
Mon Nov 26 11:04:28 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
We are trying to do monthly analysis on CTC outline type using
stat_analysis tool.
This is how I am trying to work this out:
Based on slide 5 in Tara's PPT (
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf),
in order to output CTC using stat_analysis tool, we have to generate MRP
line type using grid_stat tool. In MET user guide, I could not find
information of MPR in grid_stat tool, but on your website, Grid-stat tool
can
- Write output to ASCII statistics files and write matched pairs to
NetCDF file
Shall I just add MPR as other output types into output_flag definition in
grid_stat_config? maybe like:
output_flag = {
fho = NONE;
ctc = BOTH;
cts = BOTH;
mctc = NONE;
mcts = BOTH;
cnt = BOTH;
sl1l2 = BOTH;
sal1l2 = NONE;
vl1l2 = NONE;
val1l2 = NONE;
*MPR = BOTH;*
pct = NONE;
pstd = NONE;
pjc = NONE;
prc = NONE;
nbrctc = NONE;
nbrcts = NONE;
nbrcnt = NONE;
}
Thank you for your help
Guohui
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Question about stat_analysis tool
From: John Halley Gotway
Time: Mon Oct 29 09:58:16 2018
Guohui,
The Point-Stat tool writes the MPR line type to the ascii output file
ending in .stat. The Grid-Stat does *not* because the MPR line type
is a
very inefficient way of storing data... especially gridded data.
Grid-Stat
can write the pair information to the a NetCDF output file. However,
STAT-Analysis doesn't know how to read that NetCDF file. STAT-
Analysis
only reads the ascii ".stat" files.
But let's take a step back and figure out what you're trying to do. I
understand that you want to compute monthly summary statistics... but
a
monthly summary of what?
(1) CTC lines output from Grid-Stat that were computed over some
spatial
area?
(2) Doing a grid-to-grid comparison and computing CTS statistics
separately
for each grid point over a month's worth of data?
(3) Doing a grid-to-point comparison and computing CTS statistics
separately for each observation station?
If you can describe to me the type of data you have available and what
the
analysis you'd like to do, I should be able to recommend which MET
tools to
use.
Thanks,
John Halley Gotway
On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> Transaction: Ticket created by guohui.wang at noaa.gov
> Queue: met_help
> Subject: Question about stat_analysis tool
> Owner: Nobody
> Requestors: guohui.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
>
>
> Good morning,
>
> We are trying to do monthly analysis on CTC outline type using
> stat_analysis tool.
>
> This is how I am trying to work this out:
>
> Based on slide 5 in Tara's PPT (
>
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> ),
> in order to output CTC using stat_analysis tool, we have to generate
MRP
> line type using grid_stat tool. In MET user guide, I could not find
> information of MPR in grid_stat tool, but on your website, Grid-stat
tool
> can
>
> - Write output to ASCII statistics files and write matched pairs
to
> NetCDF file
>
> Shall I just add MPR as other output types into output_flag
definition in
> grid_stat_config? maybe like:
>
> output_flag = {
> fho = NONE;
> ctc = BOTH;
> cts = BOTH;
> mctc = NONE;
> mcts = BOTH;
> cnt = BOTH;
> sl1l2 = BOTH;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> *MPR = BOTH;*
> pct = NONE;
> pstd = NONE;
> pjc = NONE;
> prc = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = NONE;
> }
>
> Thank you for your help
>
> Guohui
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: Guohui Wang - NOAA Affiliate
Time: Mon Oct 29 10:14:33 2018
Good morning, John,
we have NetCDF data for analysis model(model) and a few forecast
models
like "HRRR", "NDFD", etc. We could be able to grid_stat tool to
generate
cnt, cts, and ctc as needed.
We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" for
stat_analysis in case that people are interested in looking at
contingency
table by monthly base. So, yes, we'd like option 1. It would be nice
if
there is a way to get it .
Thank you
Guohui
On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Guohui,
>
> The Point-Stat tool writes the MPR line type to the ascii output
file
> ending in .stat. The Grid-Stat does *not* because the MPR line type
is a
> very inefficient way of storing data... especially gridded data.
Grid-Stat
> can write the pair information to the a NetCDF output file.
However,
> STAT-Analysis doesn't know how to read that NetCDF file. STAT-
Analysis
> only reads the ascii ".stat" files.
>
> But let's take a step back and figure out what you're trying to do.
I
> understand that you want to compute monthly summary statistics...
but a
> monthly summary of what?
>
> (1) CTC lines output from Grid-Stat that were computed over some
spatial
> area?
> (2) Doing a grid-to-grid comparison and computing CTS statistics
separately
> for each grid point over a month's worth of data?
> (3) Doing a grid-to-point comparison and computing CTS statistics
> separately for each observation station?
>
> If you can describe to me the type of data you have available and
what the
> analysis you'd like to do, I should be able to recommend which MET
tools to
> use.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > Transaction: Ticket created by guohui.wang at noaa.gov
> > Queue: met_help
> > Subject: Question about stat_analysis tool
> > Owner: Nobody
> > Requestors: guohui.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> >
> >
> > Good morning,
> >
> > We are trying to do monthly analysis on CTC outline type using
> > stat_analysis tool.
> >
> > This is how I am trying to work this out:
> >
> > Based on slide 5 in Tara's PPT (
> >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > ),
> > in order to output CTC using stat_analysis tool, we have to
generate MRP
> > line type using grid_stat tool. In MET user guide, I could not
find
> > information of MPR in grid_stat tool, but on your website, Grid-
stat tool
> > can
> >
> > - Write output to ASCII statistics files and write matched
pairs to
> > NetCDF file
> >
> > Shall I just add MPR as other output types into output_flag
definition in
> > grid_stat_config? maybe like:
> >
> > output_flag = {
> > fho = NONE;
> > ctc = BOTH;
> > cts = BOTH;
> > mctc = NONE;
> > mcts = BOTH;
> > cnt = BOTH;
> > sl1l2 = BOTH;
> > sal1l2 = NONE;
> > vl1l2 = NONE;
> > val1l2 = NONE;
> > *MPR = BOTH;*
> > pct = NONE;
> > pstd = NONE;
> > pjc = NONE;
> > prc = NONE;
> > nbrctc = NONE;
> > nbrcts = NONE;
> > nbrcnt = NONE;
> > }
> >
> > Thank you for your help
> >
> > Guohui
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: John Halley Gotway
Time: Mon Oct 29 10:27:10 2018
Guohui,
Yes, definitely. So here's how the data flow would work:
(1) Each day, you run Grid-Stat to compare the analysis to HRRR, NDFD,
and
so on. The Grid-Stat for each run is written to a ".stat" file.
(2) After you have a month or so of data, you can run the STAT-
Analysis
tool to summarize the results through time. You'll run an
"aggregate_stat"
job which reads CTC lines as input and writes CTS lines as output.
Something like this type of job...
stat_analysis -lookin grid_stat_HRRR -job aggregate_stat -line_type
CTC
-out_line_type CTS \
-fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
-by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
-out_stat 2018NovStats.stat
This job will...
(1) Search recursively through an input directory named
"grid_stat_HRRR"
and read any ".stat" files it finds.
(2) Read CTC input lines... aggregate them... and write CTS output
lines.
(3) Only use lines whose valid time is between 20181101 and 20181130.
(4) Produce separate output CTS lines for each unique combination of
the
following set of MET header columns:
MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
(5) Write output to file named 2018NovStats.stat.
If you're running a single job like this, it's fine to do it on the
command
line. If you'd like to run multiple jobs all on the same set of input
data, it may be more efficient to do so using a configuration file for
STAT-Analysis.
FYI, here's online tutorial information about the STAT-Analysis tool:
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
Thanks,
John
On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
>
> Good morning, John,
>
> we have NetCDF data for analysis model(model) and a few forecast
models
> like "HRRR", "NDFD", etc. We could be able to grid_stat tool to
generate
> cnt, cts, and ctc as needed.
>
> We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" for
> stat_analysis in case that people are interested in looking at
contingency
> table by monthly base. So, yes, we'd like option 1. It would be nice
if
> there is a way to get it .
>
> Thank you
>
> Guohui
>
> On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Guohui,
> >
> > The Point-Stat tool writes the MPR line type to the ascii output
file
> > ending in .stat. The Grid-Stat does *not* because the MPR line
type is a
> > very inefficient way of storing data... especially gridded data.
> Grid-Stat
> > can write the pair information to the a NetCDF output file.
However,
> > STAT-Analysis doesn't know how to read that NetCDF file. STAT-
Analysis
> > only reads the ascii ".stat" files.
> >
> > But let's take a step back and figure out what you're trying to
do. I
> > understand that you want to compute monthly summary statistics...
but a
> > monthly summary of what?
> >
> > (1) CTC lines output from Grid-Stat that were computed over some
spatial
> > area?
> > (2) Doing a grid-to-grid comparison and computing CTS statistics
> separately
> > for each grid point over a month's worth of data?
> > (3) Doing a grid-to-point comparison and computing CTS statistics
> > separately for each observation station?
> >
> > If you can describe to me the type of data you have available and
what
> the
> > analysis you'd like to do, I should be able to recommend which MET
tools
> to
> > use.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > Queue: met_help
> > > Subject: Question about stat_analysis tool
> > > Owner: Nobody
> > > Requestors: guohui.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> >
> > >
> > >
> > > Good morning,
> > >
> > > We are trying to do monthly analysis on CTC outline type using
> > > stat_analysis tool.
> > >
> > > This is how I am trying to work this out:
> > >
> > > Based on slide 5 in Tara's PPT (
> > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > ),
> > > in order to output CTC using stat_analysis tool, we have to
generate
> MRP
> > > line type using grid_stat tool. In MET user guide, I could not
find
> > > information of MPR in grid_stat tool, but on your website, Grid-
stat
> tool
> > > can
> > >
> > > - Write output to ASCII statistics files and write matched
pairs to
> > > NetCDF file
> > >
> > > Shall I just add MPR as other output types into output_flag
definition
> in
> > > grid_stat_config? maybe like:
> > >
> > > output_flag = {
> > > fho = NONE;
> > > ctc = BOTH;
> > > cts = BOTH;
> > > mctc = NONE;
> > > mcts = BOTH;
> > > cnt = BOTH;
> > > sl1l2 = BOTH;
> > > sal1l2 = NONE;
> > > vl1l2 = NONE;
> > > val1l2 = NONE;
> > > *MPR = BOTH;*
> > > pct = NONE;
> > > pstd = NONE;
> > > pjc = NONE;
> > > prc = NONE;
> > > nbrctc = NONE;
> > > nbrcts = NONE;
> > > nbrcnt = NONE;
> > > }
> > >
> > > Thank you for your help
> > >
> > > Guohui
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: Guohui Wang - NOAA Affiliate
Time: Tue Oct 30 12:50:12 2018
Good afternoon, John,
Thank you for the details about stat_analysis. I could generate
contingency
variables in CTS files .
Another question about running stat_analysis tool is how to set fcst
var
option: I would like to run the tool on following variables "
cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
"FAR_NCU",
"CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
"HSS_BCL",
"HSS_BCU", "TOTAL" ];
cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL", "MAE_BCU",
"MSE",
"MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
"
If I don't add fcst_var in job option: -job aggregate_stat -dump_row
$statOutFile $maskList -line_type SL1L2 -out_line_type CNT -v 4, I
got
output columns like "
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE
VX_MASK INTERP_MTHD
INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA
LINE_TYPE TOTAL FBAR OBAR FOBAR FFBAR
OOBAR MAE
"
If I add fcst_var to set job like following: -job aggregate_stat
-fcst_var
ME -fcst_var MAE -dump_row $statOutFile $maskList -line_type SL1L2
-out_line_type CNT -v 4, I got no data in output file.
Do you know what I did wrong?
Thanks
Guohui
On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Guohui,
>
> Yes, definitely. So here's how the data flow would work:
>
> (1) Each day, you run Grid-Stat to compare the analysis to HRRR,
NDFD, and
> so on. The Grid-Stat for each run is written to a ".stat" file.
>
> (2) After you have a month or so of data, you can run the STAT-
Analysis
> tool to summarize the results through time. You'll run an
"aggregate_stat"
> job which reads CTC lines as input and writes CTS lines as output.
>
> Something like this type of job...
> stat_analysis -lookin grid_stat_HRRR -job aggregate_stat
-line_type CTC
> -out_line_type CTS \
> -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> -out_stat 2018NovStats.stat
>
> This job will...
>
> (1) Search recursively through an input directory named
"grid_stat_HRRR"
> and read any ".stat" files it finds.
> (2) Read CTC input lines... aggregate them... and write CTS output
lines.
> (3) Only use lines whose valid time is between 20181101 and
20181130.
> (4) Produce separate output CTS lines for each unique combination of
the
> following set of MET header columns:
> MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> (5) Write output to file named 2018NovStats.stat.
>
> If you're running a single job like this, it's fine to do it on the
command
> line. If you'd like to run multiple jobs all on the same set of
input
> data, it may be more efficient to do so using a configuration file
for
> STAT-Analysis.
>
> FYI, here's online tutorial information about the STAT-Analysis
tool:
>
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> >
> > Good morning, John,
> >
> > we have NetCDF data for analysis model(model) and a few forecast
models
> > like "HRRR", "NDFD", etc. We could be able to grid_stat tool to
generate
> > cnt, cts, and ctc as needed.
> >
> > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" for
> > stat_analysis in case that people are interested in looking at
> contingency
> > table by monthly base. So, yes, we'd like option 1. It would be
nice if
> > there is a way to get it .
> >
> > Thank you
> >
> > Guohui
> >
> > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Guohui,
> > >
> > > The Point-Stat tool writes the MPR line type to the ascii output
file
> > > ending in .stat. The Grid-Stat does *not* because the MPR line
type
> is a
> > > very inefficient way of storing data... especially gridded data.
> > Grid-Stat
> > > can write the pair information to the a NetCDF output file.
However,
> > > STAT-Analysis doesn't know how to read that NetCDF file. STAT-
Analysis
> > > only reads the ascii ".stat" files.
> > >
> > > But let's take a step back and figure out what you're trying to
do. I
> > > understand that you want to compute monthly summary
statistics... but a
> > > monthly summary of what?
> > >
> > > (1) CTC lines output from Grid-Stat that were computed over some
> spatial
> > > area?
> > > (2) Doing a grid-to-grid comparison and computing CTS statistics
> > separately
> > > for each grid point over a month's worth of data?
> > > (3) Doing a grid-to-point comparison and computing CTS
statistics
> > > separately for each observation station?
> > >
> > > If you can describe to me the type of data you have available
and what
> > the
> > > analysis you'd like to do, I should be able to recommend which
MET
> tools
> > to
> > > use.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: Question about stat_analysis tool
> > > > Owner: Nobody
> > > > Requestors: guohui.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > >
> > > >
> > > >
> > > > Good morning,
> > > >
> > > > We are trying to do monthly analysis on CTC outline type using
> > > > stat_analysis tool.
> > > >
> > > > This is how I am trying to work this out:
> > > >
> > > > Based on slide 5 in Tara's PPT (
> > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > ),
> > > > in order to output CTC using stat_analysis tool, we have to
generate
> > MRP
> > > > line type using grid_stat tool. In MET user guide, I could
not find
> > > > information of MPR in grid_stat tool, but on your website,
Grid-stat
> > tool
> > > > can
> > > >
> > > > - Write output to ASCII statistics files and write matched
pairs
> to
> > > > NetCDF file
> > > >
> > > > Shall I just add MPR as other output types into output_flag
> definition
> > in
> > > > grid_stat_config? maybe like:
> > > >
> > > > output_flag = {
> > > > fho = NONE;
> > > > ctc = BOTH;
> > > > cts = BOTH;
> > > > mctc = NONE;
> > > > mcts = BOTH;
> > > > cnt = BOTH;
> > > > sl1l2 = BOTH;
> > > > sal1l2 = NONE;
> > > > vl1l2 = NONE;
> > > > val1l2 = NONE;
> > > > *MPR = BOTH;*
> > > > pct = NONE;
> > > > pstd = NONE;
> > > > pjc = NONE;
> > > > prc = NONE;
> > > > nbrctc = NONE;
> > > > nbrcts = NONE;
> > > > nbrcnt = NONE;
> > > > }
> > > >
> > > > Thank you for your help
> > > >
> > > > Guohui
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: John Halley Gotway
Time: Wed Oct 31 16:11:37 2018
Guohui,
It looks like we're getting a little confused. Let me start by
clarifying
some terminology.
By fcst_var, I mean the contents of the column named "FCST_VAR" in the
".stat" output files from the MET tools. They should be strings like
"TMP"
for temperature or "PRES" for pressure or "APCP" for temperature.
Abbreviations like ME and MAE stand for "mean error" and "mean
absolute
error". Those are the statistics that show up in the CNT line.
To illustrate, I've attached a sample .stat file that's output from
the
Grid-Stat tool. It contains both continuous and categorical output
for
relative humidity, 2-meter temperature and 10-m winds (i.e. FCST_VAR
column) computed using multiple thresholds (i.e. FCST_THRESH column)
and
over multiple verification regions (i.e. VX_MASK column). Here's a few
STAT-Analysis jobs you could run on it:
# Run aggregate job to aggregate the SL1L2 continuous partial sums
across
all the masking regions separately for each variable
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
SL1L2 -by
FCST_VAR
# Same thing but use aggregate_stat to derive continuous stats instead
AND
WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate_stat -line_type
SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
# Aggregate contingency table counts across regions, separately for
each
variable/threshold
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type CTC
-by
FCST_VAR,FCST_THRESH
# Same as above but derive stats instead
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate_stat -line_type
CTC
-out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat job4.stat
Are these example starting to make sense?
Thanks,
John
On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
>
> Good afternoon, John,
>
> Thank you for the details about stat_analysis. I could generate
contingency
> variables in CTS files .
>
> Another question about running stat_analysis tool is how to set fcst
var
> option: I would like to run the tool on following variables "
> cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
"FAR_NCU",
> "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
"HSS_BCL",
> "HSS_BCU", "TOTAL" ];
> cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL", "MAE_BCU",
"MSE",
> "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
> "
>
> If I don't add fcst_var in job option: -job aggregate_stat -dump_row
> $statOutFile $maskList -line_type SL1L2 -out_line_type CNT -v 4, I
got
> output columns like "
> VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE
> VX_MASK INTERP_MTHD
> INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA
> LINE_TYPE TOTAL FBAR OBAR FOBAR FFBAR
> OOBAR MAE
> "
>
> If I add fcst_var to set job like following: -job aggregate_stat
-fcst_var
> ME -fcst_var MAE -dump_row $statOutFile $maskList -line_type SL1L2
> -out_line_type CNT -v 4, I got no data in output file.
>
> Do you know what I did wrong?
>
> Thanks
>
> Guohui
>
>
>
> On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Guohui,
> >
> > Yes, definitely. So here's how the data flow would work:
> >
> > (1) Each day, you run Grid-Stat to compare the analysis to HRRR,
NDFD,
> and
> > so on. The Grid-Stat for each run is written to a ".stat" file.
> >
> > (2) After you have a month or so of data, you can run the STAT-
Analysis
> > tool to summarize the results through time. You'll run an
> "aggregate_stat"
> > job which reads CTC lines as input and writes CTS lines as output.
> >
> > Something like this type of job...
> > stat_analysis -lookin grid_stat_HRRR -job aggregate_stat
-line_type
> CTC
> > -out_line_type CTS \
> > -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > -out_stat 2018NovStats.stat
> >
> > This job will...
> >
> > (1) Search recursively through an input directory named
"grid_stat_HRRR"
> > and read any ".stat" files it finds.
> > (2) Read CTC input lines... aggregate them... and write CTS output
lines.
> > (3) Only use lines whose valid time is between 20181101 and
20181130.
> > (4) Produce separate output CTS lines for each unique combination
of the
> > following set of MET header columns:
> > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > (5) Write output to file named 2018NovStats.stat.
> >
> > If you're running a single job like this, it's fine to do it on
the
> command
> > line. If you'd like to run multiple jobs all on the same set of
input
> > data, it may be more efficient to do so using a configuration file
for
> > STAT-Analysis.
> >
> > FYI, here's online tutorial information about the STAT-Analysis
tool:
> >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > >
> > > Good morning, John,
> > >
> > > we have NetCDF data for analysis model(model) and a few forecast
models
> > > like "HRRR", "NDFD", etc. We could be able to grid_stat tool to
> generate
> > > cnt, cts, and ctc as needed.
> > >
> > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL"
for
> > > stat_analysis in case that people are interested in looking at
> > contingency
> > > table by monthly base. So, yes, we'd like option 1. It would be
nice if
> > > there is a way to get it .
> > >
> > > Thank you
> > >
> > > Guohui
> > >
> > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Guohui,
> > > >
> > > > The Point-Stat tool writes the MPR line type to the ascii
output file
> > > > ending in .stat. The Grid-Stat does *not* because the MPR
line type
> > is a
> > > > very inefficient way of storing data... especially gridded
data.
> > > Grid-Stat
> > > > can write the pair information to the a NetCDF output file.
However,
> > > > STAT-Analysis doesn't know how to read that NetCDF file.
> STAT-Analysis
> > > > only reads the ascii ".stat" files.
> > > >
> > > > But let's take a step back and figure out what you're trying
to do.
> I
> > > > understand that you want to compute monthly summary
statistics...
> but a
> > > > monthly summary of what?
> > > >
> > > > (1) CTC lines output from Grid-Stat that were computed over
some
> > spatial
> > > > area?
> > > > (2) Doing a grid-to-grid comparison and computing CTS
statistics
> > > separately
> > > > for each grid point over a month's worth of data?
> > > > (3) Doing a grid-to-point comparison and computing CTS
statistics
> > > > separately for each observation station?
> > > >
> > > > If you can describe to me the type of data you have available
and
> what
> > > the
> > > > analysis you'd like to do, I should be able to recommend which
MET
> > tools
> > > to
> > > > use.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Question about stat_analysis tool
> > > > > Owner: Nobody
> > > > > Requestors: guohui.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > >
> > > > >
> > > > >
> > > > > Good morning,
> > > > >
> > > > > We are trying to do monthly analysis on CTC outline type
using
> > > > > stat_analysis tool.
> > > > >
> > > > > This is how I am trying to work this out:
> > > > >
> > > > > Based on slide 5 in Tara's PPT (
> > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > ),
> > > > > in order to output CTC using stat_analysis tool, we have to
> generate
> > > MRP
> > > > > line type using grid_stat tool. In MET user guide, I could
not
> find
> > > > > information of MPR in grid_stat tool, but on your website,
> Grid-stat
> > > tool
> > > > > can
> > > > >
> > > > > - Write output to ASCII statistics files and write
matched pairs
> > to
> > > > > NetCDF file
> > > > >
> > > > > Shall I just add MPR as other output types into output_flag
> > definition
> > > in
> > > > > grid_stat_config? maybe like:
> > > > >
> > > > > output_flag = {
> > > > > fho = NONE;
> > > > > ctc = BOTH;
> > > > > cts = BOTH;
> > > > > mctc = NONE;
> > > > > mcts = BOTH;
> > > > > cnt = BOTH;
> > > > > sl1l2 = BOTH;
> > > > > sal1l2 = NONE;
> > > > > vl1l2 = NONE;
> > > > > val1l2 = NONE;
> > > > > *MPR = BOTH;*
> > > > > pct = NONE;
> > > > > pstd = NONE;
> > > > > pjc = NONE;
> > > > > prc = NONE;
> > > > > nbrctc = NONE;
> > > > > nbrcts = NONE;
> > > > > nbrcnt = NONE;
> > > > > }
> > > > >
> > > > > Thank you for your help
> > > > >
> > > > > Guohui
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: Guohui Wang - NOAA Affiliate
Time: Fri Nov 02 09:41:32 2018
Good morning, John,
Yes, these are great examples. We can run stat_analysis for
aggregate_stat
in the right way.
But beside MSE, we do hope to get aggregate of all the statistical
output
types as follows:
ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" ];
cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
"FAR_NCU",
"CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
"HSS_BCL",
"HSS_BCU", "TOTAL" ];
cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL", "MAE_BCU",
"MSE",
"MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
We got above output types from grid-stat tool.
Is there a way to do this? Or shall we turn on certain options in
config
file?
Thanks
Guohui
On Wed, Oct 31, 2018 at 6:11 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> It looks like we're getting a little confused. Let me start by
clarifying
> some terminology.
>
> By fcst_var, I mean the contents of the column named "FCST_VAR" in
the
> ".stat" output files from the MET tools. They should be strings
like "TMP"
> for temperature or "PRES" for pressure or "APCP" for temperature.
>
> Abbreviations like ME and MAE stand for "mean error" and "mean
absolute
> error". Those are the statistics that show up in the CNT line.
>
> To illustrate, I've attached a sample .stat file that's output from
the
> Grid-Stat tool. It contains both continuous and categorical output
for
> relative humidity, 2-meter temperature and 10-m winds (i.e. FCST_VAR
> column) computed using multiple thresholds (i.e. FCST_THRESH column)
and
> over multiple verification regions (i.e. VX_MASK column). Here's a
few
> STAT-Analysis jobs you could run on it:
>
> # Run aggregate job to aggregate the SL1L2 continuous partial sums
across
> all the masking regions separately for each variable
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
SL1L2 -by
> FCST_VAR
>
> # Same thing but use aggregate_stat to derive continuous stats
instead AND
> WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
>
> # Aggregate contingency table counts across regions, separately for
each
> variable/threshold
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
CTC -by
> FCST_VAR,FCST_THRESH
>
> # Same as above but derive stats instead
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type CTC
> -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat job4.stat
>
> Are these example starting to make sense?
>
> Thanks,
> John
>
>
>
>
> On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> >
> > Good afternoon, John,
> >
> > Thank you for the details about stat_analysis. I could generate
> contingency
> > variables in CTS files .
> >
> > Another question about running stat_analysis tool is how to set
fcst var
> > option: I would like to run the tool on following variables "
> > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
> "FAR_NCU",
> > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
> "HSS_BCL",
> > "HSS_BCU", "TOTAL" ];
> > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> "MSE",
> > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
> > "
> >
> > If I don't add fcst_var in job option: -job aggregate_stat
-dump_row
> > $statOutFile $maskList -line_type SL1L2 -out_line_type CNT -v 4,
I got
> > output columns like "
> > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE
> > VX_MASK INTERP_MTHD
> > INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
> > LINE_TYPE TOTAL FBAR OBAR FOBAR FFBAR
> > OOBAR MAE
> > "
> >
> > If I add fcst_var to set job like following: -job aggregate_stat
> -fcst_var
> > ME -fcst_var MAE -dump_row $statOutFile $maskList -line_type
SL1L2
> > -out_line_type CNT -v 4, I got no data in output file.
> >
> > Do you know what I did wrong?
> >
> > Thanks
> >
> > Guohui
> >
> >
> >
> > On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Guohui,
> > >
> > > Yes, definitely. So here's how the data flow would work:
> > >
> > > (1) Each day, you run Grid-Stat to compare the analysis to HRRR,
NDFD,
> > and
> > > so on. The Grid-Stat for each run is written to a ".stat" file.
> > >
> > > (2) After you have a month or so of data, you can run the STAT-
Analysis
> > > tool to summarize the results through time. You'll run an
> > "aggregate_stat"
> > > job which reads CTC lines as input and writes CTS lines as
output.
> > >
> > > Something like this type of job...
> > > stat_analysis -lookin grid_stat_HRRR -job aggregate_stat
-line_type
> > CTC
> > > -out_line_type CTS \
> > > -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> > > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > -out_stat 2018NovStats.stat
> > >
> > > This job will...
> > >
> > > (1) Search recursively through an input directory named
> "grid_stat_HRRR"
> > > and read any ".stat" files it finds.
> > > (2) Read CTC input lines... aggregate them... and write CTS
output
> lines.
> > > (3) Only use lines whose valid time is between 20181101 and
20181130.
> > > (4) Produce separate output CTS lines for each unique
combination of
> the
> > > following set of MET header columns:
> > > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > (5) Write output to file named 2018NovStats.stat.
> > >
> > > If you're running a single job like this, it's fine to do it on
the
> > command
> > > line. If you'd like to run multiple jobs all on the same set of
input
> > > data, it may be more efficient to do so using a configuration
file for
> > > STAT-Analysis.
> > >
> > > FYI, here's online tutorial information about the STAT-Analysis
tool:
> > >
> > >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
>
> > > >
> > > > Good morning, John,
> > > >
> > > > we have NetCDF data for analysis model(model) and a few
forecast
> models
> > > > like "HRRR", "NDFD", etc. We could be able to grid_stat tool
to
> > generate
> > > > cnt, cts, and ctc as needed.
> > > >
> > > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL"
for
> > > > stat_analysis in case that people are interested in looking at
> > > contingency
> > > > table by monthly base. So, yes, we'd like option 1. It would
be nice
> if
> > > > there is a way to get it .
> > > >
> > > > Thank you
> > > >
> > > > Guohui
> > > >
> > > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Guohui,
> > > > >
> > > > > The Point-Stat tool writes the MPR line type to the ascii
output
> file
> > > > > ending in .stat. The Grid-Stat does *not* because the MPR
line
> type
> > > is a
> > > > > very inefficient way of storing data... especially gridded
data.
> > > > Grid-Stat
> > > > > can write the pair information to the a NetCDF output file.
> However,
> > > > > STAT-Analysis doesn't know how to read that NetCDF file.
> > STAT-Analysis
> > > > > only reads the ascii ".stat" files.
> > > > >
> > > > > But let's take a step back and figure out what you're trying
to do.
> > I
> > > > > understand that you want to compute monthly summary
statistics...
> > but a
> > > > > monthly summary of what?
> > > > >
> > > > > (1) CTC lines output from Grid-Stat that were computed over
some
> > > spatial
> > > > > area?
> > > > > (2) Doing a grid-to-grid comparison and computing CTS
statistics
> > > > separately
> > > > > for each grid point over a month's worth of data?
> > > > > (3) Doing a grid-to-point comparison and computing CTS
statistics
> > > > > separately for each observation station?
> > > > >
> > > > > If you can describe to me the type of data you have
available and
> > what
> > > > the
> > > > > analysis you'd like to do, I should be able to recommend
which MET
> > > tools
> > > > to
> > > > > use.
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > >
> > > > >
> > > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA Affiliate
via
> RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Question about stat_analysis tool
> > > > > > Owner: Nobody
> > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning,
> > > > > >
> > > > > > We are trying to do monthly analysis on CTC outline type
using
> > > > > > stat_analysis tool.
> > > > > >
> > > > > > This is how I am trying to work this out:
> > > > > >
> > > > > > Based on slide 5 in Tara's PPT (
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > > ),
> > > > > > in order to output CTC using stat_analysis tool, we have
to
> > generate
> > > > MRP
> > > > > > line type using grid_stat tool. In MET user guide, I
could not
> > find
> > > > > > information of MPR in grid_stat tool, but on your website,
> > Grid-stat
> > > > tool
> > > > > > can
> > > > > >
> > > > > > - Write output to ASCII statistics files and write
matched
> pairs
> > > to
> > > > > > NetCDF file
> > > > > >
> > > > > > Shall I just add MPR as other output types into
output_flag
> > > definition
> > > > in
> > > > > > grid_stat_config? maybe like:
> > > > > >
> > > > > > output_flag = {
> > > > > > fho = NONE;
> > > > > > ctc = BOTH;
> > > > > > cts = BOTH;
> > > > > > mctc = NONE;
> > > > > > mcts = BOTH;
> > > > > > cnt = BOTH;
> > > > > > sl1l2 = BOTH;
> > > > > > sal1l2 = NONE;
> > > > > > vl1l2 = NONE;
> > > > > > val1l2 = NONE;
> > > > > > *MPR = BOTH;*
> > > > > > pct = NONE;
> > > > > > pstd = NONE;
> > > > > > pjc = NONE;
> > > > > > prc = NONE;
> > > > > > nbrctc = NONE;
> > > > > > nbrcts = NONE;
> > > > > > nbrcnt = NONE;
> > > > > > }
> > > > > >
> > > > > > Thank you for your help
> > > > > >
> > > > > > Guohui
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: John Halley Gotway
Time: Fri Nov 02 15:09:49 2018
Guohui,
Judging from the list of stats you sent, it looks like you want to run
the
following types of jobs:
(1) aggregate job for the CTC lines.
(2) aggregate_stat job to aggregate CTC's and convert to CTS.
(3) aggregate_stat job to aggregate SL1L2's and write CNT.
You can run these as 3 separate calls to the stat_analysis tool on the
command line:
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type CTC
-by
FCST_VAR,FCST_THRESH -out_stat agg_ctc.stat
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate_stat -line_type
CTC
-out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
agg_stat_ctc_to_cts.stat
/usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -job aggregate_stat -line_type
SL1L2 -out_line_type CNT -by FCST_VAR -out_stat
agg_stat_sl1l2_to_cnt.stat
Or if you prefer, you could use a config file to do all 3 in a single
calls
to STAT-Analysis:
* /usr/local/met-8.0/bin/stat_analysis -lookin
grid_stat_120000L_20050807_120000V.stat -config STATAnalysisConfig *
Using the attached config file, which contains the following job
definition:
*jobs = [ "-job aggregate -line_type CTC -by FCST_VAR,FCST_THRESH
-out_stat agg_ctc.stat", "-job aggregate_stat -line_type CTC
-out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
agg_stat_ctc_to_cts.stat", "-job aggregate_stat -line_type SL1L2
-out_line_type CNT -by FCST_VAR -out_stat
agg_stat_sl1l2_to_cnt.stat"];*
Does that answer your question, or am I misinterpreting it?
Thanks,
John
On Fri, Nov 2, 2018 at 9:42 AM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
>
> Good morning, John,
>
> Yes, these are great examples. We can run stat_analysis for
aggregate_stat
> in the right way.
>
> But beside MSE, we do hope to get aggregate of all the statistical
output
> types as follows:
>
> ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" ];
> cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
"FAR_NCU",
> "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
"HSS_BCL",
> "HSS_BCU", "TOTAL" ];
> cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL", "MAE_BCU",
"MSE",
> "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
>
> We got above output types from grid-stat tool.
>
> Is there a way to do this? Or shall we turn on certain options in
config
> file?
>
> Thanks
>
> Guohui
>
> On Wed, Oct 31, 2018 at 6:11 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Guohui,
> >
> > It looks like we're getting a little confused. Let me start by
> clarifying
> > some terminology.
> >
> > By fcst_var, I mean the contents of the column named "FCST_VAR" in
the
> > ".stat" output files from the MET tools. They should be strings
like
> "TMP"
> > for temperature or "PRES" for pressure or "APCP" for temperature.
> >
> > Abbreviations like ME and MAE stand for "mean error" and "mean
absolute
> > error". Those are the statistics that show up in the CNT line.
> >
> > To illustrate, I've attached a sample .stat file that's output
from the
> > Grid-Stat tool. It contains both continuous and categorical
output for
> > relative humidity, 2-meter temperature and 10-m winds (i.e.
FCST_VAR
> > column) computed using multiple thresholds (i.e. FCST_THRESH
column) and
> > over multiple verification regions (i.e. VX_MASK column). Here's a
few
> > STAT-Analysis jobs you could run on it:
> >
> > # Run aggregate job to aggregate the SL1L2 continuous partial sums
across
> > all the masking regions separately for each variable
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
SL1L2
> -by
> > FCST_VAR
> >
> > # Same thing but use aggregate_stat to derive continuous stats
instead
> AND
> > WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
> >
> > # Aggregate contingency table counts across regions, separately
for each
> > variable/threshold
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
CTC -by
> > FCST_VAR,FCST_THRESH
> >
> > # Same as above but derive stats instead
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> CTC
> > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat job4.stat
> >
> > Are these example starting to make sense?
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > >
> > > Good afternoon, John,
> > >
> > > Thank you for the details about stat_analysis. I could generate
> > contingency
> > > variables in CTS files .
> > >
> > > Another question about running stat_analysis tool is how to set
fcst
> var
> > > option: I would like to run the tool on following variables "
> > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
> > "FAR_NCU",
> > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
> > "HSS_BCL",
> > > "HSS_BCU", "TOTAL" ];
> > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> > "MSE",
> > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
> > > "
> > >
> > > If I don't add fcst_var in job option: -job aggregate_stat
-dump_row
> > > $statOutFile $maskList -line_type SL1L2 -out_line_type CNT -v
4, I got
> > > output columns like "
> > > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> > > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV
> OBTYPE
> > > VX_MASK INTERP_MTHD
> > > INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
> > > LINE_TYPE TOTAL FBAR OBAR FOBAR FFBAR
> > > OOBAR MAE
> > > "
> > >
> > > If I add fcst_var to set job like following: -job
aggregate_stat
> > -fcst_var
> > > ME -fcst_var MAE -dump_row $statOutFile $maskList -line_type
SL1L2
> > > -out_line_type CNT -v 4, I got no data in output file.
> > >
> > > Do you know what I did wrong?
> > >
> > > Thanks
> > >
> > > Guohui
> > >
> > >
> > >
> > > On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Guohui,
> > > >
> > > > Yes, definitely. So here's how the data flow would work:
> > > >
> > > > (1) Each day, you run Grid-Stat to compare the analysis to
HRRR,
> NDFD,
> > > and
> > > > so on. The Grid-Stat for each run is written to a ".stat"
file.
> > > >
> > > > (2) After you have a month or so of data, you can run the
> STAT-Analysis
> > > > tool to summarize the results through time. You'll run an
> > > "aggregate_stat"
> > > > job which reads CTC lines as input and writes CTS lines as
output.
> > > >
> > > > Something like this type of job...
> > > > stat_analysis -lookin grid_stat_HRRR -job aggregate_stat
> -line_type
> > > CTC
> > > > -out_line_type CTS \
> > > > -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> > > > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > -out_stat 2018NovStats.stat
> > > >
> > > > This job will...
> > > >
> > > > (1) Search recursively through an input directory named
> > "grid_stat_HRRR"
> > > > and read any ".stat" files it finds.
> > > > (2) Read CTC input lines... aggregate them... and write CTS
output
> > lines.
> > > > (3) Only use lines whose valid time is between 20181101 and
20181130.
> > > > (4) Produce separate output CTS lines for each unique
combination of
> > the
> > > > following set of MET header columns:
> > > > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > (5) Write output to file named 2018NovStats.stat.
> > > >
> > > > If you're running a single job like this, it's fine to do it
on the
> > > command
> > > > line. If you'd like to run multiple jobs all on the same set
of
> input
> > > > data, it may be more efficient to do so using a configuration
file
> for
> > > > STAT-Analysis.
> > > >
> > > > FYI, here's online tutorial information about the STAT-
Analysis tool:
> > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA Affiliate
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > > > >
> > > > > Good morning, John,
> > > > >
> > > > > we have NetCDF data for analysis model(model) and a few
forecast
> > models
> > > > > like "HRRR", "NDFD", etc. We could be able to grid_stat tool
to
> > > generate
> > > > > cnt, cts, and ctc as needed.
> > > > >
> > > > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON",
"TOTAL" for
> > > > > stat_analysis in case that people are interested in looking
at
> > > > contingency
> > > > > table by monthly base. So, yes, we'd like option 1. It would
be
> nice
> > if
> > > > > there is a way to get it .
> > > > >
> > > > > Thank you
> > > > >
> > > > > Guohui
> > > > >
> > > > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Guohui,
> > > > > >
> > > > > > The Point-Stat tool writes the MPR line type to the ascii
output
> > file
> > > > > > ending in .stat. The Grid-Stat does *not* because the MPR
line
> > type
> > > > is a
> > > > > > very inefficient way of storing data... especially gridded
data.
> > > > > Grid-Stat
> > > > > > can write the pair information to the a NetCDF output
file.
> > However,
> > > > > > STAT-Analysis doesn't know how to read that NetCDF file.
> > > STAT-Analysis
> > > > > > only reads the ascii ".stat" files.
> > > > > >
> > > > > > But let's take a step back and figure out what you're
trying to
> do.
> > > I
> > > > > > understand that you want to compute monthly summary
statistics...
> > > but a
> > > > > > monthly summary of what?
> > > > > >
> > > > > > (1) CTC lines output from Grid-Stat that were computed
over some
> > > > spatial
> > > > > > area?
> > > > > > (2) Doing a grid-to-grid comparison and computing CTS
statistics
> > > > > separately
> > > > > > for each grid point over a month's worth of data?
> > > > > > (3) Doing a grid-to-point comparison and computing CTS
statistics
> > > > > > separately for each observation station?
> > > > > >
> > > > > > If you can describe to me the type of data you have
available and
> > > what
> > > > > the
> > > > > > analysis you'd like to do, I should be able to recommend
which
> MET
> > > > tools
> > > > > to
> > > > > > use.
> > > > > >
> > > > > > Thanks,
> > > > > > John Halley Gotway
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA
Affiliate via
> > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted upon.
> > > > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: Question about stat_analysis tool
> > > > > > > Owner: Nobody
> > > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Good morning,
> > > > > > >
> > > > > > > We are trying to do monthly analysis on CTC outline type
using
> > > > > > > stat_analysis tool.
> > > > > > >
> > > > > > > This is how I am trying to work this out:
> > > > > > >
> > > > > > > Based on slide 5 in Tara's PPT (
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > > > ),
> > > > > > > in order to output CTC using stat_analysis tool, we have
to
> > > generate
> > > > > MRP
> > > > > > > line type using grid_stat tool. In MET user guide, I
could not
> > > find
> > > > > > > information of MPR in grid_stat tool, but on your
website,
> > > Grid-stat
> > > > > tool
> > > > > > > can
> > > > > > >
> > > > > > > - Write output to ASCII statistics files and write
matched
> > pairs
> > > > to
> > > > > > > NetCDF file
> > > > > > >
> > > > > > > Shall I just add MPR as other output types into
output_flag
> > > > definition
> > > > > in
> > > > > > > grid_stat_config? maybe like:
> > > > > > >
> > > > > > > output_flag = {
> > > > > > > fho = NONE;
> > > > > > > ctc = BOTH;
> > > > > > > cts = BOTH;
> > > > > > > mctc = NONE;
> > > > > > > mcts = BOTH;
> > > > > > > cnt = BOTH;
> > > > > > > sl1l2 = BOTH;
> > > > > > > sal1l2 = NONE;
> > > > > > > vl1l2 = NONE;
> > > > > > > val1l2 = NONE;
> > > > > > > *MPR = BOTH;*
> > > > > > > pct = NONE;
> > > > > > > pstd = NONE;
> > > > > > > pjc = NONE;
> > > > > > > prc = NONE;
> > > > > > > nbrctc = NONE;
> > > > > > > nbrcts = NONE;
> > > > > > > nbrcnt = NONE;
> > > > > > > }
> > > > > > >
> > > > > > > Thank you for your help
> > > > > > >
> > > > > > > Guohui
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: Guohui Wang - NOAA Affiliate
Time: Mon Nov 19 18:53:03 2018
Good afternoon, John
I was following your suggestions, and did monthly stat aggregate on
variables in ctc and cts stat files. For example, here is one line
from
ctc.stat in 1810:
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY
FN_ON
V7.1 NDFD NA 060000 *20181001_060000 20181031_060000* 060000
20181001_060000 20181031_060000 APCP_06 A06 APCP_06 A
06 URMA CR NEAREST 1 >=0.254 >=0.254
NA
NA CTC 13351669 2134805 780114 53868
5 9898065
The fcst_valid_begin and fcst_valid_end are as we expected.
But in cnt.stat, we got output line with unexpected date as follows:
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR
FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL
FSTDEV_NCU
FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU
OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL
OSTDEV_BCU
PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_
BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL
ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDE
V_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE
MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE
BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU
E25
E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E
90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU
ANOM_CORR
ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU M
E2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU RMSFA
RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU
V7.1 NDFD NA 060000 *20181006_060000 20181027_060000* 060000
20181006_060000 20181027_060000 APCP_06 A06 APCP_06 A06
URMA
rnk NEAREST 1 >=12.7&&<25.4 >=12.7&&<25.4
NA
0.05 CNT 800 4.86738 4.50913 5.22563 NA
NA
5.16988 4.92837 5.43647 NA NA 14.27806
13.88944
14.66668 NA NA 5.60818 5.34619 5.89737
NA
NA -0.85018 -0.86832 -0.82976 NA
NA
NA NA 0 0 0 -9.41068 -10.12908
-8.69229 NA NA 10.3672 9.8829 10.90179 NA
NA
0.3409 NA NA 13.33006 NA NA 195.90548
NA NA 107.34451 NA NA 13.99662 NA
NA
NA NA NA NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA NA NA
NA NA NA NA NA 88.56097
NA NA -5.22878 NA NA NA NA
NA NA NA NA
V7.1 NDFD NA 060000 *20181002_060000 20181029_060000* 060000
20181002_060000 20181029_060000 APCP_06 A06 APCP_06 A06
URMA
iln NEAREST 1 >=2.54&&<6.35 >=2.54&&<6.35
NA
0.05 CNT 25146 3.30999 3.29216 3.32781 NA
NA
1.44228 1.42979 1.455 NA NA 3.35798
3.32776
3.3882 NA NA 2.44515 2.42397 2.46671
NA
NA 0.12541 0.11322 0.13756 NA
NA
NA NA 0 0 0 -0.047995 -0.081101
-0.014889 NA NA 2.67851 2.65531 2.70213 NA
NA
0.98571 NA NA 1.98773 NA NA 7.17644
NA NA 7.17414 NA NA 2.67889 NA
NA
NA NA NA NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA NA NA
NA NA NA NA NA
0.0023035
NA NA -0.20032 NA NA NA NA
NA NA NA NA
We expect to see fcst_valid_begin=20181001_060000 and
fcst_valid_end=20181031_060000 for all of the output, but we got
varies
date in this month. Do you know what I miss?
Here is how I set the jobs in my config:
"-job aggregate -line_type CTC -by FCST_VAR,FCST_THRESH,VX_MASK
-out_stat
${OUTFILE}_ctc.stat",
"-job aggregate_stat -line_type CTC -out_line_type CTS -by
FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cts.stat ",
"-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cnt.stat" This job
generated cnt.stat file.
Thank you for your help.
Guohui
On Fri, Nov 2, 2018 at 5:09 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> Judging from the list of stats you sent, it looks like you want to
run the
> following types of jobs:
>
> (1) aggregate job for the CTC lines.
> (2) aggregate_stat job to aggregate CTC's and convert to CTS.
> (3) aggregate_stat job to aggregate SL1L2's and write CNT.
>
> You can run these as 3 separate calls to the stat_analysis tool on
the
> command line:
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
CTC -by
> FCST_VAR,FCST_THRESH -out_stat agg_ctc.stat
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type CTC
> -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> agg_stat_ctc_to_cts.stat
> /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> SL1L2 -out_line_type CNT -by FCST_VAR -out_stat
agg_stat_sl1l2_to_cnt.stat
>
> Or if you prefer, you could use a config file to do all 3 in a
single calls
> to STAT-Analysis:
> * /usr/local/met-8.0/bin/stat_analysis -lookin
> grid_stat_120000L_20050807_120000V.stat -config STATAnalysisConfig *
>
> Using the attached config file, which contains the following job
> definition:
>
>
>
>
> *jobs = [ "-job aggregate -line_type CTC -by FCST_VAR,FCST_THRESH
> -out_stat agg_ctc.stat", "-job aggregate_stat -line_type CTC
> -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> agg_stat_ctc_to_cts.stat", "-job aggregate_stat -line_type SL1L2
> -out_line_type CNT -by FCST_VAR -out_stat
agg_stat_sl1l2_to_cnt.stat"];*
>
> Does that answer your question, or am I misinterpreting it?
>
> Thanks,
> John
>
>
> On Fri, Nov 2, 2018 at 9:42 AM Guohui Wang - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> >
> > Good morning, John,
> >
> > Yes, these are great examples. We can run stat_analysis for
> aggregate_stat
> > in the right way.
> >
> > But beside MSE, we do hope to get aggregate of all the statistical
output
> > types as follows:
> >
> > ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" ];
> > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
> "FAR_NCU",
> > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
> "HSS_BCL",
> > "HSS_BCU", "TOTAL" ];
> > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> "MSE",
> > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
> >
> > We got above output types from grid-stat tool.
> >
> > Is there a way to do this? Or shall we turn on certain options in
config
> > file?
> >
> > Thanks
> >
> > Guohui
> >
> > On Wed, Oct 31, 2018 at 6:11 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Guohui,
> > >
> > > It looks like we're getting a little confused. Let me start by
> > clarifying
> > > some terminology.
> > >
> > > By fcst_var, I mean the contents of the column named "FCST_VAR"
in the
> > > ".stat" output files from the MET tools. They should be strings
like
> > "TMP"
> > > for temperature or "PRES" for pressure or "APCP" for
temperature.
> > >
> > > Abbreviations like ME and MAE stand for "mean error" and "mean
absolute
> > > error". Those are the statistics that show up in the CNT line.
> > >
> > > To illustrate, I've attached a sample .stat file that's output
from the
> > > Grid-Stat tool. It contains both continuous and categorical
output for
> > > relative humidity, 2-meter temperature and 10-m winds (i.e.
FCST_VAR
> > > column) computed using multiple thresholds (i.e. FCST_THRESH
column)
> and
> > > over multiple verification regions (i.e. VX_MASK column). Here's
a few
> > > STAT-Analysis jobs you could run on it:
> > >
> > > # Run aggregate job to aggregate the SL1L2 continuous partial
sums
> across
> > > all the masking regions separately for each variable
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type SL1L2
> > -by
> > > FCST_VAR
> > >
> > > # Same thing but use aggregate_stat to derive continuous stats
instead
> > AND
> > > WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
> > >
> > > # Aggregate contingency table counts across regions, separately
for
> each
> > > variable/threshold
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type CTC
> -by
> > > FCST_VAR,FCST_THRESH
> > >
> > > # Same as above but derive stats instead
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > CTC
> > > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat job4.stat
> > >
> > > Are these example starting to make sense?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
>
> > > >
> > > > Good afternoon, John,
> > > >
> > > > Thank you for the details about stat_analysis. I could
generate
> > > contingency
> > > > variables in CTS files .
> > > >
> > > > Another question about running stat_analysis tool is how to
set fcst
> > var
> > > > option: I would like to run the tool on following variables "
> > > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR",
"FAR_NCL",
> > > "FAR_NCU",
> > > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU",
"HSS",
> > > "HSS_BCL",
> > > > "HSS_BCU", "TOTAL" ];
> > > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> > > "MSE",
> > > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL"
];
> > > > "
> > > >
> > > > If I don't add fcst_var in job option: -job aggregate_stat
-dump_row
> > > > $statOutFile $maskList -line_type SL1L2 -out_line_type CNT -v
4, I
> got
> > > > output columns like "
> > > > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> > > > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV
> > OBTYPE
> > > > VX_MASK INTERP_MTHD
> > > > INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
> > > > LINE_TYPE TOTAL FBAR OBAR FOBAR FFBAR
> > > > OOBAR MAE
> > > > "
> > > >
> > > > If I add fcst_var to set job like following: -job
aggregate_stat
> > > -fcst_var
> > > > ME -fcst_var MAE -dump_row $statOutFile $maskList -line_type
SL1L2
> > > > -out_line_type CNT -v 4, I got no data in output file.
> > > >
> > > > Do you know what I did wrong?
> > > >
> > > > Thanks
> > > >
> > > > Guohui
> > > >
> > > >
> > > >
> > > > On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Guohui,
> > > > >
> > > > > Yes, definitely. So here's how the data flow would work:
> > > > >
> > > > > (1) Each day, you run Grid-Stat to compare the analysis to
HRRR,
> > NDFD,
> > > > and
> > > > > so on. The Grid-Stat for each run is written to a ".stat"
file.
> > > > >
> > > > > (2) After you have a month or so of data, you can run the
> > STAT-Analysis
> > > > > tool to summarize the results through time. You'll run an
> > > > "aggregate_stat"
> > > > > job which reads CTC lines as input and writes CTS lines as
output.
> > > > >
> > > > > Something like this type of job...
> > > > > stat_analysis -lookin grid_stat_HRRR -job aggregate_stat
> > -line_type
> > > > CTC
> > > > > -out_line_type CTS \
> > > > > -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> > > > > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > -out_stat 2018NovStats.stat
> > > > >
> > > > > This job will...
> > > > >
> > > > > (1) Search recursively through an input directory named
> > > "grid_stat_HRRR"
> > > > > and read any ".stat" files it finds.
> > > > > (2) Read CTC input lines... aggregate them... and write CTS
output
> > > lines.
> > > > > (3) Only use lines whose valid time is between 20181101 and
> 20181130.
> > > > > (4) Produce separate output CTS lines for each unique
combination
> of
> > > the
> > > > > following set of MET header columns:
> > > > > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > (5) Write output to file named 2018NovStats.stat.
> > > > >
> > > > > If you're running a single job like this, it's fine to do it
on the
> > > > command
> > > > > line. If you'd like to run multiple jobs all on the same
set of
> > input
> > > > > data, it may be more efficient to do so using a
configuration file
> > for
> > > > > STAT-Analysis.
> > > > >
> > > > > FYI, here's online tutorial information about the STAT-
Analysis
> tool:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA
Affiliate via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > > > > >
> > > > > > Good morning, John,
> > > > > >
> > > > > > we have NetCDF data for analysis model(model) and a few
forecast
> > > models
> > > > > > like "HRRR", "NDFD", etc. We could be able to grid_stat
tool to
> > > > generate
> > > > > > cnt, cts, and ctc as needed.
> > > > > >
> > > > > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON",
"TOTAL"
> for
> > > > > > stat_analysis in case that people are interested in
looking at
> > > > > contingency
> > > > > > table by monthly base. So, yes, we'd like option 1. It
would be
> > nice
> > > if
> > > > > > there is a way to get it .
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > Guohui
> > > > > >
> > > > > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Guohui,
> > > > > > >
> > > > > > > The Point-Stat tool writes the MPR line type to the
ascii
> output
> > > file
> > > > > > > ending in .stat. The Grid-Stat does *not* because the
MPR line
> > > type
> > > > > is a
> > > > > > > very inefficient way of storing data... especially
gridded
> data.
> > > > > > Grid-Stat
> > > > > > > can write the pair information to the a NetCDF output
file.
> > > However,
> > > > > > > STAT-Analysis doesn't know how to read that NetCDF file.
> > > > STAT-Analysis
> > > > > > > only reads the ascii ".stat" files.
> > > > > > >
> > > > > > > But let's take a step back and figure out what you're
trying to
> > do.
> > > > I
> > > > > > > understand that you want to compute monthly summary
> statistics...
> > > > but a
> > > > > > > monthly summary of what?
> > > > > > >
> > > > > > > (1) CTC lines output from Grid-Stat that were computed
over
> some
> > > > > spatial
> > > > > > > area?
> > > > > > > (2) Doing a grid-to-grid comparison and computing CTS
> statistics
> > > > > > separately
> > > > > > > for each grid point over a month's worth of data?
> > > > > > > (3) Doing a grid-to-point comparison and computing CTS
> statistics
> > > > > > > separately for each observation station?
> > > > > > >
> > > > > > > If you can describe to me the type of data you have
available
> and
> > > > what
> > > > > > the
> > > > > > > analysis you'd like to do, I should be able to recommend
which
> > MET
> > > > > tools
> > > > > > to
> > > > > > > use.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John Halley Gotway
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA
Affiliate
> via
> > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted
upon.
> > > > > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: Question about stat_analysis tool
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Good morning,
> > > > > > > >
> > > > > > > > We are trying to do monthly analysis on CTC outline
type
> using
> > > > > > > > stat_analysis tool.
> > > > > > > >
> > > > > > > > This is how I am trying to work this out:
> > > > > > > >
> > > > > > > > Based on slide 5 in Tara's PPT (
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > > > > ),
> > > > > > > > in order to output CTC using stat_analysis tool, we
have to
> > > > generate
> > > > > > MRP
> > > > > > > > line type using grid_stat tool. In MET user guide, I
could
> not
> > > > find
> > > > > > > > information of MPR in grid_stat tool, but on your
website,
> > > > Grid-stat
> > > > > > tool
> > > > > > > > can
> > > > > > > >
> > > > > > > > - Write output to ASCII statistics files and write
matched
> > > pairs
> > > > > to
> > > > > > > > NetCDF file
> > > > > > > >
> > > > > > > > Shall I just add MPR as other output types into
output_flag
> > > > > definition
> > > > > > in
> > > > > > > > grid_stat_config? maybe like:
> > > > > > > >
> > > > > > > > output_flag = {
> > > > > > > > fho = NONE;
> > > > > > > > ctc = BOTH;
> > > > > > > > cts = BOTH;
> > > > > > > > mctc = NONE;
> > > > > > > > mcts = BOTH;
> > > > > > > > cnt = BOTH;
> > > > > > > > sl1l2 = BOTH;
> > > > > > > > sal1l2 = NONE;
> > > > > > > > vl1l2 = NONE;
> > > > > > > > val1l2 = NONE;
> > > > > > > > *MPR = BOTH;*
> > > > > > > > pct = NONE;
> > > > > > > > pstd = NONE;
> > > > > > > > pjc = NONE;
> > > > > > > > prc = NONE;
> > > > > > > > nbrctc = NONE;
> > > > > > > > nbrcts = NONE;
> > > > > > > > nbrcnt = NONE;
> > > > > > > > }
> > > > > > > >
> > > > > > > > Thank you for your help
> > > > > > > >
> > > > > > > > Guohui
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: John Halley Gotway
Time: Tue Nov 20 12:17:00 2018
Guohui,
I see that you ran STAT-Analysis to do a monthly summary. For the
aggregated CTC output, you saw the expected output time window:
*20181001_060000 20181031_060000*
But for CNT output the FCST_VALID_END date was short by 4 days:
*20181006_060000 20181027_060000*
The most likely explanation for this behavior is that you didn't pass
to
STAT-Analysis any SL1L2 lines for those last 4 days of October.
The fcst_valid_beg and fcst_valid_end entries in the STAT-Analysis
config
file define the search window. However, STAT-Analysis keeps track of
the
actual range of timestamps it finds at reports them in the output
FCST_VALID_BEG and FCST_VALID_END columns.
One way to figure this out is using the -dump_row job command option:
"-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cnt.stat
*-dump_row ${OUTFILE}_sl1l2_to_cnt.dump*"
That tells STAT-Analysis to write to that output dump file every input
line
that it used for this job. After you run the job, check to dump file
to
confirm that data is really missing for the last 4 days of October.
So STAT-Analysis writes the actual range of timestamps it found in the
data. But you can manually override this using the -set_hdr option:
* -set_hdr FCST_VALID_BEG 20181001_060000 -set_hdr FCST_VALID_END
20181031_060000*
The -set_hdr option manually overrides the output header columns with
whatever you specify.
Hope that helps clarify.
Thanks,
John
On Mon, Nov 19, 2018 at 6:53 PM Guohui Wang - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
>
> Good afternoon, John
>
> I was following your suggestions, and did monthly stat aggregate on
> variables in ctc and cts stat files. For example, here is one line
from
> ctc.stat in 1810:
>
> VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
> BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH
> COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY
> FN_ON
> V7.1 NDFD NA 060000 *20181001_060000 20181031_060000*
060000
> 20181001_060000 20181031_060000 APCP_06 A06 APCP_06 A
> 06 URMA CR NEAREST 1 >=0.254 >=0.254
NA
> NA CTC 13351669 2134805 780114 53868
> 5 9898065
>
> The fcst_valid_begin and fcst_valid_end are as we expected.
>
> But in cnt.stat, we got output line with unexpected date as
follows:
>
> VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
> BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
> OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR
> FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL
FSTDEV_NCU
> FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU
> OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL
OSTDEV_BCU
> PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_
> BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL
> ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDE
> V_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE
> MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE
> BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU
E25
> E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E
> 90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU
ANOM_CORR
> ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU M
> E2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU
RMSFA
> RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU
>
> V7.1 NDFD NA 060000 *20181006_060000 20181027_060000*
060000
> 20181006_060000 20181027_060000 APCP_06 A06 APCP_06 A06
URMA
> rnk NEAREST 1 >=12.7&&<25.4 >=12.7&&<25.4
NA
> 0.05 CNT 800 4.86738 4.50913 5.22563 NA
NA
> 5.16988 4.92837 5.43647 NA NA 14.27806
13.88944
> 14.66668 NA NA 5.60818 5.34619 5.89737
NA
> NA -0.85018 -0.86832 -0.82976 NA
NA
> NA NA 0 0 0 -9.41068 -10.12908
> -8.69229 NA NA 10.3672 9.8829 10.90179 NA
> NA
> 0.3409 NA NA 13.33006 NA NA
195.90548
> NA NA 107.34451 NA NA 13.99662 NA
NA
> NA NA NA NA NA NA NA NA NA NA
NA
> NA NA NA NA NA NA NA NA NA NA
> NA NA NA NA NA
88.56097
> NA NA -5.22878 NA NA NA NA
> NA NA NA NA
>
> V7.1 NDFD NA 060000 *20181002_060000 20181029_060000*
060000
> 20181002_060000 20181029_060000 APCP_06 A06 APCP_06 A06
URMA
> iln NEAREST 1 >=2.54&&<6.35 >=2.54&&<6.35
NA
> 0.05 CNT 25146 3.30999 3.29216 3.32781 NA
NA
> 1.44228 1.42979 1.455 NA NA 3.35798
3.32776
> 3.3882 NA NA 2.44515 2.42397 2.46671
NA
> NA 0.12541 0.11322 0.13756 NA
NA
> NA NA 0 0 0 -0.047995 -0.081101
> -0.014889 NA NA 2.67851 2.65531 2.70213 NA
> NA
> 0.98571 NA NA 1.98773 NA NA
7.17644
> NA NA 7.17414 NA NA 2.67889 NA
NA
> NA NA NA NA NA NA NA NA NA NA
NA
> NA NA NA NA NA NA NA NA NA NA
> NA NA NA NA NA
0.0023035
> NA NA -0.20032 NA NA NA NA
> NA NA NA NA
>
> We expect to see fcst_valid_begin=20181001_060000 and
> fcst_valid_end=20181031_060000 for all of the output, but we got
varies
> date in this month. Do you know what I miss?
>
> Here is how I set the jobs in my config:
>
> "-job aggregate -line_type CTC -by FCST_VAR,FCST_THRESH,VX_MASK
-out_stat
> ${OUTFILE}_ctc.stat",
> "-job aggregate_stat -line_type CTC -out_line_type CTS -by
> FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cts.stat ",
> "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
> FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cnt.stat" This
job
> generated cnt.stat file.
>
> Thank you for your help.
>
> Guohui
>
>
> On Fri, Nov 2, 2018 at 5:09 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Guohui,
> >
> > Judging from the list of stats you sent, it looks like you want to
run
> the
> > following types of jobs:
> >
> > (1) aggregate job for the CTC lines.
> > (2) aggregate_stat job to aggregate CTC's and convert to CTS.
> > (3) aggregate_stat job to aggregate SL1L2's and write CNT.
> >
> > You can run these as 3 separate calls to the stat_analysis tool on
the
> > command line:
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate -line_type
CTC -by
> > FCST_VAR,FCST_THRESH -out_stat agg_ctc.stat
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> CTC
> > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> > agg_stat_ctc_to_cts.stat
> > /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat
> agg_stat_sl1l2_to_cnt.stat
> >
> > Or if you prefer, you could use a config file to do all 3 in a
single
> calls
> > to STAT-Analysis:
> > * /usr/local/met-8.0/bin/stat_analysis -lookin
> > grid_stat_120000L_20050807_120000V.stat -config STATAnalysisConfig
*
> >
> > Using the attached config file, which contains the following job
> > definition:
> >
> >
> >
> >
> > *jobs = [ "-job aggregate -line_type CTC -by
FCST_VAR,FCST_THRESH
> > -out_stat agg_ctc.stat", "-job aggregate_stat -line_type CTC
> > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> > agg_stat_ctc_to_cts.stat", "-job aggregate_stat -line_type SL1L2
> > -out_line_type CNT -by FCST_VAR -out_stat
agg_stat_sl1l2_to_cnt.stat"];*
> >
> > Does that answer your question, or am I misinterpreting it?
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Nov 2, 2018 at 9:42 AM Guohui Wang - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > >
> > > Good morning, John,
> > >
> > > Yes, these are great examples. We can run stat_analysis for
> > aggregate_stat
> > > in the right way.
> > >
> > > But beside MSE, we do hope to get aggregate of all the
statistical
> output
> > > types as follows:
> > >
> > > ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" ];
> > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR", "FAR_NCL",
> > "FAR_NCU",
> > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU", "HSS",
> > "HSS_BCL",
> > > "HSS_BCU", "TOTAL" ];
> > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> > "MSE",
> > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL" ];
> > >
> > > We got above output types from grid-stat tool.
> > >
> > > Is there a way to do this? Or shall we turn on certain options
in
> config
> > > file?
> > >
> > > Thanks
> > >
> > > Guohui
> > >
> > > On Wed, Oct 31, 2018 at 6:11 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Guohui,
> > > >
> > > > It looks like we're getting a little confused. Let me start
by
> > > clarifying
> > > > some terminology.
> > > >
> > > > By fcst_var, I mean the contents of the column named
"FCST_VAR" in
> the
> > > > ".stat" output files from the MET tools. They should be
strings like
> > > "TMP"
> > > > for temperature or "PRES" for pressure or "APCP" for
temperature.
> > > >
> > > > Abbreviations like ME and MAE stand for "mean error" and "mean
> absolute
> > > > error". Those are the statistics that show up in the CNT
line.
> > > >
> > > > To illustrate, I've attached a sample .stat file that's output
from
> the
> > > > Grid-Stat tool. It contains both continuous and categorical
output
> for
> > > > relative humidity, 2-meter temperature and 10-m winds (i.e.
FCST_VAR
> > > > column) computed using multiple thresholds (i.e. FCST_THRESH
column)
> > and
> > > > over multiple verification regions (i.e. VX_MASK column).
Here's a
> few
> > > > STAT-Analysis jobs you could run on it:
> > > >
> > > > # Run aggregate job to aggregate the SL1L2 continuous partial
sums
> > across
> > > > all the masking regions separately for each variable
> > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type
> SL1L2
> > > -by
> > > > FCST_VAR
> > > >
> > > > # Same thing but use aggregate_stat to derive continuous stats
> instead
> > > AND
> > > > WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
> > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
> -line_type
> > > > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
> > > >
> > > > # Aggregate contingency table counts across regions,
separately for
> > each
> > > > variable/threshold
> > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type CTC
> > -by
> > > > FCST_VAR,FCST_THRESH
> > > >
> > > > # Same as above but derive stats instead
> > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
> -line_type
> > > CTC
> > > > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
job4.stat
> > > >
> > > > Are these example starting to make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA Affiliate
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > > > >
> > > > > Good afternoon, John,
> > > > >
> > > > > Thank you for the details about stat_analysis. I could
generate
> > > > contingency
> > > > > variables in CTS files .
> > > > >
> > > > > Another question about running stat_analysis tool is how to
set
> fcst
> > > var
> > > > > option: I would like to run the tool on following variables
"
> > > > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR",
"FAR_NCL",
> > > > "FAR_NCU",
> > > > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU",
"HSS",
> > > > "HSS_BCL",
> > > > > "HSS_BCU", "TOTAL" ];
> > > > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
> "MAE_BCU",
> > > > "MSE",
> > > > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU",
"TOTAL" ];
> > > > > "
> > > > >
> > > > > If I don't add fcst_var in job option: -job aggregate_stat
> -dump_row
> > > > > $statOutFile $maskList -line_type SL1L2 -out_line_type CNT
-v 4, I
> > got
> > > > > output columns like "
> > > > > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
> OBS_LEAD
> > > > > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV
> > > OBTYPE
> > > > > VX_MASK INTERP_MTHD
> > > > > INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH
> ALPHA
> > > > > LINE_TYPE TOTAL FBAR OBAR FOBAR
FFBAR
> > > > > OOBAR MAE
> > > > > "
> > > > >
> > > > > If I add fcst_var to set job like following: -job
aggregate_stat
> > > > -fcst_var
> > > > > ME -fcst_var MAE -dump_row $statOutFile $maskList
-line_type SL1L2
> > > > > -out_line_type CNT -v 4, I got no data in output file.
> > > > >
> > > > > Do you know what I did wrong?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Guohui
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Guohui,
> > > > > >
> > > > > > Yes, definitely. So here's how the data flow would work:
> > > > > >
> > > > > > (1) Each day, you run Grid-Stat to compare the analysis to
HRRR,
> > > NDFD,
> > > > > and
> > > > > > so on. The Grid-Stat for each run is written to a ".stat"
file.
> > > > > >
> > > > > > (2) After you have a month or so of data, you can run the
> > > STAT-Analysis
> > > > > > tool to summarize the results through time. You'll run an
> > > > > "aggregate_stat"
> > > > > > job which reads CTC lines as input and writes CTS lines as
> output.
> > > > > >
> > > > > > Something like this type of job...
> > > > > > stat_analysis -lookin grid_stat_HRRR -job
aggregate_stat
> > > -line_type
> > > > > CTC
> > > > > > -out_line_type CTS \
> > > > > > -fcst_valid_beg 20181101 -fcst_valid_end 20181130 \
> > > > > > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > > -out_stat 2018NovStats.stat
> > > > > >
> > > > > > This job will...
> > > > > >
> > > > > > (1) Search recursively through an input directory named
> > > > "grid_stat_HRRR"
> > > > > > and read any ".stat" files it finds.
> > > > > > (2) Read CTC input lines... aggregate them... and write
CTS
> output
> > > > lines.
> > > > > > (3) Only use lines whose valid time is between 20181101
and
> > 20181130.
> > > > > > (4) Produce separate output CTS lines for each unique
combination
> > of
> > > > the
> > > > > > following set of MET header columns:
> > > > > > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > > (5) Write output to file named 2018NovStats.stat.
> > > > > >
> > > > > > If you're running a single job like this, it's fine to do
it on
> the
> > > > > command
> > > > > > line. If you'd like to run multiple jobs all on the same
set of
> > > input
> > > > > > data, it may be more efficient to do so using a
configuration
> file
> > > for
> > > > > > STAT-Analysis.
> > > > > >
> > > > > > FYI, here's online tutorial information about the STAT-
Analysis
> > tool:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA
Affiliate via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> >
> > > > > > >
> > > > > > > Good morning, John,
> > > > > > >
> > > > > > > we have NetCDF data for analysis model(model) and a few
> forecast
> > > > models
> > > > > > > like "HRRR", "NDFD", etc. We could be able to grid_stat
tool to
> > > > > generate
> > > > > > > cnt, cts, and ctc as needed.
> > > > > > >
> > > > > > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON",
"TOTAL"
> > for
> > > > > > > stat_analysis in case that people are interested in
looking at
> > > > > > contingency
> > > > > > > table by monthly base. So, yes, we'd like option 1. It
would be
> > > nice
> > > > if
> > > > > > > there is a way to get it .
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > Guohui
> > > > > > >
> > > > > > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Guohui,
> > > > > > > >
> > > > > > > > The Point-Stat tool writes the MPR line type to the
ascii
> > output
> > > > file
> > > > > > > > ending in .stat. The Grid-Stat does *not* because the
MPR
> line
> > > > type
> > > > > > is a
> > > > > > > > very inefficient way of storing data... especially
gridded
> > data.
> > > > > > > Grid-Stat
> > > > > > > > can write the pair information to the a NetCDF output
file.
> > > > However,
> > > > > > > > STAT-Analysis doesn't know how to read that NetCDF
file.
> > > > > STAT-Analysis
> > > > > > > > only reads the ascii ".stat" files.
> > > > > > > >
> > > > > > > > But let's take a step back and figure out what you're
trying
> to
> > > do.
> > > > > I
> > > > > > > > understand that you want to compute monthly summary
> > statistics...
> > > > > but a
> > > > > > > > monthly summary of what?
> > > > > > > >
> > > > > > > > (1) CTC lines output from Grid-Stat that were computed
over
> > some
> > > > > > spatial
> > > > > > > > area?
> > > > > > > > (2) Doing a grid-to-grid comparison and computing CTS
> > statistics
> > > > > > > separately
> > > > > > > > for each grid point over a month's worth of data?
> > > > > > > > (3) Doing a grid-to-point comparison and computing CTS
> > statistics
> > > > > > > > separately for each observation station?
> > > > > > > >
> > > > > > > > If you can describe to me the type of data you have
available
> > and
> > > > > what
> > > > > > > the
> > > > > > > > analysis you'd like to do, I should be able to
recommend
> which
> > > MET
> > > > > > tools
> > > > > > > to
> > > > > > > > use.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John Halley Gotway
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted
upon.
> > > > > > > > > Transaction: Ticket created by guohui.wang at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: Question about stat_analysis tool
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good morning,
> > > > > > > > >
> > > > > > > > > We are trying to do monthly analysis on CTC outline
type
> > using
> > > > > > > > > stat_analysis tool.
> > > > > > > > >
> > > > > > > > > This is how I am trying to work this out:
> > > > > > > > >
> > > > > > > > > Based on slide 5 in Tara's PPT (
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > > > > > ),
> > > > > > > > > in order to output CTC using stat_analysis tool, we
have to
> > > > > generate
> > > > > > > MRP
> > > > > > > > > line type using grid_stat tool. In MET user guide,
I could
> > not
> > > > > find
> > > > > > > > > information of MPR in grid_stat tool, but on your
website,
> > > > > Grid-stat
> > > > > > > tool
> > > > > > > > > can
> > > > > > > > >
> > > > > > > > > - Write output to ASCII statistics files and
write
> matched
> > > > pairs
> > > > > > to
> > > > > > > > > NetCDF file
> > > > > > > > >
> > > > > > > > > Shall I just add MPR as other output types into
output_flag
> > > > > > definition
> > > > > > > in
> > > > > > > > > grid_stat_config? maybe like:
> > > > > > > > >
> > > > > > > > > output_flag = {
> > > > > > > > > fho = NONE;
> > > > > > > > > ctc = BOTH;
> > > > > > > > > cts = BOTH;
> > > > > > > > > mctc = NONE;
> > > > > > > > > mcts = BOTH;
> > > > > > > > > cnt = BOTH;
> > > > > > > > > sl1l2 = BOTH;
> > > > > > > > > sal1l2 = NONE;
> > > > > > > > > vl1l2 = NONE;
> > > > > > > > > val1l2 = NONE;
> > > > > > > > > *MPR = BOTH;*
> > > > > > > > > pct = NONE;
> > > > > > > > > pstd = NONE;
> > > > > > > > > pjc = NONE;
> > > > > > > > > prc = NONE;
> > > > > > > > > nbrctc = NONE;
> > > > > > > > > nbrcts = NONE;
> > > > > > > > > nbrcnt = NONE;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > Thank you for your help
> > > > > > > > >
> > > > > > > > > Guohui
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Question about stat_analysis tool
From: Guohui Wang - NOAA Affiliate
Time: Fri Nov 23 09:16:46 2018
John,
You are amazing! It is as you said. We didn't feed in enough data.
Thanks a lot! Have a great holiday
Guohui
On Tue, Nov 20, 2018 at 2:17 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Guohui,
>
> I see that you ran STAT-Analysis to do a monthly summary. For the
> aggregated CTC output, you saw the expected output time window:
>
> *20181001_060000 20181031_060000*
>
> But for CNT output the FCST_VALID_END date was short by 4 days:
>
> *20181006_060000 20181027_060000*
>
> The most likely explanation for this behavior is that you didn't
pass to
> STAT-Analysis any SL1L2 lines for those last 4 days of October.
>
> The fcst_valid_beg and fcst_valid_end entries in the STAT-Analysis
config
> file define the search window. However, STAT-Analysis keeps track
of the
> actual range of timestamps it finds at reports them in the output
> FCST_VALID_BEG and FCST_VALID_END columns.
>
> One way to figure this out is using the -dump_row job command
option:
> "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
> FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cnt.stat
> *-dump_row ${OUTFILE}_sl1l2_to_cnt.dump*"
>
> That tells STAT-Analysis to write to that output dump file every
input line
> that it used for this job. After you run the job, check to dump
file to
> confirm that data is really missing for the last 4 days of October.
>
> So STAT-Analysis writes the actual range of timestamps it found in
the
> data. But you can manually override this using the -set_hdr option:
> * -set_hdr FCST_VALID_BEG 20181001_060000 -set_hdr FCST_VALID_END
> 20181031_060000*
>
> The -set_hdr option manually overrides the output header columns
with
> whatever you specify.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Mon, Nov 19, 2018 at 6:53 PM Guohui Wang - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> >
> > Good afternoon, John
> >
> > I was following your suggestions, and did monthly stat aggregate
on
> > variables in ctc and cts stat files. For example, here is one line
from
> > ctc.stat in 1810:
> >
> > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
> > BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH
> > COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY
> > FN_ON
> > V7.1 NDFD NA 060000 *20181001_060000 20181031_060000*
060000
> > 20181001_060000 20181031_060000 APCP_06 A06 APCP_06 A
> > 06 URMA CR NEAREST 1 >=0.254
>=0.254 NA
> > NA CTC 13351669 2134805 780114 53868
> > 5 9898065
> >
> > The fcst_valid_begin and fcst_valid_end are as we expected.
> >
> > But in cnt.stat, we got output line with unexpected date as
follows:
> >
> > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD
> > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR O
> > BS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
> > OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR
> > FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL
FSTDEV_NCU
> > FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU
> > OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL
OSTDEV_BCU
> > PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_
> > BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME
ME_NCL
> > ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDE
> > V_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE
> > MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE
> > BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL
E10_BCU E25
> > E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E
> > 90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU
ANOM_CORR
> > ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU M
> > E2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU
RMSFA
> > RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU
> >
> > V7.1 NDFD NA 060000 *20181006_060000 20181027_060000*
060000
> > 20181006_060000 20181027_060000 APCP_06 A06 APCP_06 A06
URMA
> > rnk NEAREST 1 >=12.7&&<25.4
>=12.7&&<25.4
> NA
> > 0.05 CNT 800 4.86738 4.50913 5.22563 NA
> NA
> > 5.16988 4.92837 5.43647 NA NA 14.27806
13.88944
> > 14.66668 NA NA 5.60818 5.34619 5.89737
NA
> > NA -0.85018 -0.86832 -0.82976 NA
NA
> > NA NA 0 0 0 -9.41068 -10.12908
> > -8.69229 NA NA 10.3672 9.8829 10.90179
NA
> > NA
> > 0.3409 NA NA 13.33006 NA NA
195.90548
> > NA NA 107.34451 NA NA 13.99662 NA
NA
> > NA NA NA NA NA NA NA NA NA NA
NA
> > NA NA NA NA NA NA NA NA NA
NA
> > NA NA NA NA NA
88.56097
> > NA NA -5.22878 NA NA NA NA
> > NA NA NA NA
> >
> > V7.1 NDFD NA 060000 *20181002_060000 20181029_060000*
060000
> > 20181002_060000 20181029_060000 APCP_06 A06 APCP_06 A06
URMA
> > iln NEAREST 1 >=2.54&&<6.35
>=2.54&&<6.35
> NA
> > 0.05 CNT 25146 3.30999 3.29216 3.32781 NA
> NA
> > 1.44228 1.42979 1.455 NA NA 3.35798
3.32776
> > 3.3882 NA NA 2.44515 2.42397 2.46671
NA
> > NA 0.12541 0.11322 0.13756 NA
NA
> > NA NA 0 0 0 -0.047995 -0.081101
> > -0.014889 NA NA 2.67851 2.65531 2.70213
NA
> > NA
> > 0.98571 NA NA 1.98773 NA NA
7.17644
> > NA NA 7.17414 NA NA 2.67889 NA
NA
> > NA NA NA NA NA NA NA NA NA NA
NA
> > NA NA NA NA NA NA NA NA NA
NA
> > NA NA NA NA NA
0.0023035
> > NA NA -0.20032 NA NA NA NA
> > NA NA NA NA
> >
> > We expect to see fcst_valid_begin=20181001_060000 and
> > fcst_valid_end=20181031_060000 for all of the output, but we got
varies
> > date in this month. Do you know what I miss?
> >
> > Here is how I set the jobs in my config:
> >
> > "-job aggregate -line_type CTC -by FCST_VAR,FCST_THRESH,VX_MASK
-out_stat
> > ${OUTFILE}_ctc.stat",
> > "-job aggregate_stat -line_type CTC -out_line_type CTS -by
> > FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cts.stat ",
> > "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
> > FCST_VAR,FCST_THRESH,VX_MASK -out_stat ${OUTFILE}_cnt.stat" This
job
> > generated cnt.stat file.
> >
> > Thank you for your help.
> >
> > Guohui
> >
> >
> > On Fri, Nov 2, 2018 at 5:09 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Guohui,
> > >
> > > Judging from the list of stats you sent, it looks like you want
to run
> > the
> > > following types of jobs:
> > >
> > > (1) aggregate job for the CTC lines.
> > > (2) aggregate_stat job to aggregate CTC's and convert to CTS.
> > > (3) aggregate_stat job to aggregate SL1L2's and write CNT.
> > >
> > > You can run these as 3 separate calls to the stat_analysis tool
on the
> > > command line:
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type CTC
> -by
> > > FCST_VAR,FCST_THRESH -out_stat agg_ctc.stat
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > CTC
> > > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> > > agg_stat_ctc_to_cts.stat
> > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
-line_type
> > > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat
> > agg_stat_sl1l2_to_cnt.stat
> > >
> > > Or if you prefer, you could use a config file to do all 3 in a
single
> > calls
> > > to STAT-Analysis:
> > > * /usr/local/met-8.0/bin/stat_analysis -lookin
> > > grid_stat_120000L_20050807_120000V.stat -config
STATAnalysisConfig *
> > >
> > > Using the attached config file, which contains the following job
> > > definition:
> > >
> > >
> > >
> > >
> > > *jobs = [ "-job aggregate -line_type CTC -by
FCST_VAR,FCST_THRESH
> > > -out_stat agg_ctc.stat", "-job aggregate_stat -line_type CTC
> > > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
> > > agg_stat_ctc_to_cts.stat", "-job aggregate_stat -line_type
SL1L2
> > > -out_line_type CNT -by FCST_VAR -out_stat
> agg_stat_sl1l2_to_cnt.stat"];*
> > >
> > > Does that answer your question, or am I misinterpreting it?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Nov 2, 2018 at 9:42 AM Guohui Wang - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
>
> > > >
> > > > Good morning, John,
> > > >
> > > > Yes, these are great examples. We can run stat_analysis for
> > > aggregate_stat
> > > > in the right way.
> > > >
> > > > But beside MSE, we do hope to get aggregate of all the
statistical
> > output
> > > > types as follows:
> > > >
> > > > ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON", "TOTAL" ];
> > > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR",
"FAR_NCL",
> > > "FAR_NCU",
> > > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU",
"HSS",
> > > "HSS_BCL",
> > > > "HSS_BCU", "TOTAL" ];
> > > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
"MAE_BCU",
> > > "MSE",
> > > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU", "TOTAL"
];
> > > >
> > > > We got above output types from grid-stat tool.
> > > >
> > > > Is there a way to do this? Or shall we turn on certain options
in
> > config
> > > > file?
> > > >
> > > > Thanks
> > > >
> > > > Guohui
> > > >
> > > > On Wed, Oct 31, 2018 at 6:11 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Guohui,
> > > > >
> > > > > It looks like we're getting a little confused. Let me start
by
> > > > clarifying
> > > > > some terminology.
> > > > >
> > > > > By fcst_var, I mean the contents of the column named
"FCST_VAR" in
> > the
> > > > > ".stat" output files from the MET tools. They should be
strings
> like
> > > > "TMP"
> > > > > for temperature or "PRES" for pressure or "APCP" for
temperature.
> > > > >
> > > > > Abbreviations like ME and MAE stand for "mean error" and
"mean
> > absolute
> > > > > error". Those are the statistics that show up in the CNT
line.
> > > > >
> > > > > To illustrate, I've attached a sample .stat file that's
output from
> > the
> > > > > Grid-Stat tool. It contains both continuous and categorical
output
> > for
> > > > > relative humidity, 2-meter temperature and 10-m winds (i.e.
> FCST_VAR
> > > > > column) computed using multiple thresholds (i.e. FCST_THRESH
> column)
> > > and
> > > > > over multiple verification regions (i.e. VX_MASK column).
Here's a
> > few
> > > > > STAT-Analysis jobs you could run on it:
> > > > >
> > > > > # Run aggregate job to aggregate the SL1L2 continuous
partial sums
> > > across
> > > > > all the masking regions separately for each variable
> > > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type
> > SL1L2
> > > > -by
> > > > > FCST_VAR
> > > > >
> > > > > # Same thing but use aggregate_stat to derive continuous
stats
> > instead
> > > > AND
> > > > > WRITE OUTPUT TO A NEW FILE NAMED "job2.stat"
> > > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
> > -line_type
> > > > > SL1L2 -out_line_type CNT -by FCST_VAR -out_stat job2.stat
> > > > >
> > > > > # Aggregate contingency table counts across regions,
separately for
> > > each
> > > > > variable/threshold
> > > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > > grid_stat_120000L_20050807_120000V.stat -job aggregate
-line_type
> CTC
> > > -by
> > > > > FCST_VAR,FCST_THRESH
> > > > >
> > > > > # Same as above but derive stats instead
> > > > > /usr/local/met-8.0/bin/stat_analysis -lookin
> > > > > grid_stat_120000L_20050807_120000V.stat -job aggregate_stat
> > -line_type
> > > > CTC
> > > > > -out_line_type CTS -by FCST_VAR,FCST_THRESH -out_stat
job4.stat
> > > > >
> > > > > Are these example starting to make sense?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Oct 30, 2018 at 12:50 PM Guohui Wang - NOAA
Affiliate via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558 >
> > > > > >
> > > > > > Good afternoon, John,
> > > > > >
> > > > > > Thank you for the details about stat_analysis. I could
generate
> > > > > contingency
> > > > > > variables in CTS files .
> > > > > >
> > > > > > Another question about running stat_analysis tool is how
to set
> > fcst
> > > > var
> > > > > > option: I would like to run the tool on following
variables "
> > > > > > cts = [ "PODY", "PODY_NCL", "PODY_NCU", "FAR",
"FAR_NCL",
> > > > > "FAR_NCU",
> > > > > > "CSI", "CSI_NCL", "CSI_NCU", "GSS", "GSS_BCL", "GSS_BCU",
"HSS",
> > > > > "HSS_BCL",
> > > > > > "HSS_BCU", "TOTAL" ];
> > > > > > cnt = [ "ME", "ME_NCL", "ME_NCU", "MAE", "MAE_BCL",
> > "MAE_BCU",
> > > > > "MSE",
> > > > > > "MSE_BCL", "MSE_BCU", "RMSE", "RMSE_BCL", "RMSE_BCU",
"TOTAL" ];
> > > > > > "
> > > > > >
> > > > > > If I don't add fcst_var in job option: -job aggregate_stat
> > -dump_row
> > > > > > $statOutFile $maskList -line_type SL1L2 -out_line_type
CNT -v
> 4, I
> > > got
> > > > > > output columns like "
> > > > > > VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG
FCST_VALID_END
> > OBS_LEAD
> > > > > > OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV
> > > > OBTYPE
> > > > > > VX_MASK INTERP_MTHD
> > > > > > INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH
> > ALPHA
> > > > > > LINE_TYPE TOTAL FBAR OBAR FOBAR
FFBAR
> > > > > > OOBAR MAE
> > > > > > "
> > > > > >
> > > > > > If I add fcst_var to set job like following: -job
aggregate_stat
> > > > > -fcst_var
> > > > > > ME -fcst_var MAE -dump_row $statOutFile $maskList
-line_type
> SL1L2
> > > > > > -out_line_type CNT -v 4, I got no data in output file.
> > > > > >
> > > > > > Do you know what I did wrong?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Guohui
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 29, 2018 at 12:27 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Guohui,
> > > > > > >
> > > > > > > Yes, definitely. So here's how the data flow would
work:
> > > > > > >
> > > > > > > (1) Each day, you run Grid-Stat to compare the analysis
to
> HRRR,
> > > > NDFD,
> > > > > > and
> > > > > > > so on. The Grid-Stat for each run is written to a
".stat"
> file.
> > > > > > >
> > > > > > > (2) After you have a month or so of data, you can run
the
> > > > STAT-Analysis
> > > > > > > tool to summarize the results through time. You'll run
an
> > > > > > "aggregate_stat"
> > > > > > > job which reads CTC lines as input and writes CTS lines
as
> > output.
> > > > > > >
> > > > > > > Something like this type of job...
> > > > > > > stat_analysis -lookin grid_stat_HRRR -job
aggregate_stat
> > > > -line_type
> > > > > > CTC
> > > > > > > -out_line_type CTS \
> > > > > > > -fcst_valid_beg 20181101 -fcst_valid_end 20181130
\
> > > > > > > -by MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > > > -out_stat 2018NovStats.stat
> > > > > > >
> > > > > > > This job will...
> > > > > > >
> > > > > > > (1) Search recursively through an input directory named
> > > > > "grid_stat_HRRR"
> > > > > > > and read any ".stat" files it finds.
> > > > > > > (2) Read CTC input lines... aggregate them... and write
CTS
> > output
> > > > > lines.
> > > > > > > (3) Only use lines whose valid time is between 20181101
and
> > > 20181130.
> > > > > > > (4) Produce separate output CTS lines for each unique
> combination
> > > of
> > > > > the
> > > > > > > following set of MET header columns:
> > > > > > > MODEL,VX_MASK,FCST_LEAD,FCST_VAR,FCST_THRESH
> > > > > > > (5) Write output to file named 2018NovStats.stat.
> > > > > > >
> > > > > > > If you're running a single job like this, it's fine to
do it on
> > the
> > > > > > command
> > > > > > > line. If you'd like to run multiple jobs all on the
same set
> of
> > > > input
> > > > > > > data, it may be more efficient to do so using a
configuration
> > file
> > > > for
> > > > > > > STAT-Analysis.
> > > > > > >
> > > > > > > FYI, here's online tutorial information about the STAT-
Analysis
> > > tool:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/support/online_tutorial/METv6.1/tutorial.php?name=stat_analysis&category=index
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 29, 2018 at 10:14 AM Guohui Wang - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > >
> > > > > > > >
> > > > > > > > Good morning, John,
> > > > > > > >
> > > > > > > > we have NetCDF data for analysis model(model) and a
few
> > forecast
> > > > > models
> > > > > > > > like "HRRR", "NDFD", etc. We could be able to
grid_stat tool
> to
> > > > > > generate
> > > > > > > > cnt, cts, and ctc as needed.
> > > > > > > >
> > > > > > > > We are looking for "FY_OY", "FY_ON", "FN_OY", "FN_ON",
> "TOTAL"
> > > for
> > > > > > > > stat_analysis in case that people are interested in
looking
> at
> > > > > > > contingency
> > > > > > > > table by monthly base. So, yes, we'd like option 1. It
would
> be
> > > > nice
> > > > > if
> > > > > > > > there is a way to get it .
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > > Guohui
> > > > > > > >
> > > > > > > > On Mon, Oct 29, 2018 at 11:58 AM John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Guohui,
> > > > > > > > >
> > > > > > > > > The Point-Stat tool writes the MPR line type to the
ascii
> > > output
> > > > > file
> > > > > > > > > ending in .stat. The Grid-Stat does *not* because
the MPR
> > line
> > > > > type
> > > > > > > is a
> > > > > > > > > very inefficient way of storing data... especially
gridded
> > > data.
> > > > > > > > Grid-Stat
> > > > > > > > > can write the pair information to the a NetCDF
output file.
> > > > > However,
> > > > > > > > > STAT-Analysis doesn't know how to read that NetCDF
file.
> > > > > > STAT-Analysis
> > > > > > > > > only reads the ascii ".stat" files.
> > > > > > > > >
> > > > > > > > > But let's take a step back and figure out what
you're
> trying
> > to
> > > > do.
> > > > > > I
> > > > > > > > > understand that you want to compute monthly summary
> > > statistics...
> > > > > > but a
> > > > > > > > > monthly summary of what?
> > > > > > > > >
> > > > > > > > > (1) CTC lines output from Grid-Stat that were
computed over
> > > some
> > > > > > > spatial
> > > > > > > > > area?
> > > > > > > > > (2) Doing a grid-to-grid comparison and computing
CTS
> > > statistics
> > > > > > > > separately
> > > > > > > > > for each grid point over a month's worth of data?
> > > > > > > > > (3) Doing a grid-to-point comparison and computing
CTS
> > > statistics
> > > > > > > > > separately for each observation station?
> > > > > > > > >
> > > > > > > > > If you can describe to me the type of data you have
> available
> > > and
> > > > > > what
> > > > > > > > the
> > > > > > > > > analysis you'd like to do, I should be able to
recommend
> > which
> > > > MET
> > > > > > > tools
> > > > > > > > to
> > > > > > > > > use.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John Halley Gotway
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 29, 2018 at 8:46 AM Guohui Wang - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Mon Oct 29 08:45:36 2018: Request 87558 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by
guohui.wang at noaa.gov
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: Question about stat_analysis tool
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: guohui.wang at noaa.gov
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87558
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Good morning,
> > > > > > > > > >
> > > > > > > > > > We are trying to do monthly analysis on CTC
outline type
> > > using
> > > > > > > > > > stat_analysis tool.
> > > > > > > > > >
> > > > > > > > > > This is how I am trying to work this out:
> > > > > > > > > >
> > > > > > > > > > Based on slide 5 in Tara's PPT (
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://dtcenter.org/met/users/docs/presentations/MET_Tutorial_20150202/08_Stat_Analysis_Tool_Feb15.pdf
> > > > > > > > > > ),
> > > > > > > > > > in order to output CTC using stat_analysis tool,
we have
> to
> > > > > > generate
> > > > > > > > MRP
> > > > > > > > > > line type using grid_stat tool. In MET user
guide, I
> could
> > > not
> > > > > > find
> > > > > > > > > > information of MPR in grid_stat tool, but on your
> website,
> > > > > > Grid-stat
> > > > > > > > tool
> > > > > > > > > > can
> > > > > > > > > >
> > > > > > > > > > - Write output to ASCII statistics files and
write
> > matched
> > > > > pairs
> > > > > > > to
> > > > > > > > > > NetCDF file
> > > > > > > > > >
> > > > > > > > > > Shall I just add MPR as other output types into
> output_flag
> > > > > > > definition
> > > > > > > > in
> > > > > > > > > > grid_stat_config? maybe like:
> > > > > > > > > >
> > > > > > > > > > output_flag = {
> > > > > > > > > > fho = NONE;
> > > > > > > > > > ctc = BOTH;
> > > > > > > > > > cts = BOTH;
> > > > > > > > > > mctc = NONE;
> > > > > > > > > > mcts = BOTH;
> > > > > > > > > > cnt = BOTH;
> > > > > > > > > > sl1l2 = BOTH;
> > > > > > > > > > sal1l2 = NONE;
> > > > > > > > > > vl1l2 = NONE;
> > > > > > > > > > val1l2 = NONE;
> > > > > > > > > > *MPR = BOTH;*
> > > > > > > > > > pct = NONE;
> > > > > > > > > > pstd = NONE;
> > > > > > > > > > pjc = NONE;
> > > > > > > > > > prc = NONE;
> > > > > > > > > > nbrctc = NONE;
> > > > > > > > > > nbrcts = NONE;
> > > > > > > > > > nbrcnt = NONE;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Thank you for your help
> > > > > > > > > >
> > > > > > > > > > Guohui
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list