[Met_help] [rt.rap.ucar.edu #76724] History for Latest Statanal Issue
John Halley Gotway via RT
met_help at ucar.edu
Thu Jul 7 10:34:15 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John, been on vacation hence the relative quiet. I took your recommendations and now have my persistence data stored in the climo field in the MPR files and I am correctly setting the nbrsize. When I build the PCT files using the following command line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin', '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat', '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT_0Z_pct.stat', '-config', '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated', '-out_fcst_thresh ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0', '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], and the configuration file I stored on the FTP site, I get the contingency table data but with the FCST_THRESH column being blank. When I run statanal again to create the PSTD file using the following command line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin', '/h/data/global/WXQC/data!
/met/ens_cont_tbl', '-out', '/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z', '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the files in the enc_cont_tbl and gives me one set of numbers where it combined data from the three thresolds.
So, as a work around, I run Statanal with the stat file names in ens_cont_tbl wildcarded to point to the proper threshold files using the following command line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin', '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat', '-out', '/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z', '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the following error:
DEBUG 1: Creating STAT-Analysis output file "/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
DEBUG 4: Amending default job with command line options: "-job aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by FCST_THRESH"
WARNING:
WARNING: get_filenames() -> can't stat "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
WARNING:
ERROR :
ERROR : process_search_dirs() -> no STAT files found in the directories specified!
ERROR :
ERROR :
ERROR : main() -> encountered an error value of 1. Calling clean_up() and usage() before exiting.
I would prefer that the FCST_THRESH would appear in the PCT file so I can use the -by FCST_THRESH in the building of the PSTD file but can use the wildcarded names if you can tell me what I am doing wrong.
ALSO, even though I am including a climo value in the MPR file, the Brier Skill Score is coming up NA. Why is this?
I placed the datafiles on the FTP site as well.
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Latest Statanal Issue
From: John Halley Gotway
Time: Mon Jun 13 10:30:38 2016
Bob,
Unfortunately, you've discovered an issue with the current logic. In
version 5.1, we are not able to aggregate together PCT lines and
recompute
BSS because the existing PCT lines do not contain the contingency
tables
for climatology. We'll need to update the logic to better handle this
in a
future release.
Seems to me that we could either... tack the climo contingency table
on to
the end of the existing PCT line... or write the climo contingency
table in
a new PCT line immediately after the forecast one. Either way, we'll
need
to update the parsing/aggregation logic in STAT-Analysis to use the
forecast and climo contingency tables together to compute BSS.
What about when aggregating matched pairs lines and computing PSTD?
Does
that work? Do you get BSS in the output of STAT-Analysis?
Until we're able to update MET to better handle this... here's what
I'd
recommend for 5.1:
- Continue using STAT-Analysis to convert MPR lines to PCT.
- At that point, also run a second STAT-Analysis job to convert MPR
to
PSTD. That will give you a BSS value for each case.
- Then, you can look at the distribution of "daily" BSS values. For
example, look at the mean of those daily BSS scores. In the long run,
we'd
like to compute a true aggregated BSS value... but looking at the mean
of
daily values is also done frequently.
I'll create a development ticket to capture the need for MET to better
aggregate climo data.
Hope that helps.
Thanks,
John
On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Latest Statanal Issue
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724 >
>
>
> Hi John, been on vacation hence the relative quiet. I took your
> recommendations and now have my persistence data stored in the climo
field
> in the MPR files and I am correctly setting the nbrsize. When I
build the
> PCT files using the following command line options -
> ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
>
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT_0Z_pct.stat',
> '-config',
> '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> '-out_fcst_thresh
> ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
'6'], and
> the configuration file I stored on the FTP site, I get the
contingency
> table data but with the FCST_THRESH column being blank. When I run
> statanal again to create the PSTD file using the following command
line
> options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data!
> /met/ens_cont_tbl', '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
'-job
> aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
FCST_VAR',
> '-by FCST_THRESH', '-v', '6'], it reads all the files in the
enc_cont_tbl
> and gives me one set of numbers where it combined data from the
three
> thresolds.
>
>
>
> So, as a work around, I run Statanal with the stat file names in
> ens_cont_tbl wildcarded to point to the proper threshold files using
the
> following command line options - ['/h/WXQC/met-
5.1/bin/stat_analysis',
> '-lookin',
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
'-job
> aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
FCST_VAR',
> '-by FCST_THRESH', '-v', '6'], I get the following error:
>
>
>
> DEBUG 1: Creating STAT-Analysis output file
> "/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
>
> DEBUG 4: Amending default job with command line options: "-job
> aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by
> FCST_THRESH"
>
> WARNING:
>
> WARNING: get_filenames() -> can't stat
> "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
>
> WARNING:
>
> ERROR :
>
> ERROR : process_search_dirs() -> no STAT files found in the
directories
> specified!
>
> ERROR :
>
> ERROR :
>
> ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> and usage() before exiting.
>
>
>
> I would prefer that the FCST_THRESH would appear in the PCT file so
I can
> use the -by FCST_THRESH in the building of the PSTD file but can use
the
> wildcarded names if you can tell me what I am doing wrong.
>
>
>
> ALSO, even though I am including a climo value in the MPR file, the
Brier
> Skill Score is coming up NA. Why is this?
>
>
>
> I placed the datafiles on the FTP site as well.
>
>
>
> Thanks
>
> Bob
>
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76724] Latest Statanal Issue
From: robert.craig.2 at us.af.mil
Time: Mon Jun 13 11:30:18 2016
John, that answers one of my questions. I was planning to have two
sets of PSDT files, one for model data and one for persistence and
then I would calculate BSS outside of MET. In doing it this way the
other question I asked was how to handle the fcst_thresh values. As
I stated in previous emails, this field is blank in my PSDT file
though I thought I set it properly (see previous emails). Also, to
work around that issue I can include the fcst threshold in the
filename but I am having trouble getting the that to work (when using
wildcards it doesn't find my files - see previous email for details).
How can I get FST_THRESH value to show up in the PSTD file and what is
wrong with my syntax for -lookin using wild cards.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, June 13, 2016 11:31 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
Bob,
Unfortunately, you've discovered an issue with the current logic. In
version 5.1, we are not able to aggregate together PCT lines and
recompute BSS because the existing PCT lines do not contain the
contingency tables for climatology. We'll need to update the logic to
better handle this in a future release.
Seems to me that we could either... tack the climo contingency table
on to the end of the existing PCT line... or write the climo
contingency table in a new PCT line immediately after the forecast
one. Either way, we'll need to update the parsing/aggregation logic
in STAT-Analysis to use the forecast and climo contingency tables
together to compute BSS.
What about when aggregating matched pairs lines and computing PSTD?
Does that work? Do you get BSS in the output of STAT-Analysis?
Until we're able to update MET to better handle this... here's what
I'd recommend for 5.1:
- Continue using STAT-Analysis to convert MPR lines to PCT.
- At that point, also run a second STAT-Analysis job to convert MPR
to PSTD. That will give you a BSS value for each case.
- Then, you can look at the distribution of "daily" BSS values. For
example, look at the mean of those daily BSS scores. In the long run,
we'd like to compute a true aggregated BSS value... but looking at the
mean of daily values is also done frequently.
I'll create a development ticket to capture the need for MET to better
aggregate climo data.
Hope that helps.
Thanks,
John
On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Latest Statanal Issue
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724
> >
>
>
> Hi John, been on vacation hence the relative quiet. I took your
> recommendations and now have my persistence data stored in the climo
> field in the MPR files and I am correctly setting the nbrsize. When
I
> build the PCT files using the following command line options -
> ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
>
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT_0
> Z_pct.stat',
> '-config',
> '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> '-out_fcst_thresh
> ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
'6'],
> and the configuration file I stored on the FTP site, I get the
contingency
> table data but with the FCST_THRESH column being blank. When I run
> statanal again to create the PSTD file using the following command
> line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data!
> /met/ens_cont_tbl', '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
> FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the files in
> the enc_cont_tbl and gives me one set of numbers where it combined
> data from the three thresolds.
>
>
>
> So, as a work around, I run Statanal with the stat file names in
> ens_cont_tbl wildcarded to point to the proper threshold files using
> the following command line options -
> ['/h/WXQC/met-5.1/bin/stat_analysis',
> '-lookin',
> '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
> FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the following error:
>
>
>
> DEBUG 1: Creating STAT-Analysis output file
> "/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
>
> DEBUG 4: Amending default job with command line options: "-job
> aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by
> FCST_THRESH"
>
> WARNING:
>
> WARNING: get_filenames() -> can't stat
> "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
>
> WARNING:
>
> ERROR :
>
> ERROR : process_search_dirs() -> no STAT files found in the
> directories specified!
>
> ERROR :
>
> ERROR :
>
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and usage() before exiting.
>
>
>
> I would prefer that the FCST_THRESH would appear in the PCT file so
I
> can use the -by FCST_THRESH in the building of the PSTD file but can
> use the wildcarded names if you can tell me what I am doing wrong.
>
>
>
> ALSO, even though I am including a climo value in the MPR file, the
> Brier Skill Score is coming up NA. Why is this?
>
>
>
> I placed the datafiles on the FTP site as well.
>
>
>
> Thanks
>
> Bob
>
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76724] Latest Statanal Issue
From: robert.craig.2 at us.af.mil
Time: Mon Jun 13 11:36:01 2016
John, I realized I forgot to answer your question. I haven't tried to
aggregate matched pairs lines and compute PSTD since testing has shown
this could be a very slow process for several months of data - that is
why you suggested using the PCT intermediate step. I will try that on
a limited set if data to confirm that it works.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, June 13, 2016 11:31 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
Bob,
Unfortunately, you've discovered an issue with the current logic. In
version 5.1, we are not able to aggregate together PCT lines and
recompute BSS because the existing PCT lines do not contain the
contingency tables for climatology. We'll need to update the logic to
better handle this in a future release.
Seems to me that we could either... tack the climo contingency table
on to the end of the existing PCT line... or write the climo
contingency table in a new PCT line immediately after the forecast
one. Either way, we'll need to update the parsing/aggregation logic
in STAT-Analysis to use the forecast and climo contingency tables
together to compute BSS.
What about when aggregating matched pairs lines and computing PSTD?
Does that work? Do you get BSS in the output of STAT-Analysis?
Until we're able to update MET to better handle this... here's what
I'd recommend for 5.1:
- Continue using STAT-Analysis to convert MPR lines to PCT.
- At that point, also run a second STAT-Analysis job to convert MPR
to PSTD. That will give you a BSS value for each case.
- Then, you can look at the distribution of "daily" BSS values. For
example, look at the mean of those daily BSS scores. In the long run,
we'd like to compute a true aggregated BSS value... but looking at the
mean of daily values is also done frequently.
I'll create a development ticket to capture the need for MET to better
aggregate climo data.
Hope that helps.
Thanks,
John
On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Latest Statanal Issue
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724
> >
>
>
> Hi John, been on vacation hence the relative quiet. I took your
> recommendations and now have my persistence data stored in the climo
> field in the MPR files and I am correctly setting the nbrsize. When
I
> build the PCT files using the following command line options -
> ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
>
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT_0
> Z_pct.stat',
> '-config',
> '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> '-out_fcst_thresh
> ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
'6'],
> and the configuration file I stored on the FTP site, I get the
contingency
> table data but with the FCST_THRESH column being blank. When I run
> statanal again to create the PSTD file using the following command
> line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> '/h/data/global/WXQC/data!
> /met/ens_cont_tbl', '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
> FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the files in
> the enc_cont_tbl and gives me one set of numbers where it combined
> data from the three thresolds.
>
>
>
> So, as a work around, I run Statanal with the stat file names in
> ens_cont_tbl wildcarded to point to the proper threshold files using
> the following command line options -
> ['/h/WXQC/met-5.1/bin/stat_analysis',
> '-lookin',
> '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> '-out',
>
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-by
> FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the following error:
>
>
>
> DEBUG 1: Creating STAT-Analysis output file
> "/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
>
> DEBUG 4: Amending default job with command line options: "-job
> aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by
> FCST_THRESH"
>
> WARNING:
>
> WARNING: get_filenames() -> can't stat
> "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
>
> WARNING:
>
> ERROR :
>
> ERROR : process_search_dirs() -> no STAT files found in the
> directories specified!
>
> ERROR :
>
> ERROR :
>
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and usage() before exiting.
>
>
>
> I would prefer that the FCST_THRESH would appear in the PCT file so
I
> can use the -by FCST_THRESH in the building of the PSTD file but can
> use the wildcarded names if you can tell me what I am doing wrong.
>
>
>
> ALSO, even though I am including a climo value in the MPR file, the
> Brier Skill Score is coming up NA. Why is this?
>
>
>
> I placed the datafiles on the FTP site as well.
>
>
>
> Thanks
>
> Bob
>
>
>
>
>
>
------------------------------------------------
Subject: Latest Statanal Issue
From: John Halley Gotway
Time: Mon Jun 13 11:46:11 2016
Bob,
Yes, you've found that using several months of data to convert MPR to
PCT
is very slow. That's why we're doing the intermediate PCT step.
What I'd like you to do is this...
- You're already running STAT-Analysis to compute that intermediate
PCT
line.
- Now run STAT-Analysis a second time, but instead of "-out_line_type
PCT",
use "-out_line_type PSTD". And check the output PSTD line to make
sure
that the BSS value looks good.
John
On Mon, Jun 13, 2016 at 11:36 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724 >
>
> John, I realized I forgot to answer your question. I haven't tried
to
> aggregate matched pairs lines and compute PSTD since testing has
shown this
> could be a very slow process for several months of data - that is
why you
> suggested using the PCT intermediate step. I will try that on a
limited
> set if data to confirm that it works.
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, June 13, 2016 11:31 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
>
> Bob,
>
> Unfortunately, you've discovered an issue with the current logic.
In
> version 5.1, we are not able to aggregate together PCT lines and
recompute
> BSS because the existing PCT lines do not contain the contingency
tables
> for climatology. We'll need to update the logic to better handle
this in a
> future release.
>
> Seems to me that we could either... tack the climo contingency table
on to
> the end of the existing PCT line... or write the climo contingency
table in
> a new PCT line immediately after the forecast one. Either way,
we'll need
> to update the parsing/aggregation logic in STAT-Analysis to use the
> forecast and climo contingency tables together to compute BSS.
>
> What about when aggregating matched pairs lines and computing PSTD?
Does
> that work? Do you get BSS in the output of STAT-Analysis?
>
> Until we're able to update MET to better handle this... here's what
I'd
> recommend for 5.1:
> - Continue using STAT-Analysis to convert MPR lines to PCT.
> - At that point, also run a second STAT-Analysis job to convert MPR
to
> PSTD. That will give you a BSS value for each case.
> - Then, you can look at the distribution of "daily" BSS values.
For
> example, look at the mean of those daily BSS scores. In the long
run, we'd
> like to compute a true aggregated BSS value... but looking at the
mean of
> daily values is also done frequently.
>
> I'll create a development ticket to capture the need for MET to
better
> aggregate climo data.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Latest Statanal Issue
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724
> > >
> >
> >
> > Hi John, been on vacation hence the relative quiet. I took your
> > recommendations and now have my persistence data stored in the
climo
> > field in the MPR files and I am correctly setting the nbrsize.
When I
> > build the PCT files using the following command line options -
> > ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
> >
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT_0
> > Z_pct.stat',
> > '-config',
> > '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> > '-out_fcst_thresh
> > ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> > '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
'6'],
> > and the configuration file I stored on the FTP site, I get the
> contingency
> > table data but with the FCST_THRESH column being blank. When I
run
> > statanal again to create the PSTD file using the following command
> > line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > '/h/data/global/WXQC/data!
> > /met/ens_cont_tbl', '-out',
> >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> > '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-
by
> > FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the files
in
> > the enc_cont_tbl and gives me one set of numbers where it combined
> > data from the three thresolds.
> >
> >
> >
> > So, as a work around, I run Statanal with the stat file names in
> > ens_cont_tbl wildcarded to point to the proper threshold files
using
> > the following command line options -
> > ['/h/WXQC/met-5.1/bin/stat_analysis',
> > '-lookin',
> > '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> > '-out',
> >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z',
> > '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD', '-
by
> > FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the following
error:
> >
> >
> >
> > DEBUG 1: Creating STAT-Analysis output file
> >
"/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
> >
> > DEBUG 4: Amending default job with command line options: "-job
> > aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by
> > FCST_THRESH"
> >
> > WARNING:
> >
> > WARNING: get_filenames() -> can't stat
> > "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
> >
> > WARNING:
> >
> > ERROR :
> >
> > ERROR : process_search_dirs() -> no STAT files found in the
> > directories specified!
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and usage() before exiting.
> >
> >
> >
> > I would prefer that the FCST_THRESH would appear in the PCT file
so I
> > can use the -by FCST_THRESH in the building of the PSTD file but
can
> > use the wildcarded names if you can tell me what I am doing wrong.
> >
> >
> >
> > ALSO, even though I am including a climo value in the MPR file,
the
> > Brier Skill Score is coming up NA. Why is this?
> >
> >
> >
> > I placed the datafiles on the FTP site as well.
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76724] Latest Statanal Issue
From: robert.craig.2 at us.af.mil
Time: Mon Jun 13 11:55:00 2016
John, I wish I could call you since it time consuming to explain
through email. Per your suggestion below. I have two sets of PCT
files (one for model data and one for persistence). If I run statanal
and out PSTD, the BSS (Brier Skill Score) field will be blank since
the PCT files only contains data for the model and and not the
reference climo (persistence in this case). I will only have the
Brier Score in the file and not Brier Skill Score. Am I missing
something?
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, June 13, 2016 12:46 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
Bob,
Yes, you've found that using several months of data to convert MPR to
PCT is very slow. That's why we're doing the intermediate PCT step.
What I'd like you to do is this...
- You're already running STAT-Analysis to compute that intermediate
PCT line.
- Now run STAT-Analysis a second time, but instead of "-out_line_type
PCT", use "-out_line_type PSTD". And check the output PSTD line to
make sure that the BSS value looks good.
John
On Mon, Jun 13, 2016 at 11:36 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724 >
>
> John, I realized I forgot to answer your question. I haven't tried
to
> aggregate matched pairs lines and compute PSTD since testing has
shown
> this could be a very slow process for several months of data - that
is
> why you suggested using the PCT intermediate step. I will try that
on
> a limited set if data to confirm that it works.
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, June 13, 2016 11:31 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
>
> Bob,
>
> Unfortunately, you've discovered an issue with the current logic.
In
> version 5.1, we are not able to aggregate together PCT lines and
> recompute BSS because the existing PCT lines do not contain the
> contingency tables for climatology. We'll need to update the logic
to
> better handle this in a future release.
>
> Seems to me that we could either... tack the climo contingency table
> on to the end of the existing PCT line... or write the climo
> contingency table in a new PCT line immediately after the forecast
> one. Either way, we'll need to update the parsing/aggregation logic
> in STAT-Analysis to use the forecast and climo contingency tables
together to compute BSS.
>
> What about when aggregating matched pairs lines and computing PSTD?
> Does that work? Do you get BSS in the output of STAT-Analysis?
>
> Until we're able to update MET to better handle this... here's what
> I'd recommend for 5.1:
> - Continue using STAT-Analysis to convert MPR lines to PCT.
> - At that point, also run a second STAT-Analysis job to convert MPR
> to PSTD. That will give you a BSS value for each case.
> - Then, you can look at the distribution of "daily" BSS values.
For
> example, look at the mean of those daily BSS scores. In the long
run,
> we'd like to compute a true aggregated BSS value... but looking at
the
> mean of daily values is also done frequently.
>
> I'll create a development ticket to capture the need for MET to
better
> aggregate climo data.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Latest Statanal Issue
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724
> > >
> >
> >
> > Hi John, been on vacation hence the relative quiet. I took your
> > recommendations and now have my persistence data stored in the
climo
> > field in the MPR files and I am correctly setting the nbrsize.
When
> > I build the PCT files using the following command line options -
> > ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
> >
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT
> > _0
> > Z_pct.stat',
> > '-config',
> > '/h/WXQC/met-5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> > '-out_fcst_thresh
> > ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> > '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
> > '6'], and the configuration file I stored on the FTP site, I get
the
> contingency
> > table data but with the FCST_THRESH column being blank. When I
run
> > statanal again to create the PSTD file using the following command
> > line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > '/h/data/global/WXQC/data!
> > /met/ens_cont_tbl', '-out',
> >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z'
> > , '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD',
> > '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the
> > files in the enc_cont_tbl and gives me one set of numbers where it
> > combined data from the three thresolds.
> >
> >
> >
> > So, as a work around, I run Statanal with the stat file names in
> > ens_cont_tbl wildcarded to point to the proper threshold files
using
> > the following command line options -
> > ['/h/WXQC/met-5.1/bin/stat_analysis',
> > '-lookin',
> > '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> > '-out',
> >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z'
> > , '-job aggregate_stat', '-line_type PCT', '-out_line_type PSTD',
> > '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the following
> > error:
> >
> >
> >
> > DEBUG 1: Creating STAT-Analysis output file
> >
"/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
> >
> > DEBUG 4: Amending default job with command line options: "-job
> > aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR -by
> > FCST_THRESH"
> >
> > WARNING:
> >
> > WARNING: get_filenames() -> can't stat
> > "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
> >
> > WARNING:
> >
> > ERROR :
> >
> > ERROR : process_search_dirs() -> no STAT files found in the
> > directories specified!
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and usage() before exiting.
> >
> >
> >
> > I would prefer that the FCST_THRESH would appear in the PCT file
so
> > I can use the -by FCST_THRESH in the building of the PSTD file but
> > can use the wildcarded names if you can tell me what I am doing
wrong.
> >
> >
> >
> > ALSO, even though I am including a climo value in the MPR file,
the
> > Brier Skill Score is coming up NA. Why is this?
> >
> >
> >
> > I placed the datafiles on the FTP site as well.
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Latest Statanal Issue
From: John Halley Gotway
Time: Mon Jun 13 12:01:32 2016
Bob,
The MPR line contains a column for climatology (CLIMO). If FCST and
CLIMO
are both filled in, I suspect that STAT-Analysis converting MPR to
PSTD
would correctly compute a BSS value using thse FCST and the CLIMO
values.
MET version 5.1 won't be able to do anything with FCST in one PCT line
and
CLIMO in another. However, as you've said, you could aggregate them
separately and then compute BSS "manually" outside of MET.
I will likely be visiting Omaha in July for the cloud vx project. It
would
be great if we could meet for a while, review the details, and figure
out
the best long term solutions in MET for your needs.
John
On Mon, Jun 13, 2016 at 11:55 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724 >
>
> John, I wish I could call you since it time consuming to explain
through
> email. Per your suggestion below. I have two sets of PCT files
(one for
> model data and one for persistence). If I run statanal and out
PSTD, the
> BSS (Brier Skill Score) field will be blank since the PCT files only
> contains data for the model and and not the reference climo
(persistence in
> this case). I will only have the Brier Score in the file and not
Brier
> Skill Score. Am I missing something?
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, June 13, 2016 12:46 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
>
> Bob,
>
> Yes, you've found that using several months of data to convert MPR
to PCT
> is very slow. That's why we're doing the intermediate PCT step.
>
> What I'd like you to do is this...
> - You're already running STAT-Analysis to compute that intermediate
PCT
> line.
> - Now run STAT-Analysis a second time, but instead of "-
out_line_type
> PCT", use "-out_line_type PSTD". And check the output PSTD line to
make
> sure that the BSS value looks good.
>
> John
>
> On Mon, Jun 13, 2016 at 11:36 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724 >
> >
> > John, I realized I forgot to answer your question. I haven't
tried to
> > aggregate matched pairs lines and compute PSTD since testing has
shown
> > this could be a very slow process for several months of data -
that is
> > why you suggested using the PCT intermediate step. I will try
that on
> > a limited set if data to confirm that it works.
> >
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, June 13, 2016 11:31 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [rt.rap.ucar.edu #76724] Latest Statanal Issue
> >
> > Bob,
> >
> > Unfortunately, you've discovered an issue with the current logic.
In
> > version 5.1, we are not able to aggregate together PCT lines and
> > recompute BSS because the existing PCT lines do not contain the
> > contingency tables for climatology. We'll need to update the
logic to
> > better handle this in a future release.
> >
> > Seems to me that we could either... tack the climo contingency
table
> > on to the end of the existing PCT line... or write the climo
> > contingency table in a new PCT line immediately after the forecast
> > one. Either way, we'll need to update the parsing/aggregation
logic
> > in STAT-Analysis to use the forecast and climo contingency tables
> together to compute BSS.
> >
> > What about when aggregating matched pairs lines and computing
PSTD?
> > Does that work? Do you get BSS in the output of STAT-Analysis?
> >
> > Until we're able to update MET to better handle this... here's
what
> > I'd recommend for 5.1:
> > - Continue using STAT-Analysis to convert MPR lines to PCT.
> > - At that point, also run a second STAT-Analysis job to convert
MPR
> > to PSTD. That will give you a BSS value for each case.
> > - Then, you can look at the distribution of "daily" BSS values.
For
> > example, look at the mean of those daily BSS scores. In the long
run,
> > we'd like to compute a true aggregated BSS value... but looking at
the
> > mean of daily values is also done frequently.
> >
> > I'll create a development ticket to capture the need for MET to
better
> > aggregate climo data.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 9, 2016 at 1:05 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 09 13:05:28 2016: Request 76724 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Latest Statanal Issue
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76724
> > > >
> > >
> > >
> > > Hi John, been on vacation hence the relative quiet. I took your
> > > recommendations and now have my persistence data stored in the
climo
> > > field in the MPR files and I am correctly setting the nbrsize.
When
> > > I build the PCT files using the following command line options -
> > > ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > > '/h/data/global/WXQC/data/met/mdlob_pairs', '-out_stat',
> > >
'/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM_APCP_12hr_25_9_PCT
> > > _0
> > > Z_pct.stat',
> > > '-config',
> > > '/h/WXQC/met-
5.1/data/config/STATAnalysisConfig_hira_bs_updated',
> > > '-out_fcst_thresh
> > >
ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0',
> > > '-out_obs_thresh ge1', '-by FCST_VAR', '-by FCST_THRESH', '-v',
> > > '6'], and the configuration file I stored on the FTP site, I get
the
> > contingency
> > > table data but with the FCST_THRESH column being blank. When I
run
> > > statanal again to create the PSTD file using the following
command
> > > line options - ['/h/WXQC/met-5.1/bin/stat_analysis', '-lookin',
> > > '/h/data/global/WXQC/data!
> > > /met/ens_cont_tbl', '-out',
> > >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z'
> > > , '-job aggregate_stat', '-line_type PCT', '-out_line_type
PSTD',
> > > '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], it reads all the
> > > files in the enc_cont_tbl and gives me one set of numbers where
it
> > > combined data from the three thresolds.
> > >
> > >
> > >
> > > So, as a work around, I run Statanal with the stat file names in
> > > ens_cont_tbl wildcarded to point to the proper threshold files
using
> > > the following command line options -
> > > ['/h/WXQC/met-5.1/bin/stat_analysis',
> > > '-lookin',
> > > '/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat',
> > > '-out',
> > >
'/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z'
> > > , '-job aggregate_stat', '-line_type PCT', '-out_line_type
PSTD',
> > > '-by FCST_VAR', '-by FCST_THRESH', '-v', '6'], I get the
following
> > > error:
> > >
> > >
> > >
> > > DEBUG 1: Creating STAT-Analysis output file
> > >
"/h/data/global/WXQC/data/met/summary/GALWEM_APCP_12hr_50_9_PSTD_0Z"
> > >
> > > DEBUG 4: Amending default job with command line options: "-job
> > > aggregate_stat -line_type PCT -out_line_type PSTD -by FCST_VAR
-by
> > > FCST_THRESH"
> > >
> > > WARNING:
> > >
> > > WARNING: get_filenames() -> can't stat
> > > "/h/data/global/WXQC/data/met/ens_cont_tbl/GALWEM*_50_9_*.stat"
> > >
> > > WARNING:
> > >
> > > ERROR :
> > >
> > > ERROR : process_search_dirs() -> no STAT files found in the
> > > directories specified!
> > >
> > > ERROR :
> > >
> > > ERROR :
> > >
> > > ERROR : main() -> encountered an error value of 1. Calling
> > > clean_up() and usage() before exiting.
> > >
> > >
> > >
> > > I would prefer that the FCST_THRESH would appear in the PCT file
so
> > > I can use the -by FCST_THRESH in the building of the PSTD file
but
> > > can use the wildcarded names if you can tell me what I am doing
wrong.
> > >
> > >
> > >
> > > ALSO, even though I am including a climo value in the MPR file,
the
> > > Brier Skill Score is coming up NA. Why is this?
> > >
> > >
> > >
> > > I placed the datafiles on the FTP site as well.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Bob
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list