[Met_help] [rt.rap.ucar.edu #47861] History for slowness of stat_analysis
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Mon Jul 11 09:12:57 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello MET_Help
We are in the process of setting up some basic surface verification for selected vx_mask NCEP regions, based on CONUS-scale 4-km WRF forecasts and observations from MADIS processed into MET sfc files using ascii2nc.
Needless to say, there are a lot of data pairs (~77,000 per forecast output time) that lead to excruciatingly slow performance of stat_analysis.
We had a job running all weekend just to generate overall verification stats for 0-37 h forecasts for 2 different model runs for T, Td, u/v, wind speed, and PMSL over 3 NCEP verification regions. The script is STILL running this morning and will likely take another couple days to complete. And we're only running on a single forecast run - it's not even compositing a series of forecasts.
We have tried to filter out the data by forecast variable, forecast hour, vx_mask, etc., but it still takes very long. It appears that stat_analysis reads in all the *.stat files in whatever directory(ies) it is looking in, regardless of the filtered forecast hour specified.
I was wondering if you have any recommendations and examples on how to speed up the stat_analysis program so that it won't try to read ALL *.stat files in a given directory before filtering out lines of data for calculations? Also, perhaps we are not using the filter job correctly. An example on how to run that correctly is needed as well.
Thanks for any help you can provide,
Jonathan
--------------------------------------------------------------------------------------------------
Jonathan Case, ENSCO Inc.
NASA Short-term Prediction Research and Transition Center (aka SPoRT Center)
320 Sparkman Drive, Room 3062
Huntsville, AL 35805
Voice: 256.961.7504
Fax: 256.961.7788
Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
--------------------------------------------------------------------------------------------------
"Whether the weather is cold, or whether the weather is hot, we'll weather
the weather whether we like it or not!"
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #47861] slowness of stat_analysis
From: John Halley Gotway
Time: Mon Jun 27 11:49:00 2011
Jonathan,
I'm happy to help you try to figure out how to improve the speed of
STAT-Analysis. Hopefully we can help you structure the STAT-Analysis
jobs
to get them to run as quickly as possible.
But the place to start is to figure out exactly what type of jobs you
are
running. Can you please send me the command line argument (and config
file, if you're using one)? You state that you're trying to "filter"
out
the data using various stratifications, but I'm not quite sure why.
What
questions are you ultimately trying to answer? Are you trying to
summarize performance over a long period of time?
All the filter jobs do is read in a bunch of STAT lines, apply the
filtering criteria you've selected, and write the lines that meet that
criteria out to an ASCII file.
If you actually want to compute aggregated statistics over a long
period
of time, you probably want to be running "aggregate_stat" type jobs.
For example...
-job aggregate_stat -line_type CTC -out_line_type CTS (other filtering
info)
-job aggregate_stat -line_type SL1L2 -out_line_type CNT (other
filtering
info)
The first job above will aggregate together contingency table counts
and
compute the corresponding categorical statistics from those sums. The
second will aggregate together partial sums and compute the
corresponding
continuous statistics from those sums. So you should decide what
types of
jobs you want to be running.
Regarding the -lookin command line argument, you're right, STAT-
Analysis
will search recursively through that directory and read all files
ending
in ".stat". Please note the you can use the -lookin option an
arbitrary
number of times to specify directories and/or filenames to be parsed.
And if you've chosen to write out the MPR lines (matched pairs) from
Point-Stat, that'll probably slow down the analysis considerably.
Hopefully, that'll help get us started. I suspect this will take a
few
iterations to get STAT-Analysis to run faster.
Thanks,
John Halley Gotway
met_help at ucar.edu
>
> Mon Jun 27 10:09:59 2011: Request 47861 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: slowness of stat_analysis
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=47861 >
>
>
> Hello MET_Help
>
> We are in the process of setting up some basic surface verification
for
> selected vx_mask NCEP regions, based on CONUS-scale 4-km WRF
forecasts and
> observations from MADIS processed into MET sfc files using ascii2nc.
>
> Needless to say, there are a lot of data pairs (~77,000 per forecast
> output time) that lead to excruciatingly slow performance of
> stat_analysis.
> We had a job running all weekend just to generate overall
verification
> stats for 0-37 h forecasts for 2 different model runs for T, Td,
u/v, wind
> speed, and PMSL over 3 NCEP verification regions. The script is
STILL
> running this morning and will likely take another couple days to
complete.
> And we're only running on a single forecast run - it's not even
> compositing a series of forecasts.
>
> We have tried to filter out the data by forecast variable, forecast
hour,
> vx_mask, etc., but it still takes very long. It appears that
> stat_analysis reads in all the *.stat files in whatever
directory(ies) it
> is looking in, regardless of the filtered forecast hour specified.
>
> I was wondering if you have any recommendations and examples on how
to
> speed up the stat_analysis program so that it won't try to read ALL
*.stat
> files in a given directory before filtering out lines of data for
> calculations? Also, perhaps we are not using the filter job
correctly.
> An example on how to run that correctly is needed as well.
>
> Thanks for any help you can provide,
> Jonathan
>
>
--------------------------------------------------------------------------------------------------
> Jonathan Case, ENSCO Inc.
> NASA Short-term Prediction Research and Transition Center (aka SPoRT
> Center)
> 320 Sparkman Drive, Room 3062
> Huntsville, AL 35805
> Voice: 256.961.7504
> Fax: 256.961.7788
> Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
>
--------------------------------------------------------------------------------------------------
>
> "Whether the weather is cold, or whether the weather is hot, we'll
weather
> the weather whether we like it or not!"
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #47861] slowness of stat_analysis
From: Case, Jonathan[ENSCO INC]
Time: Mon Jun 27 12:36:04 2011
Hi John H-G,
Thanks for the quick reply. I was thinking of
including the job command line we're using. That would have been more
helpful for you.
For the time being, we want to simply summarize
sfc verification stats by forecast hour for 2 different model
configurations on a daily basis. To accomplish this, we run point_stat
on all forecast hours for each model run to create the various stat
files.
Now, we want summary stats for different verification
zones using stat_analysis. The job command in the config file is:
jobname="-job aggregate_stat -line_type MPR -out_line_type CNT
-fcst_lead ${fhr} -fcst_var ${var} -obtype ADPSFC -vx_mask $region"
where we are looping through $fhr, $var, and $region.
Unfortunately, the sticking point appears to be I/O. stat_analysis is
trying to read all *.stat files in the forecast directory containing
37 different forecast stat files (one per forecast hour in a 36-h
forecast run). Each *.stat file is nearly 80 mb in size due to our
large domain and numerous observations. We have about 77,000 pairs of
fcst-obs at each forecast hour.
I think a quick fix is to make
our script "smarter" by linking in only the *.stat files we want to
process and setting the "-lookin" directory to point to the directory
containing the smaller subset of *.stat files. I tested this out, and
stat_analysis ran orders of magnitude faster.
So, I suppose my
final question to you is whether the MET software can be improved to
reduce the tremendous I/O overhead associated with large domains and
large numbers of fcst-obs pairs? Specifically, it seems like the
stat_analysis program could be designed to key in on the specific
forecast hour file or files that is set in the filter options (i.e.
-fcst_lead[]), instead of trying to read every *.stat file in a
particular directory and THEN parsing out the lines of data.
Hopefully, you can follow what I'm saying.
Thanks again,
Jonathan
-----Original Message-----
From: RAL HelpDesk {for John Halley
Gotway} [mailto:met_help at ucar.edu]
Sent: Monday, June 27, 2011 12:49
PM
To: Case, Jonathan (MSFC-VP61)[ENSCO INC]
Cc: Shafer, Jaclyn A.
(MSFC-VP34)[UAH]; johnhg at ucar.edu
Subject: Re: [rt.rap.ucar.edu
#47861] slowness of stat_analysis
Jonathan,
I'm happy to help
you try to figure out how to improve the speed of
STAT-Analysis.
Hopefully we can help you structure the STAT-Analysis jobs
to get
them to run as quickly as possible.
But the place to start is to
figure out exactly what type of jobs you are
running. Can you please
send me the command line argument (and config
file, if you're using
one)? You state that you're trying to "filter" out
the data using
various stratifications, but I'm not quite sure why. What
questions
are you ultimately trying to answer? Are you trying to
summarize
performance over a long period of time?
All the filter jobs do is
read in a bunch of STAT lines, apply the
filtering criteria you've
selected, and write the lines that meet that
criteria out to an ASCII
file.
If you actually want to compute aggregated statistics over a
long period
of time, you probably want to be running "aggregate_stat"
type jobs.
For example...
-job aggregate_stat -line_type CTC
-out_line_type CTS (other filtering info)
-job aggregate_stat
-line_type SL1L2 -out_line_type CNT (other filtering
info)
The
first job above will aggregate together contingency table counts and
compute the corresponding categorical statistics from those sums. The
second will aggregate together partial sums and compute the
corresponding
continuous statistics from those sums. So you should
decide what types of
jobs you want to be running.
Regarding the
-lookin command line argument, you're right, STAT-Analysis
will
search recursively through that directory and read all files ending
in ".stat". Please note the you can use the -lookin option an
arbitrary
number of times to specify directories and/or filenames to
be parsed.
And if you've chosen to write out the MPR lines (matched
pairs) from
Point-Stat, that'll probably slow down the analysis
considerably.
Hopefully, that'll help get us started. I suspect
this will take a few
iterations to get STAT-Analysis to run faster.
Thanks,
John Halley Gotway
met_help at ucar.edu
>
> Mon Jun 27
10:09:59 2011: Request 47861 was acted upon.
> Transaction: Ticket
created by jonathan.case-1 at nasa.gov
> Queue: met_help
>
Subject: slowness of stat_analysis
> Owner: Nobody
>
Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=47861 >
>
>
> Hello MET_Help
>
> We are in the process of setting up some basic
surface verification for
> selected vx_mask NCEP regions, based on
CONUS-scale 4-km WRF forecasts and
> observations from MADIS
processed into MET sfc files using ascii2nc.
>
> Needless to say,
there are a lot of data pairs (~77,000 per forecast
> output time)
that lead to excruciatingly slow performance of
> stat_analysis.
>
We had a job running all weekend just to generate overall verification
> stats for 0-37 h forecasts for 2 different model runs for T, Td,
u/v, wind
> speed, and PMSL over 3 NCEP verification regions. The
script is STILL
> running this morning and will likely take another
couple days to complete.
> And we're only running on a single
forecast run - it's not even
> compositing a series of forecasts.
>
> We have tried to filter out the data by forecast variable, forecast
hour,
> vx_mask, etc., but it still takes very long. It appears that
> stat_analysis reads in all the *.stat files in whatever
directory(ies) it
> is looking in, regardless of the filtered
forecast hour specified.
>
> I was wondering if you have any
recommendations and examples on how to
> speed up the stat_analysis
program so that it won't try to read ALL *.stat
> files in a given
directory before filtering out lines of data for
> calculations?
Also, perhaps we are not using the filter job correctly.
> An example
on how to run that correctly is needed as well.
>
> Thanks for any
help you can provide,
> Jonathan
>
>
--------------------------------------------------------------------------------------------------
> Jonathan Case, ENSCO Inc.
> NASA Short-term Prediction Research and
Transition Center (aka SPoRT
> Center)
> 320 Sparkman Drive, Room
3062
> Huntsville, AL 35805
> Voice: 256.961.7504
> Fax:
256.961.7788
> Emails: Jonathan.Case-1 at nasa.gov /
case.jonathan at ensco.com
>
--------------------------------------------------------------------------------------------------
>
> "Whether the weather is cold, or whether the weather is hot,
we'll weather
> the weather whether we like it or not!"
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #47861] slowness of stat_analysis
From: John Halley Gotway
Time: Mon Jun 27 13:02:00 2011
Jonathan,
OK, so I was concerned that you might be taking this sort of approach.
When you use Point-Stat to dump out the MPR lines, you get very fine
control of the fcst/obs pairs. But then the downside is what you're
seeing - a ridiculous amount of output. Here's a couple of things to
try:
(1) Instead of converting MPR -> CNT, please try converting SL1L2 ->
CNT.
There are fewer statistics that can be derived in this way, but it
should
be much faster.
(2) If you want to continue with MPR -> CNT, try disabling rank
correlations and bootstrap confidence intervals:
-rank_corr_flag 0 -n_boot_rep 0 (*** I think these are correct, but
you
should check***)
Lastly, we dump out the MPR lines mostly for research and to be run on
small datasets - not in the way you're using it over a large dataset
and
time period.
Hope that helps.
John
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=47861 >
>
> Hi John H-G,
>
> Thanks for the quick reply. I was thinking of including the job
command
> line we're using. That would have been more helpful for you.
>
> For the time being, we want to simply summarize sfc verification
stats by
> forecast hour for 2 different model configurations on a daily basis.
To
> accomplish this, we run point_stat on all forecast hours for each
model
> run to create the various stat files.
>
> Now, we want summary stats for different verification zones using
> stat_analysis. The job command in the config file is:
>
> jobname="-job aggregate_stat -line_type MPR -out_line_type CNT
-fcst_lead
> ${fhr} -fcst_var ${var} -obtype ADPSFC -vx_mask $region"
>
> where we are looping through $fhr, $var, and $region.
>
> Unfortunately, the sticking point appears to be I/O. stat_analysis
is
> trying to read all *.stat files in the forecast directory containing
37
> different forecast stat files (one per forecast hour in a 36-h
forecast
> run). Each *.stat file is nearly 80 mb in size due to our large
domain
> and numerous observations. We have about 77,000 pairs of fcst-obs
at each
> forecast hour.
>
> I think a quick fix is to make our script "smarter" by linking in
only the
> *.stat files we want to process and setting the "-lookin" directory
to
> point to the directory containing the smaller subset of *.stat
files. I
> tested this out, and stat_analysis ran orders of magnitude faster.
>
> So, I suppose my final question to you is whether the MET software
can be
> improved to reduce the tremendous I/O overhead associated with large
> domains and large numbers of fcst-obs pairs? Specifically, it seems
like
> the stat_analysis program could be designed to key in on the
specific
> forecast hour file or files that is set in the filter options (i.e.
> -fcst_lead[]), instead of trying to read every *.stat file in a
particular
> directory and THEN parsing out the lines of data. Hopefully, you
can
> follow what I'm saying.
>
> Thanks again,
> Jonathan
>
>
>
> -----Original Message-----
> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Sent: Monday, June 27, 2011 12:49 PM
> To: Case, Jonathan (MSFC-VP61)[ENSCO INC]
> Cc: Shafer, Jaclyn A. (MSFC-VP34)[UAH]; johnhg at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #47861] slowness of stat_analysis
>
> Jonathan,
>
> I'm happy to help you try to figure out how to improve the speed of
> STAT-Analysis. Hopefully we can help you structure the STAT-
Analysis jobs
> to get them to run as quickly as possible.
>
> But the place to start is to figure out exactly what type of jobs
you are
> running. Can you please send me the command line argument (and
config
> file, if you're using one)? You state that you're trying to
"filter" out
> the data using various stratifications, but I'm not quite sure why.
What
> questions are you ultimately trying to answer? Are you trying to
> summarize performance over a long period of time?
>
> All the filter jobs do is read in a bunch of STAT lines, apply the
> filtering criteria you've selected, and write the lines that meet
that
> criteria out to an ASCII file.
>
> If you actually want to compute aggregated statistics over a long
period
> of time, you probably want to be running "aggregate_stat" type jobs.
>
> For example...
> -job aggregate_stat -line_type CTC -out_line_type CTS (other
filtering
> info)
> -job aggregate_stat -line_type SL1L2 -out_line_type CNT (other
filtering
> info)
>
> The first job above will aggregate together contingency table counts
and
> compute the corresponding categorical statistics from those sums.
The
> second will aggregate together partial sums and compute the
corresponding
> continuous statistics from those sums. So you should decide what
types of
> jobs you want to be running.
>
> Regarding the -lookin command line argument, you're right, STAT-
Analysis
> will search recursively through that directory and read all files
ending
> in ".stat". Please note the you can use the -lookin option an
arbitrary
> number of times to specify directories and/or filenames to be
parsed.
>
> And if you've chosen to write out the MPR lines (matched pairs) from
> Point-Stat, that'll probably slow down the analysis considerably.
>
> Hopefully, that'll help get us started. I suspect this will take a
few
> iterations to get STAT-Analysis to run faster.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>>
>> Mon Jun 27 10:09:59 2011: Request 47861 was acted upon.
>> Transaction: Ticket created by jonathan.case-1 at nasa.gov
>> Queue: met_help
>> Subject: slowness of stat_analysis
>> Owner: Nobody
>> Requestors: jonathan.case-1 at nasa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=47861 >
>>
>>
>> Hello MET_Help
>>
>> We are in the process of setting up some basic surface verification
for
>> selected vx_mask NCEP regions, based on CONUS-scale 4-km WRF
forecasts
>> and
>> observations from MADIS processed into MET sfc files using
ascii2nc.
>>
>> Needless to say, there are a lot of data pairs (~77,000 per
forecast
>> output time) that lead to excruciatingly slow performance of
>> stat_analysis.
>> We had a job running all weekend just to generate overall
verification
>> stats for 0-37 h forecasts for 2 different model runs for T, Td,
u/v,
>> wind
>> speed, and PMSL over 3 NCEP verification regions. The script is
STILL
>> running this morning and will likely take another couple days to
>> complete.
>> And we're only running on a single forecast run - it's not even
>> compositing a series of forecasts.
>>
>> We have tried to filter out the data by forecast variable, forecast
>> hour,
>> vx_mask, etc., but it still takes very long. It appears that
>> stat_analysis reads in all the *.stat files in whatever
directory(ies)
>> it
>> is looking in, regardless of the filtered forecast hour specified.
>>
>> I was wondering if you have any recommendations and examples on how
to
>> speed up the stat_analysis program so that it won't try to read ALL
>> *.stat
>> files in a given directory before filtering out lines of data for
>> calculations? Also, perhaps we are not using the filter job
correctly.
>> An example on how to run that correctly is needed as well.
>>
>> Thanks for any help you can provide,
>> Jonathan
>>
>>
--------------------------------------------------------------------------------------------------
>> Jonathan Case, ENSCO Inc.
>> NASA Short-term Prediction Research and Transition Center (aka
SPoRT
>> Center)
>> 320 Sparkman Drive, Room 3062
>> Huntsville, AL 35805
>> Voice: 256.961.7504
>> Fax: 256.961.7788
>> Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
>>
--------------------------------------------------------------------------------------------------
>>
>> "Whether the weather is cold, or whether the weather is hot, we'll
>> weather
>> the weather whether we like it or not!"
>>
>>
>
>
>
>
------------------------------------------------
Subject: slowness of stat_analysis
From: John Halley Gotway
Time: Fri Jul 08 10:22:51 2011
Jon,
I wanted to check in on this to see if you're still having issues with
how slowly STAT-Analysis is running. Is there anything I can do to
help?
Thanks,
John
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #47861] slowness of stat_analysis
From: Case, Jonathan[ENSCO INC]
Time: Sun Jul 10 07:15:12 2011
Hi John H-G,
For now, I solved the problem by simply modifying my script to
minimize the number of .stat files being read to do the computations.
That sped things up by a factor of 50-100 by only using the necessary
.stat files, linking them into the working directory.
-JonC
-----Original Message-----
From: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Sent: Friday, July 08, 2011 11:23 AM
To: Case, Jonathan (MSFC-VP61)[ENSCO INC]
Cc: Shafer, Jaclyn A. (MSFC-VP34)[UAH]; johnhg at ucar.edu
Subject: [rt.rap.ucar.edu #47861] slowness of stat_analysis
Jon,
I wanted to check in on this to see if you're still having issues with
how slowly STAT-Analysis is running. Is there anything I can do to
help?
Thanks,
John
------------------------------------------------
More information about the Met_help
mailing list