[Met_help] [rt.rap.ucar.edu #81605] History for question on weekly/monthly processing using stat_analysis
John Halley Gotway via RT
met_help at ucar.edu
Wed Sep 6 11:50:34 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
What would be the easiest way to set up a stat_analysis job for weekly or
monthly processing, using any of the commands (aggregate, summary or
aggregate_stat)? Would it be easiest to separate files into weekly
directories, or, can a time range be specified on the command line?
Thanks in advance!
Roz
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question on weekly/monthly processing using stat_analysis
From: John Halley Gotway
Time: Tue Aug 15 09:50:23 2017
Hello Roz,
I see that you're looking for advice on how to use STAT-Analysis to
filter
by time.
Yes, the STAT-Analysis jobs are very flexible in filtering by timing
information. Starting on line 2147 the file met-
6.0/data/config/README
contains a description of the job filtering options that are
supported,
including, the following for timing info:
2154 // "-fcst_lead HHMMSS"
2155 // "-obs_lead HHMMSS"
2156 // "-fcst_valid_beg YYYYMMDD[_HH[MMSS]]" (use once)
2157 // "-fcst_valid_end YYYYMMDD[_HH[MMSS]]" (use once)
2158 // "-obs_valid_beg YYYYMMDD[_HH[MMSS]]" (use once)
2159 // "-obs_valid_end YYYYMMDD[_HH[MMSS]]" (use once)
2160 // "-fcst_init_beg YYYYMMDD[_HH[MMSS]]" (use once)
2161 // "-fcst_init_end YYYYMMDD[_HH[MMSS]]" (use once)
2162 // "-obs_init_beg YYYYMMDD[_HH[MMSS]]" (use once)
2163 // "-obs_init_end YYYYMMDD[_HH[MMSS]]" (use once)
2164 // "-fcst_init_hour HH[MMSS]"
2165 // "-obs_init_hour HH[MMSS]"
2166 // "-fcst_valid_hour" HH[MMSS]
2167 // "-obs_valid_hour" HH[MMSS]
When you run STAT-Analysis, the "-lookin" command line option tells it
where to look for input data. If you pass "-lookin" the name of a
file,
it'll read STAT lines from that file regardless of the file extension.
If
you pass "-lookin" the name of a directory, it'll search recursively
through that directory looking only for files ending in ".stat".
While you could do all the filtering using the job command options,
the
more data you pass to STAT-Analysis, the slower it'll run. So I do
think
it's a good idea to organize your MET output files in some sort of
date
directory structure. And in your script, only pass to STAT-Analysis
the
directory or directories that it actually needs to process.
Hope that helps.
Thanks,
John
On Mon, Aug 14, 2017 at 1:39 PM, Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> Mon Aug 14 13:39:17 2017: Request 81605 was acted upon.
> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> Queue: met_help
> Subject: question on weekly/monthly processing using
stat_analysis
> Owner: Nobody
> Requestors: rosalyn.maccracken at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81605 >
>
>
> What would be the easiest way to set up a stat_analysis job for
weekly or
> monthly processing, using any of the commands (aggregate, summary or
> aggregate_stat)? Would it be easiest to separate files into
weekly
> directories, or, can a time range be specified on the command line?
>
> Thanks in advance!
>
> Roz
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applilcations Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: question on weekly/monthly processing using stat_analysis
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Aug 15 10:27:36 2017
Hi John,
So, you think it's better to separate my *.stat files into date
directories. Can you pass in multiple directories to look in? I
think I
remember something like that in the training class, but, I don't
remember
if it was that tool. Because the next question I would ask you is,
what if
I separate each day by directory and I wanted to do a weekly or
monthly
output (job=aggregate, correct?) or average (job=summary, correct?) ?
Can
I do that? Would I use some sort of ascii file, and have
stat_analysis
read from that?
Oh, one other question with stat_analysis. If I wanted to calculate
the
monthly average values, or differences, at each data point (from the
*mpr.txt file, because the data points overlap), would I use the
summary
job and define which directories to use, much like above? What I'm
interested in doing is 2 things: (1) plot a monthly average difference
map
of the differences between the ASCAT and GFS, and (2) plot a monthly
scatterplot of GFS vs. ASCAT to see the biases (the actual values).
I'm
not sure if python will complain at the 2nd idea, since there will be
alot
of data points. That might have to be monthly averages as
well...we'll
see. Anyway, I can do both with the "summary' job, correct?
Thanks for our help!
Roz
On Tue, Aug 15, 2017 at 11:50 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hello Roz,
>
> I see that you're looking for advice on how to use STAT-Analysis to
filter
> by time.
>
> Yes, the STAT-Analysis jobs are very flexible in filtering by timing
> information. Starting on line 2147 the file met-
6.0/data/config/README
> contains a description of the job filtering options that are
supported,
> including, the following for timing info:
>
> 2154 // "-fcst_lead HHMMSS"
> 2155 // "-obs_lead HHMMSS"
> 2156 // "-fcst_valid_beg YYYYMMDD[_HH[MMSS]]" (use once)
> 2157 // "-fcst_valid_end YYYYMMDD[_HH[MMSS]]" (use once)
> 2158 // "-obs_valid_beg YYYYMMDD[_HH[MMSS]]" (use once)
> 2159 // "-obs_valid_end YYYYMMDD[_HH[MMSS]]" (use once)
> 2160 // "-fcst_init_beg YYYYMMDD[_HH[MMSS]]" (use once)
> 2161 // "-fcst_init_end YYYYMMDD[_HH[MMSS]]" (use once)
> 2162 // "-obs_init_beg YYYYMMDD[_HH[MMSS]]" (use once)
> 2163 // "-obs_init_end YYYYMMDD[_HH[MMSS]]" (use once)
> 2164 // "-fcst_init_hour HH[MMSS]"
> 2165 // "-obs_init_hour HH[MMSS]"
> 2166 // "-fcst_valid_hour" HH[MMSS]
> 2167 // "-obs_valid_hour" HH[MMSS]
>
> When you run STAT-Analysis, the "-lookin" command line option tells
it
> where to look for input data. If you pass "-lookin" the name of a
file,
> it'll read STAT lines from that file regardless of the file
extension. If
> you pass "-lookin" the name of a directory, it'll search recursively
> through that directory looking only for files ending in ".stat".
>
> While you could do all the filtering using the job command options,
the
> more data you pass to STAT-Analysis, the slower it'll run. So I do
think
> it's a good idea to organize your MET output files in some sort of
date
> directory structure. And in your script, only pass to STAT-Analysis
the
> directory or directories that it actually needs to process.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Mon, Aug 14, 2017 at 1:39 PM, Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 14 13:39:17 2017: Request 81605 was acted upon.
> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > Queue: met_help
> > Subject: question on weekly/monthly processing using
stat_analysis
> > Owner: Nobody
> > Requestors: rosalyn.maccracken at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81605 >
> >
> >
> > What would be the easiest way to set up a stat_analysis job for
weekly or
> > monthly processing, using any of the commands (aggregate, summary
or
> > aggregate_stat)? Would it be easiest to separate files into
weekly
> > directories, or, can a time range be specified on the command
line?
> >
> > Thanks in advance!
> >
> > Roz
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applilcations Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list