[Met_help] [rt.rap.ucar.edu #42228] History for Point-Stat questions
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Wed Feb 23 14:50:00 MST 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
MET-Help,
Couple of quick questions for you:
1. Let's say I have 2-meter (above ground) observations but the WPP has
only given me usable surface values (from the forecasts) --> don't ask me
why but the post-processor is having trouble calculating the 2-meter values
(I'm still investigating that one). My question is this:
Will Point-Stat have trouble doing the stats because the observations are
at 2-meters and the forecast values are at the model surface?
2. Also, we discussed using MPR output from Point-Stat as the input to the
Stat-Analysis tool - but isn't the MPR output very large and
space-consuming? I've got about a month's worth of data at hourly
intervals out to 3.5 days for 00z and 12z cycles.
Thanks,
James
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42228]
From: John Halley Gotway
Time: Thu Nov 18 10:29:48 2010
James,
I created a separate ticket for these questions.
(1) If you're having issues running WRF-Post, I'd recommend writing
WRF-Help for assistance. But you should still be able to run Point-
Stat to perform the verification you want. I suspect that
something like the following would do the trick:
fcst_field[] = [ "TMP/Z0" ];
obs_field[] = [ "TMP/Z2" ];
message_type = [ "APDSFC" ];
Give that a shot and let me know if you get stuck.
(2) Yes, the MPR output from Point-Stat is rather large. It creates
one line of output for each matched pair. For 30 days, initialized at
00Z and 12Z, with hourly output out to 84 hours, that'd be
30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR lines,
where n_fields is the number of variables/levels you're verifying and
n_locations is the number of stations you're using.
Perhaps you could give it a shot for a few days and see if the files
are too large. After you generated MPR output for each verification
time, you'd run STAT-Analysis to aggregate those MPR values at
each station of interest and compute statistics.
If it would help, you could define a masking polyline region and pass
it to Point-Stat using "mask_poly" - then Point-Stat would only dump
out MPR information for those observations falling inside
that polyline.
Thanks,
John
On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
> Transaction: Ticket created by johnhg
> Queue: met_help
> Subject: (No subject given)
> Owner: johnhg
> Requestors: jpcipria at us.ibm.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> MET-Help,
>
> Couple of quick questions for you:
>
> 1. Let's say I have 2-meter (above ground) observations but the WPP
has
> only given me usable surface values (from the forecasts) --> don't
ask me
> why but the post-processor is having trouble calculating the 2-meter
values
> (I'm still investigating that one). My question is this:
>
> Will Point-Stat have trouble doing the stats because the
observations are
> at 2-meters and the forecast values are at the model surface?
>
> 2. Also, we discussed using MPR output from Point-Stat as the input
to the
> Stat-Analysis tool - but isn't the MPR output very large and
> space-consuming? I've got about a month's worth of data at hourly
> intervals out to 3.5 days for 00z and 12z cycles.
>
> Thanks,
> James
------------------------------------------------
Subject: Point-Stat questions
From: James P Cipriani
Time: Mon Nov 22 06:45:48 2010
Hi John,
Thanks for the suggestions. As it turns out, using different levels
in
point stat does indeed work.
As for running aggregate_stat on the point_stat output, I was
wondering:
Is it possible to run the aggregate_stat tool for one month and obtain
monthly averages (for each station) at the 24-hour or 48-hour lead
times
(taking into account each individual hour's values)? I think I know
how to
run the tool for the 24-hour or 48-hour lead times for each day of the
month; but running for the entire month all at once isn't exactly
clear.
Thanks.
From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
To: James P Cipriani/Watson/IBM at IBMUS
Date: 11/18/2010 12:30 PM
Subject: Re: [rt.rap.ucar.edu #42228]
James,
I created a separate ticket for these questions.
(1) If you're having issues running WRF-Post, I'd recommend writing
WRF-Help for assistance. But you should still be able to run Point-
Stat to
perform the verification you want. I suspect that
something like the following would do the trick:
fcst_field[] = [ "TMP/Z0" ];
obs_field[] = [ "TMP/Z2" ];
message_type = [ "APDSFC" ];
Give that a shot and let me know if you get stuck.
(2) Yes, the MPR output from Point-Stat is rather large. It creates
one
line of output for each matched pair. For 30 days, initialized at 00Z
and
12Z, with hourly output out to 84 hours, that'd be
30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR lines,
where n_fields is the number of variables/levels you're verifying and
n_locations is the number of stations you're using.
Perhaps you could give it a shot for a few days and see if the files
are
too large. After you generated MPR output for each verification time,
you'd run STAT-Analysis to aggregate those MPR values at
each station of interest and compute statistics.
If it would help, you could define a masking polyline region and pass
it to
Point-Stat using "mask_poly" - then Point-Stat would only dump out MPR
information for those observations falling inside
that polyline.
Thanks,
John
On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
> Transaction: Ticket created by johnhg
> Queue: met_help
> Subject: (No subject given)
> Owner: johnhg
> Requestors: jpcipria at us.ibm.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> MET-Help,
>
> Couple of quick questions for you:
>
> 1. Let's say I have 2-meter (above ground) observations but the WPP
has
> only given me usable surface values (from the forecasts) --> don't
ask me
> why but the post-processor is having trouble calculating the 2-meter
values
> (I'm still investigating that one). My question is this:
>
> Will Point-Stat have trouble doing the stats because the
observations are
> at 2-meters and the forecast values are at the model surface?
>
> 2. Also, we discussed using MPR output from Point-Stat as the input
to
the
> Stat-Analysis tool - but isn't the MPR output very large and
> space-consuming? I've got about a month's worth of data at hourly
> intervals out to 3.5 days for 00z and 12z cycles.
>
> Thanks,
> James
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42228]
From: John Halley Gotway
Time: Mon Nov 22 09:03:19 2010
James,
When you run the STAT-Analysis tool, you tell it where to look for
input using the "-lookin" command line argument. You can use "-
lookin" to pass a STAT file name directly or a directory name. If
you pass it a directory name, STAT-Analysis will search recursively
through that directory looking for files that end in ".stat". If
you'd like to run an analysis job over a full month's work of
data, you'll use one or more "-lookin" options on the command line to
tell the tool where to find the data.
Next, you'll want to use either the "-fcst_valid_beg" and "-
fcst_valid_end" or the "-fcst_init_beg" and "-fcst_init_end" job
command options. These are used to define a time window to be used
when
filtering the STAT lines the tool is reading. As you'd guess, the
first set of options defines a filtering window for the valid times
and the second set filters on the forecast initialization times.
I'd guess you'd want to filter on the init times, but it's up to you.
The following type of STAT-Analysis job should do what you want:
stat_analysis -lookin stat_directory -job aggregate_stat -line_type
MPR -out_line_type CNT -column_str OBS_SID KDEN -fcst_init_beg
20101001 -fcst_init_end 20101031_235900
That'll look through all the STAT files found in "stat_directory",
retain only the MPR lines whose station ID value is "KDEN" and whose
initialization time falls between 20101001 and 20101031_235900,
and use those matched pairs to compute continuous statistics.
Hope that helps.
John
On 11/22/2010 06:45 AM, RAL HelpDesk {for James P Cipriani} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> Hi John,
>
> Thanks for the suggestions. As it turns out, using different levels
in
> point stat does indeed work.
>
> As for running aggregate_stat on the point_stat output, I was
wondering:
>
> Is it possible to run the aggregate_stat tool for one month and
obtain
> monthly averages (for each station) at the 24-hour or 48-hour lead
times
> (taking into account each individual hour's values)? I think I know
how to
> run the tool for the 24-hour or 48-hour lead times for each day of
the
> month; but running for the entire month all at once isn't exactly
clear.
>
> Thanks.
>
>
>
> From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
> To: James P Cipriani/Watson/IBM at IBMUS
> Date: 11/18/2010 12:30 PM
> Subject: Re: [rt.rap.ucar.edu #42228]
>
>
>
> James,
>
> I created a separate ticket for these questions.
>
> (1) If you're having issues running WRF-Post, I'd recommend writing
> WRF-Help for assistance. But you should still be able to run Point-
Stat to
> perform the verification you want. I suspect that
> something like the following would do the trick:
> fcst_field[] = [ "TMP/Z0" ];
> obs_field[] = [ "TMP/Z2" ];
> message_type = [ "APDSFC" ];
>
> Give that a shot and let me know if you get stuck.
>
> (2) Yes, the MPR output from Point-Stat is rather large. It creates
one
> line of output for each matched pair. For 30 days, initialized at
00Z and
> 12Z, with hourly output out to 84 hours, that'd be
> 30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR
lines,
> where n_fields is the number of variables/levels you're verifying
and
> n_locations is the number of stations you're using.
> Perhaps you could give it a shot for a few days and see if the files
are
> too large. After you generated MPR output for each verification
time,
> you'd run STAT-Analysis to aggregate those MPR values at
> each station of interest and compute statistics.
>
> If it would help, you could define a masking polyline region and
pass it to
> Point-Stat using "mask_poly" - then Point-Stat would only dump out
MPR
> information for those observations falling inside
> that polyline.
>
> Thanks,
> John
>
> On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>>
>> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
>> Transaction: Ticket created by johnhg
>> Queue: met_help
>> Subject: (No subject given)
>> Owner: johnhg
>> Requestors: jpcipria at us.ibm.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>
>>
>> MET-Help,
>>
>> Couple of quick questions for you:
>>
>> 1. Let's say I have 2-meter (above ground) observations but the
WPP has
>> only given me usable surface values (from the forecasts) --> don't
ask me
>> why but the post-processor is having trouble calculating the 2-
meter
> values
>> (I'm still investigating that one). My question is this:
>>
>> Will Point-Stat have trouble doing the stats because the
observations are
>> at 2-meters and the forecast values are at the model surface?
>>
>> 2. Also, we discussed using MPR output from Point-Stat as the
input to
> the
>> Stat-Analysis tool - but isn't the MPR output very large and
>> space-consuming? I've got about a month's worth of data at hourly
>> intervals out to 3.5 days for 00z and 12z cycles.
>>
>> Thanks,
>> James
>
>
------------------------------------------------
Subject: Point-Stat questions
From: James P Cipriani
Time: Mon Nov 22 10:18:32 2010
John,
As I was reviewing the configuration once more, that's what I was
thinking
as well. I appreciate your clarification in these matters. I had my
config file (and associated scripts) almost set-up exactly as what you
indicated.
I'll let you know how it goes.
Thanks.
From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
To: James P Cipriani/Watson/IBM at IBMUS
Date: 11/22/2010 11:03 AM
Subject: Re: [rt.rap.ucar.edu #42228]
James,
When you run the STAT-Analysis tool, you tell it where to look for
input
using the "-lookin" command line argument. You can use "-lookin" to
pass a
STAT file name directly or a directory name. If
you pass it a directory name, STAT-Analysis will search recursively
through
that directory looking for files that end in ".stat". If you'd like
to run
an analysis job over a full month's work of
data, you'll use one or more "-lookin" options on the command line to
tell
the tool where to find the data.
Next, you'll want to use either the "-fcst_valid_beg" and "-
fcst_valid_end"
or the "-fcst_init_beg" and "-fcst_init_end" job command options.
These
are used to define a time window to be used when
filtering the STAT lines the tool is reading. As you'd guess, the
first
set of options defines a filtering window for the valid times and the
second set filters on the forecast initialization times.
I'd guess you'd want to filter on the init times, but it's up to you.
The following type of STAT-Analysis job should do what you want:
stat_analysis -lookin stat_directory -job aggregate_stat -line_type
MPR
-out_line_type CNT -column_str OBS_SID KDEN -fcst_init_beg 20101001
-fcst_init_end 20101031_235900
That'll look through all the STAT files found in "stat_directory",
retain
only the MPR lines whose station ID value is "KDEN" and whose
initialization time falls between 20101001 and 20101031_235900,
and use those matched pairs to compute continuous statistics.
Hope that helps.
John
On 11/22/2010 06:45 AM, RAL HelpDesk {for James P Cipriani} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> Hi John,
>
> Thanks for the suggestions. As it turns out, using different levels
in
> point stat does indeed work.
>
> As for running aggregate_stat on the point_stat output, I was
wondering:
>
> Is it possible to run the aggregate_stat tool for one month and
obtain
> monthly averages (for each station) at the 24-hour or 48-hour lead
times
> (taking into account each individual hour's values)? I think I know
how
to
> run the tool for the 24-hour or 48-hour lead times for each day of
the
> month; but running for the entire month all at once isn't exactly
clear.
>
> Thanks.
>
>
>
> From: "RAL HelpDesk {for John Halley Gotway}"
<met_help at ucar.edu>
> To: James P Cipriani/Watson/IBM at IBMUS
> Date: 11/18/2010 12:30 PM
> Subject: Re: [rt.rap.ucar.edu #42228]
>
>
>
> James,
>
> I created a separate ticket for these questions.
>
> (1) If you're having issues running WRF-Post, I'd recommend writing
> WRF-Help for assistance. But you should still be able to run Point-
Stat
to
> perform the verification you want. I suspect that
> something like the following would do the trick:
> fcst_field[] = [ "TMP/Z0" ];
> obs_field[] = [ "TMP/Z2" ];
> message_type = [ "APDSFC" ];
>
> Give that a shot and let me know if you get stuck.
>
> (2) Yes, the MPR output from Point-Stat is rather large. It creates
one
> line of output for each matched pair. For 30 days, initialized at
00Z
and
> 12Z, with hourly output out to 84 hours, that'd be
> 30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR
lines,
> where n_fields is the number of variables/levels you're verifying
and
> n_locations is the number of stations you're using.
> Perhaps you could give it a shot for a few days and see if the files
are
> too large. After you generated MPR output for each verification
time,
> you'd run STAT-Analysis to aggregate those MPR values at
> each station of interest and compute statistics.
>
> If it would help, you could define a masking polyline region and
pass it
to
> Point-Stat using "mask_poly" - then Point-Stat would only dump out
MPR
> information for those observations falling inside
> that polyline.
>
> Thanks,
> John
>
> On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>>
>> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
>> Transaction: Ticket created by johnhg
>> Queue: met_help
>> Subject: (No subject given)
>> Owner: johnhg
>> Requestors: jpcipria at us.ibm.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>
>>
>> MET-Help,
>>
>> Couple of quick questions for you:
>>
>> 1. Let's say I have 2-meter (above ground) observations but the
WPP has
>> only given me usable surface values (from the forecasts) --> don't
ask
me
>> why but the post-processor is having trouble calculating the 2-
meter
> values
>> (I'm still investigating that one). My question is this:
>>
>> Will Point-Stat have trouble doing the stats because the
observations
are
>> at 2-meters and the forecast values are at the model surface?
>>
>> 2. Also, we discussed using MPR output from Point-Stat as the
input to
> the
>> Stat-Analysis tool - but isn't the MPR output very large and
>> space-consuming? I've got about a month's worth of data at hourly
>> intervals out to 3.5 days for 00z and 12z cycles.
>>
>> Thanks,
>> James
>
>
------------------------------------------------
Subject: Point-Stat questions
From: James P Cipriani
Time: Mon Nov 22 12:59:16 2010
John,
Is there a way to output all the statistics for all the individual
stations
to one file? (instead of specifying one station per output file)?
Thanks.
From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
To: James P Cipriani/Watson/IBM at IBMUS
Date: 11/22/2010 11:03 AM
Subject: Re: [rt.rap.ucar.edu #42228]
James,
When you run the STAT-Analysis tool, you tell it where to look for
input
using the "-lookin" command line argument. You can use "-lookin" to
pass a
STAT file name directly or a directory name. If
you pass it a directory name, STAT-Analysis will search recursively
through
that directory looking for files that end in ".stat". If you'd like
to run
an analysis job over a full month's work of
data, you'll use one or more "-lookin" options on the command line to
tell
the tool where to find the data.
Next, you'll want to use either the "-fcst_valid_beg" and "-
fcst_valid_end"
or the "-fcst_init_beg" and "-fcst_init_end" job command options.
These
are used to define a time window to be used when
filtering the STAT lines the tool is reading. As you'd guess, the
first
set of options defines a filtering window for the valid times and the
second set filters on the forecast initialization times.
I'd guess you'd want to filter on the init times, but it's up to you.
The following type of STAT-Analysis job should do what you want:
stat_analysis -lookin stat_directory -job aggregate_stat -line_type
MPR
-out_line_type CNT -column_str OBS_SID KDEN -fcst_init_beg 20101001
-fcst_init_end 20101031_235900
That'll look through all the STAT files found in "stat_directory",
retain
only the MPR lines whose station ID value is "KDEN" and whose
initialization time falls between 20101001 and 20101031_235900,
and use those matched pairs to compute continuous statistics.
Hope that helps.
John
On 11/22/2010 06:45 AM, RAL HelpDesk {for James P Cipriani} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> Hi John,
>
> Thanks for the suggestions. As it turns out, using different levels
in
> point stat does indeed work.
>
> As for running aggregate_stat on the point_stat output, I was
wondering:
>
> Is it possible to run the aggregate_stat tool for one month and
obtain
> monthly averages (for each station) at the 24-hour or 48-hour lead
times
> (taking into account each individual hour's values)? I think I know
how
to
> run the tool for the 24-hour or 48-hour lead times for each day of
the
> month; but running for the entire month all at once isn't exactly
clear.
>
> Thanks.
>
>
>
> From: "RAL HelpDesk {for John Halley Gotway}"
<met_help at ucar.edu>
> To: James P Cipriani/Watson/IBM at IBMUS
> Date: 11/18/2010 12:30 PM
> Subject: Re: [rt.rap.ucar.edu #42228]
>
>
>
> James,
>
> I created a separate ticket for these questions.
>
> (1) If you're having issues running WRF-Post, I'd recommend writing
> WRF-Help for assistance. But you should still be able to run Point-
Stat
to
> perform the verification you want. I suspect that
> something like the following would do the trick:
> fcst_field[] = [ "TMP/Z0" ];
> obs_field[] = [ "TMP/Z2" ];
> message_type = [ "APDSFC" ];
>
> Give that a shot and let me know if you get stuck.
>
> (2) Yes, the MPR output from Point-Stat is rather large. It creates
one
> line of output for each matched pair. For 30 days, initialized at
00Z
and
> 12Z, with hourly output out to 84 hours, that'd be
> 30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR
lines,
> where n_fields is the number of variables/levels you're verifying
and
> n_locations is the number of stations you're using.
> Perhaps you could give it a shot for a few days and see if the files
are
> too large. After you generated MPR output for each verification
time,
> you'd run STAT-Analysis to aggregate those MPR values at
> each station of interest and compute statistics.
>
> If it would help, you could define a masking polyline region and
pass it
to
> Point-Stat using "mask_poly" - then Point-Stat would only dump out
MPR
> information for those observations falling inside
> that polyline.
>
> Thanks,
> John
>
> On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>>
>> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
>> Transaction: Ticket created by johnhg
>> Queue: met_help
>> Subject: (No subject given)
>> Owner: johnhg
>> Requestors: jpcipria at us.ibm.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>
>>
>> MET-Help,
>>
>> Couple of quick questions for you:
>>
>> 1. Let's say I have 2-meter (above ground) observations but the
WPP has
>> only given me usable surface values (from the forecasts) --> don't
ask
me
>> why but the post-processor is having trouble calculating the 2-
meter
> values
>> (I'm still investigating that one). My question is this:
>>
>> Will Point-Stat have trouble doing the stats because the
observations
are
>> at 2-meters and the forecast values are at the model surface?
>>
>> 2. Also, we discussed using MPR output from Point-Stat as the
input to
> the
>> Stat-Analysis tool - but isn't the MPR output very large and
>> space-consuming? I've got about a month's worth of data at hourly
>> intervals out to 3.5 days for 00z and 12z cycles.
>>
>> Thanks,
>> James
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42228]
From: John Halley Gotway
Time: Mon Nov 22 13:36:11 2010
James,
When you run a single job on the command line, it's output will by
default be written to the screen - or you can use the "-out" command
line option to have it written to a file. It sounds like that's
what you're doing right now.
However, you can also run STAT-Analysis using a configuration file.
In the configuration file, you set up the jobs you want to be run
using the "jobs" parameter. It sounds like you want to basically
run the same job but just for a bunch of different station names.
You'd set up "jobs" to list out each of the analysis jobs you want
run. Then when you pass STAT-Analysis the config file and specify
"-out" on the command line, all of the job output will be written to
the same output file.
Please take a look at METv3.0/scripts/test_stat_analysis.sh for an
example of this.
Hopefully that'll do the trick for you.
John
On 11/22/2010 12:59 PM, RAL HelpDesk {for James P Cipriani} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> John,
> Is there a way to output all the statistics for all the individual
stations
> to one file? (instead of specifying one station per output file)?
> Thanks.
>
>
>
> From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
> To: James P Cipriani/Watson/IBM at IBMUS
> Date: 11/22/2010 11:03 AM
> Subject: Re: [rt.rap.ucar.edu #42228]
>
>
>
> James,
>
> When you run the STAT-Analysis tool, you tell it where to look for
input
> using the "-lookin" command line argument. You can use "-lookin" to
pass a
> STAT file name directly or a directory name. If
> you pass it a directory name, STAT-Analysis will search recursively
through
> that directory looking for files that end in ".stat". If you'd like
to run
> an analysis job over a full month's work of
> data, you'll use one or more "-lookin" options on the command line
to tell
> the tool where to find the data.
>
> Next, you'll want to use either the "-fcst_valid_beg" and "-
fcst_valid_end"
> or the "-fcst_init_beg" and "-fcst_init_end" job command options.
These
> are used to define a time window to be used when
> filtering the STAT lines the tool is reading. As you'd guess, the
first
> set of options defines a filtering window for the valid times and
the
> second set filters on the forecast initialization times.
> I'd guess you'd want to filter on the init times, but it's up to
you.
>
> The following type of STAT-Analysis job should do what you want:
>
> stat_analysis -lookin stat_directory -job aggregate_stat -line_type
MPR
> -out_line_type CNT -column_str OBS_SID KDEN -fcst_init_beg 20101001
> -fcst_init_end 20101031_235900
>
> That'll look through all the STAT files found in "stat_directory",
retain
> only the MPR lines whose station ID value is "KDEN" and whose
> initialization time falls between 20101001 and 20101031_235900,
> and use those matched pairs to compute continuous statistics.
>
> Hope that helps.
>
> John
>
> On 11/22/2010 06:45 AM, RAL HelpDesk {for James P Cipriani} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>
>>
>> Hi John,
>>
>> Thanks for the suggestions. As it turns out, using different
levels in
>> point stat does indeed work.
>>
>> As for running aggregate_stat on the point_stat output, I was
wondering:
>>
>> Is it possible to run the aggregate_stat tool for one month and
obtain
>> monthly averages (for each station) at the 24-hour or 48-hour lead
times
>> (taking into account each individual hour's values)? I think I
know how
> to
>> run the tool for the 24-hour or 48-hour lead times for each day of
the
>> month; but running for the entire month all at once isn't exactly
clear.
>>
>> Thanks.
>>
>>
>>
>> From: "RAL HelpDesk {for John Halley Gotway}"
> <met_help at ucar.edu>
>> To: James P Cipriani/Watson/IBM at IBMUS
>> Date: 11/18/2010 12:30 PM
>> Subject: Re: [rt.rap.ucar.edu #42228]
>>
>>
>>
>> James,
>>
>> I created a separate ticket for these questions.
>>
>> (1) If you're having issues running WRF-Post, I'd recommend writing
>> WRF-Help for assistance. But you should still be able to run
Point-Stat
> to
>> perform the verification you want. I suspect that
>> something like the following would do the trick:
>> fcst_field[] = [ "TMP/Z0" ];
>> obs_field[] = [ "TMP/Z2" ];
>> message_type = [ "APDSFC" ];
>>
>> Give that a shot and let me know if you get stuck.
>>
>> (2) Yes, the MPR output from Point-Stat is rather large. It
creates one
>> line of output for each matched pair. For 30 days, initialized at
00Z
> and
>> 12Z, with hourly output out to 84 hours, that'd be
>> 30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR
lines,
>> where n_fields is the number of variables/levels you're verifying
and
>> n_locations is the number of stations you're using.
>> Perhaps you could give it a shot for a few days and see if the
files are
>> too large. After you generated MPR output for each verification
time,
>> you'd run STAT-Analysis to aggregate those MPR values at
>> each station of interest and compute statistics.
>>
>> If it would help, you could define a masking polyline region and
pass it
> to
>> Point-Stat using "mask_poly" - then Point-Stat would only dump out
MPR
>> information for those observations falling inside
>> that polyline.
>>
>> Thanks,
>> John
>>
>> On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway}
wrote:
>>>
>>> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
>>> Transaction: Ticket created by johnhg
>>> Queue: met_help
>>> Subject: (No subject given)
>>> Owner: johnhg
>>> Requestors: jpcipria at us.ibm.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>>
>>>
>>> MET-Help,
>>>
>>> Couple of quick questions for you:
>>>
>>> 1. Let's say I have 2-meter (above ground) observations but the
WPP has
>>> only given me usable surface values (from the forecasts) --> don't
ask
> me
>>> why but the post-processor is having trouble calculating the 2-
meter
>> values
>>> (I'm still investigating that one). My question is this:
>>>
>>> Will Point-Stat have trouble doing the stats because the
observations
> are
>>> at 2-meters and the forecast values are at the model surface?
>>>
>>> 2. Also, we discussed using MPR output from Point-Stat as the
input to
>> the
>>> Stat-Analysis tool - but isn't the MPR output very large and
>>> space-consuming? I've got about a month's worth of data at hourly
>>> intervals out to 3.5 days for 00z and 12z cycles.
>>>
>>> Thanks,
>>> James
>>
>>
>
>
------------------------------------------------
Subject: Point-Stat questions
From: James P Cipriani
Time: Mon Nov 22 14:12:32 2010
Thanks, John. I was thinking the same thing.
I do apologize for what must seem like easy questions to you.
One more question for you. Let us back-track just a bit...
We spoke before about the stat_analysis command that would do the
computation of continuous stats based on matched pairs. Given the
following command:
-"job aggregate_stat -line_type MPR -out_line_type CNT -fcst_lead
240000
-fcst_var TMP -column_str OBS_SID station_name -fcst_init_beg 20101001
-fcst_init_end 20101031_235900"
Would stat_analysis use all times up to 24 hours lead time (from the
initialization time) for all 31 days in the month and then compute
monthly
stats?
Similarly, would the following command be for all times up to 48 hours
lead
time (using all point-stat output prior to 48 hour lead time)?:
-"job aggregate_stat -line_type MPR -out_line_type CNT -fcst_lead
480000
-fcst_var TMP -column_str OBS_SID station_name -fcst_init_beg 20101001
-fcst_init_end 20101031_235900"
Basically, I want the monthly average continuous stats per station for
say
days 1 and 2. I believe the command you gave before did this for
everything up to the final lead time.
Thanks you.
From: "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
To: James P Cipriani/Watson/IBM at IBMUS
Date: 11/22/2010 03:37 PM
Subject: Re: [rt.rap.ucar.edu #42228]
James,
When you run a single job on the command line, it's output will by
default
be written to the screen - or you can use the "-out" command line
option to
have it written to a file. It sounds like that's
what you're doing right now.
However, you can also run STAT-Analysis using a configuration file.
In the
configuration file, you set up the jobs you want to be run using the
"jobs"
parameter. It sounds like you want to basically
run the same job but just for a bunch of different station names.
You'd
set up "jobs" to list out each of the analysis jobs you want run.
Then
when you pass STAT-Analysis the config file and specify
"-out" on the command line, all of the job output will be written to
the
same output file.
Please take a look at METv3.0/scripts/test_stat_analysis.sh for an
example
of this.
Hopefully that'll do the trick for you.
John
On 11/22/2010 12:59 PM, RAL HelpDesk {for James P Cipriani} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>
>
> John,
> Is there a way to output all the statistics for all the individual
stations
> to one file? (instead of specifying one station per output file)?
> Thanks.
>
>
>
> From: "RAL HelpDesk {for John Halley Gotway}"
<met_help at ucar.edu>
> To: James P Cipriani/Watson/IBM at IBMUS
> Date: 11/22/2010 11:03 AM
> Subject: Re: [rt.rap.ucar.edu #42228]
>
>
>
> James,
>
> When you run the STAT-Analysis tool, you tell it where to look for
input
> using the "-lookin" command line argument. You can use "-lookin" to
pass
a
> STAT file name directly or a directory name. If
> you pass it a directory name, STAT-Analysis will search recursively
through
> that directory looking for files that end in ".stat". If you'd like
to
run
> an analysis job over a full month's work of
> data, you'll use one or more "-lookin" options on the command line
to
tell
> the tool where to find the data.
>
> Next, you'll want to use either the "-fcst_valid_beg" and
"-fcst_valid_end"
> or the "-fcst_init_beg" and "-fcst_init_end" job command options.
These
> are used to define a time window to be used when
> filtering the STAT lines the tool is reading. As you'd guess, the
first
> set of options defines a filtering window for the valid times and
the
> second set filters on the forecast initialization times.
> I'd guess you'd want to filter on the init times, but it's up to
you.
>
> The following type of STAT-Analysis job should do what you want:
>
> stat_analysis -lookin stat_directory -job aggregate_stat -line_type
MPR
> -out_line_type CNT -column_str OBS_SID KDEN -fcst_init_beg 20101001
> -fcst_init_end 20101031_235900
>
> That'll look through all the STAT files found in "stat_directory",
retain
> only the MPR lines whose station ID value is "KDEN" and whose
> initialization time falls between 20101001 and 20101031_235900,
> and use those matched pairs to compute continuous statistics.
>
> Hope that helps.
>
> John
>
> On 11/22/2010 06:45 AM, RAL HelpDesk {for James P Cipriani} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>
>>
>> Hi John,
>>
>> Thanks for the suggestions. As it turns out, using different
levels in
>> point stat does indeed work.
>>
>> As for running aggregate_stat on the point_stat output, I was
wondering:
>>
>> Is it possible to run the aggregate_stat tool for one month and
obtain
>> monthly averages (for each station) at the 24-hour or 48-hour lead
times
>> (taking into account each individual hour's values)? I think I
know how
> to
>> run the tool for the 24-hour or 48-hour lead times for each day of
the
>> month; but running for the entire month all at once isn't exactly
clear.
>>
>> Thanks.
>>
>>
>>
>> From: "RAL HelpDesk {for John Halley Gotway}"
> <met_help at ucar.edu>
>> To: James P Cipriani/Watson/IBM at IBMUS
>> Date: 11/18/2010 12:30 PM
>> Subject: Re: [rt.rap.ucar.edu #42228]
>>
>>
>>
>> James,
>>
>> I created a separate ticket for these questions.
>>
>> (1) If you're having issues running WRF-Post, I'd recommend writing
>> WRF-Help for assistance. But you should still be able to run
Point-Stat
> to
>> perform the verification you want. I suspect that
>> something like the following would do the trick:
>> fcst_field[] = [ "TMP/Z0" ];
>> obs_field[] = [ "TMP/Z2" ];
>> message_type = [ "APDSFC" ];
>>
>> Give that a shot and let me know if you get stuck.
>>
>> (2) Yes, the MPR output from Point-Stat is rather large. It
creates one
>> line of output for each matched pair. For 30 days, initialized at
00Z
> and
>> 12Z, with hourly output out to 84 hours, that'd be
>> 30*2*84*n_fields*n_locations (or 5040*n_fields*n_locations) MPR
lines,
>> where n_fields is the number of variables/levels you're verifying
and
>> n_locations is the number of stations you're using.
>> Perhaps you could give it a shot for a few days and see if the
files are
>> too large. After you generated MPR output for each verification
time,
>> you'd run STAT-Analysis to aggregate those MPR values at
>> each station of interest and compute statistics.
>>
>> If it would help, you could define a masking polyline region and
pass it
> to
>> Point-Stat using "mask_poly" - then Point-Stat would only dump out
MPR
>> information for those observations falling inside
>> that polyline.
>>
>> Thanks,
>> John
>>
>> On 11/18/2010 09:54 AM, RAL HelpDesk {for John Halley Gotway}
wrote:
>>>
>>> Thu Nov 18 09:54:54 2010: Request 42228 was acted upon.
>>> Transaction: Ticket created by johnhg
>>> Queue: met_help
>>> Subject: (No subject given)
>>> Owner: johnhg
>>> Requestors: jpcipria at us.ibm.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42228 >
>>>
>>>
>>> MET-Help,
>>>
>>> Couple of quick questions for you:
>>>
>>> 1. Let's say I have 2-meter (above ground) observations but the
WPP
has
>>> only given me usable surface values (from the forecasts) --> don't
ask
> me
>>> why but the post-processor is having trouble calculating the 2-
meter
>> values
>>> (I'm still investigating that one). My question is this:
>>>
>>> Will Point-Stat have trouble doing the stats because the
observations
> are
>>> at 2-meters and the forecast values are at the model surface?
>>>
>>> 2. Also, we discussed using MPR output from Point-Stat as the
input to
>> the
>>> Stat-Analysis tool - but isn't the MPR output very large and
>>> space-consuming? I've got about a month's worth of data at hourly
>>> intervals out to 3.5 days for 00z and 12z cycles.
>>>
>>> Thanks,
>>> James
>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42228]
From: John Halley Gotway
Time: Mon Nov 22 14:19:25 2010
James,
I'm not entirely clear what you mean by "up to 24 hours lead time".
When you say "-fcst_lead 240000", STAT-Analysis will only use lines
that have a lead time of EXACTLY 24 hours. Alternatively, if
you say "-fcst_lead 120000 -fcst_lead 240000", it will use all lines
with a lead time of 12 *OR* 24 hours. So if you want to group
multiple lead times together, just list them out.
The options that have "_beg" and "_end" in them are used to define
matching time windows. The forecast lead time is not one of those
options.
I'd suggest just giving these jobs a try and see how it goes. I'd
suggest first running a sample job on the command line. And when you
do, use the "-dump_row" option which writes that STAT lines
that were actually used by the job to the output file name you
specify. For example, if you run a job using "-fcst_lead 240000" and
then run it again using "-fcst_lead 120000 -fcst_lead 240000", you
should be able to see the difference in which lines were used by the
job by taking a look at the dump_row output.
Thanks,
John
On 11/22/2010 02:12 PM, RAL HelpDesk {for James P Cipriani} wrote:
> We spoke before about the stat_analysis command that would do the
------------------------------------------------
More information about the Met_help
mailing list