[Met_help] [rt.rap.ucar.edu #81373] History for series_analysis: process more than one variable in config file?
John Halley Gotway via RT
met_help at ucar.edu
Fri Jul 28 16:37:47 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I am currently trying to use series_analysis to compare the averages and
difference of different variable fields between different models (ME, FBAR,
OBAR). Can the series_analysis handle more than one variable when they are
listed in forecast field in the config file? It seem to be running into an
issue trying to do this.
Listing just one variable works just fine, but if I try...
*fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; },
{ name = "UGRD" ; level = [ "Z10" ]; } ]; };*
I still only get series_cnt_ME, series_cnt_FBAR, series_cnt_OBAR and
doesn't seem to be processing all the files I have listed in my
fcst_file_list and obs_file_list (20 files).
*DEBUG 1: Length of configuration "fcst.field" = 2DEBUG 1: Length of
configuration "obs.field" = 2DEBUG 1: Length of forecast file list
= 20DEBUG 1: Length of observation file list = 20DEBUG 1: Series
defined by the "fcst.field" configuration entry of length 2.DEBUG 2:
Computing statistics using a block size of 70000, requiring 1 pass(es)
through the 144 x 73 grid.DEBUG 2: Processing data pass number 1 of 1 for
grid points 1 to 10512.DEBUG 2: Processing series entry 1 of 2: TMP/Z2
versus TMP/Z2DEBUG 2: Found data for TMP/Z2 in file:
/scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf06.ecm.2017010100DEBUG 1:
Regridding field TMP/Z2 to the verification grid.DEBUG 2: Found data for
TMP/Z2 in file:
/scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf06.gfs.2017010100DEBUG 1:
Regridding field TMP/Z2 to the verification grid.DEBUG 1: Reading stat
column descriptions:
/scratch4/BMC/dtc/MET/met-6.0/share/met/table_files/stat_column_description.txtDEBUG
2: Processing series entry 2 of 2: UGRD/Z10 versus UGRD/Z10DEBUG 2: Found
data for UGRD/Z10 in file:
/scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf12.ecm.2017010100DEBUG 1:
Regridding field UGRD/Z10 to the verification grid.DEBUG 2: Found data for
UGRD/Z10 in file:
/scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf12.gfs.2017010100DEBUG 1:
Regridding field UGRD/Z10 to the verification grid.DEBUG 1: Output file:
/scratch4/NCEPDEV/global/save/Mallory.Row/VRFY/met/out_2varsnew/plots2d/model2model/model/ecm/00Z/d01/out.nc
<http://out.nc>*
I tried...
*fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ]; field =
[ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
and...
*fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ]; };fcst =
{ field = [ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
but it only processes the second variable, UGRD. Is it possible for series
analysis to process both variables separately, to get series_cnt_ME,
series_cnt_FBAR, series_cnt_OBAR for each variable in one netcdf output
file? Am I missing the correct formatting method? Thanks!
Mallory Row
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: series_analysis: process more than one variable in config file?
From: John Halley Gotway
Time: Thu Jul 27 13:15:11 2017
Mallory,
I see that you have a question about configuring Series-Analysis.
Thia
tool is designed to process a single series of data each time it is
run.
But you're right to notice that "fcst.field" and "obs.field" can be
set as
an array of multiple fields. Series-Analysis supports multiple ways
of
defining a series...
The vast majority of the time the user specifies 1 forecast field, 1
observation field, and a time series of files from which the same
fields
should be extracted. For example, compute stats for 2-meter
temperature
for all 12-hour forecast lead times over 1 month of model
initializations.
Or, compute stats for 2-m temperature for all lead times for a single
model
initialization. Defining time series like this is the most common
logic.
However, you could instead define multiple forecast and observation
fields
and then pass in a single forecast and observation file. In this
case, the
series is defined by the multiple fields extracted from the same file.
For
example, aggregate statistics for temperature over all pressure levels
from
the same file. So this is a series of vertical levels rather than a
time
series.
When you run series_analysis, it counts up the number of input fields
and
files. Listed below is a comment I cut-and-pasted from the code
(series_analysis.cc):
// Determine the length of the series to be analyzed. Series is
// defined by the first parameter of length greater than one:
// - Configuration fcst.field
// - Configuration obs.field
// - Forecast file list
// - Observation file list
And then it prints a log message something like the following to tell
you
what logic is being invoked:
DEBUG(1): Series defined by the "fcst.field" configuration entry of
length 30.
Ultimately, I see that you want to process multiple time series. This
should be pretty easy to do by calling Series-Analysis multiple times
from
a script. Each time you run, the only things that'll change will be
the
"name" and "level" settings. So just define those using environment
variables:
{ name = "${FCST_VAR_NAME}" ; level = [ "${FCST_VAR_LEVEL}" ]; }
Then have your calling script set these variables. Using environment
variables will avoid having to create many Series-Analysis config
files.
Hope that helps.
Thanks,
John
On Thu, Jul 27, 2017 at 12:13 PM, Mallory Row - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Jul 27 12:13:03 2017: Request 81373 was acted upon.
> Transaction: Ticket created by mallory.row at noaa.gov
> Queue: met_help
> Subject: series_analysis: process more than one variable in
config
> file?
> Owner: Nobody
> Requestors: mallory.row at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81373 >
>
>
> Hi,
>
> I am currently trying to use series_analysis to compare the averages
and
> difference of different variable fields between different models
(ME, FBAR,
> OBAR). Can the series_analysis handle more than one variable when
they are
> listed in forecast field in the config file? It seem to be running
into an
> issue trying to do this.
>
> Listing just one variable works just fine, but if I try...
>
>
>
>
>
> *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; },
> { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
> I still only get series_cnt_ME, series_cnt_FBAR, series_cnt_OBAR and
> doesn't seem to be processing all the files I have listed in my
> fcst_file_list and obs_file_list (20 files).
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 1: Length of configuration "fcst.field" = 2DEBUG 1: Length of
> configuration "obs.field" = 2DEBUG 1: Length of forecast file list
> = 20DEBUG 1: Length of observation file list = 20DEBUG 1:
Series
> defined by the "fcst.field" configuration entry of length 2.DEBUG 2:
> Computing statistics using a block size of 70000, requiring 1
pass(es)
> through the 144 x 73 grid.DEBUG 2: Processing data pass number 1 of
1 for
> grid points 1 to 10512.DEBUG 2: Processing series entry 1 of 2:
TMP/Z2
> versus TMP/Z2DEBUG 2: Found data for TMP/Z2 in file:
> /scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf06.ecm.2017010100DEBUG
1:
> Regridding field TMP/Z2 to the verification grid.DEBUG 2: Found data
for
> TMP/Z2 in file:
> /scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf06.gfs.2017010100DEBUG
1:
> Regridding field TMP/Z2 to the verification grid.DEBUG 1: Reading
stat
> column descriptions:
> /scratch4/BMC/dtc/MET/met-6.0/share/met/table_files/stat_
> column_description.txtDEBUG
> 2: Processing series entry 2 of 2: UGRD/Z10 versus UGRD/Z10DEBUG 2:
Found
> data for UGRD/Z10 in file:
> /scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf12.ecm.2017010100DEBUG
1:
> Regridding field UGRD/Z10 to the verification grid.DEBUG 2: Found
data for
> UGRD/Z10 in file:
> /scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf12.gfs.2017010100DEBUG
1:
> Regridding field UGRD/Z10 to the verification grid.DEBUG 1: Output
file:
> /scratch4/NCEPDEV/global/save/Mallory.Row/VRFY/met/out_
> 2varsnew/plots2d/model2model/model/ecm/00Z/d01/out.nc
> <http://out.nc>*
> I tried...
>
>
>
>
> *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ];
field =
> [ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
>
> and...
>
>
>
>
>
>
> *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ];
};fcst =
> { field = [ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
>
> but it only processes the second variable, UGRD. Is it possible for
series
> analysis to process both variables separately, to get series_cnt_ME,
> series_cnt_FBAR, series_cnt_OBAR for each variable in one netcdf
output
> file? Am I missing the correct formatting method? Thanks!
>
> Mallory Row
>
>
------------------------------------------------
Subject: series_analysis: process more than one variable in config file?
From: Mallory Row - NOAA Affiliate
Time: Thu Jul 27 13:30:11 2017
Sounds great, thanks John!
Mallory
On Thu, Jul 27, 2017 at 3:15 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mallory,
>
> I see that you have a question about configuring Series-Analysis.
Thia
> tool is designed to process a single series of data each time it is
run.
> But you're right to notice that "fcst.field" and "obs.field" can be
set as
> an array of multiple fields. Series-Analysis supports multiple ways
of
> defining a series...
>
> The vast majority of the time the user specifies 1 forecast field, 1
> observation field, and a time series of files from which the same
fields
> should be extracted. For example, compute stats for 2-meter
temperature
> for all 12-hour forecast lead times over 1 month of model
initializations.
> Or, compute stats for 2-m temperature for all lead times for a
single model
> initialization. Defining time series like this is the most common
logic.
>
> However, you could instead define multiple forecast and observation
fields
> and then pass in a single forecast and observation file. In this
case, the
> series is defined by the multiple fields extracted from the same
file. For
> example, aggregate statistics for temperature over all pressure
levels from
> the same file. So this is a series of vertical levels rather than a
time
> series.
>
> When you run series_analysis, it counts up the number of input
fields and
> files. Listed below is a comment I cut-and-pasted from the code
> (series_analysis.cc):
>
> // Determine the length of the series to be analyzed. Series is
> // defined by the first parameter of length greater than one:
> // - Configuration fcst.field
> // - Configuration obs.field
> // - Forecast file list
> // - Observation file list
>
> And then it prints a log message something like the following to
tell you
> what logic is being invoked:
> DEBUG(1): Series defined by the "fcst.field" configuration entry
of
> length 30.
>
> Ultimately, I see that you want to process multiple time series.
This
> should be pretty easy to do by calling Series-Analysis multiple
times from
> a script. Each time you run, the only things that'll change will be
the
> "name" and "level" settings. So just define those using environment
> variables:
> { name = "${FCST_VAR_NAME}" ; level = [ "${FCST_VAR_LEVEL}" ]; }
>
> Then have your calling script set these variables. Using
environment
> variables will avoid having to create many Series-Analysis config
files.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Thu, Jul 27, 2017 at 12:13 PM, Mallory Row - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jul 27 12:13:03 2017: Request 81373 was acted upon.
> > Transaction: Ticket created by mallory.row at noaa.gov
> > Queue: met_help
> > Subject: series_analysis: process more than one variable in
config
> > file?
> > Owner: Nobody
> > Requestors: mallory.row at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81373 >
> >
> >
> > Hi,
> >
> > I am currently trying to use series_analysis to compare the
averages and
> > difference of different variable fields between different models
(ME,
> FBAR,
> > OBAR). Can the series_analysis handle more than one variable when
they
> are
> > listed in forecast field in the config file? It seem to be running
into
> an
> > issue trying to do this.
> >
> > Listing just one variable works just fine, but if I try...
> >
> >
> >
> >
> >
> > *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; },
> > { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
> > I still only get series_cnt_ME, series_cnt_FBAR, series_cnt_OBAR
and
> > doesn't seem to be processing all the files I have listed in my
> > fcst_file_list and obs_file_list (20 files).
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 1: Length of configuration "fcst.field" = 2DEBUG 1: Length
of
> > configuration "obs.field" = 2DEBUG 1: Length of forecast file
list
> > = 20DEBUG 1: Length of observation file list = 20DEBUG 1:
Series
> > defined by the "fcst.field" configuration entry of length 2.DEBUG
2:
> > Computing statistics using a block size of 70000, requiring 1
pass(es)
> > through the 144 x 73 grid.DEBUG 2: Processing data pass number 1
of 1 for
> > grid points 1 to 10512.DEBUG 2: Processing series entry 1 of 2:
TMP/Z2
> > versus TMP/Z2DEBUG 2: Found data for TMP/Z2 in file:
> >
/scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf06.ecm.2017010100DEBUG
1:
> > Regridding field TMP/Z2 to the verification grid.DEBUG 2: Found
data for
> > TMP/Z2 in file:
> >
/scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf06.gfs.2017010100DEBUG
1:
> > Regridding field TMP/Z2 to the verification grid.DEBUG 1: Reading
stat
> > column descriptions:
> > /scratch4/BMC/dtc/MET/met-6.0/share/met/table_files/stat_
> > column_description.txtDEBUG
> > 2: Processing series entry 2 of 2: UGRD/Z10 versus UGRD/Z10DEBUG
2: Found
> > data for UGRD/Z10 in file:
> >
/scratch4/NCEPDEV/global/noscrub/stat/ecm/pgbf12.ecm.2017010100DEBUG
1:
> > Regridding field UGRD/Z10 to the verification grid.DEBUG 2: Found
data
> for
> > UGRD/Z10 in file:
> >
/scratch4/NCEPDEV/global/noscrub/stat/gfs/pgbf12.gfs.2017010100DEBUG
1:
> > Regridding field UGRD/Z10 to the verification grid.DEBUG 1: Output
file:
> > /scratch4/NCEPDEV/global/save/Mallory.Row/VRFY/met/out_
> > 2varsnew/plots2d/model2model/model/ecm/00Z/d01/out.nc
> > <http://out.nc>*
> > I tried...
> >
> >
> >
> >
> > *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ];
field
> =
> > [ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
> >
> > and...
> >
> >
> >
> >
> >
> >
> > *fcst = { field = [ { name = "TMP" ; level = [ "Z2" ]; } ];
> };fcst =
> > { field = [ { name = "UGRD" ; level = [ "Z10" ]; } ]; };*
> >
> > but it only processes the second variable, UGRD. Is it possible
for
> series
> > analysis to process both variables separately, to get
series_cnt_ME,
> > series_cnt_FBAR, series_cnt_OBAR for each variable in one netcdf
output
> > file? Am I missing the correct formatting method? Thanks!
> >
> > Mallory Row
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list