[Met_help] [rt.rap.ucar.edu #91893] History for appending to a stat file

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I’m running grid_stat by variable for a specific valid time.  When I cycle over variables I’d like to append the result to the same output file, but it seems that it will only overwrite.  Is there a way to indicate append?

Thanks,
Liz

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: appending to a stat file
From: John Halley Gotway
Time: Wed Sep 04 12:58:45 2019

Hi Liz,

I moved your question over to met-help.

The short answer is no, there is no way for Grid-Stat to append to an
existing file.  Grid-Stat (and all the other MET tools) will "clobber"
any existing files.  That being said, here's some other relevant info.

First, I read that you're running Grid-Stat once for each variable
you're processing.  Alternatively, you could run Grid-Stat once and
process all of the variables in a single configuration file.  That
would result in all the stats being written to the same output file.
Instead of listing a single entry in the field array, you could list
many.

Second... perhaps you do still have a good reason for calling Grid-
Stat many times.  Maybe the data is spread across many input files, it
makes the scripting easier, or the run are more efficient?  You can
set the "output_prefix" entry in the config file to avoid the files
from being clobbered.  Whatever string you choose gets inserted in the
output files named "grid_stat_OUTPUT_PREFIX_...".  So you could
include the current variable name there to make the output files
unique.

Third, let's say you've used the output_prefix option to make many
output files for a single time, but you'd still like to 'cat' them
together into a single file.  You can run a Stat-Analysis 'filter' job
to do so.  Something like this:

stat_analysis -job filter -lookin today/grid_stat -out_stat
all_variables.stat

The idea of appending to an existing file seems simple enough, but it
opens up a can of worms.  What if multiple instances of grid_stat
attempt to write to the same file at the same time?  That can cause
file access issues and race conditions.  If everything is sequential,
it might work well, but there's no guarantee that it will be
sequential.

Hope that helps clarify.  Of course, anything is possible.  If there's
a very good reason for doing this at NRL, we could talk more about it.

Thanks,
John

------------------------------------------------
Subject: appending to a stat file
From: Satterfield, Dr. Elizabeth
Time: Wed Sep 04 13:05:18 2019

Thanks!  This helps -- ok I'll use the output prefix option.

I'm running all lead times and masking region per grid_stat call, but
since
each is a separate file I'm trying to make scripting a bit easier.

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, September 4, 2019 11:59 AM
To: Satterfield, Dr. Elizabeth
Subject: [rt.rap.ucar.edu #91893] appending to a stat file

Hi Liz,

I moved your question over to met-help.

The short answer is no, there is no way for Grid-Stat to append to an
existing
file.  Grid-Stat (and all the other MET tools) will "clobber" any
existing
files.  That being said, here's some other relevant info.

First, I read that you're running Grid-Stat once for each variable
you're
processing.  Alternatively, you could run Grid-Stat once and process
all of
the variables in a single configuration file.  That would result in
all the
stats being written to the same output file.  Instead of listing a
single
entry in the field array, you could list many.

Second... perhaps you do still have a good reason for calling Grid-
Stat many
times.  Maybe the data is spread across many input files, it makes the
scripting easier, or the run are more efficient?  You can set the
"output_prefix" entry in the config file to avoid the files from being
clobbered.  Whatever string you choose gets inserted in the output
files named
"grid_stat_OUTPUT_PREFIX_...".  So you could include the current
variable name
there to make the output files unique.

Third, let's say you've used the output_prefix option to make many
output
files for a single time, but you'd still like to 'cat' them together
into a
single file.  You can run a Stat-Analysis 'filter' job to do so.
Something
like this:

stat_analysis -job filter -lookin today/grid_stat -out_stat
all_variables.stat

The idea of appending to an existing file seems simple enough, but it
opens up
a can of worms.  What if multiple instances of grid_stat attempt to
write to
the same file at the same time?  That can cause file access issues and
race
conditions.  If everything is sequential, it might work well, but
there's no
guarantee that it will be sequential.

Hope that helps clarify.  Of course, anything is possible.  If there's
a very
good reason for doing this at NRL, we could talk more about it.

Thanks,
John


------------------------------------------------