[mpas-developers] config_write_initial_output option

[Michael Duda]

Hi, Mark.

Although the branch does avoid writing the initial time for a cold-start run
with all frames in the same output file, it appears that the output filename has
the timestamp of the first frame in the file added to it; whereas with
config_write_initial_output=.true. we might get an output file "output.nc", with
the option set to false, the output file is named, e.g.,
"output.0000-01-01_06.00.00.nc". I suspect this is done to avoid filename
conflicts when restarting a run, which is where I think what I had in mind
differs from the current behavior.

Consider the example where we write history every 6h and write 4 frames per
outfile (in other words, separate output files for each day) with files

   output.0000-01-01_00.00.00.nc
   output.0000-01-02_00.00.00.nc
   output.0000-01-03_00.00.00.nc
   ...

that contain times

  "0000-01-01_00:00:00                                             ",
  "0000-01-01_06:00:00                                             ",
  "0000-01-01_12:00:00                                             ",
  "0000-01-01_18:00:00                                             " ;

  "0000-01-02_00:00:00                                             ",
  "0000-01-02_06:00:00                                             ",
  "0000-01-02_12:00:00                                             ",
  "0000-01-02_18:00:00                                             " ;

  "0000-01-03_00:00:00                                             ",
  "0000-01-03_06:00:00                                             ",
  "0000-01-03_12:00:00                                             ",
  "0000-01-03_18:00:00                                             " ;

  ...

respectively. It may typically happen that we run for an integral number of
days, in which we end up with an output file at the final time that has just a
single output time, e.g.,

   output.0000-01-05_00.00.00.nc

with the output frame at 0000-01-05_00:00:00 only. Ideally (in my opinion), we'd
like to be able to restart the model from 0000-01-05_00:00:00 and avoid writing
the intial time, but still write the times 0000-01-05_06:00:00, 0000-01-05_12:00:00, 
and 0000-01-05_18:00:00 to the file output.0000-01-05_00.00.00.nc. Basically,
the restarted run would have output files structured in an identical way to a
single run for the complete simulation period.

To support this behavior, I've modified the IO layers to support appending to
existing files, and to seek through files to find the frame number that we need
to begin writing at. If it sounds alright to you, I can commit the changes that
I have to support appending to existing output files to the branch. The only
remaining work then would be in the code where we open and close files (the same
parts that you've modified); here, I think we'd need logic to determine whether
we want to append to an output file or to overwrite it (considering whether it
is a restart run, whether the file already exists, and the number of frames
in each file). I'd volunteer to make the additional changes in the branch, if
you'd like.

Michael


On Thu, May 10, 2012 at 03:51:34PM -0600, Mark Petersen wrote:
> Michael,
> 
> Thanks for your thoughts.
> 
> I tested the ocean core using config_write_initial_output=.false. with 
> both single and multiple frames per output file, and both worked as 
> expected.  In both cases,
>    mpas_output_state_init
> is called the first time within a
>   if(output_frame == 1) then
> branch in mpas_*_mpas_core.F.  Is calling the initialization routine 
> your concern for multiple frames?
> 
> Your more general implementation of time frames sounds good.  It is 
> mostly a question of timing.  If you are thinking of making these 
> changes in the next week or two, I'm happy to wait and you can use my 
> branch if you like.  If it is a longer term improvement, I prefer to 
> merge this config_write_initial_output flag to the trunk so I don't need 
> to maintain a branch from which to run our large simulations.
> 
> Thanks,
> Mark
> 
> 
> On 05/10/12 14:24, Michael Duda wrote:
> >Mark,
> >
> >I think this would be a useful option, though as currently implemented
> >in the branch, I'm not sure that it would work as expected when there
> >are multiple history frames written to the same file. I've also been
> >looking at implementing this, and I have some changes to the IO layer to
> >support appending time frames to an existing file that I think might be
> >useful for a completely general implementation of the
> >write_initial_output functionality. I can send these changes to you to
> >take a look at, or I can merge them into the branch; just let me know.
> >
> >Michael
> >
> >
> >On Thu, May 10, 2012 at 01:06:15PM -0600, Mark Petersen wrote:
> >>MPAS Developers,
> >>
> >>I added the flag config_write_initial_output to all cores in the branch
> >>ocean_projects/first_output_optional.  It simply controls if an output
> >>file is written upon start-up.  The ocean core needs this for our large
> >>runs, to save i/o and for simply time-averaged variables.  Because the
> >>first file initialized in driver/mpas_subdriver.F, it affects all
> >>cores.  In addition, it seems like a useful flag for everyone.
> >>
> >>I tested by compiling all four cores, and running with the ocean core.
> >>Could someone run a little test with the atmospheric cores, switching
> >>this flag on and off, and make sure it works as expected?  When I hear
> >>back, I will merge the branch to the trunk.
> >>
> >>Thanks,
> >>Mark
> >>_______________________________________________
> >>mpas-developers mailing list
> >>mpas-developers at mailman.ucar.edu
> >>http://mailman.ucar.edu/mailman/listinfo/mpas-developers