[mpas-developers] config_write_initial_output option

Michael Duda duda at ucar.edu
Tue May 15 14:00:10 MDT 2012


Hi, Mark.

I've just committed changes to the branch to support appending to
files in the IO layers. Next, I think we'll need to add more logic
at the core level to determine the correct filename to try to open
based on the number of frames per file and whether we will write the
initial time frame; I can take a look at this, too. 

Cheers,
Michael


On Fri, May 11, 2012 at 04:35:42PM -0600, Mark Petersen wrote:
> Michael,
> 
> Thanks for your work on this.  What you are suggesting is more 
> comprehensive than my changes.  I agree that appending output after a 
> restart, as if it were a continuous run, is a nice feature.
> 
> Please commit whatever changes you like to my branch.  I will go ahead 
> and test it on the ocean core when I see your commits.
> 
> Mark
> 
> On 05/11/12 15:27, Michael Duda wrote:
> >Hi, Mark.
> >
> >Although the branch does avoid writing the initial time for a cold-start 
> >run
> >with all frames in the same output file, it appears that the output 
> >filename has
> >the timestamp of the first frame in the file added to it; whereas with
> >config_write_initial_output=.true. we might get an output file 
> >"output.nc", with
> >the option set to false, the output file is named, e.g.,
> >"output.0000-01-01_06.00.00.nc". I suspect this is done to avoid filename
> >conflicts when restarting a run, which is where I think what I had in mind
> >differs from the current behavior.
> >
> >Consider the example where we write history every 6h and write 4 frames per
> >outfile (in other words, separate output files for each day) with files
> >
> >    output.0000-01-01_00.00.00.nc
> >    output.0000-01-02_00.00.00.nc
> >    output.0000-01-03_00.00.00.nc
> >    ...
> >
> >that contain times
> >
> >   "0000-01-01_00:00:00                                             ",
> >   "0000-01-01_06:00:00                                             ",
> >   "0000-01-01_12:00:00                                             ",
> >   "0000-01-01_18:00:00                                             " ;
> >
> >   "0000-01-02_00:00:00                                             ",
> >   "0000-01-02_06:00:00                                             ",
> >   "0000-01-02_12:00:00                                             ",
> >   "0000-01-02_18:00:00                                             " ;
> >
> >   "0000-01-03_00:00:00                                             ",
> >   "0000-01-03_06:00:00                                             ",
> >   "0000-01-03_12:00:00                                             ",
> >   "0000-01-03_18:00:00                                             " ;
> >
> >   ...
> >
> >respectively. It may typically happen that we run for an integral number of
> >days, in which we end up with an output file at the final time that has 
> >just a
> >single output time, e.g.,
> >
> >    output.0000-01-05_00.00.00.nc
> >
> >with the output frame at 0000-01-05_00:00:00 only. Ideally (in my 
> >opinion), we'd
> >like to be able to restart the model from 0000-01-05_00:00:00 and avoid 
> >writing
> >the intial time, but still write the times 0000-01-05_06:00:00, 
> >0000-01-05_12:00:00,
> >and 0000-01-05_18:00:00 to the file output.0000-01-05_00.00.00.nc. 
> >Basically,
> >the restarted run would have output files structured in an identical way 
> >to a
> >single run for the complete simulation period.
> >
> >To support this behavior, I've modified the IO layers to support appending 
> >to
> >existing files, and to seek through files to find the frame number that we 
> >need
> >to begin writing at. If it sounds alright to you, I can commit the changes 
> >that
> >I have to support appending to existing output files to the branch. The 
> >only
> >remaining work then would be in the code where we open and close files 
> >(the same
> >parts that you've modified); here, I think we'd need logic to determine 
> >whether
> >we want to append to an output file or to overwrite it (considering 
> >whether it
> >is a restart run, whether the file already exists, and the number of frames
> >in each file). I'd volunteer to make the additional changes in the branch, 
> >if
> >you'd like.
> >
> >Michael
> >
> >
> >On Thu, May 10, 2012 at 03:51:34PM -0600, Mark Petersen wrote:
> >>Michael,
> >>
> >>Thanks for your thoughts.
> >>
> >>I tested the ocean core using config_write_initial_output=.false. with
> >>both single and multiple frames per output file, and both worked as
> >>expected.  In both cases,
> >>    mpas_output_state_init
> >>is called the first time within a
> >>   if(output_frame == 1) then
> >>branch in mpas_*_mpas_core.F.  Is calling the initialization routine
> >>your concern for multiple frames?
> >>
> >>Your more general implementation of time frames sounds good.  It is
> >>mostly a question of timing.  If you are thinking of making these
> >>changes in the next week or two, I'm happy to wait and you can use my
> >>branch if you like.  If it is a longer term improvement, I prefer to
> >>merge this config_write_initial_output flag to the trunk so I don't need
> >>to maintain a branch from which to run our large simulations.
> >>
> >>Thanks,
> >>Mark
> >>
> >>
> >>On 05/10/12 14:24, Michael Duda wrote:
> >>>Mark,
> >>>
> >>>I think this would be a useful option, though as currently implemented
> >>>in the branch, I'm not sure that it would work as expected when there
> >>>are multiple history frames written to the same file. I've also been
> >>>looking at implementing this, and I have some changes to the IO layer to
> >>>support appending time frames to an existing file that I think might be
> >>>useful for a completely general implementation of the
> >>>write_initial_output functionality. I can send these changes to you to
> >>>take a look at, or I can merge them into the branch; just let me know.
> >>>
> >>>Michael
> >>>
> >>>
> >>>On Thu, May 10, 2012 at 01:06:15PM -0600, Mark Petersen wrote:
> >>>>MPAS Developers,
> >>>>
> >>>>I added the flag config_write_initial_output to all cores in the branch
> >>>>ocean_projects/first_output_optional.  It simply controls if an output
> >>>>file is written upon start-up.  The ocean core needs this for our large
> >>>>runs, to save i/o and for simply time-averaged variables.  Because the
> >>>>first file initialized in driver/mpas_subdriver.F, it affects all
> >>>>cores.  In addition, it seems like a useful flag for everyone.
> >>>>
> >>>>I tested by compiling all four cores, and running with the ocean core.
> >>>>Could someone run a little test with the atmospheric cores, switching
> >>>>this flag on and off, and make sure it works as expected?  When I hear
> >>>>back, I will merge the branch to the trunk.
> >>>>
> >>>>Thanks,
> >>>>Mark
> >>>>_______________________________________________
> >>>>mpas-developers mailing list
> >>>>mpas-developers at mailman.ucar.edu
> >>>>http://mailman.ucar.edu/mailman/listinfo/mpas-developers


More information about the mpas-developers mailing list