[mpas-developers] mpas output file naming conventions
Jones, Philip W
pwjones at lanl.gov
Thu Sep 8 11:45:29 MDT 2011
Michael,
Thanks for the clarification - and after I sent the last, I thought the same thing regarding a pass at requirements to see what we all really need.
Phil
On 9/8/11 11:35 AM, "Michael Duda" <duda at ucar.edu> wrote:
Phil,
just to clarify, the functionality that's in the trunk adds the time
stamp to the file name "on-the-fly"; e.g., if one specifies
config_output_name = 'foo.nc',
config_frames_per_outfile = 0,
all output will be written to a single file, foo.nc. However, specifying
config_frames_per_outfile = 1,
will write a single frame to output files that will automatically be
named, e.g., foo.2011-09-08_00:00:00.nc, foo.2011-09-09_00:00:00.nc,
foo.2011-09-10_00:00:00.nc, etc., assuming output here is performed on a
daily interval.
The requirement that the filename root be based on experiment name can
be accommodated by current functionality by changing config_output_name;
presumably, different experiments will require different input
(different initial conditions, and likely a different set of namelist
settings), so it doesn't seem too onerous to make a change to the
config_output_name variable when setting up the namelist and other input
for a different experiment, either through sed scripting or manually.
Might the ability to change the filename root based on the run
environment (specifically, environment variables) be of significant
benefit, though?
We don't currently split restart frames between different files, but
it should be straightforward to extend the functionality from just
output files to restart files as well, if this would help with jobs that
restart more than once per submission.
All of the above isn't to say that I think what we have currently is
perfect or ideal; at this point in the evolution of the MPAS framework
and infrastructure, I think it's important to try to capture all of our
requirements, and to experiment with ways that they could be
implemented, bearing in mind that a significant re-write of the MPAS I/O
subsystem is practically imminent. So, I think we're very much open to
suggestions for improvements, especially given that we may each have
different ways of using and running MPAS in the course of our work.
Cheers,
Michael
On Thu, Sep 08, 2011 at 11:09:05AM -0600, Jones, Philip W wrote:
>
> Michael,
>
> I think we'll want a more general capability, including renaming the root based on experiment name and having one frame in each file with a simulation date stamp. While some of this could be handled via post-processing, it can be difficult to manage within a queue/script structure, especially if you're doing more than one restart per submission and need to keep them separate. It's much easier to do on the fly during the simulation.
>
> Phil
>
>
> On 9/8/11 10:54 AM, "Michael Duda" <duda at ucar.edu> wrote:
>
> Hi, All.
>
> In the trunk code, we recently added an option to split output across
> multiple files at the same time we added the time manager code. The new
> namelist option config_frames_per_outfile defaults to 0, in which case
> all output frames are written to a single output file whose name is
> simply the name specified by config_output_name. However, if
> config_frames_per_outfile is set to a positive integer n, at most n
> frames will be written to an output file, and the (n+1)st frame will
> cause a new output file to be created; obviously, this requires that we
> provided a mechanism to automatically generate unique names for output
> files, and the approach we've taken is to insert the time stamp of the
> first frame in the file into the file's name, e.g.,
> output.2011-09-08_00:00:00.nc. If it would be desirable to keep all
> output frames for a run in a single file, one can simply set
> config_frames_per_outfile to a very large integer -- larger than the
> number of output frames that will be generated during the run -- and the
> first output file will still have a time stamp embedded in its name. It
> sounds like this functionality would be of help in the scenarios that
> Sara described; would it simply be a matter of porting these changes
> from the trunk into, e.g., the mpas_cam_coupling branch?
>
> Cheers,
> Michael
>
>
> On Thu, Sep 08, 2011 at 10:39:21AM -0600, Todd Ringler wrote:
> >
> > Hi Everyone,
> >
> > More broadly, the issue that Sara raises is a common headache in climate modeling. A simulations might be composed of a dozen or more queue submissions that pick up where the last submission ended.
> >
> > Sara offered one possible remedy. I am guessing Mat has also come up with solutions and that Phil has had to address this more broadly within the CESM. Can others offer up ideas on how we should build in support for production simulations that need a large number of submissions/restarts to complete?
> >
> > Cheers,
> > Todd
> >
> >
> > On Sep 8, 2011, at 10:22 AM, Sara A. Rauscher wrote:
> >
> > > Hi,
> > >
> > > I mentioned this to Todd and he suggested I post my user experience to the developer team.
> > >
> > > I've been running cam-mpas, and revision 983 brought up a small run-time hassle. Is there any way to automatically put a unique name on mpas output files instead of generic names like "output.nc", "ocean.nc" without editing namelist.input every time? For example, they could be time stamped by model time (YYYY-MM-DD-HH as in CESM) or number of time steps) or the real-world time at which the job is running.
> > >
> > > It's not a big deal during testing, but in production runs it starts to be a hassle. I have set my run script to capture the date from the latest CAM restart file and sed that into namelist.input so that the output files have a unique and meaningful name. It works fine but errors can still happen. In production runs, with one misnamed file, or writing over an output file, things get messy fast.
> > >
> > > thanks
> > > cheers
> > > sara
> > >
> > >> -------- Original Message --------
> > >> Subject: [mpas-developers] /home/subversion/mpas/model revision 983
> > >> Date: Wed, 7 Sep 2011 11:00:48 -0600 (MDT)
> > >> From: mpas-developers at ucar.edu
> > >> Reply-To: mpas-developers at ucar.edu
> > >> To: mpas-developers at ucar.edu
> > >>
> > >> ringler at lanl.gov 2011-09-07 11:00:48 -0600 (Wed, 07 Sep 2011)
> > >>
> > >> add default naming for I/O files
> > >> Modified: branches/ocean_projects/timesplitting_mrp/namelist.input.ocean
> > >> ===================================================================
> > >> --- branches/ocean_projects/timesplitting_mrp/namelist.input.ocean 2011-09-07 15:53:33 UTC (rev 982)
> > >> +++ branches/ocean_projects/timesplitting_mrp/namelist.input.ocean 2011-09-07 17:00:48 UTC (rev 983)
> > >> @@ -8,6 +8,9 @@
> > >> config_stats_interval = 100
> > >> /
> > >> &io
> > >> + config_input_name = 'ocean.nc'
> > >> + config_output_name = 'output.nc'
> > >> + config_restart_name = 'restart.nc'
> > >> /
> > >> &restart
> > >> config_restart_interval = 10000000
> > >>
> > >>
> > >> <Attached Message Part.txt>
> > >
> > > _______________________________________________
> > > mpas-developers mailing list
> > > mpas-developers at mailman.ucar.edu
> > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> >
>
> > _______________________________________________
> > mpas-developers mailing list
> > mpas-developers at mailman.ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
>
>
> ---
> Correspondence/TSPA/DUSA EARTH
> ------------------------------------------------------------
> Philip Jones pwjones at lanl.gov
> Climate, Ocean and Sea Ice Modeling
> Los Alamos National Laboratory
> T-3 MS B216 Ph: 505-500-2699
> PO Box 1663 Fax: 505-665-5926
> Los Alamos, NM 87545-1663
>
>
>
---
Correspondence/TSPA/DUSA EARTH
------------------------------------------------------------
Philip Jones pwjones at lanl.gov
Climate, Ocean and Sea Ice Modeling
Los Alamos National Laboratory
T-3 MS B216 Ph: 505-500-2699
PO Box 1663 Fax: 505-665-5926
Los Alamos, NM 87545-1663
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20110908/9e3e9fb5/attachment-0001.html
More information about the mpas-developers
mailing list