<HTML>
<HEAD>
<TITLE>Re: [mpas-developers] mpas output file naming conventions</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12pt'><BR>
Michael,<BR>
<BR>
Thanks for the clarification – and after I sent the last, I thought the same thing regarding a pass at requirements to see what we all really need.<BR>
<BR>
Phil<BR>
<BR>
<BR>
On 9/8/11 11:35 AM, "Michael Duda" <<a href="duda@ucar.edu">duda@ucar.edu</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12pt'>Phil,<BR>
<BR>
just to clarify, the functionality that's in the trunk adds the time<BR>
stamp to the file name "on-the-fly"; e.g., if one specifies<BR>
<BR>
config_output_name = 'foo.nc',<BR>
config_frames_per_outfile = 0,<BR>
<BR>
all output will be written to a single file, foo.nc. However, specifying<BR>
<BR>
config_frames_per_outfile = 1,<BR>
<BR>
will write a single frame to output files that will automatically be<BR>
named, e.g., foo.2011-09-08_00:00:00.nc, foo.2011-09-09_00:00:00.nc,<BR>
foo.2011-09-10_00:00:00.nc, etc., assuming output here is performed on a<BR>
daily interval.<BR>
<BR>
The requirement that the filename root be based on experiment name can<BR>
be accommodated by current functionality by changing config_output_name;<BR>
presumably, different experiments will require different input<BR>
(different initial conditions, and likely a different set of namelist<BR>
settings), so it doesn't seem too onerous to make a change to the<BR>
config_output_name variable when setting up the namelist and other input<BR>
for a different experiment, either through sed scripting or manually.<BR>
Might the ability to change the filename root based on the run<BR>
environment (specifically, environment variables) be of significant<BR>
benefit, though?<BR>
<BR>
We don't currently split restart frames between different files, but<BR>
it should be straightforward to extend the functionality from just<BR>
output files to restart files as well, if this would help with jobs that<BR>
restart more than once per submission.<BR>
<BR>
All of the above isn't to say that I think what we have currently is<BR>
perfect or ideal; at this point in the evolution of the MPAS framework<BR>
and infrastructure, I think it's important to try to capture all of our<BR>
requirements, and to experiment with ways that they could be<BR>
implemented, bearing in mind that a significant re-write of the MPAS I/O<BR>
subsystem is practically imminent. So, I think we're very much open to<BR>
suggestions for improvements, especially given that we may each have<BR>
different ways of using and running MPAS in the course of our work.<BR>
<BR>
Cheers,<BR>
Michael<BR>
<BR>
<BR>
On Thu, Sep 08, 2011 at 11:09:05AM -0600, Jones, Philip W wrote:<BR>
><BR>
> Michael,<BR>
><BR>
> I think we'll want a more general capability, including renaming the root based on experiment name and having one frame in each file with a simulation date stamp. While some of this could be handled via post-processing, it can be difficult to manage within a queue/script structure, especially if you're doing more than one restart per submission and need to keep them separate. It's much easier to do on the fly during the simulation.<BR>
><BR>
> Phil<BR>
><BR>
><BR>
> On 9/8/11 10:54 AM, "Michael Duda" <<a href="duda@ucar.edu">duda@ucar.edu</a>> wrote:<BR>
><BR>
> Hi, All.<BR>
><BR>
> In the trunk code, we recently added an option to split output across<BR>
> multiple files at the same time we added the time manager code. The new<BR>
> namelist option config_frames_per_outfile defaults to 0, in which case<BR>
> all output frames are written to a single output file whose name is<BR>
> simply the name specified by config_output_name. However, if<BR>
> config_frames_per_outfile is set to a positive integer n, at most n<BR>
> frames will be written to an output file, and the (n+1)st frame will<BR>
> cause a new output file to be created; obviously, this requires that we<BR>
> provided a mechanism to automatically generate unique names for output<BR>
> files, and the approach we've taken is to insert the time stamp of the<BR>
> first frame in the file into the file's name, e.g.,<BR>
> output.2011-09-08_00:00:00.nc. If it would be desirable to keep all<BR>
> output frames for a run in a single file, one can simply set<BR>
> config_frames_per_outfile to a very large integer -- larger than the<BR>
> number of output frames that will be generated during the run -- and the<BR>
> first output file will still have a time stamp embedded in its name. It<BR>
> sounds like this functionality would be of help in the scenarios that<BR>
> Sara described; would it simply be a matter of porting these changes<BR>
> from the trunk into, e.g., the mpas_cam_coupling branch?<BR>
><BR>
> Cheers,<BR>
> Michael<BR>
><BR>
><BR>
> On Thu, Sep 08, 2011 at 10:39:21AM -0600, Todd Ringler wrote:<BR>
> ><BR>
> > Hi Everyone,<BR>
> ><BR>
> > More broadly, the issue that Sara raises is a common headache in climate modeling. A simulations might be composed of a dozen or more queue submissions that pick up where the last submission ended.<BR>
> ><BR>
> > Sara offered one possible remedy. I am guessing Mat has also come up with solutions and that Phil has had to address this more broadly within the CESM. Can others offer up ideas on how we should build in support for production simulations that need a large number of submissions/restarts to complete?<BR>
> ><BR>
> > Cheers,<BR>
> > Todd<BR>
> ><BR>
> ><BR>
> > On Sep 8, 2011, at 10:22 AM, Sara A. Rauscher wrote:<BR>
> ><BR>
> > > Hi,<BR>
> > ><BR>
> > > I mentioned this to Todd and he suggested I post my user experience to the developer team.<BR>
> > ><BR>
> > > I've been running cam-mpas, and revision 983 brought up a small run-time hassle. Is there any way to automatically put a unique name on mpas output files instead of generic names like "output.nc", "ocean.nc" without editing namelist.input every time? For example, they could be time stamped by model time (YYYY-MM-DD-HH as in CESM) or number of time steps) or the real-world time at which the job is running.<BR>
> > ><BR>
> > > It's not a big deal during testing, but in production runs it starts to be a hassle. I have set my run script to capture the date from the latest CAM restart file and sed that into namelist.input so that the output files have a unique and meaningful name. It works fine but errors can still happen. In production runs, with one misnamed file, or writing over an output file, things get messy fast.<BR>
> > ><BR>
> > > thanks<BR>
> > > cheers<BR>
> > > sara<BR>
> > ><BR>
> > >> -------- Original Message --------<BR>
> > >> Subject: [mpas-developers] /home/subversion/mpas/model revision 983<BR>
> > >> Date: Wed, 7 Sep 2011 11:00:48 -0600 (MDT)<BR>
> > >> From: <a href="mpas-developers@ucar.edu">mpas-developers@ucar.edu</a><BR>
> > >> Reply-To: <a href="mpas-developers@ucar.edu">mpas-developers@ucar.edu</a><BR>
> > >> To: <a href="mpas-developers@ucar.edu">mpas-developers@ucar.edu</a><BR>
> > >><BR>
> > >> <a href="ringler@lanl.gov">ringler@lanl.gov</a> 2011-09-07 11:00:48 -0600 (Wed, 07 Sep 2011)<BR>
> > >><BR>
> > >> add default naming for I/O files<BR>
> > >> Modified: branches/ocean_projects/timesplitting_mrp/namelist.input.ocean<BR>
> > >> ===================================================================<BR>
> > >> --- branches/ocean_projects/timesplitting_mrp/namelist.input.ocean 2011-09-07 15:53:33 UTC (rev 982)<BR>
> > >> +++ branches/ocean_projects/timesplitting_mrp/namelist.input.ocean 2011-09-07 17:00:48 UTC (rev 983)<BR>
> > >> @@ -8,6 +8,9 @@<BR>
> > >> config_stats_interval = 100<BR>
> > >> /<BR>
> > >> &io<BR>
> > >> + config_input_name = 'ocean.nc'<BR>
> > >> + config_output_name = 'output.nc'<BR>
> > >> + config_restart_name = 'restart.nc'<BR>
> > >> /<BR>
> > >> &restart<BR>
> > >> config_restart_interval = 10000000<BR>
> > >><BR>
> > >><BR>
> > >> <Attached Message Part.txt><BR>
> > ><BR>
> > > _______________________________________________<BR>
> > > mpas-developers mailing list<BR>
> > > <a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><BR>
> > > <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><BR>
> ><BR>
><BR>
> > _______________________________________________<BR>
> > mpas-developers mailing list<BR>
> > <a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><BR>
> > <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><BR>
><BR>
> _______________________________________________<BR>
> mpas-developers mailing list<BR>
> <a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><BR>
> <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><BR>
><BR>
><BR>
><BR>
> ---<BR>
> Correspondence/TSPA/DUSA EARTH<BR>
> ------------------------------------------------------------<BR>
> Philip Jones <a href="pwjones@lanl.gov">pwjones@lanl.gov</a><BR>
> Climate, Ocean and Sea Ice Modeling<BR>
> Los Alamos National Laboratory<BR>
> T-3 MS B216 Ph: 505-500-2699<BR>
> PO Box 1663 Fax: 505-665-5926<BR>
> Los Alamos, NM 87545-1663<BR>
><BR>
><BR>
><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12pt'><BR>
</SPAN></FONT><SPAN STYLE='font-size:12pt'><FONT FACE="Courier, Courier New"><BR>
---<BR>
Correspondence/TSPA/DUSA EARTH<BR>
------------------------------------------------------------<BR>
Philip Jones <a href="pwjones@lanl.gov">pwjones@lanl.gov</a><BR>
Climate, Ocean and Sea Ice Modeling<BR>
Los Alamos National Laboratory<BR>
T-3 MS B216 Ph: 505-500-2699<BR>
PO Box 1663 Fax: 505-665-5926<BR>
Los Alamos, NM 87545-1663<BR>
</FONT></SPAN><FONT SIZE="1"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'><BR>
<BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12pt'><BR>
</SPAN></FONT>
</BODY>
</HTML>