[Dart-dev] [6528] DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid: At the moment this works but is not optimal.
nancy at ucar.edu
nancy at ucar.edu
Fri Oct 18 15:09:24 MDT 2013
Revision: 6528
Author: thoar
Date: 2013-10-18 15:09:24 -0600 (Fri, 18 Oct 2013)
Log Message:
-----------
At the moment this works but is not optimal.
Must explore removing the rtm namelist section.
Must explore reducing the CLM history file information.
Must explore using 80 ensemble members for Dec 2008.
Must explore using the stage_cesm_files under different scenarios.
Added Paths:
-----------
DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid
-------------- next part --------------
Added: DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid
===================================================================
--- DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid (rev 0)
+++ DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid 2013-10-18 21:09:24 UTC (rev 6528)
@@ -0,0 +1,868 @@
+#!/bin/csh -f
+#
+# DART software - Copyright 2004 - 2013 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# DART $Id: CESM1_1_1_hybrid 6396 2013-08-12 17:51:26Z thoar $
+
+# ---------------------
+# Purpose
+# ---------------------
+#
+# This script is designed to set up, stage, and build a multi-instance run of CESM
+# using an F compset where CAM and CLM are active. The initial states
+# come from a single multi-instance reference case so a CESM hybrid setup is used.
+#
+# Because the atmosphere assimilations typically occur every 6 hours, the methodology
+# here reflects that. All of CESM stops every 6 hours (requiring the ocean to couple
+# 4x per day) so that a CAM output file would be available for assimilation.
+#
+# This script results in a viable setup for a CESM multi-instance experiment. You
+# are STRONGLY encouraged to run the multi-instance CESM a few times and experiment
+# with different settings BEFORE you try to assimilate observations. The amount of
+# data volume is quite large and you should become comfortable using CESM's restart
+# capability to re-stage files in your RUN directory
+#
+# CESM/DART requires some modifications to the CESM source code EVEN IF YOU ARE NOT
+# assimilating. The modifications for CAM require a modification to the
+# CESM ${CASE}.run script. The modifications will invoke a DART script that will
+# either assimilate or 'do nothing'. It is stronly recommended that you configure
+# the ${CASE}.run script to 'do nothing' initially.
+#
+# CASEROOT/CESM_DART_config will augment the CESM case with the required setup
+# and configuration to perform a DART assimilation and modify the ${CASE}.run script
+# as mentioned previously - after it makes a backup copy.
+#
+# This script relies heavily on the information in:
+# http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/book1.html
+#
+# ---------------------
+# How to use this script.
+# ---------------------
+#
+# -- You will have to read and understand the script in its entirety.
+# You will have to modify things outside this script.
+# This script sets up a CESM multi-instance run as we understand them and
+# it has almost nothing to do with DART. This is intentional.
+#
+# -- Edit and run this script in the $DART/models/CESM/shell_scripts directory
+# or copy it to somewhere that it will be preserved and run it there.
+# It will create a CESM 'CASE' directory, where the model will be built,
+# and a run directory, where each forecast (and assimilation) will
+# take place. The short term archiver will use a third directory for
+# storage of model output until it can be moved to long term storage (HPSS)
+#
+# -- Examine the whole script to identify things to change for your experiments.
+#
+# -- Provide the CESM initial ensemble needed by your run.
+#
+# -- Run this script.
+# -- Edit the DART input.nml that appears in the $CASEROOT directory, if not done here.
+# -- Submit the job using $CASEROOT/${case}.submit
+#
+# -- Read, understand, and execute ${CASEROOT}/CESM_DART_config
+#
+# Table of contents:
+# Set up case and directory names
+# Refined grid mods
+# create_newcase
+# cd caseroot
+# xmlchanges
+# cesm_setup
+# namelist mods
+# preview_namelists
+# SourceMods
+# case.build
+# stage ICs
+# update the run script
+# Bring in parts of DART; executables, support files,...
+#
+# ---------------------
+# Important features
+# ---------------------
+#
+# If you want to change something in your case other than the runtime
+# settings, it is safest to delete everything and start the run from scratch.
+# For the brave, read
+#
+# http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x1142.html
+#
+# and you may be able to salvage something with
+# ./cesm_setup -clean
+# ./cesm_setup
+# ./${case}.clean_build
+# ./${case}.build
+#
+# ==============================================================================
+# ==== Set case options
+# ==============================================================================
+
+# the value of "case" will be used many ways;
+# directory and file names, both locally and on HPSS, and
+# script names; so consider its length and information content.
+# num_instances: Number of ensemble members
+# BUG 1384 applies here, so ocean and atm/land must be at same resolution.
+# 0.9x1.25_0.9x1.25 (f09_f09)
+# 1 deg grid for atm/lnd, gx1v6 mask. Normally used for CAM with prescribed lnd, ocn and ice.
+
+setenv case test_cam
+setenv compset F_AMIP_CAM5
+setenv resolution f09_f09
+setenv cesmtag cesm1_1_1
+setenv num_instances 4
+
+# ==============================================================================
+# define machines and directories
+#
+# mach Computer name
+# cesmroot Location of the cesm code base
+# For cesm1_1_1 on yellowstone
+# caseroot Your (future) cesm case directory, where this CESM+DART will be built.
+# Preferably not a frequently scrubbed location.
+# This script will delete any existing caseroot, so this script,
+# and other useful things should be kept elsewhere.
+# rundir (Future) Run-time directory; scrubbable, large amount of space needed.
+# exeroot (Future) directory for executables - scrubbable, large amount of space needed.
+# archdir (Future) Short-term archive directory
+# until the long-term archiver moves it to permanent storage.
+# dartroot Location of _your_ DART installation
+# This is passed on to the CESM_DART_config script.
+# ==============================================================================
+
+setenv mach yellowstone
+setenv cesm_datadir /glade/p/cesm/cseg/inputdata
+
+setenv cesmroot /glade/p/cesm/cseg/collections/$cesmtag
+setenv caseroot /glade/p/work/${USER}/cases/${case}
+setenv exeroot /glade/scratch/${USER}/${case}/bld
+setenv rundir /glade/scratch/${USER}/${case}/run
+setenv archdir /glade/scratch/${USER}/archive/${case}
+setenv dartroot /glade/u/home/${USER}/svn/DART/trunk
+
+# ==============================================================================
+# configure settings
+# The reference case has dates in it.
+# For a 'hybrid' start, these may be unrelated to the refyear, refmon, refday.
+# ==============================================================================
+
+setenv run_refcase cesm_hybrid
+setenv refyear 2004
+setenv refmon 01
+setenv refday 10
+setenv run_reftod 00000
+setenv run_refdate $refyear-$refmon-$refday
+
+# THIS IS THE LOCATION of the 'reference case'.
+
+setenv stagedir /glade/p/image/CESM_initial_ensemble/rest/${run_refdate}-${run_reftod}
+
+# setenv stagedir /glade/p/work/raeder/Models/CAM_init/${run_refcase}_${run_refdate}
+
+# ==============================================================================
+# runtime settings
+#
+# sst_dataset Data ocean file and
+# sst_grid supporting grid file. Must be consistent.
+#
+# resubmit How many job steps to run on continue runs (will be 0 initially)
+# stop_option Units for determining the forecast length between assimilations
+# stop_n Number of time units in the first forecast
+#
+# If the long-term archiver is off, you get a chance to examine the files before
+# they get moved to long-term storage. You can always submit $CASE.l_archive
+# whenever you want to free up space in the short-term archive directory.
+#
+# ==============================================================================
+
+setenv start_year 2004
+setenv start_month 01
+setenv start_day 10
+setenv start_tod 0
+setenv run_startdate ${start_year}-${start_month}-${start_day}
+setenv year_start 1850
+setenv year_end 2012
+
+# setenv sst_grid ${cesm_datadir}/ocn/docn7/domain.ocn.gx1v6.090206.nc
+setenv sst_grid ${cesm_datadir}/share/domains/domain.ocn.fv0.9x1.25_gx1v6.130409.nc
+setenv sst_dataset ${cesm_datadir}/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_1850_2011_c110307.nc
+
+setenv short_term_archiver on
+setenv long_term_archiver off
+setenv resubmit 0
+setenv stop_option nhours
+setenv stop_n 6
+
+# ==============================================================================
+# job settings
+#
+# queue can be changed during a series by changing the ${case}.run
+# timewall can be changed during a series by changing the ${case}.run
+#
+# TJH: Advancing 30 instances for 6 hours with 900 pes
+# took less than 10 minutes on yellowstone.
+# ==============================================================================
+
+setenv ACCOUNT P86850054
+setenv queue premium
+setenv timewall 0:20
+
+# ==============================================================================
+# set these standard commands based on the machine you are running on.
+# ==============================================================================
+
+set nonomatch # suppress "rm" warnings if wildcard does not match anything
+
+# The FORCE options are not optional.
+# The VERBOSE options are useful for debugging though
+# some systems don't like the -v option to any of the following
+switch ("`hostname`")
+ case be*:
+ # NCAR "bluefire"
+ set MOVE = '/usr/local/bin/mv -fv'
+ set COPY = '/usr/local/bin/cp -fv --preserve=timestamps'
+ set LINK = '/usr/local/bin/ln -fvs'
+ set REMOVE = '/usr/local/bin/rm -fr'
+
+ breaksw
+ default:
+ # NERSC "hopper", NWSC "yellowstone"
+ set MOVE = '/bin/mv -fv'
+ set COPY = '/bin/cp -fv --preserve=timestamps'
+ set LINK = '/bin/ln -fvs'
+ set REMOVE = '/bin/rm -fr'
+
+ breaksw
+endsw
+
+# ==============================================================================
+# Make sure the CESM directories exist.
+# VAR is the shell variable name, DIR is the value
+# ==============================================================================
+
+foreach VAR ( cesmroot dartroot stagedir )
+ set DIR = `eval echo \${$VAR}`
+ if ( ! -d $DIR ) then
+ echo "ERROR: directory '$DIR' not found"
+ echo " In the setup script check the setting of: $VAR "
+ exit -1
+ endif
+end
+
+# ==============================================================================
+# Create the case - this creates the CASEROOT directory.
+#
+# For list of the pre-defined component sets: ./create_newcase -list
+# To create a variant compset, see the CESM documentation and carefully
+# incorporate any needed changes into this script.
+# ==============================================================================
+
+# fatal idea to make caseroot the same dir as where this setup script is
+# since the build process removes all files in the caseroot dir before
+# populating it. try to prevent shooting yourself in the foot.
+
+if ( $caseroot == `dirname $0` ) then
+ echo "ERROR: the setup script should not be located in the caseroot"
+ echo "directory, because all files in the caseroot dir will be removed"
+ echo "before creating the new case. move the script to a safer place."
+ exit -1
+endif
+
+echo "removing old files from ${caseroot}"
+echo "removing old files from ${exeroot}"
+echo "removing old files from ${rundir}"
+${REMOVE} ${caseroot}
+${REMOVE} ${exeroot}
+${REMOVE} ${rundir}
+
+${cesmroot}/scripts/create_newcase -case ${caseroot} -mach ${mach} \
+ -res ${resolution} -compset ${compset}
+
+if ( $status != 0 ) then
+ echo "ERROR: Case could not be created."
+ exit -1
+endif
+
+# preserve a copy of this script as it was run
+set ThisFileName = $0:t
+${COPY} $ThisFileName ${caseroot}/${ThisFileName}.original
+
+# ==============================================================================
+# Record the DARTROOT directory and copy the DART setup script to CASEROOT.
+# CESM_DART_config can be run at some later date if desired, but it presumes
+# to be run from a CASEROOT directory. If CESM_DART_config does not exist locally,
+# then it better exist in the expected part of the DARTROOT tree.
+# ==============================================================================
+
+if ( ! -e CESM_DART_config ) then
+ ${COPY} ${dartroot}/models/cam/shell_CESM/CESM_DART_config .
+endif
+
+if ( -e CESM_DART_config ) then
+ sed -e "s#BOGUS_DART_ROOT_STRING#$dartroot#" < CESM_DART_config >! temp.$$
+ ${MOVE} temp.$$ ${caseroot}/CESM_DART_config
+ chmod 755 ${caseroot}/CESM_DART_config
+else
+ echo "ERROR: the script to configure for data assimilation is not available."
+ echo " CESM_DART_config MUST be present locally or in"
+ echo " ${dartroot}/models/CESM/shell_scripts/"
+ exit -2
+endif
+
+# ==============================================================================
+# Configure the case.
+# ==============================================================================
+
+cd ${caseroot}
+
+source ./Tools/ccsm_getenv || exit -2
+
+# MAX_TASKS_PER_NODE comes from $case/Tools/mkbatch.$machine
+@ ptile = $MAX_TASKS_PER_NODE / 2
+@ nthreads = 1
+
+# Save a copy for debug purposes
+foreach FILE ( *xml )
+ if ( ! -e ${FILE}.original ) then
+ ${COPY} $FILE ${FILE}.original
+ endif
+end
+
+if ($num_instances < 4) then
+
+ # This is only for the purpose of debugging the code.
+ @ atm_tasks = $ptile * $num_instances * 4
+ @ lnd_tasks = $ptile * $num_instances * 4
+ @ ice_tasks = $ptile * $num_instances * 4
+ @ ocn_tasks = $ptile * $num_instances
+ @ cpl_tasks = $ptile * $num_instances
+ @ glc_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances
+
+else
+
+ # This works, but a more efficient layout should be used
+ # CAM only; 1 node/instance; no cross node communications
+ # as long as memory is big enough.
+ @ atm_tasks = $ptile * $num_instances * 4
+ @ lnd_tasks = $ptile * $num_instances * 4
+ @ ice_tasks = $ptile * $num_instances * 4
+ @ ocn_tasks = $ptile * $num_instances
+ @ cpl_tasks = $ptile * $num_instances
+ @ glc_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances
+
+endif
+
+# echo "task partitioning ... perhaps ... atm // ocn // lnd+ice+glc+rof"
+# presently, all components run 'serially' - one after another.
+# Yellowstone: no large memory nodes, and 15 tasks/node is recommended.
+# Edwards says there's no speed up by running non-active components concurrently,
+# after ATM has run, so just run all components sequentially.
+# BUT, do arrange it so that each member(instance) spans complete nodes:
+# modulo(total pe count / number of instances, 15) == 0.
+
+echo ""
+echo "ATM gets $atm_tasks"
+echo "LND gets $lnd_tasks"
+echo "ICE gets $ice_tasks"
+echo "OCN gets $ocn_tasks"
+echo "CPL gets $cpl_tasks"
+echo "GLC gets $glc_tasks"
+echo "ROF gets $rof_tasks"
+echo ""
+
+./xmlchange NTHRDS_ATM=$nthreads,NTASKS_ATM=$atm_tasks,NINST_ATM=$num_instances
+./xmlchange NTHRDS_LND=$nthreads,NTASKS_LND=$lnd_tasks,NINST_LND=$num_instances
+./xmlchange NTHRDS_ICE=$nthreads,NTASKS_ICE=$ice_tasks,NINST_ICE=$num_instances
+./xmlchange NTHRDS_OCN=$nthreads,NTASKS_OCN=$ocn_tasks,NINST_OCN=1
+./xmlchange NTHRDS_CPL=$nthreads,NTASKS_CPL=$cpl_tasks
+./xmlchange NTHRDS_GLC=$nthreads,NTASKS_GLC=$glc_tasks,NINST_GLC=1
+./xmlchange NTHRDS_ROF=$nthreads,NTASKS_ROF=$rof_tasks,NINST_ROF=1
+./xmlchange ROOTPE_ATM=0
+./xmlchange ROOTPE_LND=0
+./xmlchange ROOTPE_ICE=0
+./xmlchange ROOTPE_OCN=0
+./xmlchange ROOTPE_CPL=0
+./xmlchange ROOTPE_GLC=0
+./xmlchange ROOTPE_ROF=0
+
+# http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c1158.html#run_start_stop
+# "A hybrid run indicates that CESM is initialized more like a startup, but uses
+# initialization datasets from a previous case. This is somewhat analogous to a
+# branch run with relaxed restart constraints. A hybrid run allows users to bring
+# together combinations of initial/restart files from a previous case (specified
+# by $RUN_REFCASE) at a given model output date (specified by $RUN_REFDATE).
+# Unlike a branch run, the starting date of a hybrid run (specified by $RUN_STARTDATE)
+# can be modified relative to the reference case. In a hybrid run, the model does not
+# continue in a bit-for-bit fashion with respect to the reference case. The resulting
+# climate, however, should be continuous provided that no model source code or
+# namelists are changed in the hybrid run. In a hybrid initialization, the ocean
+# model does not start until the second ocean coupling (normally the second day),
+# and the coupler does a "cold start" without a restart file."
+
+# TJH:
+# DART's CAM implementation causes a bit more complexity. DART only uses CAM _initial_
+# files, not RESTART files, so there are sourcemods to force a hybrid start for CAM to
+# read initial files - even when CONTINUE_RUN = TRUE.
+
+./xmlchange RUN_TYPE=hybrid
+./xmlchange RUN_STARTDATE=$run_startdate
+./xmlchange START_TOD=$start_tod
+./xmlchange RUN_REFCASE=$run_refcase
+./xmlchange RUN_REFDATE=$run_refdate
+./xmlchange RUN_REFTOD=$run_reftod
+./xmlchange BRNCH_RETAIN_CASENAME=FALSE
+./xmlchange GET_REFCASE=FALSE
+./xmlchange EXEROOT=${exeroot}
+
+./xmlchange SSTICE_DATA_FILENAME=$sst_dataset
+./xmlchange SSTICE_GRID_FILENAME=$sst_grid
+./xmlchange SSTICE_YEAR_ALIGN=$year_start
+./xmlchange SSTICE_YEAR_START=$year_start
+./xmlchange SSTICE_YEAR_END=$year_end
+
+# Do not change the CALENDAR or the CONTINUE_RUN
+
+./xmlchange CALENDAR=GREGORIAN
+
+./xmlchange STOP_OPTION=$stop_option
+./xmlchange STOP_N=$stop_n
+./xmlchange CONTINUE_RUN=FALSE
+./xmlchange RESUBMIT=$resubmit
+
+./xmlchange PIO_TYPENAME=pnetcdf
+
+# COUPLING discussion. F compsets are 'tight' coupling.
+# Only change the ATM_NCPL ... everything is based on this one value,
+# including CAM physics and dynamics timesteps.
+# Default values for coupling are preserved in env_run.xml.original
+
+./xmlchange NCPL_BASE_PERIOD=day
+./xmlchange ATM_NCPL=48
+
+# TJH FIXME ... any reason to turn ON the biogeochemistry in CLM when atmos only?
+# There is no compset that has everything we want,
+# so we are also turning on the CLM biogeochemistry.
+#./xmlchange CLM_CONFIG_OPTS='-bgc cn'
+
+if ($short_term_archiver == 'off') then
+ ./xmlchange DOUT_S=FALSE
+else
+ ./xmlchange DOUT_S=TRUE
+ ./xmlchange DOUT_S_ROOT=${archdir}
+ ./xmlchange DOUT_S_SAVE_INT_REST_FILES=FALSE
+endif
+if ($long_term_archiver == 'off') then
+ ./xmlchange DOUT_L_MS=FALSE
+else
+ ./xmlchange DOUT_L_MS=TRUE
+ ./xmlchange DOUT_L_MSROOT="csm/${case}"
+ ./xmlchange DOUT_L_HTAR=FALSE
+endif
+
+# The river transport model ON is useful only when using an active ocean or
+# land surface diagnostics. Setting ROF_GRID, RTM_MODE to 'null' turns off the RTM.
+
+./xmlchange ROF_GRID='null'
+./xmlchange RTM_MODE='NULL'
+
+# level of debug output, 0=minimum, 1=normal, 2=more, 3=too much, valid values: 0,1,2,3 (integer)
+# Early versions (late 2012) of CAM-SE could not handle these being other than FALSE and 0.
+./xmlchange DEBUG=TRUE
+./xmlchange INFO_DBUG=1
+
+# ==============================================================================
+# Set up the case.
+# This creates the EXEROOT and RUNDIR directories.
+# ==============================================================================
+
+./cesm_setup
+
+if ( $status != 0 ) then
+ echo "ERROR: Case could not be set up."
+ exit -2
+endif
+
+# ==============================================================================
+# Edit the run script to reflect queue and wallclock
+# ==============================================================================
+
+echo ''
+echo 'Updating the run script to set wallclock and queue.'
+echo ''
+
+if ( ! -e ${case}.run.original ) then
+ ${COPY} ${case}.run ${case}.run.original
+endif
+
+source Tools/ccsm_getenv
+set BATCH = `echo $BATCHSUBMIT | sed 's/ .*$//'`
+switch ( $BATCH )
+ case bsub*:
+ # NCAR "bluefire", "yellowstone"
+ set TIMEWALL=`grep BSUB ${case}.run | grep -e '-W' `
+ set QUEUE=`grep BSUB ${case}.run | grep -e '-q' `
+ sed -e "s/$TIMEWALL[3]/$timewall/" \
+ -e "s/ptile=[0-9][0-9]*/ptile=$ptile/" \
+ -e "s/$QUEUE[3]/$queue/" < ${case}.run >! temp.$$
+ ${MOVE} temp.$$ ${case}.run
+ chmod 755 ${case}.run
+ breaksw
+
+ default:
+
+ breaksw
+endsw
+
+# This is the part that modifies the run script to allow CESM to advance
+# correctly given the modifications necessary for CAM. It also copies several
+# required DART files to the caseroot directory.
+
+./CESM_DART_config
+
+# ==============================================================================
+# Update source files.
+# Ideally, using DART would not require any modifications to the model source.
+# Until then, this script accesses sourcemods from a hardwired location.
+# If you have additional sourcemods, they will need to be merged into any DART
+# mods and put in the SourceMods subdirectory found in the 'caseroot' directory.
+# ==============================================================================
+
+if ( -d ~/${cesmtag}/SourceMods ) then
+ ${COPY} -r ~/${cesmtag}/SourceMods/* ${caseroot}/SourceMods/
+else
+ echo "ERROR - No SourceMods for this case."
+ echo "ERROR - No SourceMods for this case."
+ echo "DART requires modifications to several src files."
+ echo "These files can be downloaded from:"
+ echo "http://www.image.ucar.edu/pub/DART/CESM/DART_SourceMods_cesm1_1_1.tar"
+ echo "untar these into your HOME directory - they will create a"
+ echo "~/cesm_1_1_1 directory with the appropriate SourceMods structure."
+ exit -4
+endif
+
+# The CESM multi-instance capability is relatively new and still has a few
+# implementation bugs. These are known problems and will be fixed soon.
+# this should be removed when the files are fixed:
+
+echo "REPLACING BROKEN CESM FILES HERE - SHOULD BE REMOVED WHEN FIXED"
+echo caseroot is ${caseroot}
+if ( -d ~/${cesmtag} ) then
+
+ # preserve the original version of the files
+ if ( ! -e ${caseroot}/Buildconf/clm.buildnml.csh.original ) then
+ ${MOVE} ${caseroot}/Buildconf/clm.buildnml.csh \
+ ${caseroot}/Buildconf/clm.buildnml.csh.original
+ endif
+ if ( ! -e ${caseroot}/Buildconf/rtm.buildnml.csh.original ) then
+ ${MOVE} ${caseroot}/Buildconf/rtm.buildnml.csh \
+ ${caseroot}/Buildconf/rtm.buildnml.csh.original
+ endif
+ if ( ! -e ${caseroot}/preview_namelists.original ) then
+ ${MOVE} ${caseroot}/preview_namelists \
+ ${caseroot}/preview_namelists.original
+ endif
+
+ # patch/replace the broken files
+ ${COPY} ~/${cesmtag}/clm.buildnml.csh ${caseroot}/Buildconf/.
+ ${COPY} ~/${cesmtag}/rtm.buildnml.csh ${caseroot}/Buildconf/.
+ ${COPY} ~/${cesmtag}/preview_namelists ${caseroot}/.
+
+endif
+
+# ==============================================================================
+# Modify namelist templates for each instance. This is a bit of a nuisance in
+# that we are pulling in restart and initial files from 'all over the place'
+# and each model component has a different strategy.
+#
+# In a hybrid run with CONTINUE_RUN = FALSE (i.e. just starting up):
+#
+# CAM has been forced to read initial files - specified by namelist var:ncdata
+# CICE reads from namelist variable 'ice_ic'
+# CLM builds its own 'finidat' value from the REFCASE variables but in CESM1_1_1
+# it does not use the instance string. There is a patch for clm.buildnml.csh
+# RTM reads from namelist variable 'finidat_rtm', but rtm.buildnml.csh also is buggy.
+#
+# All of these must later on be staged with these same filenames.
+# OR - all these namelists can be changed to match whatever has been staged.
+# MAKE SURE THE STAGING SECTION OF THIS SCRIPT MATCHES THESE VALUES.
+# ==============================================================================
+
+@ inst = 1
+while ($inst <= $num_instances)
+
+ # following the CESM strategy for 'inst_string'
+ set inst_string = `printf _%04d $inst`
+
+ # ===========================================================================
+ set fname = "user_nl_cam${inst_string}"
+ # ===========================================================================
+ # For a HOP TEST ... empty_htapes = .false.
+ # For a HOP TEST ... use a default fincl1
+ # inithist == 'ENDOFRUN' ensures that CAM writes an initial file every time it stops.
+ # fincl1,nhtfrq,mfilt all control the history file containing a REQUIRED PHIS field.
+
+ echo " inithist = 'ENDOFRUN'" >> ${fname}
+ echo " ncdata = 'cam_initial${inst_string}.nc'" >> ${fname}
+ echo " empty_htapes = .true. " >> ${fname}
+ echo " fincl1 = 'PHIS:I' " >> ${fname}
+ echo " nhtfrq = -$stop_n " >> ${fname}
+ echo " mfilt = 1 " >> ${fname}
+
+ # A lot of the files specified here are because the 'default' files only
+ # contain data through 2005 and we are interested in timeframes after that.
+
+ set chem_datapath = "${cesm_datadir}/atm/cam/chem/trop_mozart_aero"
+
+ # CAM5 does prognostic aerosols by default. If you want to prescribe them,
+ # use the following variables with your own settings ...
+ # Filenames with 'clim' in their names should specify aerodep_flx_type = 'CYCLICAL'.
+ # Other values are 'CYCLICAL', 'SERIAL', 'INTERP_MISSING_MONTHS', or 'FIXED'.
+
+ echo "aerodep_flx_datapath = '${chem_datapath}/aero'" >> ${fname}
+ echo "aerodep_flx_file = 'aerosoldep_rcp4.5_monthly_1849-2104_0.9x1.25_c100407.nc'" >> ${fname}
+ echo "aerodep_flx_type = 'CYCLICAL' " >> ${fname}
+ # Set if aerodep_flx_type = 'CYCLICAL'
+ echo "aerodep_flx_cycle_yr = 2000 " >> ${fname}
+
+ # ===========================================================================
+ set fname = "user_nl_clm${inst_string}"
+ # ===========================================================================
+
+ # Customize the land namelists
+ # The filename is built using the REFCASE/REFDATE/REFTOD information.
+ # Making a compact .h0. file is a good idea, since the clm restart files
+ # do not have all the metadata required to reconstruct a gridded field.
+ # 'TSA' is 2m surface air temperature.
+ # TJH ... should make monthly average file or something ...
+ # 1.3 MB is 100x bigger than CAM history file. See page 65 of
+ # http://www.cesm.ucar.edu/models/cesm1.1/clm/models/lnd/clm/doc/UsersGuide/clm_ug.pdf
+ #
+ # Every 6 hours
+ # echo "hist_mfilt = 1" >> $fname
+ # echo "hist_nhtfrq = -$stop_n" >> $fname
+ # Every month
+ # echo "hist_mfilt = 1" >> $fname
+ # echo "hist_nhtfrq = 0" >> $fname
+
+# KDR; Was the ntapes bug fixed?
+# That's where the CLM restart file always has dimension ntapes,
+# but it's set = 0 for hist_empty_htapes = .false.,
+# which prevents the next forecast from starting.
+# Kluzek fixed it in some recent (<9/2013) version, but not 1_1_1.
+# There's a supposed fix in ~/cesm1_1_1/SourceMods/src.clm/HistFileMod.F90,
+# but it only checks whether ntapes = ntapes_on_file, and is_restart() returning 'true'.
+# It doesn't check whether the size is 0.
+# That solution can't work in hybrid mode. See ~/Homme/notes:9/17/2013
+# KDR
+
+ echo "hist_empty_htapes = .false." >> $fname
+ echo "hist_fincl1 = 'TSA'" >> $fname
+ echo "hist_nhtfrq = 0" >> $fname
+ echo "hist_mfilt = 1" >> $fname
+ echo "hist_avgflag_pertape = 'A'" >> $fname
+
+ # ===========================================================================
+ set fname = "user_nl_cice${inst_string}"
+ # ===========================================================================
+ # CICE Namelists
+
+ echo "ice_ic = '${run_refcase}.cice${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> $fname
+
+ # ===========================================================================
+ # TJH FIXME ... do we need this if we turn off the river runoff model?
+ set fname = "user_nl_rtm${inst_string}"
+ # ===========================================================================
+ # RIVER RUNOFF CAN START FROM AN OLD CLM RESTART FILE
+ # you can specify the RTM filename here and override the settings from
+ # RUN_REFCASE/RUN_REFDATE/RUN_REFTOD (something you cannot do with CLM).
+
+ echo "finidat_rtm = '${run_refcase}.rtm${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> $fname
+
+ @ inst ++
+end
+
+./preview_namelists
+
+# ==============================================================================
+# Stage the restarts now that the run directory exists
+# THIS IS THE STAGING SECTION - MAKE SURE THIS MATCHES THE NAMELISTS.
+# ==============================================================================
+
+cat << EndOfText >! stage_cesm_files
+#!/bin/csh -f
+# This script can be used to help restart an experiment from any previous step.
+# The files are copied to the RUN directory.
+# Set the 'restart_time' variable below to define when to restart the experiment.
+# The value of the CESM variable 'env_run.xml:CONTINUE_RUN' is used to determine
+# the origin of the files.
+
+echo 'Copying the required files to the run directory.'
+echo 'No matter what, make sure the restart_time is set to the desired time.'
+echo ''
+echo 'With CONTINUE_RUN=FALSE, the files are copied from the initial ensemble directory.'
+echo 'When CONTINUE_RUN=TRUE, the files are copied from the short-term archive restart directory.'
+echo 'If the short-term archiver is OFF, all the files are already in the RUN directory and'
+echo 'just the pointer files need to be updated.'
+echo ''
+
+cd ${caseroot}
+source ./Tools/ccsm_getenv || exit -2
+cd ${RUNDIR}
+
+set restart_time = $RUN_REFDATE-$RUN_REFTOD
+
+#----------------------------------------------------------------------
+# This block copies over a set of restart files from
+# this experiment and exits. Intended use to restart a failed run.
+#----------------------------------------------------------------------
+
+echo "continue_run is \$CONTINUE_RUN"
+
+if ( \$CONTINUE_RUN == TRUE ) then
+ if ( \$DOUT_S == TRUE ) then
+
+ # the restarts should be in the short term archive directory
+ # http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x1631.html#running_ccsm_restart_back
+ ${COPY} \$DOUT_S_ROOT/rest/\${restart_time}/* .
+
+ else
+
+ # All the required files must still be in the run directory.
+ # Just need to update the pointer files.
+
+ @ inst=1
+ while (\$inst <= $num_instances)
+
+ set inst_string = \`printf _%04d \$inst\`
+
+ echo "${CASE}.cpl.r.\${restart_time}.nc" >! rpointer.drv
+ echo "${CASE}.clm2\${inst_string}.r.\${restart_time}.nc" >! rpointer.lnd\${inst_string}
+ echo "${CASE}.cice\${inst_string}.r.\${restart_time}.nc" >! rpointer.ice\${inst_string}
+ echo "${CASE}.rtm\${inst_string}.r.\${restart_time}.nc" >! rpointer.rof\${inst_string}
+ echo "${CASE}.cam\${inst_string}.r.\${restart_time}.nc" >! rpointer.atm\${inst_string}
+
+
+ @ inst ++
+ end
+
+ endif
+
+ # Now that we're guaranteed to have the cam initial file, link to our hardwired name.
+ @ inst=1
+ while (\$inst <= $num_instances)
+ set inst_string = \`printf _%04d \$inst\`
+ ${LINK} ${CASE}.cam\${inst_string}.i.\${restart_time}.nc cam_initial\${inst_string}.nc
+ @ inst ++
+ end
+
+ # relink the pointer file for the data ocean
+ echo "${CASE}.docn.r.\${restart_time}.nc" >! rpointer.ocn
+ echo "" >> rpointer.ocn
+ echo "${CASE}.docn.rs1.\${restart_time}.bin" >> rpointer.ocn
+ echo "" >> rpointer.ocn
+
+ exit 0
+endif
+
+#----------------------------------------------------------------------
+# This block prepares the initial run directory for a startup run.
+# The CLM, CICE, ROF namelists specify a relative path for the initial file.
+# The CAM namelist specifies a static name.
+# POP uses pointer files.
+#
+# The reference case may more instances than we need,
+# and it certainly has more files than we need.
+# This stages only the minimum number of instances and files.
+#
+# Remember - in a hybrid case - when CONTINUE_RUN = FALSE,
+# the run_refdate and the time in the filenames can be unrelated.
+#----------------------------------------------------------------------
+
+set filedir = ${stagedir}
+
+@ inst=1
+while (\$inst <= $num_instances)
+
+ set inst_string = \`printf _%04d \$inst\`
+
+ echo ''
+ echo "Staging restarts for instance \$inst of $num_instances"
+
+ ${LINK} \${filedir}/${run_refcase}.clm2\${inst_string}.r.${run_refdate}-${run_reftod}.nc .
+ ${LINK} \${filedir}/${run_refcase}.cice\${inst_string}.r.${run_refdate}-${run_reftod}.nc .
+ ${LINK} \${filedir}/${run_refcase}.cam\${inst_string}.i.${run_refdate}-${run_reftod}.nc cam_initial\${inst_string}.nc
+
+ @ inst ++
+end
+
+exit 0
+
+EndOfText
+chmod 0755 stage_cesm_files
+
+./stage_cesm_files
+
+# ==============================================================================
+# build
+# ==============================================================================
+
+echo ''
+echo 'Building the case'
+echo ''
+
+./${case}.build
+
+if ( $status != 0 ) then
+ echo "ERROR: Case could not be built."
+ exit -5
+endif
+
+# ==============================================================================
+# What to do next
+# ==============================================================================
+
+echo ""
+echo "Time to check the case."
+echo ""
+echo "1) cd ${rundir}"
+echo " and check the compatibility between the namelists/pointer"
+echo " files and the files that were staged."
+echo ""
+echo "2) cd ${caseroot}"
+echo " (on yellowstone) If the ${case}.run script still contains:"
+echo ' #BSUB -R "select[scratch_ok > 0]"'
+echo " around line 9, delete it."
+echo ""
+echo "3) If you want to simply advance CESM without using DART, make sure the"
+echo " ${case}.run script references the 'no_assimilate.csh' script."
+echo " If you want to assimilate, reference the 'assimilate.csh' script."
+echo " The default action is to assimilate."
+echo ""
+echo "4) Verify the contents of env_run.xml and submit the CESM job:"
+echo " ./${case}.submit"
+echo ""
+echo "5) After the job has run, check to make sure it worked and that"
+echo " a: the right restart files exist in the run directory,"
+echo " b: (if you're running DART) the archive dart/hist directory has the DART output,"
+echo " c: everything is working correctly ..."
+echo ""
+echo "6) To extend the run in $stop_n '"$stop_option"' steps,"
+echo " change the env_run.xml variables:"
+echo ""
+echo " ./xmlchange CONTINUE_RUN=TRUE"
+echo " ./xmlchange RESUBMIT=<number_of_cycles_to_run>"
+echo " ./xmlchange STOP_N=$stop_n"
+echo ""
+echo " and"
+echo " ./${case}.submit"
+echo ""
+
+exit 0
+
+# <next few lines under version control, do not edit>
+# $URL: https://subversion.ucar.edu/DAReS/DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid $
+# $Revision: 6396 $
+# $Date: 2013-08-12 11:51:26 -0600 (Mon, 12 Aug 2013) $
+
Property changes on: DART/trunk/models/cam/shell_scripts/CESM1_1_1_hybrid
___________________________________________________________________
Added: svn:executable
+ *
More information about the Dart-dev
mailing list