[Dart-dev] [6563] DART/trunk/models/CESM/shell_scripts: Adding setup_pmo script, best practices for all other scripting.
nancy at ucar.edu
nancy at ucar.edu
Fri Nov 1 16:21:17 MDT 2013
Revision: 6563
Author: thoar
Date: 2013-11-01 16:21:16 -0600 (Fri, 01 Nov 2013)
Log Message:
-----------
Adding setup_pmo script, best practices for all other scripting.
Modified Paths:
--------------
DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid
DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid_initial
DART/trunk/models/CESM/shell_scripts/CESM_DART_config
DART/trunk/models/CESM/shell_scripts/st_archive.sh
Added Paths:
-----------
DART/trunk/models/CESM/shell_scripts/CESM1_1_1_setup_pmo
-------------- next part --------------
Modified: DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid
===================================================================
--- DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid 2013-11-01 19:06:27 UTC (rev 6562)
+++ DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid 2013-11-01 22:21:16 UTC (rev 6563)
@@ -27,13 +27,13 @@
# capability to re-stage files in your RUN directory
#
# CESM/DART requires some modifications to the CESM source code EVEN IF YOU ARE NOT
-# assimilating. The modifications for CAM require a modification to the
-# CESM ${CASE}.run script. The modifications will invoke a DART script that will
-# either assimilate or 'do nothing'. It is stronly recommended that you configure
+# assimilating. The modifications for CAM require a modification to the
+# CESM ${CASE}.run script. The modifications will invoke a DART script that will
+# either assimilate or 'do nothing'. It is strongly recommended that you configure
# the ${CASE}.run script to 'do nothing' initially.
#
# CASEROOT/CESM_DART_config will augment the CESM case with the required setup
-# and configuration to perform a DART assimilation and modify the ${CASE}.run script
+# and configuration to perform a DART assimilation and modify the ${CASE}.run script
# as mentioned previously - after it makes a backup copy.
#
# This script relies heavily on the information in:
@@ -48,21 +48,24 @@
# This script sets up a CESM multi-instance run as we understand them and
# it has almost nothing to do with DART. This is intentional.
#
-# -- Edit and run this script in the $DART/models/CESM/shell_scripts directory
-# or copy it to somewhere that it will be preserved and run it there.
-# It will create a CESM 'CASE' directory, where the model will be built,
-# and an execution directory, where each forecast (and assimilation) will
-# take place. The short term archiver will use a third directory for
-# storage of model output until it can be moved to long term storage (HPSS)
-#
# -- Examine the whole script to identify things to change for your experiments.
#
-# -- Provide the CESM initial ensemble needed by your run.
+# -- Edit this script in the $DART/models/CESM/shell_scripts directory
+# or copy it to somewhere where it will be preserved.
#
-# -- Run this script.
+# -- Run this script. When it is executed, it will create:
+# 1) a CESM 'CASE' directory, where the model will be built,
+# 2) a run directory, where each forecast (and assimilation) will take place,
+# 3) a bld directory for the executables.
#
-# -- Read, understand, and execute ${CASEROOT}/CESM_DART_config
+# This script also executes ${CASEROOT}/CESM_DART_config to modify
+# the ${CASEROOT}/${CASE}.run script so that the SourceMods for CAM
+# are effective. CESM_DART_config will also augment the case with all
+# the pieces necessary to run DART when the time comes.
#
+# -- (if running DART) Edit the DART input.nml that appears in the ${CASEROOT}
+# directory.
+#
# -- Submit the job using ${CASEROOT}/${CASE}.submit
#
# ---------------------
@@ -87,7 +90,7 @@
# the value of "case" will be used many ways;
# directory and file names, both locally and on HPSS, and
-# script names; so consider it's length and information content.
+# script names; so consider its length and information content.
# num_instances: Number of ensemble members
setenv case cesm_6h_hybrid
@@ -102,7 +105,7 @@
# mach Computer name
# cesmroot Location of the cesm code base
# For cesm1_1_1 on yellowstone
-# caseroot Your (future) cesm case directory, where this CESM+DART will be built.
+# caseroot Your (future) CESM case directory, where this CESM+DART will be built.
# Preferably not a frequently scrubbed location.
# This script will delete any existing caseroot, so this script,
# and other useful things should be kept elsewhere.
@@ -115,6 +118,7 @@
# ==============================================================================
setenv mach yellowstone
+
setenv cesmroot /glade/p/cesm/cseg/collections/$cesmtag
setenv caseroot /glade/p/work/${USER}/cases/${case}
setenv exeroot /glade/scratch/${USER}/${case}/bld
@@ -137,7 +141,7 @@
# THIS IS THE LOCATION of the 'reference case'.
-setenv stagedir /glade/p/image/CESM_initial_ensemble/rest/2004-01-10-00000
+setenv stagedir /glade/p/image/CESM_initial_ensemble/rest/${run_refdate}-${run_reftod}
# ==============================================================================
# runtime settings
@@ -146,9 +150,13 @@
# stop_option Units for determining the forecast length between assimilations
# stop_n Number of time units in the first forecast
# assim_n Number of time units between assimilations
+# clm_dtime CLM dynamical timestep (in seconds) ... 1800 is the default
+# h1nsteps is the number of time steps to put in a single CLM .h1. file
+# DART needs to know this and the only time it is known is during
+# this configuration step. The run-time settings can lie.
#
# If the long-term archiver is off, you get a chance to examine the files before
-# they get moved to long-term storage. You can always submit $CASE.l_archive
+# they get moved to long-term storage. You can always submit $case.l_archive
# whenever you want to free up space in the short-term archive directory.
#
# ==============================================================================
@@ -160,6 +168,13 @@
setenv stop_n 12
setenv assim_n 6
+# This specifies the number of time steps available for a CLM forward obs. operator.
+# with clm_dtime = 1800 and h1nsteps = $assim_n * 3600 / $clm_dtime, the CLM .h1.
+# file is created with values every 30 minutes.
+
+@ clm_dtime = 1800
+@ h1nsteps = $assim_n * 3600 / $clm_dtime
+
# ==============================================================================
# job settings
#
@@ -219,8 +234,8 @@
# ==============================================================================
# Create the case - this creates the CASEROOT directory.
#
-# For list of the pre-defined cases: ./create_newcase -list
-# To create a variant case, see the CESM documentation and carefully
+# For list of the pre-defined component sets: ./create_newcase -list
+# To create a variant compset, see the CESM documentation and carefully
# incorporate any needed changes into this script.
# ==============================================================================
@@ -300,22 +315,22 @@
# This is only for the purpose of debugging the code.
@ atm_tasks = $ptile * $num_instances * 4
@ lnd_tasks = $ptile * $num_instances * 4
- @ ice_tasks = $ptile * $num_instances
+ @ ice_tasks = $ptile * $num_instances * 4
@ ocn_tasks = $ptile * $num_instances * 4
@ cpl_tasks = $ptile * $num_instances
@ glc_tasks = $ptile * $num_instances
- @ rof_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances * 4
else
# This works, but a more efficient layout should be used
@ atm_tasks = $ptile * $num_instances * 2
@ lnd_tasks = $ptile * $num_instances * 2
- @ ice_tasks = $ptile * $num_instances
+ @ ice_tasks = $ptile * $num_instances * 2
@ ocn_tasks = $ptile * $num_instances * 2
@ cpl_tasks = $ptile * $num_instances
@ glc_tasks = $ptile * $num_instances
- @ rof_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances * 2
endif
@@ -471,7 +486,7 @@
endsw
# This is the part that modifies the run script to allow CESM to advance
-# correctly given the modifications necessary for CAM. It also copies several
+# correctly given the modifications of CAM for DART. It also copies several
# required DART files to the caseroot directory.
./CESM_DART_config
@@ -491,7 +506,7 @@
echo "ERROR - No SourceMods for this case."
echo "DART requires modifications to several src files."
echo "These files can be downloaded from:"
- echo "http://www.image.ucar.edu/pub/DART/CESM/DART_SourceMods_cesm1_1_1.tar"
+ echo "http://www.image.ucar.edu/pub/DART/CESM/DART_SourceMods_cesm1_1_1_24Oct2013.tar"
echo "untar these into your HOME directory - they will create a"
echo "~/cesm_1_1_1 directory with the appropriate SourceMods structure."
exit -4
@@ -507,15 +522,15 @@
# preserve the original version of the files
if ( ! -e ${caseroot}/Buildconf/clm.buildnml.csh.original ) then
- ${MOVE} ${caseroot}/Buildconf/clm.buildnml.csh \
+ ${COPY} ${caseroot}/Buildconf/clm.buildnml.csh \
${caseroot}/Buildconf/clm.buildnml.csh.original
endif
if ( ! -e ${caseroot}/Buildconf/rtm.buildnml.csh.original ) then
- ${MOVE} ${caseroot}/Buildconf/rtm.buildnml.csh \
+ ${COPY} ${caseroot}/Buildconf/rtm.buildnml.csh \
${caseroot}/Buildconf/rtm.buildnml.csh.original
endif
if ( ! -e ${caseroot}/preview_namelists.original ) then
- ${MOVE} ${caseroot}/preview_namelists \
+ ${COPY} ${caseroot}/preview_namelists \
${caseroot}/preview_namelists.original
endif
@@ -527,11 +542,10 @@
endif
# ==============================================================================
-# Modify namelist templates for each instance. This is a bit of a nuisance in
-# that we are pulling in restart and initial files from 'all over the place'
-# and each model component has a different strategy.
+# Modify namelist templates for each instance.
+# CAM SourceMods force CAM to ALWAYS read INITIAL files specified by namelist var 'ncdata'
#
-# In a hybrid run with CONTINUE_RUN = FALSE (i.e. just starting up):
+# When CONTINUE_RUN = FALSE (i.e. just starting up):
#
# CAM has been forced to read initial files - specified by namelist var:ncdata
# POP reads from pointer files
@@ -578,6 +592,7 @@
# then it is required to create a .h1. file with the instantaneous flux
# variables every 30 minutes. Despite being in a namelist, these values
# HAVE NO EFFECT once CONTINUE_RUN = TRUE so now is the time to set these.
+ # Check the value of h1nsteps considering STOP_N.
#
# DART's forward observation operators for these fluxes just reads them
# from the .h1. file rather than trying to create them from the subset of
@@ -588,15 +603,14 @@
#
# FIXME ... add documentation for configuring CLM history files
- @ thirtymin = $assim_n * 2
+ echo "dtime = $clm_dtime," >> ${fname}
+ echo "hist_empty_htapes = .true." >> ${fname}
+ echo "hist_fincl1 = 'NEP'" >> ${fname}
+ echo "hist_fincl2 = 'NEP','FSH','EFLX_LH_TOT_R'" >> ${fname}
+ echo "hist_nhtfrq = -$assim_n,1," >> ${fname}
+ echo "hist_mfilt = 1,$h1nsteps" >> ${fname}
+ echo "hist_avgflag_pertape = 'A','A'" >> ${fname}
- echo "hist_empty_htapes = .true." >> $fname
- echo "hist_fincl1 = 'NEP'" >> $fname
- echo "hist_fincl2 = 'NEP','FSH','EFLX_LH_TOT_R'" >> $fname
- echo "hist_nhtfrq = -$assim_n,1," >> $fname
- echo "hist_mfilt = 1,$thirtymin" >> $fname
- echo "hist_avgflag_pertape = 'A','A'" >> $fname
-
# ===========================================================================
set fname = "user_nl_pop2${inst_string}"
# ===========================================================================
@@ -627,14 +641,14 @@
# Default copies of SourceMods/src.pop2/ocn.*.tavg.csh files are provided in the
# DART_SourceMods_cesm1_1_1.tar bundle.
- echo "init_ts_suboption = 'data_assim'" >> $fname
+ echo "init_ts_suboption = 'data_assim'" >> ${fname}
# ===========================================================================
set fname = "user_nl_cice${inst_string}"
# ===========================================================================
# CICE Namelists
- echo "ice_ic = '${run_refcase}.cice${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> $fname
+ echo "ice_ic = '${run_refcase}.cice${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> ${fname}
# ===========================================================================
set fname = "user_nl_rtm${inst_string}"
@@ -643,7 +657,7 @@
# you can specify the RTM filename here and override the settings from
# RUN_REFCASE/RUN_REFDATE/RUN_REFTOD (something you cannot do with CLM).
- echo "finidat_rtm = '${run_refcase}.rtm${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> $fname
+ echo "finidat_rtm = '${run_refcase}.rtm${inst_string}.r.${run_refdate}-${run_reftod}.nc'" >> ${fname}
@ inst ++
end
@@ -652,8 +666,6 @@
# ==============================================================================
# Stage the restarts now that the run directory exists
-# THIS IS THE STAGING SECTION - MAKE SURE THIS MATCHES THE NAMELISTS.
-# POP reads from pointer files. The others use namelist values initially.
# ==============================================================================
cat << EndOfText >! stage_cesm_files
@@ -690,9 +702,10 @@
if ( \$DOUT_S == TRUE ) then
# the restarts should be in the short term archive directory
- # http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x1631.html#running_ccsm_restart_back
- ${COPY} \$DOUT_S_ROOT/rest/\${restart_time}/* .
+ # www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x1631.html#running_ccsm_restart_back
+ ${COPY} \$DOUT_S_ROOT/rest/\${restart_time}/* . || exit -1
+
else
# All the required files must still be in the run directory.
@@ -703,7 +716,6 @@
set inst_string = \`printf _%04d \$inst\`
- echo "${CASE}.cpl.r.\${restart_time}.nc" >! rpointer.drv
echo "${CASE}.clm2\${inst_string}.r.\${restart_time}.nc" >! rpointer.lnd\${inst_string}
echo "${CASE}.cice\${inst_string}.r.\${restart_time}.nc" >! rpointer.ice\${inst_string}
echo "${CASE}.rtm\${inst_string}.r.\${restart_time}.nc" >! rpointer.rof\${inst_string}
@@ -715,14 +727,17 @@
if ( -e rpointer.ocn\${inst_string}.tavg ) then
echo "${CASE}.pop\${inst_string}.rh.\${restart_time}.nc" >! rpointer.ocn\${inst_string}.tavg
- endif
+ endif
if ( -e rpointer.ocn\${inst_string}.tavg.2 ) then
echo "${CASE}.pop\${inst_string}.rh.nday1.\${restart_time}.nc" >! rpointer.ocn\${inst_string}.tavg.2
- endif
+ endif
@ inst ++
end
+ # This has nothing to do with instances.
+ echo "${CASE}.cpl.r.\${restart_time}.nc" >! rpointer.drv
+
endif
# Now that we're guaranteed to have the cam initial file, link to our hardwired name.
@@ -737,12 +752,13 @@
endif
#----------------------------------------------------------------------
-# This block prepares the initial run directory for a startup run.
+# THIS IS THE INITIAL STAGING SECTION.
+# This block prepares the run directory for the first advance.
# The CLM, CICE, ROF namelists specify a relative path for the initial file.
# The CAM namelist specifies a static name.
-# POP uses pointer files.
+# POP uses pointer files that reference a local filename.
#
-# The reference case may more instances than we need,
+# The reference case may contain more instances than we need,
# and it certainly has more files than we need.
# This stages only the minimum number of instances and files.
#
@@ -750,8 +766,6 @@
# the run_refdate and the time in the filenames can be unrelated.
#----------------------------------------------------------------------
-set filedir = ${stagedir}
-
@ inst=1
while (\$inst <= $num_instances)
@@ -760,12 +774,12 @@
echo ''
echo "Staging restarts for instance \$inst of $num_instances"
- ${LINK} \${filedir}/${run_refcase}.clm2\${inst_string}.r.\${restart_time}.nc .
- ${LINK} \${filedir}/${run_refcase}.cice\${inst_string}.r.\${restart_time}.nc .
- ${LINK} \${filedir}/${run_refcase}.rtm\${inst_string}.r.\${restart_time}.nc .
- ${LINK} \${filedir}/${run_refcase}.cam\${inst_string}.i.\${restart_time}.nc cam_initial\${inst_string}.nc
- ${LINK} \${filedir}/${run_refcase}.pop\${inst_string}.r.\${restart_time}.nc .
- ${LINK} \${filedir}/${run_refcase}.pop\${inst_string}.ro.\${restart_time} .
+ ${LINK} ${stagedir}/${run_refcase}.clm2\${inst_string}.r.\${restart_time}.nc .
+ ${LINK} ${stagedir}/${run_refcase}.cice\${inst_string}.r.\${restart_time}.nc .
+ ${LINK} ${stagedir}/${run_refcase}.rtm\${inst_string}.r.\${restart_time}.nc .
+ ${LINK} ${stagedir}/${run_refcase}.cam\${inst_string}.i.\${restart_time}.nc cam_initial\${inst_string}.nc
+ ${LINK} ${stagedir}/${run_refcase}.pop\${inst_string}.r.\${restart_time}.nc .
+ ${LINK} ${stagedir}/${run_refcase}.pop\${inst_string}.ro.\${restart_time} .
echo "${run_refcase}.pop\${inst_string}.ro.\${restart_time}" >! rpointer.ocn\${inst_string}.ovf
echo "${run_refcase}.pop\${inst_string}.r.\${restart_time}.nc" >! rpointer.ocn\${inst_string}.restart
@@ -812,26 +826,21 @@
echo ' #BSUB -R "select[scratch_ok > 0]"'
echo " around line 9, delete it."
echo ""
-echo "3) For reasons not clear to me, the very first advance (i.e. STOP_N)"
-echo " (CONTINUE_RUN=FALSE) must be longer than *AT LEAST 2 TIMES* the coupling"
+echo "3) The very first CESM advance (i.e. CONTINUE_RUN=FALSE)"
+echo " STOP_N must be longer than *AT LEAST 2 TIMES* the coupling"
echo " frequency between the atmosphere and ocean."
echo " If coupling once a day, the first advance MUST be at least 48 hours."
echo " If coupling 4 times a day, the first advance MUST be at least 12 hours."
echo " After that, STOP_N can be as short as a single coupling frequency."
echo ""
-echo "4) If you want to simply advance CESM without using DART, make sure the"
-echo " ${case}.run script references the 'no_assimilate.csh' script."
-echo " If you want to assimilate, reference the 'assimilate.csh' script."
-echo " The default action is to assimilate."
+echo "4) The default action is to simply advance CESM ... i.e. NO ASSIMILATION."
+echo " If you want to assimilate, make sure the ${case}.run script"
+echo " references the 'assimilate.csh' script."
echo ""
echo "5) Verify the contents of env_run.xml and submit the CESM job:"
echo " ./${case}.submit"
echo ""
-echo "6) After the job has run, check to make sure it worked and that"
-echo " a: POP is creating netCDF restart files,"
-echo " b: the right restart files exist in the run directory,"
-echo " c: (if you're running DART) the archive dart/hist directory has the DART output,"
-echo " d: everything is working correctly ..."
+echo "6) After the job has run, check to make sure it worked."
echo ""
echo "7) To extend the run in $assim_n '"$stop_option"' steps,"
echo " change the env_run.xml variables:"
@@ -843,6 +852,8 @@
echo " and"
echo " ./${case}.submit"
+cat ${caseroot}/DART_instructions.txt
+
exit 0
# <next few lines under version control, do not edit>
Modified: DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid_initial
===================================================================
--- DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid_initial 2013-11-01 19:06:27 UTC (rev 6562)
+++ DART/trunk/models/CESM/shell_scripts/CESM1_1_1_hybrid_initial 2013-11-01 22:21:16 UTC (rev 6563)
@@ -50,13 +50,15 @@
# data volume is quite large and you should become comfortable using CESM's restart
# capability to re-stage files in your RUN directory
#
+# CESM/DART requires some modifications to the CESM source code EVEN IF YOU ARE NOT
+# assimilating. The modifications for CAM require a modification to the
+# CESM ${CASE}.run script. The modifications will invoke a DART script that will
+# either assimilate or 'do nothing'. It is strongly recommended that you configure
+# the ${CASE}.run script to 'do nothing' initially.
+#
# CASEROOT/CESM_DART_config will augment the CESM case with the required setup
-# and configuration to perform a DART assimilation. CASEROOT/CESM_DART_config
-# will insert a few dozen lines into the ${CASE}.run script after it makes a backup
-# copy. This, and the required setup, can be run at a later date. e.g. you can
-# advance an ensemble from 2004-01-01 to 2004-02-01 and then run
-# CESM_DART_config to augment the existing run script, modify STOP_N to 6 hours,
-# and start assimilating observations when CESM stops at 2004-02-01 06Z ...
+# and configuration to perform a DART assimilation and modify the ${CASE}.run script
+# as mentioned previously - after it makes a backup copy.
#
# This script relies heavily on the information in:
# http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/book1.html
@@ -70,21 +72,24 @@
# This script sets up a CESM multi-instance run as we understand them and
# it has almost nothing to do with DART. This is intentional.
#
-# -- Edit and run this script in the $DART/models/CESM/shell_scripts directory
-# or copy it to somewhere that it will be preserved and run it there.
-# It will create a CESM 'CASE' directory, where the model will be built,
-# and an execution directory, where each forecast (and assimilation) will
-# take place. The short term archiver will use a third directory for
-# storage of model output until it can be moved to long term storage (HPSS)
-#
# -- Examine the whole script to identify things to change for your experiments.
#
-# -- Provide the CESM initial ensemble needed by your run.
+# -- Edit this script in the $DART/models/CESM/shell_scripts directory
+# or copy it to somewhere where it will be preserved.
#
-# -- Run this script.
+# -- Run this script. When it is executed, it will create:
+# 1) a CESM 'CASE' directory, where the model will be built,
+# 2) a run directory, where each forecast (and assimilation) will take place,
+# 3) a bld directory for the executables.
#
-# -- If you want to run DART; read, understand, and execute ${CASEROOT}/CESM_DART_config
+# This script also executes ${CASEROOT}/CESM_DART_config to modify
+# the ${CASEROOT}/${CASE}.run script so that the SourceMods for CAM
+# are effective. CESM_DART_config will also augment the case with all
+# the pieces necessary to run DART when the time comes.
#
+# -- (if running DART) Edit the DART input.nml that appears in the ${CASEROOT}
+# directory.
+#
# -- Submit the job using ${CASEROOT}/${CASE}.submit
#
# ---------------------
@@ -109,7 +114,7 @@
# the value of "case" will be used many ways;
# directory and file names, both locally and on HPSS, and
-# script names; so consider it's length and information content.
+# script names; so consider its length and information content.
# num_instances: Number of ensemble members
setenv case cesm_multi
@@ -124,7 +129,7 @@
# mach Computer name
# cesmroot Location of the cesm code base
# For cesm1_1_1 on yellowstone
-# caseroot Your (future) cesm case directory, where this CESM+DART will be built.
+# caseroot Your (future) CESM case directory, where this CESM+DART will be built.
# Preferably not a frequently scrubbed location.
# This script will delete any existing caseroot, so this script,
# and other useful things should be kept elsewhere.
@@ -137,6 +142,7 @@
# ==============================================================================
setenv mach yellowstone
+
setenv cesmroot /glade/p/cesm/cseg/collections/$cesmtag
setenv caseroot /glade/p/work/${USER}/cases/${case}
setenv exeroot /glade/scratch/${USER}/${case}/bld
@@ -146,15 +152,18 @@
# ==============================================================================
# configure settings
+# The reference case has dates in it.
+# For a 'hybrid' start, these may be unrelated to the refyear, refmon, refday.
# ==============================================================================
-setenv run_refcase cesm_test
+setenv run_refcase cesm_spinup
setenv refyear 2004
setenv refmon 01
setenv refday 01
setenv run_reftod 00000
setenv run_refdate $refyear-$refmon-$refday
+# The RTM and CLM directories are hpss:~thoar/DART_CESM_ensemble_2004_01_01.tar
set RTM_stagedir = /glade/scratch/thoar/DART_POP_RESTARTS/2004-01-01-00000
set CLM_stagedir = /glade/scratch/thoar/DART_POP_RESTARTS/CLM_2004-01-01-00000
set CAM_stagedir = /glade/p/cesm/cseg/inputdata/atm/cam/inic/fv
@@ -167,6 +176,10 @@
# stop_option Units for determining the forecast length between assimilations
# stop_n Number of time units in the first forecast
# assim_n Number of time units between assimilations
+# clm_dtime CLM dynamical timestep (in seconds) ... 1800 is the default
+# h1nsteps is the number of time steps to put in a single CLM .h1. file
+# DART needs to know this and the only time it is known is during
+# this configuration step. The run-time settings can lie.
#
# If the long-term archiver is off, you get a chance to examine the files before
# they get moved to long-term storage. You can always submit $CASE.l_archive
@@ -181,6 +194,13 @@
setenv stop_n 72
setenv assim_n 6
+# This specifies the number of time steps available for a CLM forward obs. operator.
+# with clm_dtime = 1800 and h1nsteps = $assim_n * 3600 / $clm_dtime, the CLM .h1.
+# file is created with values every 30 minutes.
+
+@ clm_dtime = 1800
+@ h1nsteps = $assim_n * 3600 / $clm_dtime
+
# ==============================================================================
# job settings
#
@@ -240,8 +260,8 @@
# ==============================================================================
# Create the case - this creates the CASEROOT directory.
#
-# For list of the pre-defined cases: ./create_newcase -list
-# To create a variant case, see the CESM documentation and carefully
+# For list of the pre-defined component sets: ./create_newcase -list
+# To create a variant compset, see the CESM documentation and carefully
# incorporate any needed changes into this script.
# ==============================================================================
@@ -292,11 +312,10 @@
${MOVE} temp.$$ ${caseroot}/CESM_DART_config
chmod 755 ${caseroot}/CESM_DART_config
else
- echo "WARNING: the script to configure for data assimilation is not available."
- echo " CESM_DART_config should be present locally or in"
- echo " ${dartroot}/models/CESM/shell_scripts/"
- echo " You can stage this script later, but you must manually edit it"
- echo " to reflect the location of the DART code tree."
+ echo "ERROR: the script to configure for data assimilation is not available."
+ echo " CESM_DART_config MUST be present locally or in"
+ echo " ${dartroot}/models/CESM/shell_scripts/"
+ exit -2
endif
# ==============================================================================
@@ -322,22 +341,22 @@
# This is only for the purpose of debugging the code.
@ atm_tasks = $ptile * $num_instances * 4
@ lnd_tasks = $ptile * $num_instances * 4
- @ ice_tasks = $ptile * $num_instances
+ @ ice_tasks = $ptile * $num_instances * 4
@ ocn_tasks = $ptile * $num_instances * 4
@ cpl_tasks = $ptile * $num_instances
@ glc_tasks = $ptile * $num_instances
- @ rof_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances * 4
else
# This works, but a more efficient layout should be used
@ atm_tasks = $ptile * $num_instances * 2
@ lnd_tasks = $ptile * $num_instances * 2
- @ ice_tasks = $ptile * $num_instances
+ @ ice_tasks = $ptile * $num_instances * 2
@ ocn_tasks = $ptile * $num_instances * 2
@ cpl_tasks = $ptile * $num_instances
@ glc_tasks = $ptile * $num_instances
- @ rof_tasks = $ptile * $num_instances
+ @ rof_tasks = $ptile * $num_instances * 2
endif
@@ -400,6 +419,7 @@
./xmlchange RUN_REFCASE=$run_refcase
./xmlchange RUN_REFDATE=$run_refdate
./xmlchange RUN_REFTOD=$run_reftod
+./xmlchange BRNCH_RETAIN_CASENAME=FALSE
./xmlchange GET_REFCASE=FALSE
./xmlchange EXEROOT=${exeroot}
@@ -412,8 +432,16 @@
./xmlchange PIO_TYPENAME=pnetcdf
-# This sets the ocean coupling time to 6 hours.
+# COUPLING discussion.
+# http://bugs.cgd.ucar.edu/show_bug.cgi?id=1740
+# "In summary, users should ensure that the following is true,
+# ATM_NCPL = LND_NCPL = ICE_NCPL >= ROF_NCPL >= OCN_NCPL"
+#
+# OCN_NCPL == 4 sets the ocean coupling time to 6 hours.
# All related namelist settings are based on this value.
+# OCN_NCPL is # coupling intervals per NCPL_BASE_PERIOD (default 'day')
+
+./xmlchange ROF_NCPL=8
./xmlchange OCN_NCPL=4
# There is no compset that has everything we want,
@@ -483,12 +511,18 @@
breaksw
endsw
+# This is the part that modifies the run script to allow CESM to advance
+# correctly given the modifications of CAM for DART. It also copies several
+# required DART files to the caseroot directory.
+
+./CESM_DART_config
+
# ==============================================================================
# Update source files.
# Ideally, using DART would not require any modifications to the model source.
# Until then, this script accesses sourcemods from a hardwired location.
# If you have additional sourcemods, they will need to be merged into any DART
-# mods and put in the SourceMods subdirectory found in the 'case' directory.
+# mods and put in the SourceMods subdirectory found in the 'caseroot' directory.
# ==============================================================================
if ( -d ~/${cesmtag}/SourceMods ) then
@@ -498,7 +532,7 @@
echo "ERROR - No SourceMods for this case."
echo "DART requires modifications to several src files."
echo "These files can be downloaded from:"
- echo "http://www.image.ucar.edu/pub/DART/CESM/DART_SourceMods_cesm1_1_1.tar"
+ echo "http://www.image.ucar.edu/pub/DART/CESM/DART_SourceMods_cesm1_1_1_24Oct2013.tar"
echo "untar these into your HOME directory - they will create a"
echo "~/cesm_1_1_1 directory with the appropriate SourceMods structure."
exit -4
@@ -514,15 +548,15 @@
# preserve the original version of the files
if ( ! -e ${caseroot}/Buildconf/clm.buildnml.csh.original ) then
- ${MOVE} ${caseroot}/Buildconf/clm.buildnml.csh \
+ ${COPY} ${caseroot}/Buildconf/clm.buildnml.csh \
${caseroot}/Buildconf/clm.buildnml.csh.original
endif
if ( ! -e ${caseroot}/Buildconf/rtm.buildnml.csh.original ) then
- ${MOVE} ${caseroot}/Buildconf/rtm.buildnml.csh \
+ ${COPY} ${caseroot}/Buildconf/rtm.buildnml.csh \
${caseroot}/Buildconf/rtm.buildnml.csh.original
endif
if ( ! -e ${caseroot}/preview_namelists.original ) then
- ${MOVE} ${caseroot}/preview_namelists \
+ ${COPY} ${caseroot}/preview_namelists \
${caseroot}/preview_namelists.original
endif
@@ -538,7 +572,7 @@
# that we are pulling in restart and initial files from 'all over the place'
# and each model component has a different strategy.
#
-# In a hybrid run with CONTINUE_RUN = FALSE (i.e. just starting up):
+# When CONTINUE_RUN = FALSE (i.e. just starting up):
#
# CAM has been forced to read initial files - specified by namelist var:ncdata
# POP reads from pointer files
@@ -566,14 +600,15 @@
# ===========================================================================
# For a HOP TEST ... empty_htapes = .false.
# For a HOP TEST ... use a default fincl1
- # FIXME ... add documentation for configuring CAM history files
+ # inithist == 'ENDOFRUN' ensures that CAM writes an initial file every time it stops.
+ # fincl1,nhtfrq,mfilt all control the history file containing a REQUIRED PHIS field.
- echo " inithist = '6-HOURLY'" >> ${fname}
- echo " ncdata = 'cam_initial${inst_string}.nc'" >> ${fname}
- echo " empty_htapes = .true. " >> ${fname}
- echo " fincl1 = 'PHIS:I' " >> ${fname}
- echo " nhtfrq = -$assim_n " >> ${fname}
- echo " mfilt = 1 " >> ${fname}
+ echo " inithist = 'ENDOFRUN'" >> ${fname}
+ echo " ncdata = 'cam_initial${inst_string}.nc'" >> ${fname}
+ echo " empty_htapes = .true. " >> ${fname}
+ echo " fincl1 = 'PHIS:I' " >> ${fname}
+ echo " nhtfrq = -$assim_n " >> ${fname}
+ echo " mfilt = 1 " >> ${fname}
# ===========================================================================
set fname = "user_nl_clm${inst_string}"
@@ -587,6 +622,7 @@
# then it is required to create a .h1. file with the instantaneous flux
# variables every 30 minutes. Despite being in a namelist, these values
# HAVE NO EFFECT once CONTINUE_RUN = TRUE so now is the time to set these.
+ # Check the value of h1nsteps considering STOP_N.
#
# DART's forward observation operators for these fluxes just reads them
# from the .h1. file rather than trying to create them from the subset of
@@ -597,31 +633,36 @@
#
# FIXME ... add documentation for configuring CLM history files
- @ thirtymin = $assim_n * 2
+ echo "dtime = $clm_dtime," >> ${fname}
+ echo "hist_empty_htapes = .true." >> ${fname}
+ echo "hist_fincl1 = 'NEP'" >> ${fname}
+ echo "hist_fincl2 = 'NEP','FSH','EFLX_LH_TOT_R'" >> ${fname}
+ echo "hist_nhtfrq = -$assim_n,1," >> ${fname}
+ echo "hist_mfilt = 1,$h1nsteps" >> ${fname}
+ echo "hist_avgflag_pertape = 'A','A'" >> ${fname}
- echo "hist_empty_htapes = .true." >> $fname
- echo "hist_fincl1 = 'NEP'" >> $fname
- echo "hist_fincl2 = 'NEP','FSH','EFLX_LH_TOT_R'" >> $fname
- echo "hist_nhtfrq = -$assim_n,1," >> $fname
- echo "hist_mfilt = 1,$thirtymin" >> $fname
- echo "hist_avgflag_pertape = 'A','A'" >> $fname
-
# ===========================================================================
set fname = "user_nl_pop2${inst_string}"
# ===========================================================================
# POP Namelists
- # GIVEN: init_ts_option = 'ccsm_hybrid' when RUN_TYPE=hybrid, then
- #
- # init_ts_suboption = 'data_assim' --> non bit-for-bit restarting (assimilation mode)
+ # init_ts_suboption = 'data_assim' for non bit-for-bit restarting (assimilation mode)
# init_ts_suboption = 'rest' --> default behavior
#
+ # README:
+ # Configuring the contents of the history files for POP is best explained in
+ # the section marked "POP2: TAVG Settings" in the cesm1_1_1 pop2 namelist documentation
+ # http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/modelnl/nl_pop2.html
+ #
+ # and the CESM-specific documentation for the tavg output variables in the pop2
+ # online documentation:
+ # http://www.cesm.ucar.edu/models/cesm1.1/pop2/doc/users/node78.html
# No matter the setting of init_ts_suboption, POP always uses the
# 'data_assim' behavior when RUN_TYPE=hybrid and CONTINUE_RUN=FALSE.
# As soon as CONTINUE_RUN=TRUE, the value becomes important.
#
- # For a HOP TEST ... Would like to have restart files every day, not just for end.
- # For a HOP TEST (untested)... tavg_file_freq_opt = 'nmonth' 'nday' 'once'"
+ # In CESM1_1_1 keep the values for tavg_file_freq_opt and tavg_freq_opt identical.
+ # pop2/trunk_tags/cesm_pop_2_1_20130412 explains the issue.
#
# DEFAULT values for these are:
# tavg_file_freq_opt = 'nmonth' 'nmonth' 'once'
@@ -630,21 +671,25 @@
# The second entry indicates we get a monthly average as it is being created.
# The third entry indicates we get a daily timeslice
#
+ # Default copies of SourceMods/src.pop2/ocn.*.tavg.csh files are provided in the
+ # DART_SourceMods_cesm1_1_1.tar bundle.
+ #
# IFF values for these are:
# tavg_file_freq_opt = 'nmonth' 'never' 'never'
# tavg_freq_opt = 'nmonth' 'never' 'never'
# The first entry indicates we get a monthly average once a month, and thats all we get..
-
- echo "init_ts_suboption = 'data_assim'" >> $fname
- echo "tavg_file_freq_opt = 'nmonth' 'never' 'never'" >> $fname
- echo "tavg_freq_opt = 'nmonth' 'never' 'never'" >> $fname
+ #
+ # For a HOP TEST ... Would like to have restart files every day, not just for end.
+ # For a HOP TEST (untested)... tavg_file_freq_opt = 'nmonth' 'nday' 'once'"
+ echo "init_ts_suboption = 'data_assim'" >> ${fname}
+
# ===========================================================================
set fname = "user_nl_cice${inst_string}"
# ===========================================================================
# CICE Namelists
- echo "ice_ic = 'b40.20th.005_ens${instance2}.cice.r.2004-01-01-00000.nc'" >> $fname
+ echo "ice_ic = 'b40.20th.005_ens${instance2}.cice.r.2004-01-01-00000.nc'" >> ${fname}
# ===========================================================================
set fname = "user_nl_rtm${inst_string}"
@@ -653,7 +698,7 @@
# you can specify the RTM filename here and override the settings from
# RUN_REFCASE/RUN_REFDATE/RUN_REFTOD (something you cannot do with CLM).
- echo "finidat_rtm = 'b40.20th.005_ens${instance2}.clm2.r.2004-01-01-00000.nc'" >> $fname
+ echo "finidat_rtm = 'b40.20th.005_ens${instance2}.clm2.r.2004-01-01-00000.nc'" >> ${fname}
@ inst ++
end
@@ -662,8 +707,6 @@
# ==============================================================================
# Stage the restarts now that the run directory exists
-# THIS IS THE STAGING SECTION - MAKE SURE THIS MATCHES THE NAMELISTS.
-# POP/CAM/CICE read from pointer files. The others use namelist values initially.
# ==============================================================================
cat << EndOfText >! stage_initial_cesm_files
@@ -698,7 +741,7 @@
echo "b40.20th.005_ens\${n2}.cice.r.2004-01-01-00000.nc" >| rpointer.ice_\${n4}
echo "b40.20th.005_ens\${n2}.pop.ro.2004-01-01-00000" >| rpointer.ocn_\${n4}.ovf
echo "b40.20th.005_ens\${n2}.pop.r.2004-01-01-00000" >| rpointer.ocn_\${n4}.restart
- echo "RESTART_FMT=bin" >> rpointer.ocn_\${n4}.restart
+ echo "RESTART_FMT=bin" >> rpointer.ocn_\${n4}.restart
let inst+=1
done
@@ -741,26 +784,38 @@
echo ' #BSUB -R "select[scratch_ok > 0]"'
echo " around line 9, delete it."
echo ""
-echo "3) If you want to assimilate 'right away', configure and execute"
-echo " the ${caseroot}/CESM_DART_config script."
+echo "3) The very first CESM advance (i.e. CONTINUE_RUN=FALSE)"
+echo " STOP_N must be longer than *AT LEAST 2 TIMES* the coupling"
+echo " frequency between the atmosphere and ocean."
+echo " If coupling once a day, the first advance MUST be at least 48 hours."
+echo " If coupling 4 times a day, the first advance MUST be at least 12 hours."
+echo " After that, STOP_N can be as short as a single coupling frequency."
echo ""
-echo "3) Verify the contents of env_run.xml and submit the CESM job:"
+echo "4) The default action is to simply advance CESM ... i.e. NO ASSIMILATION."
+echo " If you want to assimilate, make sure the ${case}.run script"
+echo " references the 'assimilate.csh' script."
+echo ""
+echo "5) Verify the contents of env_run.xml and submit the CESM job:"
echo " ./${case}.submit"
echo ""
-echo "4) After the job has run, check to make sure it worked and that"
+echo "6) After the job has run, check to make sure it worked and that"
echo " a: POP is creating netCDF restart files,"
echo " b: the right restart files exist in the run directory,"
echo " c: (if you're running DART) the archive dart/hist directory has the DART output,"
echo " d: everything is working correctly ..."
echo ""
-echo "5) To extend the run in $assim_n '"$stop_option"' steps,"
+echo "7) To extend the run in $assim_n '"$stop_option"' steps,"
echo " change the env_run.xml variables:"
echo ""
echo " ./xmlchange CONTINUE_RUN=TRUE"
echo " ./xmlchange RESUBMIT=<number_of_cycles_to_run>"
@@ Diff output truncated at 40000 characters. @@
More information about the Dart-dev
mailing list